I would like to have an example of a query for database or a dataflow program in spark (or any stream processing system) that is not worth to parallelize at least one of its operators. The insigth that i have is based on "inherently serial problems" (https://en.m.wikipedia.org/wiki/Inherently_serial_problem). Imagine that I have several cameras in an airport and I want to collect images of the cameras in a determined window. I would write my query like this:

source.map(cameraId, [image, timestamp])
.window(SlidingWindow(10 min, 2 min))
.reduce(new CountPeopleReduce())

In this case it is worth to parallelize by the key. How would I create a query like this tha it is not worth to parallelize? Something where the communication between the tasks are more costly, or the merge of the results the the reduce task is more costly.

Edit: I have found this thread (https://scicomp.stackexchange.com/questions/1391/are-there-any-famous-problems-algorithms-in-scientific-computing-that-cannot-be?newreg=e693e7b2c6484367929916877709ce96) and I am trying to give some real world example for this type of problem....



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.