In the Map-Reduce implementation, the reducers start when all mappers finish their jobs. Why don't we stream the records from mappers to reducers while they are being processed instead? I think there are two reasons: (i) if we use streaming method, if a reducer fails, all the mappers have to recompute the map function because the results are lost; (ii) if we want to sort the keys in each partition, streaming will have bigger time complexity than waiting for mappers finish task and sort the key. Are these reasons correct and are there any reasons that they didn't use streaming implementation?

  • $\begingroup$ I don't know, but perhaps it is because then one can take advantage of the speed improvements from an optimized shuffle implementation. $\endgroup$ – D.W. May 18 '19 at 18:39
  • $\begingroup$ Can you give more information on that? $\endgroup$ – toantruong May 22 '19 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.