I am reading through chapter 2 of of the free textbook "Mining of Massive Datasets" (http://www.mmds.org/).

On page 28 the following is stated:

"It is reasonable to create one Map task for every chunk of the input file(s), but we may wish to create fewer Reduce tasks. The reason for limiting the number of Reduce tasks is that it is necessary for each Map task to create an intermediate file for each Reduce task, if there are too many Reduce tasks the number of intermediate files explodes."

I am not sure I follow why each Map task creates an intermediate file for ALL Reduce tasks. It seems to me that the Master (being the programmatic element that delegates tasks) would delegate the output of any given Map task to a single Reduce task. Certainly multiple Map outputs can go to the same Reduce task, but it seems wasteful to store a pointer to all of the Reduce tasks. What am I missing?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.