I am playing around with MapReduce and wanted to ask why the map function takes two inputs (key, value)? Why not just pass along the value?

Specifically, if you look at the word count example on Wikipedia page you will see that the map function is:

function map(String name, String document):
// name: document name
// document: document contents
  for each word w in document:
    emit (w, 1)

However, the function never does anything with the parameter "name". Why even pass it in?

  • $\begingroup$ This is almost a pure programming question (which would be offtopic here, but ontopic on Stack Overflow) but it seems to allow some general answers, so I'll leave it here. $\endgroup$ – Raphael Sep 16 '13 at 7:39

Let me quote the documentation (emphasis mine):

Map You write a map function that runs once for each input value. It returns a collection of name-value pairs which are passed on to the next stage. If there are many, many input values, then this function runs many, many times. The framework divides up the input so that subsets are handled in parallel on multiple instances of your application. A typical map function could count things that occur in each input value that matches some filter.

So there are two immediate uses I can imagine:

  • Use the "chunk" name in the returned names. For example, you might want to distinguish bible_author from fiftyshadesofgrey_author in the result.
  • Identify "subprocesses". If anything goes wrong (and something will go wrong) an error message like "Error processing bible: Illegal character at position 666" is obviously only useful if you include the document name.

And, of course, you may also want to use the name in the actual computation. A general API should cover that case.

  • $\begingroup$ Ah I see, so while it's strictly not needed for the word count algorithm, it is used by the general API and debugging. Got it. At least now I know I'm not missing anything! $\endgroup$ – user1357015 Sep 18 '13 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.