Are there scientific papers that compare distributed systems and big data processing systems?

Big data processing systems can be batch processing systems or streaming big data processing systems.

A batch big data system is a distributed system that:

  • loads data into the system from relational databases, log files or other sources (usually via Apache Sqoop)
  • makes some computations about that data: aggregations and machine learning algorithms to train existing models or to use some models that have already been trained (via Apache Pig or Apache Spark)
  • stores the result in some files or exports the data to a datawarehouse system or a relational database (using Apache Sqoop)

A streaming big data system is a distributed system that:

  • has a streaming mechanism that allows messages/events to be delievered to the system (usually Apache Kafka)
  • has an engine that makes elaborations on those messages, as before elaborations can include training machine learning models or using them (Apache Spark, or Apache Storm or Apache Flink)
  • stores the result of elaborations or exports that result to other systems (using Apache Kafka)

What I have found so far are a lot of scientific papers about distributed systems but they don't mention streaming systems; and a lot of papers mainly empirical or marketing-oriented about big data.

The worst thing is that often, articles about big data don't mention the fact that big data systems are part of distributed systems.

Is there some material about distributed systems that explains also the systems that I have described above?

  • 2
    $\begingroup$ That seems too broad to me. There are many such systems, and many ways they could be compared; what specifically do you want to know? (And even if you told us how you want them to be compared, no reason to expect there to exist a paper that gets at that specific comparison.) Also, "big data processing system" is not a standard term with any accepted meaning, so you'd have to tell us what you mean by that before we could answer the question. Are you able to edit the question to make it more suitable? $\endgroup$
    – D.W.
    Apr 24, 2018 at 21:51
  • $\begingroup$ @D.W. I have edited my question, adding more information about what I mean. $\endgroup$ Apr 25, 2018 at 8:11
  • 1
    $\begingroup$ What do you want to be explained about the systems you've listed? Why does it have to be a paper about distributed systems that does the explaining, rather than a paper specifically about those systems you want explained? $\endgroup$
    – D.W.
    Apr 25, 2018 at 15:46
  • 1
    $\begingroup$ Well that seems beyond what's reasonable to expect in a single answer here. Such a complete taxonomy might not even exist. The variety and diversity of ideas is enormous. I suggest you try to figure out what problem you are trying to solve (what will you do with a taxonomy?) and then ask a more focused question about that. There's certainly plenty of research papers on distributed systems that are doing streaming computation. $\endgroup$
    – D.W.
    May 1, 2018 at 0:16
  • 1
    $\begingroup$ This book is primarily about bringing these two areas together and has great references. $\endgroup$
    – testuser
    Nov 30, 2018 at 17:11


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.