I'm building a system that processes key,value pairs on a continuous basis in a loop, until terminated.

How do I schedule tasks on the leader and make slaves execute them? This is a realm of distributed objects. It will require the leader to connect to every slave via TCP/IP. I got that part.

Then, the leader should deploy tasks to slaves and be notified of progress. It is known as Job Scheduling, Distributed Task Scheduling, etc.

Computer science slides I've seen were too high level, without a working Java example. Can you point me to any example of an isolated, practical implementation? I don't want an all-in-one system like Spark. I need just the distributed job scheduling, as simple as you can. Like when you'd show me a lab-grown heart that lives and beats, without showing me a whole human.

Can you point to any such computer science example, or java code, or describe yourself how the algorithm should look in the leader and in the slave? Some CS experts know this from distributed operating systems classes. I didn't take those.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.