Cloud computing has transformed the landscape of compute operations. Of course, there are still many labs/businesses with local, large-scale compute clusters. For those businesses who keep the hardware, it would be beneficial to the users to have a tool for deciding whether or not a job should be ran locally versus on the cloud. Two important factors would be how much time the user can afford to wait and the funds available to the user.

If we were only optimizing for cost, of course, we'd send every job to the local resources. It gets interesting when you optimize for both cost and time, considering there's a finite amount of machines locally (other users' jobs creates a queue) and the cloud can provide better/faster hardware.

Ultimately, say you have some amount of money and a time frame. After providing this information and a list of jobs (each with varying CPU nodes requested, memory required, estimated job time, etc.), this tool should be able to make a near-optimal decision about which jobs to run locally and which to run on the cloud.

Considering the optimization nature of this problem, Google's OR-Tools library looks great to start with. They provide guides (check left side bar there) to several "problem types". I'm having trouble deciding which category this problem falls into.

For those unable to visit the OR-Tools link, the problem types are:

  • Linear Optimization
  • Integer Optimization
  • Constraint Optimization
  • Assignment
  • Routing
  • Bin Packing
  • Network Flows
  • Scheduling

What problem type category does this best fall into, and what are some ideas on how to formulate said problem?

  • $\begingroup$ This sounds like a non-trivial goal in the generality you present. One challenge is that it is not clear how to predict how long a job will take locally vs how long it will take on a particular cloud platform, or to predict how long it will as a function of which cloud resources it is given. (Running a job locally might well cost more in some cases.) Not every problem falls cleanly into exactly one of those categories so I don't think that asking which category it falls into is going to give you much helpful information. $\endgroup$
    – D.W.
    Jan 28 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.