algorithm for predicting procces's memory/cpu usage based on other other prior/still running jobs

Question

In a 'pool' with a lot of cpus ,jobs(processes) are running(lots of them) .

Given I have time/resource usage 'samples' from many prior/still running jobs - how would you go about trying to predict the process resource usage ?

Very abstractly , what I think should be done is as follows :

1) New job is submitted and start running and I wait for some time to gather resource usage information/samples

2) find a subset of jobs that their 'behave' (up until that point in running time) 'similarly' - how would u do that ? maybe those who are always in a specific margin and also there average distance is closest ?

3) choose one job from the subset I found at stage 2, and give prediction of resource usage based upon that job..

Are there known/standard way for my objective ? Would love to hear your ideas,direction etc ..

Answer 1

Here are the broad outlines of one general approach:

1. Determine what observable attributes of a job might help you predict the resource usage of the job. You'll have to hypothesize what these might be based on your domain knowledge of the kinds of jobs you run, as well as your knowledge of what attributes of a job can be measured and observed, before the job finishes.

2. Treat that set of observable attributes as the "feature vector" for the job. Each job gets assigned a "feature vector", based on the values of those attributes.

3. Gather training data: gather a data set, from past jobs, where you have measured the attributes of the jobs before they finished running and also measured the actual resource usage of those jobs.

4. Now use some suitable machine learning algorithm to predict the resource usage of new jobs, based on the training set and the attributes of the new job. Select some appropriate supervised learning algorithm. The choice of a specific algorithm will depend on how you expect the resource usage to vary depending on the attributes. (Is it a linear relationship? Smooth curve? Something else?) You might choose linear regression, or CART trees, or something else entirely. There's lots of literature on this -- I'll leave you to study it on your own and get up to speed on standard materials on this subject.

You'll have to work out the details on your own and play with it a bit to see how well it works in your particular setting. I can't tell you whether or not this will work well in the end; that will depend on your particular workload and the types of jobs you are dealing with.