At our company, we've started using Amazon Machine Learning to predict the likelihood of a certain segment of our customers cancelling their subscription.

We only have 500 customers in that segment so we uploaded training data of all our customers who

  • are on a subscription or
  • were on a subscription but cancelled

Amazon tells us there's an AUC score 0.70 which is just ok from what I read.

I'm new to Machine Learning so my question is the following:

Out of the 500 customers that are in the training set, can I load the ones that haven't cancelled into the prediction set to predict the likelihood of them cancelling their subscription in the future? Or can I only use the model for new customers that are not included in the training data?


The most important thing is to avoid "testing on the training set". If you trained a model on a set of 500 users and then evaluate how well it performs by testing it on some of those same users, you're committing the "testing on the training set" sin. The consequences of doing that is that your evaluation results will be misleading: they will over-estimate the effectiveness of your model.

There are various standard methods to avoid this problem. One method is to use a holdout set: you set aside, say, 67% of your data for training and the other 33% for testing. You train a model on the first 70% (the training set), and then evaluate how well your model performs on the other 30% (the testing set). This will then give you a reasonable, unbiased estimate of the performance of your machine learning model. In practice, a test+train division is often not enough; for various reasons, you'll often want three sets, a training set, a validation set, and a testing set. You can read more about that in standard resources.

Another reasonable method is to use cross-validation to estimate the effectiveness of your scheme. Basically, this involves repeatedly dividing up the data, learning a model on one portion of the data and evaluating on another portion, and then averaging the results. This is a reasonable alternative to dividing the data into train+validate+test sets, when you don't have a lot of labelled data -- as in your case.

These comments are all about how to build the model, select parameters for your model, and evaluate the model's performance (predict how effective it will be in practice). Once you've done all that and settled on a model, then you can simply apply the model to new customers to predict their likelihood of cancelling their subscription.

Finally, one last comment: your approach assumes uniformity over time: i.e., that the factors that influence behavior in the past will continue to hold in the same way in the future. This may or may not be realistic. For instance, maybe today some people are more likely to cancel cable subscriptions because of the availability of Netflix/Hulu/etc. whereas ten years ago that wasn't as much of a factor. In many areas, things can change over time. Read about concept drift to learn more about that particular challenge.

If this didn't answer your question, edit the question to clarify in more detail what you are asking about.

  • $\begingroup$ Thanks, very useful. I've updated my post to ask if, besides new customers, I can also use the model to predict the likelihood of existing customers (part of the training data) to cancel their subscriptions in the future. $\endgroup$ – migu Nov 6 '15 at 0:38
  • $\begingroup$ That's given that the factors that influence behavior in the past will continue to hold in the same way in the future. $\endgroup$ – migu Nov 6 '15 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.