1
$\begingroup$

I have a large data set consisting of ca. 40 categorical data items and a few interval data items (real numbers, less than 5 such items). Most categories should have a lot of values that repeat themselves over and over, and very few that appear very rarely. Some categories are also overcategories of others (like country and city). The outcome of each data is either 1 if the event occurred, or 0 if it did not occur.

The idea is to calibrate a machine learning model or a statistical model that can predict for every given data row the probability that the outcome is 1. The data set I will use will have at least 1 million rows.

Which machine learning approaches and statistical models will perform well on such a task? My initial thoughts are logarithmic regression and support vector machines (with extensions like random forest).

How do I deal with the interval data items? The easiest approach is obviously to convert different ranges into categories, which I think will not be a problem.

What libraries and tools can I use when my data set has a size of 10 GB? I am interested in tools/libraries that include machine learning algorithms but also statistical tools to help me find attributes with significant influence on the outcome. I can code in Java and C++ at the moment. I looked into Root, a data analysis tool from CERN for large data sets, and its machine learning module TMVA, but it can only handle real numbers and integers as input as far as I know.

Improve this question
$\endgroup$
3
  • $\begingroup$ This was also posted on Cross Validated: Event Prediction through Machine Learning. $\endgroup$
    – GEL
    Jul 3, 2013 at 15:05
  • $\begingroup$ Welcome to CS.Stack Exchange. However, please don't cross-post the same question on multiple Stack Exchange sites. That is generally frowned upon. It is especially frowned-upon to post it twice on two different sites at the same time. I suggest you pick one site, and ask the moderators to migrate the other one (by clicking on the "flag" button). $\endgroup$
    – D.W. ♦
    Jul 3, 2013 at 17:19
  • $\begingroup$ Thank your for your answer! I was not sure where the best place for my question was. After consultation with some of my colleagues I will create a vector of 0s and 1s out of each category. So instead of country I have n new attributes for each of the n different countries represented in my dataset and for each sample one of these new attributes is 1 and the rest are 0. With this preprocessing step I can use all standard machine learning algorithms. $\endgroup$
    – Alexander Dür
    Jul 3, 2013 at 17:55

2 Answers 2

Reset to default
2
$\begingroup$

As usual in these kinds of situations, the best way to find out which machine learning approaches will perform best is to try them out and see which ones work best. You can evaluate them using cross-validation.

It's unlikely that anyone here will be able to guess what will work best without knowing your data set. So I recommend that you get to work trying some likely candidates.

The Orange software tool is very convenient for trying multiple different machine learning algorithms on the same data set.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ In statistics and data mining, cross-validation is a technique of evaluating a model's performance given a set of labeled data. The set is partitioned into equal segments and one subset is used as the test set while the other sets are used to train the model. This is repeated with another subset being left out until all the subsets have been used for both training and testing. All of the data is then used to train a model and the model's performance is reported as the average of the models previously created. What you're proposing is more a simple comparison. $\endgroup$
    – Richard D
    Jul 3, 2013 at 12:05
  • $\begingroup$ @RichardD, yup, I know that, and I'm not sure what you are objecting to. I think cross-validation is a perfectly reasonable tool here to measure the performance of each method; then you can compare them, using these measures. $\endgroup$
    – D.W. ♦
    Jul 3, 2013 at 17:18
  • $\begingroup$ My point was cross-validation, as commonly used, refers to the evaluation of a single model. What you are suggesting is a simple comparison. As such, "cross-validation" has a specific meaning in this instance and it wasn't the meaning you were intending in your use of the word. $\endgroup$
    – Richard D
    Jul 3, 2013 at 18:17
  • $\begingroup$ @RichardD, I understand all that, but I don't get why you think my answer is problematic or why you are criticizing my answer. It's perfectly valid to use cross-validation as part of the way you compare models: use cross-validation to evaluate each model, then compare which model did the best. To compare each model, we need a way to estimate how accurate the model is; and I'm suggesting cross-validation is one way you can estimate how accurate each model is. (Your edit actually made my meaning less clear; my original answer implied that cross-validation is just one way to evaluate a model.) $\endgroup$
    – D.W. ♦
    Jul 3, 2013 at 18:59
  • $\begingroup$ Please feel free to change my edit if you feel I obscured your meaning and please read the first line of my answer where I say "I agree with @D.W." Please feel free to edit your own answer if I've mis-interpreted your use of the term "cross-validation." You wouldn't cross-validate models. You would run cross-validation on each model and compare the results. Since Stack Exchange is about preserving information for reference purposes, I wanted to make the distinction clear for others who would read your answer later on. Otherwise, I have no problem with your answer. $\endgroup$
    – Richard D
    Jul 3, 2013 at 20:55
1
$\begingroup$

I agree with @D.W. that you need to try a number of models and compare the results. I would include Naive Bayes as well, but there are many other algorithms that could give you the results you're looking for. You'll want to compare the performance of the model on it's accuracy, as well as precision and recall, in addition to the other measures of a model's performance, such as kappa statistic and F-measure. As far as packages, Weka might also be a good option, especially if you're already familiar with Java. Though it doesn't have ordinal types that would be useful if you're discretizing the ranges into categories, there are ways of compensating for that.

Since you're working with such a large dataset, I'd recommend sampling down to a smaller subset of your data that can be easily run in memory to test the different packages and models. This may give you a better feel for the algorithms and tools that work best, help you troubleshoot any problems with your attributes (such as your categorical attributes and over-categories, as well as your ranges) without also having to deal with memory issues and other issues unrelated to your actual data mining task.

Good luck!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.