Logarithmic Loss function in Criteolab comptetion - kaggle

The logarithm of the likelihood function for a Bernoulli random distribution.

In plain English, this error metric is typically used where you have to predict that something is true or false with a probability (likelihood) ranging from definitely true (1) to equally true (0.5) to definitely false(0).

The use of log on the error provides extreme punishments for being both confident and wrong. In the worst possible case, a single prediction that something is definitely true (1) when it is actually false will add infinite to your error score and make every other entry pointless. In Kaggle competitions, predictions are bounded away from the extremes by a small value in order to prevent this.

LogLoss=−1n∑i=1n[yilog(ŷ i)+(1−yi)log(1−ŷ i)]

Classification and Regression Trees : rpart

How to prune a decision tree ?

Prune back the tree to avoid overfitting the data. Typically, you will want to select a tree size that minimizes the cross-validated error, the xerror column printed by printcp( ).

Prune the tree to the desired size using
prune(fit, cp= )

Specifically, use printcp( ) to examine the cross-validated error results, select the complexity parameter associated with minimum error, and place it into the prune( ) function. Alternatively, you can use the code fragment

     fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"]

to automatically select the complexity parameter associated with the smallest cross-validated error. Thanks to HSAUR for this idea.

http://www.mayo.edu/hsr/techrpt/61.pdf
http://www.statmethods.net/advstats/cart.html - classification, regression trees, random forests

What is the difference between bagging, boosting and stacking ?

These are different approaches to improve the performance of your model (so-called meta-algorithms):

1. Bagging (stands for Bootstrap Aggregation) is the way decrease the variance of your prediction by generating additional data for training from your original dataset usingcombinations with repetitions to produce multisets of the same cardinality/size as your original data. By increasing the size of your training set you can't improve the model predictive force, but just decrease the variance, narrowly tuning the prediction to expected outcome.
2. Bagging reduces variance by averaging and bagging has little effect on bias. Way to reduce variance and bias both - boosting
3. Boosting 
   • Train model on train set
   • Compute error of model on train set
   • Increase weights on train cases model gets wrong
   • Train new model on re-weighted train set ( Draw a bootstrap sample from the data with the probability of drawing each example is proportional to its weight  - resampling is easier to implement than reweighting)
   • Re-compute errors on weighted train set
   • Increase weights again on cases model gets wrong
   • Repeat until tired (100+ iteraations)
   • Final model: weighted prediction of each model
4. Boosting can hurt with noisy data sets but bagging wont hurt with noisy datasets
5. As a thumb rule, bagging almost always helps, boosting helps more than bagging and is supposed to hurt as well. So, you have to remove the noise first before you can use boosting. Bagging you can keep the noise and it will still get you the results.
7. 
   
8. Stacking is a similar to boosting: you also apply several models to you original data. The difference here is, however, that you don't have just an empirical formula for your weight function, rather you introduce a meta-level and use another model/approach to estimate the input together with outputs of every model to estimate the weights or, in other words, to determine what models perform well and what badly given these input data.

As you see, these all are different approaches to combine several models into a better one, and there is no single winner here: everything depends upon your domain and what you're going to do. You can still treat stacking as a sort of more advances boosting, however, the difficulty of finding a good approach for your meta-level makes it difficult to apply this approach in practice.

Short examples of each:

1. Bagging: Ozone data.
2. Boosting: is used to improve optical character recognition (OCR) accuracy.
3. Stacking: is used in K-fold cross validation algorithms.

Excellent references from which this post is compiled : 

1. http://stats.stackexchange.com/questions/18891/bagging-boosting-and-stacking-in-machine-learning
2. http://www.cs.cornell.edu/courses/cs578/2005fa/CS578.bagging.boosting.lecture.pdf

What is Boosting ?

The concept of boosting applies to the area of predictive data mining, to generate multiple models or classifiers (for prediction or classification), and to derive weights to combine the predictions from those models into a single prediction or predicted classification (see also Bagging).

A simple algorithm for boosting works like this: Start by applying some method (e.g., a tree classifier such as C&RT orCHAID) to the learning data, where each observation is assigned an equal weight. Compute the predicted classifications, and apply weights to the observations in the learning sample that are inversely proportional to the accuracy of the classification. In other words, assign greater weight to those observations that were difficult to classify (where the misclassification rate was high), and lower weights to those that were easy to classify (where the misclassification rate was low). In the context of C&RT for example, different misclassification costs (for the different classes) can be applied, inversely proportional to the accuracy of prediction in each class. Then apply the classifier again to the weighted data (or with different misclassification costs), and continue with the next iteration (application of the analysis method for classification to the re-weighted data).

Boosting will generate a sequence of classifiers, where each consecutive classifier in the sequence is an "expert" in classifying observations that were not well classified by those preceding it. During deployment (for prediction or classification of new cases), the predictions from the different classifiers can then be combined (e.g., via voting, or some weighted voting procedure) to derive a single best prediction or classification.

Note that boosting can also be applied to learning methods that do not explicitly support weights or misclassification costs. In that case, random sub-sampling can be applied to the learning data in the successive steps of the iterative boosting procedure, where the probability for selection of an observation into the subsample is inversely proportional to the accuracy of the prediction for that observation in the previous iteration (in the sequence of iterations of the boosting procedure).

http://www.statsoft.com/Textbook/Statistics-Glossary/B/button/b#boosting
http://en.wikipedia.org/wiki/Boosting_(machine_learning)

Random Forest

Disadvantages

1. Ensembles of decision trees (such as Random Forests, which is a trademarked term for one particular implementation) are very fast to train, but quite slow to create predictions once trained. More accurate ensembles require more trees, which means using the model becomes slower. In most practical situations this approach is fast enough, but there can certainly be situations where run-time performance is important and therefore other approaches would be preferred.
2. Results of learning are incomprehensible. Compared to a single decision tree, or to a set of rules, they don't give you a lot of insight.
3. They are hard to make incremental. It can be done, but there's no natural algorithm to do so (in the same way, for instance, that it's easy to tweak the parameters of naive bayes after you add an instance, or to add a new instance to a nearest neighbour classifier).
4. Random Forest is usually less accurate than Boosting/GBM on wide range of tasks, and usually slower in the runtime.
5. Overfitting : Broadly the reason is the same as for any algorithm: fitting noise instead of signal. In decision trees, this happens when the trees are too deep. If you just kept going, eventually the tree would have a node for every distinct point and it turns into a form of 1-nearest-neighbor classifier. This fits the training data too closely and is unlikely to generalize. This is why there is usually a stopping or pruning criteria. It could be a minimum node size, such that nodes with <= N examples are not split further, or a minimum information gain, such that nodes that have no decision that decreases entropy more than a trivial amount are not split.
6. 
   

Advantages of Random Forests :

1. Random forest is robust to outliers
2.  And a large amount of classifiers is very likely to happen to generate a good classifiers by finding a decent subset of features.
3. 
   

References : 

1. http://www.quora.com/When-is-a-random-forest-a-poor-choice-relative-to-other-algorithms
2. http://www.quora.com/How-do-random-forests-work-in-laymans-terms

Comparison of machine learning algorithms

1. http://www.quora.com/What-are-the-advantages-of-different-classification-algorithms
2. http://www4.ncsu.edu/~arezaei2/paper/JCIT4-184028_Camera%20Ready.pdf
3. http://stackoverflow.com/questions/10317885/decision-tree-vs-naive-bayes-classifier

What is the run time complexity of linear regression ?

http://stackoverflow.com/questions/1955088/what-is-the-bigo-of-linear-regression

Data Preparation : kaggle Facebook Recruiting competition III

Each record in the data ends with \r, so you can replace all the \n with spaces and replace all the \r with \n.

#!/bin/bash

if [ -z "$1" ] ; then
echo "First replaces all the \\n with spaces then replaces all the \\r with \\n"
echo "usage: $0 input.csv output.csv"
exit 1;
fi

tr '\n' ' ' < "$1" | tr '\r' '\n' > "$2"




<post from kaggle forum>

head -n [number of lines] Train.csv > sample_train.csv

Python script to parse the data : 

import csv, sys

if len(sys.argv) <> 3:

    print >>sys.stderr, 'Wrong number of arguments. This tool will print first n records from a comma separated CSV file.' 

    print >>sys.stderr, 'Usage:' 

    print >>sys.stderr, '       python', sys.argv[0], '<file> <number-of-lines>'

    sys.exit(1)

fileName = sys.argv[1]

n = int(sys.argv[2])

i = 0

out = csv.writer(sys.stdout, delimiter=',', quotechar='"', quoting=csv.QUOTE_NONNUMERIC)

with open(fileName, 'rb') as csvfile:

    for row in csv.reader(csvfile, delimiter=',', quotechar='"'):

        i += 1

        if i > n: break

        else:

            out.writerow(row)

How to interpret logistic regression data ?

In a previous post, we analyzed which version of logistic regression we should be using depending on how our data looks like.

This current post is aimed at interpreting the model that you have built using logistic regression.
http://www.stat.wisc.edu/~mchung/teaching/MIA/reading/GLM.logistic.Rpackage.pdf

How do I know that I have enough data for my logistic regression ?
I think I need more data and more positive signals into the data.