Human Biases

1. http://en.wikipedia.org/wiki/Conjunction_fallacy
2. http://www.psychologytoday.com/blog/in-practice/201301/50-common-cognitive-distortions
3. 
   

Feature Hashing

http://scikit-learn.org/stable/modules/feature_extraction.html#feature-hashing
http://en.wikipedia.org/wiki/Feature_hashing

http://people.csail.mit.edu/romer/papers/TISTRespPredAds.pdf

Adaptive Learning rate in gradient descent

Depending on the cost function F that we will select, we might face different problems. When the Sum of Squared Errors is selected as our cost function then the value of θF(Wj)/θWj gets larger and larger as we increase the size of the training dataset. Thus the λ must be adapted to significantly smaller values.

One way to resolve this problem is to divide the λ with 1/N, where N is the size of the training data. So the update step of the algorithm can be rewritten as:

1	Wj = Wj - (λ/N)*θF(Wj)/θWj

You can read more about this on Wilson et al. paper “The general inefficiency of batch training for gradient descent learning”.

Finally another way to resolve this problem is by selecting a cost function that is not affected by the number of train examples that we use, such as the Mean Squared Errors.

This technique was used in the online gradient descent code by tingrtu in Criteo Ad Click Competition organized by Kaggle.

Reference : http://blog.datumbox.com/tuning-the-learning-rate-in-gradient-descent/

How to score your model using different scoring functions in Python

The scoring parameter can be a callable that takes model predictions and ground truth.

However, if you want to use a scoring function that takes additional parameters, such as fbeta_score, you need to generate an appropriate scoring object. The simplest way to generate a callable object for scoring is by using make_scorer. That function converts score functions (discussed below in Function for prediction-error metrics) into callables that can be used for model evaluation.

One typical use case is to wrap an existing scoring function from the library with non default value for its parameters such as the beta parameter for the fbeta_score function:

>>>

>>> from sklearn.metrics import fbeta_score, make_scorer
>>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
>>> from sklearn.grid_search import GridSearchCV
>>> from sklearn.svm import LinearSVC
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=ftwo_scorer)

The second use case is to build a completely new and custom scorer object from a simple python function:

>>>

>>> def my_custom_loss_func(ground_truth, predictions):
...     diff = np.abs(ground_truth - predictions).max()
...     return np.log(1 + diff)
...
>>> my_custom_scorer = make_scorer(my_custom_loss_func, greater_is_better=False)
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=my_custom_scorer)

make_scorer takes as parameters:

• the function you want to use
• whether it is a score (greater_is_better=True) or a loss (greater_is_better=False),
• whether the function you provided takes predictions as input (needs_threshold=False) or needs confidence scores (needs_threshold=True)
• any additional parameters, such as beta in an f1_score.

References : http://scikit-learn.org/stable/modules/model_evaluation.html

Criteolab kaggle challenge

1. Take a random sample of the train data
2. Decision trees work worse than even random solution in this case 
3. Logistic regression with only the independent variables work better
4. The data is huge and we cant load the whole data into memory and probably we dont need the whole data to learn a model, but we need more insights into the categorical variables.
5. Perl script to find statistics of the categorical variables : https://github.com/novieq/kaggle/blob/master/test/stats.pl
6. Convert unknown values into NA so that they are treated as missing values or Convert all the categorical variables to CTR values
7. sd

Categorical Variables

Bias Variance Tradeoff

y_noise = np.var(y_test, axis=1)

    y_bias = (f(X_test) - np.mean(y_predict, axis=1)) ** 2

    y_var = np.var(y_predict, axis=1)

MSE = y_noise + y_bias + y_var

Proof : http://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

Kaggle Solutions

1. Dont overfit : https://www.kaggle.com/c/overfitting/forums/t/593/results-auc
2. Predicting biological response : https://github.com/emanuele/kaggle_pbr : emanuele's solution
3. Criteo 3rd Place : http://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10547/document-and-code-for-the-3rd-place-finish
4. 
   

Must read kaggle forum links

1. https://www.kaggle.com/c/bioresponse/forums/t/1889/question-about-the-process-of-ensemble-learning
2. 
   

Data transformations tips

1. If variable's distribution has a long tail(left skewed distribution), apply Box-Cox transformation (taking log() is a quick & dirty way). Standardize all variables when in doubt (does not hurt anyway)
2. We want to turn categorical features into count feature because one-hot-encoding would curse us with dimensionality rendering tree-based models unmanageable. We store the counts from the train set for every unique hash inside a dictionary, and use this to replace hashes with their count occurrence.

References :

1. http://stats.stackexchange.com/questions/18844/when-and-why-to-take-the-log-of-a-distribution-of-numbers/18852#18852
2. http://itl.nist.gov/div898/handbook/eda/section3/eda336.htm
3. 
   

Python Tools

1. http://web.stanford.edu/~mwaskom/software/seaborn/
2. https://pypi.python.org/pypi/joblib
3. Matplotlib is like matlab graph plots http://matplotlib.org/users/pyplot_tutorial.html
4. Numpy Tutorial http://wiki.scipy.org/Tentative_NumPy_Tutorial
5. 
   

Naive Bayes Classifier tips

In a nutshell, the Gaussian Naive Bayes model is generally used for continuous data (where each feature is a real number), where the underlying data distribution is assumed to be a Gaussian (Normal) distribution.

The Multinomial Naive Bayes model counts how often a certain event occurs in the dataset (for example how often a certain word occurs in a document).

The Bernoulli Naive Bayes model is similar to the Multinomial Naive Bayes model, but instead of counting how often an event occurred, it only describes whether or not an event occurred (for example whether or not a certain word occurs in a document, where it doesn't matter if it occurs once or 100000 times)

Logistic Regression

Decision Trees tips

Random Forests tips

Extremely Randomized Trees : In Random Forests, the partition is chosen to maximize entropy. In extremely randomized trees, the partition is chosen randomly. This reduces the variance and increases the bias.

Regularized Greedy Forest : This will increase the bias and reduce the variance by boosting and is an additive model.

Support Vector Machine tips

1. Works better on high dimensional data than logistic regression (features >> samples)
2. Creates a maximum margin classifier - hence its prone to outliers 
3. Outlier removal can give a better model
4. C works reverse than lamba as a regularization parameter 
5. High Value of C means the model error term has to be zero and only the regularization part will stay - so this will be a high variance model
6. On extremely high dimensional data sometimes, PCA followed by SVM works well.
7. My rank 57 on African Soil Challenge competition in kaggle was an ensemble of PCA SVM, SVM on the data and I used Box Cox(log) transformation for P value. P was the hardest to predict and showed that the distribution was left skewed. So a log transform helped. This was also used as an input to the blend. 

Posterior, Prior and Likelihood

Bayes Theorem is a very common and fundamental theorem used in Data mining and Machine learning. Its formula is pretty simple:

P(X|Y) = ( P(Y|X) * P(X) ) / P(Y), which is Posterior = ( Likelihood * Prior ) /  Evidence

So I was wondering why they are called correspondingly like that.

Let’s use an example to find out their meanings.

Example

Suppose we have 100 movies and 50 books.
There are 3 different movie types: Action, Sci-fi, Romance,
2 different book types: Sci-fi, Romance

20 of those 100 movies are Action.
30 are Sci-fi
50 are Romance.

15 of those 50 books are Sci-fi
35 are Romance

So given a unclassified object,

The probability that it's a movie is 100/150, 50/150 for book.
The probability that it's a Sci-fi type is 45/150, 20/150 for Action and 85/150 for Romance.

If we already know it's a movie, then the probability that it's an action movie is 20/100, 30/100 for Sci-fi and 50/100 for Romance.
If we already know it's a book, then that probability that it's an Sci-fi book is 15/50, 35/50 for Romance.

Right now, we want to know that given an object which has type Sci-fi, what the probability is if it’s a movie?

Using Bayes theorem, we know that the formula is:

P(movie|Sci-fi) = P(Sci-fi| Movie) * P(Movie) / P(Sci-fi)

Here, P(movie|Sci-fi) is called Posterior,
P(Sci-fi|Movie) is Likelihood,
P(movie) is Prior,
P(Sci-fi) is Evidence.

Now let’s see why they are called like that.

Prior: Before we observe it’s a Sci-fi type, the object is completely unknown to us. Our goal is to find out the possibility that it’s a movie, we actually have the data prior(or before) ourobservation, which is the possibility that it’s a movie if it’s a completely unknown object:P(movie).

Posterior: After we observed it’s a Sci-fi type, we know something about the object. Because it’s post(or after) the observation, we call it posterior: P(movie|Sci-fi).

Evidence: Because we’ve already known it’s a Sci-fi type, what has happened is happened. Wewitness it’s appearance, so to us, it’s an evidence, and the chance we get this evidence isP(Sci-fi).

Likelihood: The dictionary meaning of this word is chance or probability that one thing will happen. Here it means when it’s a movie, what the chance will be if it is also a Sci-fi type. This term is very important in Machine Learning.

So why those probabilities are named like that, the observation time is a very important reason.

KDD Cup

1. https://www.kaggle.com/c/kdd-cup-2014-predicting-excitement-at-donors-choose/forums/t/9347/another-approach-to-beat-the-essay-length-benchmark
2. http://rajithayrm.blogspot.com/2014/06/simple-approach-to-kdd-cup-2014-with.html
3. 
   

Random Forest Titanic

 This particular python function requires floats for the input variables, so all strings need to be converted, and any missing data needs to be filled.
Not all types of data can be converted into floats. For example, Names would be very difficult. In these cases let's decide to neglect these columns. Although they are strings, the categorical variables like male and female can be converted to 1 and 0, and the port of embarkment, which has three categories, can be converted to a 0, 1 or 2 (Cherbourg, Southamption and Queenstown). This may seem like a non-sensical way of classifying, since Queenstown is not twice the value of Southampton-- but random forests are somewhat robust when the number of different attributes are not too numerous.

Converting from categorical strings to floats is intuitive. However, filling in data can be more tricky. Some data cannot be trivially filled (such as Cabin) without complete knowledge of every cabin and ticket price for the entire ship.  Nonetheless, Fare price can be estimated if you know the class, or the age of a passenger can be estimated using the median age of the people on board. Fortunately for us, the amount of missing data here is not too large, so the method for which you choose to fill the data shouldn’t have too much of an effect on your predictive result.

Python Notes

1. if you want to call a specific column of data, say, the gender column, I can just type data[0::,4], remembering that "0::" means all (from start to end), and Python starts indices from 0 (not 1)
2. You should be aware that the csv reader works by default with strings, so you will need to convert to floats in order to do numerical calculations. For example, you can turn the Pclass variable into floats by usingdata[0::,2].astype(np.float)
3. 


Ensemble Learning with R

http://www.r-bloggers.com/an-intro-to-ensemble-learning-in-r/
http://vikparuchuri.com/blog/intro-to-ensemble-learning-in-r/
http://vikparuchuri.com/blog/build-your-own-bagging-function-in-r/