How to score your model using different scoring functions in Python

The scoring parameter can be a callable that takes model predictions and ground truth.

However, if you want to use a scoring function that takes additional parameters, such as fbeta_score, you need to generate an appropriate scoring object. The simplest way to generate a callable object for scoring is by using make_scorer. That function converts score functions (discussed below in Function for prediction-error metrics) into callables that can be used for model evaluation.

One typical use case is to wrap an existing scoring function from the library with non default value for its parameters such as the beta parameter for the fbeta_score function:

>>>

>>> from sklearn.metrics import fbeta_score, make_scorer
>>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
>>> from sklearn.grid_search import GridSearchCV
>>> from sklearn.svm import LinearSVC
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=ftwo_scorer)

The second use case is to build a completely new and custom scorer object from a simple python function:

>>>

>>> def my_custom_loss_func(ground_truth, predictions):
...     diff = np.abs(ground_truth - predictions).max()
...     return np.log(1 + diff)
...
>>> my_custom_scorer = make_scorer(my_custom_loss_func, greater_is_better=False)
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=my_custom_scorer)

make_scorer takes as parameters:

• the function you want to use
• whether it is a score (greater_is_better=True) or a loss (greater_is_better=False),
• whether the function you provided takes predictions as input (needs_threshold=False) or needs confidence scores (needs_threshold=True)
• any additional parameters, such as beta in an f1_score.

References : http://scikit-learn.org/stable/modules/model_evaluation.html

Python Tools

1. http://web.stanford.edu/~mwaskom/software/seaborn/
2. https://pypi.python.org/pypi/joblib
3. Matplotlib is like matlab graph plots http://matplotlib.org/users/pyplot_tutorial.html
4. Numpy Tutorial http://wiki.scipy.org/Tentative_NumPy_Tutorial
5. 
   

Random Forest Titanic

 This particular python function requires floats for the input variables, so all strings need to be converted, and any missing data needs to be filled.
Not all types of data can be converted into floats. For example, Names would be very difficult. In these cases let's decide to neglect these columns. Although they are strings, the categorical variables like male and female can be converted to 1 and 0, and the port of embarkment, which has three categories, can be converted to a 0, 1 or 2 (Cherbourg, Southamption and Queenstown). This may seem like a non-sensical way of classifying, since Queenstown is not twice the value of Southampton-- but random forests are somewhat robust when the number of different attributes are not too numerous.

Converting from categorical strings to floats is intuitive. However, filling in data can be more tricky. Some data cannot be trivially filled (such as Cabin) without complete knowledge of every cabin and ticket price for the entire ship.  Nonetheless, Fare price can be estimated if you know the class, or the age of a passenger can be estimated using the median age of the people on board. Fortunately for us, the amount of missing data here is not too large, so the method for which you choose to fill the data shouldn’t have too much of an effect on your predictive result.

Python Notes

1. if you want to call a specific column of data, say, the gender column, I can just type data[0::,4], remembering that "0::" means all (from start to end), and Python starts indices from 0 (not 1)
2. You should be aware that the csv reader works by default with strings, so you will need to convert to floats in order to do numerical calculations. For example, you can turn the Pclass variable into floats by usingdata[0::,2].astype(np.float)
3. 


Pythong Tips and Tricks - learning on the fly

1. What is the difference between iterators, generators and what is the usage of yield ? 
2. sd
3. sd
4.