Introduction to R

Concepts 

1. Data types : character, numeric, integer, complex, logical
2. A vector can only contain objects of the same class
3. List is represented as a vector but can contain objects of different classes
4. Numbers in R are generally represented as numeric objects
5. If you explicitly want an integer, you need to specify the L suffix
6. Ex : entering 1 will be treated as a numeric object and 1L will be treated as an integer
7. R objects can have attributes - names, dimensions, class, other user defined attributes
8. x <- -="" 1:20="" create="" integer="" is="" operation="" sequences="" span="" the="" to="" used="">
9. The function c() is used to create vectors of objects
10. Objects can be co-erced from one class to another using as.* function
1. x <- 1:20="" as.character="" span="" x="">
11. Non sensical co-ersion results in NA
12. Matrices are vectors with dimension attribute. 
13. Matrices are created columnwise, so entries start at the upper left corner
14. Matrices can also be created from vectors by adding the dimension attribute.
1. x <- 1:10="" c="" dim="" m="" nbsp="" span="">
15. Matrices can also be created by column binding or row binding 
1. x <- 10:12="" 2:4="" cbind="" m="" nbsp="" rbind="" span="" x="" y="">
16. Factors - categorical data
17. Missing values : NA and NAN. NA can be integer NA or character NA and they have classes.
18. NAN value is NA but the converse is not true
19. Data Frames 
1. Special type of list where every element of the list has to be the same length
2. Each element of the list can be thought of as a column and the length of each element of the list is the number of rows
3. data frames can store different classes of objects in each column. Matrixes all elemets have to be of the same class
4. Data frames also have special attributes called row.names
5. Data frames are created by read.table() or read.csv()
20. R Objects can have names
1. m <- matrix="" nrow="2,ncol=2)</span">
2. dimnames(m) <- a="" b="" c="" d="" list="" span="">
21. sd
22. sd

Examples

1. How to get a list of available packages in R ?  
   

   
   
   available.packages()
   
   
    

2. How to install packages ?

   install.packages("slidify"); install.packages(c("slidify", "ggplot")); 

   
   
   
   
   source("http://www.bioconductor.org/biocLite.R");
   biocLite();
   #Place the names of the packages in a vector
   biocLite(c("GenomicFeatures","AnotationDbi"));
   
   
   
   
   
   

3. How do you load R packages ?
   
   
   library(ggplot2);
   search();
   
   
   After loading a package the functions loaded in the package will be attached to the top of the search list

4. How do you load the package in R ? library("slidify")
6. sd

1) How to read a csv file in R ?

1	data<- code="">read.csv(filename,header=TRUE)

2) How to display the first n lines of the file ?

1	head(data,n) : The default value of n is 6.

3) How to display the last n lines of the file ?

1	tail(data,n)

4) Calculate missing values in all the columns in the data set ?

1	colSums(data)

Other functions that can be used for this purpose are sapply and apply.

5) Calculate the mean of a column without the missing values ?

1 2 3 4 5 6 7 8 9

colMeans(data,na.rm=TRUE)      Ozone    Solar.R       Wind       Temp      Month        Day  42.129310 185.931507   9.957516  77.882353   6.993464  15.803922  colMeans(data)     Ozone   Solar.R      Wind      Temp     Month       Day        NA        NA  9.957516 77.882353  6.993464 15.803922  colMeans(data["Ozone"],na.rm=TRUE)    Ozone 42.12931

6) Extract the subset of rows of the data frame where Ozone values are above 31 and Temp values are above 90. What is the mean of Solar.R in this subset?

1 2 3

colMeans(subset(data,(Ozone>31 & Temp>90)))  Ozone Solar.R    Wind    Temp   Month     Day  89.5   212.8     5.6    93.4     8.2    14.5

Additional info on Subset

7) Find the mean temperature in the Month of n ?

1 2 3

colMeans(subset(data,Month==n))     Ozone   Solar.R      Wind      Temp     Month       Day     NA 190.16667  10.26667  79.10000   6.00000  15.50000

Additional Resources :
1) Filling in nas with column medians in R

2) Apply function and its variants
3) http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

How to install hadoop client ?

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.0/bk_installing_manually_book/content/rpm-chap1-3.html
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.0/bk_installing_manually_book/content/rpm-chap2-2.html

Pythong Tips and Tricks - learning on the fly

1. What is the difference between iterators, generators and what is the usage of yield ? 
2. sd
3. sd
4. 
   

Personalization of Search results - State of the art

There are two types of personalization strategies :

1. Profile based - Both long term and short term contexts are very important for profile based personalization strategies
2. Click Based - 
3. Personalization brings significant search accuracy improvements on the queries with large click entropy and has little effect on queries with small click entropy.
4. Personalization can even harm the search accuracy on some queries.
5. 
   

Data Preparation Tricks

1. perl -p -i.bak -e 's/\bNULL\b/0/g' filename
2. Print all rows - divide one column by another                                                                             awk '{print $1"\t"$3/$2}' merchant_ctr > merchant_ctr_final
3. Print the rows where the ctr value is greater than 0                                                                     awk '{if($2>0) print;}' merchant_ctr_final > merchant_ctr_final_pos
4. sd

http://www.ibm.com/developerworks/library/l-p102/index.html

What is the run time complexity of linear regression ?

http://stackoverflow.com/questions/1955088/what-is-the-bigo-of-linear-regression

How to get started with scikit-learn in python

I have had difficulties with managing the versions of different libraries like scikit-learn, numpy, matplotlib and sklearn. The best way I could figure out was to use the source code and add it to PYTHONPATH. The steps are documented below.

git clone git://github.com/scikit-learn/scikit-learn.git

export PYTHONPATH="/home/yourname/bin/scikit-learn"

python setup.py build_ext --inplace

make

Resources :

1. http://scikit-learn.org/dev/developers/index.html#retrieving-the-latest-code
2. http://stackoverflow.com/questions/12219657/upgrade-version-of-scikit-learn-included-in-enthought-distribution

Data Preparation : kaggle Facebook Recruiting competition III

Each record in the data ends with \r, so you can replace all the \n with spaces and replace all the \r with \n.

#!/bin/bash

if [ -z "$1" ] ; then
echo "First replaces all the \\n with spaces then replaces all the \\r with \\n"
echo "usage: $0 input.csv output.csv"
exit 1;
fi

tr '\n' ' ' < "$1" | tr '\r' '\n' > "$2"




<post from kaggle forum>

head -n [number of lines] Train.csv > sample_train.csv

Python script to parse the data : 

import csv, sys

if len(sys.argv) <> 3:

    print >>sys.stderr, 'Wrong number of arguments. This tool will print first n records from a comma separated CSV file.' 

    print >>sys.stderr, 'Usage:' 

    print >>sys.stderr, '       python', sys.argv[0], '<file> <number-of-lines>'

    sys.exit(1)

fileName = sys.argv[1]

n = int(sys.argv[2])

i = 0

out = csv.writer(sys.stdout, delimiter=',', quotechar='"', quoting=csv.QUOTE_NONNUMERIC)

with open(fileName, 'rb') as csvfile:

    for row in csv.reader(csvfile, delimiter=',', quotechar='"'):

        i += 1

        if i > n: break

        else:

            out.writerow(row)

How to interpret logistic regression data ?

In a previous post, we analyzed which version of logistic regression we should be using depending on how our data looks like.

This current post is aimed at interpreting the model that you have built using logistic regression.
http://www.stat.wisc.edu/~mchung/teaching/MIA/reading/GLM.logistic.Rpackage.pdf

How do I know that I have enough data for my logistic regression ?
I think I need more data and more positive signals into the data.

Count Data

http://en.wikipedia.org/wiki/Count_data
http://www.ats.ucla.edu/stat/stata/seminars/count_presentation/count.htm

When to use logistic regression and exact logistic regression

1. The general logistic regression process does not work very well for small sample set. The general logistic regression process is described here : http://www.ats.ucla.edu/stat/r/dae/logit.htm
2. For small sample sets, use exact logistic regression : http://www.ats.ucla.edu/stat/r/dae/exlogit.htm
3. To understand how the maximum likelihood estimation for logistic regression is biased for rare events, read : http://www.statisticalhorizons.com/logistic-regression-for-rare-events and http://www.cscu.cornell.edu/news/statnews/stnews82.pdf
4. sd
5. sd

When to use exact logistic regression instead of regular logistic regression?

It is used when the sample size is too small for a regular logistic regression (which uses the standard maximum-likelihood-based estimator) and/or when some of the cells formed by the outcome and categorical predictor variable have no observations. The estimates given by exact logistic regression do not depend on asymptotic results.

What is separation in the data ? 

http://en.wikipedia.org/wiki/Separation_(statistics)

When there is separation in the data we use exact logistic regression or firths logistic regression ?

When the data is small use exact logistic regression. When you have a lot of non events then use firths logistic regression as suggested here http://sas-and-r.blogspot.com/2010/11/example-815-firth-logistic-regression.html

When do you use mixed effects logistic regression model ?
When there are fixed and random effects on the data. When the data has rank bias or kadu quality score bias, then you use mixed effects logistic regression. Some of the other biases might be variable and random. 
http://www2.hawaii.edu/~kdrager/MixedEffectsModels.pdf

Python for Data Analysis

1. sudo easy_install pip - install pip on Mac OS
2. 
   
3. df
4. df
5. df
6. 
   

Big Data in news

1. Big Data in airlines and travel
2. http://www.informationweek.com/big-data/
3. 
   

Why should you view your data graphically before jumping into conculsions

http://upload.wikimedia.org/wikipedia/commons/e/ec/Anscombe%27s_quartet_3.svg
http://en.wikipedia.org/wiki/Anscombe's_quartet