Dealing with categorical variables

http://www.r-bloggers.com/r-minitip-dont-use-data-matrix-when-you-mean-model-matrix/
http://topepo.github.io/caret/preprocess.html

Why are scatterplots used ?

• Scatterplot matrices are good for determining rough linear correlations of metadata that contain continuous variables.
• Scatterplot matrices are not so good for looking at discrete variables.

http://www.r-bloggers.com/scatterplot-matrices/

Learning to Rank

http://en.wikipedia.org/wiki/Learning_to_rank

What is the difference between bagging, boosting and stacking ?

These are different approaches to improve the performance of your model (so-called meta-algorithms):

1. Bagging (stands for Bootstrap Aggregation) is the way decrease the variance of your prediction by generating additional data for training from your original dataset usingcombinations with repetitions to produce multisets of the same cardinality/size as your original data. By increasing the size of your training set you can't improve the model predictive force, but just decrease the variance, narrowly tuning the prediction to expected outcome.
2. Bagging reduces variance by averaging and bagging has little effect on bias. Way to reduce variance and bias both - boosting
3. Boosting 
   • Train model on train set
   • Compute error of model on train set
   • Increase weights on train cases model gets wrong
   • Train new model on re-weighted train set ( Draw a bootstrap sample from the data with the probability of drawing each example is proportional to its weight  - resampling is easier to implement than reweighting)
   • Re-compute errors on weighted train set
   • Increase weights again on cases model gets wrong
   • Repeat until tired (100+ iteraations)
   • Final model: weighted prediction of each model
4. Boosting can hurt with noisy data sets but bagging wont hurt with noisy datasets
5. As a thumb rule, bagging almost always helps, boosting helps more than bagging and is supposed to hurt as well. So, you have to remove the noise first before you can use boosting. Bagging you can keep the noise and it will still get you the results.
7. 
   
8. Stacking is a similar to boosting: you also apply several models to you original data. The difference here is, however, that you don't have just an empirical formula for your weight function, rather you introduce a meta-level and use another model/approach to estimate the input together with outputs of every model to estimate the weights or, in other words, to determine what models perform well and what badly given these input data.

As you see, these all are different approaches to combine several models into a better one, and there is no single winner here: everything depends upon your domain and what you're going to do. You can still treat stacking as a sort of more advances boosting, however, the difficulty of finding a good approach for your meta-level makes it difficult to apply this approach in practice.

Short examples of each:

1. Bagging: Ozone data.
2. Boosting: is used to improve optical character recognition (OCR) accuracy.
3. Stacking: is used in K-fold cross validation algorithms.

Excellent references from which this post is compiled : 

1. http://stats.stackexchange.com/questions/18891/bagging-boosting-and-stacking-in-machine-learning
2. http://www.cs.cornell.edu/courses/cs578/2005fa/CS578.bagging.boosting.lecture.pdf

What is Boosting ?

The concept of boosting applies to the area of predictive data mining, to generate multiple models or classifiers (for prediction or classification), and to derive weights to combine the predictions from those models into a single prediction or predicted classification (see also Bagging).

A simple algorithm for boosting works like this: Start by applying some method (e.g., a tree classifier such as C&RT orCHAID) to the learning data, where each observation is assigned an equal weight. Compute the predicted classifications, and apply weights to the observations in the learning sample that are inversely proportional to the accuracy of the classification. In other words, assign greater weight to those observations that were difficult to classify (where the misclassification rate was high), and lower weights to those that were easy to classify (where the misclassification rate was low). In the context of C&RT for example, different misclassification costs (for the different classes) can be applied, inversely proportional to the accuracy of prediction in each class. Then apply the classifier again to the weighted data (or with different misclassification costs), and continue with the next iteration (application of the analysis method for classification to the re-weighted data).

Boosting will generate a sequence of classifiers, where each consecutive classifier in the sequence is an "expert" in classifying observations that were not well classified by those preceding it. During deployment (for prediction or classification of new cases), the predictions from the different classifiers can then be combined (e.g., via voting, or some weighted voting procedure) to derive a single best prediction or classification.

Note that boosting can also be applied to learning methods that do not explicitly support weights or misclassification costs. In that case, random sub-sampling can be applied to the learning data in the successive steps of the iterative boosting procedure, where the probability for selection of an observation into the subsample is inversely proportional to the accuracy of the prediction for that observation in the previous iteration (in the sequence of iterations of the boosting procedure).

http://www.statsoft.com/Textbook/Statistics-Glossary/B/button/b#boosting
http://en.wikipedia.org/wiki/Boosting_(machine_learning)

Random Forest

Disadvantages

1. Ensembles of decision trees (such as Random Forests, which is a trademarked term for one particular implementation) are very fast to train, but quite slow to create predictions once trained. More accurate ensembles require more trees, which means using the model becomes slower. In most practical situations this approach is fast enough, but there can certainly be situations where run-time performance is important and therefore other approaches would be preferred.
2. Results of learning are incomprehensible. Compared to a single decision tree, or to a set of rules, they don't give you a lot of insight.
3. They are hard to make incremental. It can be done, but there's no natural algorithm to do so (in the same way, for instance, that it's easy to tweak the parameters of naive bayes after you add an instance, or to add a new instance to a nearest neighbour classifier).
4. Random Forest is usually less accurate than Boosting/GBM on wide range of tasks, and usually slower in the runtime.
5. Overfitting : Broadly the reason is the same as for any algorithm: fitting noise instead of signal. In decision trees, this happens when the trees are too deep. If you just kept going, eventually the tree would have a node for every distinct point and it turns into a form of 1-nearest-neighbor classifier. This fits the training data too closely and is unlikely to generalize. This is why there is usually a stopping or pruning criteria. It could be a minimum node size, such that nodes with <= N examples are not split further, or a minimum information gain, such that nodes that have no decision that decreases entropy more than a trivial amount are not split.
6. 
   

Advantages of Random Forests :

1. Random forest is robust to outliers
2.  And a large amount of classifiers is very likely to happen to generate a good classifiers by finding a decent subset of features.
3. 
   

References : 

1. http://www.quora.com/When-is-a-random-forest-a-poor-choice-relative-to-other-algorithms
2. http://www.quora.com/How-do-random-forests-work-in-laymans-terms

Comparison of machine learning algorithms

1. http://www.quora.com/What-are-the-advantages-of-different-classification-algorithms
2. http://www4.ncsu.edu/~arezaei2/paper/JCIT4-184028_Camera%20Ready.pdf
3. http://stackoverflow.com/questions/10317885/decision-tree-vs-naive-bayes-classifier

When to use classification and regression trees over linear and logistic regression

Tree methods are nonparametric and nonlinear. The final results of using tree methods for classification or regression can be summarized in a series of (usually few) logical if-then conditions (tree nodes). Therefore, there is no implicit assumption that the underlying relationships between the predictor variables and the dependent variable are linear, follow some specific non-linear link function [e.g., see Generalized Linear/Nonlinear Models (GLZ)], or that they are even monotonic in nature. For example, some continuous outcome variable of interest could be positively related to a variable Income if the income is less than some certain amount, but negatively related if it is more than that amount (i.e., the tree could reveal multiple splits based on the same variable Income, revealing such a non-monotonic relationship between the variables). Thus, tree methods are particularly well suited for data mining tasks, where there is often little a priori knowledge nor any coherent set of theories or predictions regarding which variables are related and how. In those types of data analyses, tree methods can often reveal simple relationships between just a few variables that could have easily gone unnoticed using other analytic techniques.

http://www.statsoft.com/Textbook/Classification-and-Regression-Trees

Debugging in R

Summary

• There are three main indications of a problem/condition: message, warning, error
  • only an error is fatal
• When analyzing a function with a problem, make sure you can reproduce the problem, clearly state your expectations and how the output differs from your expectation
• Interactive debugging tools traceback, debug, browser, trace, and recover can be used to find problematic code in functions
• Debugging tools are not a substitute for thinking!

Downloading and Reading data in R

1. If the url starts with http you can use download.file()
2. If the url starts with https, you may have to set method = curl
3. Check if a directory exists or not :
1. if (!file.exists("data")) {dir.create("data")}
4. Example : 
1. fileUrl <- span="">"https://data.baltimorecity.gov/api/views/dz54-2aru/rows.csv?accessType=DOWNLOAD"
2. download.file(fileUrl, destfile = "./data/cameras.csv", method = "curl")
3. list.files("./data")
4. dateDownloaded <- date="" li="">
5. dateDownloaded
5. Read data from the file
1. cameraData <- read.table="" span="">"./data/cameras.csv", sep = ",", header = TRUE)
2. head(cameraData)
6. read.csv sets sep="," and header=true
7. quote - you can tell R whether there are any quoted values quote="" means no quotes.
8. na.strings - set the character that represents a missing value.
9. nrows - how many rows to read of the file (e.g. nrows=10 reads 10 lines).
10. skip - number of lines to skip before starting to read
11. The biggest trouble with reading flat files are quotation marks ` or " placed in data values, setting quote="" often resolves these.
12. Reading excel files :
1. library(xlsx)
2. cameraData <- read.xlsx="" span="">"./data/cameras.xlsx",sheetIndex=1,header=TRUE)
3. head(cameraData)
13. Reading specific rows and columns
1. colIndex <- span="">2:3
2. rowIndex <- span="">1:4
3. cameraDataSubset <-read .xlsx="" span="">"./data/cameras.xlsx",sheetIndex=1,colIndex=colIndex,rowIndex=rowIndex)
14. read.xlsx2 is much faster than read.xlsx but for reading subsets of rows may be slightly unstable.
15. 
    
16. 
    

Find the mean of a column excluding missing values

mean(na.exclude(d$Ozone));

Counting missing values in R

Counting missing values

# Counting the occurrence of a particular value
x = c(3.14, 98, 0, 99, 7, NA, 0, 99)
sum(x==99, na.rm=T) # Count the occurrence of 99 in x, (omitting any NA)

# Counting the occurrence of NA
sum(x==NA) # Don't do this! Any conditional operation with NA results in NA, (NA is a special logical value)
sum(is.na(x)) # Do this to count the NA in x

# Counting the occurrence of any of a set of particular values
sum(x==c(98,99), na.rm=T) # Don't do this! It recycles c(98,99) to match x and does pairwise comparison
sum(x %in% c(98,99)) # Do this to count the occurrence of any 98 or 99 in x

# Counting incomplete cases, (rows of a data frame where one or more columns contain NA)

sum(complete.cases(data)) # Count of complete cases in a data frame named 'data'
sum(!complete.cases(data)) # Count of incomplete cases
which(!complete.cases(data)) # Which cases (row numbers) are incomplete?

The summary function of a data frame also counts the occurrence of NA in each column.

With numerical data that contain numerically coded missing values a scatter-plot is often helpful to identify missing values, especially if the missing value codes are sometimes entered incorrectly:

x = c(3.14, 98, 0, 99, 7, NA, 0, 99)
plot(x)

Similarly with character data a contingency table is helpful:

x = c("male", "female", "female", "male", 999)
factor(x)
table(x)

Re-coding particular values as NA (or any other value)

# Re-coding the occurrence of a particular value
x = c(3.14, 98, 0, 99, 7, NA, 0, 99)
x[x==99] = NA # Re-code all 99 in x as NA

# Re-coding the occurrence of NA
x[is.na(x)] = -1 # Recode all NA in x as -1. (Don't do this: x[x==NA] = -1)

# Re-coding the occurrence of any of a set of particular values
x = c(3.14, 98, 0, 99, 7, NA, 0, 99)
x[x %in% c(98,99)] = NA # Re-code any 98 or 99 in x as NA
x = c(2,5,3,5,2,0,1,3,4,4,0)
x[x %in% 0:3] = 0 # Re-code any 0, 1, 2, or 3 in x as 0

Removing NA values

# Removing NA values from a vector
x = c(3.14, 98, 0, 99, 7, NA, 0, 99)
x = x[!is.na(x)] # Drop NA from x

# Removing incomplete cases from a data frame named 'data'
na.omit(data) # In effect the same as: data[complete.cases(data), ]

The read.table function has an optional argument named na.strings that can take a vector of values which are then also mapped onto NA. For example:

data = read.table("foo.txt", header=TRUE, sep = "\t", na.strings=c("999", "-999") )

However this recodes occurrences of the given na.strings in all the variables, which can cause mistakes if codes for missing values for one variable are valid data for another.

http://forums.psy.ed.ac.uk/R/P01582/essential-1/

How to remove any row containing a NA value in a data frame

airquality[1:6,]
good <- airquality="" complete.cases="" p="">airquality[good, ][1:6, ]

To Read R

Chambers (2008). Software for Data Analysis, Springer. (your textbook)

1. Writing R Extensions
2. R Internals
3. Chambers (1998). Programming with Data, Springer.
4. Venables & Ripley (2002). Modern Applied Statistics with S, Springer.
5. Venables & Ripley (2000). S Programming, Springer.
6. Pinheiro & Bates (2000). Mixed-Effects Models in S and S-PLUS, Springer.
7. Murrell (2005). R Graphics, Chapman & Hall/CRC Press.
Other resources
• Springer has a series of books called Use R!.
• A longer list of books is at http://www.r-project.org/doc/bib/R-books.html

What is confounding ?

Shoesize - > Literacy
There might be a strong correlation between shoe size and literacy.

This does not imply causality. Pay attention to the other variables. Age might be confounding here.
As age increases, literacy increases. 

Types of Questions asked to Data Scientists

1. Descriptive - describe a set of data
• Descriptions cannot be generalized without adding statistical modelling
2. Exploratory - find relationships you didnt know about
• Exploratory analysis should not be alone used for generalizing/predicting
• Correlation doesnt imply causation
3. Inferential - use a relatively small sample of data to say something about the bigger population
• Inference is the common goal of statistical analysis
• Inference involves estimating both the quantity we care about and the certainty of that estimate
• Inference depends heavily on both the population and the sampling scheme
4. Predictive - To use the data in some objects to predict the data in other objects
• If X predicts Y then it doesnt mean that X causes Y
• More and more data works well with reasonable models
5. Causal - To find what happens to one variable when you change another variable
• Usually randomized variables are used for causation
• There are approaches to infering causation in non-randomized studies, but they are complicated and sensitive to assumptions
• Causal relationships are identified as average effects, but may not apply to every individual
• Causal models are usually the gold standard for data analysis
6. Mechanistic - physics

Introduction to R

Concepts 

1. Data types : character, numeric, integer, complex, logical
2. A vector can only contain objects of the same class
3. List is represented as a vector but can contain objects of different classes
4. Numbers in R are generally represented as numeric objects
5. If you explicitly want an integer, you need to specify the L suffix
6. Ex : entering 1 will be treated as a numeric object and 1L will be treated as an integer
7. R objects can have attributes - names, dimensions, class, other user defined attributes
8. x <- -="" 1:20="" create="" integer="" is="" operation="" sequences="" span="" the="" to="" used="">
9. The function c() is used to create vectors of objects
10. Objects can be co-erced from one class to another using as.* function
1. x <- 1:20="" as.character="" span="" x="">
11. Non sensical co-ersion results in NA
12. Matrices are vectors with dimension attribute. 
13. Matrices are created columnwise, so entries start at the upper left corner
14. Matrices can also be created from vectors by adding the dimension attribute.
1. x <- 1:10="" c="" dim="" m="" nbsp="" span="">
15. Matrices can also be created by column binding or row binding 
1. x <- 10:12="" 2:4="" cbind="" m="" nbsp="" rbind="" span="" x="" y="">
16. Factors - categorical data
17. Missing values : NA and NAN. NA can be integer NA or character NA and they have classes.
18. NAN value is NA but the converse is not true
19. Data Frames 
1. Special type of list where every element of the list has to be the same length
2. Each element of the list can be thought of as a column and the length of each element of the list is the number of rows
3. data frames can store different classes of objects in each column. Matrixes all elemets have to be of the same class
4. Data frames also have special attributes called row.names
5. Data frames are created by read.table() or read.csv()
20. R Objects can have names
1. m <- matrix="" nrow="2,ncol=2)</span">
2. dimnames(m) <- a="" b="" c="" d="" list="" span="">
21. sd
22. sd

Examples

1. How to get a list of available packages in R ?  
   

   
   
   available.packages()
   
   
    

2. How to install packages ?

   install.packages("slidify"); install.packages(c("slidify", "ggplot")); 

   
   
   
   
   source("http://www.bioconductor.org/biocLite.R");
   biocLite();
   #Place the names of the packages in a vector
   biocLite(c("GenomicFeatures","AnotationDbi"));
   
   
   
   
   
   

3. How do you load R packages ?
   
   
   library(ggplot2);
   search();
   
   
   After loading a package the functions loaded in the package will be attached to the top of the search list

4. How do you load the package in R ? library("slidify")
6. sd

1) How to read a csv file in R ?

1	data<- code="">read.csv(filename,header=TRUE)

2) How to display the first n lines of the file ?

1	head(data,n) : The default value of n is 6.

3) How to display the last n lines of the file ?

1	tail(data,n)

4) Calculate missing values in all the columns in the data set ?

1	colSums(data)

Other functions that can be used for this purpose are sapply and apply.

5) Calculate the mean of a column without the missing values ?

1 2 3 4 5 6 7 8 9

colMeans(data,na.rm=TRUE)      Ozone    Solar.R       Wind       Temp      Month        Day  42.129310 185.931507   9.957516  77.882353   6.993464  15.803922  colMeans(data)     Ozone   Solar.R      Wind      Temp     Month       Day        NA        NA  9.957516 77.882353  6.993464 15.803922  colMeans(data["Ozone"],na.rm=TRUE)    Ozone 42.12931

6) Extract the subset of rows of the data frame where Ozone values are above 31 and Temp values are above 90. What is the mean of Solar.R in this subset?

1 2 3

colMeans(subset(data,(Ozone>31 & Temp>90)))  Ozone Solar.R    Wind    Temp   Month     Day  89.5   212.8     5.6    93.4     8.2    14.5

Additional info on Subset

7) Find the mean temperature in the Month of n ?

1 2 3

colMeans(subset(data,Month==n))     Ozone   Solar.R      Wind      Temp     Month       Day     NA 190.16667  10.26667  79.10000   6.00000  15.50000

Additional Resources :
1) Filling in nas with column medians in R

2) Apply function and its variants
3) http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html