statistical computing in R

Size: px

Start display at page:

Download "statistical computing in R"

Victoria Greer
6 years ago
Views:

1 statistical computing in R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop MCS 507 Lecture 37 Mathematical, Statistical and Scientific Software Jan Verschelde, 20 November 2013 Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

2 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

3 R R is a language and environment for statistical computing and graphics, released under the GNU General Public License. R is an implementation of the S language developed at Bell Labs by Rick Becker, John Chambers, and Allan Wilks. We run R explicitly in Sage via 1 the class R, do help(r); or 2 opening a Terminal Session with R, type r_console(); in a Sage terminal. rpy2 is a Python interface to R, at sourceforge, developed by Laurent Gautier. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

4 no multiprecision R is NOT a computer algebra package: > choose(5,2) [1] 10 > choose(52,5) [1] > choose(100,30) [1] e+25 Instead of displaying an exact integer of 25 decimal places, the outcome of choose switched to a float. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

5 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

6 estimating π We estimate x 2 dx = π 4 > u <- runif( ,min=0,max=1) > 4*mean(sqrt(1-u^2)) [1] To check, we can use integrate: using 1,000,000 samples: > integrand <- function(x) { sqrt(1-x^2) } > integrate(integrand,lower=0,upper=1) with absolute error < > pi/4 [1] Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

7 Monte Carlo integration We estimate Applied to b a π/2 0 f(x)dx by (b a) 1 n 4 sin(2x)e x2 dx: x [a,b] f(x). > u <- runif( ,min=0,max=pi/2) > pi/2*mean(4*sin(2*u)*exp(-u^2)) [1] Checking: > integrand <- function(x) { 4*sin(2*x)*exp(-x^2) } > integrate(integrand,lower=0,upper=pi/2) with absolute error < 2.4e-14 Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

8 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

9 r.plot in Sage After generating 10 points with coordinates x, y [0, 1], uniformly distributed, we make a plot: sage: filename = tmp_filename() +.png sage: r.png(filename= "%s" %filename) NULL sage: x = r("runif(10)") sage: y = r("runif(10)") sage: r.plot(x,y) null device 1 sage: filename /Users/jan/.sage//temp/getafix.local/333/\ /tmp_0.png Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

10 the plot in tmp_0.png Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

11 plotting to a specific file Plotting 10 random points: sage: filename = /Users/jan/Pictures/points.png sage: r.png(filename= "%s" % filename) NULL sage: x = r("runif(10)") sage: y = r("runif(10)") sage: r.plot(x,y) null device 1 sage: filename /Users/jan/Pictures/points.png We can see the picture as specified by filename. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

12 a density function Plotting the density function of the normal distribution: sage: filename = tmp_filename() +.png sage: r.png(filename= "%s" %filename) NULL sage: r.plot( dnorm,-3,+3) null device 1 sage: filename /Users/jan/.sage//temp/getafix.local/650/\ /tmp_0.png Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

13 the plot in tmp_0.png Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

14 a distribution function With pnorm we get the distribution function of the normal distribution, to plot it with Sage: sage: filename = tmp_filename() +.png sage: r.png(filename= "%s" %filename) NULL sage: r.plot( pnorm,-3,+3) null device 1 sage: filename /Users/jan/.sage//temp/getafix.local/650/\ /tmp_1.png Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

15 the plot in tmp_1.png Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

16 using rpy2 in python Plotting 10 random points: $ python Python (v2.7.3:70274d53c1dd, Apr , 20:52:43) [GCC (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more i >>> import rpy2.robjects as robjects >>> r = robjects.r >>> r.x11() rpy2.rinterface.null >>> r.plot(r.runif(10),r.runif(10)) rpy2.rinterface.null After r.x11() a new window pops up and the figure gets show in this window by the r.plot. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

17 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

18 sampled data We take 100 samples of the normal distribution. > R = rnorm(100) > R [1] [7] > sort(r) [1] [7] A summary of sorted data is in a stem and leaf plot. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

19 stem and leaf plot > stem(r) The decimal point is at the Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

20 histogram > h = hist(r) > h $breaks [1] [7] $counts [1] $intensities [1] [7] $density [1] [7] $mids [1] [7] Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

21 in a Sage session sage: r.quartz() # for OS X NULL sage: data = r.call( rnorm,100) sage: type(data) <class sage.interfaces.r.relement > sage: r.call( hist,data) $breaks [1] \ $counts [1] Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

22 the histogram plot Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

23 making a boxplot sage: r.quartz() NULL sage: r.call( boxplot,data) $stats [,1] [1,] [2,] [3,] [4,] [5,] Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

24 the boxplot Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

25 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

26 calling lm For x we generate 50 points in [0, 10] and y equals x plus some random noise: > x <- 10*runif(50) > y <- x + rnorm(50) > gx <- lm(y ~ x) lm is used to fit linear models, the method applied is QR decomposition. > gx$coefficients (Intercept) x Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

27 plotting the data sage: filename = tmp_filename() +.png sage: r.png(filename= "%s" %filename) NULL sage: x = r("x <- 10*runif(50)") sage: y = r("y <- x + rnorm(50)") sage: r.plot(x,y) null device 1 sage: filename /Users/jan/.sage//temp/getafix.local/1312/\ /tmp_0.png Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

28 the data points Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

29 the output of lm > summary(gx) Call: lm(formula = y ~ x) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) x <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 48 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 48 DF, p-value: < 2.2e-16 Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

30 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

31 the package datasets Typing data() at the R prompt shows a list of data sets in the package datasets. The last one in the list is women, which contains "Average Heights and Weights for American Women" The output of help(women) shows that the data set appears to have been taken from the American Society of Actuaries for some unknown year earlier than Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

32 printing the entire data set > print(women) height weight Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

33 a summary of the data > summary(women) height weight Min. :58.0 Min. : st Qu.:61.5 1st Qu.:124.5 Median :65.0 Median :135.0 Mean :65.0 Mean : rd Qu.:68.5 3rd Qu.:148.0 Max. :72.0 Max. :164.0 The correlation matrix: > cor(women) height weight height weight To make a scatterplot: > pairs Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

34 a scatterplot Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

35 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

36 data about tornadoes NOAA = National Oceanic and Atmospheric Administration NOAA s National Weather Service provides free data. Downloaded from the file 2011_torn.csv is a comma separated file with data about tornadoes in The data file comes without a header in its first row. Information about the data in the columns is provided in a separate.pdf file. The file 2011_torn.csv with an extra header row is renamed into tornadoes.csv. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

37 reading data into R > torn <- read.csv("tornadoes.csv") > attributes(torn) $names [1] "number" "year" "month" [4] "day" "date" "time" [7] "timezone" "state" "statefips" [10] "statenum" "scale" "injuries" [13] "fatalities" "loss" "croploss" [16] "startinglatitude" "startinglongitude" "endinglatitude" [19] "endinglongitude" "length" "width" [22] "statesaffected" "statenum.1" "tornsegment" [25] "county1fips" "county2fips" "county3fips" [28] "county4fips" These names are the headers for the 28 columns. There are 1778 tornadoes recorded on file. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

38 selecting data > torn[,8] shows all states in column 8 > torn[8,] shows row 8. > torn[8,8] shows the state: row 8 and column 8. To select specific rows and columns: > torn[c(5,20),c(3,8,11)] month state scale 5 1 MS HI 0 > hist(torn[,11]) Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

39 a histogram of the scales of tornadoes Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

40 correlating scale, injuries, fatalities, and loss > cor(torn[,c(11,12,13,14)]) scale injuries fatalities loss scale injuries fatalities loss Plotting starting latitude and longitudes: > plot(torn[,c(16,17)]) Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

41 starting latitudes and longitudes Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

42 statistical computing with R 1 R in Sage the language and environment R Monte Carlo integration making plots with R in Sage 2 Statistical Computing with R histograms of data fitting linear models 3 Data Sets exploring available data sets looking at a downloaded data set 4 Big Data R and Hadoop Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

43 Large Data Files Process big data with parallel and distributed computing: MySQL scales well. overview of Hadoop: Hadoop has become the de facto standard for processing big data. Hadoop is used by Facebook to store photos, by LinkedIn to generate recommendations, and by Amazon to generate search indices. Hadoop works by connecting many different computers, hiding the complexity from the user: work with one giant computer. Hadoop uses a model called Map/Reduce. RHadoop by Antonio Piccolboni is a project for R and Hadoop, hosted at Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

44 the Map/Reduce model Goal: help with writing efficient parallel programs. Common data processing tasks: filtering, merging, aggregating data and many (not all) machine learning algorithms fit into Map/Reduce. Two steps: Map: Tasks read in input data as a set of records, process the records, and send groups of similar records to reducers. The mapper extracts a key from each input record. Hadoop will then route all records with the same key to the same reducer. Reduce: Tasks read in a set of related records, process the records, and write the results. The reducer iterates through all results for the same key, processing the data and writing out the results. Strength: the map and reduce steps run well in parallel. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

45 example: predicting user behavior Problem: how likely is a user to purchase an item from a website? Suppose we have already computed (maybe using Map/Reduce) a set of variables describing each user: most common locations, the number of pages viewed, the number of purchases made in the past. Calculate forecast using random forests: Random forests work by calculating a set of regression trees and then averaging them together to create a single model. It can be time consuming to fit the random trees to the data, but each new tree can be calculated independently. One way to tackle this problem is to use a set of map tasks to generate random trees, and then send the models to a single reducer task to average the results and produce the model. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

46 Summary + Exercises Available at Joseph Adler: R in a Nutshell. 2nd edition, O Reilly, Richard Cotton: Learning R. A Step-by-Step Function Guide to Data Analysis. O Reilly, Write a wrapper around r.plot which takes two arguments: a string with the argument for r.plot and the name of the file for the plot. 2 Annual U.S. Killer Tornado Statistics are listed at Use python to get the annual data from the web page and into R data sets, making one data set for every year. Project Three due Wednesday 27 November at 9AM. Homework Six is due on Wednesday 4 December at 9AM: ex 3 of L-27; ex 2 of L-28; ex 1 of L-29; ex 1 of L-30; ex 2 of L-31; ex 3 of L-32; ex 1 of L-33; ex 2 of L-34; ex 1 of L-35; and ex 1 of L-36. Scientific Software (MCS 507 L-37) statistical computing with R 20 November / 46

= = Intro to Statistics for the Social Sciences. Name: Lab Session: Spring, 2015, Dr. Suzanne Delaney

= = Intro to Statistics for the Social Sciences. Name: Lab Session: Spring, 2015, Dr. Suzanne Delaney Name: Intro to Statistics for the Social Sciences Lab Session: Spring, 2015, Dr. Suzanne Delaney CID Number: _ Homework #22 You have been hired as a statistical consultant by Donald who is a used car dealer