Exploratory Data Analysis with #PrDC16

Transcription

1 Exploratory Data Analysis with #PrDC16

2 Motivation The ability to take data to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it that s going to be a hugely important skill in the next decades, because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it. Hal Varian, Google s Chief Economist The McKinsey Quarterly, Jan 2009

3 Source: Dice 2014 Tech Salary Survey Results

4 A Flood of Data is Coming... Source: Source: Wikipedia Sink or Swim

5 Overview Introduction to R Data Munging Descriptive Statistics Data Visualization Beyond R and EDA

6 How Does This Apply to Me? As a software developer, I often: Perform log file analysis Analyze software performance Analyze code metrics for code quality Detect anomalies in source data Transform or clean data files to make them usable Help decision makers make decisions based on data

7 About Me Independent software consultant Education B.S. in Computer Science B.A. in Philosophy Community Pluralsight Author ASPInsider Public Speaker Open-Source Software

8 SPONSORS

9 Introduction to R

10 What is R? Open source Language and environment Numerical and graphical analysis Cross platform

11 What is R? Active development Large user community Modular and extensible extensions and best of all

12 FREE

13 FREE

14 Source:

15

16 Source:

17

18 Code Demo

19 Data Munging

20 Data Munging Transforming data Raw data to usable data Data must be cleaned first

21 Data Munging Tasks Renaming variables Data type conversion Encoding, decoding or recoding data Merging data sets Transforming data Handling missing data (imputing) Handling anomalous values

22 Loading Data in R File-based data Web-based data Databases Statistical data And many more CSV XML

23 Cleaning Data This step is often the: Most difficult Most time consuming TIP: Record all steps

24

25

26 Column with wrong name Rows with missing values Runtime column has units Revenue in multiple scales Wrong file format

27 Code Demo

28

29 Descriptive Statistics

30 Descriptive Statistics Describe data Provides a summary aka: Summary statistics Movie Runtime Statistic Value (minutes) Minimum 38 1 st Quartile 93 Median 101 Mean rd Quartile 113 Maximum 219

31 Statistical Terms Observations Variables Qualitative variable Quantitative variable ID Date Customer Product Quantity John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

32 Number of Variables Types of Analysis Number of variables Univariate Bivariate Multivariate Qualitative Univariate Analysis Quantitative Univariate Analysis Type of variables Qualitative Quantitative Qualitative Bivariate Analysis Qualitative & Quantitative Bivariate Analysis Quantitative Bivariate Analysis Multivariate Analysis Type of Variable(s)

33 Univariate Analysis One variable Qualitative Frequency Quantitative Central tendency Dispersion

34 Bivariate Analysis Qualitative Joint frequency Quantitative Two variables Predictor Outcome Measures Covariance Correlation

35

36 Cowboys & Space Invaders: The Musical Extended Edition

37 Code Demo

38 Cowboys & Space Invaders: The Musical Extended Edition

39 Data Visualization

40 Data Visualization Visual data representation For human pattern recognition Map dimensions to visual characteristics

41 Data Visualization ID Date Customer Product Quantity John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

42 Number of Variables Types of Data Visualizations Number of variables Univariate Bivariate Multivariate Qualitative Univariate Analysis Quantitative Univariate Analysis Type of variable(s) Qualitative Quantitative Qualitative Bivariate Analysis Qualitative & Quantitative Bivariate Analysis Quantitative Bivariate Analysis Multivariate Analysis Type of Variable(s)

43 Cowboys & Space Invaders: The Musical Extended Edition

44 Code Demo

45 The Warlord of the Rings : An Unexpected Adventure Feature Length PG

46 Beyond R and EDA

47 This is just the tip of the iceberg! This is just the tip of the iceberg!

48 Advanced Data Analysis with R Cluster Analysis Statistical Modeling Dimensionality Reduction Analysis of Variance (ANOVA)

49 Source: Nathan Yau (

50 Data Mining and Machine Learning with R

51 Alternatives to R for EDA

52 Source: Microsoft

53

54

55

56

57

58 Where to Go Next R website: R Studio: Revolutions: Flowing Data: R-Blogger:

59

60 Conclusion

61 Conclusion Introduction to R Data munging Descriptive statistics Data visualization Beyond R & EDA

62 Feedback Feedback is very important to me! One thing you liked? One thing I could improve?

63 Contact Info Matthew Renze Website: