Data Science 101: an introduction

Transcription

1 Data Science 101: an introduction Data Science Team Stanford University, Department of Statistics

2 Sexy jobs I keep saying the sexy job in the next ten years will be statisticians. People think I m joking, but who would ve guessed that computer engineers would ve been the sexy job of the 1990s? The ability to take data, to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it s going to be a at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. Hal Varian, Google s Chief Economist

3 Google trends

4 Keeping things in perspective

5 Data Science 101 Not the recipe for your future start-up Literacy for citizenship Data Science What is it? Where can we find it? How can we explore it? What does it mean to learn from data? How do we know when we are right or wrong?

6 There is a lot of data

7 Some examples of Big Data Genetics data: it is easy to assess genetic variation at millions of locations in the genome, and to sequence the entire DNA of a subject; we measure expression levels of 20,000 genes in different tissues Physics experiments: the data generated in one year is 30 petabytes (petabyte of average MP3-encoded songs would require 2000 years to play) Passively gathered data Products we buy Topics that engage us Our levels of physical activity Who we talk to

8 Some sources of Data Stats for change Data.gov GEO dbgap CERN Private data

9 Data can do a lot: sports

10 Data can do a lot: sports

11 Data can do a lot: politics

12 Data can do a lot: politics

13 Data can do a lot: recommendations NETFLIX: US based streaming and DVD rental-by-mail company 10M customers, 10K titles, 1.9M DVDs per day

14 Netflix challenge October 2006: Netflix offers $1M for an improved recommender algorithm. Training data: 100M ratings 480K users 17,770 movies 6 years of data: Test data: Last few ratings of each user (2.8M) Evaluation via RMSE: root mean squared error Netflix Cinematch RMSE: Competition: $1M grand prize for 10% improvement If 10% not met, $50K annual Progress Prize for best improvement

15 Netflix data: training

16 Netflix data: test

17 Netflix data: latent factor model

18 Netflix challenge

19 Data can do a lot: discover bias

20 Data can do a lot: medicine

21 Data can do a lot: language processing For example Google translate New York Times article.

22 Is data all we need?

23 Is data all we need?

24 Is data all we need? It is easy to find interesting patterns where none exist! How should we judge whether a pattern is interesting? When should we worry about falsely labelling patterns interesting? (E.g. Google mistranslates a sentence vs. incorrect cancer diagnosis... )

25 Is data all we need? Blindly used, machine learning algorithms can reinforce biases hidden in data.

26 Science is losing its authority?

27 Learn how to use data Explore: identify patterns Predict: make informed guesses Infer: quantify what you know

28 Data Science 101, resources A team of instructors, TA, and many more faculty that contribute to the course development Sohom Bhattacharya (Thursday 4-6) Pete Mohanty (Tuesday 2-4) Chiara Sabatti (Monday 10:30-12:30) Guenther Walter (Wednesday 2-4) Web-page: CANVAS site Piazza (linked from Canvas) for online discussion

29 Data Science 101: Evaluation Weekly homework (40% of grade) Participation (attendance, activities in labs, etc.) (10%) Midterm (30% if Midterm > Final) In class final (30% if Final > Midterm)

30 Data Science 101: Logistics Bring your laptops on Tuesdays and Thursdays Install R and RStudio for tomorrow Install extra packages this week (see course website). Lectures and RNotebooks for labs available on stats101.stanford.edu Two homeworks per module. A short one due Friday at 9:30am and a slightly longer one due the following Wednesday at 9:30am.