Crowdsourcing platforms: Mechanical Turk and CrowdFlower

Transcription

1 Crowdsourcing platforms: Mechanical Turk and CrowdFlower

2 Terminology

3 Terminology Requester (AKA You!)

4 Terminology Workers (AKA Crowdworkers/Turkers) Requester (AKA You!)

5 Terminology Workers (AKA Crowdworkers/Turkers) Platform Requester (AKA You!)

6 Terminology Microtasks (AKA Human Intelligence Tasks/HITs) Workers (AKA Crowdworkers/Turkers) Platform Requester (AKA You!)

7 Terminology Microtasks (AKA Human Intelligence Tasks/HITs) Workers (AKA Crowdworkers/Turkers) Platform Requester (AKA You!)

8 Terminology Microtasks (AKA Human Intelligence Tasks/HITs) Micropayments (AKA pennies, nickels, dimes) Workers (AKA Crowdworkers/Turkers) Platform Requester (AKA You!)

9 Terminology Platform

10 Terminology Workers sign up directly Platform

11 Terminology Workers sign up directly Work outsourced to workers on smaller platforms (e.g. gaming sites) Platform

12 How big is the market?

13 How big is the market? Study from January April 2010: 6,701,406 HITs total 165,368 HIT groups 9,436 requesters $529,259 worth of rewards

14 How big is the market? Volume of rewards over past 5 years

15 How big is the market? 1% of requesters account for 60% of rewards 90% of HITs and 50% of HIT groups pay less than $ % of HIT groups are completed in less than 16 hours Analyzing the MTurk Marketplace. Ipeirotis (2012)

16 How big is the market? 1% of requesters account for 60% of rewards 90% of HITs and 50% of HIT groups pay less than $ % of HIT groups are completed in less than 16 hours Analyzing the MTurk Marketplace. Ipeirotis (2012)

17 How big is the market? 1% of requesters account for 60% of rewards 90% of HITs and 50% of HIT groups pay less than $ % of HIT groups are completed in less than 16 hours Analyzing the MTurk Marketplace. Ipeirotis (2012)

18 How big is the market? Top requesters as of May

19 Who is doing all this work for us? Mechanical Turk CrowdFlower 3% 26% 44% 56% 71% Male Female Unspecified Panos Ipeirotis s site: CrowdFlower blog:

20 Who is doing all this work for us? Mechanical Turk CrowdFlower 19% 6% 18% 48% 12% 75% 4%4% 4%4% 6% US India UK Indonesia Canada Philippines Pakistan Others

21 Who is doing all this work for us? Mechanical Turk CrowdFlower 11% 1% 4% 15% 3% 4% 8% 34% 16% 23% 46% 35% 25 or younger or older

22 Who is doing all this work for us? Mechanical Turk CrowdFlower 8% 6% 23% 6% 4% 5% 8% 7% 17% 10% 10% 45% 6% 11% 23% 13% Less than 10K 10K-15K 15K-25K 25K-40K 40K-60K 60K-75K 75K-100K More than 100K

23 Who is doing all this work for us? Mechanical Turk* CrowdFlower 15% 4% 11% 2% 11% 34% 27% 25% 35% 8% 13% 16% High School Some College Associates Bachelors Masters PhD *Data is from 2010 and reflects only US workers

24 Why are they doing all this work for us? Good way to spend free time and earn money (e.g. instead of TV) Mechanical Turk* CrowdFlower As a primary source of income As a secondary source of income/ pocket change It is just so much fun!! *Data is from 2010 and reflects only US workers

25 Considerations for the requester

26 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Master Workers Qualification tests Custom qualifications DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

27 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Master Workers Qualification tests Custom qualifications DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

28 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Master Workers Qualification tests Custom qualifications DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

29 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Qualification tests Custom qualifications Master Workers DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

30 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Qualification tests Custom qualifications Master Workers DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

31 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Qualification tests Custom qualifications Master Workers DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

32 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Qualification tests Custom qualifications Master Workers DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

33 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Qualification tests Custom qualifications Master Workers DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

34 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Qualification tests Custom qualifications Master Workers DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way

35 Considerations for the requester Mostly US workers More flexibility with designing HITs, e.g. with Javascript Quality control largely up to the requester (although not entirely) Qualification tests Custom qualifications Master Workers DYI embedded controls/ majority voting scheme Better international coverage HIT designs are mostly plugand-play for requesters Built in mechanisms for quality control = more quality, less control Proprietary EM-style method for determine quality Prices can be hard to predict Limited to tasks that can be QCed this way