Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers

Transcription

1 Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers Toni Kaplan*, Susumu Saito*, Kotaro Hara, and Jeffrey P. Bigham (* Equal contribution)

2 Earning a decent wage is difficult for crowd workers on Amazon Mechanical Turk Most workers are not paid well [Martin et al., 2014] Workers median hourly wage is ~$2/h, while average requester pays $11+/h [Hara et al., 2018] Supporting workers for a better wage is a crucial challenge

3 What is needed to improve wage? (i.e., increasing!"#$% &$'(&) '&*$') )!"#$% #+,& -.&(# $- $ *"'/&' 1. Minimizing spent time 2. Maximizing pay per work time (Given ) [Hanrahan et al., 2015]

4 But not everyone can do this *Task listing page on Amazon Mechanical Turk* Requires time to find tasks [Chilton et al. 2010] Likely to waste working time [McInnis et al. 2016] - Too small reward - Broken task (= unable to submit) - Answer rejection after submission

5 Work Strategies Website forums and tools/scripts [Mason and Suri, 2012] Some workers are successful, but others are still not! No formal research on tool types and their prevalence per workers earnings < Our future project: Support by automation using Artificial Intelligence (AI)? Modeling successful workers behavior on their work can possibly give all workers good capabilities

6 Research Questions a) What are workers current strategies/tools/information for improving wage? b) What are design considerations for future tool development for supporting crowd workers?

7 Survey Period January 23, 2018 January 31, 2018 Subjects 360 US-based workers on Amazon Mechanical Turk 40/400 spam answers are filtered out 100 each to workers who has completed {100 1,000 5,000 10,000} HITs Task 67 survey questions (~10-30 minutes) for $3.50 General demographic information and income as workers Workers working tips Workers tools/forums usage Sentiment towards AI

8 Analysis Worker categorization by their reported total earnings in 2017 High-earning Extremes Top 10% High-earners Top 50% Low-earners Bottom 50%

9 Outline 1. Tools and strategies for workers 2. Sentiment towards automation with AI 3. Discussion: How can we assist workers better?

10 Outline 1. Tools and strategies for workers 2. Sentiment towards automation with AI 3. Discussion: How can we assist workers better?

11 Website forums Q: How often do you browse or post in Mechanical Turk related websites? [%] Always Most of the time About half the time Occasionally Never High-earning extremes High-earners Low-earners

12 Website forums Q: Which of the following Mechanical Turk related websites do you regularly read / browse / reference? MTurk Crowd Turker Hub [Reddit] AMT [Reddit] HITsWorthTurkingFor [Reddit] hnotwtf MTurkgrind MTurk Forum Turker Nation HIT Notifier

13 Website forums [%] Relatively new and active Various thread categories (e.g., Daily Work, MTurk Help & Resources ) Active Mainly Q&A / chatting Single thread category MTurk Crowd Turker Hub [Reddit] mturk [Reddit] HWTF

14 Tools Q: Are you using any browser extensions or scripts to improve your work experience/pay on Mechanical Turk? 100 [%] 50 0 Extremes High-earners Low-earners

15 Tools Q: Which, if any, of the following extensions and scripts do you regularly use while working on Mechanical Turk? Turkopticon MTurk Suite Panda Crazy HIT Scraper Turkmaster MTurk Engine

16 Tools Turkopticon [Irani et al., 2013] HIT Scraper: Auto-refreshing task search Panda Crazy: MTurk Suite Turkopticon with extra filters Panda Crazy Userscript with interface (Turkopticon ratings, for automated PandA ing of customizable blocklist ) prespecified HIT batches HIT Scraper MTurk Suite

17 Outline 1. Tools and strategies for workers 2. Sentiment towards automation with AI 3. Discussion: How can we assist workers better?

18 I would use a tool that automates some of the work in a HIT, which then lets me verify if the work was done correctly. Strongly Agree Somewhat Agree Neither Agree nor Disagree Why NOT agree? Somewhat Disagree Strongly Disagree

19 Why would you NOT use it? 78 workers expressed concerns The role of a human in Human Intelligence Tasks (18) The whole purpose of a HIT is to complete a Human Intelligence Task, which by definition is a task that cannot or should not be automated. Mistrust in the quality of AI output (17) I don t trust it, it would add more time to go back and check to see if it was right Violating the AMT terms of service (12) Violation of their personal ethics (9) Unfairness to requesters (12)

20 Outline 1. Tools and strategies for workers 2. Sentiment towards automation with AI 3. Discussion: How can we assist workers better?

21 Is there room for improvement? STATUS QUO: Sharing information among workers (Popular tasks) Tools that assist searching tasks (e.g., Turkopticon) (Choose tasks) Tools that assist accepting tasks (e.g., Panda Crazy) Microtasks are done all by workers

22 Is there room for improvement? POSSIBLE FUTURE: Collecting data for automation with AI (Popular tasks) (Similar tasks) 1. Tools that automate search & accepting microtasks 2. Tools that support doing microtasks Workers do microtasks with the support

23 Proposal 1: Automatic microtask search Find and reserve recommendable tasks as soon as they are posted Input:? Newly posted microtask batch Training data Feature: microtask HTML Label: completion time, approval/rejection Supervised ML algorithms Worker preferences - Microtask type - Work time - Reward target Output: - Hourly wage - How possible to be approved? - How long will they be available? Auto-reserve microtasks

24 Proposal 2: Iterative answer suggestion Augmenting work by automating user action Opening links as a task starts (e.g., survey tasks) Copy & pasting specified keywords in microtasks (e.g., external search tasks) Suggesting answers for repeated or commonly asked questions Input your AMT worker ID etc. (Repeatedly inputting worker ID) (Training data) Supervised ML algorithms? Answer suggestions: AXBCDEFG01234 (Output)

25 Conclusion a) What are workers current strategies/tools/information for improving wage efficiency? Use active forums and tools that evaluate requesters/tasks and reserve task batches b) What are design considerations for future tool development for supporting crowd workers? Use collected data for automation with machine learning

26 Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers Toni Kaplan*, Susumu Saito*, Kotaro Hara, and Jeffrey P. Bigham (* Equal contribution) Questions?

27

28 Website forums Q: Do you browse or post in any Mechanical Turk related websites? 80 [%] Always Most of the time About half the time Occasionally Never Extremes High-earners Low-earners

29 Website forums 100 [%] MTurk Crowd Turker Hub [Reddit] mturk [Reddit] HWTF [Reddit] hnotwtf Mturkgrind MTurk Forum Turker Nation HIT Notifier Other

30 Website forums MTurk Crowd and Turker Hub Relatively new and rapidly growing Has many discussion threads Subreddits Popular among most workers, but less attractive to high-earning extremes Dunno why? Other forums Used to be active but not anymore. HIT Notifier Still new and seems pretty easy to use, but dunno why it s not popular?

31 Tools (No userscript-related) (Userscript-related) 0 Turkopticon MTurk Suite Tampermonkey /Greasemonkey Panda Crazy

32 Tools Turkopticon and MTurk Suite Just a chrome extension and intuitive, which makes it very easy to use for everyone Used mainly for seeing requesters reputation Greasemonkey, Tampermonkey A base extension tool for userscript Panda Crazy Not very intuitive but has high functionality w/ helper scripts, once getting used HIT Scraper Intuitive, but not always good for high-earners who prefers native AMT page Turkmaster Often recognized as a substitute of Panda Crazy. Seems less useful MTurk Engine A new script being developed recently, that combines Panda Crazy and HIT Scraper Maybe not widely known yet?

33 High-earners had more access to Masters Qualification Access to more profitable HITs More opportunity, higher reward PANDA technique ( Preview and Accept ) Technique for task batch reservation More opportunity, less task search time [%] [%] Extremes High-earners Low-earners