MicroTrails Comparing Hypotheses about Task Selection on a Crowdsourcing Platform

Transcription

1 MicroTrails Comparing Hypotheses about Task Selection on a Crowdsourcing Platform Martin Becker 1 Kathrin Borchert 2 Mathias Hirth 2 Hauke Mewes 1 Andreas Hotho 1,3 Phuoc Tran-Gia 2 DMIR, Computer Sicence, Chair 6, University of Würzburg, Germany 1 Computer Science, Chair 3, University of Würzburg, Germany 2 L3S Research Center, Hanover, Germany 3

2 Crowdsourcing Platforms (1) Employers have repetitive, time extensive chores like Filling out many surveys Tagging a corpus of sentences Labelling large sets of images Task 4 Task 1 Campaign Campaigns Campaign with tasks and Tasks Task 3 Task 2 2

3 Crowdsourcing Platforms (2) Workers Choose tasks from campaigns Get rewards for finishing Campaigns and Tasks Campaign with tasks 3

4 Crowdsourcing Platforms (3) Issues How to optimize workflow? How to predict success of campaigns? How to recommend tasks for workers? Understand Survey What is your reason for choosing a task? The employer? The category? how workers? choose campaigns! 4

5 Agenda Motivation Method HypTrails Modelling worker behavior Hypotheses, Data and Results Discussion Conclusion 5

6 Agenda Motivation Method HypTrails Modelling worker behavior Hypotheses, Data and Results Discussion Conclusion 6

7 HypTrails (1): A Bayesian Approach for Comparing Hypotheses Observations Transition counts n ij Hypotheses Conditional probabilities p ij n n,1 n n,n p n,1 p n,n > p n,1 p n,n Bayes Formula Review transitions likelihood prior evidence between restaurants P D θ, H P(θ H) P D H = P θ D, H posterior Bayes factor H 1 Reviewers stay B 1,2 = P D H Reviewers review H in the same region the 1same restaurant 2 P D Hover 2 and over Philipp Singer, Denis Helic, Andreas Hotho and Markus Strohmaier, HypTrails: A Bayesian Approach for Comparing Hypotheses About Human Trails on the Web 7

8 Agenda Motivation Method HypTrails Modelling worker behavior Hypotheses, Data and Results Discussion Conclusion 8

9 Modelling worker behavior (1) HypTrails Campaign with tasks 9

10 Modelling worker behavior (2) Campaigns Log Data Trails 10

11 Modelling worker behavior (3) Campaigns HypTrails Observations Transition counts n ij

12 Modelling worker behavior (4) People stick to the same employer Survey What is your reason for choosing a task? The employer? The category? HypTrails Hypotheses Conditional probabilities p ij Everyone likes cat campaigns 12

13 Agenda Motivation Method Hypotheses, Data and Results Baselines and Availability Hypotheses Uniform Availability Data Results Worker Hypotheses Discussion Conclusion 13

14 Uniform Hypothesis Base line No self transitions (because on MicroWorkers one worker can only work on one task per campaign) 14

15 Agenda Motivation Method Hypotheses, Data and Results Baselines and Availability Hypotheses Uniform Availability Data Results Worker Hypotheses Discussion Conclusion 15

16 Availability Hypothesis - Transition Models After Overlap Campaign 3, #tasks: 2 Campaign 1, #tasks: 3 Campaign 4, #t 0 Campaign 2, #tasks: 5 0 Time 16

17 Agenda Motivation Method Hypotheses, Data and Results Baselines and Availability Hypotheses Uniform Availability Data Results Worker Hypotheses Discussion Conclusion 17

18 Data Timespan: May 2009 (founding) - January 2015 Workers: Only US workers Task release speed: MAX Covered US-Workers: 95% Covered Campaigns: 55% Campaigns: > Tasks: > 3.5 Mio. 18

19 Agenda Motivation Method Hypotheses, Data and Results Baselines and Availability Hypotheses Uniform Availability Data Results Worker Hypotheses Discussion Conclusion 19

20 Baselines and Availability - Results Overlap most realistic model for availability New base line 20

21 Agenda Motivation Method Hypotheses, Data and Results Baselines and Availability Worker Hypotheses Hypotheses Employer and Category Hypotheses Combined Employer and Category Hypotheses Value and Class Hypotheses Semantic Hypotheses Results Discussion Conclusion 21

22 Employer and Category Hypothesis Large transition probabilities for the Same employer Same category T. Schulze et al.. Exploring task properties in crowdsourcing-an empirical study on mechanical turk H. Aris. Influencing factors in mobile crowdsourcing participation: A review of empirical studies 22

23 Combination of Employer and Category Large probabilities for both same employer AND same category Medium probabilities for either same employer OR same category Small probabilities otherwise 23

24 Value and Stratified Class Hypothesis Values Payment Payment per hour Number of tasks Positions / numer of tasks Required time Stratified classes Payment Payment per hour Positions / number of tasks L. B. Chiltonet al.. Task search in a human computation market. T. Schulze et al... Exploring task properties in crowdsourcing-an empirical study on mechanical turk. M.-C. Yuen et al..task recommendation in crowdsourcing systems. 24

25 Semantic Hypothesis Probability based the cosine of TF-IDF vectors for Titles Descriptions 25

26 Agenda Motivation Method Hypotheses, Data and Results Baselines and Availability Worker Hypotheses Hypotheses Employer and Category Hypotheses Combined Employer and Category Hypotheses Value and Class Hypotheses Semantic HypothesesResults Results Discussion Conclusion 26

27 Overall Optimized combination of employer and category works best followed by Employer Semantic similarity Category Payment per hour and positions / number of tasks less prominent than expected 27

28 Value and Stratified Classes Hypotheses Stratification is necessary to beat availability baseline Overall bad performance Improve modelling? 28

29 Semantic Hypotheses New kind of hypothesis Descriptions work better than just title Probably due to defining distinguishing titles with similar content 29

30 Agenda Motivation Method Hypotheses, Data and Results Discussion Conclusion 30

31 Discussion s 1 e 1 Campaign 1, #tasks: 3 Time and Availability First-Order Markov Chains No Sessions Dataset US Choice and Modelling of Hypotheses No combination of employer and semantics No combinations of payment and emp/cat/sem Modelling of payment? 31

32 Agenda Motivation Method Hypotheses, Data and Results Discussion Conclusion 32

33 Conclusion Analyzed worker behavior using Logs on the US workers of the crowdsourcing platform MicroWorkers Modelled several hypotheses Confirmed results originally based on surveys Found a semantic component Thank you! 33