The Michelin Curse: Expert Judgment Versus Public Opinion

Transcription

1 The Michelin Curse: Expert Judgment Versus Public Opinion Marco Del Vecchio The University of Warwick ICUR 2017 Day 2

2 Outline 1 Introduction 2 The Data 3 Methodology and Hypotheses 4 Results 5 Conclusion

3 Introduction Outline 1 Introduction 2 The Data 3 Methodology and Hypotheses 4 Results 5 Conclusion

4 Introduction Context Context When choosing what restaurant to go to, what movie to watch, what book to read, or what wine to buy, observable product characteristics are unlikely to provide a complete picture of what to expect. Thus, the average consumer might rely on external sources of information to make more informed decisions about a particular good or service. Without loss of generality, these sources might take two forms: expert judgment and public opinion.

5 Introduction The Michelin Guide The Michelin Guide Sometimes referred to as the foodies bible, the Michelin Guide is considered to be the foremost judge of restaurant excellence. Every year since 1926, the French company Michelin publishes a set of guides by geographical location, where it unveils which restaurants around the world have gained (or lost) 1 ( A very good restaurant in its category ) 2 ( Excellent cooking, worth a detour ) 3 ( Exceptional cuisine, worth a special journey ) Michelin star(s) 1. 1 Or as the French like to call them macarons, not to be confused with the English word macaroons

6 Introduction TripAdvisor TripAdvisor It has been argued that the most prominent user-generated review site within the travel industry is TripAdvisor.com. In fact, TripAdvisor.com is currently visited on average, by 350 million unique users monthly. In 2017, it reached 385 million reviews covering 6.6 million accommodations, restaurants and attractions.

7 Introduction Research Questions Research Questions By using the Michelin Guide Main Cities of Europe 2016 as the source of expert opinion and TripAdvisor user-generated content (UGC) as the source of electronic word-of-mouth (ewom) in the restaurant industry, we seek to answer three questions: 1 To what extent does expert judgment, in the form of Michelin stars, and public opinion, in the form of TripAdvisor user-generated content, differ in the context of restaurants? 2 What is the most important background information about a TripAdvisor reviewer in explaining the possible differences between ewom and expert opinion? 3 Given two restaurants a and b where a has one Michelin star and b does not have any but shares the same price range, cuisine, and city, is the average TripAdvisor overall rating associated with a lower than the one associated with b?

8 The Data Outline 1 Introduction 2 The Data 3 Methodology and Hypotheses 4 Results 5 Conclusion

9 The Data An Overview An Overview Restaurants (703,305): This dataset contains information about all the restaurants taken into consideration in the analysis. Namely, it includes information about the Michelin restaurants in the Michelin Guide Main Cities of Europe 2016 (482 in total), the restaurants in the cities featured in the Guide. Users (142,942): This dataset contains information about the users who have given at least one review to a restaurant featured in the Guide. Reviews (211,609): This dataset contains information about the reviews that have been given to the restaurants featured in the Guide. Visits (2,947,227): This dataset contains information regarding which user (in Users) visited which restaurant (in Restaurants).

10 The Data An Overview An Overview Figure: Geolocation of the users

11 The Data Data Exploration: Restaurants Network Visualisation Data Exploration: Restaurants Network Visualisation Data visualisation methodology: 1 Create an undirected graph where each node is a restaurant and two nodes are joined by a weighted edge if and only if at least one person has been to both, where the weight is given by the number of people for which this holds. 2 Colour the nodes by number of Michelin stars and colour the edges by mixing the source and target colours. 3 Let the size of the nodes be given by the corresponding element in the equilibrium distribution of the underlying Markov Chain as given by the PageRank algorithm.

12 The Michelin Curse: Expert Judgment Versus Public Opinion The Data Data Exploration: Restaurants Network Visualisation Data Exploration: Restaurants Network Visualisation Figure: 1 Michelin Star, 2 Michelin Star, 3 Michelin Star.

13 The Data Data Exploration: Restaurants Network Visualisation Data Exploration: Restaurants Network Visualisation Figure: Node colour given by modularity class.

14 The Data Storage Storage In order to establish relational links between the datasets which would benefit data consistency and retrieval, a MySql relational database consisting of four tables (Restaurants, Users, Reviews and Visits) was created. Figure: Database schema.

15 The Data Collection Collection For the sake of rapid prototyping and re-usability, a web-scraper in R was implemented. The library utilised to communicate with the remote web driver was RSelenium, and the chosen browser was PahntomJS (a scripted, headless browser for automating web page interaction). The web-scraper collected the data between September 15, 2016 and February 25, 2017 whilst running on a Red Hat server.

16 Methodology and Hypotheses Outline 1 Introduction 2 The Data 3 Methodology and Hypotheses 4 Results 5 Conclusion

17 Methodology and Hypotheses Hypothesis 1 Hypothesis 1 Hypothesis 1 : When comparing the mean value of TripAdvisor ratings between one, two, and three star(s) Michelin restaurants, one should not observe any significant difference. Methodology : 1 All the ratings were recomputed using the reviews, so to be able to work with non- rounded figures (Figure on next slide). 2 A non-parametric Wilcoxon rank sum test was conducted to asses whether the location of the ratings distributions changes between 1-2, 1-3, and 2-3 Michelin Stars with the following hypothesis: H0 true location shift is equal to 0 H1 true location shift is not equal to 0.

18 Methodology and Hypotheses Hypothesis 2 Hypothesis 2 Hypothesis 2 : Information about online reviewers in the form of demographics, restaurants visits, level of activity on the online platform, and trustworthiness can help to explain as well as predict the possible differences between ewom, in terms of overall rating given to a restaurant, and expert opinion in terms of Michelin stars. Methodology : The following models were trained on a (wide) panel data set. 1 Multinomial Logit model (MLogit): used to asses the marginal effects of the user background information on the rating difference. 2 Multilayer Perceptron (MLP): used to gauge how well the difference in the ratings could be predicted.

19 Methodology and Hypotheses Hypothesis 2 Hypothesis 2 We define the difference between the number of Michelin stars and the TripAdvisor rating as d i = φ(ˆr i m j ) where ˆr i is the overall rating r i left in review i for restaurant j mapped onto the interval [1, 3], m j is the number of Michelin stars that restaurant j has, and φ is the function very positive x > 1.5 φ(x) = positive 0 x 1.5, negative x < 0

20 Methodology and Hypotheses Hypothesis 2 Hypothesis 2 Table: Panel dataset variables Variable name Description user sex The gender of the user: female (1), man (0) user number of visits one stars The number of times that a specific users has visited one Michelin star restaurants. user number of visits two stars The number of times that a specific users has visited two Michelin star restaurants user number of visits three stars The number of times that a specific users has visited three Michelin star restaurants user meancategoryprice The mean price category of all the restaurants visited by the user The mean text polarity sentiment user mean review sentiment expressed by the reviews left by the user on a continuous scale form + to - 2 user level The level of the user user hometown lon The longitude of the user hometown user hometown lat The latitude of the user hometown number of helpful votes michelin restaurants The total number of helpful votes given to the reviews left by the user rating difference φ(ˆr i m j )

21 Methodology and Hypotheses Hypothesis 3 Hypothesis 3 Hypothesis 3 : On average, restaurants with one Michelin star tend to be rated higher by the public than similar non-starred restaurants. Methodology : In order to check this hypothesis, we formalise what we mean by similar restaurants, and we define an algorithm to look at the difference in the ratings. We define two restaurants r i and r j to be similar if and only if they satisfy the following list of desiderata: 1 r i and r j are in the same city. 2 r i and r j have the same price category. 3 r i and r j have cuisine similarity s ij = C i C j C i C j ρ, where ρ [0, 1] and C k are the cuisines associated to r k, k {i, j}.

22 Methodology and Hypotheses Hypothesis 3 Hypothesis 3 Algorithm 1 Restaurant Similarity 1: Fix the value of ρ 2: Initialise A which will contain the average difference in the ratings 3: Store the information about the Michelin star restaurants into R 4: for each one-michelin-star restaurant r in R do 5: Store the rating of restaurant r into m 6: Find all similar restaurants (as defined above) that have zero Michelin stars and store their rating into M 7: Compute the mean of M and store it into m 8: Store m m into a 9: Append a to A 10: end for 11: Take the mean of A and store it into ā 12: return ā

23 Results Outline 1 Introduction 2 The Data 3 Methodology and Hypotheses 4 Results 5 Conclusion

24 Results Hypothesis 1 Hypothesis 1 Table: Wilcoxon rank sum test results As indicated in table on the right, the Wilcoxon rank sum test suggests that there is a statistically significant difference in almost all TripAdvisor ratings between 1 and 2 as well as between 1 and 3 Michelin star(s) restaurants, thus rejecting Hypothesis 1. Rating Groups P-value Overall Value Service Food Atmosphere ** ** ** ** ** ** * ** * Red values indicate a negative difference. Significances codes: 0 *** ** 0.01 *

25 Results Hypothesis 1 This proves that the reputation of Michelin stared restaurants is not homogeneous across the number of stars according to TripAdvisor users.

26 Results Hypothesis 2 Hypothesis 2 MLogit : The marginal effect on the difference between the overall rating and the number of Michelin stars is Negative w.r.t.: Average restaurant expenditure Number of helpful votes for Michelin related reviews Positive w.r.t.: Number of visits to one, two or three Michelin stars restaurants User level Mean review sentiment

27 Results Hypothesis 2 Hypothesis 2 MLP : Confusion matrix Precision Recall F1-score Support Negative Positive Very positive Avg / total The model which generated this classification, was chosen in accordance to the grid search results which indicated that the best performing model in terms of negative logarithmic loss was parametrised as follows: Four hidden fully connected layers each having 100 neurons A learning rate of A regularisation strength of 0.01

28 Results Hypothesis 2 Hypothesis 2 Overall, we have shown how certain background information about online reviewers can help explaining as well as predicting the possible differences between ewom, in terms of overall rating given to a restaurant, and expert opinion in terms of Michelin stars. Hence, we can accept Hypothesis 2.

29 Results Hypothesis 3 Hypothesis 3 By running Algorithm 1 with different values of ρ, which controls the cut-off point for considering two restaurants similar cuisine-wise we can see how the mean difference changes accordingly. From Figure 3 it can be seen how the difference is always positive regardless of the value of ρ (it fluctuates between approximately 0.12, and 0.195), thus validating Hypothesis 3.

30 Conclusion Outline 1 Introduction 2 The Data 3 Methodology and Hypotheses 4 Results 5 Conclusion

31 Conclusion Conclusion We highlight that one Michelin star restaurants are cursed in the sense that they have significantly lower overall, service, food and atmosphere ratings when compared to two and three Michelin star restaurants.

32 Conclusion Conclusion We find that the marginal effect of the average restaurant expenditure and number of helpful votes for Michelin related reviews on the difference between the overall rating and the number of Michelin stars is negative, whereas the effect of the number of visits to one, two or three Michelin stars restaurants, user level, and mean review sentiment is positive. Further, this study presents an MLP classifier which is able to classify with 70% precision whether the rating difference is negative, positive, or very positive.

33 Conclusion Conclusion We show that, on average, one star Michelin restaurants have a higher mean rating when compared to similar (same city, price category and cuisine tags) non-starred restaurants.

34 Conclusion The End The End, Thank You.