Seminar in E-Business & Recommender Systems University of Fribourg, Department of Informatics

Transcription

1 Seminar in E-Business & Recommender Systems University of Fribourg, Department of Informatics Research Paper The impact of Recommender Systems on Business and Customers in Electronic Markets. STUDENT NAMES: José A. Mancera, Philipp Bosshard STUDENT NUMBERS: , COURSE NAME: Electronic Business and Recommender Systems DEPARTMENT: Department of Informatics SUPERVISOR: ASSISTANT: DATE OF SUBMISSION: Prof. Dr. Andreas Meier Luis Terán

2 II Table of contents List of Figures... V 1. Introduction Problem statement Research Objectives and Methodology Research Questions Objectives and Output of the thesis Research Methodology Timetable Addressees Recommender Systems Overview on Recommender Systems Recommender System Categories Collaborative Recommendation Content-Based Recommendation Knowledge-Based Recommendation Hybrid Recommendation Most popular recommendation techniques per category Recommender System Algorithms Similarity Measures Collaborative Filtering Algorithms Impacts of Recommender Systems on Business and Users Offline and Online Evaluation Offline Evaluation: Online Evaluation: Accuracy Impacts of Recommender Systems on Business Impacts of Recommender Systems on Users... 18

3 III 5. Evaluation of Recommender Systems and their value for Business and Users Recommendation System Properties Prediction Accuracy Coverage Confidence Trust Novelty Serendipity Diversity Utility Risk Robustness Privacy Adaptivity Scalability Summary of the Recommendation System Properties Evaluation Metrics for Recommender Systems Accuracy Metrics Privacy Metrics Adaptivity Metrics Trust Metrics Confidence Metrics Novelty Metrics Correlation between evaluation metrics and impacts Documented Scenarios Impact of Accuracy on Online Time Impacts of Accuracy, Privacy and Adaptivity on User Preferences Impact of Accuracy on Product Views... 35

4 IV 6.2 Non-documented Scenarios Impacts of Trust on Product Sales and Satisfaction Impacts of Novelty/Confidence on Product Sales and User Preferences Analysis and Results Documented Scenarios Non-Documented Scenarios The correlation results between RS Properties Results and Impacts Recommendations Variety in Recommender System Properties Non-documented scenarios suggestions Perception of the User New Research Questions Conclusion Future Work References... 50

5 V List of Figures Figure 1: Recommender System Categories... 4 Figure 2: Collaborative Recommendation Techniques... 8 Figure 3: Content-Based Recommendation Techniques... 9 Figure 4: Knowledge-Based Recommendation Techniques... 9 Figure 5: Hybrid Recommendation Techniques:... 9 Figure 6: Impacts of Recommender Systems on Business Figure 7: Impacts of Recommender Systems on Users Figure 8: Attributes for Consumer Perceptions [12] Figure 9: Recommendation System Properties Figure 10: Accuracy Predictions Figure 11: Recommendation System Properties Figure 12: Scope of our impacts analysis Figure 13: The impact of Accuracy on Online Time Figure 14: Summary of user's tolerable waiting time for computer response Figure 15: The impact of Accuracy, Privacy and Adaptivity on User Preferences Figure 16: The impact of Accuracy on Product Views Figure 17: Average Items viewed for 1 Control Group and 3 Treatment Groups Figure 18: The impact of Trust on Product Sales and Satisfaction Figure 19: The impact of Novelty on Product Sales and User Intention Figure 20: Research Paper Categories in Recommender Systems Field Figure 21: Direction of the Recommender System Design Figure 22: Relations between Metrics, RS properties and Impacts on Documented Scenarios Figure 23: Relations between Metrics, RS properties and Impacts on nondocumented Scenarios Figure 24: Relation between RS Properties and their impact on Users and Business... 47

6 1 1. Introduction 1.1 Problem statement Overview In this age full of data and big amounts of information, people use a variety of strategies to make choices about what to buy, how to spend their leisure time, and even what to eat. Recommender systems automate some of these strategies with the goal of providing to the user affordable, personal, and high-quality recommendations in order to help the user in the decision-making process. After all, how can one be sure that the user is not being manipulated and what is the recommenders influence power on the users and on the business? Firstly, the objective of this seminar thesis is to determine the impacts that Recommender Systems can have on Business and Users from the side of consumer research and marketing. Secondly, documentation and analysis of different research papers was carried on in order to identify the most important evaluation metrics for Recommender Systems. Thirdly, the document shows the results or our analysis based on the evidence found concerning the correlation between evaluation metrics and impacts of RS. Finally, recommendations and suggestions for future work regarding the impacts of Recommender Systems on Business and Customers will be proposed. As an important remark, recommender Systems have multiple areas of application consequently the focus of this paper only lies on e-business and e-commerce segments (e.g. Amazon, ebay and other kinds of e-shops) and the position of Recommender Systems for other types of segments are not part of the scope of this document.

7 2 1.2 Research Objectives and Methodology Research Questions The next group of questions is the guideline of our study, each of them is answered in sequence during the evolution of the document. 1. What are the impacts of Recommender System on users? 2. What are the impacts of Recommender System on business? 3. What are the most important evaluation metrics that can be used to analyze influences on the Users and the Business? 4. What is the correlation between the evaluation metrics of and the influence of recommender systems on business and users 5. Which aspects have been neglected in previous research papers? Objectives and Output of the thesis Based on the research questions, the main objectives of this seminar research paper analysis are to find potential correlations between impacts and recommender system (RS) properties. In addition recommendations and suggested paths of research are also suggested further in the document, in order that other researchers can continue our work and find more evidence about the metrics, which it would be essential to design, improve or interpret the output of recommender systems. Understanding these correlations requires a profound understanding about what recommender systems are?, How do they work? and what are the influences that they have on online customers and business? Research Methodology In order to answer the questions of this research seminar paper, our analysis would be based not only on data of previous studies in the field of Recommender Systems, consumer research and marketing but also on selected handbooks or textbooks recommended by our supervisors. All these documentation would guide us to discover or identify the variables that play a main role in the impact on users and business in the segments of e-business and e-commerce.

8 3 1.3 Timetable Acceptance of working title Submission of the proposal March 2015 Continue literature Research and reading Writing Chapter 1, 2 and 3 April 2015 Writing Chapter 4 Draft of the paper Finishing the report Revision and Correction April 2015 Midterm Appointment Submission of the thesis report Presentation of the thesis report 1.4 Addressees The target audience of this paper is primarily students in the fields of computer science, e-commerce, marketing and professionals who are involved in the field of Recommender Systems. The results of this seminar document should provide the parties mentioned above not only valuable knowledge in order to better understand, analyze and improve the quality of Recommender Systems, but also a better understanding of the consequences of the algorithms on users and business. 2. Recommender Systems In order to get a better understanding and interpretation of the findings presented in this seminar paper, it is important to review some core concepts in the field of recommender systems before moving to the analysis and results. In the next two chapters, there is a briefly overview of recommender systems, characteristics and description of the algorithms considered in our analysis. 2.1 Overview on Recommender Systems The majority of the Internet users have experienced in their visits to online stores or web pages that offer their services, certain kind of recommendations from other users and suggestions after reviewing a product as Customers who bought this Item also bought, or Customers who read this book also read these. These recommendations are becoming more often in the context of e-commerce.[6]

9 4 Among all the different definitions, a Recommender System (RS) is software and techniques which provide suggestions that determine whichever articles should be shown to a particular visitor. [6] Every RS requires that the system knows something about every user and it must maintain a user model or user profile which contains for instance the user preferences and remember the activity of the user, in order to be able to predict the articles that might be interesting for the user. The way the RS collects this information depends on the particular Recommendation technique, user preferences can, for instance, be acquired implicitly by monitoring user behavior but recommender system might also explicitly ask the visitor about his or her preferences. Moreover, it is important to collect additional information, like opinions and tastes of a large community and not only individual approaches.[6] The variety in the information that can be collected is very wide but the most important is to know which information the system could exploit when it generates a list of personalized recommendations. Figure 1 gives an overview of the four main RS categories: Collaborative Hybrid Recommend er System Content- Based Knowledge- Based Figure 1: Recommender System Categories

10 5 2.2 Recommender System Categories The next subsections give a short walkthrough around these four different RS categories in order to understand the characteristics and parameters that are involved in each of them Collaborative Recommendation The core idea of this type of recommender system is that if users shared the same interests in the past and for instance the users that viewed or purchased the same books will also have similar interests or preferences in the future. As an example considers the case when user A and user B have a purchase history that is strongly similar and user A has recently bought a book that B has not yet seen, the basic rationale is to propose this book also to B. Because this selection of hopefully interesting books involves filtering the most promising ones from a large set and because the users implicitly collaborate with one another, this technique is also called collaborative filtering (CF). [6] Techniques under this category are wide used in the context of e-commerce, the advantage of these techniques is that the recommender system does not need to know what the commodity or product is about, its genre, or who create it. Based on these conditions to propose commodities that are actually similar to those the user liked in the past might be more effective. [6] Content-Based Recommendation The techniques related to this category are based on the availability of item descriptions and a profile that assigns importance to these characteristics. For instance, thinking in an online store, the possible characteristics of the products might include the genre, the specific topic, or the author. Similar to item descriptions, user profiles may also be automatically derived and learned either by analyzing user behavior and feedback or by asking explicitly about interests and preferences. [6] In the context of content-based recommendation, the techniques are focused on how the systems automatically acquire and continuously improve user profiles and how do they determine which items match, similar items or common interest among users. [6]

11 6 Content-based recommendation has two advantages: 1. It does not require large user groups to achieve reasonable recommendation accuracy.[6] 2. New items can be immediately recommended once item attributes are available. [6] Knowledge-Based Recommendation There are market segments where the products are highly sophisticated, for instance the consumer electronics, which involves not only a large number of one time buyers but also the customer, does not have all the knowledge to understand about these technologies. [6] These simple facts bring new problems such that the recommender system cannot rely on the existence of a purchase history, build user profiles or propose products that other users bought. In addition more detailed and structured content must be considered as technical and quality features. [6] The solution that brings an answer in this context is a system that exploits additional and means end knowledge to generate recommendations. In such knowledge-based approaches, the recommender system typically makes use of additional, often manually provided, information about both the current user and the available items. [6] As a simple example of techniques in this category are Constraint based recommenders which for instance in the case of the digital camera domain, a constraint-based system could use detailed knowledge about the features of the cameras, such as resolution, weight, or price. In addition, explicit constraints may be used to describe the context in which certain features are relevant for the customer, such as, for example, that a high resolution camera is advantageous if the customer is interested in printing large pictures. Moreover the system could ask the user about the relative importance of features, such as whether resolution is more important than weight in order to provide recommendations. Although there are many different techniques in this category, the recommender system should be able to answer questions as: [6] What are the mechanisms that can be used to select and rank the items based on the user s characteristics? [6]

12 7 How do we acquire the user profile in domains in which no purchase history is available, and how can we take the customer s explicit preferences into account? [6] Which interaction patterns can be used in interactive recommender systems? [6] Finally, in which dimensions can we personalize the dialog to maximize the precision of the user preferences? [6] Hybrid Recommendation The previous categories have shown the main philosophy behind each of recommender systems main categories, nevertheless combining recommendation techniques can offer a better solution and more precise recommendations to specific problems and the advantages and disadvantages depends on the problem setting. [6] For instance, community knowledge exists and detailed information about the individual items is available, a recommender system could be enhanced by hybridizing collaborative or social filtering with content-based techniques. In particular, such a design could be used to overcome the described increase of problems with pure collaborative approaches and rely on content analysis for new items or new users. Combining different approaches, the recommender systems should be able to answer: [6] Which techniques can be combined, and what are the prerequisites for a given combination? [6] Should proposals be calculated for two or more systems sequentially, or do other hybridization designs exist? [6] How should the results of different techniques be weighted and can they be determined dynamically? [6] Last but not least, the classification and understanding of the philosophy behind each category shows a general overview and it is important to remember that these techniques vary constantly. The next section shows a more detailed classification of some of these particular techniques per category.

13 8 2.3 Most popular recommendation techniques per category Although there are many recommendation techniques used currently in the market and mentioning all of them, would be impossible. This report shows some of them and as general overview in order to give a better picture to the reader about the outlook of the recommenders systems. The next figures show some of the techniques by category. User-based nearest neighbor SVD-based recommend ation Collaborative Item-based nearest neighbor User item ratings matrix Figure 2: Collaborative Recommendation Techniques Content representatio n and content similarity Text classification methods Contentbased Similaritybased retrieval

14 9 Figure 3: Content-Based Recommendation Techniques Constraintbased recommend ers Knowledgebased recommend ation Case-based recommend ers Figure 4: Knowledge-Based Recommendation Techniques Monolithic hybridization design Hybrid recommendati on approaches Pipelined hybridization design Parallelized hybridization design Figure 5: Hybrid Recommendation Techniques:

15 10 3. Recommender System Algorithms It is essential to mention that the number of recommender systems that are available for applications or in development are uncountable and it would be very extensive to treat all of them in a document, in this section for the purposes of our research, we have selected some of them which would be supportive later to make a deeper analysis of some particular scenarios. 3.1 Similarity Measures Similarity measures are normally applied as preprocessing data tools, which prepare the data in order to apply a recommender algorithm. Similarity Measures are considered as a complement of most of the algorithms. The next section shows the most relevant ones that are used in most of the research papers that were analyzed.[4] Pearson Correlation The similarity between items can also be given by their correlation which measures the linear relationship between objects. Given the covariance of data points x and y Σ, and their standard deviation σ, we compute the Pearson correlation using [4] Cosine This measure indicates vector dot product and x,which is the norm of vector x. This similarity is known as the cosine similarity or the L2 Norm. Where σ is the covariance matrix of the data. Another very common approach is to consider items as document vectors of an n-dimensional space and compute their similarity as the cosine (x) of the angle that they form: The cosine-based approach defines the cosine-similarity between two users x and y as:[4]

16 Euclidean distance: The simplest approach to measure similarity is the Euclidean distance, where d(x,y) is the degree of the distance: Where n is the number of dimensions (attributes) and x k and y k are the k th attributes (components) of data objects x and y, respectively [6] 3.2 Collaborative Filtering Algorithms During the seminar research, we found that most of the algorithms applied in different research paper cases were mostly collaborative filtering algorithms. In this subsection we mention the three main categories and the description of a particular algorithm that was cited several times in different papers, k-nearest Neighbors algorithm Item-based CF Collaborative filtering (CF) is a technique used by some recommender systems. Collaborative filtering has two senses, a narrow one and a more general one. In general, collaborative filtering is the process of filtering for information or patterns using techniques involving collaboration among multiple agents, viewpoints, data sources, etc. Applications of collaborative filtering typically involve very large data sets. Collaborative filtering methods have been applied to many different kinds of data including: sensing and monitoring data, such as in mineral exploration, environmental sensing over large areas or multiple sensors; financial data, such as financial service institutions that integrate many financial sources; or in electronic commerce and web applications where the focus is on user data, etc. The remainder of this discussion focuses on collaborative filtering for user data, although some of the methods and approaches may apply to the other major applications as well. [6] In the newer, narrower sense, collaborative filtering is a method of making automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating). The underlying assumption of the collaborative filtering approach is that if a person A has the same opinion as a person B on an issue, A is more likely to have B's opinion on a different issue x than to have

17 12 the opinion on x of a person chosen randomly. For example, a collaborative filtering recommendation system for television tastes could make predictions about which television show a user should like given a partial list of that user's tastes (likes or dislikes).note that these predictions are specific to the user, but use information gleaned from many users. This differs from the simpler approach of giving an average (non-specific) score for each item of interest, for example based on its number of votes. [4] [6] Memory-based CF This approach uses user rating data to compute the similarity between users or items. This is used for making recommendations. This was an early approach used in many commercial systems. It is effective and easy to implement. Typical examples of this approach are neighborhood -based CF and item-based/user-based top-n recommendations. For example, in user based approaches, the value of ratings user 'u' gives to item 'i' is calculated as an aggregation of some similar users' rating of the item: [4] [6] Where 'U' denotes the set of top 'N' users that are most similar to user 'u' who rated item 'i'. Some examples of the aggregation function includes: Where k is a normalizing factor defined as and is the average rating of user u for all the items rated by u. [4] [6] The neighborhood-based algorithm calculates the similarity between two users or items, produces a prediction for the user by taking the weighted average of all the ratings. Similarity computation between items or users is an important part of this

18 13 approach. Multiple measures, such as Pearson correlation and vector cosine based similarity are used for this. [4] [6] The user based top-n recommendation algorithm uses a similarity-based vector model to identify the k most similar users to an active user. After the k most similar users are found, their corresponding user-item matrices are aggregated to identify the set of items to be recommended. A popular method to find the similar users is the Locality-sensitive hashing, which implements the nearest neighbor mechanism in linear time. [4] [6] The advantages with this approach include: the explainability of the results, which is an important aspect of recommendation systems; easy creation and use; easy facilitation of new data; content-independence of the items being recommended; good scaling with co-rated items. [4] [6] There are also several disadvantages with this approach. Its performance decreases when data gets sparse, which occurs frequently with web-related items. This interfere the scalability of this approach and creates problems with large datasets. Although it can efficiently handle new users because it relies on a data structure, adding new items becomes more complicated since that representation usually relies on a specific vector space. Adding new items requires inclusion of the new item and the re-insertion of all the elements in the structure. [4] [6] Model-based CF Models are developed using data mining, machine learning algorithms to find patterns based on training data. These are used to make predictions for real data. There are many model-based CF algorithms. These include Bayesian networks, clustering models, latent semantic models such as singular value decomposition, probabilistic latent semantic analysis, Multiple Multiplicative Factor, Latent Dirichlet allocation and Markov decision process based models. This approach has a more holistic goal to uncover latent factors that explain observed ratings. Most of the models are based on creating a classification or clustering technique to identify the user based on the test set. The number of the parameters can be reduced based on types of principal component analysis. There are several advantages with this paradigm. It handles the data distribution better than memory based ones. This helps

19 14 with scalability with large data sets. It improves the prediction performance. It gives an intuitive rationale for the recommendations. The disadvantages with this approach are in the expensive model building. One needs to have a tradeoff between prediction performance and scalability. One can lose useful information due to reduction models. A number of models have difficulty explaining the predictions. [4] [6] k-nearest Neighbors algorithm The k-nn algorithm is one of the most popular among collaborative filtering (CF) recommenders. This classification method as most classifiers and clustering techniques is highly dependent on defining an appropriate similarity or distance measure. In pattern recognition, the k-nearest Neighbors algorithm (or k-nn for short) is a nonparametric method used for classification and regression.[1] In both cases, the input consists of the k closest training examples in the feature space. The output depends on whether k-nn is used for classification or regression: [4] [6] In k-nn classification, the output is a class membership. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor. [4] In k-nn regression, the output is the property value for the object. This value is the average of the values of its k nearest neighbors. [4] k-nn is a type of instance-based learning, or lazy learning, where the function is only approximated locally and all computation is deferred until classification. The k-nn algorithm is among the simplest of all machine learning algorithms. [4] Both for classification and regression, it can be useful to weight the contributions of the neighbors, so that the nearer neighbors contribute more to the average than the more distant ones. For example, a common weighting scheme consists in giving each neighbor a weight of 1/d, where d is the distance to the neighbor. [4][6]

20 15 The neighbors are taken from a set of objects for which the class (for k-nn classification) or the object property value (for k-nn regression) is known. This can be thought of as the training set for the algorithm, though no explicit training step is required. [4][6]

21 16 4. Impacts of Recommender Systems on Business and Users After a short overview of basics of recommenders systems in the previous sections, it is important to slowly go in the direction to describe what we have defined as impacts. Based on different research papers, the impact of Recommender Systems on the Business and on the User have often been neglected or understudied. Therefore, the purpose of this thesis is to acquire deeper knowledge about such impacts. In order to analyze those impacts, the first important step is to define whether the evaluation is done in Offline or Online experiments [17]. 4.1 Offline and Online Evaluation Offline Evaluation: If an experiment is performed offline, the data is collected in advance and contains the data set of user that have chosen or rated items. Offline Experiments are typically the easiest ones to conduct. The main advantage is that no real interaction with a user is required and allow us to test a wide range of recommender algorithms at a low level of cost. A major disadvantage of offline evaluation is the fact that only a small set of questions can be answered. Usually, it is the prediction power of an algorithm. In offline evaluation, the behavior of the user is not modeled and must be assumed. The influence of the Recommender System on the user s behavior cannot be measured in a direct way. As a consequence, it is not appropriate to only rely on offline evaluations [6] Online Evaluation: The aim of online evaluation is to influence the user s behavior. Online evaluation tries to measure the change of the behavior in real time when the user has an interaction with the Recommender System. Factor of the user s intent, the user s context and the way how recommendations are presented play an important role. Online Evaluation has a unique status as the measurement is done directly. The downside of online evaluations is that the comparison of recommender algorithms is costly and not always possible [6].

22 Accuracy Accuracy is one the most important properties among the quality of Recommender Systems. It decides about how the user s information needs are fulfilled. Every user s personal information needs may vary lot as goals, preferences, knowledge and contexts are on a very individual basis. While one user may be interested in the latest music charts, another one s goal might be to receive some recommendations about Classical Music or Opera. Therefore, a recommender system has to be accurate by recommending the most important items for the user [10]. 4.3 Impacts of Recommender Systems on Business In this paper, the focus lies on three types of impacts that Recommender Systems can have on business. Figure 6 gives an overview of them: Business Product Sales Product Views Satisfaction of the Recommendation Provider Figure 6: Impacts of Recommender Systems on Business Product Sales: There are several reasons why Recommender Systems have an impact on the Product Sales. A first reason is that the search costs for the user decrease as the need to search for the right product is reduced. The amount of Product sales increases as it is simpler for the user to find the product that satisfies his needs [11]. Product Views: The Product Views of a User are the most important pre condition for the Product Sales. However, the impact of Recommender Systems on Product Views is less clear. A Recommender System can bring the user faster to his product with less

23 18 product views and clicks but RS have a positive effect on up-selling, cross-selling and driving repeat visits which lead the product views to increase [11]. Satisfaction of the Recommendation Provider: A good recommender system must not only satisfy the user but the Recommendation Provider as well. One major interest of the recommendation provider is to operate and maintain the recommender system at low costs. Costs may occur in labor, memory, disk storage, CPU power and traffic [10]. Due to lack of evidence in previous research papers, the Satisfaction of the Recommendation Provider will not be taken into account in this thesis. 4.4 Impacts of Recommender Systems on Users From the user's perspective, we have enumerated four main factors that influence the user behavior (Figure 7). Users User Satisaction Online Time User Preferences User Perceptions Figure 7: Impacts of Recommender Systems on Users The User satisfaction is a main goal of Recommender Systems. It s important to keep in mind that Accuracy alone does not necessarily contribute to User satisfaction and other factors have to be included, i.e. serendipity. [10]. As an example, a recommendation from a grocery store to buy milk is indeed accurate but it will not lead to satisfaction as it s evident to everyone that milk is a standard good. A recommendation for some specific dairy products like cheese or ice

24 19 cream would satisfy the customer more. User satisfaction can be influenced by many more factors. Some of those are for example demographics, time spent till the reception of recommendations or if costs are charged for the use of Recommender Systems [10]. Online Time: The Online time defines the time the user is spending on the Web page where the Recommender System is running. User Preferences: This property is in a certain way difficult to control but interesting to analyze, given the nature that it is easier for humans to give judgments than to give scores based on their personal experiences, then we can rely on the system that had the largest number of votes. However there are some concerns to consider in this property like it assumes that all users are equal, which is not true in all the cases. For instance an e- shop website, a client may prefer the opinion of users who buy many items or the opinion of the users who buy only one single item. Therefore in this case we need to weight the votes by the importance of the user and giving this weight may not be easy. [6] As a final remark in this property, when we want to rely on user preferences, we need to compare specific properties in order to have an overview of the user. Therefore, although user satisfaction is important to measure, splitting satisfaction into smaller components is critical to understand the entire system and improve it [6].

25 20 User Perceptions: User perceptions refer to the process by which an individual selects, organizes, and interprets stimuli into a meaningful and coherent picture of the world [12.] User perceptions can be divided in attributes for non-personally oriented perceptions and for personally-oriented perceptions.(see figure 8) Attributes for nonpersonally oriented perceptions popular affordable exclusive unique manufactured luxurious uncommon superior Attributes for personally oriented perceptions exquisite leading influential successful well-regarded memorable attractive Figure 8: Attributes for Consumer Perceptions [12] Although user perception has been identified as a user impact, it is not part of the analysis of this document because given its numerous attributes to describe it, it remains unclear and it is still a an open research question from the side of recommender systems and an intense subject of study on the side of consumer research, and marketing.

26 21 5. Evaluation of Recommender Systems and their value for Business and Users As it was shown in the previous section, the impacts of recommender systems with respect users and business have an important role in our seminar analysis, nevertheless before to identify potential correlation and show some of our findings. It is important to understand, evaluate and make clear the different properties that are involved in the recommendation systems, in order to understand the trade-offs on parameters to measure properties of recommenders which in future sections would be linked in a certain way with the impacts. 5.1 Recommendation System Properties Regardless the case of an online or offline analysis, it is important to know the recommender properties in order to understand later their impacts on the business or users. In the next subsections, we will give a precise and concise overview of them. The figure 9 shows the most important properties. Scalability User Prefence Prediction Accuracy Adaptivity Coverage Privacy Robustness Recommendation System Properties Confidence Trust Risk Novelty Utility Diversity Serendipity Figure 9: Recommendation System Properties It is fundamental to mention that some of the properties in a recommender system can be trade-off, for instance giving less important to the accuracy rather than diversity, risk, privacy, etc. and analyze the effect of this change on the overall performance. As a consequence the combinations and the highly level of customization make the task very complex and the proper way of gaining such

27 22 understanding without intensive online testing or distinct to the opinions of domain experts is still an open question. Nevertheless as we discuss further in the results and also as part of our contribution, there is an strategy to find among the whole set of scenarios, some relevant ones and reduce the complexity to find new interesting scenarios to analyze. In the next subsections, we provide a short explanation of these properties in order to get a better understanding and see later their potential interconnection with respect the impacts Prediction Accuracy The property relies on a prediction engine which may predict user opinions over items (e.g. ratings of movies) or probability of usage (e.g. shopping). The core assumption is that the user will prefer a system that provides more accurate predictions. Prediction accuracy is normally independent of the user interface and it can be measured in an offline experiment. This property in a study measures the accuracy given a recommendation. It is important to emphasize that accuracy influence or causes a user behavior with comparison to the case which has no recommendations Accuracy for the purpose of our study can be viewed from three perspectives as we mention in the figure 10 and we will describe them later when we will mention more about the link of this property with the impacts mentioned before [6]. Accuracy Ratings predictions Usage predictions Rankings of items Figure 10: Accuracy Predictions

28 Coverage In this seminar paper, the term coverage refers to the proportion of items or user interactions from which the recommendation system can recommend. As a consequence, coverage can have an impact on the accuracy property reviewed before and deliver different results because of the delimitation of particular items or user interactions therefore there is an important trade-off between coverage and accuracy [6] Confidence Confidence in the recommendation context is defined as the system s trust in its recommendations or predictions. As we have described the algorithms in the previous chapter for instance, collaborative filtering recommenders tend to improve their accuracy as the amount of data over items grows. Similarly, the confidence in the predicted property typically also grows with the amount of data. It is important to mention that when the system shows confidence values in their recommendations, the users can take a further step and make better decisions, for instance if a system reports a low confidence in an item, then the user may tend to do further research on the item before making a decision [6] Trust This property refers to the user s trust in the system recommendation. For instance, it may be beneficial for the system to recommend some few items that the user already knows and likes with the purpose that the user observes that the system provides reasonable recommendations, which may increase the trust in the system recommendations for unknown items. In addition to this technique, another common way of enhancing trust in the system is to explain the recommendations that the system provides. [6]

29 Novelty Novel recommendations are recommendations for items that the user did not know about [6]. In applications that require novel recommendation, a practical approach is to filter out items that the user already rated or used. However, in many cases users will not report all the items they have used in the past. Thus, this simple method is insufficient to filter out all items that the user already knows and a more clever way to split the information should be implemented. Another method for evaluating novel recommendations uses the above assumption that popular items are less likely to be novel [6] Serendipity Serendipity is a measure of how surprising the successful recommendations are. For instance, if the user has rated positively many songs where a certain singer appears, recommending the new song of that singer may be novel, because the user may not know of it, but is hardly surprising. In the opposite direction, random recommendations may be very surprising also, and therefore the need of balance between serendipity and accuracy is required [6] Diversity Diversity goes in the opposite direction of similarity. It states that in some cases suggesting a set of similar items may not be as useful for the user, because it may take longer to explore the range of items. For example, take in consideration a recommendation for a vacation, where the system should recommend vacation packages. Presenting a list with 10 recommendations, all for the same location, varying only on the choice of hotel, or the selection of attraction, may not be as useful as suggesting 10 different locations. The user can view the various recommended locations and request more details on a subset of the locations that are appropriate to her/him [6] Utility In general perspective, we can define various types of utility functions that the recommender tries to optimize. Utility in the context of recommendations can be interpreted and integrated with other properties such as: diversity or serendipity. Nevertheless in the way that we interpret utility in this seminar paper is with respect the value that either the system or the user gains from a recommendation [6].

30 Risk A recommendation in some context may be associated with a potential risk. For instance, in the stock market when recommending stocks for purchase, users may wish to be risk-averse, preferring stocks that have a lower expected growth, but also a lower risk of collapsing [6]. On the other hand, users may be risk-seeking, preferring stocks that have a potentially high, even if less likely, profit. In such cases we may wish to evaluate not only the (expected) value generated from a recommendation, but also to minimize the risk [6] Robustness Robustness is the stability of the recommendation in the presence of fake information typically inserted on purpose in order to influence the recommendations [6]. Influencing the system to change the rating of an item may be profitable to an interested party. For example, an owner of a hotel may wish to boost the rating for their hotel. This can be done by injecting fake user profiles that rate the hotel positively, or by injecting fake users that rate the competitors negatively. Such attempts to influence the recommendation are typically called attacks. The level of protection varies depending form one protocol to another. Nevertheless, we should be aware that creating a system that is immune to any type of attack is unrealistic [6] Privacy It is important for most users that their preferences stay private, that is, that no third party can use the recommendation system to learn something about the preferences of a specific user. For instance, consider the case where a user who is interested in the wonderful and yet rare art of growing Bahamian orchids, then the user has bought a book titled The Divorce Organizer and Planner. The spouse of that user, looking for a present, upon browsing the book The Bahamian and Caribbean Species (Cattleyas and Their Relatives) may get a recommendation of the type people who bought this book also bought for the divorce organizer, thus revealing sensitive private information [6].

31 Adaptivity Real recommendation systems may operate in a setting where the item collection changes rapidly, or where trends in interest over items may shift. As a simple example of such systems is the recommendation of news items or related stories in online newspapers. In this scenario stories may be interesting only over a short period of time, afterwards becoming outdated. For instance, when an unexpected news event occurs, such as the tsunami disaster, people become interested in articles that may not have been interesting otherwise, such as a relatively old article explaining the tsunami phenomenon [6] Scalability This property relies in the ability to navigate in large collections of items without slowed down in the searches, one of the goals of the designers of such systems is to scale up to real data sets. As such, it is often the case that algorithms trade other properties, such as accuracy or coverage, for providing rapid results even for huge data sets consisting of millions of items [6].

32 Summary of the Recommendation System Properties The next Figure 11 shows a summary of all the RS-Properties considered in the analysis in Chapter 6. Dimension Metric/Technique Type(s) Accuracy Ratings: Root Mean Square Error (RMSE), Normalized RMSE Quantitative (NRMSE), Mean Absolute Error (MAE), Normalized MAE (NMAE) Ranking: Normalized Distance-based Performance Measure (NDPM), Spearman correlation, Kendall correlation, Normal- ized Discounted Cumulative Gain (NDCG) Classification: Precision, Recall, False Positive Rate, Specifity, F-Measure, Reciver Operating Characteristics (ROC) Coverage Catalogue Coverage, Weighted Catalogue Coverage, Prediction Quantitative Coverage, Weighted Prediction Coverage Confidence Neighborhood-aware similarity model, Similarity indicators Qualitative/ Quantitative Trust User studies Qualitative Novelty Comparing recommendation list and user profiles, Counting Qualitative/ popular items Quantitative Serendipity Comparing recommendation list and user profiles, ratability Qualitative/ Quantitative Diversity Diversity Measure, Relative Diversity, Precision-Diversity Quantitative Curve, Q-Statistics, Set theoretic difference of recommendation lists Utility Profit based utility function, study user intention, user study Qualitative/ Quantitative Risk Depending on application and user preference Qualitative Robustness Prediction shift, average hit ratio, average rank Quantitative Privacy Differential privacy, RMSE vs. Differential privacy curve Qualitative/ Adaptivity User studies but generally changing rate Quantitative/ Qualitative Scalability Training time, recommendation throughput Quantitative Figure 11: Recommendation System Properties Although there are many properties and we looked for papers that could include most of these RS properties, there were not too many papers that could analyze or give relevant evidence of interpretation other than accuracy, diversity and coverage 5.3 Evaluation Metrics for Recommender Systems Evaluation Metrics or quality measures are a fundamental part to make a statement about the quality of recommendation and prediction algorithms. The benefits of evaluation metrics are that different Recommender Systems can be tested on performance, compared with each other and improved. This stage provides an overview about the most commonly used quality measures [7]. The evaluation metrics can be classified into 6 main groups. The first group consists prediction metrics which are primarily used to test accuracy. Those are the Mean Absolute Error (MAE), Root of Mean Square Error (RMSE) and Normalized Mean

33 28 Average Error (NMAE). In the second group, one can find the set of recommendation metrics. Precision, Recall and Receiver Operating Characteristic fall into that group. The third group deals with rank recommendation metrics. There are two metrics which are the half-life and the discounted cumulative gain. Group Nr. 4 contains the diversity and the novelty of the recommended items. A fifth group gives information stability metrics. Here, the use of the Mean Absolute Shift (MAS) is proposed. The sixth and last group makes the use of Reliability measures [7]. Due the variety of options regarding the metrics, only the metrics that will be used in Chapter 6 of this thesis are going to be presented Accuracy Metrics Uis defined as the set of RS users, r ui as the set of predicted user-item pairs (u,i), r ui as the true Ratings. The true ratings r ui are known as they are hidden in an offline experiment [6]. Mean Absolute Error (MAE) Root of Mean Square Error (RMSE) Normalized Mean Absolute Error (NMAE): The NMAE is a version of the MAE that has been normalized by the range of (r max r min ) Privacy Metrics Although in the recommender systems literature, there are many metrics definitions to measure privacy, we find convenient to measure privacy with the k-anonymity metric which is almost a standard in the context of Swiss Government Privacy. The metric describes to which extent private information of the user is public. The parameters of k-anonymity define the degree of anonymity. In the simplest approach, the higher the value k, the more anonymity, otherwise values like k =0 means that all private information is published in the Recommender System [19].

34 Adaptivity Metrics The Adaptivity Metrics describe how fast the System can acquire new information and can adapt it in the algorithm. Although this metrics can be measured by different types of metrics, the one considered in this document is considered as changing rate Trust Metrics The Trustworthiness of a Recommender System cannot be measured in a quantitative way. However, an often used approach is to conduct qualitative user studies by asking the users whether the recommendations were reasonable to them. There is the possibility to check how frequently a recommender is applied by a user. A higher usage frequency is an indicator for a user to trust more in a Recommender System [18] Confidence Metrics If we consider an online scenario, the confidence of a recommendation can be calculated by the observation of environmental variables. Those so called confidence scores indicate how frequently a recommender is used an applied by the user. Some authors propose a calculation of confidences scores by using a neighborhood-aware similarity model. The model includes similarities by users and items to generate recommendations. The most suitable recommendation is the one that maximizes the similarity between a recommended item and similar item. The usage of Confidence scores can be applied by using confidence intervals or the probability that a value predicted to the user is true [18] Novelty Metrics The Novelty Metrics measures the difference between the items recommended to and known by the user. However, Novelty metrics do not have a standard. Bobadilla et al. suggest the following formula used by various authors [7]. Z u represents the set of n recommendations to the user u. sim (i,j) refers to the item-item memory based Collaborative Filtering similarity measures.

35 30 6. Correlation between evaluation metrics and impacts After an extensive analysis on different research papers which are related to recommender systems, we realized that in most of the cases the researchers present papers in three general categories: Technical Research Papers: Creating new or proposing variants of algorithms Applied Short Research Papers: Applying an algorithm for a case with purpose of testing Comparison Applied Research Papers: Applying different algorithms and compare results. In most of the cases they compare the results in performance of the algorithms by analyzing the metrics and in a certain way a short description of these metrics used by some algorithms with the properties related to them. The categories of papers studied for this seminar are presented in Figure 12. Comparison Applied Research Papers Applied Short Research Papers Technical Research Papers TEST Relation Results Relation Relation Relation Recommender Algorithms CASES Metrics analysis Recommender Properties Impacts Consequences Figure 12: Scope of our impacts analysis However, as we can observe in the figure 12, the relation between the metrics and the impacts has not been analyzed or mentioned in the studies. We found that in most of the cases the interpretation of the results of the algorithms applied on certain cases have absences of this analysis. In this section we present a potential series of cases where the connection between evaluation metrics and impacts is potentially present. On the one hand, in most of the cases we will support our scenarios with some research papers references and on the other hand there are cases that we consider to have some impacts but there is no literature that support these assumptions, therefore these cases will be a suggestion as a potential future work to continue the research in this direction.

36 Documented Scenarios Impact of Accuracy on Online Time Consequences Buy TEST Relation Metrics Relation RS Property Relation Impact User Recommender Algorithms CASES MAE RMSE NMAE Accuracy Online Time Keep Interest Visits Figure 13: The impact of Accuracy on Online Time Scenario: The scenario presented in Figure 13 shows the proposed correlation from the metrics that support accuracy to the Online Time impact and potential consequences on the user side. Key Assumptions: Users who shop online have a low level of patience or time tolerance in terms of waiting to receive the information, so the switching rate between websites that provide the same service is very high. A user that could not only find accurate results or receive a result after certain amount of time will change the e-commerce platform provider. Analysis: The time to retrieve results as a performance indicator in the RS algorithms has been documented and measured in different papers and it plays a main role to determine the efficiency as we can observe specially in the reference [13], where they compare the time performance of different algorithms 1-NN, 80-NN and Eigentaste-algorithm on a case that recommends funny jokes to the user. The performance response times are normally measured in milliseconds. On the other side for our assumption 2, we found evidence on reference [14] about the tolerable waiting time of users for computer response and we extracted the most relevant findings in the figure 14.

37 32 Study Findings/Recommendations Miller(1968) Delay of 2 seconds is the limit before interference with short term memory occurs Nielsen(1993,1995,1996) Delay of 0.1 second is perceived as instantaneous success Delay of 1.0 second is the limit for users flow of thought to stay uninterrupted Delay 10 seconds is the limit for keeping users attention focus on the dialogue Scheneiderman(1984) Delay of 2 seconds is the limit where response to simple commands becomes unacceptable to users Figure 14: Summary of user's tolerable waiting time for computer response In order to understand this case consider an example where an user is looking for a particular flight itinerary for vacations and it chooses an online retailer but if the user does not get a recommendation on time, he would rarely stay on the same page and change into another provider to look again for the same flight itinerary. Based on the previous studies, we can infer that an algorithm that is designed to provide accuracy in the results, it is not successful if it does not show the recommendation to the user in the waiting tolerance interval of the user. Based on the evidence of the papers and the diagram in the figure 13, the online time as an impact has some consequences for the user and although we cannot predict the exact user behavior, we propose three different potential consequences that the user can take as a next step: Consequences: Buy: The user simply gets the product. Keep Interest: He finds the product but he does not buy it and keep interested on it. Increase the Number of visits: Regardless the user buys or keeps interest on the product that he searches. The accuracy of the algorithm ensures the success of the recommendations and increases the number of visits to the page.

38 Impacts of Accuracy, Privacy and Adaptivity on User Preferences Consequences Loyalty TEST Relation Metrics Relation RS Properties Relation Impact User Recommender Algorithms CASES MAE RMSE NMAE K-Anonymity Changing Rate Accuracy Privacy Adaptivity User Preference Trust Visits Figure 15: The impact of Accuracy, Privacy and Adaptivity on User Preferences Scenario: As we reviewed in the recommender systems properties section, a recommender algorithm can consider different kind of properties that are related to certain metrics in this case we focus on the scenario where the recommender system focus on the RS properties such as accuracy, privacy and adaptability in order to have an impact on User Preferences. Key Assumptions: Privacy is very important for the user and any kind of misuse of the information has a negative impact on the service that the user is using. High accurate predictions let the user to continue using the system and have a positive impact on the service. A high changing rate in the recommended products give the sense to the user that the recommender system is giving him the last products of the market or that the system is freshly updated in the products that the e-commerce platform is offering. Analysis: In this case we have mentioned in previous chapters the metrics related to the accuracy such as: MAE, RMSE and NMAE. Nevertheless for properties as privacy and adaptability the metrics that we consider to measure these RS properties are k- anonymity and changing rate respectively.

39 34 In addition changing rate metric is related to the adaptivity of the algorithm in the sense to measure the freshness of the data or external data elements that it is considering for providing the recommendation. Firstly, the assumption regarding the privacy relies on the potential risks to divulge personal Information from the users in the recommendations because there is the direct risk that someone will get information that the user wished to keep private. For instance as supportive argument that we found in reference [15], it was found that revealing identity information could lead to identity theft. There are also indirect risks of re-identification finding information about a user in one system that could identify her in another system. These elements cause a negative impact on the user that modifies his user preferences in a negative way. Nevertheless it is not sufficient to say that privacy can totally affect the user preferences in the way that the user will stop to use the service. As an example taken form reference [15], there is an important case analysis that supports this idea and it comes from the situation where in 2004, Amazon s Canadian site suddenly accidentally revealed the identities of thousands of people, who had anonymously posted book reviews. It turned out that authors were praising their own books and trashing other authors books. The New York Times reported that many people say Amazon s pages have turned into what one writer called a rhetorical war, where friends and family members are regularly corralled to write glowing reviews and each negative one is scrutinized for the digital fingerprints of known enemies. To increase the credibility of some reviews, after these events the writers still used Amazon services. Secondly, the accuracy and adaptivity are very good complement properties to the privacy in order to make an impact on the user in the user preference side in the sense that if the recommender still provides good accuracy in the recommendations and the freshness of the information is highly appreciate it by the user then he might just be careful to not provide all his private information and still use the services of the site. [16] Finally, Based on the relation between the RS properties described above and the user preferences, we consider that there are three potential consequences due to the change of user preferences shown in Figure 15.

40 35 Consequences: Loyalty: The user is more devoted to the product, if he higher preference. Trust: The higher the User preference is, the more the user will trust in the Recommendation. Visits: The number of visits increase with higher User preference as he turns into a buyer or stays an observer on the webpage Impact of Accuracy on Product Views TEST Relation Metrics Relation RS Property Relation Impact Business Consequences Revenue Cross Selling Recommender Algorithms CASES MAE RMSE NMAE Accuracy Product Views Up-Selling Figure 16: The impact of Accuracy on Product Views Scenario: The scenario shows the potential impact of Accuracy Metrics on Product Views of the user. As mentioned before in part 4.3 under reference [11], the impact of RS on Product Views is somewhat ambiguous. Here we present a duality effect: An accurate Recommender System will decrease the time the user spends on the webpage and therefore the Product Views decline. The decrease in time that the user spends online catches the user s interest to view additional and differentiated products which increases the Product Views. Assumptions: We assume that duality effect referring to product views has a positive impact on business in both cases. Analysis: In a Research Paper case for two top movie retailers in North America, a study about the Impact of 3 different Collaborative Filtering Algorithms on Product Views was conducted (Figure 17).

41 36 Figure 17: Average Items viewed for 1 Control Group and 3 Treatment Groups In the study, the Average Number of Items viewed for a Control Group, to whom no CF-algorithm was applied, scored For a treatment group where Purchased- Based CF ( People who purchased this item also purchased ) was used, the Average Number of Items viewed increased to associated with a p-value of Error-Likelihood (statistically significant). Therefore, we can conclude that in the little case described, the use of a Purchase-based CF-algorithm has a positive impact on the amount of products views [11]. However, the use of view based CF only increase the product views by score of and was statistically not significant. The recently viewed algorithm implies a decrease of the item viewed by Consequences: Revenue Increase: The use of a Purchased-Based CF-algorithm seems to have a positive influence on the Product Views. The increase of Products viewed by users may therefore imply more potential buyers of products. However, the relationship between Viewers and Buyers was not investigated in the Research Paper study. Cross Selling: If the product views increase, the user is very like to view also ancillary items to his preferred product and will possibly buy them. Up Selling: An increase of Product views can awake the customer s interest to buy a more expensive version of a product, an upgrade or an add-on. Up Selling will be then an additional Revenue for the Business.