Stay Out of My Forum! Evaluatng Frm Involvement n Onlne Ratngs Communtes Neveen Awad and Hla Etzon. INTRODUCTION A growng number of onlne retalers are enablng and encouragng consumers to post revews of the products sold on ther webste. These customer revews often consst of a numerc value (ratng), along wth some optonal text descrbng the revewer s experence wth the product. Recent research suggests that consumers utlze onlne ratngs n makng ther purchase decsons (Chevaler and Mayzln, 00; Senecal and Nantel, 00). However, there s controversy related to the relablty of onlne revews as well as to how well they reflect the opnons of the populaton of consumers. Anecdotal evdence suggests that some of ths nformaton may be based and s sometmes provded anonymously by the product companes themselves (Whte 999; Harmon 00). Some frms have attempted to address nherent bases, wth the hope of ncreasng sales, by flterng customer revews. However, to date there s very lttle research regardng whether onlne retalers should flter customer revews, and when such nvolvement wll result n a busness advantage. In ths paper we examne retaler s nvolvement wth onlne word of mouth posted on ts webste by studyng, analytcally and emprcally, the relatonshp between posted onlne revews and sales. Emprcally, we examne the relatonshp of onlne ratngs wth purchase transactons at a large onlne retaler before and after t changed ts polcy for flterng revews. Specfcally, we examne the mpact of onlne ratngs on sales across two dfferent flterng strateges. Before March, 00, the frm s flterng strategy was to flter out all revews that reflected negatvely on the frm or ther products n any way. Thus the flterng was done by the marketng department, and the goal was to only keep revews that would enhance sales. After March, 00, the flterng was gven to the onlne experence department. The strategy of flterng changed to one of nose reducton. Thus, revews were fltered out only f they were deemed to provde no value (postve or negatve). As such, comments ncludng profantes, or comments not havng to do wth the product were fltered. Our study provdes affrmatve answers to several mportant questons: Are onlne revews correlated wth onlne transactons? Our results provde evdence for the clam that onlne ratngs are assocated wth onlne purchases. Does flterng of onlne revews affect the mpact of these revews on onlne transactons? Onlne retalers have a wde range of approaches to flterng customer product revews, wth many strugglng to fnd the correct balance. The retaler we study n ths paper changed ts flterng strategy and flterng team n March of 00, whch allowed us to emprcally compare the relatonshp of revews wth sales across two flterng strateges: a strategy that flters out most negatve revews, and a strategy that flters only nose. We fnd that a frm s flterng strategy mpacts the relatonshp between average ratng, number of extreme ratngs (postve or negatve) and sales. To address the last queston thoroughly, and examne whether t s optmal for the retaler to flter bad revews, we analytcally model an onlne retaler that sells two competng products. The products are mperfect substtutes, and therefore the demand for a product depends not only on ts prce and ratngs, but also on the substtute s prce and ratngs. We fnd that f the seller compares only two strateges: flterng bad revews and not flterng bad revews across both products, then not flterng domnates when the proporton of bad revews for the more proftable product s small enough relatve to the proporton of bad revews for the less proftable product. The rest of the paper s organzed as follows. Secton presents the data set. Sectons and descrbe our emprcal methodology and present the emprcal results. In Secton we present the analytcal model. In Secton 6 we dscuss the mplcatons of the emprcal and analytcal fndngs, conclude, and descrbe the next steps of ths work.. DATA SET Our data for ths study conssts of ndvdual product characterstcs, purchase transactons, and user revews. These data were collected from a large onlne retaler, and the dates of the data range from Aprl 6th, 999 to February nd, 006. The frm changed ts revews flterng method on March rd 00. The user revews data conssted of an nteger numercal ratng that ranged from (best) to (worst) and an optonal text revew of the product. Before these ratngs are publshed on the onlne retaler s webste, they are frst put n a queue to be assessed by the revew flterng team. As the team goes through the revews, they ether approve or reect the revew. We restrct our analyss to only the approved revews, snce those are the ones vsble to consumers. Table presents summary statstcs for the two perods: before and after the flterng strategy changed.
Table Summary Statstcs Average across products Ratng Varance of Ratng Revews s s Sales /6/99 - /0/0.66 0.0.88 0.0.8.0 /0/0- /0/06.7 0.0.6 0.0.9.078 % Change -% 8% % 0% 6% -% Table shows an ncrease of 0% n the average number of s per product. In addton the average ratng went down by %, reflectng the fact that the retaler stopped flterng revews wth low ratng. We also see that though the average number of s per product ncreased by 6%; ths ncrease s much smaller than the ncrease n the average number of s. It s not surprsng that sales went down, because n the frst perod we consder sales from years whle n the second perod we consder sales from only 0 months. We run two separate emprcal models: ) usng months of sales data from before the change n the flterng strategy along wth revew data posted between /6/99 and /0/0; and ) usng months of sales data from after the flterng change along wth revew data posted between /6/99 and /0/06. We lmt the before sales perod to month to control for the dfference n tme frame of avalable data before and after the strategy change. We also tested the model usng the full sales data set before the change, whch encompasses ust under 6 years of data. The results for ths last model are smlar n sgn and sgnfcance to the results of the before model that used only months of data.. EMPIRICAL METHODOLOGY The goal of ths study s to examne the relatonshp between purchase actvtes and the revew nformaton, before and after the frm changed the revew flterng strategy. The dependent varable s the total amount spent per product. The ndependent varables nclude: volume, measured as total number of revews per product (Godes and Mayzln, 00), valence, measured as average ratng per product, and densty, measured as number of revews per product dvded by number of transactons per product (Dellarocas et al, 00),. The theory behnd volume s that the more consumers dscuss a product, the hgher the chance that other consumers wll become aware of t. Densty takes ths theory one step further by normalzng the number of revews of a product by the number of transactons. The theory behnd valence s that the average ratng s a proxy for the qualty of the product. Our ndependent varables also nclude: number of extreme negatve or extreme postve (e.g. the number of revews that were the worst ratng ( on a to scale), or the best ratng ( on a to scale)) (Chevaler and Mayzln, 00), and varance of the ratngs per product, whch approxmates the range of dsagreement between revewers. To assess the mpact of products whch are potental complements, we also nclude category_valence, whch s the average ratng of products classfed wthn the same sub-category. Beyond the ratngs gven by the revewers, there mght be addtonal nformaton contaned n the revew s text. Codng the actual text n the revews has been shown to produce rather nosy results (Godes and Mayzln 00). Thus, n accordance wth pror research, we nclude ust revew length, measured as number of words; Pror lterature has suggested that a longer revew may suggest a more mxed revew (Chevaler and Mayzln, 00). Lastly, we categorze the products nto four groups of prces ($0-$0, $0-$0, $0-$0, $0-$0,000) usng three dummy varables Consder a product that s sold on the retaler s webste. The product belongs to the category and the department k. Purchase actvtes are set up as the followng: spendng Volume Category _ Valence Varance ( ( k)) = β 0 ( ( k)) Valence num _ rev _ 6 num _ rev _ RevewLength ProductPrce 7 8 9 Densty 0 ( prce _ category _ dummy ) + ε (()) k () ε Where ( ( k )) s the random error for product. We assume that the coeffcents of the average ratngs vary wth the standard devaton of the correspondng ratng, because the average ratng s less nformatve when there s more varaton n the ratngs. That s, we specfy β ' ( = β + γ std _ dev _ t). In addton, some unobservable characterstcs across categores and departments are hypotheszed to randomly nfluence the ntercept β 0-k. That s, β 0-((k)) =β 0 - () +ε k, and β 0() = β 0- +ε (k). Thus, β 0-((k)) =β 0 + ε k +ε (k). After these adustments, we estmate the functon: spendng Volume ( Varance )* Category _ Valence ( Category _ Varance )* Category _ Valence k (()) = β 0 k (()) β β Valence + γ Valence β + + + + γ Varance num_ rev _ 6 num_ rev _ RevewLength ProductPrce 7 8 9 Densty ( ) () 0 prce category dummy + ε k (()) + ε k () + ε k We also used number of transacton per product as the dependent varable, but due to space lmtatons, we do not nclude those results n ths draft.
As descrbed above, we run ths model wth months of sales data before the change n flterng strategy and months of sales data after the change n flterng strategy.. EMPIRICAL RESULTS The estmaton results for the above specfed model are as follows: Table Purchase amount before and after the flterng strategy change Negatve Revew Flterng (Before) Nose Reducton Flterng (After) Effect Before (β) t value (Pr> t ) After (β) t value ( Pr> t ) β 0-9.6 -.8 (0.6).676 0. (0.69) Volume.7*. (0.0) -0.78 -. (0.6) Valence.**.8 (0.00) -.8* -. (0.0) Category_Valence 7.098*.76 (0.08) 6.0 0.99 (0.0) Varance -0.997-0.79 (0.) -.6** -.99 (0.00) revew_n_.***.7 (0.000) -.88*.00 (0.0) revew_n_ -.68** -.6 (0.0).7*.88 (0.068) Revew_length -0.06-0.6 (0.6) 0.0 0.7 (0.7) Product_Prce 0.0*** 0. (<0.000) 0.*** 0.76 (<0.000) Densty -.8** -.8 (0.00) -.97* -.9 (0.0) dummy_.6 0.9 (0.).888.6 (0.08) dummy_ 9.*.6 (0.09) 9.9*. (0.00) dummy_.**.6 (0.00).8***.6 (<0.000) ***: p <0.00, **: p < 0.0, *: p<0.0 There are two man dfferences that we see n the results for before and after the change n flterng strategy. Before the change the number of ratngs s postve and sgnfcantly assocated wth the purchase amount, where as the number of ratngs s negatve and sgnfcantly assocated wth the purchase amount. We expect that ths some what counterntutve result s due to ther beng a small proporton of s when the marketng department was n charge of the flterng, therefore the presence of s actually added credblty to the valence score, and therefore was assocated wth an ncrease n purchasng. Smlarly, we expect that the negatve correlaton between the number of s and the purchase amount s due to the percepton of revew bas ncreasng wth the greater number of extreme postve revews. As a result, consumers were lkely to suspect of the revews, and therefore purchase amount was lower for products whch appeared to have more bas n ther revews. Notce that valence s postve and sgnfcantly assocated wth sales before the flterng change. That s, although extreme ratng values have the opposte effect than one would expect, sales stll ncrease wth the average ratng. Ths supports our argument that the presence of a small number of s gave valdty to a product s ratngs. In contrast, we see that after the flterng strategy changed to one of nose reducton the mpact of the extreme value revews reversed: the number of ratngs s negatve and sgnfcantly assocated wth the purchase amount, whereas the number of ratngs s postve and sgnfcantly assocated wth the purchase amount. After the change, when the frm allowed negatve revews to surface, consumers seem to perceve the revew nformaton provded as accurate, rather than bas. Therefore, snce consumers perceve the revews to be revealng true nformaton, negatve revews are havng the expected negatve effect, and postve revews are havng the expected postve effect. In addton, when the extreme value revews have ther expected effect, the valence and varance are negatve and sgnfcantly assocated wth the purchase amount. A greater varance ndcates more dsagreement among the posted ratngs. Thus consumers lkely perceved greater varance as greater rsk, resultng n a negatve assocaton wth purchase amount. The negatve assocaton of the valence score after the flterng change s a very nterestng result; we suspect that n the presence of greater varance of scores (extreme scores of both types are beng dsplayed) consumers are focusng more on the extreme value ratngs than on the valence. Thus even f the average ratng score s lower, f there are more revews, and less revews, the consumer wll be more apt to purchase that product. We further nvestgate ths potental swtch n consumer assessment through our analytcal model. Our emprcal results show a sgnfcant postve effect of category_valence on sales when the frm was flterng negatve revews (before). Ths result suggests that average ratngs of complmens or substtues have effect on sales of a product. The analytcal model further nvestgates the mpact of smlar products that are mperfect substtutes.. ANALYTICAL MODEL An onlne retaler offers two competng products. Retal frms sell products produced by other companes, and often can not control the products posted prces. However, retalers can control whch revews are beng dsplayed on ther webste. That s, they can flter the revews submtted by consumers. To smplfy the model, and to focus on the affect of the flterng
strategy, we assume that consumers who have purchased a product can submt one of two ratngs: a good ratng, G, and a bad ratng, B. A bad ratng has a score of - and a good ratng has a score of. Table summarzes the notaton. G the number of good ratngs for product B the number of bad ratngs for product N the total number of revews for product p prce of product to consumers cost of product (for the retaler) c S average ratng for product π (x,y) the seller s proft when there are x (y) bad revews for product () Table : Notaton The retaler can flter the revews. We assume that the retaler uses the same flterng strategy for both products (though, as shown later, ths mght not be optmal) and compare the proft from usng the followng two strateges: ) Strategy F: flter/don t show bad revews; and ) Strategy NF: Show all revews. The products are mperfect substtutes and the demand functons are gven by: D ( B, B ) = A bp + dp + α S δ S () D ( B, B ) = A bp + dp + α S δ S Where S =(G -B )/(G +B ) s the average ratng (score) for product. The frst three terms n the expresson for D gve the demand for the product n a market wth no feedback system. We assume that b>d, so that a product s own prce effect s stronger than the cross prce effect (f the two prces go up by the same amount, the demand for both products goes down). The last two terms n the expresson for D capture the effect of the dfference n the average ratng. The maxmum affect the feedback system mght have on the demand for a product s gven by (α+δ). That s, due to the feedback system, sales of product () can ncrease (decrease) by at most α+δ unts, whch happens when all of product s ratngs are good and all of product s ratngs are bad. An ncrease n the demand of product can be attrbuted to an ncrease n ts perceved value (due to a good average ratng). The postve valence of a product attracts consumers who ntally were not gong to buy any of the two products, as well as consumers who ntally (n the absence of a feedback system) would have preferred product. If α =δ, then when the two products have the same average ratng,.e. S =S =S, the demand for each product s the same as when there s no feedbacks at all. Here, the feedback system only transfers demand from one product to the other, but cannot ncrease or decrease total demand. What one product loses n sales s the other product s gan. On the other hand, If α >δ, then when the two products have the same average ratng, the demand for each product dffers from the demand n a market wth no feedback system by ±(α-δ)s. In addton, f the average ratng for the two products ncreases (decreases) by the same amount, the demand for both products goes up (down), and thus total sales can be hgher or lower than the total sales n a market wth no feedback system. The more dssmlar the products are, the smaller the cross affect of ratngs should be,.e. δ decreases. The seller s proft s gven by π(b, B )=(p -c )D +(p -c )D. () We compare the proft of the seller under the two flterng strateges descrbed above, F and NF, for a gven profle of submtted revews (G, B, G, B ) and products margns, (p -c ) and (p -c ). Lemma The retaler s proft ncreases as the number of bad revews for product decreases f and only the rato of the margns (margn for product dvded by margn for product ) s smaller than α/δ. Proof. π ( B, B ) / B < 0 f and only f ( p ) /( p ) < α / δ. () Accordng to Lemma, f δ=α (ratngs can only transfer demand between the products) the retaler should flter bad revews only for the product wth the hgher margn. If δ=0 (ratngs do not transfer demand between products) the retaler should flter bad revews for both products. And fnally, f δ <α (.e. α/δ >) t s proftable to flter bad revews at least for the product wth the hgher margn (because when (p -c )/(p -c )>, that s s the product wth the hgher margn, then necessarly (p -c )/(p -c )<). Notce that as δ decreases t becomes optmal to flter bad revews for both products. Though Lemma gves the optmal flterng strategy when the seller can flter revews selectvely only for one of the products, such a strategy mght not be possble due to the manufacturers response, or mght be dffcult to execute when prces of the products change frequently. Hence, consderng only the two extreme (and homogenous) strateges, F and NF, descrbed above, the frst domnates the latter f and only f =π(0,0)-π(b, B )>0, where: = ( B N ( ( p ) δ ( p )) + B N ( α ( p ) δ ( p )))/( N ) N α. (6)
Clearly f Inequalty () holds for both products- that s f t s optmal to flter bad revews for each of the products when consdered separately, then strategy F domnates strategy NF. If nequalty () holds for only one of the products, t would hold for the product wth the hgher margn. That s f (p -c ) > (p -c ) we would have: (p -c )/(p -c )>α/δ and (p - c )/(p -c )< α/δ. Then strategy NF domnates strategy F f and only f the rato (B /N )/(B /N ), where s the product wth the hgher margn, s larger then a threshold value. Proposton summarzes our fndngs. Proposton. NF domnates F f and only f (p -c )/(p -c )>α/δ, where s the product wth the lower margn, and B ( ) ( ) / N α p δ p >. (7) B / N δ ( p ) α ( p ) The RHS n (7) s postve when α/δ > because (p -c )/(p -c )>α/δ and (p -c )>(p -c ). Accordng to Proposton, f the product wth the hgher margn, n ths case product, has a small proporton of bad revews when compared wth the product wth the lower margn, then the seller s better off not flterng at all to flterng for both products. If δ =α (the products are close to beng perfect substtutes), NF domnates F f and only f (B /N )>(B /N ). If the products have the same proft margn and α/δ >, then F always domnates NF. However, what f by flterng all bad revews the retaler actually changes the structure of the demand? Next we assume that f consumers do not observe any bad revews, then nstead of usng the average ratng, they compare the number of good revews across the competng products. In such a case the demand functons are gven by: D ( B, B ) = A bp + dp ( αs δs ) + ( β )( αas δas ) D ( B, B ) = A bp + dp ( αs δs) + ( β )( αas δas) (8) Where β= f B +B >0 and β=0 otherwse, and AS =G /(G +G ) s the Adusted Score for product. Alternatvely, we can examne the optmal flterng strategy assumng consumers suspect bas for a product that has only postve revews, even f other products have bad revews. In ths case ( β S + ( β ) AS ) δ ( β S + ( AS ) D ( B, B ) = A bp + dp + α β ) (9) Where β = f B >0 and β =0 otherwse. Gven that consumers adust ther use of the feedback nformaton based on whether they perceve t to be bas or not, s t stll optmal for the retaler to flter bad revews for the product wth the hgher margn? Under what condtons flterng bad revews for all products domnate not flterng for both? Ths would be the next step n our analyss 6. DISCUSSION AND CONCLUSION Our ntal results suggest several mportant ssues. Frst, our emprcal results show postve support that onlne ratngs are sgnfcantly assocated wth actual transactons. Whle prevous studes have nferred a relatonshp between ratngs and sales through sales rank (Chevaler and Mayzln, 00), we show such a drect lnk through actual transacton data. In addton, our paper s the frst to begn to examne retaler nvolvement n flterng onlne ratngs. Our ntal results suggest that consumers use dfferent metrcs of onlne ratngs to assess the nformaton dependng upon what revew nformaton s presented to them. In the stuaton where the retaler fltered out negatve revews, valence score (average ratngs) of the product and of the product category were sgnfcantly assocated wth purchase amount. In addton, consumers seemed to use extreme ratngs as a gauge of degree of bas, rather than as an approxmaton of product qualty. However, n the stuaton where the frm dd not flter out negatve revews, consumers appear to use the extreme postve and negatve ratngs n makng ther purchase decson. We follow these emprcal fndngs wth an analytcal assessment of optmal frm flterng strateges. Interestngly, our analytcal model fnds that retalers beneft when they can selectvely flter revews n favor of more proftable products, suggestng that the bas n the revews wll be less vsble to the consumer when t s not unformly appled across all products. However, when the retaler chooses to ether flter or not across the entre webste, then t s sometmes optmal not to flter at all. In the next steps of our analyss, we plan to analytcally model what we observe n our emprcal results: that buyers are adustng ther percepton of revews, and therefore that ther demand functons can change based on the retaler s flterng strategy. REFERENCES. Dellarocas, Chrs, Awad, Neveen and Xaoquan (Mchael) Zhang. (00) Explorng the Value of Onlne Revews to Organzatons: Implcatons for Revenue Forecastng and Plannng, Proceedngs of the Internatonal Conference on Informaton Systems, December, Washngton, DC.. Chevaler, Judth A., and, Mayzln, Dna (00). The Effect of Word of Mouth on Sales: Onlne Book Revews. Yale SOM Workng Paper No's. ES- 8 & MK-.. Godes, Davd, and, Mayzln, Dna (00). Usng Onlne Conversatons to Study Word of Mouth Communcaton. Yale SOM Workng Paper No. MK-; Harvard NOM Workng Paper No. 0-; HBS Marketng Research Paper No. 0-0.. Harmon, Amy (00). Amazon Gltch Unmasks War of Revewers. The New York Tmes. New York, February.. Senecal, S. and Nantel, J. (00) The Influence of Onlne Product Recommendatons on Consumers Onlne Choces. Workng Paper. 6. Whte, E. (999). Chattng a Snger Up the Pop Charts. The Wall Street Journal October.