Errors in Customer Satisfaction Surveys and Methods to Correct Self-Selection Bias

Transcription

1 Qualty Technology & Quanttatve Management Vol. 8, No. 2, pp , 2011 QTQM ICAQM 2011 Errors n Customer Satsfacton Surveys and Methods to Correct Self-Selecton Bas Govanna Ncoln and Lucana Dalla Valle Department of Economcs, Busness and Statstcs, Unversty of Mlan, Italy (Receved November 2009, accepted July 2010) Abstract: Ths paper provdes an overvew of the man types of surveys carred out for customer satsfacton analyss accordng to the dmenson of the frm and to the data collecton method. Dfferent errors can be generated by these methods, causng bases n the results. We focus on the self-selecton error and we employ three methods to correct the bas assocated to t. A smulaton underlnes the performances of the three methodologes. Keywords: Customer satsfacton surveys, Heckman two-step procedure, herarchcal Bayesan approach, propensty score matchng, self-selecton bas. 1. Introducton S tatstcal surveys need complex procedures borne out of organzatonal efforts as well as economcal efforts that n many cases may appear too heavy for the frm. The survey may nclude every customer of the frm (census survey) or just a part of them (sample survey). The knd of survey chosen depends on many factors concernng the type of frm and ts customers [11]. Frms defne themselves through terrtoral presence and the market they operate n. Hence, these three types of frms: bg frms: they operate natonwde n an olgopolst regme or more generally n "mperfect competton". Such frms, for example, are the ones provdng telecommuncatons, electrcty, transportaton, bankng, fnance, nsurance and health care servces; medum frms: they generally operate n a smaller terrtory context, but f they operate natonwde, they are n "perfect competton" (local transports, tour operators, prvate clncs ); small frms: they locally develop a wde range of servces n a clmate of "perfect competton". Bg frms often carry out sample surveys wth bg probablstc samples, whlst medum or small frms prefer to carry out census surveys. If we focus our attenton on the types of customers, we can certanly dstngush the frms who offer servces to other frms (busness to busness) from the ones whch offer servces to the fnal consumer (busness to consumer). In the frst case, the lst of customers (n general not a huge one) s known and the survey s qute often a census survey. Conversely, n the second case, the lst of customers s not necessarly known and the survey can only be a sample survey. Consderng these two factors, have a look at Table 1.

2 168 Ncoln and Valle Table 1. Types of surveys for types of frms and customers. Frm s dmenson B2B B2C small C C/S medum C C/S large C S C= census survey; S= sample survey. However, the usefulness of a survey (be t a census or a sample survey) depends on ts relablty, namely the proxmty of the results of the survey to the real ones. If the relablty s poor the results are nevtably based and at the end the whole survey may be useless. Poor relablty may have many causes. One of these s the presence of errors. Lterature dentfes dfferent knds of errors. Some of these are related to the methods used to gather statstcal data. Frms may use varous methods accordng to the types of customers, the tmes and the costs nvolved. The fnal goal of ths work s to hghlght the features and the problems of the Customer Satsfacton Surveys (CSSs) regardng the types of frms that provde servces, the types of customers, the data gatherng methods carred out throughout the surveys and, consequently, the lkely errors connected to them (second secton). Our attenton wll be focused on a seldom studed knd of error (at least n ths knd of survey) and on the methods provded by lterature to correct t. These methods are generally borne out of dfferent ams or n dstant contexts and need to be aptly modfed n order to be appled n CSSs (thrd secton). The fourth secton shows a smulaton we have made n order to evaluate the performance of the appled methods and the changes we have ntroduced. Fnally, we gve concludng remarks. 2. Errors and Data Collecton Methods n CSSs Accordng to the lterature the relablty of the results of a survey depends on the exstence of errors, that can be non-samplng errors or errors related to data collecton methods Non-samplng Errors The errors that mght modfy the results of a sample survey and a census survey are known as coverage errors, measurement errors, unt non-response errors and self-selecton errors [9, 14]. In partcular: coverage error. It s observed when the number of customers of a frm (target populaton) and the lst of them (frame populaton), from whch nformaton are generally taken to nvolve the customers n the survey, do not concde; measurement error. It s gven by the dfference between the real value of an tem related to a surveyng unt and the value observed. Ths knd of error has been frequently connected to the presence of an ntervewer or to data chargng; unt non-response error. We have t when the selected unt does not answer or does not fll n the questonnare form. Ths non-response may be caused by the nablty to reach the customer or by the refusal of the customer, who does not want to jon the survey; self-selecton error. It s observed when no selectons have been made before the survey and the person ndependently decdes to jon the survey. If no selectons are made the survey s a census survey. Though, self-selecton determnes a certan amount of non-responses. The self-selected who have provded answers form a non-probablstc sample of the populaton.

3 Errors n Customer Satsfacton Surveys and Methods 169 Amongst the enlsted errors the measurement one may appear, n CSSs, the least relevant because n many crcumstances the questonnares are flled n by the customer hmself and because questons and answers are generally qute easy. More relevant s the coverage error whch, along wth the measurement one, may be attrbuted to the frm or to the way the survey has been carred out. The coverage error mples the exstence of a frame. We meet ths knd of error when a frm does not have a lst of customers or does not have an updated verson of t. The exstence of ths frame allows to make a census survey or a sample survey wth a probablstc sample. On the other hand, f we do not know the frame we wll not be able to carry out a census survey, whlst the sample survey wll generally use a non probablstc sample (a quota sample for example). Though, we may also thnk of a nearly-probablstc sample wth a perfectly suted samplng scheme. Conversely, the last two errors can be attrbuted to the ntervewee/customer. As far as the non-response s concerned, t s thought that ths knd of error s assocated to probablstc samples. The researcher knows the features of the people who do not answer; as a matter of fact, the weghtng methods suggested n presence of non-responses are based on models n whch auxlary varables are employed. These are qute known to both answerng and non-answerng people. What we do not know are the reasons behnd the non-responses. The reasons behnd non-responses n ths type of surveys are dfferent from those behnd surveys about senstve matters (as for example dscrmnatory dseases or prvate lfe facts), that may make some ntervewees feel uneasy. Indeed, causes for non-response n CSSs can be sheer boredom and reluctance of people to fll n the questonnare due to the number of CSSs one s asked to fll n everywhere (n hotels, after organzed tours, at conferences, after the completon of a workshop/course/study programme, etc.), often very poorly desgned questonnares, and lack of obvous mprovement n servce processes followng the mplementaton of a CSS. The self-selecton error, nstead, s assocated to the ndependent choce of the customer who decdes to take part n the survey. Ths s possble only f we carry out a census survey. As far as ths survey s consdered the questonnare s sent to all the customers, who then ndependently choose to fll or not fll n the form. Self-selected subjects are a non random sample of the populaton, as well as not self-selected subjects are a non random sample of the populaton. The ntervew refusal n a sample survey s consdered as unt non-response, whle the ntervew refusal n a census survey s called not self-selecton Errors Related to Data Collecton Methods Let us now see what are the most common data collecton methods and the types of errors assocated to them: Face-to-Face Intervewng (F2FI). Ths method s used when a survey has to be made because the lst of customers s not known yet. It s frequently used by moblty servces because for them t s possble to ntervew the customers only when they actually use the servce. It s generally carred out n a sample survey context. Ths s a very expensve method; here the presence of the ntervewer may produce a bas caused by measurement errors. Computer Asssted Telephone Intervewng (CATI). Ths method s frequently used and not very expensve. It s possble to employ ths method f a lst of customer already exsts. It can be used n census surveys and sample surveys wth probablstc samples. It s possble to keep a close watch on measurement errors, whlst coverage errors (f the lst of customer s not updated) and non-response ones are more frequent. Computer Asssted Web Intervewng (CAWI). It s the modern verson of the now rarely used mal ntervew. Ths method s correctly appled f extended to all those customers who

4 170 Ncoln and Valle have an e-mal address known to the frm. Beng the survey a global one, t s possble to verfy a self-selecton error. Conversely, f not every customer has an nternet access, the choce of ths method mples the presence of coverage errors. Web-Survey (WS). Web-Surveys may be of varous types [15] and get more wdespread every day. In the same way the number of people havng an nternet access ncreases every day. The web approach most frequently employed n the CSSs are onlne questonnares we can easly fnd on the frm webste or questonnares that pop up every x vstors on the frm webste. In the frst case (WS1), the fllng n of the questonnare, open to everyone, depends on the customer's wll. Hence, we have a self-selecton. In the second case (WS2), as a systematc sample s used, there mght be a non-response error. If every customer of the frm have an nternet access (e.g., an onlne bank), or f the survey s only destned to the vstors of the ste, a coverage error s not present. On the other hand, f not every customer of the frm s an nternet surfer the coverage error adds up to the errors we have already mentoned. Open Surveys (OS). The survey conssts n gvng the customer a paper questonnare at the end of the servce. The fllng n of the questonnare and the handng n of the flled n questonnare to the frm depends on the customer. Ths survey s very easy, t s not supported by any knd of organzaton and the self-selecton error s qute frequent. A summary of the surveyng methods wth the errors assocated to them s reported n Table 2. Table 2. Data collecton methods and errors. Measurement Coverage Non-response Self-selecton F2FI CATI CAWI WS1 WS2 OS for samplng survey; for census survey. In ths paper our focus s on census surveys n whch self-selecton errors are qute frequent. The presence of ths error alters the survey tself, whch becomes a sample survey. As a matter of fact, after the fllng n procedure, we do not have N flled n questonnares (N s the populaton sze), but n questonnares where n N. What we have n our hands s a n sze non-probablstc sample. It s well known that wth ths knd of sample t s not allowed to apply the nferental methods as far as the desgn-based approach s concerned. Ths may only be possble f the bas caused by the self-selecton s n some ways elmnated. Alternatvely, other nferental approaches are appled. 3. Methods to Correct Self-selecton A sample s based (not representatve of the populaton t s related to) when the sample dstrbuton of the varables descrbng the structure dffers from the same varables n the populaton. The bas of these varables generally mples the bas of the varables studed. It s lkely for the self-selectoned sample to be based because the self-selecton process s not a

5 Errors n Customer Satsfacton Surveys and Methods 171 random one. Ths means that t does not ndfferently nvolve every subject of the populaton. Conversely, t s thought to be qute typcal of those people who have certan characterstcs. For example, t may be thought that the gender and the age of a customer mght nfluence the level of satsfacton of a servce. If n a self-selected sample women and youths are manly observed, when compared to the dstrbuton of the populaton, the evaluaton of the satsfacton of that servce wll turn out to be altered. The methods that wll be shown here have a specfc end: to dfferently correct the bas caused by self-selecton. The frst method suggests to evaluate the satsfacton of the non-answerng people n probablstc terms. The second consders two equatons ted together by a latent factor that allows the mssng data assocated to the non-answerng subjects to be correctly estmated. Fnally, the thrd method proposes a lnear model whch s supposed to be able to show the relatons between the varable examned and a group of varables thought to have generated the observed data. If the model s easly adaptable to the observed data, t wll be possble to predct the values of the studed varable as far as those yet to be self-selected are concerned The Propensty Score Matchng The dea behnd ths method was born n the Seventes and was the work of Rubn [20]. Though, t has been tested n the Eghtes by Rosenbaum and Rubn [18] n an expermental context to evaluate the effect of a health treatment. For ths purpose we want to compare the outcomes of the nterested varable observed on a group of ndvduals subjected to a treatment to the outcomes that should have been observed was the treatment not appled. Obvously, t s not possble to make such a comparson on the same group. It s then necessary to choose a control group and compare the two outcomes. Marked wth T a dchotomous varable that takes a t 1 value (f the person receves the treatment) or a t 0 value (f the person doesn't receve the treatment), and marked wth an x a vector of covarates, called pre-nterventon varables, we call propensty score (ps) the probablty of every person to receve or not receve the treatment condtoned by the vector of the covarates x. Ths probablty s: p( x) Pr( t 1 x ); that s, 1 p( x) Pr( t 0 x) and s estmated wth the multvarate logstc regresson, wth pre-nterventon varables as regressors. ps 1 s the comparson term between the outcomes of the people from the two groups and allows us to do a "matchng" 2 between them; thus, t wll be possble to assocate to a person of the control group the outcome observed on the person of the group that has been treated and has got the same vector x, that s to say the same value of ps. Ths method has been appled n dfferent contexts, even recently, to correct the bas due to a self-selecton error n the web-surveys [1, 2, 13]. In the context of the CSSs, the group undergong a treatment s represented by those who, by self-selecton, have flled n the questonnare, whlst the control group s formed by those "not self-selected" ndvduals. Once calculated the propensty scores as far as the subjects of the two groups are concerned, the same standard of satsfacton of a self-selected person who has flled n the questonnare and has got the same ps wll be attrbuted to an ndvdual of the control group who has not flled n the questonnare. The propensty score matchng method allows us to correct the self-selecton error and to estmate the standard of satsfacton of those who have not flled n the questonnare. It s based on the assumpton that the subjects jon the research ndependently and on the assumpton that self-selecton does not depend on the target varable (Condtonal Independence Assumpton). Ths mples that the pre-nterventon varables, on whch the 1 The advantages connected to the use of ps s twofold: on one hand, a multdmensonal problem s reduced to a one-dmensonal one. On the other hand, the rsk of fndng more than one person wth the same ps s eluded, snce the propensty score s a monotone functon of the dscrmnant score. 2 The matchng procedure occurs utlzng one of these procedures: Nearest Neghbor Matchng, Calper Matchng, Mahalanobs Metrc Matchng, Stratfcaton Matchng, Dfference-n-Dfferences Matchng [18].

6 172 Ncoln and Valle matchng s nfluenced, must not affect the self-selecton [18]. In the context of CSSs, pre-nterventon varables could be dfferent accordng to the type of servce receved The Heckman Two-step Procedure Heckman [10] proposed a two-step procedure whch allows to estmate the values of the varable of nterest for not self-selected ndvduals. Heckman's dea s based on a model made by two equatons related to each other: the substantal equaton and the selecton equaton. The substantal equaton for ndvdual (for 1,..., N) s: Y1 X1 1 U1, (1) where Y1 represents the contnuous varable of nterest, that n ths case represents the satsfacton level. Supposed that one seeks to estmate equaton (1) but data are mssng on Y1 for N n observatons (the number of not self-selected ndvduals), we defne the selecton equaton, that for an ndvdual s the followng: W X22 U2, (2) where W s an unobserved latent random varable such that W 0 corresponds to self-selected ndvduals, whle W 0 corresponds to not-self selected ndvduals. Equaton (2) s the expresson of a probt model, whch can be alternatvely wrtten n the followng form: Prob( Y2 1 X2) F ( X2b2 ), (3) where Y2 s a dchotomous random varable takng value 1 for self-selected and 0 for 3 not-self selected ndvduals. For both the prevous equatons we make the followng assumptons: X1 s the -th vector 1 ( K 1) of varables known for all N subjects, X2 s then -th vector 1 K of varables known for all N subjects, 1 s a ( K 1) 1 vector of parameters, 2 s a K 1 vector of parameters, U1 and U 2 are the vectors of resduals. EU ( j ) 0, EU ( ju j'' ) jj ' for and EU ( ju j ) 0 for, where 1,..., N and j 1, 2. The jont densty of U1 and U 2 s denoted by hu ( 1, U2 ) and we assume that t has a bvarate normal densty. Hence, a relatonshp between equatons (1) and (2) exsts. Ths relatonshp s reflected n a non-zero correlaton between the error terms of the equatons for the same subject. If such correlaton s present, we cannot get robust estmates of the substantal equaton wthout takng the selecton process nto account [21]. Heckman's method allows the ntroducton of a control factor, whch lnks equatons (1) and (2) and expresses the underlyng characterstcs of customers n ther choce of beng self-selected [10]. 4 In other words the method allows us 3 If the latent varable W 0 then Y 2 1 (correspondng to self-selected ndvduals), whle f W 0 then Y2 0 (correspondng to not self-selected ndvduals). 4 Summarzng the Heckman procedure, t bascally takes the followng steps. Estmate the parameters 2 of the selecton equaton (2) usng a probt analyss. The resduals of the selecton equaton U 2 are used to calculate, the selecton based control factor, computed as the nverse of Mll's rato [12], the ndcator of the hdden motvatons of self-selecton.

7 Errors n Customer Satsfacton Surveys and Methods 173 to estmate the level of satsfacton of the not self-selected customers wth (1), that takes nto account, through (2), the self-selecton factor and the exstence of a correlaton between the resduals of the two equatons. Though, the Heckman method assumes that the varable observed s a contnuous one: ths assumpton s not verfed n a customer satsfacton context because the level of satsfacton s taken wth a categorcal varable. Thus, to respect ths assumpton a transformaton of the varable consdered s needed. For ths reason t s possble to use the Nonlnear Prncpal Component Analyss [8] or the satsfacton coeffcents of the Rasch Analyss [4, 5, 16] The Herarchcal Bayesan Approach Wth ths approach the value of the consdered varable Y on the -th unt of the populaton s not fxed but a realzed value of a random varable Y ( 1,..., N). Bayesan methods have become wdespread n customer satsfacton surveys. Ths approach ams to get estmates for the varable of nterest when the survey s unrelable because of data unavalablty. Ths technque, lke the prevous ones, calculates estmates usng nformaton that are supposed to be correlated to the varable of nterest. As t s well know n the lterature, ths technque was tradtonally developed and employed n the small area estmaton feld, as explaned n [17]. Indeed, ths s a very powerful tool when dealng wth lack of data [22]. Here we employ t successfully n CSSs, where we estmate the parameters of nterest for the ndvduals who dd not answer to the questonnare through the nformaton provded by covarates, known for every subject. Among Bayesan methods, we employed the herarchcal Bayesan model as formulated by Fay and Herrot [6]. Suppose one s nterested n estmatng the characterstc Y for every subject, and that the auxlary data are known for each ndvdual (for 1,..., N). Indeed, very often Y are not avalable for all the ndvduals,.e. n sample surveys. The Fay-Herrot methodology conssts of the lnkng model and the samplng model. The lnkng model s the followng: Y X U, (4) where U ~ N(0, ), X s a 1 K vector of auxlary varables and s a K 1 vector of parameters. The lnkng model (4) s merely a mxed lnear model where are fxed effect coeffcents, accountng for the effects of the auxlary varables X, vald for the entre populaton, whle are random ndvdual specfc effects [19]. The samplng model s 2 U Yˆ Y e, (5) where Yˆ s the estmate of Y and e Y ~ N(0, ), beng the samplng varance, whch s typcally assumed to be known. The herarchcal Bayesan model s descrbed by the followng equatons YY ˆ, ~ NY (, ). (6) Y 2, ~ N( X, 2 ). (7) The control factor s used as an addtonal covarate (the ( K 1)-th covarate) n the substantal equaton (1).

8 174 Ncoln and Valle In absence of pror knowledge we opted for flat prors for j ~ N (0,10 ) 2 1 (where j 1,..., K ) and ~ (0.001, 0.001), as suggested for example by Gamerman and Lopes [7]. In the herarchcal Bayesan model, nference on Y s straghtforward and computatonally feasble by usng MCMC based standard methods and all the model uncertanty sources are smultaneously accounted for n the estmaton process. Once the parameters are estmated, for 1,..., n we perform the herarchcal Bayesan model by means of MCMC methods and then, for n 1,..., N, we estmate Y through Yˆ wth the posteror predctve dstrbuton. 4. Applcaton In ths work we apply the Heckman, Bayesan and Propensty Score Matchng (PSM) methods to a dataset of a Customer Satsfacton (CS) samplng survey of 4561 unts. In partcular, we use the data collected n 2004 through a F2FI survey by a real arlne company, whch here (to hde ts dentty) we call "Best Flght". The customers gave personal nformaton as well as opnons about the servce. The 4561 observed unts, that were a sample for the Best Flght company, are consdered as the populaton for our applcaton. In the smulaton (whch s descrbed n secton 4.2), we choose a non random sample from ths populaton. In order to mplement the smulaton, we assume that the data are of good qualty. Concernng personal nformaton, we focus on the varables: FREQCLASS, CLUB, NATION, AGE, GENDER. Varable defntons are gven n the Appendx. Regardng the opnons about the servce, we analyze 10 varables ndcatng the servce dmensons and 4 varables ndcatng the overall customer satsfacton, for a total of 14 varables, each of them measured accordng to a Lkert scale (1=extremely satsfed, 7=extremely dssatsfed). The 10 varables of the servce dmensons are: BOOKING, CHECKIN, TRANSFER, LOUNGE, DEPARTURE, CABIN, MEAL, ENTERTAINMENT, DUTYFREE, CREW. The 4 varables of the overall customer satsfacton are: EXPERIENCE, VALUE, RECOMMENDATION, REPEAT. See the Appendx for the defntons of the varables. We remark that the analyss of CSSs s dfferent from other surveys for the types of varables employed, that are manly categorcal, often measured on a Lkert scale. Ths has to be accounted for n the mplementaton of the statstcal methodologes to the data Nonlnear PCA In order to apply the Heckman method, we need to determne the dependent varable Y, ndcatng the overall customer satsfacton for each ndvdual, thus summarzng the 4 varables of the overall satsfacton. In fact, one of the requrements of Heckman model s that the target varable Y 1 n the substantal equaton (1) has to be a quanttatve random varable. Snce generally n CSSs the varables are categorcal, our frst step s the transformaton of these varables nto one quanttatve ndcator. In the Bayesan approach, nstead, a quanttatve dependent varable s not strctly requred, snce we could formulate a dfferent herarchcal model even for categorcal varables. The Propensty Score Matchng can be mplemented wth categorcal target varables and contnuous ones as well. However, n order to make comparsons, we use a quanttatve response varable for all the three models. To compute the quanttatve dependent varable Y, we use the Nonlnear Prncpal Component Analyss (Nonlnear PCA), belongng to the class of Nonlnear Multvarate 6

9 Errors n Customer Satsfacton Surveys and Methods 175 Analyss [8]. Ths technque s an exploratory analyss, sutable for reducng the dmensonalty of ordnal varables. In our case, Nonlnear PCA computes optmal quantfcatons preservng categores order wthn each varable. We are nterested just n one dmenson, correspondng to the overall satsfacton varable Y. Nonlnear PCA computes the mnmum of a loss functon ls ( ; q1,..., q m ) under varous normalzaton condtons, as descrbed n [15]. The soluton of ths mnmzaton gves us the Y overall satsfacton scores for each ndvdual. Computed object scores are fnally used as the dependent varable Y, where the lower the value of Y, the hgher the satsfacton of the ntervewed customer Results Nonlnear PCA allow us to summarze the nformaton brought by the 4 overall satsfacton qualtatve varables nto the quanttatve varable Y. We employ ths varable as the quanttatve response n the Heckman, Bayesan and PSM methods, where the 10 servce dmenson varables play the role of ndependent varables n the frst two methods, whle they represent the pre-nterventon varables n the thrd method. In order to obtan one customer satsfacton ndex Y through the Nonlnear PCA technque, some condtons are to be verfed. In partcular, the frst egenvalue of the Nonlnear PCA soluton has to be much hgher than the others and Cronbach s, a measure of the relablty of the scale lyng between 0 and 1, has to be as close as possble to 1. Moreover, the component loadngs should have all the same sgn for each varable. Table 3 shows that the frst egenvalue s much hgher than the other (3.028), thus the one-dmenson soluton fts well the data. Cronbach's s 0.893, very close to 1, denotng that the Nonlnear PCA s an approprate methodology. Table 3. Summary of the Nonlnear PCA model. Dmensons Cronbach s Egenvalue Total Table 4. Component loadngs for two dmensons. Component Loadngs Dmenson 1 Dmenson 2 EXPERIENCE VALUE RECOMMENDATION REPEAT Table 4 lsts the component loadngs of Nonlnear PCA. For each varable, the correspondng value for the frst dmenson s always postve, ndcatng a good ft of the methodology to the data. These component loadngs represent the weght of the varable n the determnaton of the overall customer satsfacton ndcator. The hgher s the value of the component loadng, the hgher s the weght of the correspondng varable n the determnaton of the ndcator. In ths case, the values are all hgh and close to one, denotng that all the four consdered varables are mportant and well represented n determnng the

10 176 Ncoln and Valle overall satsfacton dmenson. Moreover, t s clear that the factor loadngs of the second dmenson are low and show dfferent sgns. Ths result confrms that the Nonlnear PCA one dmenson soluton s sutable for the data. The hstogram of the target varable Y calculated through the Nonlnear PCA s dsplayed n Fgure 1. Fgure 1. Hstogram of the target varable Y obtaned through the Nonlnear PCA. The fve personal nformaton varables (FREQCLASS, CLUB, NATION, AGE and GENDER) are used n all three approaches as self-selecton varables. Frst of all, we form dfferent groups of ndvduals based on the categores of personal nformaton varables. For example, the varable GENDER generates two sets of subjects: females (38.1%) and males (61.9%). We repeat ths process for each personal nformaton varable. We run some smulatons where each tme one category of one personal nformaton varable corresponds to mssng data for our target varable Y. More clearly, subjects correspondng to one category are consdered as the group of people who do not choose to fll n the questonnare (.e. a smulaton conssts n an applcaton of the models to a dataset where everyone but the frst class passengers were supposed to be self-selected). Dfferent categores of personal nformaton varables correspond to dfferent self-selecton (and not self-selecton) percentages, as explaned n Table 5. Therefore, n each smulaton we estmate the overall satsfacton varable for mssng data Yˆ (not self-selected ndvduals) wth the three methodologes presented. For example, we suppose that the frst class passengers dd not fll n the questonnare (not self-selected), then we estmate the overall satsfacton of these subjects through the Heckman, Bayesan and Propensty Score Matchng models, as explaned n secton 3.

11 Errors n Customer Satsfacton Surveys and Methods 177 Varables Table 5. Personal Informaton varables used for self-selecton. Self- Not self- Classes (not self-selected) selecton % selecton % FREQCLASS Frst Concorde, Frst, Clubworld Concorde, Frst, Clubworld, WTPlus, Worldtraveller CLUB Gold NATION French, German, Japanese Brtsh AGE GENDER Female Once employed the three methodologes, the outputs gve us the target varable Yˆ estmates ( for 1,..., N n), expressng the global customer satsfacton. We make a comparson between the estmated results Yˆ and the observed results Y amng to calculate how close s our estmator from the real value of not self-selected ndvduals (.e. frst class passengers). To reach ths goal, we calculate the estmated Mean Squared Error (MSE), through the followng formula: 1 N n 2 MSE ( Yˆ Y), (8) N n 1 and we calculate the bas of Y as a predctor of Y as: ˆ bas E( Yˆ ) Y. (9) The results of the smulatons are reported n Table 6. Here we see the dfferent percentages of self-selecton (people who answered to the CS questonnare), as llustrated n Table 5 and the results n terms of MSE and correspondng bas value for the Heckman, the Bayesan and the PSM approaches. As we can see from Table 6, the MSEs computed through the Bayesan method are very close to the one calculated accordng to the Heckman procedure, whle the PSM MSEs are hgher. However, f we focus on the bas, we note that the PSM performs better than the other two methods n the majorty of cases. Focusng on the comparson of the Heckman-Bayesan models, we see that the Heckman approach works well only for self-selecton percentages hgher than 50%. The Bayesan approach nstead works well even wth a lower percentage of self-selecton, snce t produces good estmates even wth small samples. However, the propensty score matchng outperforms the Bayesan methods n many cases, even where the sample sze dmenson s small.

12 178 Ncoln and Valle Table 6. MSE and bas for the three methods. Varables Self- Not self- Heckman Heckman Bayesan Bayesan PSM PSM selecton selecton % selecton % MSE bas MSE bas MSE bas FREQCLASS CLUB NATION AGE GENDER Concludng Remarks The lterature referred to statstcal surveys s wdespread, as from the research methodology, as from the sample theory pont of vew. However, there s not a lot of nformaton about CS surveys. Ths paper proposes ndeed a novel analyss about CS surveys, consderng the populaton nvolved (the whole populaton or a part of t), related to the type of company that supples the servce, to the knowledge of the company about ts customers and to the data collecton methods, that are assocated to non-samplng errors. A type of error whch s very common n statstcal surveys s the unt non-response, when the ntervewee refuses the ntervew. However, we need to dstngush between non-responses n sample surveys and non-responses n census surveys. The authors assocate the non-response error to the sample survey, when the subject to be ntervewed s selected, and the self-selecton error to the census survey, where there s no selecton of the ntervewees. Therefore, the not self-selecton n census surveys corresponds to the non-response n samplng surveys. In the lterature the self-selecton error has been less studed than the non-response error. We gve an overvew of the man methods able to correct self-selecton bas (the propensty score matchng, the Heckman two step approach and the herachcal Bayesan model). However, these methods have not been appled to the feld of Customer Satsfacton Survey yet. For ths reason we mplement the methodologes to a dataset of CSSs, n order to llustrate the potental of these technques n ths context. The target varable s the global satsfacton level of the consdered customers. It s derved by four categorcal varables of overall satsfacton through the Nonlnear Prncpal Component Analyss technque. We run some smulatons and we compare the estmated values of the target varable calculated through the three methodologes wth the observed ones. The results comparson shows that the PSM gves smaller bas than the other two methods n the majorty of smulatons. The Heckman methodology performs better wth bg samples, whle the Bayesan model performs better wth small samples. Therefore, when the percentage of self-selected subjects exceeds 50% we suggest the use of the Heckman approach, snce t s computatonally more effcent than the PSM n terms of smulaton tme. However, when the percentage of self-selected subjects s lower than 50%, both the PSM and the herarchcal Bayesan methods are somewhat tme consumng, but n our case the PSM gves the best results. When the number of self-selected subjects s small the estmate of the level of customer satsfacton s very naccurate. However, even f the number of self-selected subjects s hgh,

13 Errors n Customer Satsfacton Surveys and Methods 179 the estmate could stll be naccurate, snce the hgher the number of observatons the larger are non samplng errors. Therefore, the employment of technques amng to elmnate non samplng errors s always suggested to obtan relable estmates, although ths mples a great effort from the frm. Acknowledgements We are grateful to the anonymous referees, for ther nsghtful suggestons that sgnfcantly mproved the paper. The research for ths paper was supported by PRIN References 1. Bffgnand, S. and Prates, M. (2003). Potentalty of propensty score matchng n nference from web-surveys: a smulaton study. Workng Paper, Dpartmento d Statstca, Informatca a Applcazon. 2. Bffgnand, S., Prates, M. and Tonnell, D. (2003). Potentalty of propensty scores methods n weghtng for Web surveys: a smulaton study based on a statstcal regster. Proceedngs of the ISI Conference, Berln. 3. Couper, M. P. (2000). Web survey: a revew of ssues and approaches. Publc Opnon Quarterly, 64, De Battst F., Ncoln, G. and Saln, S. (2005). The rasch model to measure servce qualty. The ICFAI Journal of Servces Marketng, 3, De Battst F., Ncoln, G. and Saln, S. (2010). The rasch model n customer satsfacton survey data. Qualty Technology and Quanttatve Management, 7, Fay, R. E. and Herrot, R. A. (1979). Estmates of ncome for small places: an applcaton of James-Sten procedures to census data. Journal of the Amercan Statstcal Assocaton, 74, Gamerman, D. and Lopes, H. F. (2006). Markov Chan Monte Carlo: Stochastc Smulaton for Bayesan Inference. Chapman & Hall. 8. Gf, A. (1990). Nonlnear Multvarate Analyss. John Wley and Sons, New York. 9. Groves, R. M. (1989). Survey Errors and Survey Costs. John Wley and Sons, New York. 10. Heckman, J. J. (1979). Sample selecton bas as a specfcaton error. Econometrca, 47, Hothum, C. and Spntg, S. (1998). Customer Satsfacton Research, n the Hesomar th Handbook of Market and Opnon Research, 4 edton. (Edted by McDonald and Vangelder), Esomar, Johnson, N. and Kotz, S. (1972). Dstrbuton n Statstcs: Contnuous Multvarate Dstrbutons. John Wley and Sons, New York. 13. Lee, S. (2006). Propensty score adjustment as a weghtng scheme for volunteer panel web surveys. Journal of Offcal Statstcs, 22, Lessler, J. T. and Kalsbeek, W. D. (1992). Nonsamplng Errors n Surveys. John Wley and Sons, New York. 15. Mchalds, G. and De Leeuw, J. (1998). The gf system of descrptve multvarate analyss. Statstcal Scence, 13, Ncoln, G. and De Battst, F. (2008). Methods for Summarzng the Rasch Model Coeffcents, n Metod, Modell e Tecnologe dell'informazone a Supporto delle Decson. (Edted by D'Ambra, Rostrolla and Squllante),

14 180 Ncoln and Valle 17. Rao, J. N. K. (2003). Small Area Estmaton. John Wley and Sons, New York. 18. Rosenbaum, P. and Rubn, D. (1983). The central role of the propensty score n observatonal studes for causal effects. Bometrka, 70, Ross, P. E., Allenby, G. M. and McCulloch, R. (2005). Bayesan Statstcs and Marketng. John Wley and Sons, New York. 20. Rubn, D. B. (1974). Estmatng causal effects of treatments n randomzed and nonrandomzed studes. Journal of Educatonal Psychology, 66, Smts, J. (2003). Estmatng the Heckman two-step procedure to control for selecton bas wth SPSS. Avalable at Trevsan, M. and Torell, N. (2006). Comparng Herarchcal Bayesan Models for Small Area Estmaton. In Metod statstc per l'ntegrazone d bas d dat da font dverse, (Edted by Franco Angel), Appendx The defntons of personal nformaton varables are the followng: FREQCLASS In whch cabn do you most often travel when flyng wth Best Flght? (Concorde, Frst, Club World, World Traveller Plus, World Traveller (economy), Club Europe, Euro Traveller (economy), Domestc); CLUB If you are a member of the Best Flght Executve Club or any other frequent flyer scheme, please ndcate whch card you hold. (Executve Club Gold, Executve Club Slver, Executve Club Blue); NATION What s your natonalty? (Amercan, Brtsh, French, German, Japanese, Other); AGE What s your age? (12-15, 16-21, 22-25, 26-34, 35-44, 45-54, 55-64, 65+); GENDER Are you female or male? (male, female). The 10 varables of the servce dmensons are the followng: BOOKING Overall, how satsfed were you wth the tcket bookng process? CHECKIN Overall, how satsfed were you wth the check-n process? TEANSFER Overall, how satsfed were you wth the ease of transferrng flghts? LOUNGE Overall, how satsfed were you wth the Best Flght lounge? DEPARTUER Overall, how satsfed were you wth the departure process? CABIN Overall, how satsfed were you wth the cabn envronment? MEAL Overall, how satsfed were you wth the meal/refreshments servce? ENTERTAINMENT Overall, how satsfed were you wth the n-flght entertanment? DUTYFREE Overall, how satsfed were you wth the choce of goods for sale? CREW Overall, how satsfed were you wth the cabn crew? The 4 varables of the overall customer satsfacton are the followng: EXPERIENCE Overall, how satsfed were you wth your experence of Best Flght today, VALUE How satsfed are you wth the value for money of ths Best Flght flght? RECOMMENDATION On the bass of your experence wth Best Flght today, how lkely are you to recommend Best Flght to a frend or colleague?

15 Errors n Customer Satsfacton Surveys and Methods 181 REPEAT On the bass of your experence wth Best Flght today, how lkely wll you be to travel wth Best Flght agan? Authors' Bographes: Govanna Ncoln has been Assocate Professor of Sample Theory at the Unversty of Mlano Bcocca, now she s full Professor of Statstcs at Unversty of Mlan. Her man research areas are the followng: probablstc models, robustness on power statstcal tests, tests wth balanced errors, permutaton tests, samplng technques, customer satsfacton measures, assessment of the qualty of publc servces. Lucana Dalla Valle earned a Ph.D. n Statstcs from the Unversty of Mlano-Bcocca n Currently, she holds a Post-Doctoral poston at the Department of Economcs, Busness and Statstcs of the Unversty of Mlan. She s nterested n Bayesan statstcs, Markov Chan Monte Carlo methods, statstcal models for fnancal rsks, copula modelng for fnancal applcatons, statstcal models for nternatonalsaton and samplng technques.