Extended Web Assessment Method (EWAM) - Evaluaton Of E-Commerce Applcatons From The Customer's Vewpont (Copyrght 2002 IEEE. Publshed n the Proceedngs of the Hawa Internatonal Conference on System Scences, January 7-10, 2002, Bg Island, Hawa.) Petra Schubert IAB Insttute for Busness Economcs FHBB Unversty of Appled Scences, Basel Peter Meran-Str. 86, 4002 Basel, Swtzerland tel +41 61 279 1774 p.schubert@fhbb.ch Walter Dettlng IAB Insttute for Busness Economcs FHBB Unversty of Appled Scences, Basel Peter Meran-Str. 86, 4002 Basel, Swtzerland tel +41 61 279 1790 w.dettlng@fhbb.ch Abstract The paper presents and apples the Extended Web Assessment Method (EWAM) an evaluaton tool whch has been specfcally created for the assessment of e-commerce applcatons. EWAM bulds on the Web Assessment Method developed at the Unversty of St. Gallen, Swtzerland [12] and ntegrates fndngs from the Technology Acceptance Model [1] and several alternatve approaches. It defnes an evaluaton grd ncludng a set of crtera to apprase the qualty and success of exstng e- commerce applcatons. The focus les on consumer perspectves and the specfc features of the Internet as medum. Usng the EWAM tool, an analyss of eght commercal Web stes n two dfferent busness sectors consumer goods and e-bankng was performed. The fndngs show that accordng to the assessors most of the Web stes assessed do stll not fully meet user expectatons. Consumer goods are a local busness. Despte of ths possble dsadvantage they showed a good performance n satsfyng ther customers. E-bankng applcatons do not present dfferentated features, one can seemngly be replaced by the other wth no problem. 1 Introducton Dgtal marketng calls for new marketng paradgms and places some of the exstng rules n a new lght. It s of vtal necessty that Internet companes should also take these paradgms nto consderaton for ther Internet busness. In the onlne medum, where customers are only a mouseclck away from comparable offers, dstngushng characterstcs are the only means to bnd customers long term to one's own offer. The most mportant thng here s to concentrate above all on features whch are only possble onlne. One possblty, for example, s the settng up of customer communtes [11], whch pool ther knowledge and ther profles for communal use and render economes of scale effectve n a new way. The paper presents the Extended Web Assessment Method (EWAM), an evaluaton tool whch has been specfcally created for the assessment of e-commerce applcatons. A major problem posed when searchng for possble mprovements of e-commerce applcatons s the queston of the crtera used to determne the success of such a system. Ths s where EWAM comes n, by offerng a set of relevant crtera whch provde a bass for measurng the success of Web stes of ndvdual Internet companes. The prescrbed crtera enable exstng Web stes to be evaluated from the customer's pont of vew. To do ths assessors have an onlne tool at ther dsposal wth whose help data can be collected and evaluated. By defnng sector profles, comparsons of a Web ste wth the sector average or wth the Best Practce Profle can be made. Ths benchmarkng can ultmately be used to suggest steps for mprovement. The paper presents the fndngs of an emprcal study carred out n Aprl 2001. Twenty students from an execu-
tve tranng class n E-Busness completed the web assessment forms, evaluatng web stes of two dfferent busness sectors: consumer goods and e-bankng. Before startng ths project they were thoroughly nstructed n the use of the tool. Addtonally, they had to provde a qualtatve analyss of the eght web stes n the form of a wrtten report. The fulfllment of ths task was needed for a successful attendance of the class. Snce the study was conducted n Swtzerland, the results represent a Swss perspectve. E-busness s stll a very regonal affar, makng t hard for consumers to assess Web stes n countres other than ther home country. Nevertheless, we ncluded a German and a US company n our sample to make the study more nterestng and to add an nternatonal perspectve n a study about the nherently global Internet commerce envronment. Only Web stes whch were ntensvely used or tested by the assessors were nvestgated. The assessors beng well-nstructed Web experts, the sample was ntended to represent a small quantty of hghly-qualfed user opnons. The evaluatons resulted nto Web assessment reports whch compare the ndvdual performance profle of one company wth the best of ts class (best practce profle) and the general sector profle (aggregaton of all comp anes n the same sector). The paper starts wth a descrpton of the methodology of the Web Assessment explanng ts orgns and the latest adjustment to advances n technology and Web ste desgn. We then descrbe our emprcal study and the way n whch we prepared the results. Four companes from both sectors (consumer goods and e-bankng) are brefly ntroduced and the fndngs for the sector are dscussed n detal. The paper concludes wth some remarks about the current state-of-the-art n Web stes desgn and functonalty and ther sutablty for e-commerce transactons. 2 Theoretcal background 2.1 Orgnal Web Assessment Method The Web Assessment Method was developed n 1997 at the Competence Center for Electronc Markets (CCEM) at the Unversty of St. Gallen n conjuncton wth the company partners [12]. The orgnal mpulse arose from the dscontent felt by ndustral partners of the CCEM about unsatsfactory results of e-commerce applcatons already carred out. The method defnes an evaluaton grd wth a set of crtera to apprase the qualty and success of exstng e- commerce applcatons. Besdes the rgorous focusng on consumer perspectves, the success n mplementng the offer of products and servces s consdered wth reference to the specfc features of the electronc medum. Successful Internet busness actvty requres the followng new paradgms to be taken nto account: Electronc markets and transacton phases: The Web Assessment Model examnes the three classc transacton phases of electronc markets (nformaton, agreement and settlement). A fourth element, the communty component, s ntegrated as a lnk between the actual purchase transacton and the necessary trust relatonshp n the vrtual realm. Informaton technology / Meda-nherent Characterstcs: Where marketng aspects are concerned, the Web Assessment model focuses on the specal features that are nherent n the Internet. The assessment crtera are, besdes the transacton phases and communty component, derved from the characterstcs of the electronc medum: hypermeda presentaton, database nterface (expert system), 24-hour access, anonymty, ubquty, confguraton possblty of the user surface, ntegraton wth the customer and asynchronous communcaton. Performance marketng: The underlyng dea of performance marketng s that the clent should not only be sold the core product, but should addtonally be offered a range of complementary products to maxmze customer beneft. These addtonal servces customze the product and make t attractve for the clent. In the dffcult arena of nternatonal competton product dfferentaton s thus made possble. In an emprcal study carred out n 1997/1998 more than 70 questonnares were collected and evaluated from over 55 dfferent partcpants (researchers and practtoners) [12]. The result of the study was a comparson of the ndvdual performance profle of two companes (SwssAr and Amazon.com) wth comparable offers from compettors n the same sector. At the same tme n ths ntal applcaton of the Web Assessment Tool the specal strengths and weaknesses of the companes could be dentfed. As a result of technologcal progress and the accomp a- nyng value change among users snce the development of ths method, t was thought necessary to adapt the questonnare for the purpose of data collecton and dentfcaton of success crtera. In summer 2000 the method was fundamentally revsed; besdes takng account of new research fndngs especally n the Internet marketng feld t also ncorporated the Technology Acceptance Model (TAM) [1], establshed for the acceptance of nformaton systems, whch wll be brefly descrbed n the followng paragraph.
2.2 Technology Acceptance Model (TAM) Wth the Technology Acceptance Model Davs descrbes the effect of system features on the acceptance of users wth regard to new computer-based nformaton systems. Davs chose the Fshben Model, a psychologcal behavor model, as the bass for the development of the TAM. Ths model, specfed by Fshben n 1967, was extensvely analysed by Fshben & Ajzen [2] and further elaborated nto the Theory of Reasoned Acton. The TAM follows the causal chan of the TRA: Atttude Intenton Behavor. The hypothetcal relatonshps defned n the basc model were tested emprcally usng a questonnare (wth questons on e-mal and XEDIT) followed by regresson analyss. Out of 120 questonnares dstrbuted to employees of IBM Canada's Toronto Development Laboratory, 112 were returned completed (n = 185, of whch 109 regardng e-mal and 76 about XEDIT). The resultng evaluaton confrms several hypotheses. (H1) Atttude has drect nfluence on Actual System Use. (H2) Usefulness has sgnfcant effect on Atttude toward Usng. (H3) + (H4) Ease of Use has an effect on Atttude and Usefulness.... and provdes further fndngs whch were not expected n ths form: (H5) System has a sgnfcant effect on Ease of Use but not on Usefulness. (H6) Usefulness has a drect effect on Actual System Use. (H7) System has a drect effect on Atttude toward Usng. In the orgnal form of the TAM the Subjectve Norm of the Fshben Model s not ntegrated, as Davs dd not consder t to be relevant. Malhotra/Galletta [9] conclude n Extendng the Technology Acceptance Model to Account for Socal Influence, Theoretcal Bases and Emprcal Valdaton that Davs had underestmated the mportance of the Subjectve Norm n the TAM. Ther emprcally tested study ponts to the fact that for the user socal nfluences also play an mportant role n the acceptance and use of new nformaton technologes. The TAM s an mportant contrbuton to understandng the use, behavor and acceptance of new nformaton systems by the user. Thus t also appears sutable for further applcaton as a theoretcal base n the Extended Web Assessment Method. The motvatonal varables Atttude toward usng, Perceved Usefulness and Perceved Ease of Use serve as a lnk between system features and the behavor of an ndvdual n relaton to the use of new nformaton systems (Actual System Use). From ths pont of vew the model s successful [1]. The fact that socal components (Subjectve Norm) were not taken nto account n TAM was crtczed n ongong research [9], whch underlnes the mportance of these components. Hence the socal nfluences were also ncorporated nto EWAM n the form of crtera regardng Trust. 2.3 Alternatve Approaches The followng alternatve approaches to Web evaluaton lsted below were also taken nto account durng the revson of the Web Assessment Method: Desgn Qualty of Web stes for Electronc Commerce: Fortune 1000 Webmaster's Evaluatons [8] Perfekte Webseten we seht de Realtät aus? [7] GomezPro.com [4] JursNET [6] 3 Extended Web Assessment Methode (EWAM) 3.1 Descrpton of the method The Extended Web Assessment Method (EWAM) bulds on the orgnal Web Assessment Method and ntegrates fndngs from the Technology Acceptance Model and several alternatve approaches [3]. It defnes an evaluaton grd ncludng a set of crtera to apprase the qualty and success of exstng e-commerce applcatons, whch wll be presented below. The focus les on consumer perspectves and the specfc features of the Internet as medum. A successful e-commerce applcaton must meet the needs of the user accordng to Perceved Usefulness (Crtera USEF1-USEF15) and Ease of Use (Crtera EOU1- EOU8). Under the headword Trust (Crtera TRUST1 - TRUST2) questons about the Subjectve Norm are addtonally taken nto account. A success or qualty feature must be assgned to one of these categores. An excerpt from the lsts s depcted n fgure 1. When evaluatng an e-commerce applcaton accordng to the Extended Web Assessment Method a Web ste s frst of all allocated to a sector. Ths serves later durng evaluaton to dentfy the reference sector for benchmarkng. The success and qualty crtera are formulated n general terms and are vald n every sector, but are dfferentated by ther mportance ratng. In order to take due ac-
count of the dfferences n the ndvdual sectors, crtera are weghted correspondng to dfferent sector profles and ther relevance n the sector. Hence beng up to date wth nformaton s of greater mportance for the suppler of fnancal nformaton (e.g. Stock Brokerage, real-tme share prces) than for a suppler of consumer goods. Wth the dstrbuton of dgtal goods (e.g. software) the choce of generc servces (EOU5), for example, tracng and trackng of a parcel, s of lesser mportance than wth the delvery of a book. In order to be able to undertake specfc and hghqualty analyses, t s essental to record the level of mportance per crteron and per sector exactly. The mportance per crteron s recorded on a scale from unmportant (-2) through less mportant (-1), mportant (+1) to very mportant (+2). Trust s the sne qua non of e-commerce; wthout trust no busness s done. The creaton of a trustworthy envronment on the Internet s a major challenge for the success of e-commerce. Accordng to the Theory of Reasoned Acton [2] Subjectve Norm refers to how an ndvdual s nfluenced by a person partcularly close to them, and by ths person's opnon of a certan way of behavng. 3.2 Summary of the Methodology An EWAM crteron s frst of all assgned to a crtera category ( Ease of Use, Usefulness or Trust ) and wthn these three categores allotted to one of the four transacton phases of electronc markets (the nformaton, agreement, settlement and after-sales phase), to the communty component or to the category crtera whch concern all phases. Compared wth the orgnal Web Assessment Method, the EWAM was extended to nclude the after-sales phase and a category group crtera whch concern all phases. 3.3 Data collecton: the EWAM tool Data s collected over the Internet wth an onlne questonnare ( EWAM tool ) n whch the usual structure of the orgnal tool [13] was almost retaned. The ndvdual crtera were assgned to a transacton phase or a communty component, and two new sectons were added ( Aftersales phase and crtera whch concern all phases ). When the assessor starts the evaluaton wth the EWAM tool, as a frst step he must record the URL of the Web ste under examnaton and assgn t to a sector. The scale of the possble choces s so arranged that the assessor must decde on a postve or negatve statement wth each value. The scale has four values (++,+,-,--). The alternatve value n. a. (not applcable) can be used f a crteron s not relevant or not avalable n a partcular context. The crtera are so formulated that a postve (negatve) evaluaton wll also lead to a postve (negatve) result. I strongly agree always scores (+2), I slghtly agree (+1), I slghtly dsagree (-1) and I strongly dsagree (-2). N. a. scores nl, whch s dsregarded wth further calculatons (e.g., of averages). 3.4 Data preparaton and analyss For the drawng up of meanngful evaluatons of any Web ste under examnaton (the Web ste compared to the sector average, to best practce profle or to one of ts compettors) three profles were defned n the EWAM. Sector Profle: the profle of the relevant sector Company Profle: the profle of the Web ste Best Practce Profle: the profle of the best-of-breed n the relevant sector The EWAM judges a Web ste purely from the customer's pont of vew. The best EWAM result does not necessarly mean that ths Web ste s the most successful n realty, snce success s nfluenced by further factors Fgure 1: EWAM tool: Lst of crteras
(e.g. e-busness relevance of the offer, proftablty, backend ntegraton, fnancng aspects, etc). A Best Practce Profle can only be establshed when (a) a suffcent number of dfferent Internet busnesses per sector have been evaluated and (b) these have been compared wth ther performance n the real world, n whch case a best practce must also be effectvely successful there. Accordngly, an adequate best practce data reference base can only ensue from the combnaton of ponts (a) and (b). 3.5 Personal Web Assessment Report One of the results of a Web ste assessment s a Personal Web Assessment Report contanng the followng analyses and graphcal representatons: a) Summary of the ndvdual crtera and results n the categores 1. Informaton Phase, 2 Agreement Phase, 3 Settlement Phase, 4 After-sales Phase, 5 Communty Components, 6 Fnal Secton and a calculaton of the total score. b) Comparson of the Web stes examned wth the sector average and wth the sector Best Practce, n the form of a quanttatve and graphcal analyss, takng no account of the mportance ratng of the crtera. c) Graphc comparson of the results of the frst sx categores (a, b) wth ther mportance ratng for company and sector profle. d) Comparson as b) above, but takng full account of the mportance ratng of crtera. We dentfed three dfferent target groups whch gan the followng beneft from the personal Web assessment report: Internet merchant: Comparson of the qualty of the customer orentaton of hs Web ste wth the sector profle or wth a drect compettor. Possble suggestons for mprovement can be drawn from the result. Potental Internet merchant: Awareness of the success crtera of commercal Web stes n the target sector. Internet buyers: Examnaton of the qualty of the customer orentaton of the assessed Web ste. 4 EWAM fndngs: Consumer Goods and E- Bankng The followng paragraphs present the fndngs of our study for two dfferent sectors: consumer goods and e- bankng. The Web stes chosen stem from Swtzerland. Germany, and the USA. Our am was to also study dfferences between the countres. 4.1 Consumer Goods Web stes assessed: http://www.webvan.com (***) http://www.le-shop.ch http://www.mgros.ch http://www.mcfood.ch The companes whch were chosen for the consumer goods sample are from two dfferent countres: Swtzerland and USA. WebVan was an ambtous Amercan onlne venture wth a vson of 26 massve automated warehouses and a natonal fleet of vans that would delver groceres and just about anythng else to everyone n Amerca [5]. Based on bg money nvestments t s not surprsng that WebVan came out as Best of Class although the company had been experencng some problems [10] and had to shut down ts operatons by md 2001. Le-Shop s a Swss ndependent onlne busness whch acts as a reseller for food and drugs. Actng solely on the Internet t s not surprsng that t was voted best n Swtzerland. Mgros s a Swss store chan sellng all knds of consumer goods wth a focus on the food sector. McFood, fnally, was a small Swss newcomer whch had never been able to setup ts logstcs and was pcked as the bad example by the course nstructor. 4.2 Interpretaton of the assessment forms In order to be able to assess the Web stes we had to ntervew the assessors about ther perceved mportance ratngs for the consumer goods sector. As mentoned before, the raw data for a data set does not allow nterpretaton of the performance of a Web ste. We dstrbuted two general Web assessment forms to a selected set of people who are famlar wth usng the Web as a source of nformaton and shoppng. These people were asked to state ther expectatons for Web stes n the two dfferent sectors. Ther forms where collected and the data was entered nto the tool as the so-called mportance ratng. The mportance ratng shows the expectaton of a user for a certan crteron. If expectatons are hgh, meanng that a certan feature s mportant for a Web ste from the user perspectve, and cannot be found durng the assessment the result wll be a bad evaluaton for the respectve crteron. For the evaluaton, the crtera are then summed up and aggregated to show only one value for a certan phase. For an explanatory presentaton of our results we pcked www.webvan.com (the best practce profle). Fgure 2 shows the aggregated values for the consumer goods sector. The column labeled mportance shows the
Results Importance Company Profle Best Practce Profle Sector Profle (Range -2/+2) (BPP) (SCP) Dfference to Phase BPP SCP 1. Informaton Phase 3. 0.71 0.67 0.00 0.30 0.67 0.38 2. Agreement Phase 0.50 0.61 0.00 0.14 0.61 0.47 3. Settlement Phase 0.56 0.45 0.00 0.41 0.45 0.05 4. After-Sales Phase 2. 0.83 0.83 0.00 0.55 0.83 0.27 5. Communty Component 0.17-0.68 0.00 0.11-0.68-0.80 6. Fnal Secton 1. 1.43 1.04 0.00 0.46 1.04 0.58 7 Overall Score n.a. 0.49 0.00 0.33 0.49 0.16 Fgure 2: Aggregated data set for the consumer goods sector, company profle: www.webvan.com Result bad neutra very good 2.00 1.50 1.00 0.50 0.00-0.50-1.00-1.50 Fg. 4: Company Profle vs. Best Practce Profle and Sector Profle wth mportance weghts Company Profle Best Practce Profle Sector Profle -2.00 1. Informaton Phase 2. Agreement Phase 3. Settlement Phase 4. After-Sales Phase 5. Communty Component 6. Fnal Secton 7 Overall Score Fgure 3: Company Profle of www.le-shop.ch Result bad neutral very good 2.00 0.00-2.00 Fg. 2: Company Profle: Results of Categores vs. ther Importance strategc overkll mmedate mprovement not necessary unmportant -2.00 neutral Improvement necessary very mportant 2.00 Importance 1. Informaton Phase 2. Agreement Phase 3. Settlement Phase 4. After-Sales Phase 5. Communy Component 6. Fnal Secton Fgure 4: Company Profle of www.mgros.ch Result bad neutral very good Fg. 3: Sector Profle: Results of Categores vs. ther Importance 2.00 strategc overkll 0.00-2.00 mmedate mprovement not necessary unmportant -2.00 neutral Importance very mportant 2.00 1. Informaton Phase 2. Agreement Phase 3. Settlement Phase 4. After-Sales Phase 5. Communy Component 6. Fnal Secton
perceved user expectatons. The possble range of values s +2 to -2. For the calculaton, the mportance values are later transferred to a scale of 0 to 1. The fnal secton s the most mportant followed by the after-sales and the nformaton phase. The fnal secton contans general crtera whch apply to all phases (avalablty, user nterface, nteractvty, trust) and seems to be very mportant for consumer goods. The communty component receved a very low mportance ratng. Ths could be due to the fact that the experts whch we polled for the mportance ratng dd not thnk about the power of communty buldng on the Internet as e.g. ponted out by Schubert [11]. On the other hand, purchasng food and drugs mght ndeed not be an actvty durng whch communty-support s apprecated although examples could be recommendatons, product ratngs, aggregaton of demand to lower prces, etc. Fgure 3 shows the company profle for www.leshop.ch, the best Swss Web ste for consumer goods (second after the Amercan company WebVan). As we can see, Le-Shop scored almost as hgh as WebVan n two of the three mportant sectons, the fnal secton and the nformaton phase. There seems to be room for mprovement n the after-sales phase and especally n the agreement phase whch receved a lower value than the sector average. Fgure 4 shows a comparson of perceved user expectatons and assessment values. It contans recommendatons for generc strateges whch should be appled dependent on the results n the varous sectons. Strategc Overkll: Entres n the upper left feld ndcate (very) good results n a (rather) unmportant category. Avalable resources are possbly not beng appled effectvely. Mantan Strategy: Entres n the upper rght feld ndcate (very) good results n (very) mportant categores. No mmedate mprovement necessary: Entres n the lower left feld ndcate (very) poor results n (rather) unmportant categores. Improvement necessary: Entres n the lower rght feld ndcate (very) poor results n (very) mportant categores. As we can see from fgure 4 www.mgros.ch scored hgher than the expectatons n the category Agreement Phase. Interestngly, ths s a category whch were deemed unmportant for the consumer goods sector. The dagonal shows the ponts where expectatons meet actual assessments. Most assessments for www.mgros.ch are grouped around the dagonal. The general advce for www.mgros.ch s to focus ts future mprovements on the settlement phase and the communty component. The aggregated results for all four companes (sector profle) show values whch are lower than the ndvdual ratngs acheved by www.mgros.ch. Ths s due to the fact that one of the companes, McFood, scored unusually low n all areas because t has not really started operatons yet (wthout tellng the customer on the Web ste). 4.3 Electronc Bankng Web stes assessed: http://www.credt-susse.ch (***) http://www.deutsche-bank-24.de/ http://www.ubs.ch http://www.bekb.ch Credt Susse and UBS are the two bggest banks n Swtzerland. The Berner Kantonalbank (www.bekb.ch) s a small Swss regonal savngs bank whch, nevertheless, offers a comprehensve e-bankng applcaton. Bank-24 s the onlne venture of Deutsche Bank, the bggest bank n Euroland. It was the frst purely Internet-based bank n Germany. Deutsche Bank has a proactve Internet strategy they were the only bank to launch a project wth Ecash whch so far has not really taken off (and probably never wll snce new and better payment systems are already on the market). The e-bankng sector resulted to be dffcult to assess wth the EWAM. Snce most of our assessors dd at most already use one of the selected e-bankng servces (who has more than one bank account?) they had to rely on the demo versons whch are offered on all four Web stes. Credt Susse was chosen to be best of ts class n ths sample. The results clearly state that the four Web applcatons are very smlar to each other n terms of functonalty, look and feel, contact possbltes, and so on. The results suggest that dfferences are almost neglgble. What catches the eye are the almost zero values for the communty category. Nether dd the assessors attrbute great mportance to communty (-1.75) nor dd they report any knd of communty-supportng components on the four Web stes. Interestngly, the after-sales phase seems to be of hgh nterest. Ths s where the nteracton between company and customer often relates back to the real world, namely to the telephone. Most after-sales servces, such as the handlng of complants, are not processed va the Internet. 5 Mathematcal Dervaton The followng paragraphs descrbe the calculatons for the comparson of any Web ste examned wth the average and wth the best practce of ths sector, ncludng the mportance ratng of crtera, for the Sector Profle. The cal-
culatons for the Company Profle and Best Practce Profle are smlar. Crtera, Importance and Evaluaton Crtera: X Importance: W (Range 2 to +2) Importance: Wg (Range 0-1) Evaluaton: r (Range 2 to +2) Weghted Value: R = Wg * r X 1 W 1 Wg 1 r R 1 1 X 2 W 2 Wg 2 r2 R 2............... X 26 W 26 Wg 26 r R 26 26 Table 2: Crtera X, Importance W & Wg, Evaluaton r, Weghted Value R The crtera (X ; = 1...26 ) are grouped n sx categores (K k, k= 1...6) accordng to the three transacton phases (nformaton, agreement, settlement), the After-Sales Phase, the Communty Component, and the Fnal Secton Categore K k Descrpton K 1 1. Informaton Phase K 2 2. Agreement Phase K 3 3. Settlement Phase K 4 4. After-Sales Phase K 5 5. Communty component K 6 6. Fnal Secton Table 3: Categores K 1 -K 6 Transformaton of the mportance of a crteron: The evaluaton of the ndvdual crtera and ther mportance uses a scale from (-2) to (+2). To avod the problem of the multplcaton of two negatve values, the mportance of each ndvdual crteron (W ) s transformed nto a range from (0) to (1) (Wg ). 1 Wg = ( W + 2) 4 Average evaluaton of each crteron 1 m r = r where: m j j = 1 m = number of assessors evaluatng a crteron X r j = ndvdual result of a crteron n range (-2/+2) Multplcaton of average result wth mportance ratng: The defntve result for every crteron s arrved at by mu l- tplyng the average result of any crteron (r ) wth ts mportance ratng (Wg ). Thus a crteron only reaches the top score when the mportance ratng (W = 1) s top. Wth lower mportance ratngs (1 = W = 0) the fnal result of a crteron decreases, but does not fall below zero. R = r * Wg Addton of the ndvdual evaluatons per category: The result for the category (K k, k=1...6 ) s arrved at by addng the R per category. The example of the Informaton Phase (K 1 ) and the Agreement Phase (K 2 ) s gven below. K 1 = 8 R = 1 K 2 = 10 R = 9 Calculaton of the percentage attanment of top score for K : a) Calculaton of the mnmum (R MIN ) and maxmum (R MAX ) evaluaton for a crteron wth gven mportance n range (0-1). R = Wg * 2-2 = R MIN = 0 MIN R = Wg * 2 0 = R MAX = 2 MAX b) Calculaton of the mnmum (K MIN ) and maxmum (K MAX ) evaluaton for a category. The example gven s for the Informaton Phase (K k, k=1 ). K = R MIN 8 8 1 K = R MIN MAX1 = 1 = 1 MAX c) Calculaton of the percentage attanment of top score ( K + K ) K% = k MAX *100% smplfed: k K + ( K * 1) MAX MIN K K% = 0.5( k + 1)*100% k K MAX Transformaton of K% k n a range (-2/+2): In further analoges the result of a category (K k ) should be compared wth ts mportance ratng. Addtonally the result K% k s transformed n a range of (-2/+2) as the defntve result of ths category (KR k ). KR k = K% ( k *4) 2 100 A value (e.g. 0.96, Informaton Phase- KR 1 ) n a range from (-2/+2) ndcates that n the sector average the Informaton Phase has been evaluated by users as relatvely good (evaluaton scale: KR k = 1.0 corresponds to good ). If the
Web ste examned shows a lower value, then t comes off comparatvely worse n the Informaton Phase. Comparson of the results of the ndvdual categores (KR) wth ther mportance ratngs: Calculaton of the average value of the mportance ratngs of ndvdual crtera (W ) for a partcular category (K k ). The Informaton Phase s gven as an example: KW k 1 8 = 8 = 1 W Overall Score: The Overall Score s the fnal result of a profle (PR). It s calculated from the sum of the sx categores (KS) n relatonshp to the theoretcal maxmum result of the respectve profle. a) Calculaton of the sums of all categores (KS) 6 where o (o= 1..3) ndexes the KSo = K k k = 1 three profles. (Sector Profle: o=1) b) Calculaton of the percentage attanment of the top score for KS Calculaton smlar to 5c), but for all categores (k = 1..6). The percentage attanment of the top score of KS% o s calculated wth the followng formula: (KSo + RS ) KS% MAXo o = *100% smplfed: RS + RS 1) MAXo MINo KS o KS% o = 0.5( + 1)*100 % RS MAXo where RS MAXo = The theoretcal top score for all crtera n profle o RS MINo =The theoretcal mnmal score for all crtera n profle o c) Transformaton of KS% o n a range (-2/+2) KS% o PR o = ( * 4) 2 100 For the sector profle ths results n a value of 1.02, whch means good on the scale (-2/+2). The relevant nterpretaton s: In ths sector Web appearances are consdered good wthn the sector average. 6 Dscusson of fndngs Web assessment s a very ambtous and labor-ntense work. The assessors have to meet certan crtera: They need to understand the crtera of the Web assessment form very well, hence they must undergo a thorough nstructon They must be experenced Web users They must take the tme to go through all four transacton phases for each Web stes assessed (ncludng delvery and payment!) The two sectors whch we put to the test where not equally well-suted to be assessed wth the tool. The Web assessment method s based on the four transacton phases of a typcal purchase. Thus, e-bankng s dffcult to test wth the exstng categores. Nevertheless, we stll thnk that t s possble to evaluate almost any knd of Web ste snce most of the crtera refer to Web ste functonalty and not purely to the process. The e-bankng sector shows homogeneous Web applcatons whch do not dfferentate themselves from the competton. Ths s not surprsng snce bankng applcatons are qute generc and nvolve lttle personal nvolvement on the part of the user. The consumer goods busness s a local busness even on the Internet. Ths s qute evdent because of the nature of the products (have to be physcally delvered, are prone to spol or break). It s even more surprsng, that the Web stes sellng consumer goods seemed to be very mature. The servces were found to be qute relable, the problem of cooperaton wth logstcs partners had been solved satsfactorly. Some complants were the lmted range of products (e.g. frozen food s not avalable) or the mssng lnk to the nventory (avalablty check). Some orders arrved wth mssng tems because they had been out of stock at the tme when the package was prepared. Generally, t seems that few Internet merchants have ever tred to assess ther Web stes from a consumer perspectve. Most are stll drven by techncal possbltes. It s notceable that there s a mssng lnk to tradtonal nformaton systems, e.g. an exstng ERP, whch could help to perform an avalablty check whch s crucal to an onlne transacton. Most systems seem to be stand-alone Web applcatons. Ths can be ether due to the consderaton of securty (a bad reason from the customer perspectve) or due to the hgh costs of an ntegraton of the dfferent systems. 7 Possble restrctons to nterpretaton The authors are aware that the emprcal study wth a sample set of 20 expert opnons can only reflect a lmted and somehow based pcture of current practce n the two sectors. The bas exsts because the students share smlar opnons of e-commerce and are a homogeneous group (because they attended the same E-Busness class, ther opnons tend to be less unversal than f they were hand pcked at random). For e-commerce as a whole, 20 people from Swtzerland are not representatve of the thousands of Web users n Swtzerland s B2C feld. Nevertheless, snce
the EWAM s a very knowledge-requrng process we cannot ask a random sample of people to do the assessments. The Web stes chosen for evaluaton were not very dverse. Only four companes per sector were chosen. In the physcal world these three sectors have hundreds of companes that offer servces on the Web, nevertheless, t took us a week to fnd four Web companes for each sector that were suted for testng. 8 Conclusons EWAM lays down a conceptual framework for the evaluaton of commercal Web stes whose basc form the Web Assessment Method has already proved tself n operaton for several years. Wth the EWAM ths work s placed n the context of current developments n e- commerce and unted wth an establshed scentfc model (the TAM). Web stes can thereby be thoroughly apprased and mplctly the degree of customer orentaton can be judged. An mportant mprovement was consderng the mportance of a crteron for a specfc busness sector whch let to a more dfferentated assessment of Web stes n dfferent ndustres. However, further research and development work s needed for the successful launch of EWAM. In a frst phase the mportant thng s to gather for all sectors a quanttatvely and qualtatvely suffcent database, whereby the mportance of each ndvdual crteron per sector must be recorded and exstng Web stes must be evaluated. Together wth a suffcent database and knowledge about commercal Web stes whch have proved successful n practce, the best practce profles for each sector must be dentfed. Wth EWAM, the focus s fxed exclusvely on the customer's perspectve n the B2C feld. Further success factors such as the ntegraton of the supply chan, the lnkage of n-house nformaton systems or the consderaton of fnancng and yeld aspects are excluded from the present verson. 9 Acknowledgements The authors are ndebted to a student class who wthn the scope of ther executve tranng on E-Busness at the Unversty of Appled Scences Basel contrbuted the data to ths research paper. 10 References [1] Davs, Fred D. Jr. (1985): A Technology Acceptance Model for Emprcally Testng New End-User Informaton Systems: Theory and Results. Doctoral Thess, Sloan School of Management, Massachusetts Insttute of Technology, 1985. [2] Fshben, M.; Ajzen, I (1975): Belef, Atttude, Intenton and Behavor: An Introducton to Theory and Research, Readng, MA: Addson-Wesley, 1975. [3] Gger, Adran (2000): Erweterte Web Assessment Methode Beurtelung von E-Commerce Applkatonen aus Kundenscht, Master Thess, Unversty of Appled Scences (FHBB), Basel. [4] Gomez Advsors, Inc. (2000): [http://www.gomezpro.com/ndex.asp], [accessed 16.08.2000]. [5] Helft, Mguel (2001): Realty Check, Asle 5, n: The Standard, [http://www.thestandard.com/artcle/0,1902,24035,00.html? nl=dnt], 30.04.2001. [Zugrff: 10.06.2001]. [6] JursNET GmbH (2000): JursNET-Gütesegel, [http://www.jursnet.ch], [accessed 10.08.2000]. [7] Kamenz, Uwe (2000): Perfekte Webseten - we seht de Realtät aus, n: Pförtsch, Waldemar A. (Ed.), Lvng Web: Erprobte Anwendungen, Strategen und zukünftge Entwcklungen m Internet, pp. 21-33, Landsberg: verlag moderne ndustre, 2000. [8] Lu, Chang; Arnett, Krk, P.; Ltecky, Chuck (2000): Desgn Qualty of Webstes for Electronc Commerce: Fortune 1000 Webmaster's Evaluatons, n: The Internatonal Journal of Electronc Commerce & Busness Meda, Vol. 10. No. 2, pp. 120-129. [9] Malhotra, Yogesh; Galletta, Denns, F. (1999): Extendng the Technology Acceptance Model to Account for Socal Influence: Theoretcal Bases and Emprcal Valdaton, n: Proceedngs of the 32nd Annual Hawa Internatonal Conference on System Scences, [http://www.computer.org/proceedngs/hcss/0001/00011/0 0011006abs.htm], [accessed 08.08.2000]. [10] Platon, Kara (2001): The Last Mle, n: Eastbay Express Onlne, [http://www.eastbayexpress.com/ssues/2001-05- 25/feature.html/page1.html], 25.05.2001. [Zugrff: 10.06.2001]. [11] Schubert, Petra (2000): Vrtuelle Transaktonsgemenschaften m Electronc Commerce Management, Marketng und Sozale Umwelt, Lohmar, Köln: Josef Eul Verlag, 2000. [12] Schubert, Petra; Selz, Doran (2000): Measurng the Effectveness of Electronc Commerce Webstes wth the Web Assessment Method, n: Hunt, Bran (Ed.), E-Commerce and V-Busness, London: Imperal College, 2001. [13] Selz, Doran; Schubert, Petra (1998): Web Assessment - A Model for the Evaluaton and Assessment of successful Electronc Commerce Applcatons, n: Proceedngs of the 31st HICSS Conference, Hawa, Vol. IV, 1998, pp. 222-231.