MEASURING USER S PERCEPTION AND OPINION OF SOFTWARE QUALITY

Transcription

1 MEASURING USER S PERCEPTION AND OPINION OF SOFTWARE QUALITY Dmtrs Stavrnouds, Computer Technology Insttute, Greece Mchals Xenos, Patras Unversty, Greece Pavlos Peppas, Macquare Unversty, Australa Dmtrs Chrstodoulaks, Patras Unversty, Greece Summary: Ths paper presents a method for modelng users percepton of software qualty. The method ams to mprove the qualty of data derved from user opnon surveys and facltate the analyss of such data. Addtonally, usng aspects of Belef Revson theory, the proposed model offers a way to measure users opnon n early stages of product release and a way of predctng the opnon subsequently formed after ther opnon revsons usng the ntal measurements. The paper presents graphcal examples from modelng users percepton of software qualty and of handlng the users belef revson. Fnally, conclusons from our case studes and data analyss are presented. Dmtrs Stavrnouds, Computer Technology Insttute, Research Unt II, 3 Kolokotron street, Patras 26221, Greece, Phone: , Fax: , e-mal: stavrno@ct.gr 1. Introducton The value of surveys s well recognzed n measurng product qualty characterstcs, as the customers perceve these characterstcs. Moreover, nternatonal standards such as ISO9001 (ISO9001, 1991), IEEE (IEEE, 1989), Baldrge (Brown, 1991) and CMM (Bate, 1995), (Curts 1995) encourage software producton companes to measure users perceved qualty of ther products. Not only are surveys ndcators n themselves, but they also allow more sophstcated analyss technques whch are requred of organzatons wth hgher levels of qualty maturty. Furthermore, surveys commonly allow one to focus on just the ssues of nterest; and, as a result, are quantfable and thus provde a convenent way to brng the power of statstcs to bear. However, n surveys some dffcultes arse related to the qualty of data, the hgh cost of conductng the survey and the usual revsons of the customers opnon of the product untl these defne themselves suffcently. In addton, the procedure of measurng users percepton of qualty s tme consumng, especally when conductng the survey when the results need to be re affrmed by subsequent surveys. As a result, the followng goals must be acheved: a) mprove the accuracy of the nput from the surveys, b) mprove the analyss and the nterpretaton of the derved results from the surveys and c) fnd a way of modelng the belef revson of the customers so as, not only to extrapolate ther opnon n the early stages of product release, but also ther fnal opnon. The am of ths paper s to offer a model amed at meetng the aforementoned goals, provde the gudelnes for ensurng and mprovng the Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

2 qualty of the surveys data and, fnally, analyze the data, thus allowng the predcton of users opnon revsons. In order to acheve the frst goal, the opnon of users s evaluated n relaton to ther overall computer usng ablty and ther ablty to use a partcular product. As regards the second and the thrd goal, rules from Belef Revson theory and Grove s Systems of Spheres (Grove, 1988) have been adapted wthn the proposed method. Our goal s to represent, n a comprehensve manner, the way that the opnon of users may be revsed. Moreover, varous cases of users and how to apply the Grove s Systems of Spheres rules n each of them are presented and analysed. All the software qualty factors wth whch the end user s concerned are dealt wth n our model to present the way a revson of the opnon of users n one factor may also change ther opnon n others. In the next chapter the formulas used for measurng users opnon, the analyss and the fndngs of these measurements are presented. In chapter 3 rules of Grove s System of Spheres and the proposed model are presented. 2. Modelng users percepton measurements The concluson reached n our prevous research (Xenos, 1996) s that, although developer orented and user orented software qualty measurements are hghly correlated, satsfacton of nternal qualty standards does not guarantee a pror success n fulfllng the customers demand for qualty. Consequently, the measurement and the evaluaton of the opnon of users and percepton of a software product are essental. What must also be taken under consderaton s the dfferentaton of the users opnon of qualty over tme. The conclusons reached from the juxtaposton of ths dfferentaton between the experenced and the nexperenced users are to be analyzed n the followng sectons. 2.1 Measurng users opnon In order to measure users opnon of software qualty we focused on the user orented qualty characterstcs derved from the Factor Crtera Metrcs model (McCall, 1977). In order to collect the measurements of the users opnon, a multple choce format was used n the questonnares n order to gude the user to select predefned responses that were ordered n nterval scales (wth choce bars, percentage estmatons, etc.). Ths method can also be appled by focusng on the user-orented qualty characterstcs derved from the ISO 9126 (ISO9126, 1991) standard (functonalty, relablty, effcency, usablty). Examples of the questons used n these questonnares are the followng: What s your opnon of the product s accuracy and consstency?, Is nvald data entry properly recognzed?, Are all functons that relate to the wndow avalable when needed?, Is help avalable for each tem and s t context senstve?. Formulas (1) and (2) were used n the surveys conducted. These formulas wegh users opnons accordng to ther qualfcatons. In QWCO (Qualfcatons Weghed Customer Opnon) method (Xenos, 1995), whch uses formula (1), O, measures the normalsed measured results of user s opnon and E measures the qualfcatons of user. Fnally, n s Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

3 the number of users who partcpated n the survey. Therefore, each user contrbutes to the average accordng to hs/her qualfcatons. QWCO n = = n ( O E ) = 1 E 1 (1) In QWCO DS (Qualfcatons Weghed Customer Opnon wth Double Safeguards) (Xenos, 1997), a number of safeguards were embedded nto the questonnares. Safeguards are questons placed nsde the questonnare so as to measure the correctness of responses and not amed at measurng user perceved qualty. They are control questons amng at detectng errors. In equaton (2), S s the number of safeguards that the user has repled to correctly and S T s the total number of safeguards. Snce the use of the QWCU DS technque mples the use at least of one safeguard n the questonnare, dvson by S T s always vald. In ths method, safeguards were used not only to detect errors when measurng customer s opnon, but also to detect errors when measurng customers qualfcatons. In equaton (2), P value can be 0 or 1. The value of P s zero when at least one error has been detected when measurng the qualfcatons of customer. P value s set to 1 only f no error has been detected. Ths method results n the rejecton of a customer s responses f errors were detected whle measurng hs/her qualfcatons. QWCO DS = n = 1 S O E P ST n S E P ST = 1 (2) These methods were used n varous software products measurng the opnon of users, whose levels of experence dffered. The results of the measurements and the analyss method are presented n the followng secton. 2.2 Analysng measurements over tme In order to measure users opnon of a software product effcently, surveys n fxed tme ntervals must be conducted. Despte the fact that such a practce cannot be appled n a professonal settng due to the hgh cost, monthly surveys were conducted for the requrements of ths research usng the same sample of users for the same software products. For the analyss of the measurements, the users were dvded nto two man categores, the experenced and the nexperenced users. Fgure 1 shows the dervatons of these surveys. In ths fgure the lmts of the dfferentaton of the user s opnon over tme are llustrated. The horzontal bar represents the tme n monthly ntervals and the vertcal bar represents the user s opnon, whch was measured usng the formulas mentoned above. The user s opnon n each survey takes values from 0 to 1. The lne AvOp represents the average users opnon of the qualty of the software product, formed after the fnal opnon of users has been Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

4 measured. The opnon of experenced users over tme vares between the curves e1 and e2, whereas the opnon of nexperenced users over tme vares between the curves ne1 and ne2. Fgure 1: Boundares of users opnon The experenced users, n contrast to the nexperenced, form an opnon for the qualty of the product from the early stages of ts release, whch s very close to ther fnal opnon. On the contrary, the nexperenced users wll form an opnon close to ther fnal opnon after usng the software product for a long perod of tme. The length of ths perod depends on the complexty of the product, the number and the varety of the functons t supports, the amount of usage and the condtons under whch usage occurs, as well as usage of smlar software products. Ths perod of tme usually vares from sx to twelve months, when the user s experenced n the use of ths specfc product. After a perod of tme, the lne AvOp usually starts to declne as the user requrements usually ncrease over tme. Ths phenomenon s dependent on factors, such as the smlar software products that may be released and the advances n hardware. It was also observed that when an experenced user gves the software a hgher score than hs fnal score or vce versa, ths does not dsplay fluctuatons but s seemed to slowly close the gap between the hgh or low score and the fnal score. Amongst nexperenced users, however, such predctable varablty was not observed; opnon fluctuated between wdely ne1 και ne2. Over tme, the degree of fluctuaton receded to the users fnal opnon of the product qualty. For example, the dfferentaton of nexperenced users opnon over tme can be ntmated by the dagram of fgure 2, where UO represents an example of the changes n a user s opnon. Ths fluctuaton results from the nexperenced user ether fndng a new feature of the product, whch has remaned undscovered or has uncovered some aspect of the product, whch the user has sought and has not found up tll now and, as a result, rates the product hghly. Smlarly, f the user uncovers a flaw n the product (whether real or perceved), the user wll rate t lowly regardless of whether the aforementoned flaw could not have been avoded at the producton stage. Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

5 Fgure 2: Fluctuaton of nexperenced users opnon Software qualty factors are not clearly perceved by nexperenced users. If they dscover a characterstc ndcatng that the product fals n one partcular factor, then they consder that the product fals n all the other areas as well. On the contrary, experenced users do clearly perceve the ndependent nature of these factors. After a justfable tme perod, nexperenced users become accustomed to the new features or flaws they dscover n the product and, as a result, ther opnon begn to lean towards the fnal opnon as s the case wth experenced users. 2.3 Usng fndngs to mprove how surveys are conducted From the measurements of the surveys, t s obvous that over tme: a) the experenced users opnon of the qualty of the software product approaches ther fnal opnon and b) the devaton of the nexperenced users opnon from ther fnal opnon declnes contnuously. Thus, the more a customer uses a product, the more weght must be gven to hs opnon. In other words, the tme factor must also be taken nto account for effectve measurements of software qualty. The analyss of secton 2.2 revealed that, after a long perod of tme, nexperenced users wll form an opnon that wll be close enough to ther fnal opnon of the qualty of the product. The length of ths perod may surpass sx months. As a result, n order to defne the qualty of a software product suffcently n the early stages of ts release, the sample of the users beng asked must be restrcted to experenced users. Addtonally, n the early stages, the opnon of nexperenced users fluctuates greatly. Ther opnon can be consdered only f the sample of users s large enough to be consdered representatve, thus ensurng sound results. Moreover, the opnon of experenced users should be gven greater weght than that of the nexperenced users, regardless of ther beng fewer of the former. Furthermore, from the fndngs for ndvdual user groups partcpatng n the surveys, t was also observed that the larger the degree of fluctuaton n ther opnon, the more dffcult t was for them to learn the features whch are more relevant to ther specfc context. Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

6 3. Early predctons of users opnon From the fndngs t s clear that the weght of the customer s opnon of the qualty of a software product ncreases over tme. As a result, every new characterstc of the product detected by the customer, whch wll undoubtedly lead to the revson of hs belef for the product, must be consdered. Moreover, wth each newly dscovered feature, users become more assured of the valdty of ther opnon. However, because of the hgh cost of surveys, a smpler, faster and more automated way must be found, n order to estmate the revson of the opnon of user groups wthout needng to conduct a new survey every whle. 3.1 Usng Systems of Spheres Rules from Belef Revson theory, Grove s Systems of Spheres, have been adapted to meet the needs of ths approach (Gardenfors, 1988), (Peppas, 1996). In ths system, any belef set K can be represented by the subset [K] of M that conssts of all maxmal sets where all the sentences n K are ncluded, where M s the set of all possble worlds that can be descrbed n a propostonal language L. Thus, a system of spheres centered on [K] s a collecton S of subsets of M that can be represented fguratvely n fgure 3. In ths system the more nnermost the sphere s, the more possble the world centered on [K]. When a new sentence A appears to be true, wth A [K], and A s always accepted to be relable, the possble world must be revsed n order to encompass A. So the closest sphere (S A ) must be taken, where S A [A], n order to have mnmal changes to our frst belef state. Our new world s now C(A)=[K*A]= S A [A]. Fgure 3: Grove s System of Spheres 3.2 Desgnng the model Adaptng Grove s System of Spheres rules, an alternatve model for representng user s opnon of software qualty s presented. The ponts (e.g. the possble worlds) n ths model represent user perceptons of the software product accordng to the software qualty factors. For every belef set K the user has for the qualty of the product, there s a system of spheres S Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

7 n M centered on [K] that shows where hs belef set s presented n M. Furthermore, ths belef set K must contan the user s belefs for the product n all the software qualty factors. The frst step for desgnng ths model s to fnd varous characterstcs and events for each software qualty factor, whch ndcate the users opnon for ths factor. These characterstcs events are the sentences that determne the possble worlds of the model. The second step s to represent all these possble worlds to the model. The model must be separated wth a lne or curve nto two parts for every sngle event E. One where the event occurs (the E part) and one where does not (the not E part). If n a possble world S x the user s determned that the event E s true, then the S x must be located nto the E part and vce versa. But f the users are not determned for the event E (e.g. they don t know whether t happens or not) then the S x must be located n both. After separatng the model wth lnes for every sngle event, knowng users opnon of these events, ther possble world can be represented to the model. The thrd step s to determne how a possble world of a user group wll change n the model, when users revse ther opnon of an event. The new possble worlds must be delmted by the parts of the model n whch the event does or does not occur. Moreover, t must be determned n what way the users belef n the other events may change or not. If t does change, ths has drect and occasonally radcal ramfcatons on ther new possble world. These revsons must have mnmal changes and after fnshng wth all of them, the new possble world wll represent the new belef set of the user group about the qualty of the software product. 3.3 Modelng users categores The analyss of secton 2.2 revealed that users must be separated nto categores accordng to ther experence, snce ther opnon of a software product alters n a dfferent way. As a result, the proposed model dffers for each user category, because the revson of the opnon n one qualty crteron wll produce dfferent results n the belef set of each user category. In other words, when nexperenced users dscover that an event occurs or not, then ther opnon for the qualty of the product wll change n a hgher degree than the opnon of experenced users wll. Software qualty factors are not clearly perceved by nexperenced users and when they are ether satsfed or dssatsfed wth one, the other factors follow sut. As a result, a revson n ther opnon of one qualty factor leads to the revson n ther opnon of others. On the contrary, accordng to the experenced users percepton of qualty, the dfferent factors of qualty are not seen as beng nterdependent. In the experenced users case, a revson n the opnon of one qualty factor wll affect the opnon of the other factors only f ths revson s of a radcal nature. The dfferent level of nterdependence among software qualty factors, accordng to users opnon, leads to a model dfferently desgned for each category of users. In the case of experenced users, the boundares that declare whether an event occurs or not are presented n such a way, that a revson n an opnon of one event wll result n mnmal changes n ther belef set. In other words, these boundares are ndependent. Therefore, no areas that are dense n event boundares are observed. Otherwse, f the belef set of a user was represented by a Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

8 sphere desgned nto ths area, a revson n the opnon of one event would lead to a radcal revson, whch s not observed n the case of experenced users. The model n ths case can be ntmated by fgure 4. Fgure 4: Model of experenced users category On the contrary, n the case of nexperenced users, every new characterstc of the software product that has been detected dfferentates ther opnon of all the software qualty factors. As a result, the model n ths case s desgned n such a way, that the revson n users opnon of one factor leads to a revson of a radcal nature. The possble world of nexperenced users s not as stable as n the experenced users case. Snce nexperenced users have usually the opnon that the events of the model are nterrelated, the boundares of these events must be n close proxmty. The model has areas that are dense n event boundares and t s llustrated n fgure 5. Fgure 5: Model of nexperenced users category Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

9 Fgures 4 and 5 also llustrate the dfferentaton between the experenced users and the nexperenced users, after a belef revson. For example, n the new world C(A), derved from the revson n the event A, nexperenced users form an opnon of all the events completely dfferent from ther ntal one (opnon n world [K]). On the contrary, experenced users could revse ther opnon only n one addtonal event. In concluson, software producton companes usng ths method must adapt the proposed model to ther partcular needs. The boundares of the events must be desgned accordng to the weght gven to each software qualty factor. Further research s currently planned n order to refne ths model for each factor of the Factor Crtera Metrcs model (or for each qualty characterstc of the ISO-9126 standard). In that case, the events that determne the possble worlds of the model wll be related to the specfc crtera of one sngle factor. Furthermore, software producton companes wll be able to predct users opnon separately for each software qualty characterstc. 4. References Bate Roger, et al: A Systems Engneerng Capablty Maturty Model, Verson 1.1. Software Engneerng Insttute, CMU/SEI-95-MM-003, November Brown M. G.: Baldrge Award Wnnng Qualty: How to Interpret the Malcom Baldrge Award Crtera. Mlwaukee, WI: ASQC Qualty Press, Curts Bll et al: People Capablty Maturty Model. Software Engneerng Insttute, CMU/SEI-95-MM-02, September Gardenfors Peter: Knowledge n Flux Modelng the Dynamcs of Epstemc States. MIT Press, Cambrdge, England, ISBN , Grove Adam: Two modelngs for theory change. Journal of Phlosophcal Logc, 17, , IEEE: Standard for a Software Qualty Metrcs Methodology. P-1061/D20, IEEE Press, New York, ISO9001: Qualty Management and Qualty Assurance Standards. Internatonal Standard, ISO/IEC 9001: ISO9126: Software Product Evaluaton - Qualty Characterstcs and Gudelnes for ther Use, ISO/IEC Standard ISO-9126, 1991 McCall J. A., Rchards P. K. and Walters G. F.: Factors n Software Qualty, Vols I, II, III. US Rome Ar Development Center Reports NTIS AD/A , 015, 055, Peppas Pavlos: Well Behaved and Multple Belef Revson. European Conference on Artfcal Intellgence, Xenos M. and Chrstodoulaks D.: Software Qualty: The User s Pont of Vew. pp of Software Qualty and Productvty, Chapman & Hall, ISBN: , Organzaton for Qualty Software Commttee (EOQ-SC), Venna,

10 Xenos M., Stavrnouds D. and Chrstodoulaks D.: The Correlaton Between Developerorented and User-orented Software Qualty Measurements (A Case Study). 5th European Conference on Software Qualty, EOQ-SC, Dubln, pp , Xenos M. and Chrstodoulaks D.: Measurng Perceved Software Qualty. Informaton and Software Technology Journal, Butterworth Publcatons UK, Vol. 39, pp , June Organzaton for Qualty Software Commttee (EOQ-SC), Venna,