On Measuring the Criticality of Various Variables and Processes in Organization Information Systems: Proposed Methodological Procedure

Transcription

1 Informatca Economcă vol. 14, no. /010 5 On Measurng the Crtcalty of Varous Varables and Processes n Organzaton Informaton Systems: Proposed Methodologcal Procedure Jagdsh PATHAK 1, Abdulkadr HUSSEIN, Ram SRIRAM 3, S. Ejaz AHMED 1 Odette School of Busness, Unversty of Wndsor, Canada Math & Statstcs Dept., Unversty of Wndsor, Canada 3 Robnson School of Busness, Georga State Unversty, Atlanta, USA jagdsh@uwndsor.ca ahussen@uwndsor.ca accrss@langate.gsu.edu seahmed@uwndsor.ca Ths paper proposes methodologcal procedures to be used by the accountng, organzatonal and manageral researchers and executves to ascertan the crtcalty of the varables and the processes n the measurement of management control system. We have restrcted the valdaton of proposed methods to the extracton of crtcal success factors (CSF) n ths study. We have also provded a numercal llustraton and tested our methodologcal procedures usng a dataset of an emprcal study conducted for the purpose of ascertanng the CSFs. The proposed methods can be used by the researchers n accountng, organzatonal nformaton systems, economcs, and busness and also n other relevant dscplnes of organzatonal scences. The man contrbuton of ths paper s the extenson of Rockart s work [33] on crtcal success factors. We have extended the theory of CSF beyond the ntally suggested doman of nformaton nto management control system decson makng. The methodologcal procedures developed by us are expected to enrch the lterature of analytcal and emprcal studes n accountng and organzatonal areas where t can prove helpful n understandng the crtcalty of ndvdual varables, processes, methods or success factors. Keywords: Success Factors, Crtcalty Analyss, Perceptual Crtcalty, Crtcal Success Factors 1 On Measurng the Crtcalty of Varous Varables and Processes n Organzaton Informaton Systems: Proposed Methodologcal Procedure Informaton systems such as, Accountng Informaton Systems (AIS), collect data and mantan relevant nformaton that an organzaton can use to plan, manage, and evaluate ts performance. Over the last few decades, the AIS have transtoned from ther tradtonal accountng role nformng managers and others about transactons that took place n the past to nfluencers of the future. Ths because, wth greater nformaton technology support, AIS now able to collect, analyze, and report on such crtcal ssues as strategy formulaton, productvty, fnancal plannng, outsourcng and nsourcng. Because the nformaton collected and analyzed by an AIS are so mportant, the relablty of AIS also becomes a very mportant ssue [39]. Informaton systems, n general, are complcated systems that functon through nterconnected resources, objectves, perceptons and outcomes [19] [0]. To evaluate the relablty of such a complex system, we must dentfy the crtcal components of the system that contrbute to the system s relablty. To evaluate the effectve performance of the varous components towards fulfllng the relablty of the system, we must, n turn, dentfy the crtcal factors that affect the components [41]. In the past, most studes on nformaton system relablty approached the relablty ssue as a techncal revew of system capacty and assessed effcency by measurng nput, process, and output related factors [1]. The revew focused on such tems as data errors, control procedures, and detecton of target error classes n the accounts [5]. Whle such revews served a useful purpose, they nevertheless gnored the crtcal success factors that contrbute to other performance-

2 6 Informatca Economcă vol. 14, no. /010 related attrbutes of an nformaton system [35] [38]. The concept of Crtcal Success Factors (CSF) s ntutvely appealng because t focuses attenton on mportant organzatonal ssues as opposed to focusng only on a few techncal or technology-related tems. The CSF methodology s a top-down methodology that not only dentfes crtcal success factors but that also smultaneously dentfes the key nformaton attrbutes a decson maker must focus on [7] [9] [3]. In ths sense, CSF s an nformaton resource approach to managng nformaton systems rather than an nformaton functon approach that only deals only wth techncal revew. Although CSF s a manageral and organzatonal approach to understandng nformaton systems, the beneft of CSF methodology s that t requres managers to steer away from beng operatons managers and become nformaton rsk managers. The beneft of dentfyng and managng CSFs s that t demands the manager to focus on major ssues or concerns that an organzaton faces. At the same tme dentfyng and understandng the CSFs s smple and they can be easly communcated to others wthn the organzaton. In the long run, workng on CSFs would help the managers n controllng the factors that would contrbute to nformaton system and organzatonal successes. A number of studes have examned CSF n the context of specfc ndustres. For example, some studes examned the CSFs that are relevant to a specfc ndustry [9]. Other studes appled the CSF concept to specfc nformaton system relablty factors such as plannng, communcatons, and strategy [6] [11] [14]. A few other studes ponted out to the weaknesses n CSF measures [4]. These studes showed the weaknesses observed n CSF measures were the results of a lack of adequate conceptual foundaton that would facltate measurement development and the absence of a rgorous program of measurement valdaton [4]. Ths paper presents a measurement framework that would not only dentfy crtcal success factors but also the relatve crtcalty of the factors dentfed. The paper presents an analytcal procedure that computes the crtcalty of varables, procedures, and processes. The analytcal procedure proposes a measurement procedure that permts weghtng the perceved crtcalty of each success factor and rankng them n the order of prorty. We beleve that the paper would contrbute to the strengthenng the theoretcal underpnnng of CSFs and also provde a practcal tool to evaluate the CSFs that were dentfed durng an assessment of organzatonal nformaton systems. The measurement framework s proposed n the context of a BB e- commerce audt and consders the crtcal factors that would help an audtor n performng an e-commerce audt successfully. The crtcal success factors approach The concept of Success Factors (SF) was ntally developed by Danel [10]. Rockart [33] refned the defnton further and demonstrated the mportance of crtcal success factors when evaluatng the performance of an nformaton system. Rockart emphaszed that crtcal success factors are the most mportant attrbutes that a manager must dentfy and understand when workng towards organzatonal objectves. He ndcated that focusng on CSF s mportant because t ponts to where thngs must go rght. In a smlar ven, [6] also defned CSF as, few thngs that must go well to ensure success for a manager or an organzaton. Rockart [33] proposed four basc types of CSFs vz., ndustry CSFs, strategy CSFs, envronmental CSFs, and temporal CSFs. Rockart found that the ones that are measured get done more often than those that are not measured. Consequently, Rockart suggested that each CSF to be measured and that t be assocated wth a specfc target. Later studes used the Rockart s CSF theoretcal framework to dentfy other factors that nfluenced nformaton system performance [3] [8] [1]. These studes ponted out that, factors such

3 Informatca Economcă vol. 14, no. /010 7 as top management commtment to the use of performance nformaton, decson-makng authorty, and tranng n performance measurement technques to sgnfcantly nfluence nformaton system development and use. The studes also hghlghted techncal ssues such as nformaton system problems and dffcultes n selectng and nterpretng approprate performance metrcs n hard-to-measure actvtes to play an mportant role n system mplementaton and use. Although Rockart s study encouraged other researchers to further nvestgate the relevance of crtcal success factors for gudng organzatonal management, the Rockart s study tself had certan lmtatons. Rockart s defnton of CSFs was qualtatve and ntutve rather than quanttatve. Rockart dd not explan how he computed the SFs and how SFs were later converted nto CSFs. Ths falure to dfferentate between CSFs and SFs confounded the results of many other studes that used the Rockart s theoretcal framework [6] [13] [31] [36]. These studes, most of whch used Prncpal Component Analyss (PCA) to dentfy CSF, dd not consder the crtcalty of the factors that they dentfed as success factors [40]. The advantage of usng PCA, a mathematcal procedure that transforms a number of nterrelated varables nto a parsmonous set of varables, s relatvely easy to compute and understand. The frst PC or Prncpal Component accounts for as much of the varance as mathematcally possble, whle succeedng components account for as much of the remanng varances as possble. The PCA methodology s very useful for analyzng collnear data, where multple varables could be co-related. The multcollnearty, even when present, does not confound the results. Whle PCA, as a methodology, has several advantages, the lmtaton of pror studes was that they used the PCA methodology wthout elaboratng on the method/procedure used to assgn crtcalty. Although, collectvely, the pror studes contrbuted to the development of CSF as an nformaton system evaluaton methodology, the studes were lmted n ther contrbutons because of the ad-hoc approach they took to formalze crtcal performance measures [1]. As ponted out, most of these studes used subjectve approaches to formalze the crtcal measures [17]. Addtonally, n many cases, the researchers used ntuton and personal experences to wegh the crtcal factors and ths weakened the valdty of the results even further [35]. As [] pont out, there were several lmtatons and nconsstences ncludng, econometrc problems n the applcaton of CSF methodologes. Gven the lmtatons of CSF methodologes, we propose a CSF methodology that would be less subjectve than some of the pror CSF methodologes. We beleve that our proposed methodology would help nformaton system managers n dentfyng the crtcal success factors and n quantfyng the crtcalty of the varables that they have dentfed. Therefore, we propose a measurement procedure that correlates the relatonshp between a crtcal factor and ts contrbuton to organzatonal success. We beleve that our methodology could be appled to any emprcal study that examnes perceptons, outcomes of respondents or processes n the context of crtcalty. Snce the methodology s not ssue-specfc or ndustry-specfc, the methodology could be appled across ndustres of dfferent dmensons and technologcal or organzatonal sophstcatons. Snce CSF theory s wdely used n such dverse dscplnes as audtng, accountng, fnance, and marketng scences, we also beleve that the methodology would be found useful by researchers from multple dscplnes. The specfc objectves of ths paper are: (1) propose measurement procedures that would compute CSFs from amongst SFs (we wll use the PCA for ths purpose); () test and valdate the proposed procedures usng a numercal llustraton; and (3) valdate the results further by usng a hypothetcal and emprcal datasets.

4 8 Informatca Economcă vol. 14, no. /010 3 Methodology The methodology we propose s based on the followng factors: 1) that responses from ndvduals wll be avalable that dentfes crtcal success factors and factors that are not crtcal related to a manageral/technologcal ssue; ) that we can ndependently measure the subjectvelydetermned crtcal varables obtaned from the respondents; and 3) that by usng PCA, we can assgn weghts and use the weghts to dfferentate between crtcal and non-crtcal attrbutes. The methodology s developed on the assumpton that a researcher can obtan perceptual (or subjectve) responses from respondents on crtcal and non-crtcal factors. Such responses can be collected ether by conductng an experment or through a survey nstrument. The responses (ether n an experment or a survey nstrument) can be requred on a two separate fve pont Lkert scale nstruments. The frst Lkert scale nstrument would provde responses on whether an tem or attrbute s a success factor and the second Lkert scale nstrument would show whether the success factor that was dentfed by the respondents s also a crtcal factor. The responses from the frst Lkert nstrument would be analyzed usng PCA to understand the mportance of each success factor (based on ther relatve varances). Later, we wll apply the methodology that we have developed to the success factors dentfed from the frst stage analyss to understand the crtcalty of each success factors. The analyss wll be performed n two stages. The frst stage s a drect method; that s, the methodology would be appled to PCA weghts (loadngs) obtaned from the orgnal PCA that dentfed the success factors. The second stage s an ndrect method and requres that the nvestgator assgn weghts to the orgnal observatons (or varables) n the study and then apply the PCA to the weghted varables. Therefore, n the frst stage, the success factors are dentfed by a subjectve approach whle, n the second stage, the crtcalty of each success factor s determned usng a pre-weghted varables. Whle, the second approach also nvolves a certan amount of subjectvty, unlke n the frst approach, the pre-weghts can be modfed to sut organzatonal and technologcal objectves and strateges. The steps are further elaborated on the followng paragraphs. Suppose we have p correlated varables measurng certan characterstcs or judgments of ndvdual attrbutes n a populaton. These varables could ether be measured on a Lkert scale (1-5) or they could be contnuous varables. The p varables would form a p -dmensonal vector, x ( x1 x p) wth a covarance matrx. From here on, when applyng PCA, we wll use the populaton-based covarance matrx nstead of the sample covarance matrx. The prncpal components or PCs of the p -dmensonal populaton are defned as the uncorrelated p varables y1 y p such that each y s a lnear combnaton of the orgnal components of x and has maxmum varance among all y s that can be formed as lnear combnaton of the x s. In other words, the PCs y 1 y p are uncorrelated and carry as much of the nformaton (or varablty) from the orgnal varables x1 x p as possble. Thus, the PCA reduces a large number of varables to a small set of varables that would explan most of the varablty n the orgnal varables. Let us now denote the ordered pars of egenvalues and egenvectors of by: ( 1 e1) ( p e p), where e ( e 1 e p) s the th egenvector correspondng to the th ordered egenvalue,. The PCs are then gven by y e x e x e x, p p y e x e1x1 e px p,..., y e x e x e x p p p1 1 pp p and the components of the th egenvector are at tmes called loadngs of the th prncpal

5 Informatca Economcă vol. 14, no. /010 9 component. The egenvectors are normalzed and orthogonal to each other,.e., the product ee (whch s smply the sum of squared j loadngs for the th PC) s zero f j and 1 otherwse. It s known that, var( y ) for 1 p and, p p p var( y ) var( x ) That s, the total varablty n the orgnal varables s equal to the total varablty carred by the new lnear combnatons (.e., the PCs) and ths, n turn, s gven by the sum of the egenvalues. In the followng secton, we dscuss how we can ncorporate crtcalty of nformaton both at the prncpal component loadngs stage and at the raw varable stage. 4 Incorporatng crtcalty at the Prncpal Component Stage Suppose wth each p varable, we also measure how crtcal the varable s; that s, observe for each x another varable, say on a 0-1 scale, where 0 = non crtcal and 1 = crtcal. We could then estmate from the sample data, the lkelhood of the th varable beng crtcal P( x 1) P. The value of P would be between 0 and 1. Let us also suppose that we develop a crtcalty ndex for the th varable as p c P P (1) 1 The formula s computed as P dvded by the summaton The ndex obtaned from equaton (1) represents the proporton of crtcalty lkelhood assocated wth the th varable (as compared to the total of such lkelhoods, summed over all the varables). Ths ndex vares between zero and one and f all varables are equally crtcal then t assgns a crtcalty of 1 p (.e., recprocal of the number of varables) to each of the orgnal p varables. Furthermore, c 1 1. Now, consder the squared correlaton between the j th varable, x, and the th PC, y, defned as j p j e j jj The formula could be nterpreted as the proporton of varablty n x explaned by the th PC y [8]. The product of the squared correlaton and the crtcalty ndex c could be used as a measure that j ncorporates both the crtcalty of the j th varable and ts mportance or rank n the th PC. Therefore, through equaton (), we are able to smultaneously compute both crtcalty and varablty of the orgnal varables: p p p j j j j jj j j 1 j 1 j 1 c e c () Smlar to the matrx of squared correlatons, ( ), whch s often used to select the most j j relevant varables of the orgnal x s [], the matrx ( ) ( c ), j 1 p can be j j j j j used to select varables that are most crtcal, among the orgnal varables. The followng procedures could be used to select the most sgnfcant set of varables: Perform a PCA and compute the quanttes and j, j 1 p accordng to equaton (); Order the PCs accordng to the descendng order of magntude of ther s, and select the frst q PCs that gve a desred total of the s, (say, q 1 90 ). Ths s smlar to the selecton of the frst PCs that explan a desred cumulatve proporton of varablty. Arrange the j s of the selected q PCs n a descendng order of magntude and select (gnorng repettons) the q varables that have the largest j s. 5 Incorporatng crtcalty at the varable level The second approach to ncorporatng crtcalty nformaton at the varable level s to pre-weght the orgnal varables usng Equaton (1) and then apply the PCA to the j

6 10 Informatca Economcă vol. 14, no. /010 weghted varables. Pre-weghtng the orgnal varables s equvalent to weghtng the populaton (or sample) covarance matrx by p p dagonal matrx shown below: c C 1 0 c cp Therefore, the weghted PCs would be derved from the covarance matrx, c C C and we could denote them as Crtcalty Weghted PCs (CWPCs). In ths approach, loadngs of the CWPCs as well as the egenvalues of c (or the varances of the components) reflect both crtcalty and varablty n the CSFs. When we rank the CWPCs from the hghest to the lowest, the frst varable or CWPC s the one that carres the most crtcalty and e most varablty among all the CSFs followed by the second varable or CWPC and so on. That s, we can nterpret the egenvalues (or varances) of the CWPCs as the amount of varablty-crtcalty explaned by the components. The followng summarze the steps requred to selectng the most crtcal varable from CWPC: Multply the orgnal varables by ther crtcalty ndces c of Equaton (1) and then perform the PCA on the new varables and compute the quanttes for j 1 p. Select the frst q PCs that gve a desred cumulatve proporton of varablty j explaned (say, q p 1 j 1 j =.90). Arrange the j s of the selected q PCs n a descendng order of magntude and select (gnorng repettons) the q varables that have the largest s. j 6 Valdaton of the Methodology 6.1 Valdaton usng hypothetcal data As dscussed n the ntroducton, one of the objectves of ths paper s to valdate the methodology that we have proposed usng both a hypothetcal data and a real-world data. In ths secton, we dscuss the valdaton of the methodology usng a hypothetcal data. Consder the followng covarance matrx related to a populaton: The matrx represents three ndependent varables x 1, x, and x 3 and three component varables y 1, y, y 3. From the three ndependent varables denoted n the matrx, from the perspectve of varablty, varable number two or x s the most mportant varable (Please see Table 1 for the Prncpal Component loadngs). Accordng to Table 1, the total varablty explaned by the three component varables y 1, y, and y 3, s ( = 8.000). Therefore, the frst prncpal component y 1 explans 73% of the varablty (5.88 /8.000) and y explans 5% of the varablty. Table 1. PC analyss showng squared loadngs. Prncpal Components Varable y 1 y y 3 Squared loadngs e ) ( j x x x Varance Cum. proporton

7 Informatca Economcă vol. 14, no. / We can use several dfferent approaches to dentfy the mportant varables from the PCA. One of these approaches s to choose the most mportant PCs (e.g., those explanng a cumulatve varablty of 90% or more). We can then order the squared (or absolute) correlatons between the orgnal varables and the PCs and choose the varables wth the hghest correlaton (excludng repettons). As per ths approach, n our hypothetcal example, we wll choose PCs y 1 and y because they collectvely explan 98% of the varablty ( out of 8.000). When we order the squared correlatons, we wll then, select x as the two most relevant varables. x and 3 At ths pont, we can consder the crtcalty matrx C dag ( 6 3 1) whch shows that x to be the most crtcal varable followed 1 by x. Table reports both the quanttes and j jc j. The ndcates that when the PCs are ordered as before, the frst varable x 1 (0.51) to be the most mportant varable followed by the second varable x (0.98). Table also shows that the two PCs, x 1 x, have cumulatve of 91% ( ). We, therefore, choose x 1 x, as the two most mportant varables from varabltycrtcalty pont of vew. Table. PC analyss showng squared correlatons and squared correlatons tmes crtcalty ndces Prncpal Components Varable y 1 y y 3 Squared correlatons ( j ) x x x The quanttes ( ) jc j x x x We can now use the CWPC approach, where x1 x x 3 by the matrx C and then perform a we frst weght the orgnal varables PCA. Table 3. Crtcalty weghted PC analyss (CWPCA) showng squared loadngs correlatons Prncpal Components Varable y 1 y y 3 Squared loadngs ( e j ) x x x Squared correlatons ( j ) x x x Varance Cum. proporton

8 1 Informatca Economcă vol. 14, no. /010 Table 3 reports the squared loadngs and correlatons along wth egenvalues and cumulatve proporton of varablty explaned by each CWPC. In ths approach, the frst CWPC explans about 98% of the total varablty and as per the process explaned n Secton 3.., we wll agan 7 Valdaton usng emprcal data In ths secton, we wll dscuss the valdaton of the methodology usng a real-world data on CSFs. The data set for ths analyss was obtaned from a recently completed emprcal study conducted by the frst author. The study used 03 audtors wth experence n e- commerce audt (BB audts) to gve ther perceptons about the varous CSFs that are mportant to an audtor to perform a BB audt effcently. The measurement scale conssts of 38 tems/varables placed n random order n the questonnare and each varable s measured on a 5-pont Lkert. For smplcty, we label the varables 1 through 38. Each audtor-respondent was asked to ndcate ther perceptons on a 5-pont Lkert Scale about how crtcal each varable was for a successful performance of an E- select x1 x as the most mportant varables. Ths concluson s consstent wth our earler two approaches. We must cauton that ths type of perfect agreement among the varous approaches s not always true. We wll dscuss ths ssue further n the next secton commerce audt. Once the 03 responses were collected, we converted the correspondng crtcalty by dchotomzng the crtcalty varable nto a zero-one varable (1 f the Lkert score was 4 or 5 and 0, f otherwse). Although ths would result n losng some of the nformaton content, we resorted to ths procedure purely for the sake of brevty n explanng the methodologcal approach. However, the methodologcal process that we dscus s capable of handlng contnuous varables but, t would be far more elaborate than what s explaned n the paper. For each of the 38 dchotomzed questons, we estmated the P (correlated varables measurng certan characterstcs or judgments of ndvdual attrbutes n a populaton). Fg. 1. Plot of order of the orgnal prncpal components vs. ther order accordng to the measure n equaton ()

9 Informatca Economcă vol. 14, no. / Fgure 1 shows the orderng of the PCs accordng to ther s of equaton (). We found that the frst 0 PCs to explan 90% of the varablty. Therefore, we extended our analyss further to dentfy the frst 0 varables by mportance from among the 38 varables used n the study. Table 4 shows the rank orderng of the frst twenty varables n the order of hghest to the lowest crtcalty along wth Crtcalty Weghted PCs (CWPC). Column no. shows the rankng of the varables from hgh to low rankng and by ther varable numbers. The thrd and the fourth columns represent the rankng of each of these varables accordng to CWPC and measures, respectvely. Columns three and four, for example, pont out that, varable no. 37 s the most crtcal accordng to both CWPC and rankngs (Columns 3 and 4 respectvely). In contrast, varable no. 36 s ranked as second by the crtcalty approach (Column ) and as 3 rd by CWPC and as 5 th by the approach (Columns 3 and 4 respectvely). Varable no. 35, ranked by crtcalty approach as the 3 rd most mportant, s not even ranked wthn the top 0 varables ether by the CWPC or approaches. The CWPC approach, when compared to the crtcalty approach, fals to capture 7/0 (35%) of the twenty most crtcal varables, whle the approach fals to capture 5/0 (5%) of the twenty most crtcal varables. In Table 4, the letters NA ndcates that the varable s not ranked n the top 0 accordng to ether CWC or approaches. We must pont out that, f we are nterested n capturng a sgnfcant proporton of the varance (e.g. 90%), we must consder all 34 ranked PCs that collectvely explan 90% of the varablty and crtcalty and not restrct ourselves to just the 0 crtcal varables shown n Table 4. Column 3 and 4 report the rankngs of the varables n Column accordng to the CWPCA and the -measures, respectvely. Table 4. The most mportant 0 varables based on crtcalty alone (second column) and on PCA (ffth column) ndex Crtcalty CWPC ALPHA PCA CWPC ALPHA NA NA NA NA NA NA NA NA 18 NA NA NA NA NA NA NA NA NA NA NA 17

10 14 Informatca Economcă vol. 14, no. /010 Smlarly, columns 6 and 7 report the rankngs of the varables n column 5. Column 1 s a sequental ndex to facltate easer readng of the table. When we consder the top 34 varables ranked by each method and by PCA and crtcalty approaches, we fnd that the CWPCA approach fals to capture 4/34 (1%) of the crtcal varables; the PCA approach fals to capture 3/34 (9%) of the crtcal varables whle, the approach fals to capture only 3/34 (9%) and /34 (6%) of the crtcal varables respectvely. These results make us conclude that the measure may be the most accurate compared to the CWPCA. 8 Summary and Conclusons In ths paper, we presented a theoretcal framework for measurng the crtcalty of success factors, n the context of an nformaton system audt. We proposed an analytcal procedure that would compute the crtcalty of varables, procedures, and processes. The analytcal procedure provded a means to weght the perceved crtcalty of a success factor. We developed the analytcal procedure wth the ntent of overcomng some of the lmtatons of CSF methodologes ponted out n the lterature. The methodology that we proposed s less subjectve and could be appled to any emprcal study that examnes perceptons, outcomes of respondents or processes n the context of crtcalty. We proceeded on the bass that we could obtan perceptonal responses from ndvduals on factors that they consder crtcal. Identfcaton of success factors s subjectve and cannot be avoded. However, our methodology proceeded on the bass that we can ndependently and objectvely measure the crtcalty of the success factors dentfed by the respondents and that by usng Prncpal Component Analyss (PCA), we could assgn weghts and use the weghts to dfferentate between crtcal and noncrtcal attrbutes. The process requred that we obtan ndvduals to respond on two separate fvepont Lkert scale nstruments ndcatng whether certan attrbutes related to an nformaton system audt a BB ecommerce audt) are 1) mportant success factors and ) f so, whether the factor that they dentfed s also a crtcal factor. We analyzed the responses from the frst Lkert nstrument, usng PCA so that we can ascertan the mportance of each success factor (based on ther relatve varances). Later, usng the success factors we dentfed from the frst analyss, we proceeded wth a second stage analyss n whch we measured the crtcalty of each success factor. The frst analyss s a drect method and s used to calculate the PCA weghts (loadngs) that also produce a crtcalty ndex. The second analyss s an ndrect method and nvolves assgnng weghts to the orgnal observatons (or varables) n the study and then applyng the PCA to the weghted varables. Thus, the two-stage analyss nvolved both a subjectve and objectve determnaton of the crtcalty of varables. Once we computed the PCA weghts, crtcalty ndex, and had dentfed the crtcalty of the varables, we performed further analyss to valdate the methodology. The valdaton requred that we use both a hypothetcal data and a real-world data. The real-world data was obtaned from a recently completed emprcal study conducted by the frst author. The measurement scale conssted of 38 tems/varables placed n random order n a questonnare and each varable was measured on a 5-pont Lkert. Applyng the methodologcal procedures we had developed and explaned n the paper, usng the hypothetcal data, we were able to dentfy two varables that collectvely explaned 98% of the varablty. We concluded that the two varables that were most crtcal for the success of an nformaton system audt. We then weghted the orgnal varables and agan appled the PCA to the weghed varables. Our results showed that the two varables that we had dentfed from the earler subjectve assessment to be dentfed as crtcal varables agan. Whle ths result usng the

11 Informatca Economcă vol. 14, no. / hypothetcal data was a 100% valdaton of the methodology we had proposed, we were cautous to pont out that such perfect agreement s not always be possble. We later repeated the valdaton of the methodology usng real-world data on CSFs. We estmated the P (correlated varables measurng certan characterstcs or judgments of ndvdual attrbutes n a populaton) for the 38 questons used n the survey. Our analyss showed that the frst 0 PCs to explan 90% of the varablty and also provde a rankng of the 0 varables from the hghest to the lowest order n terms of crtcal mportance. When we compared the results to the varous measurement factors that we had used, we found the rankngs not be consstent across measurement scales. Whle some varables were consstently ranked as most mportant or crtcal, other varables were not even dentfed as the top 0 crtcal varables by some of the measurement approaches. For example, the CWPCA approach faled to capture 35% of the top 0 varables that were most crtcal accordng to the crtcalty measure whle, the measure faled to capture only 5% of those varables. On the other hand, CWPC and the measure faled to capture 5% and 10%, respectvely, of the top 0 most mportant varables accordng to the PCA. Thus, we can conclude that the measure appears to more accurate than the CWPC as t captures greater proporton of both varablty and crtcalty. Of course, PCAs and crtcalty measure are desgned to capture varablty and crtcalty alone, respectvely, and therefore, although they capture larger proporton of these two features than the measure does, they should not be preferred over the measure, as the latter s desgned as a combned measure of both features. The methodology we had proposed appeared to work well and s useful when a large number of varables have to be ordered accordng to ther mportance both from ther membershp n the success factors lst and from ther crtcalty. The proposed methodology s heurstc and needs further development. Currently, as t s proposed, t suffers from certan lmtatons. One of the major lmtatons s that we converted the fve-pont Lkert scale responses to a bnary varable. Ths could result n nformaton loss. In future, we must extend the process to be able to measure contnuous and ordnal scales. Ths would requre modfcaton of the n equaton 1. In concluson, we can state that our paper s an attempt at developng a methodology for measurng success factors and ther crtcalty wth less subjectvty than has been attempted n pror lterature. We beleve that the proposed methodology makes a contrbuton to the IS lterature n the area of nformaton system performance evaluaton. References [1] N. Agmon and N. Ahtuv, Assessng data relablty n an nformaton system, Journal of Management Informaton Systems, 1987, Vol. 4, No., pp [] N. M. Al Kandar and T. M. Jollffe, Varable selecton and nterpretaton of covarance prncpal components, Communcatons n Statstcs-Smulaton and Computaton, 00, pp [3] R. N. Anthony, Plannng and control systems: A framework for analyss, Graduate School of Busness Admnstraton, Harvard Unversty. Boston, [4] D. E. Avson and G. Ftzgerald, Where now for development methodologes? Communcatons of the ACM, 003, Vol. 46, No. 1, pp [5] T. Bell and I. Solomon, Cases n Strategc Systems Audtng - A Jont Publcaton of KPMG and Unversty of Illnos, Urbana Champagn, 00. [6] A. C. Boynton and R. Zmud, An assessment of crtcal success factors, Sloan Management Revew, 1984, Vol. 5, No. 4, pp [7] C. R. Byers and D. Blume, Tyng crtcal success factors to systems development, Informaton and Management, 1994, Vol. 6, No. 1, pp

12 16 Informatca Economcă vol. 14, no. /010 [8] K. Cavalluzzoa and C. Ittner, Implementng performance measurement nnovatons: Evdence from government, Accountng, Organzatons and Socety, 004, Vol. 9, pp [9] T. Chen, Crtcal success factors for varous strateges n the bankng ndustry, Internatonal Journal of Bank Marketng, 1999, Vol. 17, No., pp [10] R. D. Danel, Management nformaton crss, Harvard Busness Revew, Boston [11] R. L. Dlenschneder and R. C. Hyde, Crss communcatons: Plannng for unplanned, Busness Horzons, 1985, Vol. 8, No. 1, pp [1] R. G. Eccles, The performance measurement manfesto, Harvard Busness Revew, Jan.- Feb. 1991, pp [13] R. Ed, M. Trueman and A. M. Ahmed, A cross-ndustry revew of BB crtcal success factors, Internet Research, 00, Vol., pp [14] C. R. Ferguson and R. Dcknson, Crtcal success factors for drectors n the eghtes, Busness Horzons, 198, Vol. 5, No. 3, pp [15] E. G. Flamholtz, T. K. Das and A. S. Tsu, Toward an ntegratve framework of organzatonal control, Accountng, Organzatons and Socety, 1985, Vol. 10, No. 1, pp [16] S. Jntanapakanont Ghosh, Identfyng and assessng the crtcal rsk factors n an underground ral project n Thaland: A factor analyss approach, Internatonal Journal of Project Management, 004, Vol., No. 8, pp [17] L. A. Gordon and D. Mller, A contngency framework for the desgn of accountng nformaton systems, Accountng Organzatons and Socety, 1976, Vol. 1, pp [18] M. Gosseln, An emprcal study of performance measurement n manufacturng frms. Workng paper, Laval Unversty, 00. [19] S. Hamlton and N. L. Chervany, Evaluatng nformaton systems effectveness Part 1: Comparng evaluaton approaches, MIS Quarterly, 1981, Vol. 5, No. 4, pp [0] S. Hamlton and N. L. Chervany, Evaluatng nformaton systems effectveness Part II: Comparng evaluaton approaches, MIS Quarterly, 1981, Vol. 5, No. 4, pp [1] G. Hofstede, The poverty of management control phlosophy, Management control phlosophy, 1978, Vol. 3, pp [] C. D. Ittner, D. F. Larcker and M. W. Meyer, Subjectvty and the weghtng of performance measures: evdence from a balanced scorecard, The Accountng Revew, Vol. 78, No. 3, 003, pp [3] C. D. Ittner, D. F. Larcker and T. Randall, Performance mplcatons of strategc performance measurement n fnancal servces frms, Accountng, Organzatons and Socety, 003, Vol. 8, pp [4] B. Ives and M. H. Olson, User nvolvement and MIS success: A revew of research, Management Scence, 1984, Vol. 30, No. 5, pp [5] D. Kaplan, R. Krshnan, R. Padman and J. Peters, Assessng data qualty n nformaton, Communcatons of the ACM, 1998, Vol. 41, No., pp [6] S. S. Lee and J. S. Osteryoung, A comparson of crtcal success factors for effectve operatons of unversty busness ncubators n the Unted States and Korea, Journal of Small Busness Management, 004, Vol. 4, pp [7] S. Magal, Crtcal success factors for nformaton centre managers, MIS Quarterly, 1998, Vol. 3, pp [8] K. V. Marda, J. T. Kent and J. M. Bbby, Multvarate Analyss, Academc Press, London [9] B. Moore, Prncpal component analyss n lnear systems: Controllablty, observablty, and model reducton,

13 Informatca Economcă vol. 14, no. / IEEE Transactons on Automatc Control, 1981, Vol. 6, No. 1, pp [30] C. H. Park and Y. G. Km, Identfyng key factors affectng consumer purchase behavor n an onlne shoppng context, Internatonal Journal of Retal Dstrbuton Management, 003, Vol. 31, No. 1, pp [31] K. J. Press, A two-stage process for elctng and prortzng crtcal knowledge, Journal of Knowledge Management, 000, Vol. 4, pp [3] J. F. Rockart, Chef executves defne ther own needs, Harvard Busness Revew, 1979, Vol. 57, No., pp [33] J. F. Rockart, A prmer on crtcal success factors, Publshed n The Rse of Manageral Computng: The Best of the Center for Informaton Systems Research, Homewood, IL: Dow Jones-Irwn [34] J. Salmeron and J. Herrero, An AHPbased methodology to rank crtcal success factors of executve nformaton systems, Computer Standards & Interfaces, 005, Vol. 8, No. 1, pp [35] M. E. Shank, A. Boynton and R. Zmud, Crtcal Success Factor Analyss as a Methodology for MIS Plannng, MIS Quarterly, June 1985, pp [36] Z. Temtme and J. Pansr, Small busness crtcal success/falure factors n developng economes: Some evdences from Botswana, Amercan Journal of Appled Scences, 004, Vol. 1, pp [37] P. Weber and J. Weber, Changes n employee perceptons durng organzatonal change, Leadershp & Organzatonal Development Journal, 001, Vol., No. 6, pp [38] H. Xu, Crtcal success factors for accountng nformaton systems qualty - Dssertaton, Unversty of Queensland, 003. [39] H. Xu and D. Lu, The crtcal success factors for data qualty n accountng nformaton systems dfferent ndustres perspectve, IACIS, 003, pp [40] S. M. Yusof and E. M. Aspnwall, Crtcal success factors n small and medum enterprses: survey results, Total Qualty Management, 000, Vol. 11, No. 4, pp [41] F. Zahed, Relablty of nformaton systems based on the crtcal success factors formulaton, MIS Quarterly, June 1987, pp Jagdsh PATHAK s Assocate Professor of Accountng & Systems at the Odette School of Busness Admnstraton, Unversty of Wndsor, Ontaro, Canada and the lead author of ths research project. He has a PhD n Management Studes majorng n accountng and MIS from the Unversty of Goa.He s an award wnnng author n hs doman of research publshng n scholarly and professonal journals nternatonally. Hs papers have appeared n the Journal of Accountng and Publc Polcy (Elsever), Internatonal Journal of Audtng (Wley), Informaton Systems Management (Taylor & Francs), Manageral Audtng Journal (Emerald) and several other. Dr. Pathak s Edtor (North Amerca) of the Manageral Audtng Journal and Assocate Edtor (Accountancy) of the Internatonal Journal of Appled Decson Scence (Inderscence). He has publshed two research monographs from Alled (New Delh) and Sprnger Verlag (Berln) on the nformaton technology audtng. Hs current research nterests are n the areas of technology assurance, enterprse systems mpact on frms performance, nternatonal corporate governance, nvestor protecton, and nvestor behavour wth respect to nternet-based fnancal dsclosures.. He s an actve member and nomnated executve of strategc and emergng technology and the nformaton systems sectons of the Amercan Accountng Assocaton. He can be contacted at jagdsh@uwndsor.ca.

14 18 Informatca Economcă vol. 14, no. /010 Abdulkadr HUSSEIN obtaned PhD n Statstcs from Unversty of Alberta n Edmonton, Canada, n 003. He s an assocate professor at the Unversty of Wndsor, Ontaro Canada and currently vstng assocate professor at Kng Fahd Unversty of Petroleum and Mnerals, Dhahran, Saud Araba. Hs areas of research nterest are: Bostatstcs, Statstcal qualty control, Healthcare outcomes research, Mxture models, and Sequental analyss n clncal trals. He has publshed more than 0 peer revewed journal artcles. Ram SRIRAM s Controllers RoundTable Dstngushed Professor at the School of Accountancy, Robnson College of Busness, Georga State Unversty, Atlanta. He earned hs doctorate n 1987 from Unversty of North Texas. He s also a Certfed Publc Accountant (CPA). He s also a vstng professor at Caro Unversty, Egypt, Great Lakes Insttute of Management, Inda and Indan Insttute of Management, Bangalore. He s the author of more than 60 artcles n leadng journals such as Management Scence, Decson Scences, Accountng Organzatons and Socety and others. He specalzes n nformaton system theores and methodologes as they relate to accountng ssues. Ram serves n the edtoral boards of several leadng academc journals ncludng the Journal of Informaton System, Internatonal Journal of Accountng Informaton Systems Ejaz AHMED s Professor and Head of the Department of Mathematcs and Statstcs at the Unversty of Wndsor. He s a foundng Drector for Centre for Statstcal Consultng, Research, Learnng and Servces (CSCRLS). Dr. Ahmed has nducted as a Fellow of Amercan Statstcal Assocaton n 009. He s an elected member of the Internatonal Statstcal Insttute and a Fellow of the Royal Statstcal Socety. He s the book revew Edtor of Technometrcs. Dr. Ahmed serves as an Assocate edtor of several nternatonal statstcal journals, ncludng Computatonal Statstcs and Data Analyss (CSDA) and Journal of Statstcal Computaton and Smulaton (JSCS). He was a guest coedtor for Lnear Algebra and ts Applcatons (LAA). He served as a Board of Drector and Charman of the Educaton Commttee of the Statstcal Socety of Canada. Hs area of expertse ncludes statstcal nference, asymptotc theory, statstcal educaton and consultng. He has over 100 publshed research artcles n scentfc journals. Further, he co-edted two volumes and organzed many sessons and workshops. Recently, he co-authored a textbook on probablty and statstcs. He made over 100 nvted scholarly presentatons n varous countres around the globe. He supervsed several graduate students ncludng Ph.D. students.