A MULTI-CRITERIA APPROACH FOR THE ASSESSMENT OF E- GOVERNMENT SERVICE QUALITY

Transcription

1 Page 1 A MULTI-CRITERIA APPROACH FOR THE ASSESSMENT OF E- GOVERNMENT SERVICE QUALITY Xenia Papadomichelaki 1, Vicky Koutsouris 2, Dimitris Konstantinidis 3, Gregoris Mentzas 4 Abstract This paper proposes a measurement model for evaluating e- government service quality, and generates quality indicators to assist practitioners in identifying the strong and weak points of service delivery. Our statistical sample is based on questionnaires of citizens preferences for the evaluation of an ideal e-government site. To accurately evaluate the influence of dimensions and criteria on service quality our model applies the Analytic Hierarchy Process (AHP), a multi-criteria decision-making method. Our results show that Reliability is the most important quality dimension, while Citizen Support, Trust and Efficiency follow in importance. 1. Introduction Quality of service in the public sector has become an issue of great concern. Many organizations try to self-assess and measure the quality of service delivered. At the same time, significant progress has been made in the development of e-government services. While more and more citizens and enterprises use the online services, manifold problems related to quality of public e-services still exist, according to the Top of the Web survey [1]. Not being able to find the needed service/information, difficult use of e-services, the need for better help regarding the e-service provided on the website, the language understandability etc. are some of frequently reported problems. The Council for Excellence in Government [2] states that many non-governmental users do not interact with government online because of difficulty in finding the website information they want. Schultz [3] urges that e-government takes a user service approach to close the gap. It seems that e-government fails to achieve its full potential unless the importance of service quality is recognized and explicitly addressed. Current research on e-government service quality is mostly descriptive and only discusses some of the aspects inherent in service quality. Research has been conducted for e-government by collecting users opinions about the factors that characterize the quality of an e-government web page [4], [5], while some efforts try to evolve an instrument that was developed Institute of Communication and Computer Systems School of Electrical & Computer Engineering, National Technical University of Athens, 9 Iroon Polytechniou str. Zografou Campus, Zografou, Athens, Greece 1 xpg@central.ntua.gr 2 v_koutsouris@yahoo.com 3 konstadimit@hotmail.com 4 gmentzas@mail.ntua.gr

2 Page 2 A MULTI-CRITERIA APPROACH FOR THE ASSESSMENT OF E-GOVERNMENT SERVICE QUALITY originally for assessing user perceptions of the quality of e-commerce web sites [6], or identify evaluation criteria and assess user perceptions [7], [8]. Quality indicators have been introduced in e-government service to evaluate the quality delivered to citizens. To achieve this goal, a selection of a range of quality indicators appropriate for e-government service was made. Quality evaluation models in e-government usually encompass several dimensions, and each dimension is further split into numerous criteria. When choosing an appropriate range of service quality measures, it is necessary to balance these measures, to ensure that one dimension, criterion or a set of dimensions or criteria, is not emphasized to the detriment of the others. Moreover, the quality indicators selected must be measurable, and allow practitioners to monitor service quality. Although there has been a significant research on e- government service quality most of it is mainly exploratory in nature. To the best of our knowledge, there is no standard framework or benchmark defining e-government service quality. This paper proposes a measurement model for evaluating e-government service quality, and generates quality indicators to assist practitioners in identifying the strong and weak points of service delivery. Our model analyses the relationships between dimensions and criteria for assessment with a multi-attribute decision analysis method. Our statistical sample is based on 467 questionnaires of citizens preferences for the evaluation of an ideal e-government site. However, how to best balance these indicators is an important issue. An incomplete measurement model can result in inappropriate actions that may harm organizations e- government service delivery. To accurately evaluate the influence of these dimensions in terms of levels of criteria we used the Analytic Hierarchy Process (AHP). AHP is a flexible multi-criteria decision-making method used to effectively synthesize the judgments given by a team of experts in order to make better decisions in complex settings where both tangible and intangible criteria are considered [9]. The remainder of the paper is organized as follows: Section 2 contains a review of the literature pertaining to criteria for e-government service quality assessment followed by a brief presentation of multi-criteria approaches in e-government quality. In section 4 the proposed quality evaluation model is described while the paper ends with discussion and conclusions. 2 Criteria for e-government service quality assessment In order to achieve high quality websites, designers have to firstly understand the different quality dimensions that affect user expectations, and then relate these quality characteristics to specific design features [10]. Quality dimensions of the approaches that focus on the quality of the service delivered itself depend on attributes such as availability, usability, security etc. and priorities and needs of the receivers of the service. Also there is much work done on E- Commerce website quality, online service quality approaches, B2C e-commerce web site quality. It also seems that the area of quality of Web sites is so broad that some researchers decided to work on specific quality dimensions. For example, work has been done in the evaluation of information quality [4] and on handling accessibility as a quality dimension [5], [11]. Furthermore, research has also been conducted in examining tangibility as a quality factor [12]. In the area of e-government research has also been conducted on collecting users opinions about the factors that characterize the quality of an e-government web page [4], [5], and on identifying evaluation criteria and assessing user perceptions for the quality of e- government sites [6], [7], [8].

3 Error! Reference source not found. Seite 3 Taking into account the aforementioned quality features, a set of service quality dimensions and criteria were assessed reflecting all aspects of citizen perceptions in an e-government service quality model established in our previous research [13]. This model included a wide experimental survey and resulted in a set of quality features that sufficiently describe citizen perceptions and preferences. In the present paper we extend the above mentioned model and evaluate the weights assigned at the different levels of criteria, by adopting a multi-criteria decision analysis method. 3 Multi-criteria approaches in e-government quality Multi-Criteria Decision Analysis (MCDA) is a discipline aimed at supporting decision makers faced with making numerous and sometimes conflicting evaluations. MCDA aims at highlighting these conflicts and deriving a way to come to a compromise in a transparent process. Some of the most recent applications of MCDA to the quality of websites include e.g. web site satisfaction benchmarking analysis [14]; evaluation of the success of Web portals by benchmarking user perceived impact [15]; assessment of the multiple dimensions which might affect user satisfaction from a web portal [16]; citizen satisfaction from e- government web sites [17]; assessment of accessibility of federal e-government sites [18]; analysis of e-government websites in order to produce relevant rankings [19]; and website quality evaluation for measuring the performance of government websites [20]. A review of this literature shows that most research on the establishment of a set of scientific, integrated and practical evaluation criteria system is mostly descriptive and only discusses some of the aspects involved. Since the existing literature of e-government service quality is not yet rich enough to provide a sound conceptual foundation, exploratory research is required to develop a deeper understanding. Therefore, the aim of the present study is to improve the criteria weighting process in e-government site quality by means of MCDA methods. Among all the MCDA techniques widely used to prioritize alternatives many comparisons have revealed that both the AHP and the ANP possess a number of benefits over the other MCDA methods, such as: (i) they provide a realistic description of the problem, (ii) they support group decision-making, (iii) they soundly structure the decision-making process, (iv) they incorporate both quantitative and qualitative factors, (v) they clearly express the relative importance of factors, (vi) they allow the decision makers to focus on each small part of the problem, (vii) they facilitate the evaluation of alternative scenarios, by supporting what if and sensitivity analysis. For the above mentioned reasons we have chosen these two methods. Analytic hierarchy process (AHP) was first proposed by Thomas L. Saaty in 1971 and was published in 1980 [21]. It has been widely applied in areas such as multiple criteria decision making and has become a popular application for performance evaluation. AHP works well under the assumption of the independence of criteria. Therefore, in order to avoid the drawback mentioned above, Saaty developed ANP that simultaneously takes into account both the relationships of feedback and dependence [22]. In this paper we use the AHP method and then we validate our results with ANP in order to ensure that there is no interdependence among the quality measures involved. AHP is based on the three following principles [23]. (1) The experts define the elements of the problem (i.e. decision criteria) and arrange them in the form of a hierarchy of objectives with parent elements in a given level connected to their children elements in a level below. The top level of the hierarchy represents the goal of the problem, while the bottom level contains the alternatives that can be chosen to maximize the objective. The first and the last level are connected through a series of intermediate levels, which represent the sub-criteria and other concerns in which the goal is decomposed. (2) The experts assess (i.e. weight) the relative importance of criteria, sub-criteria and alternatives

4 Page 4 A MULTI-CRITERIA APPROACH FOR THE ASSESSMENT OF E-GOVERNMENT SERVICE QUALITY with respect to the elements in the higher level to which they are connected. (3) All the judgments throughout the structure are used to derive corresponding priority scales that are then synthesized to determine the overall priorities of the alternatives. 4 Quality evaluation model To establish an e-government service quality performance evaluation model, we followed the next three-step procedure: building preliminary model/scale; modifying dimensions and criteria; and building an evaluation model/scale. Step 1. Building the preliminary model/scale. In this step, this study refers to the literature reviewed in Section 2. After two evaluation rounds 33 e-government quality criteria remained in the list classified under six main criteria determined as the e-government service quality dimensions: Ease of Use (navigation personalization, technical efficiency), Trust (privacy, security), Functionality of the Interaction Environment (support in completing forms), Reliability (accessibility, availability), Contents and Appearance of Information and Citizen Support (Interactivity) [18]. Step 2. Modifying the preliminary model. In order to validate, refine, purify, evaluate the model and the scale and also test the model s and the scale s factorability two rounds of data collection and analysis took place. This process resulted in the reduction of the dimensions from 6 to 4 and in the drop of 12 criteria. The following are the four dimensions and 21 criteria in the modified model: Efficiency: The ease of using the site which has also been noted by other researchers [24], [25] and the quality of information it provides. The attributes of this dimension are: how clear and easy is to follow the structure of the site (Structur), how effective is the site s search engine (SearchEng), how well the site s map is organized (SiteMap), how well the site can be customized to individual user s needs (Customiz), whether the information displayed in the site is appropriate detailed (INDetail), whether the information displayed in the site is appropriate fresh (InUp2Dat) and whether there is enough information about the site s fields completion (FRpreFil) Trust: The degree to which the citizen believes the site is safe from intrusion and protects personal information. The importance of trust as a critical aspect of e-service has already been stressed in other studies [25], [26]. The attributes of Trust dimension are: the degree that the acquisition of username and password in the site is secure (SafeCode), the amount of personal data required for authentication on the e-government site (PDMinim), whether the data provided by users in the site are archived securely (PDSafety) and whether the data provided by users in the site is used only for the reason they were submitted on the first place (PDUse) Reliability: The feasibility and speed of accessing, using and receiving services of the site. Reliability as a critical aspect of e-service has already been stressed in other studies such as, [27], [28] and [29]. The attributes of this dimension are: the speed of downloading forms from the e-government site (FRFasrAp), whether the site is available and accessible whenever the citizen needs it (SiteAvai), the extent to which the site performs the service successfully upon first request (SRSucces), whether the services provided by the site are in time (SRInTime), how fast the site s pages are downloaded (FastDown) and whether the e-gov site works properly with any default browser (BrowsCom).

5 Error! Reference source not found. Seite 5 Citizen Support: The ability to get help when needed. Nevertheless as mentioned in the literature the Citizen Support dimension applies only when citizens experience problems. The attributes of Citizen Support dimension are: the interest shown by the employees in solving the citizen s problem (HDIntere), whether the employees give prompt replies to users inquiries (HDAnswer), whether the employees have the knowledge to answer the users questions (HDKnowle) and the ability the employees have to convey trust and confidence (HDTrust). Step 3. Building the evaluation Model/Scale. After identifying the dimensions and the criteria for evaluating e-government service quality performance we formulated the framework shown in Figure 1. The framework can be divided into three levels. The first level is the overall goal. The dimensions for evaluating the performance of an e-government website are listed in the second level; these dimensions are independent. Each dimension has several criteria that are listed in the third level. We assume that the relationship between the second level and the third level is a strict hierarchical structure in each and every dimension of the proposed framework. After building the AHP-based framework, pairwise comparisons are performed. As shown in Figure 1, the relationship between factors in the first and second levels (part A) and between the criteria of each dimension in the third level and respective dimension of the second level follow a strict hierarchical structure. Therefore, the AHP approach was applied once to part A and six times, once for every dimension of the proposed framework, in part B. In this step, citizens opinions were used in order to perform pairwise comparisons and identify the interrelationships among the dimensions and criteria. These interrelationships provide the foundation for prioritizing improvement efforts when attempting to maximize the return from performance improvement activities. More precisely, questionnaires were filled by a sample of 467 citizens-users of Greek e-government web sites, that set scores regarding to the dimensions of the ideal e-government web site both for the dimensions and the criteria of e- government quality service assessment. Saaty s 1-9 scale [30] was utilized to gauge answers, in which 1 indicates equal importance, 3 indicates moderate importance, 5 indicates strong importance, 7 indicates very strong importance and 9 indicates extreme importance. Even numbered values fall in between importance levels. The dimensions in level 2 are compared in terms of their contribution to achieving the primary objective. After each element is compared, a paired comparison matrix (A) is formed. Finally we check the consistency property. Since CR IS <0.1, the comparison matrix in the example is consistent. The eigenvectors and consistent ratios for the criteria of the respective upper level dimensions (efficiency, trust, reliability, citizen support), are given in Table 1. The weights of each criterion (DCW) in Table 1 are obtained by using the AHP approach between the criteria of each dimension in the third level and respective dimension of the second level, while the weights of each dimension (CW) are obtained by using AHP in level 1. Finally, in order to address the AHP s limitation on the interdependence among the evaluation criteria we applied ANP in order to ensure that there is no any dependent and feedback relationship. Our results showed that there was no interdependence and that the order of weights of the dimensions of the model resulted exactly the same (the statistical results are not given for reasons of brevity; they are however available from the authors upon request).

6 Page 6 A MULTI-CRITERIA APPROACH FOR THE ASSESSMENT OF E-GOVERNMENT SERVICE QUALITY Figure 1: The AHP- based framework for evaluating e-government service quality. Table 1: The relative importance weights of dimensions and criteria Efficiency CR= Trust CR= Reliability CR= Citizen Support CR= Structur SafeCode FRFasrAp HDIntere SearchEng PDMinim SiteAvai HDAnswer SiteMap PDSafety SRSucces HDKnowle Customiz PDUse SRInTime HDTrust INDetail FastDown InUp2Dat BrowsCom FRpreFil Discussion and Conclusions E-government service evaluation is affected by numerous factors, and the interdependence between these factors is complex. Various studies have tried to identify the factors that influence the service quality delivered by an e-government site and to assess the service delivered [4], [6], [13]. In most of them the assessment of e-government service is qualitative and either does not address the issue of the relative importance of criteria or they assign weights in an arbitrary manner.

7 Error! Reference source not found. Seite 7 In this study, we used the analytical hierarchy process (AHP) to determine the weight of each dimension and criterion detected in an e-government service quality model developed in our previous work. The citizens preferences for an ideal e-government site were adopted in the establishment of estimation of dimensions and criteria weights. These weights show the impact of each dimension and criterion to the overall e-government service quality according to citizens. We found that all of the four dimensions Efficiency, Trust, Reliability and Citizen Support significantly influence citizens evaluation of site quality. Users of e-government considered Reliability as the most important dimension, while Citizen Support, Trust and Efficiency follow in importance. The most important criterion concerning Reliability was the extend that the site is available and accessible whenever the citizen needs it, while for the Citizen Support dimension the most important criterion was the extend that the employees have the knowledge to answer the users questions. In the case of Trust dimension was whether the data provided by users in the site is used only for the reason they were submitted on the first place, whereas for Efficiency it was the extent that a site can be customized to individual user s needs. The model presented here triggers future research on extending the knowledge of quality dimensions affecting e-government in order to more fully develop guidelines for site development and provide researchers and practitioners with a tool to benchmark e- government sites. Through understanding the weights of the dimensions and the criteria that affect e-government service quality a public organization will stand a much better chance in serving citizens. For practitioners, our model and scale can serve a useful diagnostic tool; they can use the validated scale to measure and improve service delivery. Finally our model and scale add to the extant literature by establishing a basis for further research on electronic government service quality. References [1] egovernment Unit, DG Information Society, European Commission: Top of the web: user satisfaction and usage survey of e-government services, (2004). [2] Council for Excellence in Government. The new e-government equation: Ease, engagement, privacy, and protection, (2003). [3] Schultz, J. A. (2001). Information technology in local government: A practical guide for managers. Washington, DC. International City/County Management Association. [4] Eschenfelder, K. R.: Behind the Web site: An inside look at the production of Web-based textual government information. Government Inf. Quarterly. vol. 21, pp (2004). [5] Brebner, M. & Parkinson, M.: The accessibility of New Zealand public library Web sites to people who are blind or vision impaired. In: LIANZA Conference, Wellington, New Zealand (2006). [6] Barnes, S. J. & Vidgen, R.: Interactive E-Government: Evaluating the Web Site of the UK Inland Revenue. Journal of Electronic Commerce in Organisations. vol. 2, no. 1, pp (2003). [7] Korsten, H. & Bothma, T.: Evaluating South African government Web sites: methods, findings and recommendations (Part 1). South African Journal of Information Management, vol. 7, no 2 (2005). [8] Horan, T.A., Abhichandani, T. & Rayalu, R.: Assessing User Satisfaction of E-Government Services: Development and Testing of Quality-in-Use Satisfaction with Advanced Traveler Information Systems (ATIS). In: HICSS 06, vol. 4, pp. 83b -- 83b (2006). [9] Saaty, T.L. How to make a decision: the analytic hierarchy process, European Journal of Operational Research vol, 48 p.p (1990). [10] Zhang, P., von Dran, G.,. User expectations and rankings of quality factors in different website domains. International Journal of Electronic Commerce, vol.6, no.2, p.p (2001).

8 Page 8 A MULTI-CRITERIA APPROACH FOR THE ASSESSMENT OF E-GOVERNMENT SERVICE QUALITY [11] W3C:.Web Content Accessibility Guidelines 2.0. W3C Working Draft 17 May 2007, (2007). [12] Alzola, L. M., & Robaina, V. P.:.Tangibility as a quality factor in electronic commerce b2c. Managing Service Quality, vol. 16, p.p (2006). [13] Papadomichelaki, X. and Mentzas, G.: A Multiple-Item Scale for Assessing E-Government Service Quality. EGOV 2009, 8th International Conference on Electronic Government, Linz, Austria, Lecture Notes in Computer Science, 5693, pp (2009). [14] Evangelos Grigoroudis, Charalambos Litos, Vassilis A. Moustakis, Yannis Politis, Loukas Tsironis: The assessment of user-perceived web quality: Application of a satisfaction benchmarking approach. In European Journal of Operational Research vol. 187, pp (2006). [15] Daniel Delic, Hanz-J. Lenz: Benchmarking User Perceived Impact for Web portals success evaluation. In JIOS, vol. 32, no. 1. Freie Universität Berlin (2008). [16] Manouselis N., Sampson D.: Multiple dimensions of user satisfaction as quality criteria for web portals. In the Proc. Of the IADIS WWW/Internet Conference, Madrid, Spain (2004). [17] Pieter Verdegem, Gino Verleye: User-centered E-Government in practice: a comprehensive model for measuring user satisfaction. In Government Information Quarterly, vol. 26, no. 3, pp (2009) [18] Jaeger, P. T.: Assessing Section 508 compliance on federal e-government websites: A multi-method, user- centered evaluation of accessibility for persons with disabilities. In Government Information Quarterly, vol. 23, pp (2006). [19] Rorissa, A., & Demissie, D.: An analysis of African e-government service websites. In Government. Information Quarterly, vol. 27, no. 2, pp (2010). [20] Bueyuekoezkan, G., Ruan, D.: Evaluating government websites based on a fuzzy multiple criteria decision-making approach. In International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, vol. 15, no. 3, pp (2007). [21] Saaty, T.L.: The Analytic Hierarchy Process, Planning, Priority Setting, Resource Allocation. McGraw- Hill, New York (1980). [22] Saaty, T.L.: Decision making in complex environments: the Analytic Network Process for decision making with dependence and feedback, Pittsburgh, PA: RWS Publications (1996). [23] Saaty, T.L. Fundamentals of Decision Making with the Analytic Hierarchy Process, RWS Publications, [24] Santos, J.: E-service quality: a model of virtual service quality dimensions. In Managing Service Quality, vol. 13, no. 3, pp (2003). [25] Gefen, D., Karahanna, E. & Straub, D.: Trust and TAM in online shopping: an integrated model. MIS Quarterly, vol. 27, no. 1, pp (2003). [26] Zhao, J. & Zhao, S.: Opportunities and threats: A security assessment of state e-government websites. In Government Information Quarterly, vol.27, pp (2010). [27] Sukasame, N.: The development of e-service in thai government, (2004). [28] Anbazhagan, M. and A. Nagarajan: IBM Developerworks Web site. Understanding quality of service for web services. (2002). [29] Sumra R. and D. Arulazi: Quality of Service for Web Services-Demystification, Limitations, and Best Practices (2003). [30] Saaty, T.L.: Theory and Applications of the Analytic Network Process: Decision Making with Benefits, Opportunities, Costs, and Risks. Pittsburgh, PA: RWS Publications. (2005).