Information Systems Effectiveness. Joana Esteves Instituto Superior Técnico Taguspark. September, 2007

Transcription

1 Information Systems Effectiveness Joana Esteves Instituto Superior Técnico Taguspark September, 2007

2 Abstract The evaluation of Information Systems success is a subject that has been widely studied in recent years. The present document aims to demonstrate an analysis carried out on the two main approaches regarding this subject, as follows: from a purely academic basis, the DeLone, McLean and Pitt approach, and from a business environment basis, the ISO/IEC approach. It was also carried out a study on how the two approaches are complementary and still, held a proposal for joint use of them. In order to evaluate the success/effectiveness of Information Systems of a business group/organization, a case study was produced using a service quality performance tool SERVQUAL. Finally, an overall conclusion is carried out, bearing in mind the bridge between theory and practice. Keywords: Success, Approach, Information Systems, Service Quality, SERVQUAL, ISO/IEC Introduction Nowadays, companies invest annually billions of euros in Information Systems (IS). However, how can we assess the effectiveness of these IS? Over the years, several studies have been carried out to determine what are the factors that contribute to the effectiveness of IS and yet, what are the metrics and scales used to evaluate each of these factors. In 1992, DeLone and McLean (DeLone & McLean, 1992) acknowledged the difficulty in defining the dependable variable IS success. After performing a series of studies they created a model, defined by them as an attempt to reflect the procedural and interdependent nature of the IS success. As time passes, the role of a IS department within an organization has suffered a few changes. In 1995, it was already clear that the aims of a IS department didn t consist only on the development of products and operations management, but also in fulfilling the role of service providers. Nevertheless, by this time, Pitt et al. (Pitt et all., 1995) considered that the model developed by DeLone and McLean was incomplete and therefore added a new concept, service quality. Over the years, the model of DeLone and McLean has been tested and studied, which naturally caused some criticism and proposals for its amendment. However, it remains the model of reference used in assessments of IS success. In addition to these assessments, it has been tried to find ways to assure the efficacy and efficiency of the IS. One of them, which study is also presented in this report, is the ISO/IEC Standard (International Standards Organization/ International Electrotechnical Commission) 20000, that has the ITIL (Information Technology Infrastructure Library) as its fundamental basis.

3 ITIL provides guidance on what should be done to provide users, the appropriate IT services to support their business processes. The ITIL certification is available at the individual level, but until recently, it was not possible to prove that an organization was to work on the recommendations of ITIL. ISO/IEC was designed to fill this gap. Created by the IT Service Management Forum (itsmf) and the British Standards Institute (BSI), the standard is modelled under the same principles as ITIL and for the first time, makes it possible for organizations to ensure their IT services management, hence being the first IT Management International Standard. Unlike the existing ITIL documentation, ISO/IEC does not describe the way processes should be designed. It however defines a number of requirements that must be met to enable the certification. ISO/IEC allows service providers to understand how to enhance/improve the quality of services they provide to their customers, being these internal or external. The study of service quality is therefore subject to two types of approach that come from different sources. The "academic" and traditional, initiated by Delone, McLean and Pitt in the 90s, and the "business", which resulted in the publishing of the British Standard (BS) , currently ISO/IEC 20000, developed by the BSI Committee of Management Services, at the beginning of this millennium. IS Effectiveness Traditional Perspective After acknowledging the complex reality that involved the identification and definition of the IS effectiveness, DeLone and McLean conducted a series of studies and created a taxonomy (DeLone & McLean, 1992). The relationship between the concepts of this classification was represented by a model with six categories: system quality, information quality, use, user satisfaction, individual impact and organizational impact. The DeLone and McLean model proposes that the systems quality and the information quality independently and jointly affect the use and user s satisfaction. Additionally, use may affect the degree of user s satisfaction, and vice versa. Use and the user s satisfaction are direct precedents of the individual impact; and finally, this impact on the individual performance should lead to an organizational impact (Fig. 1) (DeLone & McLean, 1992). Fig. 1 IS Effectiveness Model (DeLone & McLean, 1992).

4 Regarding the Systems Quality and the Information Quality, these must be measured separately since these may subsequently and independently affect, or not, the Use and User Satisfaction categories. The latter are strongly linked. In procedural terms, the Use must precede the User s Satisfaction, but a positive Use experience can, in causal manner, lead to an increase in User s Satisfaction. Both categories will cause an impact on the individual who will lead in turn, an impact on the organization. For each of these categories there are several metrics that can be applied. According to DeLone and McLean, an attempt should be made to significantly reduce the number of used metrics, so that results can be compared and findings validated. IT Service Success As was said, in 1995, it was already clear that the objectives of an IS department were not only in the product development and operations management, but also in the role of service provider (Pitt et al., 1995). SOA (Service Oriented Architecture or the Targeted Services Architecture) emerged at the beginning of the twenty-first century. It has been studied due to the various changes that the information technologies experienced, driven by the need to improve the relationship between the areas that technologically support the business, and the business itself. SOA can be defined as an architecture centered on the notion that the assets of an IS at an organization are described and exposed as services (Marques, 2003). In this context, a service encapsulates reusable business functions. IS departments have always had a service role, since their duties encompass helping users to transform data into information. IT Service Quality The basis of the categorization of DeLone and McLean model is product oriented. For example, most metrics used when evaluating systems quality are related with performance issues. To assess the information quality we analyze the outcomes of a given system, such as accuracy, precision, etc. Since the systems quality and the information quality precede other metrics of IS effectiveness, all existing metrics seem to be strongly product oriented (Pitt et al., 1995). An IS department is not only a product supplier, it is also a service provider. Furthermore, this can be its most important function. The service quality of the IS department, as perceived by its users, is a IS effectiveness indicator. The main reason for measuring the users satisfaction is to improve the quality of the services provided by that department. To remedy the lack of focus on services, Pitt et al. (Pitt et al., 1995), added to the DeLone and McLean model the category Quality of Service (Fig. 2).

5 Fig. 2 Augmented IS Effectiveness Model (Pitt et al., 1995). Metrics Metrics used to determine the quality of services are based on the comparison between the customer s judgment and the real service offer. Over the years, researchers have concluded that the service quality corresponds to the discrepancy between a customer s perception and expectation for a given service (Conrath & Mignen, 1990) (Gronroos, 1982) (Parasuraman et al., 1985). Rushinek and Rushinek (Rushinek & Rushinek, 1986) concluded that user s expectations, when implemented; heavily contribute to their overall satisfaction. Zeithaml and Parasuraman (Zeithaml & Parasuraman, 1990) suggested that the service quality can be assessed by measuring the expectations and perceptions of a number of service attributes, at various levels of implementation. Thus, the actual difference between expectations and perceptions can be calculated using the average of these attributes. According to these researchers, the main expectation determinants of service quality are word-of-mouth, personal needs, past experiences, communication between user and service provider. In order to gather all these determinants, Pitt et al. (Pitt et al., 1995) conducted a model which describes the interactions among the determinants of users expectations (Fig. 3). Fig. 3 Determinants of Users Expectations (Pitt et al., 1995). A gap demonstrates the difference between the expected service and the perceived service: the discrepancy between what users expect and what they feel that are receiving. To quantify this

6 discrepancy and measure this gap, it is necessary to find a way to measure the expectation and the perception of the user. SERVQUAL SERVQUAL (Parasuraman et al., 1988) is a tool which has been refined over the years (Parasuraman et al., 1991) (Parasuraman et al., 1993). It aims to assess the expectation and the perception of service quality in an organization. It consists of two parts: The first part (Appendix A) is a 22 questions questionnaire, which measure the users expectation. The questions are made requiring that the users consider an ideal IS department, in what concerns the performance of the service delivery. The second part (Appendix B) is also a questionnaire with 22 questions on which, in this case, the objective is to measure the users perception. The issues are made taking into account the perception of the user in relation to the performance of their current service provider. The answers to the questions are classified on a scale of 1 (strongly disagree) to 7 (strongly agree). There are five common dimensions underlying all the 22 questions. According to the authors, customers use them during the evaluation of service quality, regardless of service type. These dimensions are: Tangibles: Physical facilities, equipment and appearance of personnel; Reliability: Ability to perform the promised service dependably and accurately; Responsiveness: Wilingness to help customers and provide prompt service; Assurance: Knowledge and courtesy of employees and their ability to inspire trust and confidence; and Empathy: Caring, individualized attention the service provider gives its customers. Service quality for each dimension is captured by a difference score G (representing perceived quality for that item), where G = P E and P and E are the average ratings of a dimension s corresponding perception and expectation statements respectively. ISO/IEC Business Perspective ISO/IEC is based on the BS 15000, and aims to replace it. It provides a matrix for the evaluation, specification, and adoption of an integrated approach to achieve an effective IT services management, in order to meet the business needs and the customers requirements. Like its predecessor BS 15000, ISO/IEC was originally developed to reflect a guideline of best practices contained in the structure of ITIL, despite it also supports other structures for the management of IT services. It defines the requirements for an IT services management system and adjusts the main processes for effective service delivery. Furthermore, the standard aligns itself with ITIL and specifies the following groups of key processes (Fig. 4):

7 Service Delivery Processes. Encompasses: service level management, availability management, capacity management, service continuity, budgeting and accounting for IT services. These processes are accompanied by information security management and by service reporting. Relationship Processes. There are two key processes: business relationship management and supplier management. Resolution Processes: Incident management and problem management. Control Processes: Configuration management and change management. Release Process: Release management. Fig. 4 Service management processes (BSi, 2005). ISO/IEC 20000, published for the first time in December 2005, is comprised of two parts, both related with the structure of ITIL best practices: a specification and a code of practice for the IT services management (Fig. 5). Fig. 5 Relationship between ISO/IEC and ITIL (ISO Central, 2007). It is important to recall that ITIL is not a standard. ITIL is not a criterion for audit. Rather, it is a structure of best practices, which aims to ensure that the service delivered is of quality, appropriate and efficient in terms of cost. Still, there is nothing in ITIL that assists an audit.

8 Global Management How should ISO/IEC be used during an approach to the continuous quality improvement? The use of this standard requires a balanced approach using management techniques, policies and other management instruments. As seen in ISO/IEC 20000, the specification represents the industry consensus about meeting quality standards for procedures for the IT services management processes (ITSMF, 2006). The standard can be used by service providers to: Monitor and improve service quality; Benchmark its IT service management; Serve as the basis for an independent evaluation that can lead to a formal certification; Demonstrate the ability to provide services that meet the customer requirements. Planning and implementing the service management During planning and implementation of service management one treats a number of cases, decisions, and responsibilities. The questions that arise are: what are the customer s requirements; what business processes must be supported; who will play each activity and when; what are the available financial resources and infrastructures; and when the objectives are achieved. The Deming Quality Cycle (PDCA: Plan-Do-Check-Act), can be applied to the analysis of this information and to monitor the progress in the field of IT services processes (Fig. 6). Fig. 6 PDCA methodology for service management processes, based on Deming Quality Cycle (BSi, 2005a). This model implies that to achieve an adequate level of quality, the following steps must be repeatedly performed: Plan: establish the objectives and processes needed to deliver the desired results. This step is complete with agreements that are measurable and realistic, as well as a plan on how these will be achieved; Do: implement the process;

9 Check: monitor and measure the processes and services, with reference to the policies, objectives and defined requirements; Act: identify actions to continuously improve performance. By the repeated iteration of this cycle, it is possible to ensure an incremental quality improvement. This method is known as "perpetual cycle of performance increase". Proposal The quality of a service refers to the extent to which the service fulfills the requirements and expectations of the customer (ITSMF-NL, 2006). How can we assess and ensure the quality of services and hence the IS effectiveness? As seen, there are two main approaches used to answer this question. The approach initiated by DeLone, McLean and Pitt in the 90 s and the ISO/IEC approach. The DeLone, McLean and Pitt approach is Top-Down alike. Following this approach, it is possible to assess in a short time and without high costs involved, what are the main flaws and strengths in the organization services provision. However, since this is an isolated action, there is no specific procedure or standard that should be accomplished according to the results of this analysis. This is its main disadvantage. As it was said, ISO/IEC is substantiated by ITIL. Despite this factor be largely beneficial, it is also the biggest obstacle to overcome with regard to the implementation of the standard. The use of ITIL involves a high initial investment at several levels that not all organizations can bear. Therefore, it is normal that today, the number of ISO/IEC certified companies in the world is still scarce. ISO/IEC approach is bottom-up alike. That is, before the organization services quality is verified it ensures that the processes that support it are implemented. From this point onwards, it starts a cycle of continuous improvement - PDCA. There is however, no reference to how the analysis on service quality should be performed. The quality of a service is verified depending on the extent to which the service fulfills the customer s requirements and expectations. This is the assertion that unites both approaches with regard to service quality. Taking into account the models that reflect the service quality in each of the approaches it is observed that is actually in the categories of User Satisfaction (Fig. 2), Customer requirements (Fig. 6) and Customer satisfaction (Fig. 6) that the approaches come together. It is in the organization s internal processes improvement cycle that the DeLone, McLean and Pitt model should be used, specifically during the stage of Verification (CHECK) as a way of measuring the expectation and the perception of customers and consequently, their overall satisfaction with respect to provided services. Through the cycle of continuous improvement, it is thus possible to (re)define the requirements for an IT services management system and adjust the main processes for effective service delivery.

10 Case Study One of the objectives of this work was to measure the service quality provided by a department/organization and thus measure its effectiveness. In this case, it was the IT Department of the National School of Public Health (Escola Nacional de Saúde Pública). To make the measurement according to the DeLone, McLean and Pitt approach, it was used the SERVQUAL tool. This case study is part of the proposal under performed. Nowadays, it is not yet possible to conduct a study in Portugal on the possible use of the approach of DeLone, McLean and Pitt as part of the cycle of continuous improvement of ISO/IEC This is partly because there is no company in Portugal officially certified with this standard. Table 1 presents the results obtained during the analysis conducted with SERVQUAL. In Expectation and Perception columns, average values are presented, corresponding to the answers given in each of the five dimensions. In the P - E column it is presented the mean value corresponding to the gap: the discrepancy between what users expect and what they feel they are obtaining. Table 1 Perception and Expectation Average Values, by Dimension Overall, the results are positive; we do not observe great discrepancies between the values of Perception and Expectation. Although the department has to better analyze the tangible items, it must be especially vigilant to the points in which users post their greatest expectations, that is, those related with the users confidence in the department. Conclusion The services sector has, every year, been occupying leading roles in the global economy, generating a fair number of new jobs. With the development of the sector, it is important and necessary for companies to give their clients guaranties in order to fulfil their needs. This becomes possible if the company invests on quality. This has become a very important factor for clients when deciding to take a certain product or service. The importance in measuring the given service quality, can be summarised in whether it fulfils the real client needs, trying to evaluate the level of discrepancy between the clients perceptions and expectations with relation to the service provided. A quality service is fully achieved when expectations are fully met. The effectiveness of an Information System is totally dependent on the maintenance of the service quality.

11 Through the case study, we can conclude that you can get a quality service without a regulatory infrastructure that supports it. But nothing can be concluded on maintaining and improving the levels of the quality of this service. Standards are established with a purpose and provide guarantees that justify its raison d'être. The implementation of standards can sometimes be costly, despite that we must take into consideration that the alternative may lead to situations with serious consequences in the organization. The management of the risk/safety is therefore an important factor to be taken into account with regard to maintaining the service quality. The study on the ISO/IEC approach supports the idea that to maintain an effective delivery of services, there should be defined requirements for an IT service management system and adjust the main processes. For this, it is necessary to keep alive the perpetual cycle of performance increase. By measuring the service quality and following the DeLone, McLean and Pitt approach, it is possible to contribute to this goal. As proposed, this should be used more specifically during the Check phase of the PDCA cycle as a way of monitoring. Bibliography (BSi, 2005) BSi, Information Technology Service Management Part 1: Specification, ISO/IEC, (Conrath & Mignen, 1990) Conrath, D.W., Mignen, O.P. "What is Being Done to Measure User Satisfaction with EDP/MIS", Information & Management (19:1), August 1990, pp (DeLone & McLean, 1992) DeLone, W. H., and McLean, E. R. "Information Systems Success: The Quest for the Dependent Variable" Information Systems Research (3), March 1992, pp (Gronroos, 1982) Gronroos, C., "Strategic Management and Marketing in the Service Sector", Swedish School of Economics and Business Administration, Helsingfors, Finland, (ISO Central, 2007) ISO Central, August (ITSMF, 2006) ITSMF, ISO 20000, A Pocket Guide, Van Haren Publishing, May (ITSMF-NL, 2006) ITSMF-NL Foundations of IT Service Management, based on ITIL, Van Haren Publishing, June (Marques, 2003) Marques, J. Alves, Acetatos da cadeira de Sistemas Empresariais Integrados, Instituto Superior Técnico, (Parasuraman et al., 1985) Parasuraman, A., Zeitlaml, V.A., Berry, L.L., "A Conceptual Model of Service Quality and Its Implications for Future Research", Journal of Marketing (49), Fall 1985, pp (Parasuraman et al., 1988) Parasuraman, A., Zeithaml, V.A., Berry, L.L., "SERVQUAL: A Multipleitem Scale for Measuring Consumer Perceptions of Service Quality", Journal Retailing (64:1), Spring 1988, pp (Parasuraman et al., 1991) Parasuraman, A., Berry, L.L., Zeithaml, V.A., "Refinement and Reassessment of the SERVQUAL Scale", Journal of Retailing (67:4), Winter 1991, pp (Parasuraman et al., 1993) Parasuraman, A., Berry, L.L., Zeithaml, V.A., "More on Improving the Measurement of Service Quality", Journal of Retailing (69:1), Spring 1993, pp

12 (Pitt et al., 1995) Pitt, Leyland F., Richard T. Watson, C. Bruce Kavan, Service quality: a measure of information systems effectiveness, MIS Quarterly, June 1995, v.19 n.2, pp (Rushinek & Rushinek, 1986) Rushinek, A., Rushinek, S.F., "What Makes Users Happy?", Communications of the ACM (29:7), July 1986, pp (Zeithaml & Parasuraman, 1990) Zeithaml, V., Parasuraman, A., Berry, L.L., "Delivering Quality Service: Balancing Customer Perceptions and Expectations", Free Press, New York, NY, 1990.

13 Appendix A: Questionnaire used to measure the Service Quality Expectations (Pitt et al., 1995).

14 Appendix B: Questionnaire used to measure the Service Quality Perceptions (Pitt et al., 1995).