Evaluating Implementations of GIS

Transcription

1 Geog 469 GIS Workshop Evaluating Implementations of GIS Consider your project activity in light of these ideas

2 Outline 1. How do we evaluate a GIS implementation? 2. What is a successful GIS implementation? How do we define the success of GIS implementation? 3. What are some additional perspectives on GIS implementation?

3 1. How do we evaluate a GIS implementation? Review what is a scale fo system implementation - Single-purpose GIS project - Application GIS - Departmental GIS - Enterprise GIS - Consortium GIS - Societal GIS

4 Evaluating GIS Ad-hoc method If time is limited, evaluate system based on your insight and previous interaction with clients Benefit-cost analysis Compare benefit to cost to determine whether the system is financially justifiable (at any stage) to move forward Usability testing Test whether clients needs have been met; use prepared questionnaire, interview, observation

5 Ad-hoc What does your experience tell you? Data Software Hardware Personnel Timing of Procedures and Institutional Arrangements Motivations aka purpose and intent

6 Benefit-cost analysis Enumerate all possible costs spent on this project; classify them into one-time cost (data, hw, sw) and operating cost (maintenance, overhead, salary, training) Cost of leasing workstation for 10 weeks can be obtained from estimates Enumerate all possible benefits; classify them into tangible (cost reduction, efficiency) and intangible one (improved decision making, better public image, equity); assign monetary value wherever possible Benefit of using GIS for making map can be calculated from saved time multiplied by salary rate (= actual time spent - time spent if manual method is used) Discounting: If B & C estimates are based on different time period, discount them given inflation rate BCR: Calculate the ratio of benefit to cost to determine whether the ratio is over 1 (i.e. economically justifiable) Future: If possible, project the cost/benefit into the foreseeable future; draw the curve; determine payback period

7 Usability testing Develop test objectives What am I testing? e.g. effectiveness of map design, accuracy of data geocoded Acquire participants Sampled from potential end users Screen them through pre-survey form (ask their age, education level, gender, occupation, and so on wherever appropriate) Collect qualitative and quantitative performance and preference measurements Questionnaires Which map do you prefer? Does prepared map design work? Task scenario if applicable Describe the context for use of the information product Recommend improvements to the design of information product Read materials on usability testing at

8 Use an Evaluation to Draw conclusions about whether need-to-know questions were answered, i.e., information needs met about the effectiveness of your methods about significance of findings Provide recommendations for improving the completing work for additional / future work

9 2. What is a successful GIS implementation? How can we define the success of GIS implementation?

10 Elements of GIS project success and failure Activity Success Failure Planning Rigorous Run and gun Requirements Focused Diffused Appraisal of effort Realistic Unrealistic Staffing Dedicated High turnover Funding Adequate Inadequate Time estimates Thoughtful Rushed or prolonged Expectations Balanced Exaggerated

11 Validity of implementation success Technical validity System to be implemented works no bug, data seems clean Organizational validity Congruence between organization and a system to be implemented System is accepted and used by the organizational members Organizational effectiveness Improvement in efficiency and effectiveness Improved decision making Perceived as a process of organizational change GIS can change the way an organization works From Obermeyer and Pinto 1995

12 What are some criteria for defining success of GIS implementation? Address question based on two perspectives Content model GI System is viewed as collection of elements Process model GI System is seen as an activity composed of phases

13 GIS Implementation: Content Models Focus on critical success factors Technical aspect Competent technical support Staff s competence in GIS knowledge Managerial aspect Clearly defined goals Sufficient resource allocation Top-management support Implementation schedule Adequate communication channels Feedback capabilities

14 GIS Implementation: Process Models Focus on key phases in implementation process Initiation Planning phase of the implementation process Related to strategies Implementation Actual process (or the action) of the implementation Related to tactics Viewing GIS implementation as a two-stage generalized process helps us look at dynamic nature of GIS in the context of strategic goals of organizations

15 Strategy/tactics effectiveness matrix Effectiveness of tactics high low High acceptance, misuse High probability of implementation failure low Effectiveness of strategy High probability of implementation success Low acceptance, low use high Successful GIS should be both strategically and tactically effective From Schultz, Slevin, and Pinto (1987)

16 Success in the short-term might become failure in the medium-term to long-term Early false start does not necessarily imply that the project is destined to fail Bear in mind the dynamic nature of the environment in which GIS is embedded when making assessments

17 3. What are some additional perspectives on GIS implementation?

18 The implementation of GIS is not merely a technical process, nor one that simply requires the formulation of rational management strategies. It is as much a social and political process which takes place within unique organizational cultures subject to changing conditions. Adapted from Campbell and Masser 1995

19 How do organizations interact with technology such as GIS? Depends on how you see organizations, for example, as Machine, system, or culture Three theoretical perspectives on organization activity Technological determinism Widespread adoption and implementation of GIS is dependent upon the quality of the technology Managerial rationalism GIS implementation success is based on good management strategy that produces results (e.g. corporate approach) Social interactionism Good understanding of the social organization (social attitudes or institutional developments) leads to GIS implementation success

20 Technological determinism Characteristic Propensity for adoption Reason for adoption Style of implementation Constraints on implementation Likely outcome of implementation Perception of technology Perception of organizations Technological determinism Inevitable if a good technology To solve an operational problem that has been identified Technical process Technical worth of the innovation and expertise of users Positive, greater efficiency and more rational decision-making Machine and methods Machine, organism From Campbell & Masser, GIS and Organizations 1995

21 Managerial rationalism Characteristic Propensity for adoption Reason for adoption Style of implementation Constraints on implementation Likely outcome of implementation Perception of technology Perception of organizations Managerial rationalism Inevitable if a good technology and rational management To solve an operational problem that has been identified Guided by a rational management strategy Poor management and the technical worth of the innovation Positive, greater efficiency and/or more rational decision-making Machine and methods System

22 Social interactionism Characteristic Propensity for adoption Reason for adoption Style of implementation Constraints on implementation Likely outcome of implementation Perception of technology Perception of organizations Social interactionism Uncertain, depends on trends in society To enhance symbolic status or power An organizational process that is problematic and uncertain Social and political processes Uncertain, at best a mixture of positive and negative results Machine, methods and knowledge Cultures

23 Case studies suggest A significant number of GIS implementations failed when User needs are not well-defined GIS design is not well matched to specific tasks Ownership and control of GI is not well determined Not clear how information is to be used in decisionmaking process Information sharing is not well-coordinated Organizational members are not ready to adopt GIS or resistant to using GIS

24 Overview of critical issues to successful GIS implementation Identification of applications producing information relevant to users is fundamental for success Awareness of organizational limitations in terms of accepted practices and available resources is crucial to success User-directed implementation set within a framework based on the commitment and real participation of staff throughout the relevant part of organization is crucial Ability to create stability or cope with change fosters success through adaptive management continual monitoring is critical to successful implementation. Campbell & Masser 1995