Program Evaluation and Performance Measurement An Introduction to Practice

Transcription

1 SECOND EDITION Program Evaluation and Performance Measurement An Introduction to Practice JAMES C. McDAVID University of Victoria, Canada IRENE HUSE LAURA R.L. HAWTHORN (DSAGE Los Angeles London New Delhi Singapore Washington DC

2 DETAILED CONTENTS Preface Acknowledgments xxi xxiii 1. Key Concepts and Issues in Program Evaluation and Performance Measurement 1 Introduction 3 Integrating Program Evaluation and Performance Measurement 4 Connecting Evaluation and Performance Management, 5 The Performance Management Cycle 6 What Are Programs and Policies? 9 What Is a Policy? h, 9 ' What Is a Program?. ; 10 The Practice of Program Evaluation: The Art and Craft of Fitting Round Pegs into Square Holes 10 A Typical Program Evaluation: Assessing the Neighbourhood Integrated Service Team Program 13 Implementation Concerns 13 The Evaluation 14 Connecting the NIST Evaluation to This Book 15 Key Concepts in Program Evaluation 16 Ten Key Evaluation Questions 18 Formative and Summative Evaluations 23 Ex Ante and Ex Post Evaluations 24 Causality in Program Evaluations 25 The Steps in Conducting a Program Evaluation 26 General Steps in Conducting a Program Evaluation 27 Steps in the Program Evaluation Process 27 Assessing the Feasibility of the Evaluation ' 28 Doing the Evaluation 35 Making Changes Based on the Evaluation 38 Summary 39 Discussion Questions 40 References Understanding and Applying Program Logic Models 45 Introduction 47 A Basic Logic Modeling Approach 50

3 Logic Models That Categorize and Specify Intended Causal Linkages 52 Constructing Program Logics for Program Evaluations 61 Specifying Program Objectives 61 Program Technologies: Working With Uncertainty 64 Program Theories and Program Logics: Evaluating Program Appropriateness 67 Logic Models as Embedded Theories of Change 71 Connecting Program Theories, Program Logic Models, and Program Mechanisms 72 Putting Program Theory Into Perspective: Theory-Driven Evaluations and Evaluation Practice 73 Program Objectives, Program Environments, and Organizational Objectives 74 Normative and Behavioral Goals in Organizations 75 Logic Models for Performance Measurement 75 Strengths and Limitations of Program Logics 78 Summary, 79 Discussion Questions 80 Appendixes 82 Appendix A: Applying What You Have Learned: Development of a Logic Model for a Meals on Wheels Program 82 Appendix B: A Complex Logic Model Describing Primary Health Care in Canada 82 References 86 Research Designs for Program Evaluations 89 Introduction 91 What Is Research Design? 95 The Origins of Experimental Design 96 Why Pay Attention to Experimental Designs? 101 Using Experimental Designs to Evaluate Programs: The Perry Preschool Study 102 Limitations of the Perry Preschool Study 105 The Perry Preschool Study in Perspective 106 Defining and Working With the Four Kinds of Validity in Research Designs for Evaluations 108 Statistical Conclusions Validity. 109 Internal Validity 110 Construct Validity 112 External Validity 116 Quasi-Experimental Designs: Navigating Threats to Internal Validity 117 The York Neighborhood Watch Program: An Example of an Interrupted Time-Series Research Design 121 Findings and Conclusions From the Neighborhood Watch Evaluation 123 Estimating the Incremental Impact of a Policy Change: Implementing an Admission Fee in the Royal British Columbia Museum 126

4 Testing the Causal Linkages in Program Logic Models 129 Research Designs and Performance Measurement 133 Summary 135 Discussion Questions v 136 Appendixes 138 Appendix A: Statistical Tools for Program Evaluation 138 Appendix B: Empirical Causal Model for the Perry Preschool Study 140 References Measurement for Program Evaluation and Performance Monitoring 145 Introduction 147 Introducing Reliability and Validity of Measures 149 Understanding the Reliability of Measures 151 Understanding Measurement Validity 153 Types of Measurement Validity 154 Ways to Assess Measurement Validity 155 Validity Types That Relate a Single Measure to a Corresponding Construct 155 Validity Types That Relate Multiple Measures to One Construct: Internal Structure Validity Validity Types That Relate Multiple Measures to Multiple Constructs 157 Units of Analysis and Levels,of Measurement 159 Nominal Level of Measurement 160 Ordinal Level of Measurement. 161 Interval and Ratio Levels of Measurement 161 Sources of Data in Program Evaluations and Performance Measurement Systems 163 Existing Sources of Data 163 Sources of Data Collected by the Program Evaluator 164 Surveys as a Data Source in Evaluations 166 Working With LJkert Statements in Surveys 168 Designing and Conducting Surveys 171 Specific Survey Design Considerations 172 Using Surveys to Estimate the Incremental Effects of Programs 174 Retrospective Pretests: Where Measurement Intersects With Research Design r 177 Survey Designs Are Not Research Designs 179 Validity of Measures and the Validity of Causes and Effects 180 Summary 181 Discussion Questions 182 References Applying Qualitative Evaluation Methods 187 Michelle Brady, James C. McDavid, Irene Huse, & Laura R. L. Hawthorn Introduction 189 Comparing and Contrasting Different Approaches to Qualitative Evaluation 189

5 . Understanding Paradigms and Their Relevance to Evaluation 190 Pragmatism as a Response to the Philosophical Divisions Among Evaluators 195 Alternative Criteria for Assessing Qualitative Research and Evaluations 196 Qualitative Evaluation Designs: Some Basics 198 Appropriate Applications for Qualitative Evaluation Approaches 198 Comparing and Contrasting Qualitative and Quantitative Evaluation Approaches 200 Designing and Conducting Qualitative Program Evaluations 202 Clarifying the Evaluation Purpose and Questions 203 Identifying Research Designs and Appropriate Comparisons 204 Mixed-Methods Evaluation Designs 205 Identifying Appropriate Samples in Qualitative Evaluations 208 Collecting and Coding Qualitative Data 210 Structuring Data Collection Instruments 210 Conducting Qualitative Interviews 211 Analyzing Qualitative Data 212 Assessing the Credibility and Generalizability of Qualitative Findings 215 Connecting Qualitative Evaluation Methods to Performance Measurement 217 The Power of Case Studies 219 Summary n Discussion Questions 221 References Needs Assessments for Program Development and Adjustment 225 Introduction 227 General Considerations.Regarding Needs Assessments 227 What Are Needs and Why Do We Conduct Needs Assessments? 227 Group-Level Focus for Needs Assessments 228 How Needs Assessments Fit Into the Performance Management Cycle 229 Recent Trends and Developments in Needs Assessments 230 Perspectives on Needs 231 A Note on the Politics of Needs Assessment 233 Steps in Conducting Needs Assessments 234 Phase I: Preassessment ' 236 Focusing the Needs Assessment 237 Forming the NAC or Study Team 242 Learning as Much as We Can About Preliminary "What Should Be" and "What Is" Conditions From Available Sources 242 Moving to Phase II and/or III or Stopping 243 Phase II: The Needs Assessment 244 Conducting a Full Assessment About "What Should Be" and "What Is" 244 Prioritizing the Needs to Be Addressed 253 Preparing a Document That Integrates Evidence and Recommendations 254

6 Phase III: Postassessment: Communicating and Implementing a Needs Assessment Action 257 Making Postassessment Decisions and Developing Action Plans 257 Communicating the Final Results of the Needs Assessment 257 Implementing and Monitoring 258 Needs Assessment Example: The Ottawa Share Program 258 Preassessment 259 Assessment 260 Summary 262 Discussion Questions 263 Appendixes 264 Appendix A: Examples of Resources for Those Doing Needs Assessments 264 Appendix B: Case Study: Designing a Needs Assessment for a Small Nonprofit Organization 265 The Program 265 Your Role ' 266 Your Task 266 References Concepts and Issues in Economic Evaluation 271 Lynda Gagne, Irene Huse, James C. McDavid, & Laura R. L Hawthorn Introduction 273 Why a Program Evaluator Needs to Know About Economic Evaluation 274 Connecting Economic Evaluation With Program Evaluation: Program Technologies and Outcome Attribution 275 High- and Low-Probability Program Technologies 275 The Attribution Issue 276 Three Types of Economic Evaluation 276 The Choice of Economic Evaluation Method 277 Economic Evaluation in the Performance Management Cycle 278 Historical Developments in Economic Evaluation 279 Distinguishing Budgetary Costs and Benefits From Social Costs and Benefits ' 280 Cost-Benefit Analysis 281 Standing ' 282 Valuing Nonmarket Impacts 284 Total Economic Value 285 Revealed and Stated Preferences Methods for Valuing Nonmarket Impacts 286 Steps for Economic Evaluations 287 Specify the Set of Alternatives 287 Decide Whose Benefits and Costs Count (Standing) 287 Categorize and Catalog the Costs and Benefits 288 Predict Costs and Benefits Quantitatively Over the Life of the Project 289 Monetize (Attach Dollar Values to) All Costs and Benefits 289

7 Select a Discount Rate for Costs and Benefits Occurring in the Future 289 Compare Costs With Outcomes, or Compute the Net Present Value of Each Alternative 291 Perform Sensitivity and Distributional Analysis 292 Make a Recommendation 292 Cost-Effectiveness Analysis 293 Cost-Utility Analysis 294 Cost-Benefit Analysis Example: A Study of the High/Scope Perry Preschool Program Specify the Set of Alternatives Decide Whose Benefits and Costs Count (Standing) Categorize and Catalog Costs and Benefits Predict Costs and Benefits Quantitatively Over the Life of the Project Monetize (Attach Dollar Values to) All Costs and Benefits Select a Discount Rate for Costs and Benefits Occurring in the Future Compute the Net Present Value of the Program Perform Sensitivity and Distributional Analysis Make a Recommendation 300 ( Strengths and limitations of Economic Evaluation 301 Strengths of Economic Evaluation 301 Limitations of Economic Evaluation 302 Summary 304 Discussion Questions 305 References Performance Measurement as an Approach to Evaluation 309 Introduction 310 Growth of Performance Measurement 311 Government Deficits and the Transformation of Public Expectations for Governments 313 Performance Measurement Purposes 317 Metaphors That Support and Sustain Performance Measurement 318 Organizations as Machines 319 Government as a Business 319 Organizations as Open Systems 320 A Note on Complexity 321 Comparing Program Evaluation and Performance Measurement 322 Summary 331 Discussion Questions 332 References Design and Implementation of Performance Measurement Systems 337 Introduction 339 Key Steps in Designing and Implementing a Performance Measurement System 341 Identify the Organizational Champions of This Change 342

8 Understand What Performance Measurement Systems Can and Cannot Do 343 Establish Multichannel Ways of Communicating That Facilitate Top-Down, Bottom-Up, and Horizontal Sharing of Information, Problem Identification, and Problem Solving 345 Clarify the Expectations for the Intended Uses of the Performance Information That Is Created 346 Identify the Resources Available for Designing, Implementing, and Maintaining the Performance Measurement System 348 Take the Time to Understand the Organizational History Around Similar Initiatives 349 Develop Logic Models for the Programs for Which Performance Measures Are Being Developed, and Identify the Key Constructs to Be Measured 350 Identify Any Constructs That Apply Beyond Single Programs 352 Involve Prospective Users in Reviewing Logic Models and Constructs in the Proposed Performance Measurement System 354 Measure the Constructs That Have Been Identified as Parts of the Performance Measurement System 355 Record, Analyze, Interpret, and Report the Performance Data 359 Regularly Review Feedback From the Users and, If Needed, Make Changes to the Performance Measurement System 362 Performance Measurement for Public Accountability 364 Summary 365 Discussion Questions ' 366 Appendix A: Organizational Logic Models 368 References Using Performance Measurement for Accountability and Performance Improvement 375 Introduction 377 Using Performance Results 379 Legislator Expected Versus Actual Uses of Performance Reports in British Columbia, Canada 379 High-Stakes Uses of Performance Measures 383 The British Experience With Performance Management 384 Assessing the "Naming and Shaming" Approach to Performance Management in Britain 386 A Case Study of Gaming: Distorting the Output of a Coal Mine 389 Reassessing the Performance Management Cycle: The Roles of Incentives and Organizational Politics 390 Use of Performance Measures in a Non-Adversarial Political Environment 393 Joining Internal and External Uses of Performance Information: The Lethbridge Local Government Study 393 Using Performance Information for Management: Encouraging Internal Uses of Performance Results 396

9 Increasing Uses of Performance Information by Elected Officials: Supply and Demand Improvements 398 Improving the Supply and Demand of Performance Information: Examining the Audit Strategy 399 Improving the Demand for Performance Information: Examining Legislation and Training 400 Assessing the Realities of Performance Measurement for Public Accountability, Performance Improvement, and Program Evaluation 401 Three Additional Considerations in Implementing and Sustaining Performance Measurement Systems 402 The Centralizing Influence of Performance Measurement in Public Organizations 403 Attributing Outcomes to Programs 403 The Levels of Analysis Problem: Conflating Organizational, Program, and Individual Performance 405 Summary 405 Discussion Questions ' 407 References Program Evaluation and Program Management: Joining Theory and Practice 411 Introduction 412 Can Management and Evaluation Be Joined? An Overview of the Issues 413 Evaluators and Managers as Partners in Evaluation 415 Building an Evaluative Culture in Organizations: An Expanded Role for Evaluators 417 Creating Ongoing Streams of Evaluative Knowledge 418 Obstacles to Building and Sustaining an Evaluative Culture 419 Manager Involvement in Evaluations: Limits and Opportunities 420 Intended Evaluation Uses and Managerial Involvement 420 Evaluating for Accountability 421 Evaluating for Program Improvement 423 Manager Bias in Evaluations: Limits to Manager Involvement 423 Striving for Objectivity in Program Evaluations 425 Can Program Evaluators Be Objective? 426 Looking for a Defensible Definition of Objectivity 427 A Natural Science Definition of Objectivity 428 Implications for Evaluation Practice 429 Criteria for High-Quality Evaluations: The Varying Views of Evaluation Associations 430 Summary 433 Discussion Questions 434 References The Nature and Practice of Professional Judgment in Program Evaluation 439 Introduction 441 The Nature of the Evaluation Enterprise 441

10 What Is Good Evaluation Practice? 442 Methodological Considerations 442 Problems With Experimentation as a Criterion for Good Methodologies 443 The Importance of Causality: The Core of the Evaluation Enterprise 444 Alternative Perspectives on the Evaluation Enterprise 444 Reconciling Evaluation Theory With the Diversity of Practice 445 Working in the Swamp: The Real World of Evaluation Practice 446 Common Ground Between Program Evaluators and Program Managers 448 Situating Professional Judgment in Program Evaluation Practice 448 Acquiring Knowledge and Skills for Evaluation Practice 450 Professional Knowledge as Applied Theory 450 Professional Knowledge as Practical Know-How 451 Balancing Theoretical and Practical Knowledge in Professional Practice 451 Understanding Professional Judgment 452 A Modeling of the Professional Judgment Process 454 The Decision Environment 456 Values, Beliefs, and Expectations 456 Acquiring Professional Knowledge 457 Improving Professional Judgment in Evaluation Through Reflective Practice 458 Guidelines for the Practitioner 458 The Range of Professional Judgment Skills 461 Ways of Improving Sound Professional Judgment Through Education and Training-Related Activities 461 Teamwork and Professional Judgment 463 Evaluation as a Craft: Implications for Learning to Become an Evaluation Practitioner 464 Ethics for Evaluation Practice 465 The Development of Ethics for Evaluation Practice 465 Ethical Evaluation Practice 467 Cultural Competence in Evaluation Practice 468 The Prospects for an Evaluation Profession 469 Summary 470 Discussion Questions 471 Appendix, 473 Appendix A: Fiona's Choice: An Ethical Dilemma for a Program Evaluator 473 Your Task References 474 Glossary 479 Index 499 About the Authors 533