The Indicators Crossroads Where Indicators, Logic Models, Evaluation Plans, and MIS Converge

Transcription

1 The Indicators Crossroads Where Indicators, Logic Models, Evaluation Plans, and MIS Converge Eileen Chappelle, MPH, Evaluation & Program Effectiveness Team Todd Rogers, PhD, Public Health Institute 2010 Annual Grantee Meeting September 14-16, 2010

2 Disclaimer The information presented here is for training purposes and reflects the views of the presenters. It does not necessarily represent the official position of the Centers for Disease Control and Prevention.

3 Session Learning Objectives Review of the DHDSP indicators background Describe the role of the DHDSP indicators in program logic models Convert the DHDSP indicators into SMART objectives Include indicators in evaluation plans in a meaningful way Report information related to DHDSP indicators in the MIS

4 Indicator Crossroads MIS (Objectives) MIS (Progress) Workshop 15/27 Logic Models (Outcomes) Evaluation Plans (Selected Indicators) Report Results Workshop 14 Operationalization (Defined Indicators)

5 Why are these all important? Indicators Logic Models Evaluation SMART Objectives

6 Accountability Demonstrate program worth to stakeholders Support data-driven planning and decision making Identify problem areas and program successes Support program improvement Compare patterns of performance over time and across programs Focus programs priorities Substantiate funding requests and advocate for program

7 Accountability & Results Oriented Government Objective, measurable indicators of success allow governments to be accountable. Rudy Giuliani, Former New York City Mayor

8 CDC Framework for Program Evaluation

9 Purpose of the HDSP Indicators Project Identify evidence-based outcome indicators for policy and systems changes that state HDSP programs and DHDSP can use for planning and evaluation of initiatives in priority areas relevant to heart disease and stroke prevention

10 What is an Indicator? Specific, observable, and measurable characteristics that shows progress a program is making toward achieving a specified outcome. Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way of America; 1996

11 HDSP Comprehensive Indicators Materials Evidence-based logic models and outcome indicators for each HDSP priority area In-depth information on indicators to measure progress toward outcomes Consumer Reports-type ratings to allow for flexibility in local tailoring Guidance on using indicators to integrate program and evaluation planning Participatory development process that engaged CDC and state program and evaluation staff Ladd S, Wall H, Rogers T, Fulmer E, et al. Outcome Indicators for Policy and System Change: Controlling High Cholesterol. Atlanta, GA: Centers for Disease Control and Prevention; 2009.

12 Indicators Crossroads The role of the DHDSP Indicators in program logic models

13 Logic Models Are graphical interpretations of your program. Help communicate the purpose of the program & expected outcomes. Are not static documents. valuation_guides/logic_model.htm

14 HDSP High Blood Pressure Control Logic Model Inputs Activities Outputs Box 1 Healthcare system changes: Adherence Efficiency Policies/protocols/tools Box 2 Provider changes: Awareness Average Adherence frequency to guidelines blood Box 5 Individual changes: Awareness Motivation Adherence to treatment Satisfaction Outcomes Short-term Intermediate Long-term Outcome Component 7: Reduced levels of Blood Pressure Average blood pressure levels among individuals with high blood pressure pressure levels measured among individuals with high blood pressure Degree of reduction in disparities in blood pressure levels between general and Box 3 Worksite changes: priority populations that have Policies/protocols/tools been diagnosed Environmental with changes high blood pressure Box 4 Community changes: Environmental changes Policy/legislative changes Box 6 Risk factor reduction through lifestyle and therapeutic intervention Box 7 Reduced levels of BP Box 8 Increased control of BP levels among individuals with HBP Box 9 Reduced mortality and morbidity due to heart disease and stroke Box 10 Reduced levels of disparities in heart disease and stroke Box 11 Reduced costs associated with heart disease and stroke: Healthcare Employer Societal

15 HDSP High Blood Pressure Control Logic Model Inputs Activities Outputs Box 1 Healthcare system changes: Adherence Efficiency Policies/protocols/tools Box 2 Provider changes: Awareness Adherence to guidelines Outcomes Short-term Intermediate Long-term Box 6 Risk factor reduction through lifestyle and therapeutic intervention Box 9 Reduced mortality and morbidity due to heart disease and stroke Box 5 Individual changes: Awareness Motivation Adherence to treatment Satisfaction Box 3 Worksite changes: Policies/protocols/tools Environmental changes Box 4 Community changes: Environmental changes Policy/legislative changes Box 7 Reduced levels of BP Box 8 Increased control of BP levels among individuals with HBP Box 10 Reduced levels of disparities in heart disease and stroke Box 11 Reduced costs associated with heart disease and stroke: Healthcare Employer Societal

16 Steps in HDSP Program Planning & Evaluation 1. Select an ultimate outcome that is achievable given the timeframe, resources, political climate, etc. 2. Sketch a logic model pathway leading to the ultimate outcome 3. Select relevant outcome indicators along the pathway 4. Construct project outcome objectives 5. Implement evidence-based activities that will accomplish these objectives

17 Schematic of the Chronic Care Model (CCM) Community Health System Organization of Health Care Resources and Policies Self- Management Support Delivery System Design Decision Support Clinical Information Systems Informed, Activated Patient Productive Interactions Optimal Functional and Clinical Outcomes Prepared, Proactive Practice Team

18 Example Intervention: Chronic Care Model Improving Chronic Care Illness supported by RWJ Foundation chronic care.org/index.php?p=model_elements&s=18

19 Select Box 1 (Health Care Systems) Indicators for the CCM Proportion of healthcare systems with policies to encourage a multidisciplinary team approach to enhance blood pressure control Proportion of healthcare systems with electronic medical records appropriate for treating patients with high blood pressure Proportion of healthcare systems with treatment algorithms that incorporate recommendations of current evidence-based blood pressure guidelines Number of quality improvement initiatives to increase practitioner compliance with current evidence-based blood pressure guidelines Proportion of healthcare systems with policies to encourage patient self-management of high blood pressure

20 Select Box 2 (Provider) Indicators for the CCM Indicator Name and Number Overall Quality low high Resources Needed Scientific Evidence Face Validity Utility Accepted Practice Proportion of patients with high blood pressure who receive provider-initiated recommendation and follow-up of therapeutic lifestyle modifications $$ Proportion of providers who increase monitoring and shifts in medication for patients unable to achieve blood pressure treatment goals $$$

21 Select Box 5 (Individual Changes) Indicators for the CCM Proportion of adults who have visited a healthcare provider according to clinical guidelines for treatment of high blood pressure Number of patients with missed follow-up provider appointments for treatment of high blood pressure

22 HDSP High Blood Pressure Control Logic Model Inputs Activities Outputs Box 1 Healthcare system changes: Proportion of healthcare systems with policies to encourage a multidisciplinary team approach to enhance blood pressure control Box 2 Provider changes: Proportion of providers who measure BP according to current evidence -based HBP guidelines Box 5 Individual changes: Proportion of indiv. who have visited a healthcare provider according to current evidence-based guidelines for treatment of HBP Box 3 Worksite changes: Policies/protocols/tools Environmental changes Box 4 Community changes: Environmental changes Policy/legislative changes Outcomes Short-term Intermediate Long-term Box 6 Risk factor reduction through lifestyle and therapeutic intervention Box 7 Reduced levels of BP Box 8 Increased control of BP levels among individuals with HBP Box 9 Reduced mortality and morbidity due to heart disease and stroke Box 10 Reduced levels of disparities in heart disease and stroke Box 11 Reduced costs associated with heart disease and stroke: Healthcare Employer Societal

23 HDSP High Blood Pressure Control Logic Model Inputs Activities Outputs Box 1 Healthcare system changes: Proportion of healthcare systems with policies to encourage a multidisciplinary team approach to enhance blood pressure control Box 2 Provider changes: Proportion of providers who follow current evidence-based guidelines algorithms for pharmacological therapies to treat HBP Box 5 Individual changes: Proportion of ind who have visited a healthcare provider according to current evidence-based guidelines for treatment of HBP Box 3 Worksite changes: Policies/protocols/tools Environmental changes Box 4 Community changes: Environmental changes Policy/legislative changes Outcomes Short-term Intermediate Long-term Box 6 Risk factor reduction through lifestyle and therapeutic intervention Box 7 Reduced levels of BP Box 8 Increased control of BP levels among individuals with HBP Box 9 Reduced mortality and morbidity due to heart disease and stroke Box 10 Reduced levels of disparities in heart disease and stroke Box 11 Reduced costs associated with heart disease and stroke: Healthcare Employer Societal

24 Indicators Crossroads DHDSP Indicators and SMART Objectives

25 Project Outcome Objectives Well-designed objectives indicate: What measurable change or benefit is expected Who or what is expected to change or benefit How much change or benefit is expected When the change or benefit is expected to happen Who is responsible for carrying out the activity and measuring progress How progress will be measured

26 SMART Objectives Specific Measurable Attainable/Achievable Relevant Time Bound

27 Example SMART Objective (Indicator 1.1.1) By June 2012, increase from 25% to 50% the percentage of healthcare systems in Blackwell, Brown, and Zapata counties with policies to encourage a multidisciplinary team approach to enhance high blood pressure control.

28 Strong Program Objectives Are... By June 2012, increase from 25% to 50% the percentage of healthcare systems in Blackwell, Brown, and Zapata counties with policies to encourage a multi-disciplinary team approach to enhance high blood pressure control. Specific Measurable Achievable Relevant Time-bound

29 Steps in HDSP Program Planning & Evaluation 1. Select an ultimate outcome that is achievable given the timeframe, resources, political climate, etc. 2. Sketch a logic model pathway leading to the ultimate outcome 3. Select relevant outcome indicators along the pathway 4. Construct project outcome objectives 5. Implement evidence-based activities that will accomplish these objectives 6. Design and conduct an appropriate evaluation

30 Developing Your Evaluation Who are you stakeholders? What information do you need to know about your program? How will this information be used?

31 Key Evaluation Questions What seems to be working? What needs to be changed? What have we accomplished? Are we making a difference? How do we know that we ve been successful? What s the evidence that things are working? How should we do it differently next time?

32 Incorporating Indicators into Your Evaluation Plan The indicators are evidence-based and should be incorporated into your evaluation plan. Evaluating your program is not just collecting information on the indicators.

33 Key Evaluation Questions & Indicators Key evaluation questions help focus your evaluation efforts. As you select your indicators, make sure they are addressing your key evaluation question.

34 Objective: By June 2012, increase from 25% to 50% the percentage of healthcare systems in Blackwell, Brown, and Zapata counties with policies to encourage a multi-disciplinary team approach to enhance high blood pressure control. Evaluation Questions What you want to know To what extent have clinics participating in the CCM initiative implemented multidisciplinary teams to control HBP Indicator(s) Source Method Timing What data will answer the question Where you will collect the data Policy inventory How you will collect the data Self-report from a clinic administrator or HDSP staff site visits When you will collect the data Baseline, and annually Data Analysis What type of analysis you will perform Quantitative

35 Refocusing Your Evaluation Questions Multiple perspectives and interests may lead to a spider web of evaluation questions. Stop. Reflect. Decide Will be able answer what you need to know vs. what would be nice to know?

36 Operationalization of Indicators See the presentation for Workshop 14 for more information.

37 Indicator Template Key concepts Intended population Recommended measures Data source Numerator Denominator Time period Frequency of collection Unit of analysis Level of analysis Representativeness Measurement issues Limitations Responsibility for collection & analysis

38 Data Considerations Types of Data Access to the Data Will you need data agreements? Data analysis support

39 Beyond the DHDSP Indicators Outcomes not address by the DHDSP Indicators Add to the practice-based evidence. Exploring the impact of the intervention to community referrals. Process Evaluation Are you planning to expand your pilot intervention statewide? Have you implement all components?

40 Small Group Activity & Discussion Break up into small groups See handouts (exercise and logic model) Select a scribe and someone to report back We ll reconvene as a group and review the key discussion points.

41 Steps in HDSP Program Planning & Evaluation 1. Select an ultimate outcome that is achievable given the timeframe, resources, political climate, etc. 2. Sketch a logic model pathway leading to the ultimate outcome 3. Select relevant outcome indicators along the pathway 4. Construct project outcome objectives 5. Implement evidence-based activities that will accomplish these objectives 6. Design and conduct an appropriate evaluation 7. Reflect on your experiences and share your results

42 Reflect and Report Consider your results in light of a problem solving approach Revise, Replace, Replicate Innovation is good Share your experiences MIS Papers and presentations

43 Linking Program & Evaluation Planning Logic Modeling Indicator Selection Project Outcome Objective Evaluation Question / Hypothesis Evaluation Design Measurement Methods Sampling Reporting

44 Indicators Crossroads DHDSP Indicators and MIS

45 Current Status of State HDSP Programs State HDSP programs are good at selecting indicators for logic models Increased use of parenthetical notation of indicators in MIS from to work plans Some integration in evaluation plans

46 Structure of the Work Plan in the MIS Intervention Objectives Long term Objectives Supporting Objectives Objective Title Objective Description Objective Progress Activities

47 What to include in MIS? Indicator numbers Write out the word Indicator Reach and impact Smart objective from indicators Key evaluation findings in objective progress Upload updated evaluation plans and evaluation reports.

48 Utilization of Evaluation Results Program Improvement Informed decision-making Monitoring progress Accountability

49 Summary Points The value of indicators and importance of selecting short, intermediate and long term. Converting indicators into SMART objectives for your work plans Moving on from indicator selection to collecting the DHDSP indicators Evaluation is not just collecting the DHDSP indicators Other indicators to consider beyond the DHDSP indicators. Uploading evaluation plan, tools, and results into MIS

50 My question is: Are we making an impact?

51 Questions Eileen Chappelle Todd Rogers