National Tuberculosis Indicators Project (NTIP): An Initiative for Collaborative Impact Evaluation Kai H. Young, MPH, CHES Field Services and Evaluation Branch Division of Tuberculosis Elimination NCHHSTP/CCID/CDC National TB Conference June 2009 Disclaimer: The findings and conclusions in this presentation have not been formally disseminated by CDC and should not be construed to represent any agency determination or policy. Overview Background NTIP development 2010 TB Cooperative Agreement requirements How NTIP will support these initiatives Collaborative impact evaluation NTIP: Future plans What is NTIP? A performance monitoring system Indicator reports - inform progress toward national objectives - focus program evaluation efforts - provide performance targets as benchmark for assessment 1
Goals An user-friendly system that enhances collaboration Increase the use of existing data Prioritize efforts for program evaluation Integrate monitoring and evaluation into routine program practice 14,15 7,9 4,5,10,12 3 1 8 6 2 11 13 2
CDC Framework for Program Evaluation Steps Engage stakeholders Ensure use and share lessons learned Justify conclusions Standards Utility Feasibility Propriety Accuracy Describe the program Focus the Evaluation design Gather credible evidence The Stakeholders Stephen Hughes New York State Department of Health Deborah Sodt and Wendy Sutherland Minnesota Department of Health Barbara Stone Colorado Department of Public Health and Environment Katie Garman and Erin Holt Tennessee Department of Health Standardizing Performance Measures How do we measure success? Are the measures valid and reliable? Are these measures fair? Reasonable? Applicable to all programs? 3
Standardizing Performance Measurement (cont) Follow a simultaneous iterative process - DTBE and states - State and state - Between DTBE branches Intensive review with larger external partners Designing Indicator Reports National TB Program Objectives California TB Indicators Project Templates National TB Indicators Project (NTIP) Designing Indicator Reports Is the report clear and useful? How will the users interpret the results? How will the stratifications inform progress? What are the implications for practice? 4
National objective State performance graph State data Method Sample State Performance Trend Graph Eligible Patients Who Completed Treatment Within 12 Months, Minnesota, 2000-2003 Percent 100.0 80.0 60.0 40.0 20.0 0.0 Year 2000 2001 2002 2003 -- 2015 Completed Treatment 85.8 84.4 85.3 89.5 -- Within 12 Months (%) State Targets (%) -- -- -- 82.0 -- 90.0 National Results (%) 80.8 81.0 80.9 84.3 -- National Targets (%) -- -- -- 88.0 93.0 Treatment Completion for Eligible Patients, by Treatment Outcomes, Minnesota, 2003 N=200 179; 89.5% 14; 7.0% 6; 3.0% 1; 0.5% 0; 0.0% 0; 0.0% Completed Within 12 Months Completed After 12 Months Moved Lost to Follow-Up Refused Unknown/Other 5
Results Developing a system that Utilizes existing reportable data Standardizes measurements for tracking progress toward objectives Will be used at the national, state, and local levels Will guide program evaluation efforts Will reinforce the national priorities for TB programs Summary: NTIP Development NTIP development process is Establishing mutual understanding of and agreement on the national objectives - Developing a common language - Establishing performance targets - Standardizing performance measurements Strengthening collaboration between DTBE and partners - Enhancing the effectiveness of assistance provided by DTBE Providing Service to Partners State TB Programs Case data CDC Reporting Progress NTIP Report 6
Interpreting Results To facilitate discussion, education, and problem solving - DTBE and program areas - Program managers and staff - Programs and community partners To track program progress Enhance ability to provide guidance and technical assistance Provides a way to collaboratively detect and understand barriers, and improve program effectiveness Integrating NTIP into Program Practice Monitor progress toward objective Met target Did not meet target Best practices Implement improvements Provide insights to barriers develop plan for evaluation (if needed) Refine/develop program activities based on findings (CoAg application) Provide evaluation updates Complete evaluation How is this different from before? Moving toward Standardization Enhancing ability to assess impact Building consensus on performance measures 7
NTIP Implementation Formal Implementation in 2010 Launched in March 2009 with Feb. 6 data All CoAg recipients have received access Option to pilot NTIP for 2008 progress reporting New update with final data set (May 15) in June Use for 2010 program planning/coag application How do we make a greater impact? Prioritize and focus efforts TB Cooperative Agreement, 2010 Monitor progress using NTIP Report performance toward national TB program objectives Identify and prioritize evaluation areas Other program evaluation activities: Collaborative evaluation with local jurisdictions Cohort review 8
Collaborative Evaluation with Local Jurisdiction State programs to provide leadership and technical assistance to local jurisdiction in evaluation NTIP reports will be provided at regional and county level Cohort Review (CR) Challenges: Results from NTCA survey Preparation is time consuming Lack of time Analyzing and summarizing data Presenting data in meaningful way Identifying cases for presentation Case review and cohort review repetitive process Cohort Review + NTIP Complimentary processes CR: review of individual case > ensure each case is given appropriate standard of care (aka. program objectives) NTIP: review of indicators > identify cases not meeting standards 9
Benefits of Integrating NTIP with Cohort Review A forum for discussion and problem solving Engage stakeholders step of evaluation Provide a way to identify cases for presentation Acknowledge outstanding case management Review challenging cases Data summary Communicate common goals Inform practice Understand how individual cases contribute to larger mission Project: Integrating NTIP and Program Evaluation in CR 2 components - CR evaluation (current process) - Define the intended effects of CR - Establish baselines - Inform guidance development on CR - NTIP and CR - To find a working process - Pilot in Seattle, WA and other areas Future NTIP Development Providing county level report, quarterly Line-listed data Real time data 10
NTIP Breakout Session Thursday, June 18, 2009 Progress reporting Idaho - Ellen Zager Collaborative evaluation with local jurisdictions California - Melissa Ehman New York State - Steve Hughes Cohort review Washington - Kim Field Readdressing the National TB Program Priorities National State Local 2009 2010 Implementation Timeline Pilot CoAg progress reporting Identify priorities for program and evaluation planning for CoAg application TB PEN (Workshop and Training) Reporting progress using NTIP County level reports available to support State-local level collaborations Real time data reporting 11
Conclusion NTIP will Reinforce the national priorities Measure progress and impact using existing data Prioritize efforts for program improvement, reporting and technical assistance Facilitate evidence based practices Enhance collaboration among partners at all levels Acknowledgements Sheanne Allen Subroto Banerji Jose Becerra James L Beebe Janice Boutotte Melissa Ehman Kim Field Katie Garman Dorthy Gibson Phil Griffin Cathy Goldsborough Christopher Hayden Marge Higgins Carol Hoff Erin Holt Stephen Hughes Kashef Ijaz John Jereb Billie Juni Awal Khan Beth Kingdon Linda Leary Phil LoBue Beverly Metchock Heather Morrow-Almeida Massa Narita Tom Navin Eyal Oberlin Robert Pratt Valerie Robison Maria Rodriguez Tom Shinnick Teresa Smith Deborah Sodt Barbara Stone Wendy Sutherland Zach Taylor Wanda Walton Maureen Wilce NTIP Breakout Session Thursday, June 18, 2009 Progress reporting Idaho - Ellen Zager Collaborative evaluation with local jurisdictions California - Melissa Ehman New York State - Steve Hughes Cohort review Washington - Kim Field 12