LCAP Evaluation workshop. Snigdha Mukherjee, PhD. Louisiana Public Health Institute October 28, 2010

Transcription

1 LCAP Evaluation workshop Snigdha Mukherjee, PhD. Louisiana Public Health Institute October 28, 2010

2 Introductions

3 Agenda for the day Part I: Overview of Program Evaluation Part II: Types of Evaluation Part III: Develop an Evaluation Plan Part IV Data Collection Closing and feedback

4 Part I: Overview of Program Evaluation

5 What is Evaluation? The systematic collection of information about the activities, characteristics, and outcomes of a program to assess its strengths and weaknesses, improve program effectiveness and/or inform decisions about future programming. - Michael Quinn Patton In essence, to evaluate anything means to assess the merit or worth of something again criteria or standards

6 Monitoring & Evaluation Monitoring: What are we doing? Tracking inputs and outputs to assess whether programs are performing according to plans (e.g., people trained, condoms distributed) Evaluation: What have we achieved? Assessment of impact of the program on behavior or health outcome (e.g., condom use at last risky sex, HIV prevalence) Surveillance: monitoring disease Spread of HIV/STD (e.g., HIV prevalence among pregnant women)

7 Why are evaluations done? Understand and improve programs Tell a program s story Be accountable Inform the field Support fundraising efforts

8 Program Evaluation A typical program evaluation consists of the following activities: Posing questions about the program Setting standards of effectiveness Designing the evaluation and selecting participants Collecting data Analyzing data, and Reporting the results

9 The logic of evaluation Establishing criteria of merit Constructing standards Measuring performance Synthesizing and integrating data into a judgment of merit Rossi, Freeman & Lipsey, 2004 pp. 70

10 Stake holders & Audiences Stakeholders (those affected by the program) Consumers Service providers Families Community members Clinics Audiences (sees and/or uses evaluation results) Organization Directors State Department of health Funders Administrators Potential advocates Tax payers

11 Stakes or importance of a program Program cost programs that are expensive need to be proven effective, if not improved or abandoned Importance of outcomes some programs have serious implications for failure E.g. participants in CPR courses are tested for proficiency because outcomes may mean the difference between life & death Perceived importance of program/outcomes by stakeholders and audiences some programs are evaluated due to a request from stakeholders or audiences

12 Evaluation context What are the time & resource constraints? Are there any hidden or not so hidden political agendas? Has the program had the opportunity to be effective? Will the evaluation results be challenged? What is the historical context of the program? How would the context affect information collected?

13 Part II: Types of Evaluation

14 Designing the evaluation Consider the type of evaluation that fits your needs Types of evaluation Develop an evaluation calendar

15 Types of evaluation Formative Outcome Process

16 The Activities Types of Evaluation Formative evaluates a program and its process during development or formation Process evaluates the process fidelity, implementation of the program compared to the design The Effect Outcomes evaluates effectiveness in terms of programmatic outcomes Impact evaluates effect on community and oither institutes

17 HIV/AIDS Monitoring &Evaluation MONITORING Process Evaluation Inputs Pipeline Outputs EVALUATION Effectiveness Evaluation Number of Projects All Most Resources Staff Funds Materials Facilities Supplies Training Condom availability Trained staff Quality of services (e.g. STI, VCT, care) Knowledge of HIV transmission Outcomes Some Short-term and intermediate effects Behavior change Attitude change Changes in STI trends Increase in social support Impact Few Long-term effects Changes in : HIV/AIDS trends AIDS-related mortality Social norms Coping capacity in community Economic impact Levels of Evaluation Efforts

18 Formative Evaluation What, Why, When Helps to identify or clarify the needs the new program is meant to address Helps identify gaps in service Tests initial reaction to the program design Used to pre-test a design before full implementation

19 Formative evaluation questions Sample questions: - What is the most efficient way to recruit participants? - What types of program activities are desired? - What are the preferences of consumers? - What is the best way to deliver the program?

20 Process evaluation Used for program monitoring Assessment og how resources are used Answers questions about adherence to design of the program Can help identify early on problems in recruitment, program delivery, and staff issues

21 Process (monitoring) Evaluation Answers the questions -What, Why, When Looks at what activities, services or intervention is being implemented Accountability determine alignment with program s final design or purpose; for monitoring Program improvement mid-course corrections, changes in outreach, recruitment, or data collection Replication clarify the ingredients before replicating or taking to scale

22 Process evaluation questions Develop evaluation questions on: Reach, coverage - target population (characteristics, proportions served, outreach efforts) Dose, Duration services or intervention (what services, how often, by who, cost) Context other factors that influence how program gets implemented (neighborhood, additional services) Fidelity how well program adheres to plan Sample questions -Who is the intended target population of the program? - What elements of the program have actually been implemented - What barriers did clients experience in accessing the services? - Was the program delivered as planned? - What was the average cost per person who received the program?

23 Process Evaluation data Collect credible (quantifiable) evidence Client demographics age, race, gender, socioeconomic status Client s prior status or behavior previous alcohol abuse, exercise, condom usage, Client outreach method of contact, mode of transportation Staff demographics, training, turnover rate Program intervention number of training sessions, # condoms distributed, frequency and attendance of services

24 Outcome evaluation What, Why, When Measures the effect on clients, a population, or the community changes in knowledge, attitude or behavior Improve the service delivery of the program by focusing on key tasks Identify effective practices with in the program Usually conducted after the program has been implemented for enough time to anticipate results Used for decision-making (e.g., continue funding, increase funding, modify and try again, discontinue funding

25 Outcome evaluation Questions - Are participants more knowledgeable about the subject after their training? - Has there been a change in behavior (increase in condom use, decrease in risk behaviors) since the intervention began? - To what extent has the project met its objectives? - How effective has the project been at producing changes? - Are there any factors outside of the project that have contributed to (or prevented) the desired change? - Has the project resulted in any unintended change?

26 Impact Evaluation What, Why, When Measures the effect on clients, a population, or the community Changes in knowledge attitude or behavior or condition Very similar to outcomes evaluation Sample questions - What is the effect of the program on the long term condition of a group or population? - What is the collective effect of similar programs? - How have these programs affected the system of services related to this need?

27 A typical example of the time-scale for a two-year evaluation

28 Part III: Develop an Evaluation Plan

29 Stages in a program evaluation Stages in a program evaluation Stage I Planning the evaluation Project Manager, stakeholders and/or organization Stage II Carrying out the evaluation Evaluation team Stage III Following-up the evaluation Organization, Funders, stakeholders

30 PLANNING THE EVALUATION Financial resources Thinking through the focus of the evaluation Finding a suitable evaluation team & checking their suitability with stakeholders Organizing the logistics training, hiring additional staff etc. Briefing the evaluation team CARRYING OUT THE EVALUATION The evaluation teams take over and the gets the job done, but Project Manager keeps in touch Receiving the draft report USING THE EVALUATION Getting the management response Decisions and action based on report Publishing the report

31 Why develop an evaluation plan? It provides a road-map Guides evaluation activities Explains what, when, how, why, who Focuses the evaluation Documents the evaluation process for all stakeholders Ensures implementation fidelity

32 Developing of an Evaluation Plan

33 Step 1 Engage Stakeholders Who needs to be at the planning table? Who will use the evaluation information? What information do they want/need? What are their interests? How will you get these people to the planning table?

34 Step 2 Know your program well Background and description of program What problem does your program address? What are the causes and consequences of the problem? What is the magnitude of the problem? What changes or trends are occurring that affect the problem? Context What are the political, social or other factors that affect your program

35 Step 3 Define your intervention What is your strategy? What is your goal? A broad statement that describes what you hope to achieve through your efforts What are your objectives? Specific, measurable steps that can be taken to meet the goal

36 Step 4 Focus Your Evaluation Define key questions for your evaluation. What do you want to know? What do other stakeholders want to know? What is the purpose of your evaluation?

37 Step 4 Focus Your Evaluation What types of evaluation data do you need? Formative What do I need to get started? Process What information/activity do I want to track or monitor? Outcome What will happen as a result of the service or intervention?

38 Step 4 Focus Your Evaluation Define your outcomes: What changes do you expect to see? Short-term Change in knowledge, attitudes, beliefs Intermediate Change in behaviors Long-term Change in conditions, status

39 Step 4 Focus Your Evaluation What are your indicators? Observable evidence of accomplishments, changes made, or progress achieved Measurements that help you answer whether or not you have been successful in achieving your outcomes

40 Step 4 Focus Your Evaluation Things to consider when selecting Methods: 1) Indicators what methods will best measure your outcomes? 2) Target population - what methods are most appropriate/feasible? 3) Resources How much money do we have for data collection?

41 Targets and Benchmarks Target A numerical objective for a program s level of achievement on an indicator a projection Benchmark Performance data used for comparison past agency data or industry standard

42 Targets and Benchmark example Outcome Indicator Target Benchmark Initial Parents learn what their children are capable of doing # number of parents that demonstrate increased knowledge of child development through pre-post test on 5 key issues after attending workshops 200 Generally we see parents each year. We belive that 90% of parents will show some improvement on the pre-post test. Last year 95% of parents showed increased knowledge

43 Family Health International - FHI

44

45

46 Goals & Objectives Goals & objectives provide us the basis to judge whether a program is a success or failure Goals: ultimate results of the project, sometimes unreachable in short term Goal represents a general, big-picture statement of desired results. Usually written in broad-based statements Objectives: measurable, time-specific results that you expect to accomplish Based on realistic expectations and more narrowly defined than goals

47 SMART Goals and Objectives S Specific M Measurable A Attainable or Achievable R Relevant T Time-bound

48 Goals When writing goals answer the following: What impact does the problem have? Does it have national, regional and/or local significance? What are the benefits to the community if the problem is solved? What will happen if the problem is not solved?

49 Objectives When writing objectives, address the following: State in quantifiable terms Describe the outcomes Include only one result per objective Clearly identify the group benefiting from the planned project Include statements that are realistic and capable of being accomplished Do not describe methods

50 Objectives Describe specific outcomes that, if achieved enable one to see that the objective has been achieved A statement of the specific behaviors and accomplishments Specification of the success criteria for these behaviors

51 Writing useful objectives Four techniques helpful for writing useful objectives Using strong verbs: describe an observable or measurable behavior eg. To increase the use of educational materials Stating only one purpose or aim: Most programs will have multiple objectives but within each objective only one aim should be stated eg. start three prenatal classes for pregnant women Specifying a single end-product or result: eg. To establish a subcontract with the City Hospital (a lower order objective) Specifying the expected time for achievement :eg for the above start a specific time frame sometime between March 1 st to May 30 th

52 Examples of objective-setting Objective Operational indicator Provide information on hypertension A 25% improvement in post-test treatment to improve physicians versus pretest exam scores. knowledge of treatment of hypertension. Ninety percent of all treated pat- ients should have blood pressures within age-adjusted normal levels within 12 months Increase the percentage of recommended physician practices in the treatment of hypertension. Lower the age-adjusted blood pressure of patients with hypertension. A 30% increase in the percentage of recommended physician pract- ices performed from postprogram versus preprogram.

53 Programs and Evaluations A program is a theory and an evaluation is its test. In order to organize the evaluation to provide a responsible test, the evaluator needs to understand the theoretical premises on which the program is based (p. 55). Carol Weiss (1998)

54 Program theory Program theory - program theory describes what the intended intervention is expected to do and an explanation of the underlying rationale (whether explicit or implicit) for achieving the expected results A cause-effect relationship a theory-based approach allows a much more in-depth understanding of the workings of a program or activity the program theory. Logic models flow from program theory

55 Logic Models A logic model is a diagram and text that describes/illustrates the logical (causal) relationships among program elements and the problem to be solved, thus defining measurements of success. We use these resources For these activities To produce these outputs So that these clients can change their ways Which leads to these outcomes Leading to these results! Your planned work Your intended results

56 Components of Logic Model Resources human, financial, organizational, community for a program (also known as Inputs) Program activities processes, tools, events, technology, and actions Outputs direct products of the activities; may include types, levels or targets of services delivered by program Outcomes specific changes in program participants knowledge, skills, behaviors, functioning Short term goals may be achieved between 1-3 years; long-term 4 to 6 years Impact fundamental intended or unintended chages occurring in organizations, communities or systems as a result of activities in a span of 7 to 10 years

57 Clarification - Outputs vs. outcomes Example: Number of patients discharged from state mental hospital is an output. Percentage of discharged who are capable of living independently is an outcome Not how many worms the bird feeds its young, but how well the fledgling flies (United Way of America, 1999)

58 Why create logic models? Logic model development offers the following benefits * : Clearly identify program goals, objectives, activities, and desired results Clarify assumptions and relationships between program efforts and expected outcomes Communicate key elements of the program Help specify what to measure in an evaluation Guide assessment of underlying project assumptions and promotes self-correction *

59 If-then relationships Underlying the logic model is a series of if-then relationships that express the program s theory of change IF then IF then IF then IF then IF then University of Wisconsin-Extension, Program Development and Evaluation

60 Logical chain of connections showing what the program is to accomplish INPUTS OUTPUTS OUTCOMES Program investments What we invest Activities Participation Short Medium What we do Who we reach What results Longterm University of Wisconsin-Extension, Program Development and Evaluation

61 Example of a Training Model Inputs Activities Outputs Outcomes Impacts Resources Services Products Benefits Changes Money Staff Volunteers Supplies Space Training Education Counseling Total # of classes Hour of service Number of participants completing course New knowledge Increased skills Changed attitudes New employment opportunities Trainees earn more over five years than those not receiving training Trainees have a higher standard of living than the control

62 Components of a HIV programme / Project Program level Population level Inputs Processes Outputs Outcomes Resources Staff Drugs, FP Supplies Equipment Functions, Activities Training Logistics Services % of facilities offering service % of communities with outreach # of trained staff Utilization: # of new clients # of return clients Short-term Contraceptive use # of HIV+ on ART Long-term Infection rate Mortality Fertility

63 How will activities lead to desired outcomes? A series of if-then relationships Tutoring Program Example IF then IF then IF then IF then IF then We invest time and money We can provide tutoring 3 hrs/week for 1 school year to 50 children Students struggling academically can be tutored They will learn and improve their skills They will get better grades They will move to next grade level on time 64 University of Wisconsin-Extension, Program Development and Evaluation

64 Basic Logic Model Outcomes and Impacts should be SMART: Specific Measurable Action-oriented Realistic Timed

65 Example Program Outputs Outcomes Crime control Highway construction Hrs of patrol # responses to calls # crimes investigated Arrests made Project designs Highway miles constructed Highway miles reconstructed From Poister, 2003 Reduction in crimes committed Reduction in deaths and injuries resulting from crime; Less property damaged or lost due to crime Capacity increases Improved traffic flow Reduced travel times Reduction in accidents and injuries 66 University of Wisconsin-Extension, Program Development and Evaluation

66 Part IV: Data Collection

67 Data collection Methods Collect Program or policy Evaluation Questions Design Approaches

68 Data collection Establish a clear timeline for data collection Be clear about your sampling strategy Everyone vs. subsample Think about the pros and cons in beginning Develop clear protocols Train staff Periodically monitor quality of data Protect confidentiality of participants

69 Indicators Indicators are the specific information that track a programs success. It is how you know that something has changed. Effective indicators Measureable Meaningful Manageable Clear As unbiased as possible Sensitive to change Acceptable to stakeholders

70 Outcome indicators - Example Outcome Initial Parents learn what their children are capable of doing Intermediate Parents participate in their child s education Long term Children are ready for school Indicator # of parents that demonstrate increased knowledge of child development through pre-post test on 5 key issues after attending workshops # of parents that attend one school-based event in addition to PTA conferences # of children that are developmentally ready based on standardized child development assessment tool

71 Steps to writing a good indicator Identify exactly hope to benefit (WHO?) Identify specific, observable change or accomplishment (WHAT?) Determine when the outcome is expected to occur (BY WHEN?) Indicator example WHO ---- # parents WHAT --- demonstrate increased knowledge of child development BY WHEN ---- after attending the workshop

72 The Evaluator s dilemma "Not everything that counts can be counted and not everything that can be counted, counts." - Albert Einstein

73 Generating Evaluation questions Meaningful questions are the heart of evaluation Qualities of good questions include: Relevant to the purpose of the evaluation and program goals so that they are useful for important decisions; Important to the identified audience(s); Comprehensive enough to provide adequate information about what is being evaluated; Constructed in ways that information is balanced and not bias; Answerable with realistic means and at a reasonable cost.

74 Three kinds of questions Descriptive Seek to describe the program or process Answer questions, such as: who? what? where? when? how? and how much/many? Normative Compare the current situation against a specified target, goal or benchmark. There is a standard or criterion against which to compare achieved performance. look at what is and compare it to what should be Cause-Effect - Cause-effect questions seek to determine the effects of a program Is it the intervention that has caused the results or something else?

75 Data sources & Data collection Data sources Who or where will I get the information from? Data collection method What is the tool or method for collecting the data? How is the tool administered? How often is information collected

76 Data for Evaluation Can be produced using any of the standard social-science research techniques Surveys Focus groups In-depth interviews Client-provider observation Key informants Surveillance Plus, analysis of program data (administrative, financial, service statistics)

77 Data sources pros & cons Data Source Example Advantages Disadvantages Program records (yours or others) Report cards, certificates, referrals Available, accessible Value of data dependents on how carefully it is recorded (completeness, updated etc) Specific individuals or trained observers Teachers reports on student behavior, case manager, client Provides first hand account Can be biased by interpretation or perceived pressure Mechanical measures Blood test, scale, Relatively objective Findings affected by accuracy of device

78 General rules for data collection Use available data if you can If using available data be sure to find out how earlier evaluators Collected the data Defined the variables Ensured accuracy of data If you much collect original data Establish procedures & follow them Maintain accurate records of definitions & coding Pre-test, pre-test, pre-test Verify accuracy of coding, data input

79 Key issues about measurements Are the measures credible? Trustworthy or believable data collection Are the measures valid? Are the questions asked giving you information about the issues you want to know about? Are the measures measuring what counts? As opposed to what is easiest Are the measures reliable? Stability of measures - that it measures the same thing the same way in repeated tests Are the measures precise? How the language in data collection matches the measure

80 Quantitative & qualitative data Quantitative approach More structured Attempts to provide precise measures Reliable easier to analyze Qualitative approach Less structured Provides rich data Challenge to analyze Labor intensive to collect Usually generates longer reports

81 When to use quantitative vs. qualitative approaches If you: Want to do statistical analysis Then use this approach: Quantitative Want to be precise Know exactly what you want to measure Want to cover a large group Want anecdotes or in-depth information Qualitative Are not sure what you want to measure Do not need to quantify

82 Data Collection Methods The Data Collection Method Provide a description of the process for collecting the information Example A Annual review of program records of referral sent for housing subsidy Example B Caseworkers rate the family each month during home visit Example C Tool = Self-administered Questionnaire Distribution = sent via mail with stamped return envelope Frequency = sent 90 days after completion of program

83 Data Collection Example Outcome Indicator Target Data source Data collection methods Initial Parents increase knowledge # parents that demonstrate increased knowledge of child development through pre-post test on 5 key issues after attending workshops 200 The parents who participate in at least 2 sessions Written or online survey that is distributed at 1 st class and again at last class. Parents who did not complete both tests are not included in final results

84

85

86 Human Subjects Protection Human Subjects protections Vulnerable populations Confidentiality HIPAA Data Security Data Storage

87 Closing thoughts The only man who behaves sensibly is my tailor; he takes my measurements anew every time he sees me, while all the rest go on with their old measurements and expect me to fit them - George Bernard Shaw

88 Questions?