Agenda June 10th, :00am-1:00pm

Transcription

1 Developing Your Theory of Change Presented by HTA Sponsored by City of Richmond Agenda June 10th, :00am-1:00pm Welcome and Introductions The What and Whys of a Logic Model Logic Modeling 101: My First Logic Model Break Logic Modeling 201: Choosing a Detailed Approach Next Steps: Using your Logic Model Resources and Wrap Up Hand in Hand: An NGO Community Network for a Thriving Richmond

2 Introduction to Logic Models What and Whys of Logic Models What is a Logic Model A logic model is an outcomes-based approach to program planning and evaluation. It is a logical and visual way to present the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve. It provides a roadmap from your identified need to your planned work, and finally to your intended results. Why Should you Develop a Logic Model Logic models provide stakeholders (e.g. program staff, funders, partners, community members and program participants) with a common understanding and language. It can help to communicate a complex program to external stakeholders, and at the same time bring detail to an organization or programs broad goals. Your logic model clarifies the difference between activities what you do and outcomes the impact you have. Finally, it creates a chain of reasoning from resources through outcomes that you can use to test assumptions and inform your evaluation. Types of Logic Models and Components Components of a Logic Model Environment (What you know) The Context and Situation The Planned Work Intended Results Assumptions (What you think) Inputs Activities Outputs Short and long-term Outcomes Impact Outcomes A description of your target population, including the demographics, geography, community needs, assets and challenges. A description of best practices or research, or other influencing factors that may help of hinder your program. The human, financial, organizational, and community resources a program has available to direct toward the work. What the program does with the resources. The activities, services, tools, events, and products that are an intentional part of the program. The direct product of the activities, i.e. the types, levels and targets service numbers. The specific changes in program participants behavior, knowledge, skills, status or level of functioning that are affected by the program activities. Outcomes are grouped into short term (1-3 years) and long-term (4-6 years) Impact outcomes (7-10 years) are fundamental changes occurring in organizations, communities or systems as a result of the program activities Hatchuel Tabernik & Associates Page 1 of 2

3 Theory Approach is the big picture logic model. It details the theory of change that is at the core of your program design. This type of logic model illustrates the how and why of your program design by describing: the problem or issues to be addressed the reasons for selecting certain strategies how to connect proven strategies to potential activities assumptions the planners hold that influence effectiveness Activities Approach is the implementation logic model, as it links activities together to map the process of program implementation. These logic models describe what a program intends to do in a detailed fashion, addressing each activity separately, with its outputs and outcomes. Outcomes Approach is the most results focused logic model. It illustrates the connection between resources and activities to outcomes. Outcomes are often dived into short-term (1 to 3 years), long-term (4 to 6 years), and impact (7 to 10 years). While outcome logic models are developed with a theory of change in mind it is not usually emphasized explicitly. Resources W.K. Kellogg Foundation The Kellogg Foundation has a wide range of useful resources for evaluation and programming. Downloadable resources include English and Spanish versions of a Logic Model Development Guide and an Evaluation Handbook, which provides a framework for thinking about evaluation as a relevant and useful program tool. Harvard Family Resource Project takes you to a downloadable brief on logic models and how to construct them Hatchuel Tabernik & Associates Page 2 of 2

4 Logic Models: Making the Case for Your Program The Context and Situation The Planned Work The Intended Results What you Know What You Think Inputs Activities Outputs Short-term Outcomes Longer-term Outcomes Your environment, target population, challenges, community needs, and assets Your assumptions, best practices or research, influencing factors that can help or hinder Resources you invest, e.g. funding, personnel, inkind, partners What activities, services, supports or products you offer & how often Who you serve, numbers and level of participation Results you will achieve in 1-3 years Results you will see after 3 years and long-term impact Adapted by Hatchuel Tabernik & Associates, 2008

5 Logic Model Sample The Context and Situation The Planned Work The Intended Results What you Know What You Think Inputs Activities Outputs Short-term Outcomes Target population - K-8 students, 1-2 grade levels below in academics, from low-income neighborhoods Needs: academic support safe and nurturing place after school leadership opportunities physical activity Challenges: Language and income barriers Pressure on schools and after-school programs to meet NCLB Strengths: Cultural diversity among target population, staff, and partners Support from foundations, parents, schools, and city/county Belief that our services can impact these needs based on prior evaluation data and reports research on afterschool programs including promising practices 7 years experience Budget changes in state-funded afterschool programs Cost of basic operations = $ Value of in-kind contributions = $ Request grant funding for expanded programming = $ Partnerships with xxx Volunteers from xxx Daily homework hour Daily enrichment activity Daily recreation activities Small group Academic Intervention Service Learning Club Health and Nutrition classes Staff development in target areas Family learning nights 180 students participate in 4 hrs per week of Homework hour for 32 weeks during the school year. 70% attend 30 or more program days that include Homework Support. A minimum of 12 staff will receive 6 hrs of training; 80% will report high satisfaction Improved homework completion Improved standardized test scores Increased school engagement and attendance rates Increased leadership skills Improved relationships with adults and peers Increased physical activity Longer-term Outcomes Higher graduation rates among target population Decrease in juvenile delinquency in after school hours Increased muscle tone and physical fitness Increased civic engagement among young people in target communities Hatchuel Tabernik & Associates, 2009

6 The Context and Situation What you Know What You Think Your environment, target population, challenges, community needs, and assets Your assumptions, best practices or research, influencing factors that can help or hinder Adapted by Hatchuel Tabernik & Associates, 2009

7 The Planned Work Inputs Activities Outputs Resources you invest, e.g. funding, personnel, in-kind, partners What activities, services, supports or products you offer & how often Who you serve, numbers and level of participation Adapted by Hatchuel Tabernik & Associates, 2009

8 The Intended Results Short-term Outcomes Results you will achieve in 1-3 years Longer-term Outcomes Results you will see in 4-6 years Impact Outcomes Fundamental changes will you see in your community, systems, and/or organization Adapted by Hatchuel Tabernik & Associates, 2009

9 Action Plan Completing and Using Your Logic Model Who else should be involved in developing your Logic Model? Think about your stakeholders, i.e. staff, Board, participants, partners, community members When will you work on the Logic Model? When do you want to have the Logic Model completed? Do you need to set specific meetings? When and where will you meet? How will you use your Logic Model? Think about the main purposes of logic models, i.e. create common language among stakeholders (who are they?), articulate your vision and theory of change, strengthen your program design, clarify your outputs v. outcomes, identify evaluation points, communicate with funders, marketing and outreach to participants, families, partners, etc. Purpose Stakeholder(s) Timeframe 2009 Hatchuel Tabernik & Associates

10 Program Planning Template Strategies 5 Assumptions 6 Influential Factors Problem or Issue 4 1 Desired Results (outputs, outcomes, and impact) 3 Community Needs/Assets 2 Sample template from the W.K. Kellog Foundation Logic Model Development Guide

11 Program Implementation Template Resources Activities Outputs In order to accomplish our set of activities we well need the following: In order to address our problem or asset we will accomplish the following activities: We expect that once accomplished these activities will produce the following evident or service deliver: Short & Long Term Outcomes We expect that if accomplished these activities will lead to the following changes in 1-3 then 4-6 years: Impact We expect that if accomplished these activities well lead to the following changes in 7-10 years: Sample template from the W.K. Kellog Foundation Logic Model Development Guide

12 Program Evaluation Template Project Goal: Services What You Do, When & How Often Directions: Describe the services (Major project deliverables) you will carry out that will lead directly to your desired outcomes. Output #1 Participation Who & How Many You Serve Directions: For each service describe the participation levels you expect, including who and how many you anticipate serving. Output #2 Quality How Well You Serve Them Directions: For each service, describe the level of quality you expect as indicated by participant satisfaction, self-assessment rating, environ-mental rating, and/or ability to meet field established standards of quality. Services What You Do, When & How Often Directions: Describe the services (Major project deliverables) you will carry out that will lead directly to your desired outcomes. Example Service 1 Parenting Class (10 weeks, 2 hours week) Participation Output parents will complete all sessions of parenting class Quality Output 1. 75% of parents rated the expertise & responsiveness of parent educators as high Outcome 1. 80% of parents will increase their knowledge of child development and parenting skills Measure: # of parents completing class Target: 25 Data Source: Sign-in Sheets Measure: % parent giving high ratings to parent educators Target: 75% Data Source: Satisfaction Survey Measure: % parent Increasing knowledge and skills Target: 85% Data Source: Pre/Post Surveys Sample template from the San Francisco First 5 Commission

13 Program Evaluation Template Project Goal: Services What You Do, When & How Often Service 1 Output #1 Participation Who and How Many You Serve Output #2 Quality How Well You Serve Them Participation Output 1 Quality Output 1 Outcomes What Will Change Outcome 1 Measure: Target: Data Source: Measure: Target: Data Source: Measure: Target: Data Source: Service 1 Participation Output 1 Quality Output 1 Outcome 1 Measure: Target: Data Source: Measure: Target: Data Source: Measure: Target: Data Source: Sample template from the San Francisco First 5 Commission

14 PROGRAM DEVELOPMENT Planning Implementation Evaluation Cooperative Extension Program Development & Evaluation UW-Extension provides equal opportunities in employment and programming, including Title IX and ADA.

15 Documenting Progress and Demonstrating Results: Evaluating Local Out-of-School Time Programs September 2002 by Priscilla Little, Harvard Family Research Project, Sharon DuPree, Consultant and Sharon Deich, The Finance Project Harvard Family Research Project 3 Garden Street Cambridge, MA (fax) New York Avenue, NW, Suite 800 Washington, DC (fax)

16 Part II: Logic Models Part II: Logic Models In designing an evaluation, it is important to be clear about the goals of the program and to be realistic about the expectations for which it can be held accountable. In Part I, Section 2, we noted that a logic model is a useful tool to assist programs in defining their goals and figuring out the focus of their evaluation. Many organizations rely on logic models to help guide planning and program improvement efforts. This part of the guide describes how to develop a logic model. measure performance, it is one of the key methods used to assist organizations in tracking program progress. Since creating a logic model requires a step-by-step articulation of the initial goals of the program and the proposed activities that will be conducted to carry out those goals, it also helps a program chart the resources it needs to carry out the evaluation process. Information for logic models can come from a variety of sources. The PROJECT FOR NEIGHBORHOOD AFTERCARE in Nashville, Tennessee, interviewed program staff to learn what the goals of the program were, how the program intended to achieve these specific goals, and what program actions were considered key to achieving the goals of the program. The resulting program logic models or flow charts were then presented individually to the staff for confirmation that the logic models accurately reflected how the program was intended to make a difference in the lives of students. The program s goals and logic models are expected to evolve with the program. For more information on the Project for Neighborhood Aftercare evaluation, see the Harvard Family Research Project Out-of-School Time Evaluation Database. Available on the Internet at: projects/afterschool/evaldatabase.html. Why Develop a Logic Model? There are many different logic models, just as there are many frameworks for learning about results-based decision making. Programs may want to consider several approaches before choosing or modifying a particular model to fit their circumstances. Once the concepts are familiar, the same understanding may be applied to any logic model. The logic model, like an architect s scale model of a building, is not meant to be a detailed blueprint of what needs to happen. It is a place to lay out the major strategies, to see how they fit together and whether they can be expected to add up to the changes the OST stakeholders want to see. Creating a logic model at the beginning of the evaluation process not only helps program leaders think about how to conduct an evaluation, but also helps programs choose the programmatic aspects they want to evaluate. Practitioners who have created logic models for evaluations have been most successful when they have included administrators, staff, and, when possible, youth in the process. While a logic model is not the only tool needed to What Are the Elements of the Logic Model? Based on information from a guide published in 2000 by the Child Care Partnership Project, a logic model has several elements. 3 Result: The logic model starts with the program s desired result-what is its vision, what does it ultimately want to happen or exist in its community? Conditions and Causes: The next step is to examine the story behind the results-what are conditions, causes, circumstances, or factors that need to change in order to move toward the result? They are what cause the result to happen or prevent it from happening. These conditions should be based on research or, at least, specific experience about what works. Strategies: The next step is to examine the broad approaches or strategies that will affect those conditions or causes in ways that help the organization reach the desired results. Both the strategies and the activities need to include all partners who will work to implement the plan. page 17

17 Evaluating Programs Harvard Family Research Project The Finance Project DEVELOPING A LOGIC MODEL Developing a logic model essentially requires filling out its elements (represented by the boxes to the right) with details based on your OST program. While there is no correct order in which to do this, it is suggested that you start with the left column of the logic model, and move counterclockwise. ❶ Decide on your desired result. Decide what result (goal or vision) you are trying to achieve. This result should express your vision for children, youth, and families. PROGRAM ❶ Desired Results ❷ Motivating Conditions and Causes ❸ Strategies ❹ Activities OUTCOMES ➏ Indicators ❺ Performance Measures ❷ Decide on the major conditions and causes behind that result. Starting with the desired result, discuss the indicator data, research, and experience that suggest the major conditions, causes, and factors that affect your result. Think broadly before narrowing your scope. ❸ Develop strategies. Strategies are broad approaches to child, youth, and family well-being. Since you may not be able to implement all of the desired strategies at once, it may be helpful to prioritize them based on your list of indicators. Once you have prioritized your strategies, you can develop a more detailed action plan and performance measures for those activities you plan to implement first. ❹ Develop activities. Activities are the individual services or interventions your program uses to implement your strategies. Ask yourself: On a day-to-day basis, what does the staff in my organization do? What services do we provide? Work plans may be useful for identifying this list (e.g., homework help and tutoring, mentoring, rap sessions). Then develop an initial list of activities to implement the strategy. ❺ Develop performance measures. Considering the activities you plan to pursue, develop some performance measures that can track progress on the activities. These measures include measures of effort (amount of activities you are implementing), as well as measures of effect (the changes in the child, the youth, or the family population you are serving). ➏ Decide on the major indicators you want to affect. Using the indicators from step two, choose a small number of indicators that you want to affect. Remember, indicators reflect community-wide changes in children, youth, and families, so you will need partners to change these numbers. (See page 22 to continue the logic model steps) page 18

18 Part II: Logic Models ELEMENTS OF THE MODEL THE PROGRAM Desired Results Improve the physical, social, and emotional well-being of children. Improve children s academic development and performance. Motivating Conditions and Causes Many parents working outside of the home Children with unstructured and unsupervised time in the after school hours Low academic performance among low-income children Lack of positive adult-youth relationships Children at greater risk for involvement in crime and substance abuse in the hours after school Television as the most common activity for children after school AS MEASURED BY THE OUTCOMES Indicators Ultimate Indicators Reduced rates of substance use among teens Reduced teen pregnancy rates Reduced numbers of violent acts among adolescents and teens Reduced dropout rates Require Increased percentage of community-wide students graduating from effort to affect high school Increased percentage of students attending college Interim Indicators Improved test scores in reading, math, or science Reduced numbers of antisocial behaviors or behavior problems Decreased student suspensions Improved grades OST Program Strategies Youth development and leadership Academic enrichment Curriculum development and enrichment Collaboration OST Program Activities Homework help and tutoring Mentoring Rap sessions Arts activities Recreation activities Technology training Literacy activities Career counseling and development Community service or work projects Intergenerational activities Conflict resolution training Program Performance Measures Measures of Effect Development of emotionally supportive relationships with adults Increased emotional adjustment Increased social competence Higher self-esteem and Can confidence be affected by Improved study habits the OST program Higher numbers of honors or awards received Improved peer relationships Improved attitudes toward school Improved school attendance and decreased truancy Measures of Effort Number of children served in the OST program and participant demographics Number of classes, sessions, or trainings held Number and type of products developed Measures of program cost-effectiveness Parent and child satisfaction rates with the OST program page 19

19 Evaluating Programs Harvard Family Research Project The Finance Project Activities: The final step identifies the activities, programs, interventions, and services that need to be implemented to carry out the strategies. Each activity the program pursues may change only one or a few of the conditions and causes that would need to change in order to reach the desired result. Each activity may also affect only a subset of the total target population. So the program leaders need to think about how much activity program staff need to implement to bring about the desired improvements. Each element should also be based on research or experience that explains why a bad condition exists or a good condition does not exist (the conditions and causes) and what will fix it (strategies and activities). On the previous page is an example of a logic model for an out-of-school time program. 4 Following this example is a matrix of sample performance measures and possible data sources. What Performance Measures Do Out-of-School Time Programs Use? Once a program has created and articulated its desired results, motivating conditions and causes, its OST strategy, and its program activities, the next step in logic model development is to determine how to use performance measures to assess progress. As previously stated, an indicator is a measure, for which there are data, that gauges community-level progress toward agreed-on results. Because results are broad statements of what communities want for their children, no single indicator is likely to signal full attainment of any given result. Further, it is unrealistic to expect that a single OST program can move the needle on these community-wide indicators. Rather, programs can affect performance measures that track progress on program activities and purposes. Choosing performance measures that are reflective of program activities ensures natural methods of measurement of short-term performance and more accurately shows clear correlations between the goals of the program and the results. Two families of performance measures are familiar to most OST practitioners. One type involves academic achievement measures. These measures assess individual performance related to school achievement, such as report grades or scores on standardized tests. The other type involves youth development measures, such as the measurement of decision-making skills or civic responsibility. Youth development measures are related to the less tangible effects a program has on an individual, but have the potential of making a long-term impact on an individual s academic achievements. Compiled from a review of city-level OST evaluations, the table on page 21 illustrates the range of performance measures OST programs use to measure academic achievement and youth development, as well as the data sources used to obtain information about the measures. 5 It clearly shows that there are many ways to define and measure academic achievement and youth development. The selection of measures to use should be tied to the goals of the program and the activities it provides. page 20

20 Part II: Logic Models Measures and Data Sources for Outcome Areas OUTCOME AREA Academic Achievement Academic performance in general Attendance or absenteeism Attendance in school related to level of program participation Attendance in school related to achievement Attitude toward school Behavior in school* Child s ability to get along with others Child s liking for school Child s communication skills Child s overall happiness Cooperation in school Effectiveness of school overall Effort grades English language development Expectations of achievement and success Family involvement in school events Grade point average Grades in content areas (e.g., math, reading) Homework performance Learning skills development Liking school more Motivation to learn Reading Safety viewing school as a safe place Scholastic achievement assessed by knowledge about specific subjects Standardized test scores Youth Development Adults in the OST program care about youth Awareness of community resources Behavior change toward new program component Child s self-confidence Exposure to new activities Facing issues outside of OST program Interaction with other students in OST Interest in nonacademic subjects (e.g., art, music) Leadership development and opportunities Opportunities to volunteer Productive use of leisure time Sense of belonging Sense of community Sense of safety Sources of support for youth DATA SOURCE Parent report, principal report School records, parent report, principal report School records School records, standardized tests Standardized behavior scales by teachers Parent report Parent report Parent report Parent report Principal report School records, teacher report Principal report School records School records, parent report Parent report, principal report Teacher report Parent report, teacher report, principal report, test scores Parent report SAT-9, state assessments Parent and child report Parent report Principal report Measures and data sources compiled from a review of findings from 13 city-level evaluations; for brevity, child refers to youth of any age participating in the OST program. * School behaviors included in the scales are frustration, tolerance, distraction, ability to ignoring teasing, nervousness, sadness, aggression, acting out, shyness, and anxiety. page 21

21 Evaluating Programs Harvard Family Research Project The Finance Project HOW DO PROGRAMS USE LOGIC MODELS FOR EVALUATION AND LEARNING? Logic models help OST programs describe their activities and identify outcomes and measures that will help them assess progress and results (i.e., steps one through six on page 18). However, they can also help programs move forward with evaluation in terms of putting plans in place to collect data on the identified measures and to use that data and the logic model for learning. Four additional elements (steps seven through ten) that can be added to the logic model toward that end are identified below. 6 The figure on the next page illustrates the relationships between those four elements and the six elements of the logic model discussed earlier. ❼ Data Sources and Methods These are the sources for the data needed to track indicators and performance measures. Ask yourself: Now that I have identified my measures, how will I get the needed data in the most resource-efficient way? If you used the criterion that data should already be available for the indicators you have chosen, then you should already know their data sources and how often they are available. However, you also need to determine how often to report that information, how to deliver it, and who will get it. Performance measures will likely require additional data collection that either you or your evaluator will conduct. Some of that information, such as the measures of effort, you can probably track on your own. However, you may need to use an external evaluator to collect data on the measures of effect (e.g., sources: standardized testing, state or local government databases; methods: surveys, focus groups, interviews). ❽ Evaluation Questions These are the questions you want to have answered by the data or decisions that you want to make based on your data. You should be able to make decisions based on your indicators and performance measures. Ask yourself: What strategic decisions can I make based on the information that will be generated? What consequences should be tied to achievement for good or bad performance? (e.g., Are the indicators showing progress and, if not, does that mean the OST program needs to be modified?) ❾ Stakeholders These are the individuals who have a vested interest in the OST program. They need to know the answers to the evaluation questions and to be involved in learning from the data being collected. Ask yourself: Who is interested in or will benefit from knowing my program s progress on its indicators and performance measures (e.g., board members, funders, collaborators, program participants, community members, and other individuals or organizations)? ➓ Mechanisms for Learning These are the periodic or regular opportunities that exist for pulling together and reporting the data being collected and for bringing together stakeholders to learn from and make decisions based on the data. Ask yourself: What opportunities exist or need to be created to focus on and learn from the evaluation (e.g., staff, stakeholder, or board meetings; regular evaluation reports; strategic retreats)? Coffman, J. (2002). Learning from logic models in out-of-school time. Cambridge, MA: Harvard Family Research Project. Available on the Internet at: page 22

22 Evaluating Programs Contents Logic Models for Learning and Evaluation PROGRAM Desired Results OUTCOMES EVALUATION AND LEARNING ❼ Data Sources and Methods Where the data needed to track the indicators and performance measures will come from. Motivating Conditions and Causes Indicators ❽ Evaluation Questions Questions, based on indicator data and movement, that determine whether the strategy needs to be modified. Questions, based on the performance measures, that determine whether the project is working as intended, what lessons have been learned, and how the project may need to be modified in order to achieve better results. Strategies Performance Measures ❾ Stakeholders The funders, collaborators, participants, and other individuals or organizations with a vested interest in the program who need to be involved in learning from the data being collected. Activities ➓ Mechanisms for Learning The opportunities for stakeholders to come together to learn from and make decisions based on the data about the program. Coffman, J. (2002). Learning from logic models in out-of-school time. Cambridge, MA: Harvard Family Research Project.Available on the Internet at: page 23

23 LOGIC MODEL WORKSHEET ELEMENTS OF THE MODEL The Program! DESIRED RESULTS What is my long-term vision for the children, adults, or families in my community, or for my community as a whole? # INDICATORS AS MEASURED BY The Outcomes Ultimate Indicators How will I know if my program s desired results have been achieved in the long-term? " MOTIVATING CONDITIONS AND CAUSES What are the factors, issues, or problems that my program is trying to improve or eliminate? Interim Indicators If my program is successful, what changes do I expect to see in my community in the next few years? Require community -wide effort to affect $ OST PROGRAM STRATEGIES What are the broad categories of services or approaches that my program provides? & PROGRAM PERFORMANCE MEASURES Measures of Effect How will I know that the children or families I work with in my program are better off? % OST PROGRAM ACTIVITIES On a day-to-day basis, what do staff in my organization do? What services do we provide? Measures of Effort What does my program generate, what levels of activity do we produce? Can be affected by the OST program Diagram from Learning From Logic Models in Out-of-School Time by Harvard Family Research Project at