What frames do you use to understand system change in order to allow you to evaluate?

Transcription

1 & What frames do you use to understand system change in order to allow you to evaluate? Mark Cabaj From Here to There Consulting Jess Dart Clear Horizon Kate McKegg Developmental Evaluation Institute

2 Evaluating systems change initiatives 1. Define or frame the thing you are evaluating Frame 2. Frame the purpose of the evaluation Frame 3. Collect useful data What happened? 4. Analyse, interpret, judge, learn So what? 5. Determine what to do next What now?

3 Framing initiatives with systems change ambitions So how can we frame or define the thing we are hoping to evaluate? Three different frames that can be used independently or blended: a) Principles and strategy continuum b) Critical systems approaches c) Theory of change

4 & Principal based approach Mark Cabaj

5 Lower Certainty & Agreement Higher Certainty & Agreement INCLUDING PRINCIPLES!!!

6 Strategy As Principles Many social innovators working on complex issues are driven by principles, as much as theories of change and plans. Principles: Reflect deeply held values and beliefs about what should be done for ethical reasons (e.g., that people most affected by an intervention should be involved in selecting, designing and evaluating it, also known as 'nothing about us without us ); and, Improve the chances of making progress on complex issues (e.g., Housing First vulnerable persons who find themselves homeless should be given safe and adequate housing BEFORE they can be expected to wrestle with their other vulnerabilities).

7 Rules Versus Principles Rule Principle Teaspoon of Salt Season to Taste

8 Example: Housing First #evalcomplexity2018

9 Example: Tackling Youth Homelessness in Minnesota: Guided by Principles

10

11 11 #evalcomplexity2018

12 Evaluating Principles Questions 1. To what extent are principles meaningful for people? 2. To what do people manifest or apply these principles? 3. What factors enable and/or undermine their ability to implement principles? 4. To what extent does following principles contribute to our ability to achieve desired outcomes? 5. What should we change in our principles? Some Methods Case Study Expert or Opinion Observation Self-Assessment

13 Another Example: The Paris Declaration The Paris Declaration (2005) is a practical, action-oriented roadmap to improve the quality of aid and its impact on development. It gives a series of specific implementation measures and establishes a monitoring system to assess progress and ensure that donors and recipients hold each other accountable for their commitments. (OECD) #evalcomplexity Ownership: Developing countries set their own strategies for poverty reduction, improve their institutions and tackle corruption. 2. Alignment: Donor countries align behind these objectives and use local systems. 3. Harmonisation: Donor countries coordinate, simplify procedures and share information to avoid duplication. 4. Results: Developing countries and donors shift focus to development results and results get measured. 5. Mutual accountability: Donors and partners are accountable for development results.

14 Evaluation of Principles International Evaluation Team Indicators of Application: Distance, Direction & Speed Findings Country ownership has advanced farthest Alignment and harmonization improved unevenly. Mutual accountability and managing for results lagging most Action on mutual accountability is now the most important need - backed by transparency and a realistic acceptance and management of risks and

15 & Critical Systems Approach all about ethics, boundaries and reflection Kate McKegg

16 A systems approach begins when first you see the world through the eyes of another" (C. West Churchman, 1968) A Critical Systems approach entails carefully and critically considering the practical choices professionals make in their practices and potential consequences of these choices. (Gates, 2017)

17 a framework for reflective practice based on practical philosophy and systems thinking (Werner Ulrich, 2005, p1)

18 For example. Facts Values What is (actual) What facts are considered relevant and which ones are left out? What values are considered relevant and which ones are left out? What ought to be (ideal) What facts ought to be considered relevant and which ones should be left out? What values ought to be considered relevant and which ones should be left out?

19 Now let s consider. Motivation Where does a sense of purposefulness and value come for the system change? Whose interests are / or should be being served? How do we (or should we) determine success? Power Who is (ought to be) in control of what is going on and what is needed (e.g., resources) for success? Knowledge What experience and expertise support (or ought to) the claims we are making? What counts (or should) as relevant knowledge? Legitimacy Where does legitimacy for our claims lie? What gives the system the legitimacy to carry out its tasks? Who argues (or should argue) the case for those affected but not involved? What worldview is (or ought to be) determining?

20 3 core questions: What s at stake? What ought to be at stake? E.g., access to water, economic development, employment, traditional customs and sacred sites, access to education, health services etc Who are the (involved and affected) stakeholders? Who ought to be the stakeholders? What are the key issues or problems related to what s at stake? Taking economic development or employment, the issues or problems depend on different perspectives e.g., for a local business owner, it could be lack of skilled labour, whereas for the local tribe it could be loss of young people to urban centres

21 For more information on critical systems Martin Reynolds and Bob Williams, Systems Thinking and Equity-Focused Evaluations. pp in Evaluation for equitable development results, UNICEF Ulrich, W. (2005) A Brief Introduction to Critical Systems Heuristics (CSH) Gates, E. (2017) Toward Valuing With Critical Systems Heuristics, American Journal of Evaluation.

22 & Theory of Change Approach Jess Dart

23 What is theory of change? A diagram or narrative that unpacks how change happens in a context Focuses on cause and effect Unpacks key assumptions

24 A dynamic tool Adapts to change Is refined over time with new knowledge Enables a shared understanding Enables a clear view of the system

25 Theory of change? #evalcomplexity2018

26 & 3 challenges and 3 ways theory of change can help

27 Challenge 1 Tension when evaluating systems change initiatives: Long-term nature of change Need for results to provide legitimacy/ funding

28 How can theory of change help? Theory of change can help you create a theory of what needs to be in place to move the needle on population results You can spread expected outcomes over time, and build and refine your model as you proceed Then you measure the expected preconditions and trust on your theory that the rest is coming!

29 Across time #evalcomplexity2018 Year zero Early years 1-3 Middle years 2-5 Late years 5-9 Baseline condition at population level Tracking change Systems ripples Impacts at population level Instances of impact System changes: solve local problems; join up; community agency; better rules Changes in capability of systems influencers Lessons from set-up Effectiveness, efficiency of anchor agency and funders to catalyse, convene and build capacity

30 Challenge 2: Clarifying boundaries and scope When working on complex systems change initiatives, boundaries are much looser and emergent than in a program You may chase a problem down from one sector to another So how to define boundaries and decide when to stop?

31 Global theory of change can help In taking a systems change approach to theory of change, we can start with a big, fat, loose global theory of change (everything that needs to be in place) And decide which pieces to work on Whilst keeping an eye on the bigger system

32 Challenge 3: Telling a story of contribution and navigating in your sphere of influence In complex programs with many moving parts, it s hard to show that you have contributed to the change How can you understand your contribution? You ll need a clear theory of how you contributed to necessary preconditions.

33 Sphere of influence and concern #evalcomplexity2018

34 Theory of change can Help clarify the boundaries of your pieces of work Keep an eye on the bigger system outcomes you desire Tell a story of progress at different phases Help tell a story of contribution

35