Realizing More Benefits From Logic Models: Background

Transcription

1 Realizing More Benefits From Logic Models: What Logic Modeling Can Learn from Benefits Realization Presentation to Canadian Evaluation Society National Capital Region Chapter June 19, 2007 Background Evaluation is intended to inform Deputies, senior government managers and taxpayers Are outcomes/benefits being realized Value for money Logic Models are a useful Evaluation planning tool: display program theory, desired outcomes or benefits that program is trying to realize, and identifying issues But Logic Models do not tell senior managers what needs to be done to realize benefits or who should be doing it (i.e., accountability) How can Logic Models overcome these weaknesses and be even more beneficial to senior managers? June 19,

2 Outline 1. What is a Logic Model? 2. Weakness of Traditional Logic Models 3. What is Benefits Realization? 4. Benefits Realization within Government of Canada (GC) 5. Comparison of Evaluation Logic Model and Benefits Realization 6. What Evaluators can Learn from Benefits Realization Results Chain 7. Conclusions June 19, Logic Model provides... Schematic of the program theory, i.e., how program is supposed to work to achieve intended changes and outcomes, and how it will make a difference for the affected stakeholders Visual of high level road map linking program activities, outputs, affected stakeholders and what you want to achieve Way to facilitate thinking, planning, communication and shared understanding about program and its intended outcomes June 19,

3 1. Logic Models are used... As the basis for: identifying outputs/outcomes, coming up with indicators and data sources for ongoing performance measurement, and developing evaluation issues and methods In GC context, integral part of planning: government program Business Case, and its performance measurement, and evaluation of program, policy or initiative as part of Evaluation Framework or Results-based Management Accountability Framework (RMAF) Policy Responsibility Centre of Excellence for Evaluation, TBS June 19, A Logic Model often looks like this... Objective Activities Outputs Immediate Outcomes Intermediate Outcomes Ultimate Outcomes Some (or all) Canadians benefit or are better off along one or more dimensions (health, wealth, literacy, security, safety, understanding who we are) June 19,

4 2. But Traditional Logic Models do have weaknesses... Logic Models are good for identifying outputs/outcomes and evaluation issues at high level, but do not clearly identify: Exactly what has to be done to achieve intended outcomes Accountability, i.e., who is responsible for achieving outcomes Attribution and external factors Outcome box size does not reflect effort, time or resources required Assumptions and risks associated with achieving outcomes Developing Logic Model may be a strategic process, but not really used as management tool once evaluation issues are determined June 19, The Miracles of Logic Models June 19,

5 2. Overcoming Logic Model Weaknesses Benefits Realization can help overcome Logic Model prescriptive and accountability weaknesses 1. What is Benefits Realization? 2. Where does it come from? 3. Can Benefits Realization be applied to programs and help Logic Models realize benefits? Over last year or two, TBS Chief Information Officer Branch (CIOB) piloted Benefits Realization re-labelled as Outcome Management June 19, Outline 1. What is a Logic Model? 2. Weakness of Traditional Logic Models 3. What is Benefits Realization? 4. Benefits Realization within Government of Canada (GC) 5. Comparison of Evaluation Logic Model and Benefits Realization 6. What Evaluators can Learn from Benefits Realization Results Chain 7. Conclusions June 19,

6 3. Benefits Realization what it is Traditional IT Project Management - deliver project on time, within budget and quality targets, but business benefits often not realized Benefits Realization is set of activities, methods, processes, and tools for understanding, planning, tracking, managing and realizing desired outcomes from IT initiatives or projects Benefits Realization methodology Created in Canada in 1990s Published in The Information Paradox Realization of intended outcomes or benefits includes business function beyond the IT project Logic of process to realize benefits is represented by Results Chain Helps manage stakeholder expectations Benefits Realization developed with no knowledge of Program Evaluation But trying to solve similar problems June 19, Benefits Realization What it can look like A Key Deliverable: Results Chain links project initiatives to outputs & outcomes Legend Assumptions / Risks Project Initiatives Stakeholder Initiatives GC Outcomes Project Outcomes Key Outcomes OGD Initiatives June 19,

7 Simplified Example: 3. Benefits Realization A closer look Results Chain model linking new on-line tracking tools that track data on OGD compliance to new policy and integrate with data from external stakeholders Assumptions / Risks Project Initiatives GC Outcomes Project Outcomes I-5 Stakeholders modernize their systems A-4 TBS modernizes policies Stakeholder Initiatives OGD Initiatives Key Outcomes A-1 OGD senior management commitment O-5 Time and effort required to produce reports decreased O-6 Tracking costs reduced O-7 Reallocation of resources to higher government priorities I-2 Develop on line tools to track policy outcomes O-2 On-line monitoring tools created O-4 Use of tracking tools increased O-8 Standardization of data and reports increased O-9 Compliance to policy increased O-10 Achieve greater compliance to policy I-3 Develop training protocol and plans O-3 Number of trained employees increased A-2 Data from third party sources will be available on a timely basis I-4 OGDs shift resources from administration to analysis A-3 Staff can be refocused I-1 OGDs authorize skilled resources June 19, 2007 to attend training Benefits Realization within Government of Canada (GC) GC IT/IM projects: deliver on time, on budget to required specifications but still challenge of achieving intended business outcomes CIOB exploring how Benefits Realization could increase project success and help achieve intended business outcomes In June 2005, ADM-level TBS Service Management Board (SMB) mandated GOL Initiative to: promote the Benefits Realization methodology endorsed by the TBS CIOB has piloted use of Benefits Realization, e.g., Shared Travel Service Initiative (STSI) BR viewed as supporting complex horizontal or business transformation initiatives by breaking down silos and crossing traditional organizational boundaries to achieve intended outcomes June 19,

8 4. Benefits Realization Potential Benefits Realization Results Chain has potential to help overcome weakness of Logic Model: 1. Lays out how downstream business benefits or outcomes can be achieved 2. Supports project, program and initiative levels 3. Supports an enterprise and horizontal approach 4. Details and assigns accountabilities for realizing outcomes 5. Facilitates on-going and ex post evaluation of hard and soft benefits June 19, But even Benefits Realization has limitations Limitations of BR Results Chain A lot of information to collect and to convey in a set of diagrams Too complex to present in a single diagram without extensive explanation Few senior managers are prepared to deal with this level of detail Senior Executive version not always developed for each project June 19,

9 Outline 1. What is a Logic Model? 2. Weakness of Traditional Logic Models 3. What is Benefits Realization? 4. Benefits Realization within Government of Canada (GC) 5. Comparison of Evaluation Logic Model and Benefits Realization 6. What Evaluators can Learn from Benefits Realization Results Chain 7. Conclusions June 19, Comparing Evaluation Logic Model and Benefits Realization (1 of 4) Evaluation Logic Model Benefits Realization / Results Chain TBS CEE TBS CIOB silo Outcome Management Government Program driven Mandated by TBS and Departmental evaluation committee IT Project driven Recommended by project manager and stakeholders Planning tool - can help manager and stakeholders understand logic of program and theory of how outcomes can be achieved Planning tool - can help IT project manager and stakeholders understand logic of how desired business outcomes can be achieved and prescribe what has to be done to achieve outcomes June 19,

10 5. Comparing Evaluation Logic Model and Benefits Realization (2 of 4 cont d) Evaluation Logic Model Benefits Realization / Results Chain Linked to Evaluation culture help determine if outcomes have been achieved Linked to Management Improvement culture help determine what to do to achieve intended outcomes Does not address scope of change required by Public Servants and stakeholders to achieve outcomes Chaos and hope no clear accountability for achieving Outcomes Prescribes Business Transformation by Public Servants and stakeholders to achieve desired outcomes Chaos and accountability clear accountability of what has to be done to achieve intended Outcomes and who has to do it June 19, Comparing Evaluation Logic Model and Benefits Realization (3 of 4 cont d) Evaluation Logic Model Benefits Realization / Results Chain Ongoing without time limit toward Ultimate Outcome Finite time horizon Desired outcomes achieved Useful in planning Evaluation (Framework) but also maybe in Evaluation Useful as planning and ongoing management tool, and for monitoring and measuring achievement Once developed for Evaluation Framework, not likely to change during Evaluation cycle Updated and maintained as environment changes and phases of project are completed June 19,

11 5. Comparing Evaluation Logic Model and Benefits Realization (4 of 4 cont d) Evaluation Logic Model Benefits Realization / Results Chain Does not explicitly address assumptions and risks no sense of urgency - Although RMAF text can address these Explicitly addresses assumptions and risks conveys sense of urgency and what has to be done and who has to do it Visual display is relatively easy to follow Visual display is complex; not so easy to follow Deputies and senior managers can benefit from an Evaluation Deputies and senior managers can benefit from the application of the Benefits Realization methodology June 19, Outline 1. What is a Logic Model? 2. Weakness of Traditional Logic Models 3. What is Benefits Realization? 4. Benefits Realization within Government of Canada (GC) 5. Comparison of Evaluation Logic Model and Benefits Realization 6. What Evaluators can Learn from Benefits Realization Results Chain 7. Conclusions June 19,

12 6.What Evaluators / Logic Modellers Can Learn (1 of 2) 1. Benefits Realization Results Chain includes different stakeholders, initiatives, and assumptions and risks impacting achievement of desired outcomes or benefits 2. BR Results Chain lays out the complex sequence of initiatives and outputs/outcomes and lays out the accountabilities 3. BR Results Chain is more detailed and comprehensive than a Logic Model it tells a more complete story consistent with program theory 4. BR Results Chain is powerful tool for dealing with: horizontal initiatives, projects or programs, and new initiatives and change initiatives June 19, What Evaluators / Logic Modellers Can Learn (2 of 2 cont d) 5. Explicitly identify what project/program, department and external stakeholders are expected to do to help achieve outcomes, i.e., realize benefits 6. Explicitly identify underlying assumptions about how program activities yield outputs and how outputs are transformed into outcomes 7. Explicitly identify external factors (i.e., risks and barriers) that may inhibit or accelerate program achieving outcomes and how to deal with them 8. Colour code related initiatives to convey sense of urgency June 19,

13 7.Conclusions Current Logic Models were developed in a simpler age Logic Models are useful for understanding link between program and outcomes, but today face real challenges accountability, external factors, risks, horizontality BR Results Chains, coming at these challenges from a different direction, has developed techniques and processes for addressing these challenges BR Results Chains are useful for understanding how to achieve desired outcomes Opportunity exists to modernize Logic Models and to increase their value by incorporating aspects of BR Results Chains June 19, Arthur Rabinovitch, CMC Maxar Consulting Inc. (formerly with Government Consulting Services) maxar@rogers.com For further information on the BR methodology, please contact: David Heath Principal Fujitsu Consulting david.heath@ca.fujitsu.com During June and July 2007 James McGibbon Principal Fujitsu Consulting ext 4401 james.mcgibbon@ca.fujitsu.com June 19,