Moving Towards Purchasing Outcomes

Transcription

1 Moving Towards Purchasing Outcomes Measuring the Impact: Local Authority Investment in West Lothian

2 Outcomes Outcomes are the effects or changes brought about by the activities provided by an organisation. Measurements of outcomes show the degree of its effectiveness, rather than its size, efficiency or productivity. 1 Charities Evaluation Service Outcome: the changes resulting from an activity 2 A Guide to Social Return on Investment A distinction is often drawn between outcomes as the impact or immediate result of services or interventions, and outcomes as the end condition or final state achieved. These are sometimes described as 'stage one' and 'stage two' outcomes. 3 Local Government Improvement and Development Website 1 vol.org.uk/index.cfm?pg=113 Retrieved February A Guide to Social Return on Investment Cabinet Office, office of the Third Sector, Retrieved February

3 Executive Summary Forth Sector Development, on behalf of the Scottish Government s SROI Project, has in partnership with West Lothian Council devised a model of purchasing services from third sector suppliers by outcomes. The model is derived from West Lothian Council strategy and policy, and initially aims to build higher level outcomes for priority populations identified in the Single Outcome Agreement and Life Stages Outcomes Planning Model into the purchasing process from the service design stage. Further to this, the model will provide an evaluation and monitoring system by which the Council can evidence the effectiveness of spend in achieving these outcomes and the impact the investment has on the wider community. It is anticipated that the introduction of an outcomes based evaluation strategy will enable the council to collate the necessary data to undertake impact measurement on a yearly basis. This will enable the Council to better understand the value of social return generated by any investment in social and community services and in the third sector in particular. The Purchasing Model includes: a model for the purchasing of services; an overview of a three phase transition from funding to commissioning outcomes; an introduction to the importance of service user and service provider input in service design; and a monitoring and evaluation framework. To test the model West Lothian Council intends to initiate a pilot with suppliers of Older Peoples Services. At present, nine suppliers are in receipt of 629,191 for provision of services to this population, including day care services to elders suffering from dementia to more arms length support including provision of social activities and a care attendant 3

4 scheme. At present, neither West Lothian Council, nor their third sector partners are ready to engage in a full tendering process for the provision of services. The model aims to move both the council and service provider closer to a full commissioning process over three phases, with the initial phase intended to be carried out through the funding cycle April 2011 to April Phase 1 of the model aims to move service providers from their current Funding Service Agreement to an Outcomes based Service Funding Agreement, and to introduce an outcomes based evaluation and monitoring system to run through the 2011 funding cycle. It is anticipated that Phase 2 will move both commissioner and service provider forward to work jointly through a Public Social Partnership to co produce the core service requirements. It is assumed that phase 3 would be a final transition for commissioner and service provider to a full commissioning process around the delivery of services to contract. The initial pilot will focus on 8 of the services currently providing Day care services in the region (the ninth service receives less than 1,000 from the council and decided not to participate). The pilot will run from April 2011 for six months, and will focus on transitioning these services onto the new funding agreement and working with them to identify core outcomes against which monitoring will be undertaken to generate an evidence base of outcomes data with which the council can begin to assess the social value being created by its investment. It is intended that, where the initial migration of these services to an outcomes based monitoring and evaluation system has been achieved, other service providers will be encouraged to adopt the same approach, contributing to this evidence base and providing a means by which the commissioner can begin to assess the full social impact and value of investment into service provision across the third sector. 4

5 The pilot will test the monitoring and reporting tools provided within the model, and provide indication of the challenges facing both the Council and service providers moving towards purchasing outcomes. Forth Sector Development, on behalf of the Scottish Government s SROI project contributed to establishing the pilot through the development of the commissioning model and the evaluation strategy and mechanisms. Tools to assist in outcome collection were offered to the pilot participants as was a method of reporting these back to the council. All participating providers of Older Peoples Services currently funded by West Lothian Council were migrated onto an interim funding agreement. Within this agreement was a requirement to monitor against outcomes identified in the lead up to the pilot. West Lothian Council will now undertake a six month pilot of the evaluation mechanism and assessment of the reporting mechanism which will inform anticipated development of Phases 2 and 3 of the three year investment strategy. 5

6 Contents 1 Introduction West Lothian Council Life Stages Social Return on Investment The SROI project Background Sustainable Commissioning Sustainable Commissioning Model Adapting the SCM for West Lothian Council West Lothian Council Outcomes Purchasing Framework Best Value Best Value and Purchasing Outcomes Incorporating Best Value within Purchasing Outcomes West Lothian Council Monitoring and Evaluation Framework Background and Definitions Outcomes Based Evaluation Outcome Indicators The Outcomes Evaluation Framework Evaluation Cycle West Lothian Pilot: Older Peoples Services Pilot Participants Implementing the Purchasing Framework Stakeholder Analysis Identifying Service Level Outcomes Identifying Outcomes Indicators Identifying Outcome Measurement Tools Reporting Back to the Council Data Collection at the Council Pilot Readiness Next Steps and Recommendations Appendices Appendix 1: Life Stages Outcomes Planning Model Appendix 2: Outcomes Based Funding Agreement Template

7 1 Introduction 1.1 West Lothian Council Located in Scotland s central belt, West Lothian is the 10th largest local authority area in Scotland and has a population of over 167,800. Recent large scale investment in the area has resulted in new employment, new housing and results from the 2001 census show that West Lothian is the youngest and fastest growing population in Scotland. With around 8,000 staff providing more than 100 different services, West Lothian Council is the biggest employer in the area managing part of the central belt bounded by Edinburgh on the East and Falkirk to the West. The Council prides itself on its innovative and proactive approach to delivering local government services and has been recognised for its achievements by winning the UK Council of the Year award in West Lothian Council s mission in providing public services is striving for excellence working with and for our communities. The Council is one of 17 partner organisations in the West Lothian Community Planning Partnership which sets out the strategic direction for West Lothian and defines the key targets and priorities for the area. The new Community Plan Towards 2020 was published in December Life Stages The Life Stages Outcome Planning Programme is a substantial, strategic programme of change led by West Lothian Council working across the Community Planning Partnership (CPP). The overall aim of the programme is to enable the CPP to plan and 7

8 deliver more effective interventions to tackle social inequalities and build successful communities across West Lothian using a Life Stage approach targeting deprived geographical areas. The Life Stages model identifies five priority populations: Early years School Age Young People in Transition Adults of Working Age Older People For each of these priority populations, short, medium and long term outcomes have been identified to underpin service provision by Community Planning partners. The outcomes selected for the Older People Population group can be found in Appendix 1: 1.2 Social Return on Investment Social Return on Investment (SROI) is a framework for measuring a concept of value that is much broader than simply financial. It tells the story of how change is being created by measuring social, environmental and economic outcomes and uses monetary values to represent them. SROI can encompass all types of outcomes but it is based on involving stakeholders in determining which outcomes are relevant. SROI was developed from social accounting and cost benefit analysis, and has a lot in common with other outcomes approaches. However, SROI is distinct from other approaches in that it places a monetary value on outcomes, so that they can be added up and compared with the investment made. This results in a ratio of total benefits (a 8

9 sum of all the outcomes) to total investments. For example, an organisation might have a ratio of 4 of social value created for every 1 spent on its activities. SROI is a practical tool that can help commissioners focus on the impact of their decisions. Its use can help commissioners to demonstrate and improve their contribution to important policy objectives including meeting the needs of end users and other stakeholders The SROI project The SROI Project is a Scottish Government funded programme is designed to develop, promote and support the use of Social Return on Investment across the third sector in Scotland. The work is being undertaken by a consortium of organisations with Forth Sector Development as the lead partner. The Scottish Government and The Office of the Third Sector (OTS) recognise that demonstrating added social, economic and environmental value is important for third sector organisations, their funders, investors and commissioners and is becoming increasingly important for the public and private sectors. The Scottish Government and the OTS have therefore funded complimentary programmes to develop and disseminate SROI. 9

10 2 Background West Lothian Council currently invests approximately 22.5m in social and communitybased services that are provided in partnership through third sector organisations. A significant proportion of this investment into the sector is already based on Best Value service level agreements or contracts, particularly within Social Policy and Education. In the light of a changing national policy context from grant aid to commissioning the Council wants to purchase these services by focusing on outcomes as outlined in the Third Sector Investment Strategy ( ). West Lothian Council will set out its priorities for the purchase of outcomes and allocate a level of investment for those priorities and it will be for suppliers to determine how they respond to these priorities. The council will look to providers to design and deliver sustainable, high quality services that meet council s priorities, address the needs of service users, achieve agreed outcomes, improve efficiency and effectiveness and meet Best Value criteria. West Lothian Council, through Community Regeneration and Community Care, engaged with the Scottish Government s SROI project to develop a model for purchasing outcomes to aid the transference of service provision from a funding to commissioned context. The model aligns with the timescales of the Third Sector Investment Strategy, and is intended to be implemented through three phases over three years. The model also includes an Outcomes Based Evaluation Framework, designed to enable services to support engagement with service providers in the design of services. 10

11 3 Sustainable Commissioning Sustainable Commissioning is an approach to tendering for service provision that aims to ensure Best Value for the buyer by focusing on the purchase of outcomes, rather than outputs. In order to ensure the best possible service is designed the service user and provider are both involved in the commissioning process. This refocusing on outcomes enables the purchaser to capture and measure the wider impacts of its investment. 3.1 Sustainable Commissioning Model The model developed for West Lothian Council has been based on a commissioning process devised by nef (The New Economics Foundation) for the London Borough of Camden. The Sustainable Commissioning Model (SCM) was designed to be outcomes focused and to capture the impacts of commissioned services on service users and on the wider community. The model also monitors the achievement of the purchased outcomes and enables the Council to better understand the longer term impact of it s spend. The SCM contains two key elements: 1. An Outcomes Framework to ensure social, economic and environmental impacts are accounted for in the tendering process and delivery 2. A Valuing Model which tracks social, economic and environmental outcomes and includes a financial savings component 11

12 Together, these two components create an outcomes focussed performance management approach which is embedded throughout the purchasing process through to contract management and evaluation. The Model is built on core principles: Focus on the value of public spending Focus services on outcomes not outputs Links service level outcomes to wider social, economic and environmental outcomes (Community Outcomes) Levels the playing field for smaller providers Embeds co production within services Creates opportunities for co operation and innovation between providers. Camden Sustainable Commissioning Model: Provider left free to describe activities and outputs Commissioner & service user priorities Community strategy & Corporate priorities 1. Activity 2. Output 3. Service level outcomes 4. Community outcomes social economic environmental 5. Value Quantitative Qualitative Monetizeable Where value accrues: To service Community wide Wider Public Sector National outcome frameworks 12

13 Fundamental to the SCM is the reliance on a co production approach to the development of services to the community. Co production involves everyone who is affected by the service in design of the service. Service users, providers and the wider community have input into the service design and help define service level outcomes. The specification issued by the purchaser stipulates a set of service level outcomes (column 3) which are established with service users and commissioners according to the principle of co production. Column 4 describes the relationship between the service level outcomes and the Councils broader priorities (Community Outcomes) drawn from wider policy and strategy. In this way, a real and measurable relationship between service level outcomes and higher level strategic outcomes of the commissioning body can be established. In their response to the Council s requirement suppliers describe the activities they will deliver (column 1) and how this leads to a set of outputs (column 2). The model allows both service level and corporate outcomes to be built into the procurement process from the start and enables the service provider to be innovative when designing services. The commissioner will not stipulate service activity, or outputs from the service. It is up to the service provider to design a service which will achieve service level outcomes while contributing to wider community priorities stipulated in the specification. The second element shows how the Council will monitor the value and benefits created through delivery of this service. Value can be measured in qualitative, quantitative and financial terms. Value accrues to the service, but also across the council, its partners in community and to the wider public sector. 13

14 3.2 Adapting the SCM for West Lothian Council The Camden SCM supports the commissioning and tendering process for the acquisition of services and differs fundamentally from the process developed with West Lothian Council, as at the present time, service provision for older people in the region will still be undertaken to a funding agreement and will not be provided under tendered contract. The SCM has been adapted in the first instance to support a gradual, three phase transition from a funded to commissioned context for services supporting older people in West Lothian. The model developed for West Lothian has drawn from the Community Planning Partnership Single Outcomes Agreement for 2010/11 and the Life Stage Outcomes Planning Model (LSOPM) developed by the Community Partnership. The adapted model incorporates the outcomes identified for Older People in West Lothian under the LSOPM. Service evaluation and evaluation of the Councils investment will focus on the contribution that service providers make towards attainment of these outcomes for this priority group. In the pilot phase participants will be moved to a new funding agreement which will include a requirement to monitor outcomes. All providers will be asked to report back to the council using a stipulated reporting mechanism designed for the pilot. The reporting mechanism is discussed fully in Section 6. As the phased change progresses the Council will be able to move providers onto a fully outcomes based funding agreement and templates can be found in Appendix 2. The Outcomes based Service Funding Agreement (OBSFA) is a two part document. Part 1 is the three year funding proposal from the Council which stipulates the service level outcomes to be achieved and the community outcomes the service should contribute to. 14

15 It will also contain the service specification and required outputs highlighting Best Value quality priorities. All should be stipulated explicitly to enable the service provider clarity of objective for the service. The Service Provider Funding Response (part 2) is completed by the service provider in response to the proposal. The response should detail the service activity to be provided, the outputs of the service, and the required inputs from the Council, as well as note of any additional input provided by the provider that is not accounted for under the funding agreement (i.e. volunteer time). The response will also define service level outcomes against the key stakeholders of the service. These will be few in number and will include the service users as a priority; they will also include any other group that may experience significant outcomes as a result of the service. The service provider should align the service level outcomes, where possible, to appropriate community outcomes, and should also provide two or more indicators for each outcome, against which data can be collected to evidence that these outcomes are being progressed. The OBSFA will be updated each year of a three year agreement as part of the evaluation cycle and the update will be triggered by the commissioner approval of the service improvement plan generated by the provider at the end of each delivery year. The revised agreement will incorporate any progress towards long medium or long term outcomes from the previous year, updating service level outcomes in relation to the previous year achievements or underperformance. The element of co production within the original model is here addressed through a Public Social Partnership (PSP) approach to service development. PSPs are an innovative approach to public service delivery, based upon the public sector and third sector working together to design and deliver excellent public services. Further 15

16 information on PSPs can be found at the Scottish Government s PSP Project website. It is anticipated that a formal PSP can be established between West Lothian Council and third sector providers of Older Peoples Services in the second phase of the transition from funded to commissioned services. Initially, the development of new Outcomesbased Service Funding Agreements will follow a PSP approach to determine service level outcomes and core service components incorporated into the agreement. 3.3 West Lothian Council Outcomes Purchasing Framework Overview of Three Phase Transition from Funding to Commissioning Phase 1 (Year 1) Identification of key Community Outcomes for priority population from Older People Life Stages Model and Single Outcomes Agreement Identification with service providers and service users of service level outcomes appropriate for evaluation of current service provision Migrating Service Providers to new Funding Agreement containing an outcomes monitoring requirement Implementing Evaluation Strategy: Continual monitoring of Service level outcomes and Quarterly Reporting; End of Year evaluation of service delivery against service level outcomes; measurement and reporting of impact of investment by Council into service delivery 16

17 West Lothian Council Outcomes Purchasing Framework: Detail Phase 1 1 Commissioner identifies Community Outcomes at strategic and policy level (Single Outcomes Agreement; Community plan; Life Stages Model) 6 Commissioner and Service Provider consider service level outcomes and identify and agree appropriate outcomes indicators by which these outcomes can be monitored. These are written into the OBSFA. 2 Commissioner groups Community Outcomes into appropriate Service Delivery Lots 7 Service Provider completes OBSFA detailing how outcomes indicator data will be collected. Commissioner and Service Provider agree on OBSFA and funding is awarded. 3 Commissioners engage with individual Service Providers to establish Service Level Outcomes and core service requirements for each Service Delivery Lot 8 Service delivery begins 4 Commissioner develops an Outcomes Based Service Funding Agreement (OBSFA), incorporating service level outcomes and core service requirements. 9 Service monitors delivery against agreed outcomes continually, reporting to commissioner quarterly 5 Service Provider designs a response to the OBSFA detailing service activity and outputs. 10 Commissioner undertakes a year end assessment of service and is in a position to undertake an evaluation of provision against Community Outcomes and measure the impact of investment in the services funded, individually and as a whole. 17

18 Phase 2 (Year 2) Review service level outcomes Migrate service providers to Outcomes based Service Funding Agreement Improvements to service design in line with service level outcomes Introduction of PSP approach to Service Design Move from Individual Service Agreements with Individual Providers to service design based on the delivery of outcomes; Encourage innovative responses to service design by providers and potential for partnership working/joint service delivery under funding agreements Phase 3 (Year 3) Services for Older People in West Lothian put out to tender Third Sector service providers engaged fully in PSP with West Lothian Council to design core service requirements and establish service level outcomes Third Sector service providers fully prepared to submit applications to tender for services and to be awarded service provision under contract 18

19 4 Best Value The Local Government in Scotland Act 2003 established Best Value as a statutory requirement for councils. The act defines Best Value as continuous improvement in the performance of the authority s functions. The objective of Best Value is to ensure that councils deliver better and more responsive public services. It is about: Balancing the quality of services with cost Continuously improving the services provided Being accountable and transparent, by listening and responding to the local community Achieving sustainable development in how the council operates Ensuring equal opportunities in the delivery of services The Accounts Commission launched the audit of Best Value and Community Planning in July The audit work is delivered by Audit Scotland and results in a statutory report by the Controller of Audit to the Accounts Commission. The Controller s report and the Commission s findings are reported in public and have attracted a great deal of media coverage. All Best Value reports received widespread coverage in local media and many received national coverage. This has served to raise the public profile and increase understanding of council performance. 19

20 4.1 Best Value and Purchasing Outcomes Relevant key messages of Best Value that have a direct bearing on West Lothian s move towards purchasing outcomes are: Councils work well with partners at a local level, but need to show what this is achieving Effective performance management is essential to improve services Good councils scrutinise performance, challenge existing ways of doing things and demonstrate value for money Good customer care, listening to and working with local people helps improve services Sustainable development activity Under Best Value Councils have a responsibility to ensure that services are designed and delivered in a way that secures value for money for service users and the wider community. This means that services should be reviewed regularly and this should be seen as a routine part of running the council. The most common problem with service reviews has been a lack of fundamental challenge and creative thinking about how services could be shaped more effectively around the needs of the people who use them. There was strong evidence that reviews had resulted in some service improvements but in only a few cases had they resulted in fundamental change. 20

21 Sustainable development is about councils managing the social, economic and environmental impacts of their activities. Best Value legislation introduced a requirement for councils to contribute to the achievement of sustainable development. This has led councils to consider more explicitly the wider impact of their activities on their communities and the environment. There are many examples across Scotland of council funded projects and initiatives which contribute to sustainable development. In general, however, councils are not good at capturing and measuring the impact. 4.2 Incorporating Best Value within Purchasing Outcomes Ensuring Best Value within West Lothian Council s move towards purchasing outcomes is achievable. The key messages of Best Value resonate within the outcomes purchasing framework. Best Value needs creative thinking to shape services. Ensuring those who engage with and supply services have a say in their design is an essential part of challenging current delivery and leads to new and innovative methods of provision. Stakeholder engagement is a fundamental part of the outcomes purchasing framework. This engagement at service user and service provider level leads to closer working at a local level and achieves better customer care. Working with those directly involved improves services. New monitoring systems looking at the change created by a service allows the Council to report against its strategic outcomes but also to scrutinise the effectiveness of the service. Ongoing performance management is key to Best Value and is essential in improving services. The framework provides an ongoing monitoring system which challenges providers to ensure that outcomes experienced by the client are measured. 21

22 The valuing method of the purchasing outcomes framework also assists in evidencing the wider impact of Council services. Being able to also monitor the social impact of investment in services meets the requirement of sustainable development activity. Each step of the outcomes purchasing framework ensures best value: Outcomes Purchasing Framework Best Value Stakeholder Engagement Listening and responding to the local community, good customer care Design services with service providers Challenge existing ways of doing things, balancing the quality of services with cost Ongoing monitoring system Scrutinise performance, effective performance management, continuously improving the services provided Valuing the wider impact of services Achieving sustainable development 22

23 The Accounts Commission is now introducing future Best Value Audits (BV2) following consultation with councils, consumer organisations and the Scottish Government. BV2 complements wider policy developments in the local government sphere, such as the Concordat and has a greater focus on outcomes. 23

24 5 West Lothian Council Monitoring and Evaluation Framework 5.1 Background and Definitions A key component of the transition from funding to purchasing services against outcomes is the evaluation and monitoring of services: to ensure that agreed service level outcomes are being met; to determine and evidence the link between delivery of service level outcomes and progress for the commissioner towards strategic, or community outcomes; to support service improvement and development of future services; to support future strategic and policy development by collecting data on both service level outcomes and their impact on wider community outcomes and progress towards these Outcomes Based Evaluation Outcomes Based Evaluation (OBE), sometimes known as outcomes based accountability, is a results focused approach to the design and management of service provision in a broad range of contexts. A more detailed definition of the approach is 24

25 available from the Local Government Improvement and Development website 4, and further information of a can be found at The Harvard Family Research website. 5 In brief, OBE is an approach to the planning, commissioning, design and management of services and their assessment that focuses on the results achieved by that service, rather than process and efficiency. Where policy, and therefore community strategy, is focused on progressing change within a community towards stated outcomes, it should also be possible to gear the development and provision of service within the community towards these outcomes. To determine the effectiveness of any intervention or service, the outcomes generated by that service are evaluated in the wider strategic context, and data gathered at service level to evidence progress towards outcomes at both service and community level. 5.2 Outcome Indicators Outcome indicators provide evidence that a certain condition exists or certain results have or have not been achieved. Outcome indicators measure the broader results achieved through the provision of services. These indicators can exist at various levels: community; service; and activity within a service. Community level indicators relate to community outcomes and can help evidence and measure changes for a population, and the wider community within which it sits. Changes in community level indicators are often long term results of the efforts of a number of different services, agencies and initiatives. 4 retrieved February resources/publications series/reaching results/overview of results based accountability components of rba Retrieved February

26 Service level indicators measure results for which a service is responsible; activity indicators measure the results for which a particular aspect of a service is responsible. Service level outcomes indicators are often defined more narrowly than those pertaining to the community as a whole and will rely on data collected directly from service users and immediate stakeholders of a service. Depending on the outcome to be measured, it may be necessary to collect data on one or more indicators to evidence change. This is true particularly in the case of softer outcomes where a direct relationship between an outcome and one indicator may not be clear. Indicators can be both objective and subjective. Objective indicators tend to be numeric, and can relate to outputs of an activity, or represent an evidential change in the behaviour of a service user. For example, if an outcome for an employability service is that more service users find employment, a sensible indicator for this outcome would be an increase in the number of service users to find employment. Subjective indicators tend to be expressed by the service users as changes in their feelings or thoughts on something. For example, a softer outcome for an employability service might be increased social confidence of service users. Finding objective indicators to support this outcome will be challenging; however, employing a subjective and a more objective, though less direct indicator could provide a measurement of this outcome. Indicators that might work for the example might be the number of service users reporting an increase in confidence (subjective; reported by the service user), and the number of service users who now regularly attend a social activity or club (objective; demonstrates an observable change in behaviour). In the case of the West Lothian framework, a precedent for the selection of indicators against which to measure community outcomes already exists within the Single Outcome Agreement and Life Stages planning models. Identifying indicators by which 26

27 service level outcomes can be measured will happen once service level outcomes have been identified, and the indicators selected will be appropriate for these As part of the West Lothian Pilot a framework for monitoring and evaluating services against set service level and community outcomes was developed. The framework includes an evaluation cycle which encourages continual improvement at service level regarding progressing service level outcomes and enables the Council to assess changing need at delivery level which may influence community planning and policy. 5.3 The Outcomes Evaluation Framework The framework aims to provide a mechanism by which West Lothian Council can: Evaluate commissioned services contribution to progressing community level outcomes Monitor service management and delivery against service level outcomes and enable continual improvement of service delivery in this respect Provide an evidence base generated at service level that supports development of community policy and strategy Enable a better understanding of the wider social impact of investment in services and it s value to the community 27

28 The Framework is designed to assist the Council in answering the following questions: How is the service performing against service level outcomes? How are the service level outcomes contributing towards higher level outcomes, and how can this be evidenced? What else is happening for service users/populations? What is the value of the service level impacts for all stakeholders, at service and community level? The framework consists of an Evaluation Cycle, designed to run across the three phases of the transition, an Evaluation and Monitoring Model and suggested tools for the collection of data at service level relating to evidencing progress towards service level outcomes. 5.4 Evaluation Cycle The Evaluation Cycle will take place over the three years that a service funding agreement or contract is in place with a service provider. Triggers have been built in to enable both the Council and the service provider to initiate processes as appropriate to improve, redesign or replace services while being able to report on progress towards service level outcomes and strategic higher level outcomes. The cycle is concerned with capturing and processing data collected against service level outcomes and with aiding service providers and commissioners to better understand the full social impact of their investment into a service and the cumulative impacts of investment into a spread of services. 28

29 The key points of the Evaluation Cycle are illustrated in the following tables. The Evaluation Cycle suggested aims to answer the following questions for the Council and service providers, as illustrated in Table 1 Tables 2 4 show the Evaluation Cycle across three years, in line with the suggested implementation of the purchasing framework. The Cycle can be condensed and triggers are flexible; in particular, the trigger in year 3 for the commissioner to assess whether the service should be re commissioned, redesigned or replaced could fall at any point through year 2 or 3. 29

30 Table 1: Evaluation Cycle Answering Questions Commissioner Question about service Service Provider Are improvements to delivery required? Should service be recommissioned, redesigned or replaced? How is the service performing against Service level Outcomes? How can we improve service delivery? Distance travelled towards higher level outcomes Are the Outcomes being measured appropriate? How are the Service Level Outcomes contributing to Higher Level Outcomes? Where do we fit in? Next Steps planning how can this inform future strategy/community outcomes? Can this feed into policy? What Else is happening for service users/priority population? Next Steps planning how can this inform future service design? What is the cumulative social value generated from our investment into service for this priority population? Can this feed into policy? What is the VALUE of the service impact for all Stakeholders? What is the real VALUE of investment into our activity? How can we increase our social impact? Can this feed into Strategy? 30

31 Table 2: Evaluation Cycle Year 1 Commissioner Timeline Service Provider April Service Delivery Begins Begin Outcomes Monitoring from Stakeholders May Outcomes Monitoring Collate Quarterly Outcomes Data June Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring Feedback to Service Provider Progress to Higher Level Outcomes July Outcomes Monitoring Aug Outcomes Monitoring Collate Quarterly Outcomes Data Sept Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring 31

32 Table 2 Continued: Evaluation Cycle Year 1 Commissioner Timeline Service Provider Feedback to Service Provider Progress to Higher Level Outcomes Oct Outcomes Monitoring Service Assessment to Inform Service Improvement Plan Collate Quarterly Outcomes Data Response to Service Improvement Plan/Approve? Feedback to Service Provider Progress to Higher Level Outcomes Nov Dec Jan Outcomes Monitoring Develop Service Improvement Plan Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring Outcomes Monitoring Feb Outcomes Monitoring Collate Quarterly Outcomes Data March Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring End of Year Reporting End of Year Reporting Distance Travelled, Community Outcomes Measure and Value cumulative social value generated cross service/investment April End of Year Reporting Service Performance Distance Travelled, Service Level Outcomes Measure and Value social impact of service 32

33 Table 3: Evaluation Cycle Year 2 Commissioner Timeline Service Provider April May Initiate Service Improvement Plan Begin Outcomes Monitoring from Stakeholders Outcomes Monitoring Collate Quarterly Outcomes Data June Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring Feedback to Service Provider July Outcomes Monitoring Progress to Higher Level Outcomes Aug Outcomes Monitoring Collate Quarterly Outcomes Data Sept Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring 33

34 Table 3 Continued: Evaluation Cycle Year 2 Commissioner Timeline Service Provider Feedback to Service Provider Progress to Higher Level Outcomes Oct Outcomes Monitoring Service Assessment to Inform Service Improvement Plan Collate Quarterly Outcomes Data Response to Service Improvement Plan/Approve? Feedback to Service Provider Progress to Higher Level Outcomes Nov Dec Jan Feb Outcomes Monitoring Develop Service Improvement Plan Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring Outcomes Monitoring Outcomes Monitoring Collate Quarterly Outcomes Data March Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring End of Year Reporting End of Year Reporting Distance Travelled, Community Outcomes Measure and Value cumulative social value generated cross service/investment April End of Year Reporting Service Performance Distance Travelled, Service Level Outcomes Measure and Value social impact of service 34

35 Table 3: Evaluation Cycle Year 2 Commissioner Timeline Service Provider Assess Service Need and Service Provision: Decision re commission service, re design service, or replace service? April May Initiate Service Improvement Plan Begin Outcomes Monitoring from Stakeholders Outcomes Monitoring Collate Quarterly Outcomes Data June Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring Feedback to Service Provider Progress to Higher Level Outcomes July Outcomes Monitoring Aug Outcomes Monitoring Collate Quarterly Outcomes Data Sept Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring 35

36 Table 3 Continued: Evaluation Cycle Year 2 Commissioner Timeline Service Provider Feedback to Service Provider Progress to Higher Level Outcomes Oct Outcomes Monitoring Service Assessment to Inform Service Improvement Plan Collate Quarterly Outcomes Data Response to Service Improvement Plan/Approve? Feedback to Service Provider Progress to Higher Level Outcomes Nov Dec Jan Feb Outcomes Monitoring Develop Service Improvement Plan Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring Outcomes Monitoring Outcomes Monitoring Collate Quarterly Outcomes Data Mar Outcomes Monitoring Quarterly Reporting on Outcomes Monitoring End of Year Reporting End of Year Reporting Distance Travelled, Community Outcomes Measure and Value cumulative social value generated cross service/investment April End of Year Reporting Service Performance Distance Travelled, Service Level Outcomes Measure and Value social impact of service End of Service Reporting Evaluate Social Impact of Service / Cumulative Impact of Investment 36

37 The Outcomes Evaluation and Monitoring Framework provides a mechanism by which the outcomes experienced by key stakeholders in a service can be evidenced and measured. It also enables The Council to understand how an investment in a service can result in progress towards higher level, strategic or community outcomes, and to evidence the impact of spend in respect of these. Services will continually monitor delivery against service level outcomes, collecting outcomes data from service users and other agreed key stakeholders, which will be collated and reported to the commissioner on a quarterly basis. The Council can use this information to evaluate the performance of the service at specified points in the cycle; a six month Service Assessment and an End of Year Assessment. These evaluations can provide the service provider with input to assist with continual service improvement and direction regarding a six monthly service improvement plan, and provide the commissioner with a basis for decisions around continuing or redesigning service provision at the end of each cycle.. 37

38 6 West Lothian Pilot: Older Peoples Services West Lothian Council will run a pilot from April 2011 April 2012 involving providers of services for Older People in the community. The purposes of the pilot are to: Migrate service providers onto an outcomes based funding agreement, incorporating service level outcomes for monitoring and evaluation Support service providers to establish continual monitoring against service level outcomes Support quarterly reporting on service level outcomes Generate baseline data for service provision to older people population in West Lothian Test evaluation and reporting process at both service and Council level; How able are services to capture and report on their outcomes? How useful is this data for the commissioner to evidence progress towards community outcomes? Does this monitoring and reporting structure enable evaluation of service provision against community outcomes? Can baseline data support development of services for later commissioning? Provide initial outcomes data to enable later valuation of impacts of individual services, and service impact cumulatively, against the investment into the service and spread of services. 38

39 6.1 Pilot Participants There are currently nine service providers funded to support the older people of West Lothian. Five of these are day centres providing varying support to service users, including service users requiring intensive support. Four are smaller, volunteer led organisations providing low intensity support to a smaller number of service users in a community setting. Eight of the providers agreed to participate in the pilot, the ninth declined due to the fact that the organisation is based in Edinburgh and receives less than 1,000 per annum from the council. The eight participating organisations are: Answer Day Care Service The Answer Day Care Service provides dementia day care for older people living in the west of West Lothian. Answer provides services for 24 people with dementia and their carers three days per week. MOOD Project MOOD is a voluntary organisation covering the west of West Lothian identifying and working with older people who have or may develop depression. Rosebery Centre The Rosebery Centre offers specialist activities, support and advice for 45 people with all types of dementia and all age groups. St Michael s Day Care Centre St Michael s Day Care Centre provides social therapeutic activities and support for up to 16 frail elderly service users daily. 39

40 6.2 Implementing the Purchasing Framework The Framework designed for West Lothian Council would be piloted from April 2011 but before this could be initiated a number of activities with the local authority and the providers of older people services had to take place. Outcomes for the pilot to measure against had to be defined and methods of capturing data to evidence these had to be implemented. Finally a way of reporting progress back to the Council was developed. The following sections look at the preliminary work undertaken before launching the pilot. At this stage the Social Impact Scotland (SIS) website was utilised to help in defining outcomes and indicators. The website is the legacy of the Scottish Government s SROI project and has a number of useful resources which could be used by other purchasers in replicating this work. As an example the work undertaken with St Michaels day care centre is provided at the end of this section Stakeholder Analysis Working with the service providers the key stakeholders for the pilot were determined. All eight participants agreed that, as part of the pilot, they would work with the following stakeholders: Older people accessing the services Carers of older people Volunteers For each of these stakeholders outcomes would be determined that the providers could monitor their progress against. 40

41 6.2.2 Identifying Service Level Outcomes Again working with the providers, outcomes for each of the key stakeholders were determined. As this work progressed it became clear that shared outcomes could be developed where service providers are working towards achieving collective change. These outcomes will be tested as part of the pilot and as the service providers engage with their stakeholders they may find that these outcomes are not being achieved or different changes are occurring. This will form an important part of the pilot helping both the Council and service providers understand exactly what is happening and what changes are being experienced. Service providers were reassured that this monitoring of outcomes is not, at this early stage, designed to change their services but instead to determine a clear picture of what is happening Identifying Outcomes Indicators With the outcomes defined indicators were determined with service providers. Where possible a subjective and objective indicator was used to measure the progress against the desired outcome. Indicators were based on the information offered by the service providers and from the indicator bank on the SIS website Identifying Outcome Measurement Tools With both the service level outcomes and their indicators determined thought had to be given to how this information would be collected. There are many different ways of collecting outcomes data and a review was carried out to identify methods suitable for the requirements of the pilot. Outcomes monitoring tools would need to be robust enough for the Council to monitor services against them and to this end previously 41

42 evaluated and validated methodologies were sought. Tools would also have to be complimentary to service providers existing data capture methods. All involved were keen not to burden service providers with extra and unnecessary work and outcomes measurement tools would have to reflect this. Finally the tools would have to be suitable for the client group and be easily administered. Research reports were accessed looking at available tools and those highlighted on the SIS website were considered. A menu of potential tools was created and each supplier was given the opportunity to determine which would be suitable for use within their services. The following tools were agreed upon for use: Older People s Outcomes Star Short Version Perceived Stress Scale Caregiver Strain Index Employability Skills Wheel These tools are useful within a broad range of stakeholders, however, some of the providers engage with older people with varying levels of dementia. In these cases more specialised tools were required designed specifically for this client group. The following methodologies were agreed upon: Quality of Life AD DEMQOL QUALID All the tools identified have been tested and validated but are also easy to administer each requiring less than 15 minutes of the service users time. 42