Measuring your Impact This briefing reflects the good prac ce showcased at our recent Measuring your Impact event in September It looks at the w

Transcription

1 Measuring your Impact Good prac ce briefing

2 Measuring your Impact This briefing reflects the good prac ce showcased at our recent Measuring your Impact event in September It looks at the ways different projects can measure the effects of what they do, and how these results can be captured. A range of different case studies and good prac ce examples are highlighted which show the importance not only of measuring hard outcomes, but also why so er achievements are as important and can also be recorded. In harsh economic mes such as these, it is even more important that organisa ons can provide evidence to funders and other stakeholders showing both the value of their ac vi es, and the impact their work is having. Some of the models men oned in this briefing can be used to assign a monetary value to a par cular ac vity, measuring the mul plier effect which can add value to local economies. Increasingly Welsh Government (WG) sees the applica on of an outcome based approach through systems such as the Results Based Accountability (RBA) method as an important tool in establishing a line of sight from results back to means. Recent examples include the Housing Department's Self Assessment process and the proposed approach to delivering the new Communi es First programme. i2i examined these and related issues in a series of regional events across Wales during September 2011 a ended by over 100 people. The Measuring Your Impact seminars showcased best prac ce from hous- ing associa ons, local authori es and third sector organisa ons where a variety of methodologies were examined, as well as case studies from different disciplines such as financial inclusion, health and regen- era on. It is important to remember one basic rule, that we are all in the business of producing outcomes through our ac vi es - it is rela vely easy to set up systems which measure outputs i.e. the specific out- comes which relate to our project, but it requires a more structured approach to measure the things that happen in the longer term as a result of achieving those outputs. Evidencing that change is the true mark of a successful project or ac vity in our communi es - but in doing so it is vital to fit the tool around your outcomes - not the other way around.

3 Methodology The Crew Toolkit Background informa on The Centre for Regenera on Excellence in Wales (CREW) was established in 2010 with the aim of promo ng integrated sustainable regenera on prac ce in Wales. CREW iden fies disseminates and promotes good prac ce developing cross-disciplinary skills for regenera on prac ce for regenera on prac oners. The Research The CREW toolkit is a piece of research funded by the Joseph Rowntree Founda on which considers how wider impacts can be measured based on the experiences in different parts of the UK. Some of the key findings underlined the importance of management prac ce and the importance of corporate backing to what is being delivered. Barriers to successful measurement include: the skills of the researchers, and the corporate buy in of the organisa on If the organisa on is not commi ed then the project will not be able to show clear outcomes. This must be thought about at the earliest possible stage in the project as there is no point in ge ng half way and realising that there is no corporate "buy in" for the work. Key points How you measure your impact must be built into the design of the tool. Delivery must be measured and monitored at all stages to be successful. Resources for the process are a key requirement and many projects undervalue this right at the start. CREW advise that most projects should allocate 6 7 % of project cost for measuring and evalua on. Context is important as nothing we do is in isola on- it's important to a ribute and account not only a project's own achievements but what others have achieved also.

4 The CREW toolkit itself recommends a three pronged approach considering Atmosphere, Landscape and Horizons - the ALH model. Atmosphere refers to the feel of a place If someone told you that you had to live in a certain place tomorrow how would you feel about it? If you walked around the community and asked people who were there how they felt while out and about, what would they say? Would it be be er or worse than 10 years ago? Is it safe, dangerous, welcoming or hos le? Is the community celebra ng itself or is it concerned? Landscape access the physical nature of the neighbourhood What is its layout like? Quality and ap- pearance of the housing? Are gardens, green spaces, and quality public spaces available? Are there play and sports spaces, community buildings, and things that improve the quality of people s lives? Is the community well connected to the external world by road, public transport and the internet? Horizon refers to the social horizon which residents of communi es feel How far do residents look outside their communi es? Do people feel equipped to travel outside their area (socially and physically)? Do young people live en rely in their family and peer culture or are there social and cultural experiences which challenge and change these ideas?

5 How it works The toolkit itself stresses the need for both qualita ve and quan ta ve data. The inclusion of na onally approved sta s cs is important but needs to be counter balanced by talking to people whose lives have been impacted- the involvement of residents in this regard is cri cal. The processes of se ng a base line, providing a medium term and final evalua on are also important but it is important to use the right lan- guage Part of the research also included a study of a group of indicators and the final report will highlights 22 indicators which passed the test in terms of spa al and me series availability. It was important to look for data that could be con nually updated. Hence census data may not the best approach as it becomes outdated by the me the results are publicly avaliable, however it can be helpful and a good source of informa on if there is a need to measure longer term impacts. The research also highlighted the im- portance of Community Surveys and audits, a good example being the New Deal for Communi es pro- gramme in England which undertook MORI polls every two years. Techniques community surveys, focus groups, community panels, event based consulta ons and interview with long term residents, community leaders, elected members and on the ground workers All of these examples provide useful evidence. Whatever the approach- be it sta s cal capture or qualita ve evalua on, it is important to meas ure the result through Triangula on with at least THREE data sources to ensure findings were accurate. The repor ng framework needs to be iden fied at the project outset and not (as commonly happens) at the end. In so doing it needs to use the atmosphere, landscape, and horizon structure and can then in- form programme delivery and ac on planning if used at the beginning of a project. The detailed report is expected to be launched later in the year. Contact : Dave Adamson: dave.adamson@regenwales.org

6 Social Return on Investment (SROI) "SROI is a framework for measuring and accoun ng for this much broader concept of value; it seeks to reduce inequality and environmental degrada- on and improve wellbeing by incorpora ng social, environmental and economic costs and benefits. SROI Guide, 2009 Primarily SROI has been developed to measure the VALUE of ac vity, logging the CHANGE that has taken place. It relies on qualita ve, quan ta ve and financial data and can be accredited and used to illustrate the financial, social and organisa onal worth of an ac vity SROI can help organisa ons understand, manage and communicate the value of their work in a clear and consistent way to customers, beneficiaries and funders. It can help to manage risks and iden fy opportu- ni es and raise finance. It will throw up poten al improvements to services, informa on systems and the way a businesses governs itself. Because SROI is built on principles, it is very flexible. Different organisa ons create value in many differ- ent ways. A consistent approach to understanding value means that you can communicate clearly in a credible way where and how you create it. SROI Principles: Involve stakeholders Understand what changes Value the things that ma er Only include what is material DON T over claim Be transparent Verify the result Key Stages Establish the scope and the stakeholders, and To map the outcomes that have resulted from the ac vity Outcomes are evidenced and give a value through the development of a financial proxy this is the most difficult part. The impact of the ac vity is then established - how much would have happened without the ac vity? The SROI is then calculated and applied it can also be accredited as a means of cross checking the result.

7 Issues to consider when calcula ng the SROI Deadweight? how much would have occurred without your ac vity? Displacement? How much of your ac vity displaced other programmes/ac vi es? A ribu on? how much of the outcome was caused by other people s contribu ons? Drop off? don t keep dining out on your ac vity forever Impact (quan ty x financial proxy deadweight, displacement and a ribu on = SROI) SROI Map Stage 1 & 2 SROI Map Stage 2 & 3 Top ps Keep it simple one small ac vity to start Resource properly Don t get bogged down by all the other interconnected ac vi es Don t oversell your impact Support the researcher Get it accredited

8 Value Wales Community Benefits Tool Designed to be used by contract / project managers and primary contractors to measure benefits of Welsh public sector contracts The Welsh Community benefits tool developed by Value Wales is designed to be used by contract / project managers and primary contractors to measure community benefits of Welsh public sector contracts. It has been developed through a series of pilot studies in 3 Pathfinder areas and the publica on Community Benefits-delivering maximum value for the Welsh pound (2010). In many ways the tool can be seen as a prac cal applica on of an SROI approach to procurement in Wales, which assigns monetary values to ac vi es using financial proxies and assump ons tailored to the Welsh economy The tool is an excel spread sheet which captures a standard set of data for each procurement project. The tool has been developed primarily to examine the community benefits derived from construc on and in- frastructure contracts. For contracts under a year, the tool is intended for comple on at the end of the contract. For contracts longer than one year, the tool is intended for annual comple on, enabling the iden fica on of future opportuni es. All informa on collected should be rela ng to contract related ac- vity only, rather than overall company opera ons. It is possible to use the tool for other contract areas, by merely filling in the enterprise and employment worksheets. The tool captures the socio-economic value of Community Benefits interven ons by: Using minimal raw data Being easily understood and applied Using a common robust methodology Having a Welsh focus Measures and records factors such as : Training, Educa on, Use of Welsh companies in delivering contracts (SMEs, Third Sector organisa ons and Supported Businesses), Cost avoidance on Waste to landfill, Contribu on to reducing Climate Change gas emissions and the value to local communi es

9 The tool is easily completed by obtaining figures from the contractor & contract itself. It has 6 worksheets for data collec on, with each sec on of the worksheet clearly explained as to which informa on should be entered at that point. The tool will also provide you with a comple on figure which demonstrates the local mul plier effect that your investment has created. Example of Populated Spread sheet The local mul plier shows the rela ve impact showing for every 1 spent on this contract, the following amount of money has been reinvested in the Welsh economy. This figure is based on whether the contract was spent on a contractor based in Wales and then how much that contractor, regardless of loca- on, re-spent the contract income. These two figures are added together and divided by the contract value to create a ra o showing how much was spent on the contract to create the added income for the Welsh Economy. The tool is a rela vely easy way of calcula ng the mul plier effect from raw data which is easily collated. However at present the tool only measures how much of the contract is spent in Wales. Many organisa- ons need to demonstrate economic value more locally than this and may need to adapt the tool further to provide the results in the format that they require. Contact: Lisa Thomas-Lewis Lisa.thomas-lewis@wales.gsi.gov.uk

10 Financial Inclusion Case Study Bron Afon Community Housing Bron Afon Community Housing is a community based housing associa on managing and improving over 8000 homes in Torfaen. It has iden fied financial inclusion as a priority and provides face to face financial advice through a dedicated provision and advice worker. It is part of the wider strategy to assist tenants in financial difficul es which includes a team of energy advisers helping tenants with fuel bills. The ac vity has been running 2 years. Hard facts: Nearly 500 Clients visited Over 50,000 p.a. in unclaimed Benefits secured. 8 Clients saved from imminent evic on Over 80,000 of debt wri en off Over 100,000 of debt to be wri en off imminently Over 35 referrals of complex cases to partner agencies Over 40 serious mul debt arrangements with creditors made Measuring these hard outcomes are key, but the so outcomes are also important. One case resulted in a saving of and a debt order being stopped, but the other real impact was the increase in the tenant's self-confidence in dealing with their life. Another tenant had a very large debt wri en off due to an individual bankruptcy, but an important change was the fact that that person no longer hid when there was someone at the door and they felt that they had got their life back. So Outcomes: Case studies Recording the difference it makes to people s lives capturing quotes and more recently video Pu ng a story behind the facts and figures

11 Be er IT systems are required to be able to track the outcomes that the service is delivering to people, and linking it to the housing management systems. So er outcomes are recorded by on- going contact with clients, and tracking surveys. Recording the experience via video and capturing quotes was also useful, although in so doing one always had to be mindful about the sensi vity of disclosing people's financial situa on A key ques on for the future focused on being more proac ve and flagging up people who are vulnerable and who will get into debt in the future before they reached that stage. Bron Afon have started working with people before they take up a tenancy, focusing par cularly on working with young people who are taking on a home for the first me. Other next steps include plans to train up other front line staff to support tenants around money and benefits advice. Contact: Cath Stenson Cath.Stenson@bronafon.org.uk Steve Nelson Steve.Nelson@bronafon.org.uk

12 Health Impact Study Case Study Carmarthenshire County Council Carmarthenshire County Council is undertaking a strategic long-term longitudinal study on the effect that the improvements on its housing stock will have on the health of the popula on who live there. How? The local authority developed a pilot study with Swansea and Cardiff Universi es in order to get a base line regarding the health of tenants comparing tenant health with current property condi on A survey was undertaken by a market research company and a Health ques onnaire developed along with face to face interviews. The ini al ques ons that were asked were based around the types of problems people face as a result of poor housing, these were found to mainly be respiratory problems, men- tal health and well being, food hygiene issues, health and safety, and energy efficiency causing excess winter deaths. As work progressed the authority decided to find out the different circumstances that people live in versus the illnesses they have. From these responses the local authority began to look at the proper es which had already been improved to WHQS against those that had not. They found a striking correla on between improvement works and improved mental health. The Feeling Fine Healthier Homes Health Im- pact Study showed that: 21% of tenants of unimproved proper es reported being treated for depression, compared with only 7% of the tenants living in completed proper es. Improvement in Respiratory Infec on 67% of tenants living in completed proper es reported they were free from respiratory infec on like coughs and colds compared with 52% in unimproved proper es. Less people going to their GP 23% of respondents in unimproved proper es visited their GP more than four mes in a three month period compared with 14% in completed proper es.

13 Using this approach an annual survey of tenants health will be carried out over the next four years to establish if there were any long term improvements. The first stage of this four year plan will begin in February The study has collected baseline data between so that it can con nue the project over the next 25 years which will be parallel to the WHQS improvement works that are being carried out. Alongside this, an evalua on of the Suppor ng People programme in Carmar- thenshire has been carried out iden fying that: Suppor ng People funding totalling 6.5m This realises over 16m public sector savings Health ( 657k) Social Care ( 8.5m) Residen al Care ( 11m) Community Safety ( 3.4m) Housing ( 1.4m) For every 1 spent, 2.30 is saved.

14 Next steps: Annual Survey over the next 4 years to track changes to health Parallel Work-Accessing actual health data Assessing the cost benefits to health services from housing improvement Understanding the wider benefits of the health impact study which may include educa onal a ainment and economic benefits and also the so er outcomes which could include sense of community and place, well being benefits, social care benefits and happiness. The report of the health impact study is available on the Carmarthenshire website. h p:// Contact: Robin Staines rstaines@carmarthehshire.gov.uk Tel: Housing Website: h p://

15 Measuring the Wider Value of Housing Investment Case Study Cartrefi Conwy & CREST waste collec ons Cartrefi Conwy sees the self assessment process as an important business tool to ensure they are mee ng the needs of their tenants, and also as an important way of providing evidence to ensure it is making a difference. They use an outcome matrix which measures financial viability and management, service delivery & overall corporate governance which in turn measures effort, the effect of said efforts and the difference this has made to the business and the stakeholders. Cartrefi Conwy was an enthusias c adopter of i2i's Can Do Toolkit approach to procurement meaning Targeted Recruitment & Training and associated ac vi es were adopted very early in the life of the organisa on. Their main contractor delivering internal and external improvement programmes have a local labour base of 96% in North Wales, with 5 appren ceships progressing to NVQ level 2. They have also linked closely with the local college to provide 6 pathways to appren ceships opportuni es and the Future Jobs programme to provide work experience for 4 administra on staff. Their other contractors are also delivering high percentages (over 80%) of North Wales workers within the labour forces as well as sourcing materials from local suppliers. Targeted Recruitment and Training Main contractor local labour base - 96% in North Wales 5 appren ceships progressing to NVQ Level 2 6 Pathways to appren ceships 4 administra on staff from the back to work programme Other contractors also delivering high local labour (over 80%) of North Wales workers within labour force & sourcing materials from local suppliers

16 Following the ethos of the Can Do toolkit lead to a par cularly innova ve partnership with Crest Coopera ve, a social enterprise based in Llandudno Junc on. Crest has 3 different enterprise arms for recycling and working with unemployed people - especially those groups facing challenges in entering the labour market e.g. disabled people and offenders. This partnership has delivered 270 work experience placements, with 7 of these moving onto full me employment with the main contractor- G Purchase Construc on Ltd. Due to the segrega on at source, the project was also delivering impressive outputs in waste management with 98% of waste being recycled or reused between Knowledge transfer partnership programme accessed in order to measure impact of environmental improvements to: Develop and establish an assessment system that is consistent, comprehensive and non-discriminatory for iden fying poten al sites. Create a tool that Cartrefi Conwy can use to measure the impact of environmental improvements. Ensure that tenants play key role in the projects success & sustainability This has lead to the crea on of a ra ng system which: Enables comparison between sites. Gives a maximum ra ng score that reflects individual criterion significance. Each criterion has its own calcula on to achieve a ra ng score. We have benefited by being able to: Plan efficient, crea ve & sustainable management of external environment Develop an environmental strategy and best prac ce approach to environmental improvements A ain the WHQS Environmental Standard Contact: Gwynne Jones Cartrefi Conwy gwynne.jones@cartreficonwy.org Rod Williams - CREST rod@crestcoopera ve.co.uk Tel: Tel:

17 Measuring Social Impact and Background Return on Investment Case Study - RCT Homes RCT Homes was established December 2007 and is the largest social landlord with over 10,300 rented homes, and over 600 leaseholders. RCT Homes has been successful in engaging tenants to become members of the first Community Mutual in Wales. To date there are 4,700 members with a further 860 awai ng membership approval. RCT Homes operate a group structure, incorpora ng Meadow Prospect (charity Regenera on Company), GrEW (Social Enterprise including Young Wales) and Homeforce (gas servicing). 86 promises were made to tenants in its offer document, and these promises were proac vely used as levers for crea ng added value in its regenera on ambi ons. This approach has been vital given the context of depriva on in which RCT Homes operates. Highlights Pioneer of Targeted Recruitment and Training in Wales Over 1100 training opportuni es across the group created to date 80 of 308 new jobs created were for new entrants to labour market 16 supported by JobMatch 17 trainee posts Measuring Impact RCT Homes has led and par cipated in some recent innova ve ini a ves which seek to evaluate the success and monitor outcomes. These include; the Value Wales Community Benefits Toolkit, Community Impact Tracking and RCT Homes/Hyder Open Space Audit. Value Wales Community Benefit Toolkit shows RCT Homes investment on one project had delivered a mul plier effect of 2.36 for every 1 of spend in the local community

18 The Open Spaces audit is based on 3 key areas of considera on; namely Landscape (physical elements), Horizon (community) and Atmosphere (the overall result of Horizon and landscape) measured against a backdrop of stakeholder focussed indicators. This approach takes forward and further develops the approach and methodology originally set out by Professor Dave Adamson. RCT Homes sought to understand the impact of its social enterprise subsidiary Grow Enterprise Wales. A number of different methods were reviewed including PQSSO,SROI, and SIMPLE (Social IMPact on Local Economies). Whilst all methods had merit, it was the SIMPLE method that offered the most flexible approach for whole organisa on assessment. Framework to assess what GrEW is doing against its mission Help us plan for change Help us assess and develop rela onships with stakeholders Support the development of local knowledge Provide a basis for other social impact assessment approaches

19 Evidence for use in marke ng materials and funding bids This approach allowed RCT Homes to build up its evidence base and understand the priori es, opportuni es and constraints of the organisa on before establishing so# and hard outcome measures and impact indicators. RCT Homes chose impact indicators based upon economic, environmental and social factors. Some of these have been able to be mone sed, for example the amount of saved to the public purse as a result of providing in work training and the posi ve impact on future employability. The approach developed within the company has held to make the whole organisa on think differently about some of these issues because the assessment was happening first before projects were undertaken. RCT Homes plan to undertake the SIA once more in 2012 and compare against the base lines and impacts established in Considera on is also to be given to undertaking an SROI on individual products and ac vi es.

20 Contact Rod Keeble - rodk@rcthomes.co.uk Lisa Balfe - lisab2@rcthomes.co.uk

21 Thank you to all of the organisa ons who presented at the Measuring your Impact events in September:

22