Measuring the social value of outdoor learning by pupils at risk of becoming NEET

Transcription

1 Measuring the social value of outdoor learning by pupils at risk of becoming NEET One of the aims of the ACEWild project was to research, develop or adapt a cost-benefit TOOL to measure financial savings from preventing NEET (not in employment, education and training). This short report describes and critically reviews this part of the project. Our ACEWild approach aims to change the mindset and behaviour of pupils at risk of becoming NEET by providing and jointly reflecting on their outdoor learning experiences. We believe that the social value of outdoor learning can help to avoid young people being NEET. The ACEWild approach uses the outdoors in the school grounds, gardens, woodlands and local enterprises to change the behaviour of the pupils and to monitor this change with the pupils, and with the partnership of specialist local enterprises and educators. This selective and alternative education for pupils at risk of becoming NEET is more costly than normal, more academic, class-based education. It includes the additional staff from the school or partner organisations, the transport to enterprises, the equipment, practical space and resources. We believe it is more cost effective overall to spend money on this ACEWild approach to outdoor learning while the young people are at school than when they leave school and are not in education, employment or training. However we need some more solid evidence for funders, educators and parents to prove the value of this more selective and alternative education approach. It is very difficult to look at these costs holistically over the lifetime of the young person in terms of costs to society, the funders, schools, families, and the young person. However it is now expected that this cost effectiveness should be judged using one of a number of social accounting methods. The NEET problem in Europe The problem is that over 15% of year old young people in the EU are NEET not in education,employment or training.

2 In the UK it is 12%. In the Netherlands it is 5% - the lowest in Europe. However this figure is rising across Europe. It's solution is a priority for the Erasmus + programme. The average individual life-time public finance cost of NEET in the UK was estimated as 56K (Univ of York 2009). These costs are to the public services of Government, local authority social services and housing, health services, police, probation and prison services. However the average individual cost of losses to the economy, to individuals and their families was estimated as 104K. However there are also non-economic costs which are social and political. The loss to the economy varies considerably across the EU but is more than 1% of its GDP.

3 Teachers already know many of the risk factors of becoming NEET. Schools can reduce this risk by selectively educating those pupils that are likely to become NEET. Social Return on Investment (SROI) One of the most popular social accounting methods, but not for the education sector, is Social Return on Investment (SROI). It is an analytic tool for measuring and accounting for a much broader concept of value. It incorporates social, environmental and economic costs and benefits into decision making, providing a fuller picture of how social value is created. It is able to assign a monetary figure to the social and environmental value which is created. It calculates the costs of the investment (eg. 1 for the outdoor learning approach/pupil) and the value of the social return (eg. 10 saved over the lifetime of the young person in not being NEET). Rightly educators, like conservationists, would say that we cannot put a financial value on everything. But the process of finding this ratio is more important that any spectacular ratio itself. Of course there are so many unknowns and variables. SROI explores the process of change in this case the pupil's personal, social, environmental and entrepreneurial behaviour, mindset, attitudes and values. It identifies the change in terms of: Inputs (or Investment) Activity Outputs Outcomes (or social return) Impact The inputs are based on the main Stakeholders in the process of change. Identifying the key outcomes is the key to SROI. For each stakeholder (e.g. teachers, local partners, pupils) we look at: Inputs - resources invested in the activity Outputs the description of the activity e.g. 20 young people employed Outcomes - changes to people resulting from the activity, i.e., a new job, increased income, improved stability in life, improved quality of life Indicators of change how do we know change has happened Quantities of change how many of the stakeholder group experience change Financial proxies how we value the change Impact = Quantities times proxies, less reductions to reflect that some change happens anyway and some change is created by other factors.

4 The ACEWild SROI Stakeholders We identified five stakeholders in priority order: 1. Pupils 2. Teachers 3. Funders 4. Parents,Carers 5. Local Educator and Enterprise Partners (LEEPs) Outcomes For the ACEWild approach we have identified up to several outcomes, again in priority order. However, as described earlier, the stakeholders have not been consulted at the beginning or end of the project. So these outcomes have been selected by the project team, and modified after consultation with the SROI Network. Pupils Linked to well-being and lifestyle 1. More able to deal with issues which cause harm in their own lives (stress, mental illness, physical, sexual abuse, self-harm, neglect) change measured by personal behaviour from the PEMS + and the SDQ questionnaire. 2. More able to make and keep friendships and personal relationships based on mutual respect - change measured by social behaviour from the PEMS + and the SDQ questionnaire. 3. Better overall feeling of wellbeing change measured by personal behaviour, Wellbeing measure (where the programme is valued by the pupil/ young person?) Linked to employability and livelihood 4. More able to access their chosen options (yr 9) or further learning routes (yr 11) change measured before and after the social outdoor learning activities. 5. More commitment and plans for work experience, volunteering or paid work related to the outdoor learning activities - change measured before and after the social outdoor learning activities. Teachers 1. More rewarding teaching as they learn another side to the pupils, and have a less stressful relationship with pupils - change measured before and after the social outdoor learning activities. 2. More social integration of pupils likely to become NEET with the whole school community - change measured before and after the social outdoor learning activities. Funders Government and other public services 1. Less cost of offering public services for young people who are NEET - change measured by cost of public services. 2. More action research and adoption of the ACEWild approach by schools and LEEPs as a replicable model change measured by adoption of the ACEWild approach as a result of dissemination. Parents, Carers 1. Have a less stressful, more rewarding parent/carer to child relationship change measured by consultation before and after the outdoor learning activities.

5 Local Employment and Enterprise Partners (LEEPs) 1. Have better partnerships with schools change measured by consultation before and after the outdoor learning activities. Indicators We have only identified for the first three outcomes above related to changes in the pupils' wellbeing and lifestyle. Three of the evaluation tools that were developed and tested during the action research could provide the quantitative evidence for change. Evaluation tool of Pupil's progress Well-Being Measure Strengths and Difficulties Questionnaire PEMS + Behaviour Scale Example measure as SROI Indicator Satisfaction with friends, Satisfaction with family, 5 point scale Emotional symptoms, Peer relationship problems, 3 point scale Personal behaviour, 5 point scale By Educator Timing Measuring Online, by pupil On form, by educator On form, by educator Before and After Pupil feelings and attitude Before and After Pupil behaviour Every alternate session Pupil behaviour The partially completed ACEWild SROI Impact map can be found on the SROI section of the ACEWild website; It is based on a template provided by the SROI Network. Also supplementing this paper; a powerpoint reviewing SROI for ACEWild Critical review of our approach 1) Consulting stakeholders about the intended and unintended outcomes We started our project by designing the inputs and indicators to measure the outcomes (eg. SDQ Questionnaires, PEMS+) before identifying the intended outcomes - probably like many educators! We should have identified the stakeholders and asked them to identify the intended outcomes. We should also have asked the stakeholders to identify these outcomes in their own words. Also we could have identified any unintended outcomes at the end of the project, again by consulting stakeholders. For example, a parent might have said I'm now sleeping at night. We now realise that we should have consulted stakeholders and identified these desired or intended outcomes at the beginning of our 2 year project, and then planned backwards to the desired outputs, as well as the indicators and inputs required to achieve these outputs and outcomes. We also could have consulted stakeholders about unintended outcomes at the end of the project.

6 2) Time period of the SROI The intended outcomes may only be achieved after several years, and after the pupils leave school and are difficult to trace and consult. Feedback from pupils, as well as teachers, is only going to be possible up to the age of leaving school at 16. So we can only forecast the social value. So ACEWild chose to do a forecast SROI. A forecast SROI is not as accurate or convincing as an evaluative SROI. 3) Accurate financial proxies Finding a financial value for the change is very difficult, especially as so few SROIs have been been applied to formal or special needs education in schools. Financial proxies can be researched in various general and specialist databases and reports (eg. Global Value Exchange Alternatively financial proxies can be identified by researching or estimating the cost of delivering an alternative education approach and social service. The ACEWild project was not able to find a wide range of financial proxies so limiting the range outcomes that could be included in the SROI. 4) Costs of the investment The ACEWild project was also not able to accurately cost the investment of the ACEwild approach, especially the considerable staff costs in the schools, enterprises, universities and local authorities. This cost was probably considerably more than the EU grant of 124,000 Euros, as staff time was underestimated on staff timesheets for the project. 5) Our evaluative toolkit as SROI Indicators We believe that the outdoor learning activities with pupils have made significant changes in their behaviour and mindset, but this could also be partly due to the considerable investment of staff time. Our chosen indicators to measure the outcomes are based on a few of the 12 or more methods of peer, self and teacher evaluation (eg. SDQ Questionnaires, PEMS+) that we researched, tested and developed as a key part of the project. Our evaluative toolkit has enabled both staff and pupils to regularly reflect on the pupil's behaviour change. So the major success of the project has been in trialling and adapting these evaluation tools. However only a few of these tools are sufficiently quantitative and costed for them to be used as SROI Indicators. 6) Stakeholders More of the pupil's time is spent in the home with parents/carers than in the school with teachers. So the home environment, often without outdoor learning experiences, is crucial in terms of both the investment by parents/carers as well as the impact on parents/carers. 7) Mixed classes The Dutch and German partners did their action research with mixed classes, where pupils at risk of becoming NEET were in the same group as pupils who were predicted to leave the school and go straight into further education, training or employment. However the UK partners did their action research with a special group of pupils who were all at risk of becoming NEET. Using the ACEWild approach with a mixed class proved very challenging partly because of stricter timetabling, but also because of less teacher support. So the ACEWild SROI has only been based on the UK action research.

7 Review in the context of the seven principles of SROI 1. Involve stakeholders: Measuring social value is all about identifying and quantifying the changes experienced by the people involved in your project (your stakeholders). Therefore, it is essential that stakeholders are involved in the SROI process at all stages to ensure that the analysis accurately reflects their views and experiences. 1. The project staff were involved in the SROI process towards the end of the project but the pupils, some of the teachers and LEEPs were not clearly involved in the SROI process. 2. Understand what changes: It is important to understand and articulate what changes (or outcomes) are being brought about by your project. These may be positive or negative, intended or unintended. 1. As described above, the outcome were identified at the end of the project and by the project staff. 3. Value the things that matter: SROI analysis must make an attempt to value in financial terms any changes that matter to stakeholders, even if they are things that would not normally be given a financial value. 1. As described above, it is difficult to find a financial value for many of the soft outcomes that were identified, especially when they are identified and described by pupils. 4. Only include what is material: A SROI assessment should only include information and evidence which is material, in the sense that a person would make a different conclusion and/or decision about the project if that information were excluded. 1. Finding the essential outcomes for the stakeholders has not been possible. However the project staff have identified those that they believe were key evidence of change in the pupils' mindset and behaviour. 5. Do not over-claim: The SROI assessment should avoid attributing the creation of value to a particular intervention or project, unless there is reasonable certainty that these impacts were not caused by other factors. 1. There is a risk that we claim that the ACEWild approach to outdoor learning is more successful at avoiding pupils becoming NEET than other approaches such as through sport and adventure, music and drama, community engagement, boot camps etc. 6. Be transparent: It is essential that any data, assumptions or methods on which your analysis is based are properly explained and documented to ensure that conclusions and decisions are founded on accurate evidence and any uncertainties with the analysis are fully understood. 7. Verify the result: Developing a SROI will always involve an element of subjectivity so external independent assurance of the analysis will help to ensure that the assessment is reasonable and unbiased. If the budget allows, it is worth considering the assurance service provided by the SROI Network. Assessment of the ACEWild approach to SROI We believe that we are one of the first projects in Europe to explore the use of SROI in the context of special education and NEET. We have identified a process for measuring a few outcomes, with their financial proxies, that we could recommend to other educators for future use, and that we could use ourselves. However without an approach to SROI that honestly follows all the seven principles we are not yet able to demonstrate the costeffectiveness of the staffing for the outdoor learning activities and management of the evaluation tools.

8 Ideal SROI process 1. Decide the stakeholders 1. Who can mainly help the process of the pupil's change, and who will be mainly impacted by it? - Pupils and parents/carers are probably central to this process. 2. Decide the time period 1. How long might the impact last of the social outdoor learning experience? - until leaving school, up to 5 years. 2. Should evaluate retrospectively or forecast ahead? 3. Decide the outcomes 1. What do the main stakeholders honestly want, in their own words, as an outcome in this time period? A one-one chat, focus group 2. What are the most important and key outcomes? 3. At the end of the period what were the unintended outcomes? 4. Plan the outputs and inputs needed to deliver those outcomes 1. What activities, plans, staffing, partnerships with LEEPs, equipment etc are needed? 5. Measure the cost of the inputs 1. What is the cost, esp. in staff time, of providing 6. Decide the indicators 1. What are the quantitative and qualitative indicators of change in the group of pupils, in the group of parents/carers, in the group of teachers and in the group of LEEPs? 2. What evaluation tools (eg. PEMs +, SDQ) can measure this change? 7. Research the financial proxies 1. What are the costs of delivering an alternative service? - in the school, in other schools, nationally 8. Calculate the SROI 9. Verify the results and evidence Some examples of indicators that could be used All these indicators are quantitative and use terms like more, less, greater. Number of pupils appearing more confident. Number of pupils being more communicative in class. Number of pupils seen by peers to be more adventurous (raising confidence further.) Number of teachers reporting more resilient pupils; contributing more in schools, in class and in groups. Number of pupils reported by teacher as working well in team and showing original thoughts/inputs Number of pupils displaying less disruptive behaviour in class e.g. crying when things are difficult. Number of pupils beginning to eat a greater variety of foods, changing eating habits considerably, as reported by parents/carers Number of pupils that report they feel more confident to avoid behaviours which may be detrimental to themselves and break the law etc

9 Some examples of financial proxies The average unit cost to the NHS in the UK of treating someone with depression is 2,026. Source: McCrone, P., Dhanasiri, S., Patel, A., Knapp, M. and Lawton-Smith, S. (2008) Paying the price: The cost of mental health care in England until The King s Fund Report. The average cost per year old not in education, employment or training ( NEETs) is per NEET per year. The Troubled Families Cost Database compiled by New Economy, Refs Eurofound, European Monitoring Centre on Change, University of York, 2010 Thanks We are very grateful to Dave Spence, Chief Executive of the Scottish Outdoor Education Centres, Helen Campbell of Social Value UK, and Elaine McCorriston, Accredited SROI Practitioners, and Co-ordinator of Social Enterprise East of England for their support and helpful comments about the ACEWild SROI. Author Adam Cade SustEd ACEWild Project Team August 2016 This project has been funded with support from the European Commission (Erasmus). The website reflects the views only of the ACEWild team, and the Commission cannot be held