EVALUATING HEALTH AND WELLBEING INTERVENTIONS Q&A

Transcription

1 April 2015 EVALUATING HEALTH AND WELLBEING INTERVENTIONS Q&A This Q&A lists all the questions asked during the webinar and includes answers to those questions we did not have time to answer. Q. Have you heard of any organisations in the UK that have used fitness bands to support staff health and wellbeing and been able to download the user information to get evidence of improved activity levels? A. I have not heard of anything formal using fitness bands. Sheffield Teaching Hospitals has run a project in collaboration with Sheffield Hallam University looking at various health markers in staff and then retesting six months later. Q. What does the physiotherapy service offer that is different to the service patients get? A. The service is more accessible with self-referral and clinics in the workplace. It is able to assess the workplace and suggest/oversee/monitor workplace adjustments. It is staffed by physiotherapists with expertise in ergonomic and occupational factors. Q. Can we get a copy of the questionnaires that were used please? NHS Employers response: A. research activity. However, as part of our work with Zeal we are 1

2 looking to make available a number of evaluation diagnostics currently being accessed by NHS trusts. Q. It was interesting that you described the main purpose of occupational health (OH) is as a management tool. In my organisation OH is the middle between the employer and employee OH is absolutely there for the benefit of the employee! Where do you see your OH service and how does it connect with HR, OH and the employee? A. I did not mean that it is the sole purpose of OH, but in our experience many staff view being referred to OH as part of the organisations formal method of managing absence, and this does not always have a positive effect. OH does not actually offer any treatment to staff referred, it is only advisory. I do see our development very much linked to both OH and HR and think that consistency of advice from the three of us is very important. Q. Did you gain board agreement to the evaluation criteria prior to the pilot? A. oard but it had the support of the senior management teams. Q. Have there been any key changes made to the PhysioPlus service as a direct result of the evaluation? A. We have started a new system that enables us to use the electronic staff record (ESR) to contact every member of staff who has been off sick for more than two weeks with a musculoskeletal disorder (MSD) to ask them if they would like to self-refer. It has enabled us to continue with self-referral when there was some pressure internally to have access to the service via management referral. 2

3 Q. How long did it take to undertake this evaluation? A. From the start of the design to the conclusion, about 18 months. Q. Did the team do any work relating to how much demand they would have for the service? What was the level of demand? A. We did but it was very difficult to predict. Demand is generally high but if we do anything internally to promote the service it goes through the roof. More recently we have been working on systems that enable us to identify and contact staff in our higher priority groups e.g. those who have been off work for about two weeks or likely to go off work soon. Q. How were the physiotherapists allocated? Were staff physiotherapists included in the service? A. The staff were recruited to the service and selected against the specific job description and person specification. They are permanent posts to this service. Q. Were all MSDs included or did it jus problems? A. All MSDs and we also see other conditions now post mastectomy, post general surgery, occasional neurological disorders. We are also looking at doing some work around cancer survivorship. Q. You said this service is a programme for staff, as a service for their needs, not a programme for absence management or a management tool. How did you achieve this? A. This was one of our fundamental principles that was stated from the outset in our business cases, it is backed up by the views of Dame Carol Black and Dr Steve Boorman. 3

4 Q. Can you give some examples of the criteria you could look at to measure or evaluate other than sickness absence figures or subjective measures? A. Before you can decide on the criteria you need to consider the purpose of the intervention. For example, if the intervention is a training programme and is being introduced to help managers manage sickness absence better then you need to explore how the training is going to help them manage sickness absence better (e.g. is it through knowledge or particular skills) and then you can start to think about how to evaluate this type of intervention. You must also question whether the intervention is likely to have an immediate impact on outcomes (e.g. sickness) or whether the outcome is likely to be seen in the longer term. You can select any type of outcome that you think could be measured, however, the key is choosing the correct criteria. The only way this can happen is by outlining in some detail the purpose of the intervention and also considering how the intervention will be implemented and how the process of implementation may impact on any outcomes that are measured. NHS Employers response: A. Tony is happy for you to contact him directly to discuss your specific needs so he can guide you. Please contact tony on: tony@zealsolutions.co.uk and use reference: NHSEevalweb in your . Q. Can you say something more about the outcome measures and their robustness? A. The outcome measures used in this study were appropriate for the intervention and the general aims of the evaluation. The criteria are robust in that they have excellent psychometric properties (e.g. high reliability and validity). We were also able to validate the evidence that was collected through follow-up interviews with staff. 4

5 Q. Can you explain the difference between service evaluation and research? Service evaluation is something that all organisations should be conducting. These are evaluation of operational services and are in line with best practice. You need to know whether and how something is or is not working as it should. In the strictest to improve knowledge and understanding of the scientific community and to progress theory. However, the difference between the two are very blurred. For me, service evaluations are research exercises. We apply the same principles of good research design to service evaluation. If you are conducting operation service evaluations (e.g. evaluating a training intervention that is provided to staff) using simple before and after questionnaires you no longer need to obtain ethics approval. If your evaluation is to use other assessments which might capture physiological data (e.g. saliva samples) from staff or might involve patients/service users then you will need to seek ethical approval to ensure the wellbeing of your participants are protected. The NHS health research authority has lots of information and guidance that can be accessed to support your decision making. Q. You may not have to be a rocket scientist but what skills do you need? A. Some of the key skills required include: psychometric skills that will help with question, survey and interview, design communication skills (written and verbal) to ensure you can document and promote the process appropriately organisational and planning skills to ensure you can coordinate and manage the evaluation quantitative analysis skills (e.g. statistics) to ensure you can make sense of any numerical data you have collected qualitative analysis skills (e.g. content analysis) to ensure you can make sense of any written information you have collected interpersonal skills to support the building of relationships and working with participants. 5

6 A. This is a small illustrative list rather than exhaustive list of skills. From the PhysioPlus side you need leadership and communication it can be a bit of an ordeal collecting data and staff and patients need to maintain their motivation. My view was really that once we had agreed on the evaluation design our job s job was to process and analyse it. Q. We b health and wellbeing events and activities both to those that attend but also across staff in general who are pleased that the trust is making some attempt to consider health and wellbeing how could we evaluate that? A. You could issue a survey to all staff to assess their perspectives on the events and also gain a general sense of the health of the organisation (however, you would need to tailor the assessment for this as, for example, the generic NHS staff survey, is not sensitive enough to pick up the impact of this form of intervention. You could also consider asking staff to complete an assessment before and after they attend the event, and also issue a follow up with some staff at some point in the future to see if any benefits have been maintained. Q. Did you have to demonstrate return on investment? A. Yes but we wanted to be able to demonstrate the breadth of the benefits of the service and not be held to the simple, but incorrect assertion that the only return is a decrease in overall MSD absence. Q. Will you continue to capture staff satisfaction of the service? A. Yes we are continuing to do so based on some of the lessons from the evaluation. 6

7 Q. Where can I download the report? NHS Employers response: A. Please see report here. Q. What response rate did Sheffield Teaching Hospitals have with their questionnaires? To get 100 match sets, how many sets did they need to send out in the first place? A. Approximately 300 surveys were issued. A. between the completion of treatment and the three month follow up we had to do quite a lot of chasing up! Q. The hallmark of a gold standard for evaluation is having a control or comparison group, can you talk a bit about the various ways of building a control group. A. Longitudinal studies where participants are followed over time (e.g. before and after they participate in an intervention) is actually one of the most robust designs. In this instance, the participant becomes their own control. This is why it is useful to collect lots of matched sets of data. This enables you to assess the impact of an intervention with a great deal of confidence. You can also run studies where you compare individuals who undertake an intervention with those that do not and then compare the results. However, and unless you are using a randomised assignment to either the intervention or control group, you will need to ensure (within reason) that the composition of the control group is as similar as possible to the intervention group. Otherwise any differences in your data could intervention itself. 7

8 Q. What happens if you have someone off sick that lives out of there e.g. A. It is uncommon but does happen. Depending on the distance and the circumstances we might still visit or offer telephone support. We have gone as far as 50 miles to do a home visit you have to bear in mind the cost of absence when considering the level of support to offer. Q. We have already stated a wellbeing intervention what can we do now that will still allow us to measure the impact this has had after the event? A. You can conduct retrospective evaluations, asking staff that have attended to complete an assessment of some kind. You can also compare this with staff who have yet to participate in the intervention to compare any results. However, we would recommend that you start to gather evidence before and after the attendance/involvement in the intervention as soon as possible. NHS Employers response: Tony is happy to offer any guidance on this. Please contact tony on: tony@zealsolutions.co.uk and use reference: NHSEevalweb in your . Q. We do not have any health and wellbeing budget what can I do to ensure this is robust without any funding? A. You will need to be realistic about what can be achieved. However, an evaluation can help to secure a future budget for your health and wellbeing activities. You can still design an assessment to capture some evidence to illustrate value and impact. The key is understanding your aims and objectives and then ensuring a realistic evaluation is designed to help meet your needs taking into consideration your constraints. 8

9 Q. We do a lot of health promotion campaigns in the trust and, measure, we know that more people could be benefiting from seeing the information. How would you recommend we might be able to quantify this? A. Again, it would be useful to know the details of any specific campaigns (e.g. any specific aims or goals). This information can then be used to develop some key evaluation questions and then help to identify the criteria that can be used to evaluate the campaign. For example, if a campaign is to be used to increase awareness of the importance of keeping healthy, then you could develop some questions to assess awareness of health and wellbeing and check whether there has been an increase as a result of your campaign. Although you can conduct retrospective evaluations, in this situation, the ideal scenario would involve wellbeing awaren you could decide to track a few people before and after a specific campaign (e.g. on physical health) to see if there has been a change in health behaviour (e.g. engaging in more physical exercise after the campaign). The key with all of this is to clearly specify the purpose of the campaign/intervention. Once specified, the criteria to help you evaluate impact or success can then be identified. For more information please healthandwellbeing@nhsemployers.org 9