Partner Telephone Survey Report May 5, 2014
EVALUATION SUMMARY Partner Telephone Survey Author: Shannon Robinson, Dorian Lunny Date: May 5, 2014 Approved by: Dr. Kit Young Hoon Page 1 of 20 Table of Contents Executive Summary... 2 Introduction... 3 Methods... 3 Pre-Testing... 4 Sample... 4 Table 1: Survey sample by sector... 5 Data Collection... 5 Analysis... 6 Results... 6 Awareness of health unit programs and services... 6 Awareness of how to contact the health unit... 7 Perception of the health unit... 7 Partnership with the health unit... 8 Visits to www.nwhu.on.ca... 8 Input into health unit priorities... 8 Limitations... 9 Discussion... 9 Conclusion... 10 Appendix 1: Request for Proposals... 11 Appendix 2: Partner Organization Pre-Notification Letter... 17 Appendix 3: Partner Pre-Notification Letter... 18 Appendix 4: Partner Questionnaire... 19 1
Executive Summary In November and December 2013 the Northwestern Health Unit commissioned a survey to assess partner awareness and perception of our organization. The survey also collected key baseline information related to strengthening partnerships that will assist us in measuring progress on our 2013-2016 Strategic Plan. The survey sample consisted of over 500 partners from the Northwestern Health Unit catchment area. A partner was defined by staff members as having been in contact with or working with the health unit more than four times in the last year. 274 partners completed the telephone survey, conducted by Oraclepoll Research Limited. The largest groups of respondents came from the government (34%) and not for profit (34%) sectors. The data was collated and analyzed using Microsoft Excel and STATA. Key findings include: 57% of respondents were aware of the programs and services offered by the health unit. A high percentage of partners interviewed (93%) were aware how to contact the health unit. A high percentage of respondents (82%) had a positive view of the health unit. A very low number of partners had a negative opinion of the health unit (2%). 78% of respondents rated their partnerships with the Northwestern Health Unit as positive. 16% indicated more communication and more information on programs could help strengthen the partnership. 58% of respondents had visited the NWHU website. To gather input into what the public believes NWHU strategic priorities should be, an openended question was posed. 33% of respondents did not know where the health unit should focus its work, 10% felt the focus should be on public awareness and education, 9% on programs for youth, and young people, 7% on prevention, 6% on healthy living and physical activity, and 6% on immunization programs. The results provide us with information that will help us determine areas in need of improvement and where we should focus efforts to accomplish the objectives set out in our strategic plan. The survey will be repeated in 2016 and the results will be compared to help us measure the impact of our activities. View the full report at www.nwhu.on.ca. 2
Introduction On March 1, 2013 the Northwestern Health Unit put out a Request for Proposals: Surveys to Assess Progress on the Strategic Plan (Appendix 1). The request sought an independent company to complete a series of surveys to help us measure our 2013-2016 Strategic Plan. There are two parts to the project, each with similar objectives: Telephone survey the general public to: Assess awareness of NWHU and how to contact us. Assess perception of the health unit. Assess changes based on our healthy eating and physical activity strategy. Gather input into strategic priorities. Telephone survey partners to: Assess awareness of NWHU and how to contact us. Assess perception of the health unit. Gather input to strengthen partnerships. Gather input into strategic priorities. Oraclepoll Research, based out of Sudbury, ON was the vendor awarded the contract. The initial general population telephone survey took place in May and June 2013, and the complete report is available at www.nwhu.on.ca. This report summarizes the results of the survey of partners in the Northwestern Health Unit catchment area November 20 December 13, 2013. This survey will be repeated again in 2016 and the two sets of results will be compared for significant differences. Methods A number of strategies were put in place to help ensure a robust response rate and reduce the likelihood of non-response bias. A letter was mailed to 68 partner organization to let them know that their staff had been identified as part of the sample and that they could contact us with any questions or concerns. The letter was not sent to private practices or businesses and was only sent to those organizations within which multiple employees would be contacted. A copy of the letter can be found in Appendix 2. A pre-notification letter was sent to all of the contacts in our sample for which we had mailing addresses. A copy of this letter can be found in Appendix 3. Other communication activities included putting up posters in the waiting rooms of each of our 13 offices, posting information at www.nwhu.on.ca, a media release, and preparing staff with information to respond to any questions from the public. The media release was picked up by 3 news outlets in the region, The questionnaire was designed to be short and non-invasive. 3
Multiple contact days and times were scheduled, and multiple follow ups were planned if unable to reach on first attempt. The opportunity to reschedule the interview to a more convenient time was offered, respondents could also provide email address to complete the survey electronically. Sponsorship of the survey was identified, and a validation source was offered if required. Respondents were given assurances of confidentiality and that responses would not be linked to names or organizations. Pre-Testing To test the face validity of the survey tool, a thorough review of the survey tool was done by the NWHU management team, Planning Officer, Epidemiologist and Continuous Quality Improvement Officer. The group went through the questions to assess clarity, appropriateness and if the questions appeared to be collecting information that would correspond to the indicators identified in the strategic plan. Some of the survey questions contained the same questions as asked in the general population telephone survey, which was pre-tested then implemented successfully with 1000 participants. To further test the questionnaire tool a pilot test was completed by Oraclepoll with members of the sample. No issues were identified, and the survey tool was implemented as designed. Sample The Northwestern Health Unit provided Oraclepoll with contact information for 517 partners from a list maintained by staff. Partners were included on the list if they were in contact/work with NWHU staff more than four times per year. As a result of the pre-notification letters, a number of revisions (both additions and deletions) to the contact list were made. A number of pre-notification letters were also returned by the post office, indicating incomplete or inaccurate contact information. Employees of the Northwestern Health Unit were excluded from the contact list. Partners outside of the catchment area (i.e. Thunder Bay or Toronto), and those on the list that did not have telephone contact information were also excluded. During the pre-test process it was decided that if the partner had left the organization the survey could be offered to the person that replaced them. New respondents were added to the sample during the survey process if they had replaced the person holding the position originally identified by NWHU staff. A total of 274 partners surveys were completed, out of a possible 408 eligible telephone numbers. 4
Table 1: Survey sample by sector Sector Number of respondents Government 94 Not for Profit 92 Health Care 31 Retail 24 Other* 13 Media 11 Education 9 Total 274 *Includes Hospitality, Manufacturing, Transportation, Social Services and Professional Services Data Collection The telephone surveys were conducted by Oraclepoll between November 20 th and December 13 th, 2013. Up to 10 attempts were made to reach a contact. Calls were made between 8:30 a.m. and 6:00 p.m. There was at least one attempt to reach respondents after 6 pm and on a weekend if there was no weekday response. Appointments were arranged for those requesting a more convenient time to complete the survey. Respondents were offered an option to provide an email address to fill out the survey electronically. The survey was conducted using computer-assisted techniques of telephone interviewing (VOXCO CATI survey software). 20% of all interviews were monitored by Oraclepoll, and the management of Oraclepoll Research Limited supervised 100%. No respondents completed the survey electronically. The average length of each survey was 4.5 minutes. A copy of the survey questions can be found in Appendix 4. A total of 517 numbers were called in order to achieve 274 completed surveys. The estimated total number of eligible participants was 402, giving a survey response rate of 68.2%. Table 2 outlines a breakdown of the calls that were made. Table 2: Call breakdown Result Number of calls Survey completed by an eligible person 274 Ineligible 76 No answer 75 Refused by an eligible* person 58 Out of service/wrong number 17 Prolonged absence 14 Fax number 3 Total Calls 517 *an eligible person is one who is reached and who meets the selection criteria 5
Analysis At the end of the telephone research stage the data was extracted from VOXCO into STATA. The data was then validated and cleaned and then coded by Oraclepoll. Open ended responses were content-analyzed by researchers at Oraclepoll and assigned into categories. Frequencies were run and cross-tabulations were prepared in an Excel report. Tests of statistical significance and regression analyses were performed using STATA. Results Awareness of health unit programs and services Partners were first asked the following awareness question. "Using a scale from 1-5 with 1 being not at all aware and 5 being very aware, how would you rate your awareness of the types of programs and services offered by the Northwestern Health Unit?" 57% of respondents were aware or very aware of the programs and services offered by the Northwestern Health Unit, while 32% of respondents were unaware or very unaware and 11% had a neutral opinion (neither aware nor unaware). The proportion of partners who were aware of the health unit s programs and services was significantly higher than the general public, which was reported at 41% during our general population survey. About 24% of respondents who did not fall in the sectors of government, retail, not for profit or health care (the most frequent categories) claimed they were unaware or very unaware of the health unit s programs and services. This includes those in education, media, hospitality, manufacturing, transportation, social services and professional services. This is significantly higher than what was reported by those in the health care and not for profit sectors. Table 3: Partner awareness of NWHU programs and services Proportion of respondents (%) Very aware 17.2 Aware 39.8 Neither aware nor unaware 32.1 Unaware 10.2 Very unaware 0.7 6
Awareness of how to contact the health unit Partners were next asked about their awareness of how to contact the Health Unit. "Using the same scale from 1-5 with 1 being not aware at all and 5 being very aware how would you rate your awareness of how you can contact the Northwestern Health Unit?" A high percentage of respondents (93%) were aware or very aware of how to contact the Northwestern Health Unit. This is also significantly higher than the general population, where 77% reported that they were aware of how to contact the health unit. Table 4: Partner awareness of how to contact the NWHU Proportion of respondents (%) Very aware 69.3 Aware 24.1 Neither aware nor unaware 4.7 Unaware 1.5 Very Unaware 0.4 Perception of the health unit Respondents were asked about their perception of the Health Unit "Using a scale of 1-5 with 1 being very negative and 5 being very positive, how would you rate your overall perception/view of the Northwestern Health Unit?" A high percentage of respondents had an overall positive or very positive view of the Health Unit (82%). This proportion is higher than the 72% of the general population who reported having a positive view, and the difference is statistically significant. A very low number of partners had a negative opinion of the health unit (2%), while 16% of respondents had a neutral view. Table 5: Partner overall perception of the NWHU Proportion of respondents (%) Very positive 42.7 Positive 39.1 Neither positive nor negative 16.7 Negative 1.5 Very negative 0 7
Partnership with the health unit Partners were asked Using a scale of 1-5 with 1 being very negative and 5 being very positive, how would you rate your partnership with the Northwestern Health Unit? A total of 78% of respondents rated their partnerships with the Northwestern Health Unit as positive. Only 6% rated the partnership as negative, while 15% had a neutral view. Table 6: Partner ratings of their partnership with the NWHU Proportion of respondents (%) Very positive 46.4 Positive 31.8 Neither positive nor negative 16.0 Negative 4.4 Very negative 1.4 They were also asked is there anything that the Northwestern Health Unit can do to strengthen the partnership that it has with you? 65% of partners did not know how the Northwestern Health Unit could strengthen the partnership. 4% indicated nothing. 16% indicated more communication and more information on programs. Visits to www.nwhu.on.ca Partners were also asked whether they had ever been to the NWHU website. 58% of respondents claimed that they have visited the NWHU website. This is significantly higher than the general population, of which 18% reported visiting the website in the past. 40% of our partners had not visited the website, and 2% did not know if they had. Input into health unit priorities Partners were asked where they believe the Northwestern Health Unit should focus its work. 33% of respondents did not know where the health unit should focus its work. 10% felt that the health unit should focus on public awareness and education, 9% on programs for youth, and young people, 7% on prevention, 6% on healthy living and physical activity, and 6% on immunization programs. 8
Table 7: Partner perceptions as to what the NWHU priorities should be Response Proportion of respondents (%) Don t know 32.5 Public health awareness/education 9.9 Programs for youth/young people 8.8 Preventative care/information 6.2 Immunization programs 5.8 Addiction programs 4.0 Healthy living 4.0 Focus on families 2.9 Limitations As with all surveys, frequencies and proportions estimated from the study are reliant on the self-reporting of individuals in the sample. Self-reported data will always have concerns about its accuracy, as it is dependent on the individual interpretations of the questions from those in the survey sample. Factors such as social desirability bias may be present with this type of survey. Contact lists are very hard to keep up to-date and only represent a point in time. For many of the contacts, information was incorrect or out-of-date, reducing the size of our sample. Sample size is a limitation when it comes to the analysis of this data. Discussion An objective in the NWHU strategic plan is to increase the percentage of community partners who report that they know about, and how to contact, the health unit by 10% relative to the baseline by December 2016. Two questions in this survey assessed this objective and provide the baseline numbers for measuring change. 57% of respondent partners were aware of the programs and services offered by the Northwestern Health Unit, and 93% were aware how to contact the Northwestern Health Unit. The results show room to increase partner awareness. A follow up question was asked about perception of the health unit, and a high percentage of respondents had an overall positive view of the health unit (82%). Knowing what partner perception of our agency is will help guide our messaging strategy as we work to improve awareness of our programs and strive for quality public health services. Partners are generally more aware of and have a more positive view of the health unit when compared with the general population in our catchment area. Although we will strive to 9
improve our presence and relationships with our community partners, the greatest room for improvement appears to be with our general public. Partners cite improved communication and information about NWHU programs and services as key to strengthening partnerships. Tactics to address this will also help increase awareness of our programs and services and relationships with our partners. The survey provided an opportunity to add a supporting indicator to measure the success of our website. As no other measure of website use currently exists for the health unit, we now know that 58% of partners can report visiting the Northwestern Health Unit website. The NWHU will be launching a new website in 2014, with a new look and new features accompanied by a promotional strategy designed to increase its use. To gather input into what the public believes NWHU strategic priorities should be, an openended question was posed. 33% of respondents did not know where the health unit should focus its work, 10% felt the focus should be on public awareness and education, 9% on programs for youth, and young people, 7% on prevention, 6% on healthy living and physical activity, and 6% on immunization programs. Conclusion This survey has provided us with baseline numbers that can be used to monitor our progress on strategic objectives from 2013 to 2016. Results of this survey provide valuable information to inform our planning over the next few years, including work in raising awareness about the health unit, the www.nwhu.on.ca website, and strengthening our work with partners. This survey will be repeated in 2016 and will allow us compare results to know if we have been successful at meeting the objectives in our strategic plan. 10
Appendix 1: Request for Proposals Request for Proposals: Surveys to Assess Progress on the Strategic Plan March 1, 2013 The Northwestern Health Unit is issuing this request to select qualified vendors related to general public and partner telephone surveying during 2013, 2014, 2015, and 2016. The surveys will assess outcome objectives identified in our strategic plan. You can view our strategic plan at www.nwhu.on.ca. Contact: Shannon Robinson Planning Officer 807-468-3147 (ext. 3248) srobinson@nwhu.on.ca Deadline for submission: March 15, 2013; 4:30 p.m. (CST) 11
SCOPE OF WORK Project Management The project will be managed at the Northwestern Health Unit by the Planning Officer. The Vendor will work in collaboration and consultation with the Project Manager. The Project Manager retains final decision-making authority. The vendor will assign a liaison to work with the Health Unit Project Manager. Project Description There are two parts of this project, each with similar objectives: Telephone survey the general public to: Assess awareness of NWHU and how to contact us. Assess general public perception of the Health Unit. Assess awareness change based on a multi-year communication campaign. Gather input into strategic priorities. Telephone survey partners to: Assess awareness of NWHU and how to contact us. Assess general partner perception of the Health Unit. Gather input to strengthen partnerships. Gather input into strategic priorities. Vendor Qualifications Vendor must: 1. Be willing to sign a contract with the Northwestern Health Unit. 2. Have a centralized facility in Canada to perform the services. 3. Provide all human resources and equipment to complete the deliverables. 4. Store data in Canada. 12
13 Deliverables and Timeline Timelines can be negotiated before the project begins. Vendor Deliverables Project 1 (Public) Timeline Regular communication with the Project Manager 2013-2016 Provide feedback on the draft survey provided by NWHU By April 5, 2013 Finalize survey, in collaboration with the Project Manager By April 12, 2013 Draw the population survey sample April 2013 Prepare CATI questionnaire By April 19, 2013 Prepare a call protocol based on survey design By April 19, 2013 Train interviewers By April 26, 2013 Pre-test the questionnaire with NWHU staff By May 10, 2013 Revisions to CATI questionnaire or call protocol (if required) By May 17, 2013 Collect data Beginning May 20, 2013 Analyze data June November, 2013 Prepare report, according to specifications By December 20, 2013 Train interviewers for follow-up survey January 2016 Collect data Beginning Feb 1, 2016 Analyze data March May, 2016 Prepare report, according to specifications By June 24, 2016 Vendor Deliverables Project 2 (Partners) Timeline Regular communication with the Project Manager 2013-2016 Provide feedback on the draft survey provided by NWHU By Oct 25, 2013 Finalize survey, in collaboration with the Project Manager By Nov 8, 2013 Draw the sample from a list of partners provided by NWHU By Nov 8, 2013 Prepare CA TI questionnaire By Dec 20, 2013 Prepare a call protocol based on survey design By Dec 20, 2013 Train interviewers January, 2014 Pre-test the questionnaire with NWHU staff By January 17, 2014 Revisions to CATI questionnaire or call protocol (if required) By January 31, 2014 Collect data Beginning Feb 3, 2014 Analyze data March-June Prepare report, according to specifications By August 1, 2014 Train interviewers for follow-up survey August, 2015 Collect data September 2015 Analyze data October December 2015 Prepare a technical report, according to specifications By Dec. 18, 2015
14 Communication During the project, the vendor liaison will work in collaboration and consultation with the Project Manager to resolve issues that arise. The liaison will ensure regular communication with Project Manager about project status. During data collection periods, the Program Manager should receive weekly updates about: Number of surveys completed Average length of survey Number of refusals Population Survey - Sample The vendor will provide a sample, using the following criteria, to provide a statistically representative sample for each demographic stratum. The sample must reflect the distribution of NWHU region population by: Sex Ethnicity Age groups: 18 to 24 years, 25 to 44 years, 45 to 64 years and 65 years and older People living in different parts of the region (Kenora District; Rainy River District) Inclusion criteria: Resident of the NWHU catchment area (list of locations to be provided by NWHU). Aged 18 or older. Able to speak and complete the telephone interview in English The sample must be large enough to compare for small significant differences between 2013 and 2015. The vendor will also calculate the sampling error. NWHU will consider the inclusion of a (complementary) alternate data collection strategy to reach 18-24 year olds who may be under-surveyed through telephone sampling. This could include promotion of an online survey through social media. Partner Survey - Sample The vendor will draw a sample from an organizational contact list provided by NWHU. The sample must be large enough to compare for small significant differences between 2014 and 2015. The vendor will also calculate the sampling error.
A call protocol must be developed for each survey and should include: Timing and number of call attempts. Description of project. Screening and selecting respondents. Obtaining consent. Standard responses to frequently asked questions. Interviews for each survey must be: Conducted in English. Between 5-15 minutes in length. Appropriately monitored by vendor to ensure professional standards and compliance with call protocol. Data Processing and Analysis The vendor will work with the project manager to develop a code book for each survey question with open-ended response. There is one open-ended question on the draft public survey, and two on the draft partner survey. The vendor will clean the data for each project, after each data collection cycle and create final data sets in STATA. The vendor will encrypt the data sets and deliver one copy of each to the Project Manager. For each project the vendor will provide descriptive analysis for each question, by demographic variables (ages, sex). For each project the vendor will provide analysis to determine if there are statistically significant differences between the initial data and the follow-up data. Reporting Over the course of the project, the vendor will prepare 4 electronic reports, in Microsoft Word, documenting the project. Reports will include: Overview Sampling strategy Data collection procedures Response rate Data processing procedures Results for each survey question (by age, sex) The second set of reports should include comparison to the first data collection cycle. 15
Record Retention Processes must be in place to ensure confidentiality and the protection of personal information The vendor shall maintain all project documentation in a single secure location for the duration of the project and for a period of one (1) year following project completion. The vendor must ensure that all project documentation and collected data are securely destroyed at the end of the archive period and that notification of this destruction is sent to the Project Manager. 16
Appendix 2: Partner Organization Pre-Notification Letter 17
Appendix 3: Partner Pre-Notification Letter 18
Appendix 4: Partner Questionnaire SURVEY - PARTNERS PREAMBLE Good morning/afternoon/evening, my name is <$> and I am calling from the research firm Oraclepoll on behalf of the Northwestern Health Unit. May I speak to <$>? CONTACT PROCEED NOT AVAILABLE - ESTABLISH CALL BACK TIME You have been identified as a partner working with the Northwestern Health Unit. We are conducting a brief survey of Northwestern Health Unit partners that will take approximately five minutes of your time and your input is very important in helping to plan and to prepare future programs in the area. Please be assured that all information gathered will be kept in strict confidence and only overall results will be reported. REFUSED POLITELY TERMINATE BUSY - ESTABLISH CALL BACK TIME REFUSED TELEPHONE APPROACH OFFER ONLINE OPTION VERIFY EMAIL TO SEND LINK Please keep in mind that this survey is voluntary and that you can stop it at any time or refuse to answer any questions. We are looking for your honest answers and I would like to mention once again that all individual survey responses will be kept confidential. Q1. Using a scale from 1 5 with 1 being not at all aware at all and 5 being very aware, how would you rate your awareness of the types of programs and services offered by the Northwestern Health Unit? 1-not at all aware 2-not aware 3-neither aware nor unaware 4-aware 5-very aware Don t know Refused Q2. Using the same scale from 1 5 with 1 being not aware at all and 5 being very aware how would you rate your awareness of how you can contact the Northwestern Health Unit? 1-not at all aware 2-not aware 3-neither aware nor unaware 4-aware 5-very aware Don t know Refused Q3. Using a scale of 1-5 with 1 very negative and 5 being very positive, how would you rate your overall perception/view of the Northwestern Health Unit? 1-very negative 2-negative 3-neither negative nor positive 4-positve 5-very positive 100-Don t know 99-Refused 19
Q4. Using a scale of 1-5 with 1 being very negative and 5 being very positive, how would you rate your partnership with the Northwestern Health Unit? 1-very negative 2-negative 3-neither negative nor positive 4-positve 5-very positive 100-Don t know 99-Refused Not applicable option Q5. Is there anything that the Northwestern Health Unit can do to strengthen the partnership that it has with you? RECORD VERBATIM 97-Open 100-Don t know 99-Refused Q6. Have you ever been to the Northwestern Health Unit website (www.nwhu.on.ca)? 1-yes 2-no 100-Don t know 99-Refused Q7. The Northwestern Health Unit seeks feedback from its partners about what its priorities should be. Where do you think that the Northwestern Health Unit should focus its work to help make communities healthier? RECORD VERBATIM 97-Open 100-Don t know 99-Refused D1. What best describes the sector or category that your organization falls into? CODE APPROPRIATE CATEGORY Health care Education Emergency services Business / private sector Government Charitable organization / not for profit Religious OTHER RECORD 20