Newfoundland and Labrador Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit Mid Term Evaluation

Transcription

1 Newfoundland and Labrador Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit Mid Term Evaluation Final Report January 17, 2017 Prepared for: NL SUPPORT Unit Prepared by: R.A. Malatest & Associates Ltd.

2 Table of Contents R. A. Malatest & Associates Ltd. Evaluation Lead: Carmelle Goldberg Project Manager: Rob Assels 1. EXECUTIVE SUMMARY Overview NL SUPPORT Context and Profile Findings Conclusions BACKGROUND The Newfoundland and Labrador SUPPORT Unit Mid Term Evaluation of the Newfoundland and Labrador SUPPORT Unit METHODOLOGY Characteristics of the NL SUPPORT network Key Informant Interviews Online Survey Document Review Analysis Strengths and Limitations of the Evaluation Strengths Limitations ADDRESSING LOCAL NEEDS AND PRIORITIES FOR POR (Question 1) Alignment with jurisdictional priorities Engagement and Representation Benefits and Challenges EFFICIENCIES AND ECONOMIES (Question 2) Organizational Structure Planning processes Community Engagement Processes Training and Capacity Building Planning Matching Patients to Research Projects Matching Decision and Policy Makers to Research Projects Partnership development PROGRESS TOWARD OPERATIONAL OBJECTIVES (Question 3) Data platforms and services are operational Methods support and development are offered Health systems research, implementation research, and knowledge translation have commenced Pragmatic / real-world clinical trials are emerging Career development in methods and health services research is operational Consultation and research services are operational DEGREE OF INTERACTION WITH SPOR NETWORKS AND SUPPORT UNITS (Question 4) CONCLUSIONS APPENDIX A: INTERVIEW GUIDES Interview Guide for the SUPPORT Team Interview Guide for Researchers Interview Guide for Partners APPENDIX B: ONLINE SURVEY... 33

3 1. EXECUTIVE SUMMARY 1.3 FINDINGS 1.1 OVERVIEW 1.2 NL SUPPORT CONTEXT AND PROFILE This report presents the findings of the Newfoundland and Labrador Support for People and Patient-Oriented Research and Trials (NL SUPPORT) Unit mid-term evaluation. The mid-term evaluation covers the period from February 1, 2014 to September 30, It examined the relevance, effectiveness and efficiency of the NL SUPPORT Unit to meet accountability requirements and additionally to inform program decision-making. Due to the SUPPORT Unit s early stage of evolution, the evaluation focused on the deployment of the unit s six core functions and short-term progress toward operational objectives. The purpose of the NL SUPPORT Unit is to create resources that facilitate Patient-Oriented Research (POR) in consultation with local stakeholders to provide patients with the latest and most innovative practices, therapies and policies for improved health outcomes and enhanced patient experiences. The new geographical unit was established in Newfoundland and Labrador following the submission of a successful business plan to the Canadian Institute of Health Research for federal funding and confirmation of matched jurisdictional funding from the NL provincial government and IBM. NL SUPPORT was designed using an integrated knowledge translation model to train health system users and to be directly involved in local research decision-making. The NL SUPPORT Unit is still in early stages of being implemented. The emphasis was placed on planning, design, partnership development and foundational elements of deployment. During the first two years of operation, NL SUPPORT has formed a collaborative leadership and governance structure to move forward on a jurisdictional business plan founded upon six core functions: 1. Data platforms and services; 2. Methods support and development; 3. Health systems research, implementation research and knowledge translation; 4. Pragmatic / real-world clinical trials; 5. Career development in methods and health services research; and 6. Consultation and research services. The evaluation found that NL SUPPORT is relevant to addressing local needs and priorities for patient-oriented research. Key findings include: Analyses of healthcare expenditures in Canada reveal that Newfoundland and Labrador has a lot to gain from evidence-based research embedded in healthcare delivery and patient care. Healthcare expenditures in NL for is projected to be over $3 billion, with the highest healthcare spending per person among the provinces, representing approximately 28% more than the rest of Canada.1 There is alignment between NL SUPPORT goals and objectives with those of the Government of Newfoundland and Labrador. Alignment is strongest in the shared priority to refocus healthcare providers and researchers on realizing outcomes and incorporating input from healthcare system users. NL SUPPORT encompasses the elements and activities required to address areas in need of support for POR in its jurisdiction. These include: enhancing access to health utilization data; developing POR tools and supports; methodological training; supporting applied health implementation research and knowledge translation; increasing collaborations among stakeholders on common interests; and facilitating patient involvement in research. Stakeholder representation to be further targeted includes: indigenous populations, healthcare providers and associations (e.g., NLMA, ARNNL, PANL), advocacy groups, broader demographics (younger citizens, citizens living in rural communities, citizens with varying levels of types of education, marginalized citizens), and deeper infiltration of key partner organizations (such as Regional Health Authorities). Design and delivery found to be effective include: Governance and leadership team appear to be experienced, knowledgeable and effective at leveraging buy-in for POR. This is achieved by fostering integration, respect and mentorship among patients, researchers, health practitioners and policy members. Priority setting processes are believed to have been well organized and effective at identifying local needs and aligning priorities using a broad community engagement approach to solicit interest and priorities for health system reforms. NL SUPPORT employed a multi-modal engagement approach including: eight town halls to identify priorities for health research that were translated into funding call for applications; an online needs assessment survey targeting healthcare professionals, researchers, students, decisionmakers and patients to identify training needs and used to develop NL SUPPORT s Training and Capacity Building Plan; and successful engagement of patients for an advisory council through advertisements in the newspaper, intercept engagement at shopping malls and coffee shops, word-ofmouth and some ad hoc social media outreach. Peer review processes for research grant applications in 2016 included provisions for methodological support for implementing patient engagement after submitting an expression of interest. The inclusion of patient representatives in the 2015 round further strengthened the review process. NL SUPPORT trainings and webinars are believed to have enhanced capacity and capabilities by providing meaningful opportunities for patients and the public to participate and influence research that is of value to them. CIHR and NL SUPPORT successfully collaborated to remove administrative barriers to funding POR according to values of self-management by third party partners such as First Nation communities. Each of these six functions have been developed and implemented to varying degrees and timeframes. The results of this evaluation need to be interpreted with this condition in mind. 1 National Health Expenditure Database, CIHI, NL SUPPORT Unit Mid- Term Evaluation 1 2 Prepared by: R.A. Malatest & Associates Ltd.

4 Although NL SUPPORT is in its initial phases of implementation, it has achieved progress toward meeting operational objectives in most of its core functions: Data platforms and services processes have been established with Regional Health Authorities and the Newfoundland and Labrador Centre for Health Information making data easier to access. Some improvements are required for NL SUPPORT to provide one-stop access to linked health data platforms that integrates existing and new datasets in an efficient manner in partnership with existing organizations. Methods support and development services are currently being offered with access to specialized expertise in research methodologies to pursue POR and help lead reforms in response to locally-driven healthcare needs. Health systems research, implementation research, and knowledge translation have commenced with the majority of stakeholders reporting agreement that NL SUPPORT offers effective incentives for applied health systems research. NL SUPPORT has begun to support pragmatic real-world clinical trials including funding one through the NL SUPPORT grant competition in Another has been submitted to CIHR for funding consideration. Most key informants disclosed that NL SUPPORT increased their awareness of POR and built their capacity to execute POR. There is a high degree of satisfaction with stakeholder trainings and professional development activities. NL SUPPORT was instrumental in the development of the first POR graduate diploma program in Canada to be launched at Memorial University upon approval. Prior to the diploma being approved by Memorial University, NL SUPPORT staff will be offering a credited course they developed in POR. This program and its associated courses are aimed at medical doctors, allied health professionals, and university students to improve health outcomes and enhance patient experiences. There is a high degree of client satisfaction with consultation and research services by NL SUPPORT stakeholders. In terms of progress toward intermediate outcomes, NL SUPPORT is early in its influence on health research with most POR projects in early implementation with research findings expected sometime next year. The intermediate and longterm benefits of NL SUPPORT have not been realized at this point in time. The alignment and integration of the activities of NL SUPPORT s core functions, enhanced engagement of key stakeholders (patients, physicians, healthcare providers, decision-makers/policy-makers, and funders), stronger communication and marketing, and early wins will help to increase buy-in for POR and support the further implementation and achievement of intended results. 1.4 CONCLUSIONS The NL SUPPORT Unit is relevant. It is effectively addressing an ongoing need for POR and is achieving above average performance given it is early in its implementation phase. It has put in place the foundational elements to ensure the efficiency of future investment. Some improvements are required to further strengthen its design and delivery. There are eight areas in which effectiveness and efficiency can be improved, including: 1. Communications overall require improvement, including the need for further clarification between TPMI and NL SUPPORT (mandates and organizational structure); general awareness and marketing for NL SUPPORT; bi-annual face-to-face meetings of the patient advisory council rather than teleconference which limits stakeholder engagement; the inclusion of a communication portal to formalize communication (access to documents, decision and planning in real-time rather than keeping track of threads); and tailored consumable communications targeted to different stakeholders. Improvement areas include: Communication On-boarding Engagement Networking Investment Awareness of SPOR Networks Regional disparities 2. On-boarding of stakeholders requires improvement, including the simplification of information and diagrams, and a reduction in the number of documents provided. Patients would like the information to be more accessible to them and more consumable (simple PowerPoint, common every-day language and pictures). They see the benefit in presenting the information in person on a regular basis. 3. Patient engagement is still a work in progress with enhancements needed to support training and mentoring, as well as determining how best to recruit patients and sustain their interest over time without suffering burn out. Capacity development to learn about research and the role patients can play in it beyond providing initial insights is suggested. Capacity development for researchers on engaging with patients is similarly required to get more out of patient engagement and build value and understanding in patients contribution to the research process. 4. Stakeholder engagement (physician, healthcare provider, decision-maker and funder) is also a work in progress with enhancements needed to incentivise engagement perhaps through small scale research funding competitions (e.g., a few thousand dollars to conduct client impact survey). 5. Informal networking during the deployment phase has worked well but stakeholders would increasingly like to see the implementation of a formalized decision-making process specifying roles and responsibilities (including time commitment), and conflict resolution processes for increased transparency. 6. NL SUPPORT stakeholders would like to see increased investment in the popularization of research. Some stakeholders suggested that incentives be offered for best plain language presentation or poster at National SPOR conferences and jurisdictional events in NL. Similarly they would like to see research presentation events designed for general public consumption and discussion. 7. There is a perceived lack of awareness of the SPOR networks as a resource pool to draw upon. This may improve as stakeholders participate in the SPOR Summit in Ottawa. The summit was appreciated by members of the patient advisory council for fostering patient engagement, disseminating SPOR materials (e.g., evaluation materials and templates), and developing ongoing collaboration structures for evaluation with other jurisdictional SUPPORT units. 8. Disparities in access to POR exist between Newfoundland and Labrador Regional Health Authorities. Efforts to reduce these disparities by engaging in research in (and with) rural communities are recommended. NL SUPPORT Unit Mid- Term Evaluation 3 4 Prepared by: R.A. Malatest & Associates Ltd.

5 2. BACKGROUND Canada s Strategy for Patient-Oriented Research (SPOR) is a pan-canadian research strategy dedicated to improved healthcare outcomes through reforms that are locally-driven and fostered by engaged users of the healthcare system. Key components of this strategy are the National Research and Knowledge Translation Network and nationally distributed specialized support units called Support for People and Patient-Oriented Research and Trials (SUPPORT) Units. Each of these components is funded through cross-matching commitments characterized by public-private partnerships between the Canadian Institutes for Health Research (CIHR), jurisdictional, and private-sector partners. In order to meet funding requirements a mid-term and final evaluation must be undertaken. These evaluations set the future direction of these jurisdictional programs and help to inform the national agenda regarding ongoing funding. This report constitutes the mid-term evaluation of the Newfoundland and Labrador SUPPORT Unit, a methodological research support centre that trains, aligns, and prepares a range of healthcare stakeholders including patients, policy-makers, researchers, funders and health-care professionals. C O R E 6 I O N S F U N C T Mid Term Evaluation of the Newfoundland and Labrador SUPPORT Unit The NL SUPPORT has formed a collaborative leadership and governance structure to form and execute its jurisdictional business plan founded upon six core functions: 1. Data platforms and services; 2. Methods support and development; 3. Health systems research, implementation research and knowledge translation; 4. Pragmatic / real-world clinical trials; 5. Career development in methods and health services research; and 6. Consultation and research services. R.A. Malatest and Associates Ltd. was hired to conduct a mid-term evaluation to set the future direction of NL SUPPORT and help inform the Canadian Institutes for Health Research s (CIHR) national agenda regarding ongoing funding for SUPPORT Units. The mid-term evaluation examined the relevance, effectiveness and efficiency of NL SUPPORT during the initiation, design and development phases of the research centre, encompassing February 2014 to September THE NEWFOUNDLAND AND LABRADOR SUPPORT UNIT The purpose of the NL SUPPORT Unit is to create resources that facilitate patient-oriented research in consultation with local stakeholders to provide patients with the latest and most innovative practices, therapies, and policies for improved health outcomes and enhanced patient experiences. The objectives of NL SUPPORT are to: Identify and address the needs of patients and knowledge users by facilitating research; Provide specialized and multidisciplinary methodological expertise in patient-oriented research and its application; Assist decision makers and investigators to identify and design research studies, conduct bio statistical analyses, manage data, provide and teach project management skills, and ensure studies meet regulatory standards; Advance methods and training in comparative effectiveness research and develop the next generation of methodologists; and Provide timely access to data including linked datasets and integrate existing or new databases. 2 The evaluation examined four main research questions: 1. Does the NL SUPPORT Unit program address local needs and priorities for Patient-Oriented Research (POR)? a. To what extent is there alignment between the NL SUPPORT Unit program and jurisdictional priorities, roles and responsibilities? 2. To what extent does the NL SUPPORT Unit offer efficiencies and economies? a. To what extent has the NL SUPPORT unit enhanced the capacity for patient-oriented research? b. To what extent has the Unit mobilized existing expertise? c. Are there alternative ways to deliver research services and products to achieve the same or better results? d. What are the perceived benefits and challenges of management and implementation structures? e. To what extent are diverse resources aligned, integrated and coordinated? f. What are the most effective and economical patient recruitment strategies for patient-oriented research? 3. To what extent has the NL SUPPORT Unit achieved progress toward operational objectives in each of its core six functions (degree of satisfaction, challenges and suggested improvements)? 4. To what extent has NL SUPPORT collaborated with SPOR networks and other jurisdictional SUPPORT Units to ensure learning and sharing of best practices? The NL SUPPORT Unit is in its third year of operation. It has completed its initiation, design and development phases respectively. Efficiencies & Economies Operational Objectives 2 Information retrieved online on September 3, 2016 from: Address Local Needs Collaboration NL SUPPORT Unit Mid- Term Evaluation 5 6 Prepared by: R.A. Malatest & Associates Ltd.

6 3. METHODOLOGY A mixed-method research design was used to examine the relevance, effectiveness, and efficiency of the NL SUPPORT Unit from six key stakeholder perspectives involved in patient-oriented research. These perspectives include: the NL SUPPORT team and staff; patients/community members; policy-makers/decision-makers; healthcare providers; researchers; and private sector funders. Three lines of evidence were triangulated to answer each of the four research questions and provide insight into the future direction of NL SUPPORT. 3.1 CHARACTERISTICS OF THE NL SUPPORT NETWORK Six key stakeholder perspectives involved in patient-oriented research were identified and mapped within the NL SUPPORT network to gain a sense of both the size and characteristics of the network at this developmental stage of the program. Table 3-1 NL SUPPORT Network Perspective Size Characteristics NL SUPPORT team/ staff 12 Experienced multidisciplinary team. Approximately half of the key leadership of NL SUPPORT have worked closely together for a number of years. Patients 22 Members engaged in the Patient Advisory Council, with some patients also participating in research projects. This group represents a highly educated subgroup of the population, of which over a third have a healthcare background. The majority are female over the age of 65 years old who did not know one another prior to joining the council. Researchers 74 Diverse group including researchers who are funded by NL SUPPORT, who have attended capacity building training, received supports (including funding, methodological consultation, data analysis services), and/or who are students. Largely an unconsolidated group with few connections to one another other than TPMI and key NL SUPPORT staff emerging as connectors in the network. Decision-makers/policy-makers 4 Decision-maker and policy-makers represent key senior level officials in Regional Health Authorities and the Provincial Government, Department of health and community services. No elected municipal or provincial decision-makers are participating in the network at this point in time. Healthcare providers 5 Experienced interdisciplinary group. Private sector funder 2 The network includes two representatives from one private sector funder (IBM). Total 119 The characteristics of the NL SUPPORT network shown in Table 3-1 do not represent mutually exclusive categories. Stakeholders may represent more than one network perspective and self-identify with one category more than another differently from what is represented above. As the network is a living entity, this representation is but a snapshot of the network s existing size, diversity, and connectivity Key Informant Interviews Malatest selected nineteen interviewees from the list of NL SUPPORT network members provided to them, reflecting a broad range of perspectives involved in patient-oriented research. The sampling plan included a purposeful sample of: 5 members of the NL SUPPORT team; 4 researchers (including 2 student researchers); 4 decision-makers, policy-makers and funders; 3 healthcare providers; and 3 patients. Selected key informants were recruited by to inform the interviewee of the upcoming project and invite them to participate in a 45 minute telephone interview. Respondents who reported being interested and available to be interviewed during the data collection period were scheduled. Interviews took place between November and December, Of those initially selected, one interviewee was not available during the data collection time period and another declined to participate as they had limited experience with NL SUPPORT and did not feel they could provide insightful information to the evaluation. These two pre-selected interviewees were replaced with participants from the same stakeholder perspective in accordance with the sampling plan. In advance of each interview, the interviewee was provided with one of three semi-structured interview guides so that they could consider their responses to the questions. Three semi-structured interview guides were developed: one for NL SUPPORT team members, one for researchers, and one for partners (including decision-makers, policy-makers, funders, healthcare providers and patients). Each guide addressed 4-5 themes grounded in the research questions. Themes included but were not limited to program relevance, strategic alignment, efficiencies and economies, barriers, changes in the social and learning environment, and improvements. See Appendix A for interview guides. 3.2 ONLINE SURVEY The CIHR SUPPORT logic model was adopted by NL SUPPORT. It was used to identify 18 outcome indicators to measure the extent to which the research center had reached intended goals and objectives. A 5-minute online survey was developed. It captured the extent to which stakeholders agreed with affirmative statements. Indicators examined included: a. stakeholder beliefs regarding patient-oriented research; b. interaction patterns among network stakeholders; c. effectiveness of the programs core functions; and d. stakeholder capabilities. See Appendix B for online survey and aggregate results. The draft survey was validated with members of NL SUPPORT, programmed for online administration, and extensively tested for online and mobile device self-completion. All NL SUPPORT network stakeholders available during data collection were sent an invitation containing a secure link to the survey and a unique password to access the survey. The survey was accessible online between October 18, 2016 and November 4, During that time, two reminder s were sent to network stakeholders who had not completed the survey. Overall, 69 of 108 available network stakeholders completed the survey, representing a response rate of 64%. NL SUPPORT Unit Mid- Term Evaluation 7 8 Prepared by: R.A. Malatest & Associates Ltd.

7 3.2.1 Document Review A systematic analysis of related secondary documents was used to develop primary data collection tools and to contextualize primary data collection findings where available. 3.3 ANALYSIS 3.4 STRENGTHS AND LIMITATIONS OF THE EVALUATION Once data collection was complete the Consultant cleaned the quantitative data and compiled it into a SPSS database for analysis. To protect the anonymity of NL SUPPORT stakeholders, all identifying information was removed from the database (e.g., name, address). Further, data was analyzed and reported at the network level as reporting responses by respondent type could compromise the anonymity of survey respondents due to small sample sizes. Key informant interviews were analyzed thematically to respond to the four evaluation research questions and triangulated with the online survey results and document review. The synthesis of multiple lines of evidence placed greater confidence and emphasis on findings where all lines of evidence aligned Strengths Early stage of development: The evaluation was conducted at a time early enough to collect real-time feedback on the NL SUPPORT structures and processes to set the future direction of NL SUPPORT and help inform the national agenda for SUPPORT units. Coverage of stakeholders: All NL SUPPORT stakeholder perspectives were included in primary data collection methodologies (key informant interviews and an online survey). This included patients, healthcare providers, policy-makers and decision-makers, researchers, private sector funders, and the NL SUPPORT team/staff Limitations Early stage: The analysis is mainly limited by the very early stage of the operations of NL SUPPORT. Thus, the interview findings should only be taken as early considerations that may affect the future direction of the unit. Potential bias: In mapping the NL SUPPORT Unit the consultant collected administrative lists and addresses from stakeholders who have participated in NL SUPPORT activities, committees or trainings. The consultant relied on the NL SUPPORT Unit s categorization of network members to select the appropriate network perspectives as detailed in the methodological design. The intention was to select respondents who had enough experience and interaction with the NL SUPPORT Unit to be able to offer opinions on all key indicators of the interview guide. It is possible that the opinions and perspectives of the sample group were biased due to closeness with key leadership staff of the NL SUPPORT Unit. However, the possibility of this limitation is believed to be minor given the candid nature of interviewees responses. Sample bias: While the overall response rate for the online survey was 64%, it over-represents researchers and NL SUPPORT team/staff. Responses under-represent the views of patients/public, policy-makers/ decision-makers, and healthcare providers. Absence of comparison group: The evaluation did not include a comparison group of key POR stakeholders not on administrative lists to examine program relevance and distinctions in the quantitative rating questions on key performance indicators. Depth of inquiry to evaluation questions: The evaluation methodology relied heavily on key informant interviews to provide a rich understanding of program relevance, effectiveness and efficiency. Evidence from these open-ended questions were then triangulated with quantitative rating questions that assisted the study team in understanding the general tenor of responses and aided in the roll up of data for analysis. Validation process of evaluation findings and conclusions: All evaluation tools and findings were presented to the NL SUPPORT s Scientific and Administrative Directors to provide the opportunity to challenge evaluation findings, provide additional evidence and/or clarify contextual findings. NL SUPPORT Unit Mid- Term Evaluation 9 10 Prepared by: R.A. Malatest & Associates Ltd.

8 4. ADDRESSING LOCAL NEEDS AND PRIORITIES FOR POR (Question 1) Analyses of healthcare expenditures in Canada reveal that Newfoundland and Labrador has a lot to gain from evidence-based research embedded in healthcare delivery and patient care. Healthcare expenditures in NL for is projected to be over $3 billion, with the highest healthcare spending per person among the provinces, representing approximately 28% more than the rest of Canada. 3 Its relatively small population size makes it an ideal testing ground to target change in healthcare provider practices whilst empowering the general population to seek the right care, provided by the right providers, in the right place, at the right time, resulting in optimal quality care. 4.1 ALIGNMENT WITH JURISDICTIONAL PRIORITIES The following results from the online survey demonstrate the relevance of POR in guiding health system reforms in Newfoundland and Labrador: Over 90% of respondents believe that patientcentered research is a way to improve the healthcare system, patient experiences, and health outcomes. Over 85% of respondents agree that patientoriented research improves the quality of research and increases the use of research into evidence. 70% of respondents believe that patients influence and accelerate decision-making, translation and uptake of new practices. There is alignment between NL SUPPORT goals and objectives with those of the Government of Newfoundland and Labrador. Alignment is strongest in the shared priority to refocus healthcare providers and researchers on realizing outcomes and incorporating input from healthcare system users. Evidence for this can be found in Newfoundland and Labrador s Department of Health and Community Services Strategic Plan ( ); Regional Health Authority Strategic Plans; Strategic Health Workforce Plan; the NL Primary Healthcare Framework; NL Policy Framework for Chronic Disease Prevention and Management; and other provincial drivers for change such as Choosing Wisely NL steering committee. Many key informants however call into question the adequacy of resources to meet the expressed commitment of key stakeholders for patient-centered care and the associated research necessary to drive health system reforms. Key informants illustrate the aforementioned point in the following way: The expenditures devoted to this program are quite low given the magnitude of healthcare practices that need to be evaluated. NL SUPPORT gets much less than 1% of healthcare expenditures to try to address the problem with the most promising evidence-based tools we have. What we need is more funding to make a difference. While the government is putting more money in applied science, the main challenge is the sheer volume of interventions that need to be evaluated. We are talking about thousands of interventions that need evaluation against best practice standards for known and actionable problems in Newfoundland and Labrador. Not enough money is being dedicated for research and development and evaluation. NL SUPPORT encompasses the elements and activities required to address areas in need of support for POR in its jurisdiction. These include: enhancing access to health utilization data; developing POR tools and supports; methodological training; supporting applied health implementation research and knowledge translation on jurisdictional POR priorities; increasing collaborations among stakeholders on common interests; and projects facilitating community involvement in research. 4.2 ENGAGEMENT AND REPRESENTATION NL SUPPORT adopted a broad multi-modal community engagement approach to foster interest in POR and solicit priorities for health system reforms from the general population. Their initial approach took the form of town hall sessions in eight rural and urban communities across the province. 4 In line with most research on patient priorities, timely and easy access to healthcare emerged as the top priority in the town hall sessions. Chronic disease prevention, health promotion and efficient use of healthcare expenditures were among the leading priorities for POR. These topics were directly aligned with jurisdictional priorities identified by a local working group composed of decision-makers, healthcare professionals and researchers and were used to set the research agenda for NL SUPPORT s funding call for applications. POR requires ongoing stakeholder engagement in order to diversify representation and intensify engagement. Key informants in NL SUPPORT mentioned the need to continue to target the following: 1. indigenous populations; 5 2. healthcare providers and their associations (e.g., NLMA, ARNNL, PANL); 3. broader demographics (e.g., younger citizens, citizens living in rural communities, citizens with varying levels and types of education, marginalized citizens); and 4. deeper infiltration of key partner organizations through organizational champions (e.g., Regional Health Authorities). 4 The eight town halls gathered insight from 68 citizens across NL. 5 See page 21 for engagement with indigenous population engagement spotlight. (Efficiencies and Economies section) 3 National Health Expenditure Database, CIHI, NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

9 These efforts will be complemented by implementing an environmental scan. The scan will use key informant interviews to explore POR capacity and the training needs of different schools and departments in the region. Further, training and capacity building activities will continue to be evaluated using confidential and anonymous online survey tools after each activity. Findings will be used to identify future training needs and to improve existing training activities. 4.3 BENEFITS AND CHALLENGES The NL SUPPORT Unit is in the early days of stakeholder engagement in all four Regional Health Authorities due to the broad geographic landscape that includes many small rural communities with limited access to healthcare services. Some groundwork to build awareness of POR beyond the Avalon Region has been established with all four Regional Health Authorities invited to participate in the NL SUPPORT steering committee, 6 and with some (albeit limited) involvement in funding applications received from outside the Eastern Regional Health Authority. Strategies to engage stakeholders in all regions of the province are required to address the unique needs and priorities. This is particularly true in small rural communities that experience different barriers to accessing healthcare services and have poorer overall health outcomes. Many key informants engaged in POR mention that it is hard to get patients engaged in a meaningful way and to maintain their interest in the research process. A researcher mentioned challenging himself for the first time. To really make research interesting and relevant to healthcare users and added this is all new to me and it can be frustrating as much as it can be rewarding. One key informant defined the key challenge in the following way: Everyone has to learn to step out of their comfort zone a little and give up something they need in order to work together. It is too early to see if POR will have the impact we are hoping for. For now we are all learning about patient engagement and experimenting with it. I can however say that I intend to use NL SUPPORT s knowledge translation services when my project reaches that stage. The intent of POR is to have an impact on the health system so my goal is not simply to publish results and hope for change. I want to leverage the relationships created by NL SUPPORT to see the results used by decision-makers. The leading challenges to POR according to key informants include: time, lack of a shared non-technical vocabulary, lack of enabling infrastructure or support within partners organizations, and an overall mindfulness to frame the research problem and methodological approach in an accessible fashion to enable collaboration over time. Many key informants described how their experiences in POR have improved over time: I used to just listen to what others would say and provide an ad hoc reaction when asked. I really didn t know what I was doing at the meetings and felt out of place. I gave it a try and started by making comments and little suggestions. That was a start. It helped me and others see what I could bring to the table. Now I get it a little more, and I have the confidence to say more. I am helping to shape the messaging for Choosing Wisely and I think what I am doing is making a difference. At first I didn t feel like I had any influence on decisionmaking processes in the committee, but now I have grown into my role. Another key informant suggested that: Opportunities for meaningful collaboration can happen if we learn to watch each other better because body language and what is not said when we work together is so crucial in POR. There was consensus that all stakeholders were learning a lot from one another and building their relationships. There was also consensus among key informants that the most significant benefits to POR were yet to be realized. 6 All RHAs have been invited to participate in the NL SUPPORT steering committee but not all have accepted the invitation at this point in time. NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

10 5. EFFICIENCIES AND ECONOMIES (Question 2) The NL SUPPORT Unit is still in the early stage of being implemented. Emphasis has been placed on planning, designing, developing partnerships and putting in place the foundational elements required for deployment. During the first two years of operation, the NL SUPPORT Unit has formed a collaborative leadership and governance structure to move forward on a jurisdictional business plan founded upon six core functions. Overall, key informants described governance and leadership teams as experienced, knowledgeable, and effective at leveraging buy-in for POR. While structural efficiencies and economies exist at an administrative level within the NL SUPPORT Unit, there is a degree of stakeholder confusion that hinders some connections from forming. Key informants describe the organizational structure of NL SUPPORT as complex and confusing. They are unclear why NL SUPPORT is tied to TPMI when other geographical SUPPORT units are standalone organizations. There was an undertone to several key informant interviews (supported by findings from the online survey) that suggests perceived favouritism for funded projects under a senior member of the leadership team. Only 36% of respondents agreed that the NL SUPPORT Unit has effective processes to align and address potential conflict. The majority (61%) of respondents reported being neutral on this issue; however, the strength of opinion among those who expressed their views suggests that clarification is warranted. TPMI and NL SUPPORT mandates; their organizational structures; and their decision-making processes would benefit from a more formalized and transparent approach. Key informants would like roles and responsibilities clarified and conflict resolution processes developed. 5.1 ORGANIZATIONAL STRUCTURE The organizational structure of the unit within the Translational and Personalized Medicine Initiative (TPMI) was described as efficient by NL SUPPORT staff and funders. Given that TPMI and NL SUPPORT emerged out of the same federal, provincial and private sector priorities for the health sector, both initiatives were linked by shared projects, personnel, and partner agency collaborations within the same time period. The main advantage of housing the NL SUPPORT Unit within TPMI is the ability for both initiatives to be physically located in the Craig L. Dobbin Research Center. Being in the same building is thought to save time and facilitate collaboration between stakeholders, including NL SUPPORT s sister organization, the Centre for Health Informatics and Analytics (CHIA) that supplies the computational infrastructure for POR through administrative datasets. CHIA and NL SUPPORT are as a result intertwined, and rely on each other with personnel that co-manage both organizations for coherence and financial accountability. These streamlined structures help foster centralized and integrated resources that leverage POR in TPMI projects, and opens communication lines between network members who did not know one another prior to establishing NL SUPPORT. Key informants articulated the aforementioned issues in the following way: I really don t understand the value of NL SUPPORT being housed in TPMI. It takes people a long time to understand the organizational structure of NL SUP- PORT due to its link to TPMI. Roles and responsibilities are blurred; key concepts get blurred and get used interchangeably when they shouldn t. The organizational charts of TPMI and NL SUPPORT are way too complicated. It is hindering people from getting to know one another. There is one person who consistently uses TPMI and NL SUPPORT interchangeably and this is a real problem. Efforts are needed to correct this issue. Key informants illustrate the aforementioned points in the following way: TPMI and NL SUPPORT bring researchers, health professionals and decision-makers together. They do an excellent job of connecting stakeholders to one another. By aligning TPMI and NL SUPPORT priorities we are able to keep communication channels open and make more connections to each other s work. Before TPMI and NL SUPPORT I wasn t aware of others. I didn t know who they were or what they were doing. Now we can see where we can make connections in order to build momentum. 5.2 PLANNING PROCESSES Priority setting processes were described as well-organized and effective at identifying local needs and aligning priorities using a broad multi-modal community engagement approach. Planning structures and processes are in their early stages of design and implementation. Results of the following sections need to be interpreted with this condition in mind. Summative findings from the online survey reveal that: 88% of respondents agreed that the NL SUPPORT Unit fosters integration, respect and mentorship among patients, researchers, health practitioners and policy members. 87% of respondents agreed that the NL SUPPORT creates meaningful opportunities for patients and the public to participate and influence research that is of value to them. NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

11 5.3 COMMUNITY ENGAGEMENT PROCESSES Broad community engagement was planned in conjunction with a local Strategy for Patient-Oriented Research (SPOR) network and Primary Healthcare Research and Integration to Improve Health System Efficiency (PRIIME) which similarly had patient engagement as core activities. The significant time and budget to plan and execute eight town hall consultation sessions in urban and rural communities was reduced by SPOR network collaboration. Further, lessons learned from these engagements were co-constructed and will be retained for future application by both networks. 5.5 MATCHING PATIENTS TO RESEARCH PROJECTS Members of the Patient Advisory Council (PAC) were recruited through advertisements in local newspapers, intercept approaches directed to the general population in malls and coffee shops, word of mouth, and through ad hoc social media outreach. Each engagement technique fostered lessons learned with different segments of the population. NL SUPPORT staff members were more likely to report the following as the important lessons learned from the public engagement initiatives: going to the public rather than expecting the public to come to you; recognizing that language matters (namely framing discussions on health concerns rather than soliciting information about research); and strong facilitation of discussions is essential. 5.4 TRAINING AND CAPACITY BUILDING PLANNING Planning processes associated with NL SUPPORT s Training and Capacity Building Plan included an online survey to identify skills and knowledge gaps in POR from local health professionals, researchers, students, and decision-makers. Members of the local training and capacity working group identified a sample of respondents from their respective organizations and were sent an invitation to participate in a training needs assessment survey in April 2016, resulting in 140 respondents. A similar survey was developed with and for patient advisors engaged in POR in July 2016, and was completed by all patient advisors. 7 Results from the online survey were translated into the Training and Capacity Building plan and used to develop webinars. NL SUPPORT trainings and webinars were widely believed to have enhanced capacity and capabilities by providing meaningful opportunities for patients and the public to participate and influence research that is of value to them. It is recommended that NL SUPPORT continue to build researcher capacity to learn about the different roles patients can play in research that extends beyond providing initial insights. The inclusion of patient advisors in the peer review process for research grant applications in 2015, for example, was said to strengthen the review process and build network capacity. Summative findings from the current online survey reveal that: 8 75% of respondents agreed that they were involved in meaningful engagement in the identification of priorities and relevance of research. 52% of respondents agreed that they were involved in meaningful engagement in grant writing and peer review processes. 57% of respondents agreed that they were involved in meaningful engagement in design and management. 58% of respondents agreed that they were involved in meaningful engagement in data collection and analysis. 57% of respondents agreed that they were involved in meaningful engagement in the implementation of research findings into practice/ policy. 45% of respondents agreed that they were involved in meaningful engagement in evaluating impact and report writing. The efficiency of one engagement approach over the other is, however, a matter of opinion with divergent opinions from key informants. Intercepts at malls and coffee shops were more likely to be cited by NL SUPPORT staff as effective methods of recruitment and engagement than other stakeholder types who expressed a certain level of discomfort about the appropriateness and effectiveness of these approaches. The patient engagement approach thus far has led to the successful engagement of 22 patients in the Patient Advisory Council. The NL SUPPORT Unit recruited and pre-screened each member in order to match them with research projects. Researchers, decision-makers, and policy makers describe this matching process as very efficient, reducing both time and money to engage patients in their research processes. Processes to sustain engagement, however, were mentioned by some key informants as potential barriers to the engagement of diverse patient perspectives. Key informants expressed waiting too long for CIHR to develop guidelines for patient honorariums to compensate participants for their time and travel expenses. Clearly defining the roles and expectations of patient advisors (e.g., amount and length of time commitment) in research activities (including capacity building training) is similarly identified as a desired practice to sustain engagement. This will ensure that patients understand the commitment they are making so they feel good about their engagement and avoid the conflict and patient burnout associated with over involvement in research processes or committees. This was of particular concern for patient advisors who are 65 years and older with severe health problems. 9 Lastly, patient advisors identified the need to streamline on-boarding processes in order to avoid overwhelming new members with complicated information and diagrams. Patients suggest the substantial reduction of documents shared with new members of the PAC, and the simplification of materials to make them more accessible and consumable (e.g., simple PowerPoint, use of common every-day language, pictures). 7 At the time of survey administration, there were 25 patient advisors who completed the survey (100% response rate). 8 While the overall response rate for the online survey was 64%, it over-represents researchers and NL SUPPORT team/ staff. Responses under-represent the views of patients/public, policy-makers/decision-makers, and healthcare providers. 9 The PAC is largely made up of seniors 65 years of age and older with medical conditions. NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

12 Key informants illustrated some of the aforementioned points with the following quotes: I was completely overwhelmed with the amount of information provided to me when I joined the PAC. The documents were very technical, very confusing, and not clearly written. There was so much to read that I must admit that I couldn t get through it. I didn t have the time or the patience. I was already giving them my time by participating in the meetings never mind spend hours on my own trying to make sense of a ton of information. So the people involved in NL SUPPORT are great. They made me feel welcome and were really interested in my experiences and what I have to say. When I looked at the materials I was really surprised they wanted me to go through it. There has got to be a better way. Please, walk me through it, give me an hour presentation and ask me if I have any questions. informants that perhaps small scale funding competitions (e.g., a few thousand dollars to conduct client impact surveys for example) for healthcare professionals in rural communities may incentivize clinical engagement and be relevant to reducing disparities in POR in rural communities. This may be a first step at redressing perceived disparities in POR in the region. Key informants illustrate some of the aforementioned points with the following ways: There is only so much I can do to help build connections as one individual within my organization. I share the information about NL SUPPORT training and webinars to my staff and some have started to participate. That is a great start. This is however still limited exposure with no ties being fostered. I feel like I could use supports and assistance to enable engagement deeper in within my organization from NL SUP- PORT. This task should not be solely my burden. The whole rural health experience is being missed. This is an important missed opportunity that I hope will be addressed by NL SUPPORT in the near future. 5.6 MATCHING DECISION AND POLICY MAKERS TO RESEARCH PROJECTS Stakeholder engagement (decision-makers, policy makers, and healthcare providers) is a work in progress. At present there appears to be a lack of engagement activities that match decision makers and policy makers to research projects at the early planning stages. Persistent engagement of key stakeholders over the past two years, however, has resulted in all four Regional Health Authorities participating in NL SUPPORT s steering committee to various degrees. The Unit intends to leverage Choosing Wisely Newfoundland and Labrador to engage other decision-makers (e.g., municipal governments), healthcare providers and their associations. At the time of key informant interviews, senior officials within Regional Health Authorities were identified as the sole or main connector between NL SUPPORT and partner organizations. Key informants from partnering organizations mentioned that while there is buy-in for POR within their organizations and a clear mandate for engagement, there is a lack of support on how to go about creating connections and communication processes within and between organizations that will enable close collaboration. It is understood that close collaboration is required to change existing relationships; identify what is working; and determine why some initiatives fall short of expectations. When key informants were asked to expand on what was not taking hold as well as they would have thought, many key informants mentioned the lack of equal access to POR throughout Newfoundland and Labrador. There is a perception among Regional Health Authorities that regional disparities in POR stem from a greater investment in the Eastern Health jurisdiction. It was suggested by key 5.7 PARTNERSHIP DEVELOPMENT The NL SUPPORT Unit is well underway to building partnerships with key stakeholders involved in POR. Summative findings from the online survey reveal that: Over 70% of stakeholders agreed that they are valued partners in POR and are able to influence decision-making process in the projects/committees they are involved in. Over 80% of stakeholders agreed that reciprocal relationships are forming among stakeholders and they feel optimistic about their current and future projects. Areas in which effectiveness can be improved have been discussed in previous sections where applicable. A more general area in need of continued improvements is communication. Leading suggestions identified by key informants include: 1. awareness and marketing of NL SUPPORT (including improvements to the NL SUPPORT website and social media strategy); 2. bi-annual face-to-face meetings rather than teleconferences which limit stakeholder engagement; 3. the inclusion of a communication portal to formalize communication and decision-making processes including access to documents (from job descriptions, communication templates, training materials, workgroup meeting minutes, etc. ); and 4. decision and planning in real-time rather than keeping track of threads, and tailored consumable communication targeted to different stakeholders. NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

13 Another area identified for improvement is increased investments in the popularization of research. Key informants would like to see incentives for best plain language presentations or posters at National SPOR conferences and jurisdictional events in Newfoundland and Labrador. Similarly they would like to see research presentations and events designed specifically for public consumption and discussion. There was a perceived lack of awareness of other SPOR networks as a resource pool to draw upon, but this was said to be improving after the 2016 SPOR Summit in Ottawa. Individuals who participate in these large events felt inspired to forge forward in their projects with a deeper appreciation for the work they are doing. They mentioned feeling part of something important and meaningful. 6. PROGRESS TOWARD OPERATIONAL OBJECTIVES (Question 3) Although the NL SUPPORT Unit is in its initial phases of implementation, it has achieved progress toward meeting operational objectives in most of its core functions. Key informants illustrate the importance of this with quotes such as: We are fighting a culture of entitlement where healthcare expenditures are not being based on need. We need to change the way we engage clinicians and the public. We need to empower both of them and frame our message differently. We need to draw people in with clear and relevant message that has meaning to them. We need to become the trusted third party and brand our message better. Our behavior modification models don t work; we need to rethink how we do this entirely. Maybe presentations of research findings are not the way to go. What do we know about knowledge translation? I am no expert, but I think patients need to be the face of this movement. They need to frame the terms of reference and be the main drivers for change. I see some researchers teaching them our terms of reference as if they need to catch up but it is really the other way around. 6.1 DATA PLATFORMS AND SERVICES ARE OPERATIONAL Data platforms and service processes are established with Regional Health Authorities and the Newfoundland and Labrador Centre for Health Information making data easier to access. To provide one-stop access to linked health data platforms that integrate existing and new datasets in an efficient manner, NL SUPPORT in partnership with existing organizations will require some improvements. Summative findings from the online survey reveal that: 67% of respondents agreed that the NL SUPPORT unit implemented a strategy that makes the best use of enhanced data in NL. This includes a broad directory of data sources that are useful to POR. 49% of respondents agreed that NL SUPPORT unit provides one-stop access to linked health data platforms and integrates existing and new datasets in an efficient manner in partnership with existing organizations. Engagement Spotlight NL SUPPORT staff members were very proud to discuss their engagement with First Nation communities in Newfoundland and Labrador. Knowing that self-management of POR is essential to First Nation engagement; they sought to develop a funding application by developing working relationships with leaders to identify POR priorities in their communities. The 2016 CIHR Summit in Ottawa provided resources and tools to support this type of engagement. NL SUPPORT staff worked with CIHR to remove the main barriers to collaboration, allowing the First Nation community to self-administer funds for POR in their community. 6.2 METHODS SUPPORT AND DEVELOPMENT ARE OFFERED Methods support and development services are currently being offered with access to specialized expertise in research methodologies to pursue POR and help lead reforms in response to locally-driven healthcare needs. Summative findings from the online survey reveal that: 6.3 HEALTH SYSTEMS RESEARCH, IMPLEMENTATION RESEARCH, AND KNOWLEDGE TRANSLATION HAVE COMMENCED 80% of respondents agreed that the NL SUPPORT Unit provides access to specialized expertise in research methodologies (e.g., epidemiology, biostatistics, economic analysis, mixed methods, ethics) of international calibre already available in NL to pursue POR and help lead reforms in response to locally-driven healthcare needs. Health systems research, implementation research, and knowledge translation have commenced. Summative findings from the online survey reveal that: 83% of respondents agreed that the NL SUPPORT Unit provides effective incentives for applied health systems research (e.g., funding, awards). NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

14 6.4 PRAGMATIC / REAL-WORLD CLINICAL TRIALS ARE EMERGING The NL SUPPORT Unit has begun to support pragmatic real-world clinical trials including funding one through the NL SUPPORT grant competition in Another has been submitted to the CIHR for funding consideration. 7. DEGREE OF INTERACTION WITH SPOR NETWORKS AND SUPPORT UNITS (Question 4) 6.5 CAREER DEVELOPMENT IN METHODS AND HEALTH SERVICES RESEARCH IS OPERATIONAL 6.6 CONSULTATION AND RESEARCH SERVICES ARE OPERATIONAL Most key informants disclosed that the NL SUPPORT Unit increased its awareness of POR and increased its capacity to execute POR. There is a high degree of satisfaction with stakeholder training and professional development activities. The NL SUPPORT Unit was instrumental in the development of the first POR graduate diploma program in Canada (to be launched at Memorial University upon approval). Prior to the diploma being approved by Memorial University, NL SUPPORT staff will be offering a credited course they developed in POR. This program and its associated courses are aimed at medical doctors, allied health professionals, and university students to improve health outcomes and enhance patient experiences. Summative findings from the online survey reveal that: 86% of respondents agreed that The NL SUPPORT Unit supports the training (researchers, clinicians, patient partners and managers) and the quality of evaluation processes for clinical trials and demonstration projects. 72% of respondents agreed that the NL SUPPORT Unit brokers and connects policy and practice communities by building capacity in areas that lack expertise. 86% of respondents agreed that they are satisfied with the supports and services offered by the NL SUPPORT Unit. In terms of progress toward intermediate outcomes, most POR projects are in the early implementation stage with research findings expected sometime in The intermediate and long-term benefits of the NL SUPPORT Unit have not been realized at this point in time. The alignment and integration of the activities of NL SUPPORT s core functions, enhanced engagement of key stakeholders (patients, physicians, healthcare providers, decision-makers/policy-makers, and funders), stronger communication and marketing, and early successes will help to increase buy-in for POR and support the further implementation and achievement of intended results. NL SUPPORT had limited interaction with SPOR networks and other jurisdictional SUPPORT Units to ensure learning and sharing of best practices within its first two years of deployment. The local SPOR network with the closest ties to NL SUPPORT is PRIIME, which collaborated on the planning and implementation of eight town hall consultation sessions in urban and rural communities. There is a perceived lack of awareness of the pool of SPOR resources to draw into POR. It is expected that the establishment of a communication portal may help resolve existing limitations as outreach strategies become more formalized. Some collaboration with the Nova Scotia SUPPORT Unit was established to support the peer review selection process of funding applicants. The NL SUPPORT Unit also participates in all of the national working groups including the Knowledge Translation Working Committee, the Training and Capacity Development Committee, the Patient Engagement Working Committee, and the National Evaluation Workgroup. These national working groups have regular interaction via teleconferencing, and share materials, tools, and best practices. Over 70% of respondents to the online survey agreed that the NL SUPPORT Unit catalyzes and supports collaboration on jurisdictional priorities and decision-making with other SPOR networks. The SPOR Summit in Ottawa in 2016 was well attended and appreciated by NL SUPPORT stakeholders, including patient advisors. A collaboration spotlight between the CIHR and the NL SUPPORT Unit reveals a successful removal of funding barriers to First Nation self-managed POR activities. In conclusion, there is room for improved interaction between the NL SUPPORT Unit and SPOR networks and other jurisdictional SUPPORT networks but progress on establishing collaboration structures and processes has been made. NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

15 8. CONCLUSIONS The NL SUPPORT Unit is relevant. It is effectively addressing an ongoing need for POR and is achieving above average performance given it is early in its implementation phase. It has put in place the foundational elements to ensure the efficiency of future investment. Some improvements are required to further strengthen its design and delivery. We recommend improvements in eight areas: 1. Communications overall require improvement, including the need for further clarification between TPMI and NL SUPPORT (mandates and organizational structure); general awareness and marketing for NL SUPPORT; bi-annual face-to-face meetings of the patient advisory council rather than teleconference which limits stakeholder engagement; the inclusion of a communication portal to formalize communication (access to documents, decision and planning in real-time rather than keeping track of threads); and tailored consumable communications targeted to different stakeholders. 2. On-boarding of stakeholders requires improvement, including the simplification of information and diagrams, and a reduction in the number of documents provided. Patients would like the information to be more accessible to them and more consumable (simple PowerPoint, common every-day language and pictures). They see the benefit in presenting the information in person on a regular basis. 3. Patient engagement is still a work in progress with enhancements needed to support training and mentoring, as well as determining how best to recruit patients and sustain their interest over time without suffering burn out. Capacity development to learn about research and the role patients can play in it beyond providing initial insights is suggested. Capacity development for researchers on engaging with patients is similarly required to get more out of patient engagement and build value and understanding in patients contribution to the research process. 4. Stakeholder engagement (physician, healthcare provider, decision-maker and funder) is also a work in progress with enhancements needed to incentivise engagement perhaps through small scale research funding competitions (e.g., a few thousand dollars to conduct client impact survey). 5. Informal networking during the deployment phase has worked well but stakeholders would increasingly like to see the implementation of a formalized decision-making process specifying roles and responsibilities (including time commitment), and conflict resolution processes for increased transparency. 6. NL SUPPORT stakeholders would like to see increased investment in the popularization of research. Some stakeholders suggested that incentives be offered for best plain language presentation or poster at National SPOR conferences and jurisdictional events in NL. Similarly they would like to see research presentation events designed for general public consumption and discussion. 7. There is a perceived lack of awareness of the SPOR networks as a resource pool to draw upon. This may improve as stakeholders participate in the SPOR Summit in Ottawa. The summit was appreciated by members of the patient advisory council for fostering patient engagement, disseminating SPOR materials (e.g., evaluation materials and templates), and developing ongoing collaboration structures for evaluation with other jurisdictional SUPPORT units. 8. Disparities in access to POR exist between Newfoundland and Labrador Regional Health Authorities. Efforts to reduce these disparities by engaging in research in (and with) rural communities are recommended. 9. APPENDIX A: INTERVIEW GUIDES Newfoundland and Labrador Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit Mid Term Evaluation NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

16 9.1 INTERVIEW GUIDE FOR THE SUPPORT TEAM NL SUPPORT is currently undergoing a mid-term evaluation of its services. This is a CIHR requirement; the results of the evaluation will set the future direction of NL SUPPORT and help inform the national agenda re ongoing funding for SUPPORT units. The focus of the evaluation will be on the relevance, effectiveness and efficiency of NS SUPPORT s six core functions: Data platforms and services Methods Support and Development Health Systems Research, Implementation Research and Knowledge Translation Pragmatic / Real-world Clinical Trials Career Development in Methods and Health Services Research Consultation and Research Services R.A. Malatest & Associates has been commissioned to implement the evaluation. The evaluation team would like to conduct telephone interviews with key NL SUPPORT stakeholders. Participation is voluntary. A member of the evaluation team will be contacting you in the near future to schedule an interview. Your input will be anonymous. The interviews will require minutes of your time and will be scheduled at your convenience. Your input will be amalgamated with that of other stakeholders to help improve NL SUPPORT s future service delivery. Again, participation is voluntary. If you have any questions about the evaluation, please contact Carmelle Goldberg, Lead Evaluator from R.A. Malatest & Associates Ltd. at c.goldberg@malatest.com or ext Do you have any questions before we begin? Contact Information First and Last Name: Position: Telephone Number: Date and Time of Interview: Strategic Alignment 1) Could you please describe your role and responsibilities within the NL SUPPORT Unit? How long have you been involved in patient-centred research and where does your primary expertise lie? 2) How does the SUPPORT Unit program address local needs and priorities for patient-oriented research? [Probe: Do the centre s jurisdictional priorities align with stakeholder need assessments? Has the SUPPORT Unit needs assessment reached a broad spectrum of the population, including all major stakeholder groups (advocacy/community groups, healthcare providers, seniors, disadvantaged groups, etc.)? Are there any perspectives missing?] 4) Are there enough opportunities and/or formal structures to address potential conflict between stakeholder priorities and perspectives that affect the strategic orientation of the centre? Efficiencies and Economies 5) Does the NL SUPPORT program provide efficiencies and economies? If so, what are they? [Probe: common contracts to streamline research processes; one-stop access to linked data platforms; research navigation; methodological expertise available to complete research projects in a timely manner; increased suitability/acceptability of research; risk mitigation; increased value and/or use of research findings; etc.] 6) What are the most effective and economical patient recruitment and retention strategies used by the unit (town halls, peer to peer recruitment, social media outreach, honorariums, reimbursement for time & travel)? 7) How effective has the unit been at engaging and retaining other key stakeholders (decision-makers, policy makers, healthcare providers, researchers, SPOR network members) in applied research? Barriers 8) Are there any barriers to patient or stakeholder engagement? [Probes: frustration with the length of training or length of research process; transportation, attendance barriers, extra time to complete research, incremental funding for patient engagement, token engagement or false appearance of inclusiveness?] 9) What are the impacts of these barriers on reaching the goals and objectives of the SUPPORT unit? [Probes: high absenteeism of stakeholders, lack of buy-in for research and/or results, missed deadlines, fewer scientific publications?] 10) Do you have potential solutions to these barriers? Changes in the Social and Learning Environment 11) Have there been any changes in the social and learning environment of key stakeholders implicated in the SUPPORT unit? [Probe: Are these changes valued, acknowledged, or incentivized? Are they sustainable?] 12) Have any best practices been identified in the initiation phase of the NL SUPPORT Unit? If so, please describe. Closing 13) Are there areas where improvements are needed? If so, please describe. 14) Is there anything else you would like to share about the NL SUPPORT Unit? Thank very much for sharing opinions and experiences. 3) To what extent are resources efficiently aligned, integrated and coordinated? Do you think changes are required to adapt to the implementation phase of the research centre? If so, what are they? NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

17 9.2 INTERVIEW GUIDE FOR RESEARCHERS NL SUPPORT is currently undergoing a mid-term evaluation of its services. This is a CIHR requirement; the results of the evaluation will set the future direction of NL SUPPORT and help inform the national agenda re ongoing funding for SUPPORT units. The focus of the evaluation will be on the relevance, effectiveness and efficiency of NS SUPPORT s six core functions: Data platforms and services Methods Support and Development Health Systems Research, Implementation Research and Knowledge Translation Pragmatic / Real-world Clinical Trials Career Development in Methods and Health Services Research Consultation and Research Services R.A. Malatest & Associates has been commissioned to implement the evaluation. The evaluation team would like to conduct telephone interviews with key NL SUPPORT stakeholders. Participation is voluntary. A member of the evaluation team will be contacting you in the near future to schedule an interview. Your input will be anonymous. The interviews will require minutes of your time and will be scheduled at your convenience. Your input will be amalgamated with that of other stakeholders to help improve NL SUPPORT s future service delivery. Again, participation is voluntary. If you have any questions about the evaluation, please contact Carmelle Goldberg, Lead Evaluator from R.A. Malatest & Associates Ltd. at c.goldberg@malatest.com or ext Do you have any questions before we begin? Contact Information First and Last Name: Position: Telephone Number: Date and Time of Interview: Relevance 1) How long have you been involved in patient-centred research and where does your primary expertise lie? 2) Is there a need for a specialized, multidisciplinary research service centre in Newfoundland and Labrador that will provide expertise and services to those engaged in patient-oriented research? What value does this research service center have for researchers? 4) How have you worked with patients, healthcare providers, policy makers & decision-makers as partners in your research? Which research processes have they been engaged in (e.g., identifying priorities or the relevance of research questions, grant writing and review processes, design and management, data collection and analysis, implementation of research findings into practice/ policy, evaluating impact & report writing)? 5) What were the benefits experienced by establishing meaningful patient/stakeholder engagement? [Probe: improved quality and relevance of research; increased networking opportunities, increased opportunities for field research; accelerated decision-making, translation or uptake of new practices?] Barriers 6) What were the most significant challenges associated with establishing meaningful patient/ stakeholder engagement? Did the NL SUPPORT Unit help you overcome these challenges? If so, how? 7) Are there any barriers to conducting patient-oriented research (e.g., length of time required to engage and complete research, implications for large number of collaborators on publications, reduced impact factor/influence of publications, decreased opportunities for publication)? What (if any) are the impacts of these barriers on career development or advancement? 8) Do the benefits of patient-centred research outweigh the challenges? If so how? Efficiencies and Economies 9) Does the NL SUPPORT program provide efficiencies and economies for your research projects? If so, what are they? [Probe: common contracts to streamline research processes; timely onestop access to linked data platforms; research navigation support; methodological expertise; risk mitigation strategies, etc.] 10) Were you satisfied with the services, products and supports provided by the NL SUPPORT unit? 11) Have funding opportunities (research, grants, fellowships, awards) created by the NL SUPPORT Unit increased your motivation or ability to conduct patient-oriented research? 12) Have these funding opportunities increased the calibre of researchers and students involved in applied health systems research, implementation research and/or knowledge translation? Improvements 13) Are there areas where the NL SUPPORT Unit can make improvements to better serve your needs? Thank you for sharing your opinions and experiences. 3) Has your involvement in the NL SUPPORT Unit changed your awareness, attitude or behaviours toward applied patient-centred research? If so, how? [Probe: have your experiences with patients increased the value of their experiential knowledge in the research process?] NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

18 9.3 INTERVIEW GUIDE FOR PARTNERS NL SUPPORT is currently undergoing a mid-term evaluation of its services. This is a CIHR requirement; the results of the evaluation will set the future direction of NL SUPPORT and help inform the national agenda re ongoing funding for SUPPORT units. The focus of the evaluation will be on the relevance, effectiveness and efficiency of NS SUPPORT s six core functions: Data platforms and services Methods Support and Development Health Systems Research, Implementation Research and Knowledge Translation Pragmatic / Real-world Clinical Trials Career Development in Methods and Health Services Research Consultation and Research Services R.A. Malatest & Associates has been commissioned to implement the evaluation. The evaluation team would like to conduct telephone interviews with key NL SUPPORT stakeholders. Participation is voluntary. A member of the evaluation team will be contacting you in the near future to schedule an interview. Your input will be anonymous. The interviews will require minutes of your time and will be scheduled at your convenience. Your input will be amalgamated with that of other stakeholders to help improve NL SUPPORT s future service delivery. Again, participation is voluntary. If you have any questions about the evaluation, please contact Carmelle Goldberg, Lead Evaluator from R.A. Malatest & Associates Ltd. at c.goldberg@malatest.com or ext Do you have any questions before we begin? Contact Information First and Last Name: Type of research partner: (position if relevant): Telephone Number: Date and Time of Interview: Relevance 1) How long have you been involved in patient-centred research and what are your primary motivations for collaborating with the NL SUPPORT Unit? 2) Could you please describe the nature of your collaboration with NL SUPPORT Unit? [Probe: roles, responsibilities, committees and/or projects you are involved with] 3) Based on your experiences with NL SUPPORT Unit, would you say there is a need for a specialized, multidisciplinary research service in Newfoundland and Labrador to provide expertise and services to those engaged in patient oriented research? What is the primary value of this research service centre to reaching your (or your organizations) goals? 5) Are there alternative ways for the NL SUPPORT Unit to build awareness of their research centre to form meaningful partnerships with key actors in the healthcare system? If so, what are they? Value of Patient-Oriented Research 6) Has your involvement in the NL SUPPORT Unit changed your awareness, attitude or behaviours toward applied patient-centred research? If so, how? [Probe: do you value the role of research in improving patient experiences, health outcomes, and guiding decision-making?] 7) How have you been involved in patient-oriented research? Which research processes have you been engaged in (e.g., identifying priorities or the relevance of research questions, grant writing or reviewing processes, design and management of research, data collection and analysis, implementation of research findings into practice/policy, evaluating impact and report writing)? 8) Do you feel valued as a partner in the research process by all the research stakeholders? Were you able to influence decision-making processes in the projects and committees you are involved in? 9) Are all key stakeholder (e.g., researchers, patients, policy makers, healthcare providers, funders) perspectives effectively engaged in the NL SUPPORT Unit projects and committees? Are perspectives missing? 10) Does the NL SUPPORT Unit have effective processes to align and address potential conflict linked to strategic planning or project implementation? If so, what are they? 11) What are the most significant benefits to your collaboration(s) with the NL SUPPORT Unit? Barriers 12) What challenges, if any, did you experience when collaborating with the NL SUPPORT Unit? Were these challenges overcome? If so, how? 13) Did you experience any barriers to collaborating with the NL SUPPORT Unit? [Probes: frustration with the length of training; lack of common vocabulary for collaborative work; insurmountable differences in perspectives/priorities, token engagement or false sense of inclusiveness?] 14) What are the impacts of these barriers on collaboration as equal partners in research that is importance to you? [Probes: high absenteeism of specific stakeholders, lack of buy-in for research and/or results, lack of participation/retention of key stakeholders, lack of influence or decision-making power] 15) Do you have any potential solutions to overcome these challenges or barriers? Improvements 16) Are there areas where the NL SUPPORT Unit can make improvements to better serve your needs? 17) Is there anything else you would like to share about your experiences with the NL SUPPORT Unit? Thank you very much for sharing your opinions and experiences. 4) How did you become aware of the NL SUPPORT Unit? NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd.

19 Respondent 1) Which perspective best describes your participation in the NL SUPPORT Unit? Patient/public (n=9) Healthcare provider (n=5) Policy maker/decision-maker (n=4) Researcher (n=37) Funder (n=2) SUPPORT Team/staff (n=12) Below are a number of statements regarding your opinions and experiences with the NL SUPPORT unit. Please read each one and indicate to what extent you agree or disagree with the statement. Beliefs 1 Strongly Disagree 2 Disagree 3 Neutral 4 Agree 5 Strongly Agree 6 Don t Know 2. Patient-centred research improves the quality of research and increases the use of research into evidence. 3. Patient-centred research is a way to improve the healthcare system, patient experiences, and health outcomes. 4. Patients influence and accelerate decision-making, translation and uptake of new practices. 2.9% 0% 7.2% 24.6% 62.3% 2.9% 1.4% 1.4% 2.9% 21.7% 69.6% 2.9% 1.4% 2.9% 21.7% 31.9% 37.7% 4.3% 10. APPENDIX B: ONLINE SURVEY Newfoundland and Labrador Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit Mid Term Evaluation Interaction Patterns Among Stakeholders 1 Strongly Disagree 5. The NL SUPPORT Unit fosters integration, respect and mentorship among patients, researchers, health practitioners and policy members. 6. The NL SUPPORT Unit creates meaningful opportunities for patients and the public to participate and influence research that is of value for them. 7. The NL SUPPORT Unit catalyzes and supports collaboration on jurisdictional priorities and decision-making with other Strategy for Patient-Oriented Research (SPOR) networks. 8. The NL SUPPORT Unit has effective processes to align and address potential conflict. NL SUPPORT Unit Mid- Term Evaluation Prepared by: R.A. Malatest & Associates Ltd. 2 Disagree 3 Neutral 4 Agree 5 Strongly Agree 6 Don t Know 1.4% 0% 5.8% 36.2% 52.2% 4.3% 0% 1.4% 4.3% 42% 44.9% 7.2% 1.4% 0% 7.2% 31.9% 40.6% 18.8% 1.4% 1.4% 13% 20.3% 15.9% 47.8%