Results-based Management Accountability Framework (RMAF)

Transcription

1 Results-based Management Accountability Framework (RMAF) Prepared by: Cathexis Consulting Inc. Contact: Martha McGuire

2 CDO RESULTS-BASED MANAGEMENT ACCOUNTABILITY FRAMEWORK TABLE OF CONTENTS 1. Introduction Background Purpose Overview of CDO Mission, Vision and Values Strategic Priorities CDO Logic Model Progress Measurement Framework Progress Measures Data Collection Framework Evaluation Strategy Evaluation Questions Evaluation and Monitoring Activities by Year...14 Appendix A: CDO Programs College of Dietitians of Ontario i

3 1. INTRODUCTION 1.1 BACKGROUND In the winter of 2011, the College of Dietitians of Ontario set out to renew its Strategic Plan. The Council established a Strategic Planning Committee and a consulting firm was engaged to facilitate the strategic planning process. What resulted was a report of key themes and strategic goals for the next 4 years (through to 2015). Following the renewal of the Strategic Plan, the College sought to revise its Resultsbased Management Accountability Framework (RMAF) to reflect the new strategic priorities. Cathexis Consulting was engaged to facilitate the RMAF revision process. Phone interviews with College staff and management were conducted in addition to a half day workshop during which staff contributed to the developed an overall logic model and key progress indicators. This document presents the measurement tools developed throughout this consultation process. 1.2 PURPOSE The Results-based Management Accountability Framework (RMAF) is intended to help CDO management measure and communicate the organization s progress towards its strategic goals. Conducting regular monitoring and evaluation using the framework will help achieve the following objectives: Increase the ability of CDO to be accountable to its Council, members and the general public; Learn about what is working well and what changes are needed in order to support program improvement; and Gain information that can assist with future resource allocation. College of Dietitians of Ontario 2

4 2. OVERVIEW OF CDO The College of Dietitians of Ontario was founded in 1991 through the Dietetics Act, and became a regulatory body on December 31, Its mandate under the Regulated Health Professions Act, and the Dietetics Act, is to regulate the profession of dietetics in Ontario in the interest of the public and public protection. 2.1 MISSION, VISION AND VALUES Mission A mission is a statement of organizational purpose. CDO s mission is: The College of Dietitians of Ontario exists to regulate and support all Registered Dietitians in the interest of the public of Ontario. We are dedicated to the ongoing enhancement of safe, ethical and competent nutrition services provided by Registered Dietitians in their changing practice environments. Vision A vision describes what our organization will look like if it succeeds in achieving its goals and its full potential. CDO s vision is: People of Ontario are confident that the College demonstrates regulatory excellence in the public interest. Values The Council has set out these values to guide our behaviour and decision-making. We are dedicated in all our actions and decisions to these core values: Public Interest Integrity Collaboration Accountability and Transparency College of Dietitians of Ontario 3

5 2.2 STRATEGIC PRIORITIES The Strategic Plan The Strategic Plan sets out the vision, end-goals, objectives and strategies of the College of Dietitians of Ontario. It focuses the organization s activities for the next four years and reflects the College s commitment to ensure that it continues to demonstrate excellence in the regulation of the dietetics profession in the public interest. The focus of this Plan is on the following key themes: Strengthening our regulatory standards framework to respond to our changing practice environments Collaborating internally and externally to ensure our members demonstrate effective practice in care settings which are increasingly interprofessional Assuring entry-level and continuing competence Maintaining our priority on good governance, effective operations and being an open and accountable organization End Goals End Goals define what the organization must accomplish, taking into account all of its obligations. They prescribe the results that the Council has set out for the CDO to achieve over the Plan Horizon ( ): End Goal 1: An effective regulatory framework for the quality and safety of dietetic practice End Goal 2: Competent members who are compliant with CDO standards End Goal 3: Informed and knowledgeable registered dieticians engaged in effective practice in their environments End Goal 4: Support the attainment of an adequate supply of registered dietitians End Goal 5: An effective organization with optimal use of resources College of Dietitians of Ontario 4

6 3. CDO LOGIC MODEL Evaluating a program or constructing a set of progress measures entails understanding how the organization operates and what it is intended to accomplish. The CDO logic model provides a succinct visual of how the College is expected to achieve its intended outcomes (or End Goals). The logic model lays out the College s activities, implementation objectives, outputs, enablers, outcomes and impacts. In the interest of being succinct, the inputs (or resources) required to support the organization s activities are not included in the visual. These resources include: human resources (staff and management), financial resources, governance (Council and committees), and stakeholders (CDO members, Minister of Health and Long-Term Care, Ontario Fairness Commissioner, general public) The activities are stated in general terms and do not capture all activities of the College. Activities are carried out within CDO programs and multiple programs contribute to the attainment of the outcomes and impacts. A list and description of each CDO program is presented in Appendix A. The implementation objectives focus attention on the operational goals required to produce outputs. Implementation objectives express program activities in a way that reminds us that successful program implementation is a prerequisite for achieving outcomes. The implementation goals found in the logic model were pulled directly from the Strategic Plan. The outputs indicate the work done. They are countable and are the first and most tangible results of a program (e.g., # of provisional certificates, # of education resources etc). Enablers explain what needs to happen to transition from the work done to the intended outcomes. Not all logic models have this element. It was thought to be appropriate to include them here in order to clearly articulate the intended cause and effect linkages between the outputs and outcomes. Outcomes are observed characteristics of the target population or social conditions. They relate to the benefits of the College s products and services for CDO members and the general public. The outcomes presented in the logic model are pulled directly from the Strategic Plan End Goals. The impacts represent the Colleges intended long-term contribution to improve public welfare. The grey box at the base of the logic model includes the activities, implementation objectives, outputs and outcomes that are required to operate an effective organization. They provide a foundation for all other College programs, activities and services. College of Dietitians of Ontario 5

7 CDO Logic Model College of Dietitians of Ontario 6

8 4. PROGRESS MEASUREMENT FRAMEWORK 4.1 PROGRESS MEASURES Progress measures are a useful tool for ongoing monitoring of the College s progress towards its Strategic goals. However, since indicators are frequently quantitative in nature, additional sources of qualitative information, such as interviews or focus groups, will add to greater understanding, looking at contextual intervening or contributing factors. This framework provides two levels of progress measures: Implementation progress measures, which monitor the College s progress towards accomplishing its intended activities and implementation objectives. These measures are of interest to College staff and management. Outcome progress measures, which monitor the College s progress in achieving its strategic goals. These measures are of interest to the executive leadership, and the Council. College of Dietitians of Ontario 7

9 Progress Measures Framework Implementation Progress Measures Implementation Objectives Implementation Indicators Data Source Frequency 1.1 Develop a CDO Regulatory Standards Framework entailing that, in its standards and programs, CDO will: Define a regulatory standard Ensure mechanisms for continuous monitoring of issues for which standards are required Set priorities for regulatory attention with high regard for risk and public safety, and develop necessary standards Demonstrate a collaborative process for standards development Interpret standards for various practice settings and within the context of changing scopes of practice and practice environments. 1.2 Improve assessment mechanism for entry to practice qualifications ensuring continued adherence to the principles for transparency, objectivity, impartiality and fairness. 1.3 Enhance the effectiveness of the Quality Assurance program relevant to all emerging and high risk areas of practice. 1.4 Develop mechanisms of Quality Assurance to enable nonactive members to continue in a non-restricted, or General Certificate of Registration. Progress towards (on track according to timelines): A definition for "regulatory standards" is created and approved Process established for the identification and monitory of risk There is an agenda for the development of standards that are based on practice issues associated with risk to the public Educational material developed for every new standard Favourable evaluation by the OFC Members are able to identify emerging and high risk areas in their practice Processes are developed to enable non-active members to continue in a non-restricted or General Certificate of Registration. Internal interviews, meeting agendas, meeting minutes and memos Evaluation from OFC Members Survey 2.1 Improve assessment of qualifications for registration revising policies and ensuring objectivity of processes Monitor and identify how new Registered Dietitians qualify given new models of dietetic education. 2.2 For practice activities, in which risk is identified (prioritized in our regulatory framework above), describe competencies and develop appropriate tools to enable self and peer assessment. 2.3 Ensure effective and timely enforcement of applicable laws and regulations through accessible policies and processes. # of files returned to the Registration Committee to clarify questions arising from the Decision and Reasons High level of inter-rater reliability # of complaint decisions issued within 150 days % RD and public who have been involved in proceedings that feel the proceedings were transparent, fair and effective # of assessment tools for high risk activities # of RDs who report being aware of standards # of RDs assessed in second step of PPA for competence issues related to the application of standards related to laws and regulations # of RDs cautioned or given action other than dismiss in an ICRC proceeding Registration Data Audit process File review Survey of members and public Evaluation of JKAT, periodic survey College of Dietitians of Ontario 8

10 3.1 Provide specific and relevant support to Registered Dietitians in all areas of dietetic practice. 3.2 Create ways for Registered Dietitians to identify the issues in their practice environments for which College guidance or policies may be required. 3.3 Increase utilization by Registered Dietitians of processes and tools intended to enhance their learning and quality improvement Develop and facilitate use of tools and educational products using new technological processes and resources Ensure relevant content, consideration of different learning styles, and ease of use Create tools to help Registered Dietitians identify and measure issues in their individual practice to be addressed through CQI and lifelong learning 4.1 Collect and transfer data to support the mapping of dietetic workforce information enabling health human resources planning in Ontario and Canada. 4.2 Support the transition to the new dietetic education model with focus on assurance of competency outcomes. 4.3 Prepare for implementation of the new provisional class of membership. 5.1 Ensure excellent governance of the College through the work of Council. 5.2 Build and strengthen human resources capacity to deliver on College goals. 5.3 Update and ensure effective technology supporting delivery and evaluation of programs and communication with the public, members and stakeholders. 5.4 Continue to ensure ongoing planning, oversight, decisionmaking, and program development is based on information and evidence, and reflects commitment to our End Goals and their objectives. # of tools in place guiding RDs in the application of laws and regulations # of hits for each online product # of RDs using Practice Advisory Service # of new issues for which College guidance or policies may be required are identified # of RDs in all areas of practice attending CDO workshops # of RDs evaluating relevance and usefulness of information received from PAS as positive # of RDs evaluating relevance and usefulness of information from workshops, resources and other services as positive # of RDs in all areas of practice who report relevance in the QA tools (SDL Tool, PPA, JKAT) # of RDs reporting completion or active pursuit of Professional Improvement Plans # of data transfers # of events and projects enabled by CDO to support implementation of the ICDEP Policies and tools are in place to support the new provisional class # of policies, tools and procedure to support the new extended temporary class implemented # of provisional certifications issued # of extended temporary certificates issued Governance review is completed # of governance improvement tools are developed or revised # of evaluation and assessment tools are developed or revised # of policies or actions around staff learning needs Staff and committees are using intranet imis upgrade is complete Processes are in place to begin assessing cost effectiveness of tools/programs and their components New audit approach is in place Web stats imis report PAS tracking File review Members Survey, PAS satisfaction survey Members Survey Members Survey; JKAT and PA evaluations, Annual SDL data Registration Program data Internal documents and reports Internal documents and reports imis data imis data Internal documents and reports IT manager reporting Report from Registrar & ED College of Dietitians of Ontario 9

11 Outcomes Progress Measures Long-Term Outcomes Indicators Measurement Methods Frequency 1. An Effective Regulatory Framework for the Quality and % of RDs demonstrate improved practice in the PPA results and Safety of Dietetic Practices areas of safety, competency and ethical practice Members Survey 2. Competent Members who are Compliant with CDO Standards # of RDs who comply with their QA obligations imis (e.g. # of RDs referred for non-compliance) # and nature of complaints and reports 3. Informed and Knowledgeable Registered Dietitians Engaged in Effective Practice in their Environments # of RDs in all areas of practice who report an impact of professional improvement Plans in the practice and client care % of RDs demonstrate improved practice in the Members Survey PPA results 4. Support of the Attainment of an Adequate Supply of Registered Dietitians areas of safety, competency and ethical practice # of applicants applying to the College # RDs obtaining registration Number of RDs per capita in Ontario, compared to the number of RDs per capita in other provinces 5. An Effective Organization with Optimal Use of Resources Activities in Strategic Plan/ Tactics Document undertaken on time and on budget Registration Program data CDO registration data compared with population census data. Registration data from other provincial dietetic regulatory bodies compared with population data in other provinces. Program and committee reports of accomplishment and budgets. College of Dietitians of Ontario 10

12 4.2 DATA COLLECTION FRAMEWORK Implementation Indicators Members Survey Internal documents, meetings, policies, tools etc Goal 1: An Effective Regulatory Framework for the Quality and Safety of Dietetic Practices A definition for "regulatory standards" is created and approved Process established for the identification and monitory of risk There is an agenda for the development of standards that are based on practice issues associated with risk to the public Educational material developed for every new standard Favorable evaluation by the OFC Members are able to identify emerging and high risk areas in their practice Processes are developed to enable non-active members to continue in a non-restricted or General Certificate of Registration. Goal 2: Competent Members who are Compliant with CDO Standards # of files returned to the Registration Committee to clarify questions arising from the Decision and Reasons High level of interater reliability # of complaint decisions issued within 150 days % RD and public who have been involved in proceeding that feel the proceedings were transparent, fair and effective Registration Data imis data PAS, SDL, JKAT tracking/ satisfaction surveys # of assessment tools for high risk activities # of RDs who report being aware of standards # of RDs assessed in second step of PPA for competence issues related to the application of standards related to laws and regulations # of RDs cautioned or given action other than dismiss in an ICRC proceeding PPA Results Goal 3: Informed and Knowledgeable Registered Dietitians Engaged in Effective Practice in their Environments # of hits for each online product # of RDs using Practice Advisory Service Evaluation from OFC College of Dietitians of Ontario 11

13 # of new issues for which College guidance or policies may be required are identified # of RDs in all areas of practice attending CDO workshops # of RDs evaluating relevance and usefulness of information received from PAS as positive # of RDs evaluating relevance and usefulness of information from workshops, resources and other services as positive # of RDs in all areas of practice who report relevance in the QA tools (SDL Tool, PPA, JKAT) # of RDs reporting completion or active pursuit of Professional Improvement Plans Goal 4: Support of the Attainment of an Adequate Supply of Registered Dietitians # of data transfers # of events and projects enabled by CDO to support implementation of the ICDEP # of policies, tools and procedures developed to support the new provisional class # of policies, tools and procedure to support the new extended temporary class implemented # of provisional certifications issued # of extended temporary certificates issued Goal 5: An Effective Organization with Optimal Use of Resources Governance review is completed # of governance improvement tools are developed or revised # of evaluation and assessment tools are developed or revised # of policies or actions around staff learning needs Staff and committees are using the intranet Processes are in place to begin assessing cost effectiveness of tools/programs and their components New audit approach is in place College of Dietitians of Ontario 12

14 Outcome Indicators Members Survey Internal documents, meeting, policies, tools etc Registration Data imis data PPA Results Goal 1: An Effective Regulatory Framework for the Quality and Safety of Dietetic Practices % of RDs demonstrate safe, competent and ethical practice Goal 2: Competent Members who are Compliant with CDO Standards # of RDs who comply with their QA obligations (e.g. # of RDs referred for non-compliance) and registration obligations # and nature of complaints and reports Goal 3: Informed and Knowledgeable Registered Dietitians Engaged in Effective Practice in their Environments # of RDs in all areas of practice who report an impact of professional improvement Plans in the practice and client care % of RDs demonstrate improved practice in the areas of safety, competency and ethical practice Goal 4: Support of the Attainment of an Adequate Supply of Registered Dietitians # of applicants applying to the College # RDs obtaining registration Number of RDs per capita in Ontario, compared to the number of RDs per capita in other provinces Goal 5: An Effective Organization with Optimal Use of Resources Activities in Strategic Plan/ Tactics Document undertaken on time and on budget College of Dietitians of Ontario 13

15 5. EVALUATION STRATEGY It is anticipated that CDO will conduct evaluation and monitory activities each year. An evaluation plan will be developed for each evaluation. The specific questions addressed will vary from year to year, depending on the purpose and timing of the evaluation. The following questions are intended to guide the thinking in the development of each specific evaluation plan. Cumulatively over the next five years, the evaluations should produce answers to all of these questions, and possibly some additional questions that become important during that time. 5.1 EVALUATION QUESTIONS Implementation Evaluation Questions 1. Have we done the activities we said we would do according to our Goals and Tactics Plan? If not, why not? 2. Are we meeting and staying within our legislated obligations/requirements? 3. Are we being more cost-effective and efficient, while still maintaining a healthy work environment? Outcome Evaluation Questions 4. Do the activities of CDO result in better practice? What activities contribute to better practice? What activities need improvement or need to be eliminated? 5. How successful are we at achieving our end goals? 6. To what extent do RD s patients feel they can trust that RDs are safe, competent and ethical and provide quality services? 5.2 EVALUATION AND MONITORING ACTIVITIES BY YEAR This should include an indication of how you are going to approach evaluation what will be done each year and a schedule for the next five years. For example: Year Evaluation Goals 2012 Monitor completion of intended activities Conduct governance review 2013 Monitor completion of intended activities Conduct Public Survey to measure progress on: o Public understand the public protection role of CDO o Public is aware that RDs are regulated experts in nutrition o Public accesses services of CDO Determine if meeting & staying within our legislated requirements 2014 Monitor completion of intended activities College of Dietitians of Ontario 14

16 Conduct Member Impact Survey 2015 Monitor completion of intended activities Determine whether advancement towards the goals has occurred Assess cost-effectiveness and cost-efficiency College of Dietitians of Ontario 15

17 APPENDIX A: CDO PROGRAMS Patient Relations Program The Patient Relations Committee advises Council with respect to the Patient Relations Program. The programprovides information and services to support the public s access to RDs and CDO s services and to prevent sexual abuse of patients. The Patient Relations Program is responsible for: Public education / communications Member education about prevention and reporting of sexual abuse of patients Developing information regarding value-added of RD services Linking service Maintaining system for funding for counselling and therapy of sexually abused clients Staff training for handling complaints of a sexual nature Registration Program The primary goal of the Registration program is to ensure that only qualified and competent people become RDs in Ontario, that only CDO members use the RD title, and that registration practices are fair, objective, transparent and impartial. The Program also ensures that the College has accurate information and financial resources to regulate and support RDs in the interest of the public. The other focus is CDO s collaborative work to increase the supply of RDs through collaborative activities that address the educational system issues affecting the ability of students to access practicum opportunities. It is responsible for the following: Establishing the standards of qualification for persons to be issued a Certificate of Registration Providing info to public/applicants Assessing qualifications Assessing applicant qualifications Issuing certificates of registration College of Dietitians of Ontario 16

18 Renewing Registrations information collection and payment. Using Registration information to support Health Human Resources Planning and advocacy Investigating and handling inappropriate use of RD title Standards and Compliance Program Standards and Compliance develops and maintains programs and standards to assist individuals to exercise their rights under the Regulated Health Professions Act, 1991 (RHPA) and to ensure that RDs practice competently and ethically in keeping with the standards and laws that affect them. Standards and Compliance ensures a fair, effective, transparent and legal way to handle issues of conduct, competency/fitness to practice in keeping with the RHPA. It relates to the Discipline Committee, the Fitness to Practice Committee and the Complaints Committee. It works with Patient Relations to prevent and address sexual abuse of patients. It provides the following services: Developing standards of professional practice and conduct Screening Case management Investigations Hearings Issuance of decisions and reasons regarding remediation and undertakings Quality Assurance (QA) Program The QA Program develops, operates and monitors learning and assessment tools to support Registered Dietitians in continuing competency and education to enhance the delivery of high quality dietetic services to the Ontario public. The QA Committee is also working towards defining quality practice with reference to enablers and detractors. The QA Program components are: Jurisprudence Knowledge and Assessment Tool Peer and Practice Assessment Self Directed Learning Tool Directed Remediation College of Dietitians of Ontario 17

19 Practice Advisory Program Practice Advisory Program supports compliance with standards, provides advice, assistance and education to members on practice issues such as standards, laws, ethics and regulations. It provides the following services: One-to-one advice to members Practice articles published in résumé and other publications Workshops/presentations to members, students and interns Online and webcast resources College of Dietitians of Ontario 18