Review of the Evaluation Framework for the Immigrant Settlement and Adaptation and Host Programs FINAL REPORT

Transcription

1 Review of the Evaluation Framework for the Immigrant Settlement and Adaptation and Host Programs FINAL REPORT Submitted to Van-Khanh Nguyen André Bernier Citizenship and Immigration Canada 31 March 2001

2 TABLE OF CONTENTS 1.0 Purpose of the Review Page Steps in Review Process Overview of Evaluation Approaches for Community-based Organizations Measuring Program Outcomes, United Way of America Guide to Project Evaluation: A Participatory Approach, Health Canada A Hands-On Guide to Planning and Evaluation, Canadian Hemophilia Society Outcomes Evaluation Program, Family Service Canada Ontario Trillium Foundation Guide for New Grant Recipients Organizational On-Site Analysis, United Way of Canada-Centraide Canada Shared Features of the Approaches Comments on the Proposed Evaluation Frameworks Positive Features of the Proposals Summary of Concerns about the Proposals Consultant s Observations and Recommendations Focus on Settlement Outcomes and Indicators Respect Client Confidentiality Strengthen Relationships Incremental and Phased Introduction of the Evaluation System 12 BIBLIOGRAPHY Attachments Update (Voluntary Sector Initiative Newsletter) Sample Indicator Forms from Family Service Canada Outcomes Evaluation Program 2

3 1.0 Purpose of the Review It is understood that the overall purpose of the performance measurement and evaluation process proposed by Citizenship and Immigration Canada (CIC) is to provide useful information for CIC and for the service provider organizations (SPOs) who deliver the programs. Performance measurement is intended to help ensure that the programs are as effective and efficient as possible to assist newcomers settle in Canada. The purpose of this project is to review and comment on two of the Evaluation Frameworks that have been prepared by Goss Gilroy Inc. Management Consultants for Citizenship and Immigration Canada (CIC) for evaluation of the Host Program and the Immigrant Settlement and Adaptation Program. This report includes suggestions and recommendations for addressing some of the issues and concerns raised by the SPOs. It also includes recommendations for an evaluation process that makes appropriate and realistic demands on the SPOs and provides practical, useful results for improving the effectiveness of the programs. 2.0 Steps in the Review Process The consultant has undertaken the following steps in preparing this report: reviewed Evaluation Frameworks and other relevant documents and materials met with key CIC officials and consulted with some SPO representatives to understand the issues and concerns identified by the non-governmental SPOs prepared an overview of the nature and scope of the United Way evaluation process and other similar approaches compared the features and strengths of the Evaluation Frameworks and the other approaches assessed the SPO issues and concerns developed recommendations for responding to the SPO concerns including suggestions for modifying the evaluation process 3

4 3.0 Overview of Selected Evaluation Approaches for Community-based Organizations: 3.1 Measuring Program Outcomes: A Practical Approach, United Way of America United Way of America has developed a system for measuring program outcomes designed to assess whether programs make a difference in the lives of people and to provide a basis for improving client programs and services. 1 The manual provides a step-by-step approach to developing a system for measuring program outcomes as well as using the results. The manual explains the eight step process and includes worksheets, tools and examples to help organizations plan an appropriate outcome measurement system for their own programs. The main steps in the process include - developing an evaluation plan and clarifying roles and responsibilities - choosing the outcomes to be measured - specifying indicators for the outcomes - preparing to collect data on the indicators - trying out the measurement system - analyzing and reporting findings - improving the system - using the findings The manual and training materials stress that agencies should use whatever model provides the kind of information that will meet their needs. United Way of America urges all funders (including local United Ways) to respect this approach by focusing on the merits of a program measurement process rather than considering their own system as the only or the most valid approach. United Way of Canada-Centraide Canada has provided training workshops on the United Way of America system for staff and volunteers from local United Way Centraides in Canada. Many local United Ways have in turn offered workshops for their member agencies using these materials. Most United Ways in Canada require that agencies provide information about the impact of services in clients lives. However few, if any, prescribe specific outcomes evaluation approach. 1 Measuring Program Outcomes: A Practical Approach. p 4 4

5 3.2 Guide to Project Evaluation: A Participatory Approach, Health Canada This guide is designed to provide an easy-to-use, comprehensive framework for project evaluation and to assist in the development and implementation of effective project evaluation. 2 The participatory evaluation recognizes the following principles: - the focus is on learning, success and action - evaluation must be useful to the people who are doing the work that is being evaluated - the evaluation process is on-going - project sponsors (not Health Canada or other funders) are responsible for defining the specific evaluation questions, the indicators of success and realistic timeframes for implementing the evaluation plan. The guide includes a series of skill-building exercises for the project personnel who will be carrying out the evaluation activities. Five key evaluation questions are identified. What did we do? What did we learn about what worked and what didn t? What difference did it make that we did this work? What could we do differently? How will we use the evaluation findings? Five process steps are explained in the manual: - define the project work - develop success indicators and their measures - collect evaluation data - analyze and interpret the data - use the evaluation results. 3 The Guide does not indicate that all sponsoring organizations that receive funds from Health Canada are required to follow these steps. Rather, the intent seems to be to motivate organizations to incorporate evaluation as an essential activity in planning, implementing and managing effective programs. 3.3 A Hands-On Guide to Planning and Evaluation, Canadian Hemophilia Society The first purpose of this Guide is to provide a process by which community-based organizations can plan results-oriented programs. The manual provides a step-bystep process for building a program model in which desired behaviour changes are linked to changes in knowledge and attitudes and these in turn are linked to specific program activities. 2 Guide to Program Evaluation: A Participatory Approach p 3 3 Ibid, p 11 5

6 In publishing this manual, the Canadian Hemophilia Society says it was driven by the need of the practitioner to learn what did or did not work and how to change the program for the future. The Guide includes sample worksheets and models designed specifically for AIDS education and prevention programs. However, the process can and has been applied to a very wide range of community-based programs. Instructions and worksheets focus on four key tasks: preparing a logic model, collecting data, analyzing data, and interpreting results. The manual explains four types of evaluation including needs assessment, process evaluation, outcome evaluation and causal evaluation. The Guide provides a means for community-based groups to demonstrate the links between their activities and changes in participants attitudes or knowledge and, by extension, changes in their behaviour that address the problem situation. 4 The guide highlights the following common evaluation questions for practitioners: - Is the program meeting client needs? Has our understanding of these needs changed since the program was first planned? - Are program activities being carried out as planned? - What modifications have been made and why? - Is the program achieving the desired impact? Are clients experiencing anticipated changes? Are there unintended impacts? - Which components of the program are most effective in achieving desired results? 3.4 Outcomes Evaluation Program, Family Service Canada In 1998, Family Service Canada 5 introduced an Outcomes Evaluation Program for its member agencies. The system was originally developed by the Catholic Family Counselling Centre in Kitchener, Ontario to respond to questions from the Board of Directors about the nature and scope of the impact the agency was able to have in helping people. CFCC s first step was to consult widely within the organization and with clients regarding the outcomes that they intended or desired to have. They identified four core outcomes that they would hope to achieve for all clients regardless of their presenting problems or the program or service they received. All adult clients are asked to complete four core indicators including: - functioning at home, work, school and in the community 4 A Hands-On Guide to Planning and Evaluation, p 13 5 Family Service Canada is a national association representing a network of family service agencies across Canada. The member agencies provide a wide range of services to individuals and families to assist and strengthen people in their daily lives and relationships. 6

7 - general satisfaction with life - satisfaction with relationships and - distress. Clients complete the specially designed indicator forms at the time of intake and at other key stages of receiving services including at closure. In addition to the four core indicators, clients are asked to complete a limited number of additional indicators depending on the problems they are experiencing. For each indicator, seven scenarios are provided and clients are asked to select the one that most closely reflects their current circumstances or feelings. Agencies input client data using special software and forward the data to Family Service Canada for the preparation of agency reports and an aggregate report summarizing the data from all agencies. For each indicator, agency and aggregate reports indicate the number of clients who report that their situation has improved or deteriorated during the time they have been receiving services. In the data provided to Family Service Canada, clients are identified by number and Family Service Canada is not able to link any of the information to a particular client. Agencies can request information about specific clients using the number they have assigned that client. The agency and aggregate reports have been used to improve client services, to introduce new services and to help in making the case for funding and other community support. 3.5 The Ontario Trillium Foundation Guide for New Grant Recipients Community-based organizations that receive grants from the Ontario Trillium Foundation are asked to provide information about the impact of their program on the intended beneficiaries. However, the Ontario Trillium Foundation does not provide or recommend any specific approach to program evaluation. A few years ago, the Trillium Foundation offered workshops or presentations on evaluation to highlight the principles and guidelines that the Foundation believed should apply in evaluating funded programs and projects. The Foundation s assessment of the success of a particular project or program is based on responses to four key evaluation questions. Are our programs effective? Are they getting better? Are we gathering and using information that will help to improve our work? 7

8 The Trillium Foundation encourages evaluation based not on activities but on what difference the activities made. The purpose of evaluation has been seen as learning and improving, not judging. 3.6 Organizational On-Site Analysis, United Way of Canada-Centraide Canada The United Way of Canada has offered a process called Organizational On-Site Analysis to local United Ways across Canada and to other community-based organizations. The process is highly participatory and follows a fairly standard process for reviewing organizational trends over a period of years. The same step-by-step process is applied to analyzing trends in client profile, services and programs, human resources, finances, organizational structure, etc. On-Site Analysis can be adapted to review and analyze facts and develop recommendations to strengthen a whole organization or a particular program, department or service. The process involves reviewing statistical information, making observations, assessing strengths and weaknesses in the trends, drawing conclusions and developing recommendations based on the conclusions. Organizational On-Site Analysis is not an outcome evaluation or organizational performance process but does provide an opportunity for organizations to review facts about various aspects of their operation and to develop recommendations based on facts. Participants also develop useful skills in statistical analysis and interpretation. The steps in the On-Site Analysis process result in a series of practical recommendations and solutions to problems for which there is often a high degree of ownership and motivation. 3.7 Shared Features of the Approaches The concerns that the SPOs have identified and other concerns that may be anticipated are the same concerns of most community based organizations when they are asked by a funder or external stakeholder to evaluate their programs and services. (See section 4.0 below.) The evaluation approaches described above have tried to address these concerns shared by many community-based organizations. The funders or other sponsors have typically encouraged and supported agencies in designing their own evaluation systems. The community-based organizations have then prepared reports for their own planning and management and to meet funders requirements. 8

9 While CIC has specific information needs and requirements, lessons can be learned from the approaches taken by other funders. The various evaluation systems and practices suggest opportunities and strategies for addressing the SPO concerns. Some of the features of the above approaches that have been specifically introduced to address the concerns of the community-based organizations include the following: The confidentiality of client information and the trust relationship between client and counsellor are respected. The sponsors of these approaches often funders do not prescribe a specific evaluation approach. The primary focus is on client or program participant outcomes: What difference have these programs made in the lives of the clients? Outcomes are usually skill development, acquisition of knowledge, change in attitude, or change in circumstances. Behaviour change or prevention of undesired behaviour is seen as longer-term impacts influenced by many factors. Agencies are encouraged to identify desired outcomes they can reasonably expect to achieve and not to hold themselves accountable for outcomes or impacts that are beyond the scope of their programs. For example, programs that are educational in nature can reasonably expect that participants will have new knowledge or skills as a result of their participation. However, behaviour change may not be a reasonable expectation because it is affected by many factors. The purpose of evaluation is seen as improving improve client outcomes not to judge organizational or individual performance. Clients themselves are the important (but not only) data source. Client satisfaction is helpful information but not a substitute for outcome information. The evaluation process is most effectively driven by the curiosity and wisdom of those who work directly with clients. Special expertise is not necessarily required. It is recognized that evaluation is a new activity in many agencies and requires new skills and processes. Clarifying and confirming the desired outcomes and identifying indicators are recognized as very challenging steps for social service agencies. 9

10 Client involvement in identifying and confirming desired outcomes and in the development of indicators is regarded as essential. 4.0 Comments on the Proposed Evaluation Frameworks The following comments on the performance measurement system proposed by management consultants Goss Gilroy include the perspectives of service provider organizations that have had an opportunity to review the proposals. 6 Relevant comments from other stakeholders are also reflected. 4.1 Positive Features of the Proposals - the implicit recognition in the proposals that desired client outcomes may vary from client to client, from community to community, and from agency to agency - recognition of a strong, shared interest among agencies in the development of a national data base - recognition that the new system would provide common definitions and facilitate sharing of information and ideas and collaboration among agencies - flexibility in the system to allow agencies to gather and include data to meet their own information requirements - progress in developing national standards for settlement programs 4.2 Summary of Concerns about the Proposals - lack of focus on defining settlement and how to measure it - desired outcomes and indicators of successful settlement not defined or explored - requirement to link client names and other identifiers with client data - pre-supposition that performance measurement will demonstrate the need for increases in skills, capacity and performance - emphasis on assessing what happened in the past rather than on the use of data to strengthen programs and to improve client outcomes and agency effectiveness and efficiency - lack of clarity about who is the intended beneficiary of evaluation process - use of focus groups involving SPOs for wide range of topics some of which seem outside SPO area of expertise and experience - program logic models mix outcomes and outputs - mandatory participation - technology requirements may be a barrier for some agencies - data will be of limited value in identifying unmet needs for individual clients or groups of clients or in a particular region 6 Evaluation Framework for the Host Program and Evaluation Framework for the Immigrant Settlement and Adaptation Program prepared for Citizenship and Immigration Canada, 2000 by Goss Gilroy Management Consultants. 10

11 5.0 Consultant s Observations and Recommendations 5.1 Focus on Settlement Outcomes and Indicators The performance measurement system will be stronger if it includes data regarding client outcomes and their progress in settlement and integration. Recommendations 1. The Performance Management Advisory Committee should oversee a collaborative process involving SPO management, direct service workers and clients and appropriate CIC personnel to identify a shared or common set of desired client outcomes and indicators of settlement and integration that could be endorsed by all SPOs and by the Department. 2. A simple survey form or other tool should be designed to gather information about these outcomes and indicators Individual agencies could expand this set of outcomes and indicators to address the particular circumstances and needs of the clients they serve and their diverse communities. 4. The performance measurement system should include these agreed-on settlement indicators. 5.2 Respect Client Confidentiality It seems reasonable to accept that the SPO relationship with clients will be undermined and their capacity to bring about community change will be affected if clients are asked to sign information release agreements during their first visit to an agency. Some SPOs indicate that they and their colleagues would be willing and able to provide individual client data provided individuals couldn t be identified by name or number. The Department does not seem to have made a convincing case to the SPOs that client names and other identifiers are essential information for Departmental planning purposes. The agencies do not have the same eligibility requirements as the Department. Most, if not all, any seek alternate funding in order to respond to the needs of clients who do not meet CIC eligibility requirements. The total cost of gathering and maintaining data about the eligibility of individual clients to receive services funded by the Department would be significant. Recommendations: 5. The Department should not require that agencies include client names or other identifiers in reporting individual client information. 7 Sample indicator forms from the Family Service Canada Outcomes Evaluation Program are attached. 11

12 6. The Department should explore the potential for meeting its information needs program evaluation and planning, monitoring service delivery, ensuring agency accountability, and allocation or re-allocation of resources without including client names and other identifiers in the national data base. 7. Agencies might be willing to provide a list of clients served including names and/or other identifiers in a separate database if the names and identifiers cannot be linked with other client information. 8. Determining what data will be gathered and reported by agencies and included in the national database should be driven by the information needs of the Department and the agencies. 5.3 Strengthen Relationships The current Voluntary Sector Initiative (VSI) of the federal government and the voluntary sector speaks to new collaborative relationships between the federal government and voluntary agencies. New accountability mechanisms for agencies that receive federal funding are being developed through the VSI. The VSI was founded on the principle that collaboration and cooperation are more promising and appropriate approaches than enforcement mechanisms and threats for strengthening the voluntary sector s ability to operate effectively and to support voluntary organizations in carrying out their mandates. Recommendations 9. The Department should contact the VSI Secretariat and/or the Privy Council Office for suggestions for building a strong and productive relationship between the Department and the SPOs. 10. The newly established CIC Performance Measurement Advisory Committee should consider lessons that might be learned from the VSI Joint Tables The Department should obtain information from the VSI regarding new accountability mechanisms and should continue its efforts to ensure that programs meet the needs of immigrants and refugees and that funds are being allocated and used as effectively and efficiently as possible. 5.4 Incremental and Phased Introduction of the Evaluation System The agencies and the Department share a strong interest in having a national database and better information for planning and allocation of resources for programs and services for immigrants and refugees. Agencies support the development of a national database. Some agencies have very limited technological capacity, resources and skills. Even a limited national database will provide new, potentially powerful information for monitoring service delivery and for Departmental and agency program planning and allocation of resources. 8 Update No. 1, VSI Secretariat, March

13 Recommendations 12. The Department should take an incremental and developmental approach in introducing the new performance measurement system starting with the agreed-on data set to get system up and running. 13. The agencies and the Department perhaps through the Performance Measurement Advisory Committee can jointly evaluate the new system, identify its strengths and weaknesses, and develop recommendations for expanding and improving it. 14. The Department and agencies should consider the extent to which information needs can be met with the range of data currently gathered and reported by agencies in the larger jurisdictions (Ontario and British Columbia) excluding the client names and other identifiers. 13

14 BIBLIOGRAPHY Evaluation Framework for the Host Program; Goss Gilroy Management Consultants. Citizenship and Immigration Canada, Evaluation Framework for the Immigrant Settlement and Adaptation Program, Goss Gilroy Management Consultants. Citizenship and Immigration Canada, Guide to Project Evaluation: A Participatory Approach. Population Health Directorate, Health Canada. Ottawa, 1996 A Hands-On Guide to Planning and Evaluation, How to Plan and Evaluate Programs in Community-Based Organizations; Durhane Wong-Reiger, Ph.D. and Lindee David. Canadian Hemophilia Society. Ottawa, 1993 Measuring Program Outcomes: A Practical Approach. United Way of America, On-Site Analysis: A practical approach to organizational change, Third Edition; Robert J. Myers, Peter Ufford, Mary-Scot Magill. OSCA Limited. Etobicoke, Ontario, 1993 Ontario Trillium Foundation, Guide for New Grant Recipients. Toronto, Outcomes Evaluation Program. Family Service Canada. Ottawa, 1998 Update, No. 1: Voluntary Sector Initiative Secretariat. Ottawa, March