Re: Request for Proposals for Immigrant Women Services Ottawa Settlement and Integration Services

Transcription

1 The Steering Committee of IWSO-SIS 219 Argyle Avenue Ottawa, Ontario K2P 2H4 Innovation. Evaluation. Results February 7, 2015 Re: Request for Proposals for Immigrant Women Services Ottawa Settlement and Integration Services Apex Consultants is pleased to respond to the Request for Proposals to present an evaluation plan for the Immigrant Women Services Ottawa Settlement and Integration Services (IWSO-SIS). As a recognized leading evaluation firm, we are confident that our proposal offers an innovative and feasible approach to meeting your evaluation needs. Our detailed proposal provides: An overview of IWSO-SIS and key stakeholders A critique of the existing logic model A draft logic model based on our understanding of the program Our planned evaluation approach including evaluation questions Our proposed methodology with rationale and analysis plans An evaluation matrix highlighting key evaluation questions, data sources, indicators, and collection methods Possible strategies and solutions for anticipated challenges A list of Credentialed Evaluator competencies Apex demonstrates in its proposed evaluation Apex Consultants has been committed to producing high-quality, innovative results in evaluation over the past 10 years. Our work is reflective of the competencies for Canadian evaluators and upholds a high standard of evaluative practice. We believe this will be inherent in our proposal. We would like to thank you for the opportunity to prepare this proposal and we look forward to providing you with evaluation support. To best meet your needs, we recommend a meeting to discuss the proposed plan and evaluation needs in further detail. Should you have any questions or concerns regarding the proposal, please do not hesitate to contact us. Sincerely, Apex Consultants 321 Main Street, Forest City, XY, B1A 3C2 T: Evaluate@ApexConsultants.ca

2 CONSULTANTS Evaluation Proposal for Immigrant Women Services Ottawa Settlement and Integration Services Prepared for: Immigration and Women Services Ottawa February 7, 2015

3 Table&of&Contents&! 1.0$Introduction$...$1! 1.1$Program$Stakeholders$...$1! 2.0$Critique$of$the$Current$Logic$Model$...$2! 2.1$Revised$Logic$Model$...$3! 3.0$Evaluation$Design$...$3! 3.1$Evaluation$Purpose$...$3! 3.2$Evaluation$Type$...$4! 3.3$Evaluation$Approach$...$4! 3.4$Evaluation$Context$...$4! 4.0$Methods$for$Evaluation$...$5! 4.1$Data$Collection$Methods$...$5! 4.2$Data$Analysis$...$7! 5.0$Anticipated$Challenges$&$Mitigation$Strategies$...$8! 6.0$Demonstrated$Core$Evaluation$Competencies$...$8! 7.0$Conclusions$...$9$ Appendix A: Draft Logic Model for IWSO Settlement and Integration Services Appendix B: Proposed Evaluation Matrix

4 1.0&Introduction&& Immigrant Women Services Ottawa (IWSO) is a registered charity, established in 1988, serving immigrant and visible minority women in Ottawa and the surrounding area. By providing culturally-responsive and integrated services, the IWSO aims to help immigrant women overcome violent situations, and empower them to reach their full potential. In , IWSO expanded their services to include the Settlement and Integration Services (SIS) and the Language Interpretation and Translation program. The IWSO-SIS serves women from a wide range of cultures with unique settlement needs that are grounded in their life history (language, education, personal experiences), and reasons for immigration. As identified in the Request for Proposal (RFP), IWSO-SIS uses an empowerment focus to enable clients to become actively involved in their community and contribute socially, economically and culturally. Services include: Needs Assessment and Referral to tail information and referrals to specific needs Newcomer Orientation, including one-on-one counselling, group information/orientation, and social and recreational activities Employment Services, including job search workshops, French language skills, computer training courses and child-care services The needs assessment referrals and newcomer orientation sessions make up the core services of IWSO-SIS and the number of clients served by these activities has increased from 185 clients in the first year to an average of 355 clients in the previous three years. Job search workshops are delivered in partnership with World Skills, and a total of 16 were delivered 2013/14, with an average of 149 clients served per year. IWSO-SIS is funded by Citizenship and Immigration Canada, and receives $293,752 annually. The program employs four full-time employees (two settlement counsellors and two full-time ECE staff), 0.2 full-time management/supervisory staff, and 0.2 full-time computer instructors. The program has volunteers to deliver their Social and Recreational Activities and French Language Skills circles. While IWSO-SIS services are promoted by other agencies and outreach events, a number of women connect with the program prior to immigration through the IWSO website to receive pre-arrival services. 1.1#Program#Stakeholders# Based on information provided in the RFP, we have identified a preliminary list of key stakeholders (Table 1) for IWSO-SIS and have begun identifying the information needs of each group. We recommend that members from each of the key stakeholder groups be involved in the evaluation by forming an Evaluation Advisory Committee (EAC). Each stakeholder group should have at least one representative present in the EAC to direct evaluation planning. The first activity of the EAC will be to identify any additional stakeholders and to verify their respective information needs. In addition to the EAC, we recommend that an Executive Evaluation Team (EEC) be gathered; the EEC will consist of the IWSO s board of directors and senior management and will be involved in all evaluation planning and decision making in order to assist them in developing evaluation capacity for future evaluations. 1

5 Table 1. List of Key Stakeholders for IWSO-SIS and Anticipated Information Needs Stakeholder Group Information Needs Primary SIS Staff and Volunteers - How can we improve the program? - What skills and tools are necessary to help clients, and are my current skills sufficient? SIS Clients and their Families - What is the value of the program? - What benefits will I gain for my family and me? IWSO Board of Directors and Senior Management - What skills are needed for future evaluations? - What resources are required for successful operation? Secondary Community Partners (World Skills, Catholic Centre for Immigrants) - Do IWSO s objectives align with ours? - Do we benefit by collaborating with IWSO? IWSO Crisis Intervention and Counseling Program, and Language Interpretation and Translation Program - How effective are the promotional activities to reaching new immigrants? - What challenges to women experience navigating across programs? Tertiary Community at large - Are community members aware of IWSO-SIS services? - Do community members support new women immigrants? Potential Employers - Are the skills provided through IWSO sufficient for current employment opportunities? 2.0&Critique&of&the&Current&Logic&Model& A program logic model is an evaluation tool that is used to illustrate the main components and activities of the program, identify relevant indicators for each activity, and the connections and linkages between the components as they relate to the shortterm, intermediate and long-term program objectives. IWOS-SIS has developed a logic model; however, based on the information provided in the RFP, Apex Consultants believes that the current logic model can be revised to better reflect the program. Overall Design: We found the present logic model to be well designed. However we note that while the second row of the logic model (e.g., describing the target group as client group ) demonstrates the programs understanding of each component, it is redundant and could be removed in order to provide a better aesthetically pleasing visual. Context and External Factors: The present logic model did not take into consideration the context that the program operates within, or external factors that may impact the success of SIS. We propose adding this section to the logic model to aid in articulating the social and political climate in which the intended outcomes will occur. Assumptions & Risks: Assumptions reflect events/conditions that are needed in order for the program linkages to be actualized, while risks are external events/conditions that may put that those links at risk. A number of the outcomes identified in the existing logic model serve as program assumptions (e.g., through program participation, clients have accurate information to make informed settlement choices), and should be articulated as such. Further, describing assumptions and risks, will aid in the development of a theory of change, which will be critical in assessing long-term outcomes. Inputs: Program inputs should be placed in the logic model before program activities, as the inputs are describing the resources that are available in order to offer the program. We found that IWSO-SIS has sufficiently described their staffing levels, however, the remaining inputs identified (e.g., utilizing existing technology) are not inputs into the 2

6 program activities. Furthermore, available funding is an important input that we recommend be added to this section. Target Group: We believe that the current logic model adequately captures the target group as immigrant women, which include permanent residents, convention refugees, and live-in caregivers. However, it is important to note that some women connecting IWSO- SIS may be receiving pre-arrival assistance. Components: The current logic model does not specify program components separately from the program activities. We believes components should be added to the logic model, as it helps to clearly articulate the delineations of the program activities, so it will be easier to demonstrate how specific activities lead to certain outcomes. Activities: After examining the program activities identified in the current logic model, we feel that the current descriptions reflect the program. However, some descriptions lack detail and do not concretely reflect the services that are offered by IWSO-SIS. For example, computer training, French conversation circles, and job search workshops are important activities that could be added to reflect IWSO-SIS services. Outputs: The current logic model articulates outputs as their targets for We believe that all components of the logic model should reflect the general operation of the program across years, and recommend that the targets for be removed. We also believe that the linkages between program activities and outputs could be more clearly stated to demonstrate which activities can be tracked by specific outputs. Outcomes: After examination of the outcomes provided in the current logic model, we believe that many of the stated outcomes are better reflected as activities (e.g., clients receive information and referrals), and that there are a number of cases where stated outcomes were misclassified (e.g., developing a positive attitude reflects long-term outcomes, not short-term). We also noted that the ultimate goal of IWSO-SIS was not stated and recommend that it be added as this shapes the overall structure of the program. Apex Consultants would like to present the current critique to the EET to build evaluation capacity by demonstrating how we formulated each component of our critique. 2.1#Revised#Logic#Model# Based on the critique of the current logic model, Apex Consultants would like to propose a revised logic model to demonstrate how program objectives relate to the activities and outcomes (see Appendix A). This logic model begins to illustrate our understanding of the assumptions and risks underpinning SIS that can contribute to the theory of change. Apex Consultants would like to review this draft logic model with both the EET and EAC to ensure the logic model reflects the program. 3.0&Evaluation&Design& 3.1#Evaluation#Purpose# IWSO-SIS has identified a need to evaluate the program s performance, with a focus on long-term outcomes, any potential delivery improvements, and relevance. Based on the primary objectives outlined in the RFP, we have identified the following preliminary evaluation questions. 3

7 1) How well is IWSO-SIS performing? 2) How can IWSO-SIS delivery be improved? 3) Is IWSO-SIS relevant to clients? In consultation with the EAC, the above evaluation questions will be appropriately refined to ensure that we are meeting the evaluation needs of IWSO-SIS. We will also work with the EET in order to discuss the process we undertook to identify and articulate the proposed evaluation questions. Appendix B outlines an evaluation matrix that connects our proposed evaluation questions with indicators, sources of data, proposed methodologies and sample information. These methods will be discussed in detail in section #Evaluation#Type# In order to meet the needs of IWSO-SIS, Apex Consultants is proposing a two-stage evaluation. The first stage will be a process evaluation, which will: systematically evaluate the activities in place; identify the strengths and weaknesses of daily operations; provide insight into the experiences of client utilizing the program, including participation levels and barriers to access. The second stage will be a short-term and intermediate-term outcome evaluation to assess the extent to which desired client outcomes are being attained. While the RFP requests a focus on long-term outcomes, Apex Consultants believe the program has not been running for enough time to assess long-term changes. However, we have identified indicators and proposed methods in our matrix (Appendix B), which will assist the Board of Directors and program managers to conduct an evaluation of long-term outcomes at a later date. 3.3#Evaluation#Approach# IWSO-SIS is committed to providing practical and accurate services that empower immigrant women to overcome barriers to successful settlement in the Ottawa area. As such, Apex Consultants aims to adopt a participatory approach to our evaluation to ensure the insights and experiences of program clients, staff and volunteers are captured. Furthermore, by directly involving the key stakeholders of IWSO-SIS, including senior management and the Board of Directors, we can assist IWSO-SIS in building evaluation capacity. Given the interest of the IWSO-SIS to increase their own capacity, Apex Consultants plans to actively engage the EET in the evaluation planning process, as well as present our logic model critique and proposed logic model for additional feedback. Finally, Apex Consultants also aims to adopt a culturally responsive approach, focusing on the influences of the diverse cultural context of IWSO-SIS. We ensure to examine the program and any connections through a culturally sensitive lens to ensure IWSO-SIS goals and objectives meet the diverse needs of their clients. 3.4#Evaluation#Context# IWSO-SIS offers services to clients with diverse cultural backgrounds who come from a variety of contexts and therefore have unique needs for and expectations of immigration to Canada. For example, IWSO-SIS serves women who have experienced violence or abuse, as well as those who have not. Furthermore, many clients may not speak English as a first language; therefore, Apex Consultants has drafted this proposed evaluation 4

8 under the assumption that IWSO-SIS has translation resources available for the evaluation team to utilize during the evaluation process. Currently, the program has not undergone a formal evaluation. However, IWSO-SIS has collected client feedback on a variety of their services, including their Free Yoga for Women classes, French Conversation Circles, and the Computer Training Program. The program has also measured participation (number of participants over years, number of workshops) of their Newcomer Orientation, Job Search Workshops, and Computer Training Program. Because of this, Apex Consultants has prepared this proposed evaluation with the assumptions that IWSO-SIS collects ongoing program registration and attendance, and that we will have access to previous program feedback. Although a budget for the evaluation has not been provided, Apex Consultants has prepared this proposal under the assumption that adequate funding is available to support the evaluation activities. When developing our proposal, Apex Consultants has taken into account the timeline outlined in the RFP (i.e., days) and believe that the proposed evaluation activities can be completed in that time. 4.0&Methods&for&Evaluation&& Apex Consultants believes that to evaluate the IWSO-SIS, a variety of information sources should be consulted: (1) key program documents; (2) published literature; (3) program managers, staff, and volunteers; (4) program clients. In order to gather information from these sources, Apex Consultants has identified 7 methods that will aid in collecting the required data. The Executive Evaluation Team (EET) will be called upon at the initial stages of and throughout the evaluation for their support as a way to support evaluation capacity building. 4.1#Data#Collection#Methods# (1) Document Review: A document review will be used to: (1) understand the history and operation of the program particularly characteristics of individuals participating in the IWSO-SIS; (2) assess the logic and validity behind the intended plan of implementation (e.g., goals, activities, outputs); (3) ensure readability of materials offered to clients; (4) ensure materials offered to clients contain culturally appropriate information. The document review will entail the review of program records (e.g., administrative data regarding program use), policies and procedures, previous feedback from clients, promotional materials, and program materials offered to clients. Documents will be requested from program managers and staff members (e.g., settlement counsellors and early childhood educators). Canada Census data will be used to identify immigrants entering Canada and settling in the Ottawa area since 2009 to assist with determining the reach of the program and identifying potential non-program users. Strengths of method: Low cost, gather existing information, low burden. (2) Literature Review: A review of the published and unpublished literature will be completed to: (1) assess the program theory; and (2) gather information about the operation of similar immigrant settlement programs and their outcomes. The inclusion criteria are immigrant settlement programs from developed nations that accept immigrants and refugees (e.g., Canada, United States, Australia, United Kingdom), as 5

9 well as settlement programs that are designed for women. The exclusion criteria will be settlement programs that focus on men. The EET and EAC will be asked to identify other inclusion and exclusion criteria to assist with the literature review. Strengths of method: Low cost, objective & existing information, validate evaluation. (3) Key Informant Interviews: Key informant interviews will be conducted to assess the IWSO-SIS operations. Semi-structured interviews will be conducted with the program managers and staff members in a private space at the IWSO-SIS main office, and should each take approximately 15 to 30 minutes. The interviews will be voice recorded and transcribed. Potential interview questions should speak to the following topics: (1) What are some challenges that program staff face in delivering the IWSO-SIS services?; and (2) How do program staff perceive how their clients are progressing through the settlement and integration process? The EET and EAC will be asked to identify questions and how they will be phrased to ensure appropriateness of the interview questions. Strengths of method: Develop relationship with program staff, rich data can be collected. (4) Site Observations: Site observations will be used to (1) gather information about how program staff and volunteers implement the IWSO-SIS; (2) compare actual program functions with intended program functions/plan; and (3) ensure evaluators have the appropriate cultural competencies required to properly interact with clients. Site observations will entail collection of information around program operations and processes, stakeholder interactions, implementation of procedures related to the IWSO- SIS. For pragmatic reasons (e.g., time availability) four (4) site observations will be conducted over the span of one week. Each of the 4 site observations will be used to observe interactions in the following settings / activities: (1) the reception area where clients come in and have their initial interaction the program staff; (2) group information/orientation sessions; (3) one social activity (e.g., Arts and Crafts Circles); and (4) one job search workshop. A checklist of potential information to capture will be developed in with the EET and EAC so that site observations can be standardized. Potential checklist components include: (1) How the referral process occurs; (2) How clients interact in a social setting. Strengths of method: No participant interaction required, snapshot of how the program actually functions, generate additional questions about program operation. (5) Focus groups: Focus groups with program volunteers will be conducted to: (1) gain in-depth understanding of their impressions regarding the relevance of services offered and the utility of services by clients; and (2) gather suggestions for improvement of the service delivery. Focus groups will be conducted in person at the IWSO-SIS site. Two (2) focus group sessions will be conducted with 6-8 participants each. Focus groups will be facilitated by 2 consultants; one will be ask questions and the other will note observations within the session. The sessions will be voice recorded and transcribed. Potential focus group questions to be answered include: (1) Do you believe the social and recreational activities are relevant to the clients; (2) How are clients interacting with each other in the social activities? Additional questions and specific phrasing of the focus group questions will be revised in collaboration with the EET and EAC. Strengths of method: Gather rich data about program, involves participant interaction. 6

10 (6) World Café: A World Café will be conducted with clients of the IWSO-SIS to: (1) gain in-depth understanding of common perceptions and experiences of clients; and (2) identify social network connectedness. The World Café will be conducted over 2 half day sessions (~3 hours each) to include as many clients as possible. We anticipate approximately participants per session. Essentially, clients will be split into 5-10 groups, depending on how many clients participate in the World Café. Each group will be given a question to respond to and they will be asked to draw or write terms associated with each question or topic of discussion and then verbally reflect on the question/topic. Each group will include 2 facilitators (i.e., Apex Consultants and EET will be paired together); one facilitator will record notes of the clients verbal reflections and another will facilitate the discussion and questions. The EET will be trained to facilitate the sessions. Language translators will be available to assist clients as necessary. World Café participants will be recruited through the IWSO-SIS services by program staff and volunteers, program outreach and promotional materials, word of mouth, and social media. Participants will be asked to provide verbal consent prior to participating in the World Café. Potential topics of discussion include: (1) what independence looks like for clients; (2) what life in Canada looks like; (3) skills clients already have; (4) skills clients would like to develop; (5) agencies and community supports clients currently use. Additional topics will be identified in collaboration with the EET and EAC. Strengths of method: Gather rich data, efficient way to collect data, no issues around literacy, involves interaction and exchange of ideas among participants. (7) Social Network Analysis: An analysis of clients social network is planned to: (1) determine the key players (e.g., partners, employers, community agencies) in the integration of clients into the community; (2) determine how clients are connected to the community through their social network; and (3) identify the connectedness of clients and the IWSO on the social media platforms (e.g., Facebook). The social network analysis will characterize the structures within a network to understand the flow of factors, and will visually represent the relationships among the clients participating in the IWSO-SIS. In addition, topics of discussions will be noted to contribute to identification of needs. Apex Consultants sees the value in this method to assist the IWSO CIC determine the connectedness of the clients and the community as a whole. This method is described to provide the IWSO-SIS with a place to start and framework to work with, while building evaluation capacity among the EET so that they could follow-up with this method and accomplish it with the evaluation skills developed through our proposed evaluation. Strengths of method: Improve ways to disseminate information, identify partnerships and connections in the community, easy to access information. 4.2#Data#Analysis# Quantitative data will be analyzed using descriptive statistics and Statistical Package for Social Sciences. Qualitative data including pictures will be analyzed via thematic analysis through the use of NVivo 10. Triangulation from both qualitative and quantitative data will be applied to develop a more comprehensive understanding of the evaluation findings. The document review, key informant interviews, and site observations will be triangulated for program operations, while the World Café results and focus groups will 7

11 be triangulated to determine clients experiences with the IWSO-SIS program. Triangulation will aid Apex Consultants in assessing the validity, reliability and trustworthiness of the data. It should account for limitations of methods and will provide valuable information on the relationships within the program. Analysis and interpretation of the data will be conducted with the EET and EAC to ensure the report findings are specific and relevant to the IWSO-SIS program. 5.0&Anticipated&Challenges&&&Mitigation&Strategies& While we have made a strong effort to propose an evaluation that is practical and informative while still meeting the unique needs of IWSO-SIS, we understand that there are a number of challenges that may arise. As shown in Table 2, we have anticipated a variety of challenges that may occur and have suggested possible mitigation strategies. Table 2: Anticipated Challenges and Mitigation Strategies Anticipated Challenge Diverse or Vulnerable Women Confidentiality and Ethics Considerations Time Constraints Difficulty contacting stakeholders Mitigation Strategies - The EAC will be consulted to determine what translation services are required to ensure that relevant evaluation materials are accessible to clients; a readability analysis will be conducted to ensure that documents are prepared at a grade 6 reading level or lower. - Proposed methodologies with clients include a World Café that reduces the need to read or write. Active verbal consent, using appropriate translators, will be obtained to further minimize the need to read or write. - All evaluators will have vulnerable sector police records check prior to the beginning the evaluation and have participated in training workshops to enhance skills needed to work with diverse and vulnerable populations. - Culturally appropriate snacks and various food options (e.g., vegetarian, vegan, halal) will be available at the World Café. - All members of Apex are familiar with the ethics and standards of Canadian Evaluation Society and the evaluation will be conducted within these standards. - Active informed consent will be obtained prior to evaluation activities. - Apex will ensure all participants are aware that they can discontinue or stop participating in the evaluation at any time. - Apex will de-identify all participants in evaluation results. - Evaluation results will be stored in a secure database. - Apex strives for integrity and honesty (core competencies). - Evaluation activities in consultation with EAC will be discussed to adjust expectations. - We will utilize existing data through record reviews and will conduct site observations to ensure we have a thorough understanding of how the program operates. - Given the large number of program participants, the proposed World Café will be offered on two occasions to promote participation. Given its nature, this proposed method can accommodate a larger group of participants. - Weekly meetings with EAC (which encompasses a representative from all key stakeholder groups) can assist in identifying these barriers to contact. - Diverse knowledge dissemination plan will be able to reach most stakeholders. 6.0&Demonstrated&Core&Evaluation&Competencies&& Apex Consultants has demonstrated competencies for Canadian evaluation practice throughout our proposal in each of the five domains: (1) Reflective Practice; (2) Technical Practice; (3) Situational Practice; (4) Management Practice; and (5) Interpersonal Practice. 8

12 Reflective practice competencies are demonstrated by Apex Consultants through our commitment to respect all stakeholders (1.3). While we have developed a preliminary evaluation plan for the current proposal, Apex Consultants have proposed revising our plans through collaboration with an EAC. Further Apex has considered human rights and public welfare (1.4) as evidenced by our cultural considerations in section 5.0. Technical Practice competencies of Apex Consultants are exemplified through the framing of our preliminary evaluation questions (2.5) (see matrix) and the development of a novel proposed evaluation design (2.6) specific to the needs of the IWSO-SIS. Situational Practice competencies are evidenced by addressing the uniqueness of this site (3.1). The identified challenges and mitigation strategies in section 5.0 show explicit consideration of the program s complex target population and sociocultural context. Additionally the use of an EAC and EET demonstrate commitment to identifying interests of all stakeholders (3.4). Management Practice competencies of Apex Consultants are displayed through our commitment to ongoing meetings with the EAC to report progress and results (4.6). Additionally we display management practice by attending to issues of evaluation feasibility (4.2) by selecting methodologies that are appropriate for a day consulting window. Finally, Interpersonal Practice competencies are exemplified through the use of a mixed methods design for data collection that requires strong listening (5.3), and facilitation skills (5.6). These skills will be important in garnering feedback and gathering input from both program users and other stakeholders. 7.0&Conclusions& Apex Consultants has reviewed IWSO-SIS s existing logic model and has proposed a number of recommendations to enhance the comprehensiveness and to ensure it accurately reflects the true components of the services being provided. This proposal has outlined a process for evaluating the IWSO-SIS services and developing evaluation capacity among IWSO senior management and the Board of Directors. We have identified data collection methods for the proposed evaluation, as well as describing a method for future evaluation work. We look forward to meeting with you and the EAC to review this proposed evaluation plan and to ensure it meet the needs of your organization and your stakeholders. 9

13 Appendix A: Draft Logic Model for IWSO Settlement and Integration Services PROGRAM CONTEXT & EXTERNAL FACTORS: Language, cultural background & history of participants. Political & social context & stigma surrounding immigration. ASSUMPTIONS: Program matches with the needs of the broader community and employers. PROGRAM COMPONENTS OUTCOMES Inputs Program staff 4.0 FTE staff/year 0.2 FTE supervisors 0.2 FTE computer instructor Volunteers Materials Technology (computers) Partnerships Funding Follow-up info from clients Target Groups Immigrant women Permanent residents Convention refugees Live-in caregivers *Note Potential immigrants connect with the program for pre-arrival assistance Components Activities Outputs Short-Term Intermediate Long-term Overall Needs Assessment and Referral Newcomer Orientation Support Services Employmen t Services Computer Training Care for Newcomer Children - Initial intake & needs assessment - Referrals based on need - Settlement plans - 1-on-1 counseling - Friendship circles - Arts & Crafts circles - Social/ Recreational activities - Job search workshops -1-on-1 assessments w/ employment counselors - French conversation circles - 3 levels of computer training - Interpretation & translation - Childcare services # Initial needs assessments, referrals, settlement plans, women served # Info & counseling sessions, crafts & art sessions, & social activities # Workshops, assessments, French circles, trainings, translators, children in childcare! in # women reached! awareness in services & community agencies! knowledge about life in Canada! connections to community - Develop new skills to assist in the job market (incl. computer & language)! access to meeting w/ staff by offering childcare! access to settlement services! self-efficacy to make choices for settlement! Use of community resources & info on life in Canada Necessary skills! self-efficacy & confidence for the job market (incl. computer & language skills) Women broaden their networks (social & professional) Become resourceful in adapting to life in Canada & giving back to the community (e.g. peer mentoring) Improved SES through meaningful employment Empower clients to integrate into their community and enable them to contribute socially, economica lly, & culturally. Theory of Change Assumption: Funding, training, materials, & technology are adequate to meet program needs. Risks: Interest in SIS is low. Assumption: Program is relevant & meets the needs of the clients. Risks: Clients may not be interested/engaged with the services & thus, may not attend. Assumption: Target groups will utilize program activities, which will lead to S/T goals. Assumption: Sufficient, accurate info is provided & understood by clients to make informed decisions. Risks: Info is inaccurate, misinterpreted, or not used by clients. Assumption: Skills & behaviours obtained are relevant & maintained over time. Risks: Program clients do not desire change and this does not align with their values. Risks: Activities may not meet the needs of the target group. 10

14 Appendix B: Proposed Evaluation Matrix Evaluation Question/Problem Indicator Data Source Method How is the program Performing? Are clients meeting short term program objectives? Has awareness of community services increased? Has Program reached target clients? Have participants increased skills relevant to job market? Has social integration increased among clients? % of newly immigrated women who access services Total # of referrals to/from community services Staff/Volunteer perceptions of skill development among clients Client perceptions of skill development. Client perceived social integration Program Documents: - Registration info - Attendance Records Census Data: newly landed immigrants in Ottawa area Program management & Full time staff Program Volunteers Program Clients Document review Key Informant interviews Focus Groups- Volunteers World Café- Clients To what extent is the program meeting intermediate goals? Has use of community services increased? Has confidence for accessing employment increased? Has self-efficacy regarding settlement improved? Number of community services accessed by clients Client perceptions of employment opportunities Client self-efficacy regarding settlement (e.g., locating schools) Staff/Volunteer perceptions of client selfefficacy Program documents: - Outgoing Referrals Client reports Client reports Staff report Document Review World Café- Clients Key Informant interview: staff Focus Groups: Volunteers Are clients of the program on track to meet long term outcomes? Have clients broadened social and professional networks? Have clients secured meaningful employment? Are clients adapting to life in Canada and giving back to the new immigrant community? Number of connections made in the community # of clients who have secured meaningful employment # of current and former clients who report successfully adapting to life in Canada. # of clients who report working new immigrants Program Documents - Outgoing referrals Client reports - Reported connections - Reported adaptation to Canadian life - Reported working with new immigrants Staff and Volunteers perception Social Network analysis Document review World Café- Clients Key Informant Focus Group- Volunteers Key informant interview- Staff 11

15 2.0 How could program delivery be improved? How could the program delivery be improved? Degree of consistency with similar successful programs Degree to which program delivery is consistent with intended delivery Wait times for clients to access desired resources Delivery Challenges of staff Reported access challenges of clients Published and Unpublished Literature Observation of program activities Time between first contact and first attended program Management and fulltime staff Client report Literatures review site observation Document review Key Informant interviews- staff Focus groupsvolunteers World Cafe-Clients 3.0 Is the program relevant to the needs of program clients? 3.1 Are the programs relevant to the needs of clients? Does training complement the skills participants already possess? Do skills learned help individuals access employment/ self-efficacy/ independence? Client/ Staff/ Volunteer perceptions of training adequacy Client/ staff perceptions of training relevance to existing market place Client perceptions of training improving self-efficacy/independence Staff report/opinion Volunteer report/opinion Client report/opinion Key informant interviews-staff Focus groups- Volunteers World Cafe- Clients 3.2 Are the programs sufficient to meet the needs of IWSO-SIS clients? What additional skills may help clients achieve employment, self-efficacy, and independence? # of training domains staff/volunteers report as being necessary to achieve desired outcomes, not currently offered # of training domains clients report as being necessary to achieve desired outcomes, not currently offered Staff report/opinion Volunteer report/opinion Client report/opinion Key informant interviews-staff Focus groups- Volunteers World Cafe- Clients 12