An Evaluation Framework for United Way of Greater Toronto s Building Strong Neighbourhoods Strategy. Prepared by

Transcription

1 An Evaluation Framework for United Way of Greater Toronto s Building Strong Neighbourhoods Strategy Prepared by April 2007 i

2 Table of Contents INTRODUCTION... 1 INTRODUCING THE BUILDING STRONG NEIGHBOURHOODS STRATEGY... 1 METHODOLOGY FOR DEVELOPMENT OF THIS FRAMEWORK... 4 HOW THIS EVALUATION FRAMEWORK IS ORGANIZED... 4 INTRODUCING THE COMPONENTS... 6 COMPONENTS THAT FUNCTION AT A BROAD COMMUNITY OR SYSTEMS LEVEL... 6 Public Awareness and Marketing: Description... 6 Influencing Public Policy: Description... 8 Resource Development: Description... 9 Targeted Funding: Description Research & Learning: Description Summary of System and Broad Community-level Components COMPONENTS THAT FUNCTION AT A NEIGHBOURHOOD LEVEL Action for Neighbourhood Change: Description ANC Capacity Building: Description ANC Quick Start: Description Neighbourhood Hubs: Description EVALUATION PLAN PART 1: EVALUATING THE BROAD COMMUNITY AND SYSTEMS- LEVEL COMPONENTS OF THE STRATEGY METHODS COMMON TO ALL COMPONENTS AND METHODS TO TRACK THE OVERALL PROGRESS OF BSNS Common Evaluation Questions PUBLIC AWARENESS AND MARKETING: EVALUATION PLANS Key Evaluation Questions: INFLUENCING PUBLIC POLICY: EVALUATION PLANS Key Evaluation Questions: RESOURCE DEVELOPMENT: EVALUATION PLANS Key Evaluation Questions: TARGETED FUNDING: EVALUATION PLANS Key Evaluation Questions: RESEARCH & LEARNING: EVALUATION PLANS Key Evaluation Questions SUMMARIZING THE EVALUATION PLAN FOR SYSTEMS AND BROAD COMMUNITY LEVEL COMPONENTS: 34 EVALUATION PLAN PART 2: EVALUATING THE NEIGHBOURHOOD-LEVEL COMPONENTS OF THE STRATEGY METHODS THAT RELATE TO ALL NEIGHBOURHOOD-LEVEL COMPONENTS ACTION FOR NEIGHBOURHOOD CHANGE: EVALUATION PLANS Key Evaluation Questions: ANC CAPACITY BUILDING: EVALUATION PLANS Key Evaluation Questions ANC QUICK START: EVALUATION PLANS Key Evaluation Questions NEIGHBOURHOOD HUBS: EVALUATION PLANS Key Evaluation Questions: ii

3 Index of Tables Table 1: Common Evaluation Methods for System Level Components Table 2: Proposed Evaluation Methods for Public Awareness & Marketing Component Table 3: Proposed Evaluation Methods for Influencing Public Policy Component Table 4: Proposed Evaluation Methods for Resource Development Component Table 5: Proposed Evaluation Methods for Targeted Funding Component Table 6: Proposed Evaluation Methods for Research & Learning Component Table 7: Major Tasks for Systems Level Evaluation Table 8: An Overview of Proposed System-Level Evaluation Methods Table 9: Proposed Evaluation Methods for Quick Start Component Table 10: Proposed Evaluation Methods for Neighbourhood Hubs Component Table 11: Major Tasks for Neighbourhood Level Evaluation Table 12: An Overview of Proposed Evaluation Methods for ANC Table 13: An Overview of Proposed Evaluation Methods for Quick Start and Neighbourhood Hubs This evaluation framework was commissioned by The United Way of Greater Toronto This evaluation framework was prepared by Centre for Research and Education in Human Services 73 King St. W., Kitchener, On N2G 1A CREHS Team Members: UWGT Team Members: Andrew Taylor, Jason Newberry & Suzanne Field Susan MacDonnell & Michi Motomura iii

4 INTRODUCTION Introduction Introducing the Building Strong Neighbourhoods Strategy The Building Strong Neighbourhoods Strategy is UWGT s plan for strengthening and revitalizing the city s high-need, inner suburban neighbourhoods. It is a five-year, crossorganization plan, which mobilizes all of UWGT s resources, including fundraising, fund allocation, research and advocacy, public policy, capacity building, community development, convening, and communications. At its broadest level, the Strategy aims to influence public and political thinking about neighbourhoods and their importance to the long-term health and prosperity of the city. At the neighbourhood level, the Strategy aims to build local leadership and influence over the investments that are brought into communities, and to strengthen the system of local services and infrastructure. The purpose of the BSNS work at four distinct levels; WIDER SOCIETY: To increase public and political understanding of need for strong neighbourhoods FUNDERS AND POLICY MAKERS: To mobilize and leverage donor and other funding SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods RESIDENT LEVEL: To Increase the Leadership and Influence of Local Residents over neighbourhood conditions, and build collaborative action UWGT s Building Strong Neighbourhoods Strategy is an exciting example of what evaluators have come to call complex interventions. 1 Complex interventions are largescale service delivery undertakings in which one or more large funders work with many different organizations in different communities to manage and deliver a cluster of components or strategies intended to address shared outcome objectives in a variety of complementary ways. In order to ensure that services are adapted to local needs, local partners take on at least some of the responsibility for needs assessment, program design, program management, and outcome measurement. The Strong Neighbourhood Strategy s emphasis on local control and neighbourhood engagement reflects this principle. The table below provides examples of the work of the Strategy at each of its four levels; 1 Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review a new method of systematic review designed for complex policy interventions. Journal of Health Services Research and Policy, 10 (1), Building Strong Neighbourhoods Strategy: Evaluation Framework p. 1

5 INTRODUCTION Levels of Evaluation & Key Objectives WIDER SOCIETY: To increase public and political understanding of need for strong neighbourhoods FUNDERS AND POLICY MAKERS To mobilize and leverage donor and other funding SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods RESIDENT LEVEL To Increase the Leadership and Influence of Local Residents over neighbourhood conditions, and build collaborative action Core Short term Outcome Objectives Increased engagement, involvement and buy-in to the work of the Strong Neighbourhood Strategy on the part of politicians, businesses and donors Increased awareness and understanding in the wider society of the need for a placebased approach UWGT seen as a credible, reliable and valuable resource and leader in the community Increased donor identification with, involvement in and connection to the project and its goals Knowledge and practice of funders and policy makers is grounded in evidence Increased knowledge, awareness and practical skills for staff, boards and animators Development of networks between agencies within and neighbourhoods More effective identification of community issues and needs by agencies Evolution of mission statement from service orientation to change orientation More strategic and effective investment of resources Increased accessibility and use of services Increased resident influence in agency development of priorities and programs Increased resident leadership and capacity building opportunities through agency structures and programming Individual: Positive changes in attitude and greater sense of hope for change Improved knowledge, skills and self esteem Sense of belonging to neighbourhood Team: Increased involvement, ownership, buy-in and commitment to a residents association Increased pool of diverse, skilled leaders Resident Association: Improved plans, priorities and practices within communities Increased capacity to communicate, articulate needs and advocate for solutions Core Long Term Outcome Objectives Deeper understanding in the wider society of social issues, challenges and possible solutions Increased capacity among partners for strategic planning and action Increased donations to UWGT Improved trust and coordination between UWGT, donors and broader community More appropriate and strategic neighbourhood and organizational vision Improved agency responsiveness, inclusiveness and accessibility Improved sustainability of community change processes within recipient agencies Greater geographic and needs based balance in UWGT funding Development of cross neighbourhood strategic networks Increased resident involvement and ownership of neighbourhood/community-based services Group cohesion across diverse community residents, leaders and partners Increased sense of community identity and belonging Shift from bonding capital within cultural groups to bridging capital across neighbourhoods Building Strong Neighbourhoods Strategy: Evaluation Framework p. 2

6 INTRODUCTION Overall Approach to Evaluation For complex interventions, the principle of shared control must also be reflected in the evaluation process. This creates unique challenges for the development of an evaluation framework. Both programs and short-term outcome objectives are likely to differ across neighbourhoods. As a result, data collection tools and methods may not be consistent across sites. Still more challenges will arise because control over the outcome evaluation process is intended to be shared. Because partners are being given some responsibility for program planning, they will also have an interest in the foci of the outcome measurement part of the work. This means that outcome evaluation plans and reports will have multiple authors and multiple audiences. It also means that the timing and the quality of evaluation activities is very likely to vary from place to place. Complex interventions often have long implementation chains involving many levels of intervention and many stakeholders. In the case of the UWGT Building Strong Neighbourhoods Strategy, the four levels of intended impacts reflect this aspect of the intervention s theory of change. Often the success of later steps in the chain assumes that different people have already had success at earlier stages. These implementation chains can influence one another in a variety of ways. Sometimes the actions of staff running a local, front-line program lead to changes in the behaviour of policy makers or granting organizations. The outcome objectives of complex interventions tend to evolve over time and require managers that are skilled at interpreting information and adapting quickly. In this kind of program demonstrable achievement of pre-determined goals and generalizability of results become less meaningful, and the emphasis of evaluation shifts to learning to think and act evaluatively in a constantly changing context. Despite the dynamic nature of outcomes in complex interventions, developing theories of change, outcome objectives and measurement plans remains pivotally important because it provides an opportunity for stakeholders to continually connect with one another to reflect critically on their progress. Specific short term outcomes may change, but the commitment to focus on outcomes and outcome measurement remains constant. Developing Theories of Change: Outcome evaluation is centrally important within complex interventions like the Building Strong Neighbourhoods Strategy, because it is seen as a means of maintaining accountability while encouraging local ownership and adaptation. This framework is based on the belief that the key to successful monitoring and evaluation for complex interventions is ongoing efforts to develop a shared understanding of the intervention's theory (or theories) of change. Developing Performance Measurement Strategies: The logical relationships between the four levels of outcomes in the Building Strong Neighbourhoods Strategy are neither linear nor simple. Each level will require a distinct approach to evaluation. When an attempt is made to develop a single, centrally coordinated evaluation plan that considers many levels of outcomes in a fully integrated way, evaluation plans tend to become complex, time consuming and difficult for stakeholders to understand. In order to Building Strong Neighbourhoods Strategy: Evaluation Framework p. 3

7 INTRODUCTION increase the efficiency and flexibility of the overall design, this framework includes distinct performance measurement strategies for each of the four levels. In order to accommodate this possibility, each section of our performance measurement strategy is accompanied by an appendix organized in a tabular form where possible indicators and measurement strategies are organized according to outcome objectives. In this way, the strategy will identify areas where several different kinds of data from different communities might fruitfully be brought together to generate new insights. This performance measurement strategy identifies key process and implementation questions as well as outcome questions. For each component, milestones are identified to plot the progress of the component through stages of implementation over time. Methodology for Development of this Framework This evaluation framework was developed through consultation with United Way staff teams involved in the Strong Neighbourhoods Strategy. Each team met to develop a program logic model for their component. Draft models were then circulated to the team and to the Research Department for feedback. A second round of consultations used the revised logic models to develop preliminary measurement plans. Finalized logic models for each component are appended. These measurement plans were further developed by the evaluation team, and are appended to this framework as measurement tables. How this Evaluation Framework is Organized This framework includes three major sections. Following this introduction is a section that provides a basic introduction to the 8 major components or programs that make up the Building Strong Neighbourhoods Strategy. For each component, a basic description is followed by a list of the major activity clusters that constitute the component. Most subsections also include a list of the anticipated milestones in the development of this component over time. This information is drawn from Component Guidelines documents that have been prepared for each component (available through United Way), and from a set of program logic models for each component that are included as appendices to this framework. These introductory comments are especially important for this evaluation project because evaluability assessment and process evaluation are crucial components of this measurement plan. The Strategy is moving forward quickly, clarifying and deepening its theory of change in the process. These introductory sessions are intended to provide the evaluation team with an overall understanding of the activities that make up the Strategy and the ways in which these activities are evolving over time. Following the Introducing the Components section are two Evaluation Plan sections. These sections deal, respectively, with those components that function at a broad community or systems level, and those that work at a neighbourhood level. In this framework, each cluster is given a distinct and relatively independent evaluation plan. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 4

8 INTRODUCTION The first Evaluation Plan section focuses on those elements of the Strategy that function primary at a broad community or systems level. For each of the five components in this category, comments are made about the most appropriate approach to evaluation. A list of key questions to guide evaluation is provided, and there is also a table that recommends a set of data gathering methods to address these questions. The information contained in these sections is drawn from detailed measurement tables that have been prepared for each component. These measurement tables include much more information about the specific indicators and questions that should be included in various measurement tools, and they are appended to this framework. Because the evaluations of these five components are not completely independent, this section concludes with a synthesis that lays out a comprehensive workplan, with timelines, for evaluation of all five components. The section Evaluation Plan section deals with the components that function at a neighbourhood level. It follows the same basic structure as the systems level evaluation plan. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 5

9 INTRODUCING THE COMPONENTS Introducing the Components The Building Strong Neighbourhoods Strategy is made up of 10 components. Nine of the components are described in this evaluation plan. The 10 th component funding to member agencies for community development is still in development and will be incorporated into the overall evaluation plan at a later date. The nine components are briefly described below. This evaluation framework is divided into two main sections. The first deals with those components of the Strong Neighbourhood Strategy that function primarily at a broad systems level or an agency level. The second section focuses on those components that function primarily at a neighbourhood level. Components that function at a broad community or systems level Five of the components in the Strong Neighbourhoods Strategy function primarily at a systems or broad community level. They are intended, in various ways, to build support for the notion of neighbourhood development, and to embed neighbourhood development principles in practices of funders, policy makers, donors, opinion leaders, politicians, service providers and others. Within this cluster of five components, four are closely inter-related. Public Awareness and Marketing and Research and Learning both transform the lessons being learned at a neighbourhood level into messages and communication strategies that can be applied in various ways by the Influencing Public Policy component and the Resource Development component. Targeted Funding is more distinct, with a focus of altering the way United Way allocates resources in order to strengthen social service agencies serving inner suburbs. Public Awareness and Marketing: Description The public awareness and marketing component of the Strong Neighbourhood Strategy works to raise public and donor awareness of the importance of strong neighbourhoods to the health and long-term prosperity of Toronto. It has already served to further understanding and laid the groundwork for commitment to the importance of building local leadership, engaging residents and mobilizing investments to strengthen the system of local services and social infrastructure. As UWGT s Neighbourhood Strategy evolves in later half of 2006 and throughout 2007, the marketing and communications focus builds on the initial awareness objective and begins to demonstrate the foundational role and high-potential impact of ANC and Community Hub initiatives in the overall Neighbourhood Strategy and at the specific community level. Concurrent with the launch of new ANC and Community Hub sites in Toronto, UWGT marketing and communications efforts will also be directed to providing technical assistance support for capacity building and facilitating community-based communications strategies. In addition, specific communications strategies will be Building Strong Neighbourhoods Strategy: Evaluation Framework p. 6

10 INTRODUCING THE COMPONENTS developed to support the peer-learning networks and to share their progress and stories of change with key stakeholders. The Public Awareness and Marketing component advances two of the Strong Neighbourhood Strategy s overall objectives: WIDER SOCIETY: To increase public and political understanding of need for strong neighbourhoods FUNDERS AND POLICY MAKERS: To mobilize and leverage donor and other funding The logic model recently developed for the marketing component includes three main clusters of activities or strategies: Public awareness campaign. This cluster of activities focuses on the development of core messages about the Strong Neighbourhood Strategy on the basis of information from several sources within and outside the United Way. It also involves a variety of strategies intended to disseminate these messages to the general public and in particular to opinion leaders. The intended short-term outcomes of these efforts are that the messages appear in the media, and that these appearances are clear, consistent, and in-depth. Donor Campaign: This cluster of activities focuses on developing materials and messages (packaged as a case for support that resource development staff and others within the United Way can use in their efforts to raise funds and build support for the strong neighbourhood strategy. These efforts are intended to increase the capacity of resource development staff to communicate the case for support clearly and persuasively. Neighbourhood Capacity Building: Marketing and communications staff members are also involved in efforts to build capacity within ANC site teams and neighbourhood associations. Although most of this component works at a systems or broad community level, the neighbourhood capacity building cluster of activities functions at the neighbourhood level. It is intended to increase the ability of residents to deliver key messages to the wider community and speak to the media. The long-term outcomes of this component s work include an increased awareness & knowledge of the Neighbourhood Strategy (among both donors and the public at large), as well as increased awareness of its impact on the community and the value of a placebased approach more generally. More information about the intended outcomes of the marketing and communications component is included in the appended program logic model. Milestones For Public Awareness and Marketing For the public awareness and marketing component, milestones are less obvious, because their work is contingent upon specific implementation plans of ANCs and hubs. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 7

11 INTRODUCING THE COMPONENTS Influencing Public Policy: Description The BSNS component Strategies for Influencing Public Policy (IPP) is designed with two main purposes in mind. The first is to ensure ongoing and sustainable multi-source funding dollars for Neighbourhood Hubs and investment in initiatives that are placebased in focus. The second purpose is to promote and engage in dialogue with multiple partners (levels of government, etc.) to foster a long-term policy commitment to community development and neighbourhood-focused social interventions. These components certainly overlap, but policy commitments to a neighbourhood agenda carry benefits that go beyond basic direct funding dollars. The Influencing Public Policy component advances two of the Strong Neighbourhood Strategy s overall objectives: WIDER SOCIETY: To increase public and political understanding of need for strong neighbourhoods SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods The logic model recently developed for the public policy component includes three main clusters of activities or strategies: Procure Financial Support for Community Hubs. Under this activity category UWGT staff members on the Influencing Policy team are engaged in developing key policy messages for use in a government and community relations strategy. Hub leaders and representatives of service partners will be actively involved in order to communicate messages about implementation and the unique hub models developed in each neighbourhood. UWGT and hub leaders will identify key policy- and decision-makers and opportunities for partnership around hub investment. Short-term outcomes are focused on improved policy dialogue and partnerships between UWGT, policy-makers, and agencies, concerning hub development and sustainability. Promote a Neighbourhood Agenda. Promotion of a Neighbourhood Agenda requires a broader examination of relevant municipal, provincial, and federal government policies regarding community development strategies. UWGT wishes to forge relations and partnerships with government and other policy actors to prompt changes in public policy that reflect and support a neighbourhood lens and a place-based approach. To accomplish these objectives, UWGT will need to have up-to-date information on relevant policies and be able to pursue opportunities for partnership and dialogue as they evolve. In general, the goal is to establish a presence and voice at the most influential policy tables to move policy decisions in favor of the neighbourhood agenda. Long-term outcomes. In the longer-term, IPP hopes to improve political understanding of the need for strong neighbourhoods, greater investment in identified neighbourhoods (including hub support), and improved and expanded services where there is need. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 8

12 INTRODUCING THE COMPONENTS Milestones for Influencing Public Policy The sequence of events for IPP is yet to be specifically developed; however several activities, outputs, and outcomes comprise recommended milestones for the component. Corresponding evaluation data is also highlighted: Detailed government and community relations plan is developed. The plan should be assessed for its consistency with the logic model and should provide future actions. This should occur as early as possible, in the front half of Funder-community partnerships are in place. UWGT staff should track potential and actual partnerships and involvement of key actors. Activities ideally linked to the G&C relations plan should be specified. Although timelines for this component are unclear, much of the groundwork will be initiated by mid Quarterly reporting and tracking, as described above. It is difficult to set concrete milestones for IPP outcomes. A useful evaluation milestone is the compilation of quality tracking data and critical reflection (policy team focus groups) on impacts of the engagement strategy to date, with an agenda to improve practice and identify future opportunities. Collection of tracking data should begin alongside IPP activities, with the first reporting round occurring 2 or 3 months afterward. Resource Development: Description The Resource Development (RD or Targeted Gifts ) component attempts to secure significant individual and corporate donations and ongoing donor commitment to the Building Strong Neighbourhoods Strategy. The focus is being placed on individuals and foundations able to donate at minimum gift levels of $ Donor stewardship and awareness-raising is a key element in the approach, in order to demonstrate to donors the value of broad, long-term neighbourhood-based investment. The Resource Development component advances one of the Strong Neighbourhood Strategy s overall objectives: FUNDERS AND POLICY MAKERS: To mobilize and leverage donor and other funding The logic model recently developed for the resource development component includes three main clusters of activities or strategies: Prospect Identification & Relationship Cultivation. Prospect identification will be driven by UW major gifts staff and will be complemented by a volunteer Chairperson and the larger Major Individual Giving Cabinet. A Case for Support document and a range of proposals have been developed for use with prospective donors. UW major gifts staff will participate in a learning program regarding the strategy. A program of events will be developed to support the cultivation of prospect interest in the neighbourhood Strategy. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 9

13 INTRODUCING THE COMPONENTS Relationships will also be pursued to provide gifts-in-kind (e.g. volunteer hours, professional services). Solicit and Secure Resources. Following effective identification and cultivation, RD will solicit and secure donations. This activity includes the development of targeted proposals, strategically matching specific donors to specific initiatives within BSNS. Stewardship and Engagement. Resource Development also aims to nurture and sustain targeted gifts over the long-term. Donor stewardship will be given to donors who commit more than $ to the strategy. Donors will be engaged by RD through initiative updates and opportunities to become involved in other ways and to feel a greater connection to the work of the Strategy. The long-term outcome of the resource development component is to enhance the financial and non-financial resources available to implement the Neighbourhood Strategy. Milestones for Resource Development Develop a detailed messaging plan and train RD staff/volunteers on the essential elements of the Neighbourhood Strategy. The ongoing education of the sales staff will be essential to ensure they are able to share current information to their donor prospects. The first and second focus group with RD staff/volunteers and Marketing. This should ideally occur once volunteers have had an opportunity to test the messaging with prospective donors (perhaps 2 to 3 months). A second group at 6 or 9 months can follow to reflect on practice. Donation tracking. Financial and non-financial resources committed to Neighbourhood Strategy will be reported semi-annually. While this is the number one indicator of success, attempts may be made to analyze patterns as they relate to the impact of the cultivation programs. Targeted Funding: Description This component of UWGT s Neighbourhood Strategy was created to respond to the growing needs and lack of community services and programs in Toronto s inner suburbs. Through this initiative, UWGT will expand sustained investment in communities that have demonstrated a rapid growth in need, and by so doing, increase the availability of social services in Toronto s inner suburbs. The Targeted funding component of the Strong Neighbourhood Strategy is intended to build social service infrastructure in the inner suburbs, where relatively few agencies capable of providing services currently exist. The Targeted Funding component advances one of the Strong Neighbourhood Strategy s overall objectives: Building Strong Neighbourhoods Strategy: Evaluation Framework p. 10

14 INTRODUCING THE COMPONENTS SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods Allocations staff and volunteers will achieve this purpose through three activity categories. Bring new agencies into membership. Allocations staff will work to identify potential member agencies based in the inner suburbs, and will work with them to help them meet the criteria for membership. Ultimately, this process will lead to the recruitment of new member agencies. Allocate 75% of new funds to inner suburbs. Allocations staff will ensure that 75% of new monies allocated to member agencies each year will be invested in core operating costs and programs serving inner suburbs. These programs will be conceived, planned and run by existing member agencies. Targeted Grant Funding. This component will also invest through one-time grant funding aimed at inner suburbs. Funding applications will be evaluated with respect to the degree to which they embody the principles of the Strong Neighbourhoods Strategy. Long-term Outcomes of Targeted Funding. In the longer term, these new investments are intended to shift the overall distribution of United Way investments towards a balance that more closely geographic distribution of people in need. In turn, this approach will make services and supports more available to people living in the inner suburbs. Milestones for Targeted Funding Milestones can be used to track the progress of the allocations team in implementing the plan expressed in the guidelines document and the program logic model. The activity clusters described above suggest that the following are key milestones for the targeted funding approach. The evaluation process should use data from the methods described above to report regularly on the progress that the allocations team has made with respect to the following key milestones: Within the first six months, potential member agencies in each the inner suburbs have been identified and a strategy for working with them has been developed. By the end of the first year, there is a proportionate increase in core operating, program proposals, and grant proposals from inner suburbs (dependent on availability of new grant programs). By the end of the third year, strong applications for membership have been received from inner suburban communities. By the end of the second year, the proportion of UW funds invested in the inner suburbs will have increased. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 11

15 INTRODUCING THE COMPONENTS Research & Learning: Description The Research, Learning and Evaluation Plan is aimed at deepening our understanding of neighbourhood trends in the City of Toronto and of building and sharing knowledge of how United Way of Greater Toronto and others can play effective roles in neighbourhood revitalization. This component of the Neighbourhood Strategy is aimed at understanding the forces at play in city neighbourhoods, and also, of building knowledge and understanding of how UWGT and others can help to strengthen the City s affected neighbourhoods and how UWGT s Neighbourhood Strategy, in particular, is impacting the health and stability of high-need, inner suburban neighbourhoods. The Research and Learning component may advance all four of the Strong Neighbourhood Strategy s overall objectives: RESIDENT LEVEL: To Increase the Leadership and Influence of Local Residents over neighbourhood conditions, and build collaborative action SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods FUNDERS AND POLICY MAKERS: To mobilize and leverage donor and other funding WIDER SOCIETY: To increase public and political understanding of need for strong neighbourhoods To achieve these goals, the Research, Learning & Evaluation Plan involves a range of activities; Research on Neighbourhood Trends and Key Learnings includes commissioning research studies, reports on successful neighbourhood revitalization practices. Already, the research team has begun to create reports for each ANC neighbourhood that summarize available census data and other key indicators of neighbourhood health. Cross-Site Learning for ANC Site teams and partner Agencies. The research component also incorporates the creation of learning networks, provision of special learning events, and the production of on-going evaluation and learning, especially for those actively involved in ANC. Evaluation of The Building Strong Neighbourhoods Strategy. This component also includes evaluation activities, including the development of this framework. However, no evaluation of the evaluation activities is planned, and so this activity cluster is not discussed in this framework. The Research and Learning Component functions at both the broad community/systems level and the neighbourhood level. It has been included in the first category here because it is at this level that the research work requires its own evaluation methods. Most of the neighbourhood work undertaken by this team will be evaluated via ANC methods. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 12

16 INTRODUCING THE COMPONENTS Milestones for Research & Learning production of at least one report annually on neighbourhood trends production of a minimum of two reports on key ANC learnings annually coordination of a minimum of four learning and reflections sessions annually Summary of System and Broad Community-level Components The process that led to the development of this evaluation framework included intensive reflection on the BSNS s underlying theory of change. Through this process, it was possible to identify some of the core ideas that knit the strategy together. Especially in the first year of the Strategy s implementation, evaluation work needs to document and explore these underlying validity assumptions if it is to help the Strategy grow and succeed. Specifically, the systems component of the BSNS rest on the following key assumptions: The four components of the BSNS that work primarily at a systemic level are interconnected and rely on one another for success. Collectively, they also rely on the neighbourhood-level components. The Public Awareness and Marketing component, for example, can only develop messages if it gets strong examples and ideas from the ANC and Hubs components. The resource development team can only be persuasive if they are equipped with clear and well-developed messages from the Marketing team. Tracking inter-departmental communication about BSNS, and in particular tracking the movement of reflective, evaluative information will be key. The BSNS is intended to influence the way key decision makers think about neighbourhood-based approaches, so that they begin to use a neighbourhood lens in their work. In this evaluation plan, many different kinds of behaviours are proposed as reasonable grounds for inferring a change in mindset or worldview on the part of many different kinds of people. Being open to unexpected manifestations of the neighbourhood lens will also be important. Like any complex intervention, the Strong Neighbourhood Strategy continues to evolve and develop. Some components will undoubtedly alter, clarify or rethink their approaches in the coming year. This evolution is expected and encouraged, through a management approach that emphasizes milestones and movement through developmental stages. The evaluation design needs to emphasize careful tracking of activities, stories and processes as well as results. Intensive qualitative data collection and analytic methods that track evolution in thinking and behaviour over time are necessary. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 13

17 INTRODUCING THE COMPONENTS Components that Function at a Neighbourhood Level Action for Neighbourhood Change: Description Action for Neighbourhood Change (ANC) is a key component of the Building Strong Neighbourhoods Strategy involving direct and foundational community development work in 13 priority neighbourhoods in the inner suburbs of Toronto. ANC is a neighbourhood engagement initiative that aims to improve local neighbourhood conditions. The investment being made in social capital and infrastructure is significant, beginning with a diverse range of grassroots neighbourhood outreach engagement activities. The success of ANC relies on broad citizen participation and the development of meaningful and sustainable resident leadership. A community driven approach to identify core neighbourhood issues, problems, and solutions requires broad support from the local social service system, businesses, government, and other citizen groups and organizations. Local participation and the creation of resident associations are augmented by organizational, financial, and capacity-building support of UWGT. The goal of ANC is to transform neighbourhoods into healthy, vibrant, and inclusive communities that celebrate their own strengths, assets and diversity. ANC aims to build communities where residents view their neighbourhoods as a place to connect to one another, share in their accomplishments, and plan for their future growth. A particular emphasis of the ANC planning process is placed on the voices of newcomers and youth. Each participating ANC site is physically located and collaboratively directed via a community-based partner agency. Partner agencies are local organizations that have legitimacy and presence in the neighbourhood and who are perceived to be an appropriate organization to facilitate resident development. With support from UWGT, these agencies are responsible for project management, neighbourhood outreach, organizing, strengthening resident capacity, and connecting local plans to the broader community. Action for Neighbourhood Change advances two of the Strong Neighbourhood Strategy s overall objectives: RESIDENT LEVEL: To Increase the Leadership and Influence of Local Residents over neighbourhood conditions, and build collaborative action SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods ANC can be understood as a sequence of community development stages, each one building on the work of the last. However, in reality, many of these stages continue on an on-going basis, albeit, at a less intense level. The activity clusters of ANC can be grouped into two broad categories - those that establish the resident-led foundation for change, and those that initiate the change process: Building Strong Neighbourhoods Strategy: Evaluation Framework p. 14

18 INTRODUCING THE COMPONENTS Activities that Build the Foundation for Improving Neighbourhood Conditions Building Knowledge of the Neighbourhood. Effective neighbourhood planning requires a firm understanding of neighbourhood characteristics demographic make-up, strengths, assets, needs, gaps, etc. The gathering of planning information is already underway, including resident surveys and interviews; meetings with existing service providers, businesses, and community groups; service inventories, and analysis of secondary data sources (e.g. census data). This upfront information about community assets and gaps will be used by UWGT and partner agencies to plan further outreach strategies to residents and other community partners (e.g., recreation centres, community groups, businesses, etc.). Outreach and Communication. The next stage in ANC is to raise the profile and overall awareness of ANC in local neighbourhoods and to attract residents broadly to get involved in the process and to encourage positions of leadership from within the broader group. This involves extensive canvassing of neighbourhood stakeholders, and wide variety of community outreach activities, such as community meetings and conversations with local service providers, representatives of local institutions such as schools, faith centres, business owners, and existing community leaders. Neighbourhood Organizing & Planning. In this stage, the formation of a group of active resident participants becomes more focused and intensive, and leader. Core groups of residents are identified to engage in neighbourhood planning and support and guidance is provided to help form a representative and functional resident association. With the formation of a resident association, planning begins to develop a neighbourhood vision, plan, and set of priorities. Activities that Initiate the Change Process Building Connections with Decision Makers. In ongoing way, ANC will link the plans and priorities to a broader set of community, government, and corporate partners. This requires UWGT and ANC partner agencies working together to establish connections and collaborations with local businesses, external funders, all levels of government, existing service infrastructures, and planning bodies. It is through these connections, that local plans will be put into action and result in a range of potential improvements to local neighbourhood conditions, including improved access to existing infrastructure, investment and provision of new services and infrastructure, improved safety, etc. Gaining wide endorsement of ANC and BSNS is a central objective of this ongoing subcomponent of ANC and is key assumption underlying ANC s effectiveness. Process Milestones for ANC Within the first 3 months, each ANC project should have amassed a welldeveloped profile of the neighbourhood, including its socio-demographic characteristics, assets, networks, associations, services, businesses, etc. Building Strong Neighbourhoods Strategy: Evaluation Framework p. 15

19 INTRODUCING THE COMPONENTS By the end of the third month, each site should have an engagement plan that reflects the diversity of the neighbourhood. By the 6 th to 8 th month, the beginnings of a core group of residents should have emerged, with an understanding that they will work together to build a plan and priorities for how they want their neighbourhood to change. By months, a more formalized resident-led association should have emerged, with some form of structure, mandate, and regular meeting schedule. By months, more formalized plans and priorities from the emerging resident-led association should start to take shape Between the 6 th and 24 th months, collaborations and connections between the emerging resident group and decision makers may be formed to act on resident issues and priorities (In other words, finding solutions to resident issues that are identified early in the engagement and organizing work does not have to wait until the resident association and its plans and priorities become more formalized). ANC Capacity Building: Description Capacity building support to the ANC initiative (beyond the skills available within the partner agencies, their site teams or the UWGT ANC Development Coordinators) will be developed and coordinated by UWGT s Organizational Capacity Building Unit to enhance and supplement the skills of partner agencies and residents.. This support will have two broad goals: Increase the capacity of individual residents to develop a common vision and work together for change Increase the knowledge, skills and abilities of ANC partner agencies and their ANC site teams. Leadership training and specialized capacity building supports are critically important supports to the ANC initiative. ANC will be successful only when resident leaders have the confidence and skills to influence decision makers to make the kinds of investments and changes that they want for their neighbourhoods. And, the ANC initiative will only be successful if partner agencies are able to support the empowerment of local residents. Capacity Building for Leadership. The UWGT Capacity Building team provides support, consultation, training, and mentoring to leaders and the resident association in collaborative leadership strategies, negotiation and communication skills, governance, and technical needs. Capacity Building for Partner Agencies. Beyond support to leaders and the resident association, there are capacity building needs for partner agencies and their site teams. This may include support around governance, negotiation, communication, visioning, planning, and marketing, conflict resolutions, and working cross-culturally. Capacity Building advances two of the Strong Neighbourhood Strategy s overall objectives: Building Strong Neighbourhoods Strategy: Evaluation Framework p. 16

20 INTRODUCING THE COMPONENTS RESIDENT LEVEL: To Increase the Leadership and Influence of Local Residents over neighbourhood conditions, and build collaborative action SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods Process Milestones for Capacity Building As RLNA s begin to build momentum, each site will be assessed for readiness for RLD. In some sites RLD may occur parallel to the development of the RLNA, in others it may help with the initiation of the RLNA. As ANC begins to support the leadership development of residents, Capacity Building will begin to assess site needs and deliver training, coaching and technical assistance across the ANC project sites. The following milestones are relevant: By 6 months, the Capacity Building team has collected information from all ANC sites and the development coordinators to determine initiation strategy for RLD At 12 months, a progress report on the activities of the Capacity Building component will be submitted, including relevant reflections from staff observations and journaling. By 12 and 18 months, a first wave of training and technical assistance will have been delivered to agencies and resident leaders/associations, respectively. Participating sites will complete appropriate training evaluation surveys. By 18 to 24 months, focus groups and reflection sessions will be held with resident leaders/associations (see also ANC measurement plan for other purposes of these focus groups). At 12 and 24 months, key informant interviews conducted with management and staff of partner agencies regarding organizational understanding of ANC, related planning activities, and reflections on changes to policy and practice. Timing, however, is flexible, and could be semi-annual. It should be linked to timing of training and support. ANC Quick Start: Description The Quick Start Fund provides a pool of funds that allow for the quick implementation of resident-identified projects in Action for Neighbourhood Change (ANC) communities. These funds will be a catalyst for change and increase the groundswell of support for the ANC initiative, creating a critical foundation for all future work in these communities. Quick Start Funds will bring immediate benefits to neighbourhoods, and strengthen the legitimacy of resident-led leadership and actions through a series of quickly implemented projects addressing priority needs. Quick Start is intended to help ANC neighbourhoods transform loose groupings of people and ideas into a coalition capable of action. Whether Quick Start becomes more than a way to fund a few small and symbolic projects and truly acts as a catalyst for more profound action for change in a neighbourhood depends to a large degree on whether a strong and representative neighbourhood Building Strong Neighbourhoods Strategy: Evaluation Framework p. 17

21 INTRODUCING THE COMPONENTS association is beginning to develop. Quick Start cannot achieve all of its intended outcomes unless other components of ANC are successful as well. Quick Start advances two of the Strong Neighbourhood Strategy s overall objectives: RESIDENT LEVEL: To Increase the Leadership and Influence of Local Residents over neighbourhood conditions, and build collaborative action SERVICE AGENCY & NEIGHBOURHOOD INFRASTRUCTURE: To Strengthen the Social Infrastructure, in high-need neighbourhoods Three main clusters of activities comprise the Quick Start component of ANC. Community Outreach and Engagement. The first phase of the Quick Start process involves letting community members know about the Quick Start opportunity. This includes raising awareness at partner agency, site team, and neighbourhood levels and includes using the fund as an opportunity to raise awareness about residents priorities, issues and needs. Proposal Development. ANC site staff will work with residents interested in submitting Quick Start proposals to build their skills and ensure that their proposals match the neighbourhood vision and priorities and benefit diverse groups of residents. Although a sponsoring agency must support each application, residents must be involved in the planning and the implementation of the proposed project. The intended outcomes of this process include increased skills and increased understanding of neighbourhood issues among applicants. Proposal Selection and Implementation. Local residents will also be involved in establishing the process and criteria for review of proposals. A formal review committee may be established, or the ANC neighbourhood association may play this role. ANC site staff will provide training and support for reviewers. Once neighbourhood funding recommendations are forwarded to the United Way, final review and decision making will take place. ANC site staff will also play a role in supporting the implementation, monitoring and promotion of projects funded through Quick Start. The intended outcomes of this activity cluster include increased skills and increased understanding of neighbourhood issues among committee members and reviewers, increased clarity of neighbourhood vision, and buy-in to the process among a greater number of residents. Milestones for Quick Start Promotional activities designed to let potential applicants know about the Quick Start opportunity have taken place. Capacity building events for potential applicants and local reviewers have taken place. # of applications that have been received (after 9 months). #of applications have been funded and implemented (by the end of year 1). Building Strong Neighbourhoods Strategy: Evaluation Framework p. 18