Get In Chicago Capacity Assessment. Summary Report. Fred Wulczyn Emily Rhodes Ava Weiss LaShaun Brooks

Transcription

1 Get In Chicago Capacity Assessment Summary Report Fred Wulczyn Emily Rhodes Ava Weiss LaShaun Brooks 2016

2 Get In Chicago Capacity Assessment: Summary Report Fred Wulczyn Emily Rhodes Ava Weiss LaShaun Brooks Recommended Citation Wulczyn, F., Rhodes, E., Weiss, A., & Brooks, L. (2016). Get In Chicago capacity assessment: Summary report. Chicago, IL: Chapin Hall at the University of Chicago. ISSN: Chapin Hall at the University of Chicago Chapin Hall at the University of Chicago 1313 East 60th Street Chicago, IL (phone) (fax)

3 Acknowledgments We would like to thank the organizations funded by Get In Chicago for participating in the study. We also thank Get In Chicago for providing background information on the organizations and reviewing the research framework and tools. We would also like to thank Professor Greg Aarons for permission to use the Implementation Leadership Scale, Implementation Climate Scale, and Evidence-Based Practice Attitudes Scale. The John D. and Catherine T. MacArthur Foundation funded the project through the Crime Lab at the University of Chicago. We are grateful for their support.

4 Table of Contents Executive Summary... 1 The Context... 1 The Question... 1 The Analysis... 2 The Findings... 2 The Implications... 3 Introduction... 4 Research Questions and Framework... 5 Research Methods... 9 Findings Implementation Timeline Structure, Process, and Human Capital Respondent Interviews: Common Themes Discussion and Recommendations Structures Processes Human Capital Appendix A: Coding Guides Appendix B: Raw Scores References and Selected Bibliographies Bibliography for Research Framework Bibliography for Research Methods Bibliography for Structures Bibliography for Processes Bibliography for Human Capital... 32

5 List of Figures Figure 1. The Continuous Quality Improvement Cycle... 8 Figure 2. Structures to Support Implementation Figure 3. Processes that Support Implementation Figure 4. Human Capital Requirements of Implementation... 16

6 List of Tables Table 1. The Stages of Implementation Completion... 7 Table A-1. Coding Guide for Organizational Structures Table A-2. Coding Guide for Process Measures Table A-3. Coding Guide for Human Capital Measures Table B-1. Cross-Organization Means by Domain, Feature, and Quality Indicator... 25

7 Executive Summary The Context Get In Chicago is a public/private partnership that brings together diverse stakeholders for the purposes of combatting youth violence and building safer communities. In May 2014, Get In Chicago awarded the first round of grants to eleven community-based organizations in Chicago. Grant recipients in this cohort were funded to provide services to youth and families in four program areas: cognitive behavioral therapy (CBT), mentoring, parent leadership, and innovation. When the RFP was issued, applicants were asked to select a program area and describe their plan for implementing an evidence-based intervention (EBI) targeting at-risk youth. As part of its effort to strengthen the service infrastructure, Get In Chicago also enlisted Chapin Hall at the University of Chicago to examine the grant recipients capacity to implement evidence-based interventions. The Question The study framework is built on literature in implementation science, research evidence use (REU), and continuous quality improvement (CQI). A common tenet of the literature in these areas is that it is not sufficient to have an EBI in hand organizations must also have the capacity to implement that EBI with fidelity to the model in order to achieve the intended outcome. To that end, the study focuses on these core organizational capacities: What structures do the organizations have in place to support evidence-based interventions? What processes are in place to support the implementation and delivery of evidence-based interventions? Is the human capital needed to implement evidence-based intervention in place? The study assesses these core questions relative to the gap between what organizations are doing now and what they need to do to maximize impact with the resources provided. Chapin Hall at the University of Chicago 1

8 The Analysis Results for this summary report are based on surveys (n = 110) and interviews (n = 36) with staff members at various levels of each organization as well as observations of group programming at eight of the eleven organizations. Researchers also reviewed various planning and program documents from all organizations to support interview and survey responses. Results are based on a combination of selfreports and researcher observations. The Findings Structures Overall, the grantee organizations capacity (i.e., structures) to collect evidence and provide feedback is in place, but the structures in place to support intake assessment are not as strong as they might be. This is reflected in the fact that the core outcomes expected from the intervention were also not as well defined as might be expected. Implementation depends on a tight alignment between the target population and the services being provided, so a second look at these structures may be warranted. Processes Although structures for evidence collection are in place, the processes needed to promote use of the evidence collected are not as strong. Fidelity monitoring and continuous program monitoring show considerable variation across the organizations. This may be a function of the ambivalence we heard when we talked to staff about their use of evidence. Successful interventions depend on how well caseworkers know their clients. Day to day, the relevance of program data to caseworkers is not as clear as it could be; workers prefer to use their relationship with a young person to guide their work. The underlying issue may be a gap between the actual value of the evidence generated from program data and its perceived value particularly in the eyes of frontline staff. Regardless of why the gap exists, organizations have to integrate the various narratives used to define the work, creating considerable work for what is fundamentally an issue of translation. Human Capital Without human capital, structures and processes have little utility. We measured five human capital requirements. Staff have a good mix of skills and attitudes and there appears to be readiness for change and innovation. When we asked specifically about communication with the public, organizations gave different answers. Though some organizations are well connected, others struggle to make a connection Chapin Hall at the University of Chicago 2

9 with external support. We also asked about learning culture. We found that although leadership supports learning, work remains when it comes to staff perceptions of their organizations culture. The Implications Get In Chicago and its community partners believe that investments in evidence-based interventions provided by community-based organizations will help curb youth violence in Chicago neighborhoods. It s an important undertaking; the benefits of growing up in safer communities last throughout the life course. Investments in evidence-based interventions pose a particular set of challenges for the participating organizations. Evidence-based interventions depend on fidelity to a model; without fidelity it is much harder to say whether using the interventions pays off relative to business as usual. Fidelity management requires most organizations to think differently about how they use their capacity. If the success of Get In Chicago s investment in EBIs depends on upgrades to existing capacity, then an investment in capacity building may be necessary. Chapin Hall at the University of Chicago 3

10 Introduction Get In Chicago is a public/private partnership that brings civic leaders from government, business, faithbased organizations, educational institutions, and foundations together for the purposes of reducing rates of youth violence and disrupting the adverse outcomes associated with youth violence. Through its grantmaking efforts to community-based organizations, Get In Chicago seeks to support the expansion of programs that can bring about population-level impact for youth at greatest risk of violence and strengthen existing community assets that create safer neighborhoods. In May 2014, following a competitive application process, Get In Chicago awarded the first round of grants to eleven community-based organizations in Chicago. Grant recipients in this cohort were funded to provide services to youth and families in four program areas: Cognitive Behavioral Therapy (CBT), mentoring, parent leadership, and innovation. When the RFP was issued, applicants were asked to first select a program area and then describe their plan for implementing an evidence-based intervention targeting at-risk youth. As part of its effort to strengthen the service infrastructure, Get In Chicago also enlisted Chapin Hall at the University of Chicago to examine the grant recipients capacity to implement evidence-based interventions. To that end, the study focuses on these core organizational capacities: What structures do the organizations have in place to support evidence-based interventions? What processes are in place to support the implementation and delivery of evidence-based interventions? Is the human capital needed to implement evidence-based intervention in place? Data collection was organized using three interrelated frameworks. The Stages of Implementation Completion (SIC) organizes the steps organizations go through when mounting an evidence-based intervention (Saldana, 2014). In the case of Get In Chicago (GIC), the applicant organizations were asked Chapin Hall at the University of Chicago 4

11 to integrate an evidence-based intervention into their ongoing operations. Implementation is a process and the SIC provides a framework for following implementation through each of three stages: preimplementation, implementation, and sustainability. The study specifically considers the programs funded by GIC within the context of larger organizational and administrative systems. Data were collected on the GIC-funded programs from September to November of 2014 and the results reflect the organizations implementation progress as of that time interval. We also relied on the continuous quality improvement (CQI) framework (Wulczyn, Alpert, Orlebeke, & Haight, 2014). Adapted from industrial process management and quality control, the CQI framework is finding widespread applicability in health and social services. CQI is a deliberate, iterative problemsolving process that connects investments in the process of care, the quality of care, and the capacity to deliver care with fidelity to the outcomes an organization or program hopes to achieve on behalf of the people it serves. The CQI process relies heavily on feedback, so we were particularly interested in how the selected organizations approached organizational learning. Research evidence use (REU), the third framework we adapted, focuses on how service organizations acquire, process, and apply research evidence. Research evidence, which is defined broadly to include evidence gathered systematically by either qualitative or quantitative means, is a key input at each step of the CQI cycle. In our study, we were interested in each organization s internal capacity to generate or acquire evidence staff could use to support decision making, at either a clinical or programmatic level. It is important to note that our study of organizational capacity is not an evaluation of the selected organizations or the success of these particular interventions. Rather, the goal here is to better understand each organization s capacity to use evidence to continuously improve their programs so that stakeholders, including Get In Chicago, understand what is needed to build strong organizations capable of mounting sustainable interventions that reduce violence in Chicago neighborhoods. Research Questions and Framework Human service organizations are increasingly invested in evidence-based interventions that have passed a rigorous test of their efficacy. In terms of what we expect from human services, this is a positive development. That said, introducing evidence-based interventions one might say installing evidencebased interventions does pose challenges for the organizations asked to participate. The connection between an intervention and its effect on children and young people is inextricably tied to how well the intervention as delivered follows the treatment protocol. Fidelity management the capacity to implement an intervention with fidelity and measure the results requires most organizations to think differently Chapin Hall at the University of Chicago 5

12 about how they use their capacity. In this report, we document how the first cohort of Get In Chicago grant recipients was positioned relative to those challenges. To do this, we studied each organization with three capacity-related questions in mind. First, we set out to establish current capacity by understanding the structures, processes, and human capital needed to support the implementation of evidence-based interventions. Evidence gathered from observations of organizations deep in the implementation process suggest these are areas that require mastery on the part of the organization. The second question focused on the gap between what the organizations are doing and what might need to be done in order to maximize impact with the resources provided. Judging from the implementation literature, there is no specific playbook describing how an organization goes from A to Z on the path to a sustainable commitment to evidence-based practices. Nevertheless, facility with using evidence is a core component of fidelity management, so we examined each organization s capacity to generate, process, and apply evidence, among other processes. Implementation Science The framework we used to organize our data collection is grounded in implementation science and related best practices continuous quality improvement and research evidence use, with specific reference to youth violence prevention when relevant. Inasmuch as the organizations selected by Get In Chicago were in the midst of putting their funded programs in place when we met with them, we wanted to situate their efforts within an implementation framework. For that purpose, we selected the Stages of Implementation Completion (SIC) (Saldana, 2014). The SIC was developed to track implementation process of evidencebased practices. As shown in Table 1, the SIC framework is divided into three phases with a total of eight stages. The first phase preimplementation involves engagement, consideration of feasibility, and readiness planning. The second stage is implementation, characterized by staff hiring and training, creating an adherence monitoring system, beginning to provide services and consultation, and ongoing activities (service provision, fidelity monitoring and feedback). The third and final stage is sustainability. Sustainability essentially means the intervention is embedded in the routine operation of the organization. With reference to the Get In Chicago cohort, when we visited with the organizations, they were all in the implementation phase. Chapin Hall at the University of Chicago 6

13 Table 1. The Stages of Implementation Completion Phase & Stage Key Steps Preimplementation 1. Engagement Site informed of grant opportunity, interest indicated (i.e., LOI). 2. Consideration of Stakeholders discuss feasibility and identify program model feasibility to fund that will address a problem in their community. 3. Readiness planning Site reviews costs, staff recruitment, and participant referral criteria. Written implementation and communications plans are developed. Implementation 4. Staff hired and trained 5. Adherence monitoring in place Frontline staff and managers are hired/reallocated and trained. If appropriate, partners (e.g., schools) are trained. Metrics for tracking outcomes are established, and a formal system for implementation fidelity monitoring is developed and tested (e.g., database, fidelity checklist, etc.). 6. Services begin First clients are served, first team (managers and frontline staff) meeting to review implementation fidelity held. 7. Ongoing services, fidelity monitoring, and feedback Formal implementation review and fidelity assessment. Review evidence of participant progress and adjust intervention as necessary. Sustainability 8. Competency Program is meeting its long-term strategic goals. The program model has been formally evaluated and is achieving intended outcomes. Continuous Quality Improvement The second part of the framework used for the study borrows from the continuous quality improvement (CQI) literature. CQI is a cyclical process of problem-solving activities built around the deliberate use of evidence. Evidence is or should be used at every stage of program implementation: defining a target problem and selecting the appropriate intervention (plan), implementing the program (do), monitoring the program for fidelity (study), and adjusting the intervention as necessary based on what monitoring reveals (act). The CQI cycle is shown in Figure 1. Chapin Hall at the University of Chicago 7

14 Figure 1. The Continuous Quality Improvement Cycle During the study phase in particular, organizations actively manage fidelity using evidence generated from data collected for that purpose. Based on this measurement feedback, organizations then act to adjust the intervention. Research Evidence Use Because evidence use is so central to the CQI process, we next turned to the research evidence use literature to clarify our use of the term evidence. Generally, there are two ways to think about evidence. The traditional, narrower view emphasizes evidence-based intervention and everything that phrase implies. A second perspective broadens the definition of evidence to include evidence generated through data collection and analytic procedures that are objective, systematic, and open to scrutiny. Evidence can be created through an organization s own internal, systematic analysis of the data it has collected (e.g., using a database, surveys, or focus groups), or acquired from external sources (e.g., peer-reviewed research articles, program evaluations, or consultations with experts). Research evidence use is itself a process that involves acquiring evidence, processing evidence, and applying evidence. Acquisition happens either by reaching out to sources external to the organization or through self-generated reports using management information. Processing evidence is the act of meaning making. Not all evidence is good evidence during evidence processing it must be determined if the evidence is valid and reliable. Once acquired and processed, evidence needs to be relevant. The narrative Chapin Hall at the University of Chicago 8

15 that emerges from the meaning of the evidence is then applied to decisions facing the organization. Naturally, we were interested in each organization s capacity to support evidence use. Research Methods In order to deepen our understanding of each grantee organization s capacity, we considered the organizations from a variety of angles, using a five-step data collection process that encompassed a number of methodological approaches. The data collection steps were as follows: Screening interview: We conducted over-the-phone interviews with the primary organizational contact(s) (provided by GIC), during which we gathered preliminary information on the organizations and organized subsequent data collection steps. Document review: During the screening interview, we requested that each organization send us documentation of structures and processes within the organization and the funded programming. Documents submitted with the organization s request for proposal (RFP) were also included. We reviewed these documents for insight into key organizational features. Online surveys: All staff and administrators involved in GIC-funded programs at each organization were sent an online survey. Depending on their role at the organization, respondents completed surveys pertaining to organization structures and processes for evidence collection, how they apply evidence when working with youth, and to what extent their organizational culture supports learning, innovation, and evidence use. Observation: Where appropriate, we observed group sessions that were part of the GICfunded programming at each organization. The observations were followed by brief interviews with the session s instructor(s). In-person interview: Finally, we met with the individual(s) the organization identified as most informed about the GIC-funded programming and asked them a number of open-ended questions about their organization s structures and processes to support evidence use. Following collection, data were analyzed using both quantitative and qualitative methods. Quantitative data were synthesized across research tools and standardized with z-scores using SAS statistical analysis software. All quantitative results are averaged to a score of 5 with a standard deviation of 1 across all GIC-funded organizations. A score above 5 represents a capacity advantage over the GIC average, and a score below 5 represents a relative capacity gap. Qualitative information collected from interviews was coded using ATLAS.ti qualitative analysis software. Researchers then reviewed coded interview responses and compared responses to available documentation sourced from the organization s written processes and practices. As a final step, each organization was assessed relative to a guide meant to Chapin Hall at the University of Chicago 9

16 highlight practice expectations. The differences between the observed capacity and the idealized capacity were regarded as a gap. Results were provided to each organization; this summary report looks across organizations to form a general impression of the sector. Results for this summary report are based on surveys (n = 110) and interviews (n = 36) with staff members at various levels of each organization, and observations of group programming at eight of the eleven organizations. Researchers also reviewed various planning and program documents from all organizations to support interview and survey responses. Results are based on a combination of selfreport and researcher observations. In the analysis that follows, quantitative results for all GIC-funded organizations are compared to the average z-score of 5 and both quantitative and qualitative results are compared to the relevant quality standard. The analysis also describes trends among the grantee organizations. Chapin Hall at the University of Chicago 10

17 Findings We have divided the findings of our review into three parts. 1 The first part addresses the implementation timeline. Implementation is a process that each organization approaches with its own sense of pace. An organization s decision about pace should balance ongoing operations with the requirements an evidencebased intervention impose. We logged the pace of implementation against what we learned about the structures, processes, and human capital assets either in place or being brought on board. In the second part, we formed a composite view for each organization using a spider graph to show how each organization compared to the mean of the others. In this way, we gained an appreciation for strengths and weaknesses along multiple dimensions within the structure process human capital framework. The third and last set of findings draws on interviews with the people working for the eleven grantee organizations. If implementing an evidence-based interventions means changing what the work looks like on a day-today basis, the individuals being asked to change how they approach their work are an important source of insight. Using their voices, we discuss some of the on-the-ground challenges. Implementation Timeline In May 2014, Get In Chicago awarded nearly two million dollars in grants to eleven Chicago nonprofit organizations with promising programs designed to address the problem of youth violence from a variety of angles. Of the eleven programs selected, five were funded to implement programming using Cognitive Behavioral Therapy (CBT) as their approach; four organizations were funded to provide mentoring; one organization was funded under the Innovation program category; and one organization was funded under the Parent Leadership category. The eleven grantees used GIC funds to serve youth and families across Chicago, in Austin, Englewood, Humboldt Park, North Lawndale, South Shore, and West Englewood. 1 In addition to this high level summary, each of the eleven organizations was provided with a detailed report assessing their capacity to implement their Get In Chicago funded program. Chapin Hall at the University of Chicago 11

18 At the time of the assessment (between September and November 2014), all organizations were expected to have made progress with the first two phases of implementation and to show some progress toward sustainability (e.g., have systems in place for program evaluation and administrative processes for long term financial and strategic planning). However, this report only covers implementation during the early months (following the grant award); therefore, conclusions with reference to sustainability are beyond the current scope. Preimplementation Planning The preimplementation planning process began when Get In Chicago announced the funding opportunity on February 10, Organizations then submitted Letters of Interest (LOIs) by February 26, Those who submitted LOIs engaged in a variety of preimplementation planning steps that were documented in the Request for Proposal. Then, applications were submitted by March 17, 2014; these applications included a definition of specific problems to address, target populations, and outcomes of interest; the development of logic models and strategic plans; and the creation of MOUs and data linkage agreements. Implementation Organizations were awarded grants from Get In Chicago on May 7, 2014, after a two-stage review process of all applications was completed. They were then required to submit program work plans by June 20, The expected implementation timeline following the award of these grants included hiring and training staff between May and August Startup was slated for either July 1 or, for school-based programs, the start of the school year. Most of the grantee organizations hired their staff in July (within the expected timeline) and trained them thereafter. Several organizations encountered delays related to transitioning staff from previous positions, lengthy bureaucratic requirements for hiring at their organizations, or related obstacles. Furthermore, a few organizations experienced staff turnover, which extended the staff hiring/training process. Although many organizations began programming on schedule, several experienced small to moderate delays in the start of programming. These delays were due to a variety of factors, including delays in staff hiring and training as well as challenges in coordinating with schools and referral sources. By the end of data collection, ten of the eleven grantees had programming fully underway. Sustainability Moving forward from implementation, we observed progress towards sustainability on the part of many of the grantee organizations. This included organizations having systems in place for program evaluation, long-term strategic plans, and strong support for capacity building. Chapin Hall at the University of Chicago 12

19 Structure, Process, and Human Capital Coding guides for how we rated structure, process, and human capital are included in the Appendix. In essence, using the available literature, we sought to establish a reasonable standard against which we could judge the presence of certain capacities thought to be important when beginning evidence-based interventions. For the summaries that follow, we relied mostly on the surveys returned to us by staff, as well as closed-ended interview responses that could be scored (e.g., Do you currently have an analytic database? ). In addition, we talked at length with staff in each of the organizations using open-ended interview questions and we observed programs in operation to gain a deeper understanding of the organizational context. When appropriate, we combined insights across the various data types to form a composite view of capacity at the organizational level. Structures to Support Implementation Organizational structures are concrete features that support program operations. These include infrastructure to support evidence collection and processing (e.g., databases and surveys of youth) and measurement feedback (e.g., regular reporting and evidence analysis). Programmatic structures include intake assessment of youth (to determine risk and need), a clearly defined problem the intervention is expected to address and outcomes it is expected to achieve, and an intervention/service model that is based on evidence for improving the outcomes of the target population. Summary information describing each organization s position relative to the others in the sample is displayed in Figure 2. The spider graph has five anchor points, one for each of the elements of interest. At each anchor point, the spider graph shows the distribution of organization scores relative to the GIC average z-score of 5. Scores for evidence-based service models and structures for measurement feedback exhibited the least variation, with scores ranging from 4.3 to 5.5 and from 4.3 to 5.4, respectively. With regards to structures for evidence collection, there was slightly more variation, with scores ranging from 3.9 to 5.5. The most variation in organizational structures to support implementation appears in grantee organizations capacities for intake assessment and clearly defined problems and core outcomes, both of which have scores ranging from 2.2 to 5.5. Chapin Hall at the University of Chicago 13

20 Figure 2. Structures to Support Implementation Evidence Based Intervention/Service Model Structures for Evidence Collection Structures for Measurement Feedback Clearly Defined Problem and Core Outcomes Intake Assessment Processes that Support Implementation If organizational structures support program operations, processes translate structure into the work of the organization. The identification and recruitment of a target population is integral to the proper utilization of evidence-based interventions. The process of training staff and providing ongoing support and learning opportunities reinforce fidelity and fidelity management. Sustainability planning, which includes strategic planning efforts, planning for funding stability, and building stakeholder support, is another key process tied to successful, long-term sustainability. Summary information describing each organization s position relative to the others in the sample is displayed in Figure 3. Compared to the GIC average z-score of 5, organizations administrative planning processes were the most consistent, with scores ranging from 4.3 to 5.4. There is slightly more variation in organizations processes for monitoring implementation process and fidelity (with scores of 4.2 to 5.9), continuous program monitoring (with scores of 3.5 to 5.4), and staff training and ongoing learning (with scores of 3.4 to 5.4). The organizational processes for target population identification and recruitment varied the most between GIC-funded organizations, with scores ranging from 3.0 to 5.7. Chapin Hall at the University of Chicago 14

21 Figure 3. Processes that Support Implementation Monitor Implementation Process and Fidelity Administrative Planning Processes Target Population Identification and Recruitment Continuous Program Monitoring Staff Training and Ongoing Learning/Support Human Capital In concert with an organization s structures and processes, strong human capital is necessary to support evidence-based interventions. Components of an organization s human capital include effective leadership. This means having leaders that support learning, manage change, and are knowledgeable about evidence-based practice and staff members skills, abilities, and attitudes towards using evidence. External support through collaborations, support from other organizations, and a supportive political climate also factors into an organization s human capital resources. Human capital also comes in the form of readiness for change and innovation and an organizational culture and climate that supports learning and change. Compared to the GIC average z-score of 5, there is limited variation in grantee organizations scores for most features of organizational human capital (see Figure 4). Staff skills and attitudes and organizational readiness for change and innovation are the most clustered around the mean, with scores ranging from 4.5 to 5.4 and 4.3 to 5.2, respectively. There is slightly more variation in organizations capacities for leadership that supports learning, with scores between 4.4 and 5.6, and organizational culture that supports learning, with scores between 4.4 and 5.7. The most variation occurs in organizations scores for external support, which range from 3.8 to 5.4. Chapin Hall at the University of Chicago 15

22 Figure 4. Human Capital Requirements of Implementation Culture that Supports Learning Leadership that Supports Learning Staff Skills and Attitudes Readiness for Change and Innovation External Support Respondent Interviews: Common Themes The findings reported in the prior section were weighted in favor of what we learned through the surveys, document review, scored interview responses, and observations. The mix of perspectives gleaned from the in-person interviews, though not at odds with the other evidence we collected, does reflect the unique context of each organization and the nuance of work on the ground as it was happening. From those conversations, we came away with observations spanning the nature of relationships and fluency with the use of evidence. Building Relationships Strong, functional relationships are a key to how organizations manage relationships. In their conversations with us, respondents talked at some length about the demands relationships place on the organizations. Building (and maintaining) relationships takes time, intent, and clear communication. Respondents talked about the relationship-building process and a pattern of increasing returns with respect to participant outcomes. As one respondent put it, working with people to address the problems that lead to violence is not a quick fix because there is no cookie-cutter approach to building relationships with people we all have different paths, pasts. Building relationships with young people is one way the paths taken reveal themselves to caseworkers. Chapin Hall at the University of Chicago 16

23 Relationships with parents are also important, though they can be difficult to build because parents often have complicated schedules, worry about stigma, and can be reluctant to open their home to visitors. To better engage parents, respondents talked about embedding themselves in the community, reasoning that appealing, well-publicized activities normalize the place organizations take in the community. Working that way with communities is itself dependent on relationships. For grant recipients with schoolbased programs, relationships with the partner schools are of the utmost importance. Schools are structurally and logistically complex institutions, and it can be quite challenging for organizations to secure their commitment to providing adequate referrals, space for programming, students time, and other program needs. Of the organizations that worked with schools, the more successful ones used formal agreements that specify resource commitments. However, formal agreements do not take the place of the more informal relationships that unfold each day. Buy-in of school staff at all levels principals, administration, school counselors, teachers, etc. is essential; the disposition of even one person can shape how the collaboration turns out. For the organizations that rely on referrals from external sources, relationships are as critical. No matter how strong the program is, if no youth are referred, the program will not succeed. The organizations receiving adequate referrals tended to have both formal agreements and strong personal relationships with gatekeepers. A clear definition of a program s target population and criteria for participant eligibility that is clearly communicated facilitates this relationship, as does a clearly documented and disseminated referral process with mechanisms in place to ensure that the process works. Finally, voluntary collaborations between organizations appear to strengthen all organizations involved. The organizations engaged in collaborations report that, for the most part, these relationships are beneficial and bolster the work that they do with youth and families. For example, respondents talked about referring participants to other organizations in their network. Collaborations fill service gaps, making it easier to share knowledge, grow programs through professional development, and deepen community embeddedness. Nonetheless, collaborations can also be challenging to establish and maintain because, like all other relationships, they require time, attention, and maintenance. Again, organizations with long-running collaborations seemed to have an advantage over those striving to insert themselves into or create a new network. Evidence Use and Buy-In Public and private agencies are increasingly evidence dependent, a development that takes two forms: a growing awareness of and interest in evidence-based interventions and a commitment to using evidence about performance to guide strategic decision making. The transition to and use of continuous quality Chapin Hall at the University of Chicago 17

24 improvement processes, which involves the deliberate use of evidence, appears to be driving these changes. Though most organizations spoke about using data in some way, most (if not all) of the organizations had work to do on their data system, the mechanism used to capture and return data to the end user. Moreover, questions about buy-in to the use of data and evidence generated a great deal of feedback. Data systems support two basic processes: capturing data and reporting data back to an end user. Data capture requires hardware and software solutions; reporting data back involves summarizing the captured data and giving it meaning in a decision-making context. With respect to data capture, few of the organizations had a fully integrated data system. Instead, organizations relied on a mix of approaches consisting of one or more databases, electronic filing, and paper records. In some cases, the centralized approach to collecting data gave way to the fragmented requirements of funders. Based on conversations with respondents, it became clear that having a coherent data system and a coherent clinical program go hand-in-hand. Organizations with clear structures, process (i.e., well defined target populations, assessment criteria, etc.), and outcomes had a better understanding of what data are important and how that data should be collected and then reported. Though progress in the organizations is evident, the quantity of data and the ability to use evidence productively did worry some respondents. In particular, the time needed to engage with data was cited as a factor that limits practice and evidence integration. The use of data and evidence generated a wide range of opinions centered on the idea of buy-in. For some respondents, awareness of and commitment to using evidence was growing. As one respondent noted, We re just finally starting to see what it can do, because our staff are finally invested in putting the data in. For others, there was more ambivalence regarding the value of data and evidence relative to other ways of knowing what one needs to know. For example, respondents sometimes compared quantitative data to data gathered as an outgrowth of their relationship with clients. In their view, it is the relationships with participants in the context of their community that better informs their work. In the words of one respondent, I don t need that [data] to tell me I m doing my job. It comes from the kids. Responses pointed to an underlying current of concern about organizational cultures becoming too data driven, instead of driven by the hopes, needs, and experiences of the clients. In part, ambivalence toward the use of data is fueled by an implicit tradeoff. Although evidence use can inform practice, it also requires significant investment of time and other resources. For small organizations, the tradeoff is particularly hard. When data entry is duplicative, the benefits are harder to see. For some, the time is simply better spent working with young people, especially if the data are not accurate, a fairly common complaint. Chapin Hall at the University of Chicago 18

25 Discussion and Recommendations Get In Chicago and its community partners are betting that investments in evidence-based interventions provided by community-based organizations will help curb youth violence in Chicago neighborhoods. It s an important undertaking; the benefits of growing up in safer communities last throughout the life course. Investments in evidence-based interventions pose a particular set of challenges for the participating organizations. Evidence-based interventions depend on fidelity to a model; without fidelity it is much harder to say whether the using the interventions pay off relative to business as usual. Fidelity management requires most organizations to think differently about how they use their capacity. If the success of Get In Chicago depends on upgrades to existing capacity, then a second, different type of investment may be necessary. To better understand the local infrastructure, Get In Chicago asked Chapin Hall to study the capacity of the eleven organizations chosen in the first round of awards. Using an implementation/cqi/research evidence framework, we asked the organizations to tell us about the structures, processes, and human capital in place to support the implementation of evidence-based interventions. To collect data, we did inperson interviews, observed services being delivered, and asked staff to complete a brief survey. The findings are fairly straightforward and expected in the sense that the organizations are not clones of one another. The organizations do indeed differ with respect to how well they are positioned vis-à-vis putting up an evidence-based intervention. Structures The capacity (i.e., structure) to collect evidence and measure feedback is in place, but the structures in place to support intake assessment are weak a problem reflected in the fact that core outcomes and problems were not well defined. Implementation depends on a tight alignment between the target population and the services being provided, so a second look at these structures may be warranted. Chapin Hall at the University of Chicago 19

26 Processes Although structures for evidence collection are in place, the processes needed to promote use of the evidence collected are not as strong. Fidelity monitoring and continuous program monitoring show considerable variation across the organizations. This may be a function of the ambivalence we heard when we talked to staff about their use of evidence. Successful interventions depend on how well caseworkers know their clients. Day to day, the relevance of program data to caseworkers is not as clear as it could be; workers prefer to use their relationship with a young person to guide their work. The underlying issue may be a gap between the actual value of the evidence generated from program data and its perceived value particularly in the eyes of frontline staff. Regardless of why the gap exists, organizations have to integrate the various narratives used to define the work, creating considerable work for what is fundamentally an issue of translation. Human Capital Without human capital, structures and processes have little utility. We measured five human capital requirements. Staff have a good mix of skills and attitudes and there appears to be readiness for change and innovation. When we asked specifically about communication with the public, organizations gave mixed answers. Though some organizations are well connected, others struggled to make a connection with external support. We also asked about learning culture. We found that although leadership supports learning, that support may not translate into a culture that staff perceive as being a learning culture, even though staff rated their organizations as adaptable. Chapin Hall at the University of Chicago 20

27 Appendix A: Coding Guides Table A-1. Coding Guide for Organizational Structures Organizational Feature Structures for Evidence Collection Structures for Measurement Feedback Intake Assessment Clearly Defined Problem and Core Outcomes Expected from Intervention Intervention/ Service Model Based on Evidence for Improving Outcomes for Target Population Quality Standard The organization has their own analytic database that all managers and frontline staff have access to. There are formal structures for evidence collection and processing, including a structure to collect feedback from participants (e.g., a survey). The organization has formal infrastructure for regular reporting (e.g., dashboards, tools for monitoring participant progress) on program and staff performance. Formal measurement feedback is regularly provided to frontline staff and stakeholders. For every participant the program conducts a standardized intake assessment that measures the level of risk and identifies service need. Assessment results are then systematically used to develop individualized treatment plans. The organization has identified specific problem(s) in their community, and has clear organizational goals to address these problems (e.g. a theory of change, mission/vision statement). The program has a clearly documented target population with specific risk factors targeted for treatment. Furthermore, the program has identified expected outcomes with specific metrics to track progress toward achieving these outcomes (e.g., a logic model). The organization has selected an intervention based on evidence concerning its validity, reliability, and relevance to their target population. There is a formal curriculum/treatment protocol based on this evidence developed internally or adapted from an externally developed EBP. This curriculum/treatment protocol is broken out by session/lesson with intended dosage and program duration. Furthermore, the organization has physical structures (e.g., adequate space) to support programming. Chapin Hall at the University of Chicago 21

28 Table A-2. Coding Guide for Process Measures Organizational Feature Administrative Planning Processes Target Population Identification and Recruitment Staff Training and Ongoing Learning/Supp ort Continuous Program Monitoring Monitor Implementatio n Process and Fidelity Quality Standard There is a clear strategic plan for how the GIC-funded program is integrated into the organization s operations, with excellent administrative and staffing support. The organization has internal and external champions invested in the goals of the GIC program, and these champions are committed to sustaining the program with financial and community resources. The program has a clearly documented strategy for identifying the target population (e.g., a referral process). Furthermore, the program has successfully established relationships for referrals through formal agreements and ongoing relationship management. In addition, any necessary training is provided to field staff recruiting participants. All program staff have intensive, curriculum-based training prior to program implementation with ongoing learning opportunities to update skills. There are clear education/certification requirements for frontline staff appropriate for the program model. The program regularly reviews their evidence base to determine if the program model is achieving identified goals and meeting the needs of the community. Measurement feedback is systematically used by program administrators to inform and improve program operations. Additionally, frontline staff routinely use evidence and measurement feedback to inform how they are working with participants (e.g., treatment planning) and to track participant progress. There is a clear implementation timeline for the GIC-funded program with planning, program implementation, and sustainability activities. This implementation plan is disseminated to frontline staff, who are actively engaged in the process of monitoring the implementation process. There is a formal process for implementation fidelity monitoring (e.g., a fidelity checklist). Chapin Hall at the University of Chicago 22

29 Table A-3. Coding Guide for Human Capital Measures Organizational Feature Leadership that Supports Learning Staff Skills and Attitudes External Support Readiness for Change and Innovation Culture that Supports Learning Quality Standard The program s leadership consistently supports staff efforts to learn about and use evidence. Leadership is knowledgeable about and able to answer staff questions concerning evidence use and EBPs. Leadership actively encourages open communication to discuss and address challenges. Staff understand and value evidence use staff are flexible, adaptable, and open to new types of interventions. Staff are engaged in the process of looking for new and better ways to do things, and have the skills to identify and implement evidence-based programs. The organization communicates the program s purpose and demonstrates the program s effectiveness to the public. The program is well integrated into the community with active support from diverse community leaders. The program has demonstrated ability to adapt the service model and experiment with new service offerings. Staff and leadership are able to constructively respond to diversion from service protocol and implementation challenges. The organization has staff policies and practices to support learning, and values staff that are invested in learning and improvement. There is a culture across the organization of openly sharing and discussing challenges. Staff have adequate time to learn and reflect. Chapin Hall at the University of Chicago 23