Camp Fire USA Quality Improvement Field Test Report

Transcription

1 Camp Fire USA Quality Improvement Field Test Report Leah Wallace, Samantha Sugar & Amanda Sutter David P. Weikart Center for Youth Program Quality February 2011

2 Table of Contents Introduction and Background... 3 Findings and Recommendations... 5 Fidelity and Participation... 7 Program Quality- Observational Program Self Assessment... 9 Program Quality- Organizational Program Self Assessment Survey Responses Appendix A: Data Table Camp Fire PQA - Form A Appendix B: Data Table Camp Fire PQA - Form B Appendix C: Open-ended Survey Responses

3 Introduction and Background The Camp Fire USA Quality Improvement Field Test began in November 2009 when the Camp Fire USA National Office commissioned the David P. Weikart Center for Youth Program Quality (Weikart Center) to help build and implement a quality improvement system for Camp Fire Councils across the United States using the Youth Program Quality Intervention as a model. The Camp Fire USA National Office is committed to assuring all Camp Fire USA programs meet system-wide standards and as a result have sponsored the research and development of the Camp Fire USA Program Standards and the Camp Fire USA Program Quality Assessment. Use of these standards and tools annually will help to systematize assessment and reporting across the diverse array of Camp Fire programs, while also supporting improvement processes that can benefit all young people in these programs. The Youth Program Quality Intervention (YPQI) is the Weikart Center s experimentally validated assessment-driven continuous improvement process. The YPQI is designed to improve the quality of afterschool services by: (a) building managers continuous quality improvement skills; (b) increasing the quality of instructional practices delivered in afterschool programs; and ultimately, (c) increasing youths engagement with program content and their skill-building opportunities. The Youth Program Quality Intervention (YPQI) The YPQI model defines instructional quality as a set of professional practices that, in combination, increase participating youths access to positive developmental experiences. The components of instructional quality emerge directly from developmental science (Eccles & Gootman, 2002; Gambone, Klem, & Connel, 2002) and the ongoing research around the Youth Program Quality Assessment (Youth PQA; High/Scope Educational Research Foundation, 2005; Smith & Hohmann, 2005; Blazevski, Smith, Devaney & Sugar, 2008), a standardized observational measure of instructional practice. The Youth PQA, from which the Camp Fire PQA is derived, is composed of four domains that comprise 18 scales (summarized in Figure 1.2) and 60 observable items. Higher scores on these observational items, particularly in the domains of interaction and engagement, are associated with higher levels of youth selfreports of engagement, while very low levels of quality are associated with youth disinterest. In addition, programs with high quality instruction provide youth with opportunities to practice emerging social and emotional skills (e.g., efficacy, communication, empathy, problem solving) that support success in adolescence and early adulthood. The Youth Program Quality Intervention follows the Assess-Plan-Improve sequence depicted in Figure 1.1 to help program staff improve the quality of instruction that they provide for youth. While the Youth Program Quality Intervention is designed to produce changes in both policies and organizational settings, the ultimate goal is to improve quality in the instructional setting the places where youth program experiences occur. This approach to quality improvement and workforce development is currently being implemented in several thousand agency, school, and community-based settings in over 20 states. The Youth Program Quality Intervention (YPQI) ASSESS Conduct Program Self- Assessment and External Assessment with the Youth PQA. PLAN Create an improvement plan based on data. IMPROVE Carry out improvement plan to improve pointof-service quality. 3

4 The Camp Fire PQA The Weikart Center partnered with Camp Fire USA to create a custom version of the standardized assessment tool called the Camp Fire Program Quality Assessment (PQA), which is aligned with the Camp Fire USA National Program Standards. The Camp Fire PQA developed for the field test was composed of two forms: (1) Form A, which has 18 scales and 61 items and (2) Camp Fire PQA Form B, which are composed of 19 scales and 90 items. Form A consists of a set of observable standards for best practices for staff instruction in youth development programs. Form B assesses the quality of organizational supports for the youth programs assessed in Form A. Figure 1.2 The Camp Fire PQA Pyramid of Program Quality Access 4

5 Findings and Recommendations The findings below are described in detail throughout the remainder of the report. Findings Summary Participation was high. Forty Camp Fire USA Councils out of 62 councils that ordered box sets completed program self assessments using the Camp Fire PQA Form A and/or Form B. The intervention was light. Councils were not required to submit improvement plans or participate in Youth Work Methods trainings, elements that are often a part of more intensive interventions and support the improvement process. Utilization of available supports was low. Telecommunication provided access to trainings and supports when live applications were not feasible. Some Council staff attended live trainings, but the majority of support was delivered via online trainings or webinars. Additional ongoing supports such as the help desk and phone/ support were not fully utilized by Councils. Most council staff found the process helpful. The majority of surveyed staff found the project worth their time and effort. Many said that it led to an improvement in staff skills and program quality. Some Councils struggled with staffing and time. Some Councils reported that they felt pressed for time and struggled with staffing limitations when completing the observations, assessments, and improvement planning. 5

6 Recommendations For investments to improve the front-line workforce and volunteers: Quality assessment data is most importantly considered at the level of the individual staff member or program for which it was collected. However, based on the information obtained during the CFPQA program self assessments across all councils, three of the lowest scoring areas could benefit from increased professional development opportunities: Culture: Camp Fire USA values the cultures of their participating youth. Providing intentional opportunities for youth to discuss and share their family culture can affirm youth, expand world views and encourage respect Leadership and communication skill building: All youth have the potential to be leaders and to use their words and ideas to shape their reality. Providing youth with opportunities to lead, to mentor other youth, and to make presentations can make the youth program a context in which leadership and communication skills can emerge Planning and reflection: The skills of making plans for the future and learning from the past can help youth succeed in school and in life. These skills are tied into what brain scientists call executive functions, and play an important role in directing attention to tasks and decision making that connects to understanding consequences. For investments in management skills: A Camp Fire USA director s ability to enact continuous quality improvement is an important factor in the successful delivery of high quality programming. Establishing the best timeline for the YPQI within the annual CFUSA cycle will require input from Councils. Offering multiple or flexible entry points to the cycle may assist Councils in meeting requirements. Together, Weikart Center and CFUSA could offer advice for customization to meet the needs of individual councils. An emphasis on low-stakes accountability and supports in the form of professional development and coaching will help encourage strong buy-in from Councils. Through CFUSA building a cadre of local or regional trainers that can hold workshops and trainings, Council staff will have increased support for their improvement process. For expansion of the Quality Improvement System: Increase capacity to provide supports and coaching. CFUSA could build human capacity at the national and regional levels in the form of a cadre of endorsed CFPQA external assessors and Youth Work Methods trainers, who could serve as coaches and mentors. These endorsed assessors, trainers and consultants could be used to formally support Councils through the CFUSA s full Assess-Plan-Improve sequence and processes. Identify councils that are passionate about the work and use them as mentors and ambassadors. CFUSA could identify a small cohort of early adopter Councils and invite them to pilot a deeper intervention more closely aligned with the full Youth Program Quality Intervention (includes the addition of live trainings and external assessments.) Support regional Continuous Quality Improvement Work That Involves Camp Fire Councils. CFUSA could work with the Weikart Center to identify opportunities for Councils to connect with local or regional initiatives. CFUSA could encourage Councils to take full advantage of available training and technical assistance. Add a Youth Outcome Metric. CFUSA could integrate Search Institute s Development Assets Profile (DAP) into the CFUSA quality improvement process. This could leverage the analysis and reporting infrastructure that the Weikart Center is developing via its Quality Assessment and Asset Building project through the Ready by 21 partnership. Encourage use of Technical Assistance time. Technical Assistance time was underutilized by Councils in the field test. Weikart Center staff are available to support the quality improvement process. 6

7 Fidelity and Participation In a recent randomized field trial, funded by the William T. Grant Foundation, the Youth Program Quality Intervention model produced positive and sustained effects on both managers continuous improvement practices and the quality of instruction delivered by individual staff. Notably, these effects were strongest in sites that implemented all elements of the model (Smith et al., in preparation). Table 1.1 describes elements of the Camp Fire quality improvement process and compares its core elements to the elements of the Youth Program Quality Intervention. Fidelity to the Youth Program Quality Intervention is somewhat limited, with the Camp Fire process concentrating on program self assessment. Table Alignment between Youth Program Quality Intervention & Camp Fire USA Quality Improvement Field Test Element YPQI CFUSA Notes on Training and Action External assessment at baseline (Youth PQA) Program self assessment at Baseline (Youth PQA) Improvement Planning Youth Work Methods trainings (High/Scope Active Participatory Approach aligned to Youth PQA) TA Coaching for site managers (focused on continuous improvement practices, managers receive support in the YPQI process) Quality Coaching for staff (focused on instruction, managers receive coaching workshop and support frontline staff through strengthsbased feedback using the Youth PQA) Councils completed assessment of 123 programs in 40 councils during the spring and summer of Planning with Data webinars in June and September 2010 used program self assessment data. Program improvement plans were not collected. Up to 2 hours of phone and technical assistance coaching from Weikart Center staff was available for each Council. 7

8 Participation in the Quality Improvement Process Table 1.2 shows the level of participation in the various elements of the Camp Fire USA Quality Improvement Field Test. Sixty-two councils volunteered to be a part of the field test and received the Camp Fire PQA Box Set, a collection of materials and access to online training and webinars. A total of 40 councils, or 61% of the councils that volunteered to participate, submitted completed program self assessments by September Table 1.2 Participation levels in Camp Fire Quality Improvement Field Test PQA Element Live PQA Training at the GROW Conference Total # of Councils Participating Online PQA Basics Training 14 Details people in attendance 51 people participating in the short Online PQA Intro Course Webinar Attendance n/a 171 total participants at 7 webinars Phone and technical Assistance from the Weikart Center 37 Total TA Time: 4 hrs 50 minutes (~8 min/council) Completion of CF PQA Form A Council Programs Assessed Completion of CF PQA Form B Council Programs Assessed Average # of Staff involved in program self assessment Average # of staff hours required for program self assessment 4.64 staff Self-reported from survey staff hours Self-reported from survey 8

9 Camp Fire PQA Score Program Quality- Observational Program Self Assessment Overall Quality- Camp Fire PQA Form A Program self assessment is the cornerstone of the YPQA quality improvement process because it gives programs the opportunity to evaluate their practices and consider areas of improvement. The scores are not as important as the process itself. Most crucial is for programs to consider best practices and make changes; the scores simply provide information to better understand their own practice. Figure 2.1 shows the overall quality scores for the 4 domains in the Camp Fire PQA Form A. Consistent with youth program assessments conducted using the original Youth PQA, Camp Fire Councils scored highest in the Safe Environment domain and lowest in the Interaction and Engagement domains. This indicates that Councils have a strong foundation in creating safe and supportive environments for youth, while having less consistency in creating opportunities for youth to develop a sense of belonging and practice higher order cognitive skills including planning, choice and reflection. Overall, scores for each domain are quite high, as is typical for self assessments during a pilot year. Scores are likely to be lower in year 2 of the intervention due to increased familiarity and comfort with the tool, which commonly causes assessors to have a more critical eye when doing program self assessment. See Appendix A for the full details of the Form A data. Figure 2.1. Overall Observational Quality Scores (Camp Fire PQA Form A) I. Safe Environment II. Supportive Environment III. Interaction Domain CampFire Councils Form A (N=40) IV. Engagement 9

10 Low-Scoring Items- Camp Fire PQA Form A During the program self assessment process, 10 items scored a 1 on the Camp Fire PQA Form A in 20% or more of the observed offerings. Scoring a 1 on the Youth PQA means that particular practice was not observed at all during the offering, indicating that the developmental experience or staff practice was not present or only available to a small percentage of participants. For example, for item II-F Ambient Item: Staff are sensitive to youth s culture this staff practice received a score of a 1 in 36% of the Youth PQA forms. Therefore, self assessment showed that this practice was not present for 36% of young people observed. Table 2.1 Low-Scoring Items in Observational Assessment % of Scoring a "1" Item N=123 IIF Ambient Item: Staff are sensitive to youths' culture. 36% IIM3 Youth identify with program offering 35% IIF1 Opportunities to share family culture 34% IVP1 Plans for projects and activities 32% IVP2 Planning strategies 28% IVP Ambient Item: Staff guide youth in deliberate planning or goalsetting. 27% IIF2 Staff respect youth's culture 26% IIG Staff are prepared for the activity. 25% IVR1 Youth reflect on what they are doing 23% IIG2 Materials and supplies ready 22% 10

11 High Scoring Items- Prevalence of Quality Instructional Practices Table 2.2 presents high-scoring items from the Form A program self assessment. These are Camp Fire PQA items where at least 80% of offerings scored a 5, indicating that the practice is present at a high level. 1 For example, 92% of self assessments reported that staff smile, use friendly gestures and make eye contact the majority of the time. The table below represents programming elements in which Camp Fire councils can be said to consistently excel. All of the high scoring items are concentrated in the Supportive Environment domain. This indicates that youth program staff is able to consistently provide emotional and material support to youth and lay the foundation for cognitive and social development. Table 2.2 High Scoring Items in Observational Assessment- Camp Fire PQA Form A % of offerings scoring a "5" Item N=123 IIE4 Staff smile, use friendly gestures, make eye contact 92% IIE5 Ambient Item: Staff provides welcoming experience for youth. 92% IIE3 Staff tone of voice and language 90% IIG3 Enough materials and supplies for all youth 83% IIE1 Emotional Climate is Positive 82% IIK1 Staff actively involved with youth 82% III1 Youth engage with materials or ideas 81% 1 Note that this does not include items in the Safe Environment domain as they do not directly address staff instructional practices. 11

12 Camp Fire PQA Form B Score Program Quality- Organizational Program Self Assessment Overall Quality- Camp Fire PQA Form B Figure 2.2 summarizes overall quality scores measured by the Camp Fire PQA Form B, which is structured as an organizational interview and designed to assess the quality of organizational policies and practices in place to support youth programs. Councils scored lowest in the Youth Centered Policies and Practices domain, which measures the involvement of youth and families in creating program structures. They scored the highest in the Access domain, which measures the program s accessibility to all youth. See Appendix B for the full details of the Form A data. Figure 2.2: Overall Organizational Quality Scores- Camp Fire PQA Form B V. Youth Centered Policies & Practices VI. High Expectations for Youth & Staff CampFire Councils (N=30) VII. Access 12

13 Low-Scoring Organizational Items- Camp Fire PQA Form B During the program self assessment process, 10 areas of program structure scored a 1 on the Camp Fire PQA Form B in 30% or more of the programs assessed. Scoring a 1 on the Youth PQA means that particular program structure is not in place. For example, for the item VD4 Youth and families are involved in staff training and evaluation, self assessments reported that 89% of programs do not have a policy or regular practice to include youth/families in training and evaluation. It is noteworthy that the three items with the highest percentage of programs scoring a 1 are related to youth and family involvement in organizational decision making. Table 2.3 Low Scoring Item- Camp Fire PQA Form B % of Council Programs Scoring a "1" Item N=51 VD4 Youth and families are involved in staff training and evaluation 89% VI03 Key stakeholders are involved in decision making 71% VC1 Youth and staff share decisions about physical environment 54% VD7 Youth and families are involved in governing bodies. 47% VIN6 Short-term staff participate in self-evaluation 39% VIJ4 Stakeholders and/or the general public give feedback 36% VIF1 New staff have preservice orientation 32% VIG1 Young people identify with organization 32% VIJ3 Feedback from past participants is collected 31% VIJ6 Programs are offered based on data 30% 13

14 High Scoring Organizational Items- Prevalence of Quality Program Structures Table 2.3 presents organizational items in the Camp Fire PQA in which at least 80% of assessed Council programs scored a 5, meaning that the program organizational practice is in place at a high level. For example, for the item Staff identifies and reduces risk factors, 100% of self assessments found this practice to be a regular occurrence. The highest scoring items are concentrated in the following two scales: (1) Program Polices Enhance Health and Safety and (2) Staff Qualification Support a Positive Youth Development Focus. In addition, all assessed Councils programs reported that they consistently identify and reduce risk factors including those Councils that provide transportation for youth reporting that they have transportation safety polices in place. Table 2.3 High Scoring Organizational Items- Camp Fire PQA Form B % of Programs Scoring a "5" Item N=56 VE4 Staff identifies and reduces risk factors 100% VE5 Transportation safety 100% VE2 Adequate staff coverage in case of emergencies 98% VE3 Crisis and risk management plan in place 98% VE1 Consistent with Accreditation Requirements 93% VA3 Program Director: youth development job experience 89% VA1 Administrator: youth development experience 86% 14

15 Percent in Agreement N=40 Survey Responses Participant Satisfaction and Staff-Reported Impacts In December 2010, 79 staff from 62 councils were invited to participate in a survey about their participation in the Camp Fire USA Quality Improvement Field Test. Of the Council staff invited, 47 staff (59%) completed the survey. Complete responses to open-ended questions are listed in Appendix C, starting on page 18. Response to the field test was overwhelmingly positive, with large majorities responding that the process was worth their time and effort, it helped the council staff understand quality, it led to improvements in staff training, and it improved the overall quality of youth programming. Fewer Councils reported that the process was helpful in a fundraising or marketing capacity. Figure 3.1 Percent of Councils Answering Agree or Strongly Agree to Questions about the Quality Improvement Field Test s Value It has helped our staff understand quality It has helped up improve the quality of youth programs It has been useful in discussions with funders It has helped us describe our programs in marketing materials It has led to changes or improvements in staff training It has been worth our time and effort When asked to give open-ended response to questions about what changes they have noticed in both staff and youth in their programs, the response was again overwhelmingly positive, with many councils reporting strong youth engagement. One respondent noted, Students have become more invested in the program, because of the leadership opportunities presented to them. Another stated that there is more trust in youth and better relationships with families, They trust the staff more. The families are more engaged and responsive to the youth workers when phone calls are made home (from our after school program). Respondents also reported positive changes in staff, specifically, that staff were more aware of standards of quality and that the Camp Fire PQA had provided a common language for discussions of quality. One said, I have seen the staff change their own definition of quality programming and focus on those areas where we have found weaknesses. The language that the staff uses to define program has shifted to a more PQA focused language and the culture of improvement has developed. 15

16 In addition to the fact that Councils overwhelmingly felt that the CFPQA had a positive effect on their programming, 30% of Councils were able to market their participation in the process to encourage funders or potential program members. Considering the challenges of marketing quality improvement work, this is a significant number of councils. Feedback and Suggestions for Future Quality Improvement Work When asked what additional supports they would benefit from or what changes they would like to have made to the quality improvement process, respondents repeatedly emphasized the pressures of staffing and time. One respondent said, I think that forms A and B were very well thought out and helpful from a youth development approach and showed our strengths and weaknesses and where improvement was needed I think that the amount of time required was unrealistic for me and the staff. For us it was also a little unrealistic to think that staff and volunteers would grasp the concept in a short period of time and get reliable results. There's definitely a learning curve. I do think that continuous program improvement is a must if we're committed to our mission... When reporting the number of staff and staff hours involved in the project, there was wide variation. Table 3.1 provides both the number of staff and number of staff hours for the program self assessment and improvement process. It is possible that some Councils may have been less efficient in their use of time and resources and this is not uncommon for the first round of implementation. With the learning experience of the field test and with additional coaching, implementation time can almost certainly be reduced for many Councils, or better, implementation can be intensified within the same overall time commitment. Table 3.1 Range in Staff and Total Staff Hours Required Element Low # High # # of Staff Involved in Self Assessment 2 15 Total # of Staff Hours Required to Complete Self 2 15 Assessment # of Staff Involved in Improvement Planning 2 29 Total # of Staff Hours Required for Improvement Planning 2 60 Feedback on Supports, Training and Coaching Flexible phone and technical assistance time was underutilized by Councils, with only 5 hours used out of 130 available hours. The majority of feedback indicated that Councils were highly satisfied with the technical assistance. One Council reported, We felt very supported in this. The Weikart Center was always very responsive when we had questions. However, one Council reported a negative experience with the technical assistance, describing missed phone calls and saying, I wish that my administrative support/ta contact would have been more responsive. This negative experience and the limited use of the technical assistance time could indicate that a more a formal coaching and support process, driven more by the needs of the Councils would be more helpful. As sites continue in this process and it becomes institutionalized through annual assessments, each site will increasingly build capacity to complete their self assessments. Not only will staff be more able to deal with the technical elements of filling out and reporting on the CFPQA, but they will build instructional skills that will really sustain the continuous improvement process. Many Councils asked for more tangible support in the form of live training, coaching and support with the creation of the improvement plans. Several survey respondents echoed this request, I would love to see more regional trainings, but I do not see that as a barrier. 16

17 There is a lot of opportunity in the fact that Councils have a generally positive perception of external assessment. Most Councils, or 49% of those that participated are in support of having an external assessor visit or are at least interested in learning more information about the process. This suggests that Camp Fire USA should consider engaging a small number of committed Councils in the external assessment process accompanied by strong coaching supports. Table 3.2 Support for External Assessment Is your council interested in considering an external assessor for the coming year? % Response (N=40) No 41% We would want to know more about this before committing Yes 49% The recommendations in this report reflect the desire for greater supports and Council s concerns regarding time and staffing. 17

18 Appendix A: Data Table Camp Fire PQA - Form A Data Table for Camp Fire USA PQA All Camp Fire N=134 Cares n=34 Camp n=50 Contributes n=13 Connects n=21 I Safe Environment A. Health and Safety Practices Health and safety Sanitation Ventilation and lighting Temperature B. Appropriate emergency & safety procedures Emergency procedures Fire extinguisher First aid kit Appropriate emergency & safety procedures Youth are supervised C. Program Space and Furniture accommodate the activities Sufficient space Space can be modified Suitable space Furniture D. Healthy food and drinks are provided Drinking water Adequate food and drinks Healthy food and drinks Food preparation is safe and sanitary II Supportive Environment E. Staff provide a welcoming atmosphere Emotional Climate Staff greet youth and family Staff tone of voice and language Staff smile, use friendly gestures, make eye contact Overall: Staff provides welcoming experience for youth F. Staff show sensitivity to youths' family and culture Opportunities to share family culture Staff respects youth's culture Staff and youth show respect and inclusion There are no biased comments Overall: Staff are sensitive to youths' culture G. Staff are prepared for activities Start on time Materials and supplies ready

19 Data Table for Camp Fire USA PQA All Camp Fire N=134 Cares n=34 Camp n=50 Contributes n=13 Connects n=21 3. Enough materials and supplies for all youth Overall: Staff are prepared for the activity H. Program activities are appropriately challenging Appropriate amount of time Activities are appropriately challenging Balance high- and low-energy activities Overall: Staff appear to have provided appropriate challenge I. Activities support active engagement Youth engage with materials or ideas Tangible products or performances Youth talk about what they are doing Balance concrete and abstract Overall: Staff provide active learning experiences for youth J. Staff support youth in building new skills Youth encouraged to try new skills Mistakes allowed Overall: Staff support youth in building new skills K. Staff support youth with encouragement Staff actively involved with youth Staff use specific, non-evaluative language Open-ended questions Overall: All staff are engaged with youth all the time L. Staff encourage youth to resolve conflicts Approach calmly Staff seek input from youth Relationship between actions and consequences Staff follow up with parent/guardian Overall: Staff handle youths' feelings and conflicts supportively III. Interaction M. Youth have opportunities to develop a sense of belonging Get to know each other Inclusive relationships Youth identify with program offering Publicly acknowledge achievements Overall: Staff make an effort to make all youth feel a part N. Youth have opportunities to collaborate Work towards shared goals Work cooperatively Interdependent tasks Overall: Staff provide collaborative experiences

20 Data Table for Camp Fire USA PQA All Camp Fire N=134 Cares n=34 Camp n=50 Contributes n=13 Connects n=21 O. Youth have opportunities to act as group leaders Group conversation Opportunities to lead a group Overall: Staff provide youth with opportunities to lead IV. Engagement P. Youth have opportunities to set goals and make plans Plans for projects and activities Planning strategies Overall: Staff guide youth in deliberate planning Q. Youth have opportunities to make choices Share age-appropriate control Open-ended choices Take activity in a new direction Overall: Youth have a choice in how they spend their time R. Youth have opportunities to reflect Youth reflect on what they are doing Staff support or guide reflection Youth give feedback on the activities Overall: Youth have opportunities to look back TOTAL

21 Appendix B: Data Table Camp Fire PQA - Form B Data Table for Camp Fire USA PQA Form B All Camp Fire Councils N=30 V. Youth Centered Policies and Practices 3.84 A. Staff qualifications support a positive youth development focus Administrator: youth development experience Program Director: youth development training/ education Program Director: youth development job experience Staff: relevant education/training Staff: job experience Staff: reflect demographics of program participants 4.40 B. Program offerings tap youth interests and build multiple skills Influence format or content of program offerings Program offerings: programmatic focus Organization: programmatic focus 4.63 C. Youth have an influence on the setting and activities in the organization Shared decisions about physical environment Shared decisions about program offerings Youth lead sessions 3.24 D. Youth and family members have an influence on the structure and policy of the organization Staff respond to family questions Staff create opportunities for families to participate Youth and family members participate in program quality reviews Youth and families involved in staff training and evaluation Youth and families involved in recruitment Youth and families involved in community outreach Youth and families involved in governing bodies E. Program policies and procedures exist to enhance the health and safety of all participants Consistent with Accreditation Requirements Adequate staff coverage in case of emergencies Crisis and risk management plan in place Staff identifies and reduces risk factors Transportation safety Smoking and drug/alcohol use not allowed Background check References and experience for all youth workers All staff are trained in reporting requirements for child abuse Policies are in place to guide staff interactions with youth Emergency information for each participant is on file There are current, signed agreements with providers 4.28 VI. High Expectations for Youth and Staff 3.99 F. Organization promotes staff development

22 Data Table for Camp Fire USA PQA Form B All Camp Fire Councils N=30 1. New staff pre-service orientation Professional development within organization Staff skills and training Adult education principles used when training staff Support for staff to attend courses, conferences, workshops Professional development outside the organization Staff meet at start of program cycle Staff meet during program cycle 4.36 G. Organization promotes supportive social norms Young people identify with organization Expectations for personal and social interaction Encouraging youth to share personal concerns Staff pursue knowledge on issues and cultures of the youth 3.85 H. Organization promotes high expectations for young people High expectations for young people Acknowledge achievements 4.20 I. Organization is committed to ongoing program improvement Assesses youth outcomes Staff evaluations Assesses program quality Program improvement based on assessment 4.17 J. This program is an integral part of the council's program service's plan Feedback from young people in program Feedback from parents/guardians Feedback from past participants Feedback from stakeholders and/or the general public Demographic information Programs offered based on data Primary audience for each program based on data 3.31 K. There are intentional strategies that support an overall pattern of growth in services to youth Intentional strategy that supports growth in services Intentional strategy to increase participants 2.74 L. For each program, there is a program business plan which includes Camp Fire USA mission statement, and core values Program activities relate to desired outcomes Written outcomes for youth Plan supports the needs of the target population M. There is a supervision plan in force that includes a ratio of youth to adults to maximize safety Ratio of adults to youth Staff circulate Minimum number of adult staff 4.00 N. Program administrators assess job performance and satisfaction among staff Program staff are supported in their work

23 Data Table for Camp Fire USA PQA Form B All Camp Fire Councils N=30 2. Team work Staff demonstrate a sense of purpose Staff receive continuous feed back Year-round staff participate in self-evaluation Short-term staff participate in self-evaluation Supervisors receive training 3.59 O. The program administration provides for sound leadership and management Strategies to support sustainability Welcoming atmosphere for staff Key stakeholders are involved in decision making Compliance with standards and licensing P. Each program is evaluated for consistency with the goals and business plan Goals for sound financial management Population and participation projections Results of youth development outcomes 3.86 VII. Access 4.23 Q. Staff availability and longevity with the organization supports youth-staff relationships Staff with the program the entire program cycle Staff have returned from previous program cycle 4.69 R Barriers to participation are addressed Availability of eligible youth Distance, transportation, neighborhood safety barriers Cost barriers 4.38 S. Organization communicates with families, other organizations, and schools Mechanisms for communication Communication with other organizations and schools 4.56 T. Parents are informed of the program's purpose, schedules Structured opportunities for families to be involved Formal communication with families Family members involved in decision-making Schedules are completed and announced in advance 4.60 TOTAL

24 Appendix C: Open-ended Survey Responses 34. As a result of participating in this program quality improvement process, what changes, if any, have you seen for your staff? (i.e. in turnover, morale, quality of instruction, etc.) (N=28) Consistency of measures and jargon across varied programs and insight via assessments and observation. This assessment has not been useful to us because it is so black and white. Most individual sessions do not fit the format. For example- We observed a Teens in Action meeting. It was a planning meeting only with one girl in charge. Therefore, leadership was not distributed, they didn't have any hands-on activities, there was no room for expression of beliefs, etc, etc, etc. That doesn't mean that the program isn t well balanced over a whole year. You'd have observed programs over and over and over in order to get a true picture. At first there were groans and some volunteers and my other staff member felt it was going to create more work. Then morale was a little better with the understanding that our programs are improving. They are more direct on the activities that they do. Becoming more involved and modeling for the youth. Our turnover rate and morale seem to be better. Thanks to our instruction improvements, our training was more to the point and covered new materials. The staff seem to be better equipped to handle most situations. We were already using YPQA prior to the CF National pilot. However, we used the pilot as opportunity to expand the types of programs with which we use the tool. I have seen increased discussion of issues of quality and best practice across our program models due to use of the tool, and subsequent changes in policies and practices to support quality. We've done a review with them on the need for it and how it truly helps us assess our performance to know where we are strong and where we need to improve. We have not been using the tool long enough to determine changes. We have done assessments, created improvement plans which we plan on implementing this Spring. Better program quality--more prepared, organized. It has helped us to identify what qualities we want in our staff during hiring processes and in volunteer recruitment. n/a This process hasn't resulted in any changes in staff by itself. The process used with other in-services and trainings has resulted in a minor change in staff. Some staff that were a part of the process are no longer with us due to budget cuts and downsizing and volunteers that were a part of the process are no longer with us due to school schedules etc so there's not really a way to measure what effect the process has had. We only have 2 part-time direct service staff at the present time and only one some indirect experience with the process. We haven't really seen any changes for staff however, this has been a tool to generate discussion and it has been a resource for our program quality specialist. Staff are excited that their questions and concerns regarding program quality are being addressed. They seek structured advice on how to improve their programming and the toolkit gives them just that. It has allowed the entire team to work on ways to improve our skills in various areas. I have seen the staff change their own definition of quality programming and focus on those areas where we have found weaknesses. The language that the staff uses to define program has shifted to a more PQA focused language and the culture of improvement has developed. Staff are trying new things and focusing more on youth led programming and initiatives. They also understand what expectations we have of them. We have been able to train staff better on the issues we have found not working. Having the pyramid with the different levels allowed us to see where we were according to National standards and as a program. We obviously want to be able to deliver a program using the tip top but need to obtain more skills. This is my goal as program administrator. We increased the training of staff that works directly with the children. Greater consideration and attention to child & youth contribution to the program and program activities. Also, a more purposeful interest in family needs and family/youth-wellness--as it impacts program involvement. 24

25 The morale has always been good, but the turnover rate has not improved due to the hiring of college students Understanding the focus of quality in programming, tailoring a position to focus on quality, observing the ripple effect down to leaders, specific direction of quality best practices Being more aware of the quality that the Council expects and the National Headquarters. It was certainly a learning tool for new employees in regards to Camp Fire USA as a whole. It also helped all staff to focus on quality and the Board was certainly pleased with the effort put in by the staff. Understanding the why's of doing things a certain way. (Risk management, youth planning, etc.) Particularly in the staff that helped with the assessment. I am the only paid staff; our volunteer staff did not have much to do with this process and only were involved in being assessed by our assessment team. Findings were discussed with our board of directors who seemed interested in discussing ways to improve our program quality, but no changes in training, or staff morale have occurred They understand their roles better and how to show up as a youth worker. Quality of instruction has increased immensely among those that attend methods trainings. We already used the YPQA so in terms of our direct delivery staff, this had little impact. The use of an outside partner to assess our programming has helped us improve quality at the management level. We have not completed the full process, and I don't feel that I can respond to most of these questions. I have every expectation that this will be a useful process and I look forward to continuing work on it this winter and on a go-forward basis. 35. As a result of participating in this program quality improvement process, what changes, if any, have you seen for the youth with whom you work? (Outcomes, motivation, attachment to program, etc.) (N=23) I feel that our internal assessments produce much more detailed feedback and are able to show improvements or "failures" in much smaller increments. I have not seen very many changes yet. They seem to be more interested and focus on what is being offered because they have had more input in what they are doing. We haven't done outcomes yet so we don't really know but attachment to program has never been a problem. Our traditional clubs have grown this year and hopefully it is in part due to new training. Too early to tell yet for the CF pilot. Same as above. More excitement with program. Not anything tangible as of yet as we are moving into this process. n/a We conduct a 5 day a week afterschool program with youth who have been attending for over 5 years. Yearly program quality improvement has been an ongoing process to some degree, so this process in itself hasn't resulted in major changes for the youth. This process along with our internal plan and outcome measurements required from our funders has resulted in programming that is more intentional; outcome based and better meets the needs of the youth we serve. In our council we already have an approach to quality that includes implementing National Afterschool Association and American Camping Association standards into each of our programs. CFPQA was just a piece. It is difficult to gage results based solely on CFPQA impact. Youth are excited to attend programming and maintain a high level of engagement in program activities. Students have become more invested in the program, because of the leadership opportunities presented to them. One of the greatest areas of improvement is the level of involvement of the youth in programming and in other decisions. Staff are much more engaged in supporting this role in the youth. Youth seem more engaged and more motivated to participate consistently. 25

26 We have involved the children more in decision making and this really adds investment by them. More direct involvement from youth in program activity ideas. More opportunities for leadership among youth (Jr. Counselors) Greater interest in program participation. We use our outcomes to measure the changes in our youth but have not done ones yet this year. There were areas of improvement in quality which were different from our first year of our pilot assessment. They appear to be more involved with the activities and appreciate the program overall. Not to say they did not enjoy prior, but they respect the organization and happy to be a part of it. I do not directly work with the youth except at special events We have ALWAYS had great amounts of success with our youth they are always motivated, they are always excited about the program and can't wait to do anything we ask of them, there are not any ways that they could be better They trust the staff more. The families are more engaged and responsive to the youth workers when phone calls are made home (from our after school program). We were able to share best practices across one site because partners were able to observe other sites. 36. Overall, what would you change, if anything, about the quality improvement process that you participated in? (N=28) Not due I would like more and quicker access to national results or averages - both CF and other. Second it remains important to support the importance and utility of internal assessments (vs. external). I understand the strength of external assessments and have benefited from it, however I do not think this is an option (or a sustainable option) for many, if not most councils. The ratings need to cover a large scale- like 1 to 10. There is no room here to show small improvements or regressions. Nothing The training on conducting the assessments was wonderful and thorough. However, I didn't feel that there was adequate training on creating the improvement plan. In fact, I didn't realize the process was completed. I was waiting on more info for what I thought would be "Phase 2" of the project. In fact, I thought that message was communicated at the National Leadership Summit in November after findings from Phase 1 (assessment) were reported to the group. Try and make it less time consuming. It took a very long time to observe and even longer to score. We needed more help in identifying our problems and correcting them. We are considering expanding the process to include a wider range of stakeholders - youth, parents, alumni, board members etc...beyond site supervisors and program directors. To combine it with the one we currently do instead of duplicating efforts. It's hard to say - it's difficult to measure some of our outdoor programs with this tool, which is challenging. implemented the changes made in the improvement plan I'll have a better idea. The assessment takes too much time. Once we've Having more support from council board to help make this process happen. Because of everything, this has had to take a back seat whether or not I would choose to make it a first or secondary priority. More specific ways to improve areas where we are weak. For me, the 6 hours of training wasn't as useful as the written materials, i.e. the training guide and forms A and B so I would have preferred to have the option to choose the method of training. I understand the concept of a team approach but the reality is that direct service staff doesn t have the time to do the observations-there's no one left to deliver the program and supervise the youth. Using volunteers has its problems i.e. background and objectivity. Time was a critical issue for me. There just wasn't time for staff to get trained, do observations, do the scoring and participate in the improvement plan. Our staff are parttime and budget restrictions prohibit overtime. I can see our direct service staff and volunteers being a part of the process, but not to the extent that was recommended for this. I would have liked to pick and choose the method of training and if I had a problem or question, have a contact person to e mail. Time was the biggest problem-this definitely took away from my daily 26