Training Evaluation in Kentucky

Transcription

1 Training Evaluation in Kentucky Anita P. Barbee, MSSW, Ph.D. University of Louisville CB Evaluation Summit, Washington DC May 27-29

2 Louisville Child Welfare Training Evaluation Model

3 Predictors of Training Effectiveness What an individual brings to training affects their willingness and ability to transfer training concepts back on the job. These factors include past education, learning readiness, conscientiousness, self efficacy and perceived support. So we measure these variables. What the supervisor and team mates do to support training and the worker affects a worker s willingness and ability to transfer training concepts back on the job What the organization does to support best practices in terms of work environment, caseloads, career ladders affects a worker s willingness and ability to transfer training concepts back on the job. So we measure supervisory, team and organizational variables.

4 Trainee Characteristics Conscientiousness (Goldberg, 1992) predicts learning which in turn predicts training transfer as measured through case file review of assessments and case plans (Barbee & Yankeelov, 1996) High PSS (perceived social support- Cutrona & Russell, 1987) from supervisors and co-workers predicts training transfer (Barbee & Yankeelov, 1996) Learning Readiness (Coetsee, 1998; van Zyl & van Zyl, 2000) predicts gains in learning which in turn predicts training transfer as measured through complete case file review of cases (Antle, Sullivan, Barbee & Christensen, in press Child Welfare) BSW prepared social workers in a specialized child welfare education program (PCWCP) are more confident and engage in best practices (Barbee, et al, in press Journal of Social Work Education; Barbee et al, in press Child Welfare)

5 Importance of Training Context Perceiving that co-workers and supervisors support training affects a worker s willingness and ability to transfer training concepts back on the job and to stay on the job (Yankeelov, Barbee, Sullivan, & Antle (in press) Children and Youth Services Review) What the organization does to support best practices in terms of the work environment (Yankeelov, et al in press), caseloads (Barbee & Yankeelov, 1996), and career ladders (Ellett & Ellett, 2006) affects a worker s willingness and ability to transfer training concepts back on the job. So we measure supervisory, team and organizational variables in our training evaluation model.

6 Training Transfer Model Individual Characteristics: Conscientiousness Learning Readiness Organizational Factor Perceived Training Support Lower Caseload Supervisor and Co-Worker Support Learning Transfer: Assessment Case Plans Other casework factors

7 Training Cycle How a trainer behaves is important in transmitting knowledge and skills. Use of Adult Learner methods, being competent in delivering training content and following the lesson plans all affect training effectiveness The content of the training and environment in which the training is delivered affect training effectiveness

8 Structure of Training Delivery Structure of training delivery affects training outcomes Delivering classroom training interspersed with on the job training is more effective in terms of satisfaction, learning and transfer than classroom training alone-especially when the classroom training is for an extended block of time (Yankeelov, Barbee, Barber & Fox, 2000) Following up classroom training with structured training reinforcement such as our FTS program (Barbee, et al, in press Child Welfare)

9 Training teams together (supervisor and their team) is more effective than not training teams together (Antle, 2002) Conference format may enhance training participation (Special Issue in Training and Development in Human Services 2008) Retreat format may enhance training participation, especially for foster parents and adoptive parents (Sar, 2008, personal communication) Still need to test effects of distance learning

10 Content of Training Matters Create curricula and fill with material from the research literature. Then test to ensure it is evidence based in terms of impacting practice and outcomes DACUM methodology for curriculum content ITIP for curriculum design 3 rd Party Review Observation Tool to ensure excellent curriculum execution Louisville Training Evaluation Model for evidence of training effectiveness through a chain of evidence Practice Model: SBC

11 Training Outcomes: Kirkpatrick Developing a Chain of Evidence Level One: Participant reactions- affective reactions predict attitude change and utility reactions predict knowledge gain and training transfer Level Two: Gains in knowledge and skill affect training transfer and job performance Level Three: Transfer of knowledge and skill to the job predicts outcomes for families and children Level Four: Change in the Organization: Training can affect organizational development and effectiveness Level Five: Change in Clients: Training can affect client outcomes

12 Level One: Participant Reactions Affect predicts attitude change (Barbee, Schloemer, & Taylor, 1996). In a training on Substance Abuse in Families found that when the trainees liked the trainer more, they were more likely to change their attitudes in a positive direction towards substance abusers. Attitude change about substance abusers predicted willingness to work with them at the end of training and perceived success at a 3 month follow-up (Barbee, et al, 1996) Affect does not predict learning (Barbee, et al, 1996) Utility reactions predict learning (Antle, 2002; Antle, et al, in press Child Welfare)

13 P&P Courses across all 16 years there have been high levels of satisfaction IL- The level I ratings have shown high utility (Average of 4.0 on a 5 point scale), Undoing Racism- High levels of satisfaction, especially for people of color

14 Level Two: Gain in Knowledge and Skills (Learning) Core P&P curricula (1992-present) On average, in child welfare training workers start with about 64% knowledge and significantly improve to about 80% by the post-test. Coaching and Mentoring Significant gain in learning from pre-test (average score of 58%) to post-test (average score of 78%) by supervisors who took the coaching and mentoring training. Solution Based Casework for Supervisors See charts Independent Living: Significant increase from pre to post (9%)

15 Level 3: Training Transfer Measure through triangulation of worker, supervisors and FTS self reports using a behavioral anchor rating scale Measure through self reports of attitude change, behavioral change usually using quality measures Measure through supervisor (customer) satisfaction with what the employee learned Measure through observation of key behaviors Measure through chart file reviews, including CQI measures of cases

16 P&P training: Average 3.49 on a 5 point scale on the behaviors PCWCP: Average 3.89 on a 5 point scale on the behaviors- demonstrate minimal competence PCWCP: Outcomes better (see earlier slides) FTS program utilizes observation and improvement over time Solution Based Casework for Supervisors

17 Undoing Racism Training Evaluation Satisfaction Race Literacy Quiz Color-Blind Racial Attitudes Scale (CoBRAS), developed by Neville, Lilly, Duran, Lee, and Browne (2000) Racial Privilege Institutional Discrimination Blatant Racial Issues 422 participants in the training completed measures All but 13% decreased their denial of racial discrimination on the 3 measures of the CoBRAS this was particularly true of those with more education and those who were younger

18 Level Four: Organizational Impact Worker Competence (could also be seen as a level 3 training transfer measure) Worker Retention PCWCP Prospective Study on retention

19 PCWCP Evaluation Results 2 Year Follow-Up 360 graduates have been out 2 years or more 310 or 86% are still with the Cabinet A random sample of other workers from the same time period shows a 54% retention rate, Chi Square = 5.79, p <.05 ROI = $2,000,000 per year for these graduates 26% have been promoted to higher positions 14% have completed or are pursuing a master s degree. 96% are in Social Work, 4% are in Public Administration. The average lag time is 1.2 years to begin a master s degree

20 Longitudinal Trends on Retention 93% of the 2004 PCWCP class has remained employed with the Cabinet (2 year retention) 79% of the 2003 PCWCP class has remained employed with the Cabinet (3 year retention) 63% of the 2002 PCWCP class has remained employed with the Cabinet (4 year retention) 59% of the 2001 PCWCP class has remained employed with the Cabinet (5 year retention) 53% of the 2000 PCWCP class has remained employed with the Cabinet (6 year retention) 57% of the 1999 PCWCP class has remained employed with the Cabinet (7 year retention) 62% of the 1998 PCWCP class has remained employed with the Cabinet (8 year retention)

21 Level 4: Training Transfer Affects Worker Retention Yankeelov et al. (in press) Quasi-experimental, prospective research design Measures taken within the first 3 months of employment All new child welfare employees hired between 1996 and 2000 completed surveys 2001 personnel data was used to determine retention and rank in the organization 976 entry level employees had data that was useable for this study

22 Results: Categories of Organizational Movement Stayers: 66% Remained in the Organization 48% stayed in their present job 43% had already been promoted 9% churned in the organization (meaning they moved from location to location or move to various types of jobs several times) until they found a home in the organization. Leavers: 32% Left the Organization 72% were from their original position. 54% within the first year 25% had received promotions (25%). 3% churned before leaving Returners: 2% Left and Then Returned 81% returned to original level job 19% returned to a higher level job

23 Comparisons Stayers are more likely to be promoted (43%) than Leavers (25%) which may speak to differences in skills, openings in the organization at the time, or other workplace factors. Perhaps some people leave because they are frustrated that they were not promoted or don t see a way to be promoted. Stayers are the biggest group to churn in the organization. These individuals seem to need to move around until they find the right niche for themselves, then they stay. Most Leavers do so in the first year, thus programs to reduce turnover in the first year are important. Returners may have life circumstances that affect their willingness or ability to stay on the job, but eventually they come back to the same or a higher position.

24 Take Away Message on Building an Effective Training System With Strong Training Transfer What trainees bring to the training impacts transfer (conscientious,ready to learn, values- BSW) What trainers do impacts transfer. Making training relevant and strategies that lead to greater learning of knowledge and skills enhances transfer. How training delivered impacts transfer. Team training with supervisors present and use of adult learning principles enhances transfer What happens back in the office on the team impacts transfer. Co-workers who are helpful and supportive enhance transfer. Structured reinforcement by mentors or frequent case consultation enhances transfer. What happens in supervision impacts transfer. Supervisors who are more approachable, supportive, and take an interest in training. Organizational Culture

25 Training as an Organizational Intervention Should see ourselves like an internal Resource Center- to develop, deliver and evaluate training at all levels of the organization (leadership, supervisors, front-line workers both new and veteran) and to provide technical assistance Technical assistance can be in the form of TOTs on particular areas of expertise, help in developing tools (risk assessment tool), help deliver additional services like deeper assessments (CATS, FORECAST) help in strategic planning, solving particular practice issues, addressing problems like disparity and disproportionality, help in bringing about organizational change

26 Workforce Solutions Employee Selection Increasing number and quality of potential employees Behavioral Interviewing In-basket tasks for selection (Graef & Hill, 2000) Personnel selection testing: securely attached, conscientious, ready to learn, open to experience, agreeable, good values (Yankeelov, et al, under review) Enhancing education of employees PCWCP Program in Kentucky (Fox, et. al, 2000, 2003)

27 Workforce Solutions Training and Development Developing Infrastructure University-Agency Partnerships (Zlotnik, 2003) Learning Organization Principles and Training (Fox, Barbee et.al, 2002) Evidence-Based Training Practices Field Training Specialist Model (Barbee, et.al., in press) Extensive training evaluation system (Barbee & Yankeelov, 2001) Use of evidence-based practice models (Antle, et. al., 2008) Use of multi-disciplinary assessment centers (Brenzel, Kaak, Clark & Sprang, 2005)

28 Workforce Solutions Workplace variables Using mechanisms like Accreditation to lower caseload size and team size Organizational culture Learning Organization Supervision and Teamwork Ensuring competence and support of supervisors Retention study results Building high functioning teams ASFA Supervisory-Team Training Curriculum

29 Contribution to Knowledge This line of research illuminates the importance of adopting a state wide practice model for child welfare practice which is grounded in policy, supported by managers and supervisors, taught rigorously in training, reinforced in the field, evaluated in terms of day to day practice through case reviews and evaluated through rigorous on-going evaluation for pre-employment and post-employment training.

30 Results and Conclusions We have been able to repeatedly demonstrate that When trainees are ready to learn and conscientious, training is of high quality, and reinforced in the field (particularly when using standardized programs that ensure high fidelity) then Participants tend to receive the training positively, Gain in knowledge and skills, Execute those skills in the field Produce quality outcomes for their clients (safety and wellbeing) and their organizations (retention).