Early Childhood Family Mental Health (ECFMH) Credential: Lamoille County Pilot Cohort Evaluation Report, March 2013

Early Childhood Family Mental Health (ECFMH) Credential: Lamoille County Pilot Cohort Evaluation Report, March 2013 Lamoille Family Center (LFC) Children s Integrated Services (CIS) staff sought out information about the Early Childhood Family Mental Health Credential in the summer of 2010. The LFC director learned about the credential at a meeting with Parent Child Center Directors. Staff interest and the fact that the LFC was also a pilot CIS site, led him to invite an ECFMH Credential committee member (Northern Lights staff) to give an orientation about the credential to LFC employees and other staff working with young children and families in the region. Two orientation sessions were held over the summer and fall of 2011 at the LFC before the group decided to be an ECFMH Cohort Pilot. The ECFMH Credential committee continued to finalize the credential process even as the Pilot Cohort launched. The LFC director hired the Reflective Practice Consultant (RPC) to work with the LFC staff seeking the credential. She provided individual consultation to them, as well as monthly group consultation to all candidates. Three of the original candidates were employed by other agencies, and had individual consultation by a different RPC. The LFC director also supported candidates to use their completed ECFMH selfassessments to identify common training needs. He then budgeted and arranged time for staff to attend the trainings, on a schedule they recommended. The ECFMH Credential committee provided the funding for the instructor, and notebooks with hard copies of the ECFMH documents and guidance. Some documents were also on the Northern Lights website and additional documents were sent by email to candidates, the LFC director and the Reflective Practice Consultants. ECFMH Committee members met with the cohort twice during the 15 months of their work on the credential, in order to evaluate the process and address questions. Northern Lights staff maintained email and phone contact with candidates, the LFC director and the Reflective Practice Consultants, as well. Additional documents with examples were created and shared with the cohort, as a result of these interactions. The final decisions about the ECFMH portfolio organization and review process were shared with the cohort nine months into the process. LFC hosted a final celebration and reflection of the credential in December 2012. The first portfolios were submitted for review in November 2012. Tools used to evaluate the Pilot and write this report include: o Intent to apply (Initial) Application from all candidates (11) and outreach to potential candidates o Training Maps, Self- Assessments and IPDPs from candidates completed over the first 8 months o Exit interviews with one candidate and her RPC when she left the Cohort o Survey of Candidates and interviews with RPCs six months after beginning the process o Final Applications/ surveys from the five candidates who applied to date o Final survey from RPCs and the LFC Program Director o Group interviews with applicants and reviewers at the end of the process o Portfolios completed (five to date) This report is divided into five sections: I. Basic Information II. Time to Complete the Credential III. Process and Tools after 6 months and at the end IV. Support and Professional Growth V. Results from Accomplishing the Credential VI. The Reflective Practice Consultant perspective VII. The Review Process and Project Evaluation 1

I. Basic Information Eleven candidates began the ECFMH Credential process. Two have a Master Degree and nine have a Bachelor degree, in the fields of Counseling, Psychology, Social Work and Education The group who began the credential, average 21 years of experience in the field (range 17 to 33). Experience in their current job averages 5.7 years, with a range from less than one month to 12 years. Nine worked for Lamoille Family Center (LFC) and 2 worked for other agencies; but all contributed to the Children s Integrated Services team. All had group RPC meetings monthly with the RPC who worked with LFC staff. As of March 2013, there are eight women in the pilot cohort, who have either completed the process or are near completion. o Three did not complete the process for the following reasons: Increased workload and the stress of not being co-located with the LFC team (1) Left job at LFC (2) o The three who did not complete the pilot had from 17 to 33 years of experience in the field, and experience in their current position ranged from less than a month to 7 years. Current status of the Cohort is outlined in this chart below COHORT education Employer Role Individual RPC Completed process and 3 Bachelor LFC -Infant toddler teacher & LFC approved (4) 1 Master director -Early Intervention coordinator & Developmental Educator -Family Support worker & Home visitor (2) Pending (1) Master Not LFC - CIS consultant and Not at LFC Will submit portfolio in 30 days (3) Program Manager 3 Bachelor LFC - Child Care Coordinator - Developmental Educator & Service Coordinator - Developmental Educator LFC Several people expressed interest in the credential and attended an orientation, but did not pursue the process. All of them were not Lamoille Family Center (LFC) employees and would have been going through the credential process individually. Seven of the eight completers work at LFC, and represent almost a quarter (23%) of the total LFC staff of 35 people. 2

What the data suggests: Group cohesion, particularly the group RPC meetings with the direct support of the host agency director contributed to success. The group had a high level of completed education and experience; there was no correlation between education, experience and applicants dropping out of the cohort. Within the limitations of the sample, it appears that the ECFMH Credential competencies can be met by professionals providing a range of services to young children and their families. Recommendations: Candidates should be part of a cohort; attending common group RPC meetings. Evaluate future cohorts to determine if there is any relationship between successful completion of the credential with level of education and experience of candidate, or work setting and role. II. Time to complete the Credential Length of time to complete: Ten candidates were accepted into the process in early August 2011. One dropped out before the midway point and one joined after the midway point as a new LFC employee. Nine completed the midway survey in mid February 2012 (six month point). Six months into the process, two predicted they would complete the process in less than six months and seven predicted six to eight months. o No one s prediction was correct. It took everyone longer than she thought it would. o Completed 9 ½ - 11 ½ months later (5) o Have not yet submitted applications 13 months later. (3) Total time to complete from acceptance to final application: o 15 ½ months (1); o 16 months (3); o 17 months (1); o 19+ months (3) What the data suggests: The process takes 16 to 20 month to complete. Completion takes longer than candidates anticipate after six months into the process. This may be due to the amount of work/time it takes to complete and assemble the portfolio, and indicate a need for more support. Recommendations: Adjust expectations of how long the credential process takes. Include new estimates in candidate instructions (18 months). Revamp materials to expedite portfolio process. IIIA. Process and Tools - Midway From Midway Survey of cohort candidates February 2012 (9 responders) 3

Six months into the process, nine candidates rated the process and tools as follows: The initial information and process to help candidates understand and prepare for the Credential process was helpful. (1 not helpful - 4 very helpful) o 2.8 usefulness of initial information o 3.0 usefulness of initial process (filling out Initial Application, talking with Northern Lights staff, identifying RPC) o Comment: it was helpful to do with a group (1) See Chart below (9 responses total) The training map was moderate in difficulty (1 difficult - 4 easy) 2.7 o Comment: Difficult and time consuming to locate information (1) o See chart below ( 9 total responses) 4

NOTE: chart above are responses from 9 people The self assessment was somewhat difficult (1 difficult - 4 easy) 2.0 o Comment: Very time consuming (7) The Individual Professional Development Plan (IPDP) based on the ECFMH Competencies self- assessment was completed by seven of nine candidates, six months into the process. Five got help with the IPDP from the RPC and coworkers; no one got help from her supervisor. o Challenges: time (4), format (1), setting goals (4) o Benefits: setting goals (6), determining strategies (5) The Credential process in the first six months: o Challenges: time (7), understanding the process start to finish (1) o Rewards: group sessions/support (7), Pat Messerle group training on topics identified by the cohort (2), learning/growth (4) What the data suggests: The tools are somewhat challenging to use. The IPDP was not created directly after the self- assessment was completed, as the Credential committee assumed it would be. Setting goals is both challenging and beneficial Candidates have difficulty finding time to work on the credential Group support fosters confidence and completion of steps 5

Recommendations: Revise instructions and format for all tools Consider the support structure for completing the tools. For example, structuring group time to work on them, training for RPCs in using the tools, instructive sessions with ECFMH Credential staff support IIIB. Process and Tools at the End From Final Applications of candidate, December 2012/January 2013 (5 responders) 17 months from start of the process (about 10 months after the midway survey) five candidates rated the process and tools as follows: The process of preparing and submitting the final application was somewhat confusing (1 confusing 5 very clear ) average 1.8 o Paperwork, directions and expectations were confusing (5) o Relationship of self-assessment to IPDP was not smooth (2) Note in chart above: 5 total responses 20% = 1 person; 40% = 2 people The ease of completing each requirement ranged from hard (1) to achievable (4) o Completed education 4.0 easily achieved o Relevant training 3.6 o RPC Time 3.4 o Training Map, Self-Assessment, IPDP (the tools) 2.2 Comment: time consuming and somewhat confusing o Portfolio Organizing 1.8 6

Note: Numbers in chart below indicate number of individuals out of 5 respondents Note: Numbers in chart above indicate number of responses out of 5 total Relevancy of the requirements: Most of the candidates perceived that the requirements were germane to meeting the competencies, with the exception of the portfolio organization. (1 least relevant - 4 very relevant) o Completed education 3.6 o Relevant training 3.0 o Experience 3.4 o RPC Time 3.4 o Portfolio Organizing 2.6 Note: Numbers in chart below indicate # of individuals out of 5 respondents 7

Note: Numbers in chart above indicate # of responders out of 5 total From the RPC Perspective (2 responders) (rating: 1 challenging to use and understand 4 clear and easy to use) Training Map, Self-Assessment, IPDP, and creating/collecting evidence all were somewhat clear and easy to use 3.0. Organizing the portfolio was rated as somewhat unclear and challenging to use (1) by one RPC, and challenging to use and understand (2) by the other. Overall rating 1.5 What the data suggests: o The candidates found the tools somewhat confusing and time consuming; however this is not the case from the RPC point of view. This difference in perception may indicate that the RPC was not very involved in using these tools. o Most of the candidates found the requirements (education, experience, time with RPC, completion of the tools) relevant to the Credential. o Six months into the process, the tools were also somewhat difficult to understand and very time consuming to complete. o Both candidates and RPCs found it confusing and challenging to organize the portfolio, including preparing and submitting the final application. 8

Recommendations: o Improve instructions and support for use of the tools and particularly organizing the portfolio. Suggest how much time will be needed to complete the tools. Consider the RPC s role and the group cohort meetings in how they might support completion of the tools. o Provide a thorough orientation to the Credential, with stories from the Pilot candidates to illustrate the process and lessen confusion. o Ensure that candidates receive a thorough orientation, to ensure they understand the content and expectations of the credential process. IV. Support and Professional Growth The five candidates that submitted their final application, reported that they received the following Professional Support (outside of the RPC relationship) Supervision (5) Part of a professional network (3) Support of a regular consultant (1) Another form of professional support (2) None had a mentor/coach All worked with several community partners (see chart) 9

Help with the Credentialing Process came from various sources Colleagues at work (4) Candidates supervisor (3) Northern Lights staff (2) Candidate s family (1) The following chart describes who initially helped the five candidates with their IPDP. Help with IPDP at beginning of the credential process People helped candidates by: Sharing ideas, organizational skills, and information to get portfolios finished Offering an objective view of the candidate and her work Providing training opportunities (The cohort identified training topics and LFC arranged for Pat Messerle to provide the training; these trainings got very positive reviews from all candidates.) The host program director s perspective: Supports provided: Work time and money for professional development and reflective practice Limits to support: Time required for start up, and funding needs Recommended supports: A team of people and additional funding 10

What the data suggests: All five candidates are well connected in their professional community and receive support at their place of employment. Generally, they re a well supported group, however the credential process, and particularly the portfolio organization took a great deal of time in an already busy work setting. They identified the need for additional help creating the final portfolio. The credential process requires time and money from the host agency, and possibly other financial resources to complete. Recommendations: Additional support from the ECFMH Credential committee in the form of instructions and models to organize the portfolio. Use the pilot cohort examples for the next cohorts. Consider how the RPC group time might be used to share strategies and resources to develop and organize the portfolio. Also consider how online tools could support candidates to share and reflect together as they work on their documentation. Help with time management related to the credential process for host agency and candidates. Provide flexible financial support to cohorts, to meet needs they identify. V. Results from accomplishing the credential When (5) candidates were beginning the process, they said they pursued the credential to: Gain recognition for years of training and experience Feel more a part of CIS team Learn about resources Learn more about early childhood and mental health (4) They believed they would benefit from: Reviewing the training they have already received to set goals Increased opportunities for networking, supervision, and training Becoming more effective in their practice When candidates finished the process and reflected back, they said they pursued the credential in order to: Broaden knowledge of specific childhood mental health topics Share their previous training with others Receive clinical supervision Grow and learn Two candidates reported their expectations were fully met; one mostly, one somewhat, creating an overall rating of 3.0 (4 is fully met) expectations were mostly met One candidate expected more attention to the deep issues in families with mental health challenges One candidate felt her many years of training and experience were not credited in the credential process 11

Note: chart above represents 5 total responders (20% = 1 person; 40% = 2 people) Overall benefits and impact of the credential process More confidence in and improved ability to communicate with parents, particularly about difficult issues More knowledgeable and confident in the job overall More cohesive team New strategies for working with children experiencing mental health challenges Better organizational and time management skills Direct quotes from candidates I have grown tremendously The impact has been huge This was a worthwhile experience for me and the families I work with I feel more knowledgeable and competent I have expanded my areas of expertise to better help my clients My expectations were fully met. This process actually exceeded my expectations! 12

A total of 144 children benefitted from the impact of this process on these five professionals. Benefits to the host organization Strengthened team work Enhanced knowledge base Team focus on common practices The director rated the experience as very beneficial for the organization (rating of 4.0 out of 4) What the data suggests: The process resulted in increased knowledge, improved practice and confidence, and strengthened team cohesiveness. Recommendations: Share the above information with prospective candidates to encourage and motivate them. VI. The Reflective Practice Consultant Experience Basic Information There were two RPCs who went through the entire credential process with candidates. There was a third RPC supervising the candidate that began, but dropped out of the process. Seven of the eight candidates that completed (or are completing) the process had the same individual RPC, who was also the group RPC, under contract with LFC. Time 87.5% or 7 candidates met individually with the RPC monthly; one met weekly Individual RPC meetings were about an hour long All candidates met as a group with the RPC monthly, for about two hours Candidates found completing the required RPC time achievable to easily achieved (3.4 out of 4) They considered this time to be relevant to very relevant (3.4 out of 4) Process and Tools Both RPCs felt they had enough time with the candidate/s to recommend her for the credential. One RPC needed a better understanding of his role The RPC who provided both group and individual support felt the paperwork was the most confusing part for the candidates The RPC who provided both group and individual support felt it was valuable to have both roles Support and Professional Growth Six months into the credential process, candidates reported: The individual RPC time was very effective (7) in supporting professional growth; one said it was effective (overall 3.4 rating out of 4) The group RPC time was mostly effective to effective (overall 2.6 rating out of 4) 13

Eight candidates had been observed once by the RPC. The words or phrases used to describe the impact of this were: o Agenda, reflective, enlightening, supportive, thoughtful, timeliness, focused, helpful, empowering, informative, validating, organized, positive outcomes o What she noticed about my technique with children o It helped me to get another opinion on my practice o It helped me to be more aware of the process I use when working with clients o How I talked with parents Note: Chart above represents a total of 8 respondents (12.5% = 1; 25% = 2; 37.5 = 3; 87.5=4) 14

After completing the process, candidates rated how helpful the RPC was with the various tools. (Note: The IPDP is the only tool that was included in the description of the RPC s role) Scale: 1 not helpful 4 very helpful Creating the IPDP 3.0, helpful Fulfilling the IPDP 2.8, somewhat helpful to helpful Meeting the competencies 2.8, somewhat helpful to helpful Completing the portfolio 2.6, somewhat helpful to helpful Supporting professional growth 3.4, helpful to very helpful Note: Chart above shows 5 total responders (1=20%; 2=40%, 3=60%, 4=80%) RPC perspective on the role RPC suggested further guidelines or support to enrich the RPC role: o Checklist of paperwork needed throughout process o Clearer understanding of role Both RPCs felt the match was good with candidates and there were not any difficulties. One candidate didn t feel RPC time addressed deeper aspects of family mental health needs. 15

Results The RPC who provided both individual and group consultation, felt the role was: Very beneficial to her (4.0) It supported her personal and professional growth and learning She would recommend the role to others; she enjoyed it The RPC that worked with only one candidate felt the role was: Somewhat beneficial to him (3.0 out of 4) He learned about the candidate and her role in the system of care He is unsure if he would recommend the role He thinks there was some sort of disconnect since the candidate was not initially approved for the credential, because some paperwork was missing that she needed to complete first. The host program director cited these benefits of having the RPC work with staff: Shared understanding of complex cases Interpersonal learning Clinical insights What the data suggests: It was effective for both the candidates and RPC to have the individual RPC also be the group RPC, and to be well connected to the host program RPCs were most helpful with professional growth and less so with portfolio development Group times with the RPC were beneficial, and individual times with the RPC were more so Recommendations: ECFMH Committee provide an more extensive orientation to help RPCs have a clearer understanding of their role and the candidate s process RPCs might benefit from a more intentional relationship with each other to identify and support issues and strategies for cohort members VI. The Review Process and Project Evaluation Five candidates submitted their final application and portfolio. Northern Lights staff, completed an initial review of the portfolio for completeness and accuracy. They also documented how the candidate met the education, relevant training and experience requirements. Some of the documents were then copied and sent to the review committee members, ahead of the meeting for their preparatory review. Each portfolio was reviewed by a team of three people who met face to face. Nancy Sugarman was the consistent member of each review team. There were a total of seven reviewers (including Nancy) that formed three teams, reviewing a total of five portfolios. It took review teams from one to four hours per candidate, to review a portfolio. Prior to meeting, reviewers were given guidance to prepare for the process. They had an email orientation, were given consistent criteria, and had a face to face orientation at the time of the review. Teams needed to reach consensus that there was strong evidence of competency for each subdomain, in 16

order for the competency to be approved. Competency was assessed based on these primary sources of evidence: RPC letter Letter from a parent Final IPDP Final Self- assessment with descriptions Reflective Summary statement of their competency by domain and evidence documents for the subdomain Secondary sources of evidence included: Transcript Resume Training map and certificates Notes were taken during the review process, summarizing the strengths and sometimes suggestions for the candidate. If a candidate didn t show strong evidence, she was given guidance and suggestions about how to provide what was needed. Often the evidence was not clear ( It s not that you don t know it, it s that you don t show it ). Review Analysis The results of each review were analyzed as follows to create the summary chart below: Each domain has several subdomains. Candidates were rated by the review team, as approved or pending, in each subdomain. The number of candidates that were approved in a given subdomain were added up and this determined the score for that subdomain These subdomain scores were then added up creating a total score for each domain area. This score was converted into a percentage representing how many candidates were approved or pending for that domain area. Results of five reviews Domain Approved Pending Philosophy & Professional Orientation 68% 32% Family Systems 73% 20% Child Development 65% 35% Assessment 80% 20% Addressing Challenges 53% 47% Systems Resources 80% 20% Most subdomains had either three or four people that were approved in that area. There were not any subdomains in which all five candidates were approved or all five candidates were pending. 17

Only two people were initially approved in the subdomains Risk and resilience in children and families and Specialized knowledge of vulnerable populations, which both fall under the domain Addressing Challenges. What the data suggests: It s difficult to determine at this point, to what extent the scores in the chart above represent candidates : o Training needs o Problems understanding how to provide evidence of competency in particular subdomains o Issues with the wording of the competency Despite the challenges candidates reported about organizing their portfolios, those portfolios reviewed to date were excellent examples of portfolio organization and provided an impressive range of evidence. Much of the evidence was supported by the diverse collaborations, training and supports that staff receive from the LFC. It will important to continue to track how evidence is provided in different cohorts. Portfolios with missing or less clear documentation, took longer to review. This is consistent with other portfolio review processes, unrelated to this credential. The reviewers were willing to review more portfolios and found the process thoughtful, educational and relevant to their work. Review of the portfolios submitted, shows that a wide array of available training is relevant to the ECFMH competencies in the past five years. It would be useful to continue to track this over other time with other cohorts, to determine if there continues to be sufficient training available to meet the needs of candidates. The process seemed to support recognition of competence and professional growth, even in these very highly qualified and experienced professionals. The credential was designed for people in a variety of roles. Through the review process, questions arose concerning how well the credential competencies can be fully met and/or documented by professionals in different roles and settings. For example, a teacher can more easily document curriculum related competencies; a teacher working in a large agency may have a more difficulty documenting competency in planning budgets. The ECFMH credential committee should continue to monitor and evaluate how well the ECFMH competencies are met when there s a larger candidate pool representing more diverse roles and work settings. In most cases, when there was missing documentation, reviewers thought that the candidate did have competency but was not adequately showing documented evidence. Reviewers wondered if more experienced people had more trouble showing how they know what they know. Sometimes when skills and knowledge are deeply ingrained, it s more difficult to tease out how we do what we do. Project Evaluation Recommendations 1. Revamp Evaluation Surveys a. Group questions according to these categories, and include the category headings in the survey: Basic Information, Tools and Process, Support and Professional Growth, Results/Impact b. Establish a consistent Likert scale where the high number is the best score and the word descriptions are standard across survey questions 18

c. Reenter new survey format on Survey Monkey d. Repeat in the final survey, some of the questions posed at the midway survey e. Given that the credential likely takes longer than a year, implement the midway survey at 7-8 months, instead of 6 months from the beginning of the process. 2. Follow up with cohort members after they complete the credential a. Create and implement a process b. Evaluate ongoing impact of the credential c. Suggest that approved candidates serve as reviewers, or as mentors to current candidates 3. Create and use a survey for reviewers a. Were they prepared enough for the review process? b. Did they have enough information about the candidate to come to a conclusion about competency? c. What did reviewers think about the process of synthesizing the information provided by the various tools? 4. Standardize the process of evaluation analysis a. All surveys are completed on Survey Monkey (candidates, RPCs, and directors) b. Results are reported by categories (see above) incorporating all respondents, to get an overall picture of how the credential program is developing c. Finalize the evaluation collection and review process in preparation for subsequent cohorts. Additional questions posed in the evaluation plan, will need to be included in the evaluation process as new cohorts develop. Identify how this data will be collected and included in the evaluation process. Early Childhood and Family Mental Health Credential Committee Northern Lights Career Development Center CCV; PO Box 489 Montpelier VT 05601 802-828-2876 Nancy.sugarman@ccv.edu This and other documents about the Early Childhood Family Mental Health Credential can be found on the Northern Lights website: www.northernlightscd.org March 2013 19