RESEARCH RESULTS CLIENT RESEARCH RESULTS BY: MIKE KEMP CONSULTANT, CABER REALIZATION RESULTS: Compared to the previous hiring process, the current hiring process was rated more than three times better at hiring individuals with good leadership potential and more than four times better at hiring individuals who are more likely to stay with NYP. An estimated $40,000 was saved due to the decrease in the number of applicants interviewed and amount of hiring manager time needed to conduct interviews. The number of weeks it takes to fill nurse positions was reduced, resulting in an estimated savings of nearly $1.4 million in overtime that would have otherwise been paid to cover vacancies. HEALTH CARE HIRING HITS BOTTOM LINE INDUSTRY Health Care DDI PRODUCT USED Success Profiles TM ; Targeted Selection ; Nurse Career Battery BUSINESS NEED With five campuses staffed by more than 20,000 people, NewYork-Presbyterian (NYP) annually treats over a million patients in metropolitan New York, earning $3.5 billion in revenue. It has been recognized as one of the most comprehensive university hospitals in the world, employing the leading specialists in every medical field. Because of its dedication to continually improve the patients experience, as it is part of the very competitive health care industry, NYP recognized the importance of hiring and retaining the best caregivers. As a result NYP sought to design a selection system that would identify and enable the retention of its top-tier talent. It partnered with Development Dimensions International (DDI) to create Success Profiles TM for five key positions that defined the clinical knowledge, skills, abilities, and personal attributes essential for job success. NYP wanted to first focus on the nurses because of their direct impact on the patient experience. With more than 200,000 nursing applications received annually, the organization needed a selection system that could quickly and efficiently identify the right talent while providing a professional, courteous experience for every applicant. SOLUTION To create a selection process for nurses, NYP first defined the clinical skills as well as the motivation that its top-tier nurses possess. Along with this career profile, NYP incorporated into its hiring process DDI s Nurse Career Battery (NCB) pre-employment test and Targeted Selection (TS) interviewing process. The hospital also worked with DDI to design a comprehensive measurement strategy to ensure these new hiring tools were having a positive impact on its business. RESEARCH DESIGN To design the measurement strategy, DDI and NYP outlined the hiring process changes and the expectations of those changes in a measurement model (Figure 1). Metrics were identified to ensure the expectations were being met. The appropriate data was then collected in three phases: A criterion validation study of the NCB ensured the Success Profiles TM test for nurses was effective in predicting job success. After NYP s hiring managers attended the TS Interviewer training, a reactions survey evaluated the relevance and effectiveness of the program for making better hiring decisions. DDI s Hiring Checkpoint SM survey evaluated the utility and impact of the NCB and TS on NYP s nurse-hiring process.
Figure 1: Hiring Process Measurement Model RESULTS This research leveraged three studies to confirm the expectations outlined in the measurement model: A criterion validation study of the NCB, evaluation of the Targeted Selection Interviewer training course reactions, and DDI s Hiring Checkpoint SM survey, encompassing the majority of this report, to evaluate the outcomes and impact of the hiring process changes (i.e., the NCB and TS ). Criterion Validation Study In 2009, the criterion validation study of the NCB was conducted. As part of the study, 117 registered nurses already working for NYP completed the NCB pre-employment test, while at the same time, their managers completed onthe-job performance appraisals evaluating the same competencies identified in the NCB. The relationship between the NCB test scores and performance evaluation scores demonstrated the NCB s ability to predict onthe-job performance. The correlation (also known as the validity coefficient) between NCB scores and overall job performance was found to be 0.39 (statistically significant). The US Department of Labor 1 interprets validity coefficient values above 0.35 (the highest range cited) to be Very Beneficial to making hiring decisions. Additionally, Figure 2 demonstrates that those who scored in the top 25 percent of the NCB were almost twice as likely to be rated as above-average performers compared to those who scored in the bottom 25 percent. 1 U. S. Department of Labor (2000). Testing and assessment: An employer s guide to good practices. Retrieved April 6, 2007, from http://www.onetcenter.org/dl_files/emptestasse.pdf. 2
Percent Rated as Above-Average Performers Figure 2. Percent of Above-Average Performers by NCB Score Group 100% 80% 60% 46% 54% 83% Top 25% 40% Mid 50% 20% Bottom 25% 0% Low Score NCB Moderate Score Group Score High Score As an additional result of the criterion validation study, DDI was able to recalibrate the scoring of the NCB to be maximally effective in predicting job success for NYP nurses. Therefore, referring to the model in Figure 1 as a framework, the criterion validation study was able to show that the competencies evaluated using the NCB were critical to job success while recalibrating the NCB ensured implementation success. TS Interviewer Training Reactions In order to conduct effective interviews, hiring managers were required to attend a two-day training session on the Targeted Selection interviewing process. Following the training, hiring managers completed a course evaluation survey to gather their reactions. Figure 3 presents the results of the course reactions, averaged across hiring managers. Figure 3. Targeted Selection Interviewer Training Course Reactions Length of Training Training Materials Course Content Instructor Would Recommend this Course to Others 4.3 4.4 4.5 4.7 4.5 1 2 3 4 5 Five-point Rating Scale (Strongly Disagree to Strongly Agree) 3
All five areas evaluated in the course reactions survey received ratings greater than 4.0 out of 5, indicating very favorable reactions to the interviewer training. The training program and resulting reactions were strong indicators that implementation of the TS interviewing process would be a success at NYP. Moreover, the Hiring Checkpoint SM survey also evaluated perceptions of the interviewer training program and its results after managers had a chance to apply their new interviewing skills back on the job. Hiring Checkpoint SM Survey The Hiring Checkpoint SM survey focused on three main topics to evaluate the use and impact of the NYP hiring process since the NCB and TS interviewing tools were added. Specifically, the study evaluated the effectiveness of the interviewer training and the interview experience, the quality of new hires, and the overall perceptions of improvements in the hiring process. Essentially, the survey was used to evaluate the outcomes and impact of the new tools at NYP outlined in the hiring process measurement model (Figure 1). As an online survey, Hiring Checkpoint SM was completed by hiring managers and new hires. The survey was administered within one year after the new tools were introduced into the hiring process. Hiring managers were asked to compare the current hiring process to the previous hiring process (before the new tools were implemented), while new hires were asked to compare their hiring experience to those with other organizations. Surveys were sent to 96 hiring managers who experienced both the current hiring process (including the NCB and TS ) and the previous hiring process for nurses. Twenty-seven of those completed the survey, resulting in a 28 percent response rate. Surveys were also sent to 282 new hires (hired within the past 6 months using the current hiring process). Seventy-nine of those new hires completed the survey, resulting in a 28 percent response rate. Both hiring managers and new hires represented eight different specialty areas (Figures 4 and 5). Figure 4. Percent of Hiring Managers by Specialty Area ACN 3.7 3.7 14.8 11.1 7.4 37.0 14.8 7.4 Critical Care ED Medical/Surgical Peds Periop Specialty Area Women's Health 4
Figure 5. Percent of New Hires by Specialty Area Critical Care 15.2 3.8 13.9 13.9 3.8 25.3 17.7 6.3 ED Medical/Surgical Peds Periop Specialty Area Step Down Women's Health The Hiring Interview In addition to the training reactions survey that managers completed immediately after the twoday program, perceptions of the interviewer training were also gathered from hiring managers in the Hiring Checkpoint SM survey to evaluate its impact after they had a chance to apply their new skills. Figure 6 below presents the percent of hiring managers who agree or strongly agree with each aspect of the interviewer training. Figure 6. Hiring Manager Interviewer Training Reactions The two-day training session I attended was the right amount of time to spend on learning the concepts and practicing the process. 60.9 The skills/knowledge gained from the training are useful to me when conducting interviews. 73.0 I am motivated to use the skills taught in interviewer training. 75.0 I would recommend the TS interviewer training program to fellow employees. 76.9 The training session I attended adequately prepared me to conduct my first interview. 77.0 I am confident in my ability to conduct a good interview. 100.0 0 10 20 30 40 50 60 70 80 90 100 Percent Agree or Strongly Agree 5
Encouragingly, the majority of hiring managers positively rated all aspects of the interviewer training outlined in Figure 6. More than three of every four managers feel the training program adequately prepared them to conduct their first interview, are motivated to use the skills taught in the program, and would recommend the TS Interviewer training program to fellow employees. Moreover, 100 percent of hiring managers felt confident in their abilities to conduct a quality interview. The lowest ratings regarding the training program were related to the length of the program. Approximately 40 percent of hiring managers did not agree that the two-day program provided the right amount of time needed to learn and practice the concepts in the TS interviewing process. Additional information should be gathered to understand whether some hiring managers feel that the program is too short or too long. Overall, these ratings are encouraging because it is critical for managers to recognize the value and relevance of the training program. If hiring managers do not feel the training program is valuable or relevant to the hiring process they will be less likely to apply what they learned back on the job, compromising the effectiveness of the hiring process to identify top-tier nursing talent. Hiring managers were also asked to identify potential barriers they have encountered to conducting successful interviews. The good news is that the majority of hiring managers (more than 80 percent) did not experience the barriers outlined in Figure 7. The most significant barrier reported involved hiring managers having few opportunities to conduct interviews since attending the training (reported by about 19 percent of hiring managers). What is most notable in this circumstance is that having trained managers who are not conducting interviews is certainly more beneficial than having untrained managers who are conducting interviews. Additionally, about 10 percent of hiring managers suggested their peers do not support or reinforce the interviewing skills that were learned, they do not have all of resources to conduct an effective interview, and that the TS interviewing process does not match their personal interaction style. Even though these ratings represent a small minority of survey participants, communicating the value of the TS interviewing process to the entire organization can help the process become more engrained in NYP culture and thus more supported on the whole. Furthermore, it is important to continually follow up with hiring managers to ensure they have the resources needed to conduct effective TS interviews. 6
Figure 7. Potential Barriers to Successful Interviews I have low personal motivation to follow the interview approach learned in training. This type of interviewing does not fit or align with my department culture (values, ways of interacting and doing business). 3.7 3.7 I'm not confident in my ability to conduct an interview and would like additional training. My department has not followed up on the training to make sure that the interviewing skills trained are being used. 7.4 7.4 My peers do not support or reinforce the use of the interviewing skills learned. I do not have all the resources (information, materials) I need to conduct an effective interview. This type of interviewing does not match my personal interaction style. 11.1 11.1 11.1 Since participating in the training, I have had few opportunities to conduct interviews. 18.5 0 5 10 15 20 Percent Identifying as a Barrier Quality of New Hires The Hiring Checkpoint SM study also evaluated the quality of hire from both new hires and hiring managers perspectives. New hires were asked to rate perceptions about their current job with NYP (Figure 8) while hiring managers were asked to compare the quality of new hires using the previous hiring process and the current hiring process (Figure 9). The majority of new hires (at least 75 percent) positively rated their current jobs with NYP. Specifically, they reported that NYP provides enough support and information for them to be effective and ramp up quickly in their jobs. Moreover, they report being engaged in their new jobs, indicated by high ratings in job satisfaction, a high sense of loyalty to NYP, and experiencing personal meaning and fulfillment in their work. Therefore, it appears the updated hiring process is effective in hiring and supporting engaged employees who would recommend employment at NewYork- Presbyterian to their friends and family. 7
Figure 8. New Hire Perceptions of Their Current Job NewYork-Presbyterian has provided enough support and information to help me get "up to speed" quickly in my job. NewYork-Presbyterian's values and priorities are consistent with my own values and priorities. 75.7 77.9 I feel a sense of loyalty to NewYork-Presbyterian. 83.5 I am satisfied with my job. 84.8 I find personal meaning and fulfillment in my work. I would recommend employment at NewYork- Presbyterian to my friends or family. I know enough about the facts processes and priorities of NewYork-Presbyterian to be effective in my job. Increased engagement reported by new hires speaks to the improvement in the quality of hires. Hiring managers were also asked to rate various aspects about the quality of new hires by comparing the current hiring process to the 84.8 85.8 87.4 0 20 40 60 80 100 Percent Rating Agree or Strongly Agree previous hiring process (before the NCB and TS were implemented). Figure 9 presents the percent of hiring managers rating the hiring process as excellent in each aspect of new hire quality. Figure 9. Percent of Hiring Managers Rating the System as Excellent Overall fit between abilities and job requirements Overall productivity Overall job performance Promotion/leadership potential Likelihood of staying with the organization 8.3 9.5 12.5 8.7 8.3 24.0 30.4 32.0 37.5 40.0 0 10 20 30 40 50 Previous System Current System 8 Development Dimensions International, Inc., MMVII. All rights reserved.
The current hiring process was rated substantially higher in every aspect of new hire quality. Specifically, the current hiring process was rated as more than three times better at hiring individuals with good leadership potential and more than four times better at hiring individuals who are more likely to stay with NYP. Hiring Process Efficiency New hires were asked to rate their overall experience with the NYP hiring process, including the hiring interview. Figure 10 presents the percent of new hires who agree or strongly agree with each positive aspect of the hiring experience. The majority of new hires (at least 60 percent) provided positive ratings regarding their NYP hiring experience. The most positive aspect of their hiring experience was with regard to the thoroughness of the interview. If applicants do not feel they have had the opportunity to communicate their entire set of relevant skills and experiences, they will be more likely to walk away with a negative perception of the experience and the organization. It is encouraging that more than three out of four applicants felt that they received enough information about the job and the organization during the hiring process. Providing applicants the opportunity to communicate their entire skill set as well as to learn about the job and organization will result in a more positive view of the organization. Additionally, providing applicants with a positive impression of the organization and a realistic job preview will have a significant impact on NYP s ability to recruit and hire top-tier nurses who are more likely to receive multiple job offers. As such, providing a positive reaction to the hiring process and solid understanding of the position will help ensure that NYP is considered as a good fit for the best nurses in the industry. Figure 10. New Hire Reactions to the Hiring Experience The hiring process left me with a positive impression of NewYork-Presbyterian. The recruiter took the time before the interview to help me feel comfortable and relaxed. I found out enough about the job and NewYork- Presbyterian to have a good idea of the opportunity before being hired. The hiring manager(s) took the time before the interview to help me feel comfortable and relaxed. The hiring process made a thorough assessment of my skills knowledge and abilities. The recruiter asked questions to make sure they truly understood my accomplishments in relation to the job. The hiring manager(s) asked questions to make sure they truly understood my accomplishments in relation to the job. 62.8 68.0 75.4 75.9 75.9 79.5 84.8 0 20 40 60 80 100 Percent Rating Agree or Strongly Agree 9
New hires were also asked to rate the NYP hiring experience compared to the hiring processes they have experienced with other organizations. Figure 11 presents the percent of new hires that rated aspects of NYP s hiring process as better or worse than hiring experiences of other organizations. Those that rated NYP s process as the same as other hiring experiences were not included in Figure 11. Figure 11. New Hires Experience at NYP Compared to Other Organizations The hiring manager(s) took the time before the interview to help me feel comfortable and relaxed. The recruiter took the time before the interview to help me feel comfortable and relaxed. The hiring manager(s) asked questions to make sure they truly understood my accomplishments in relation to the job. I found out enough about the job and the organization to have a good idea of the opportunity before being hired. The recruiter asked questions to make sure they truly understood my accomplishments in relation to the job. The hiring process made a thorough assessment of my skills knowledge and abilities. 25.8 25.4 16.4 25.8 25.4 30.8 13.8 30.8 22.4 32.8 31.3 38.8 The hiring process left me with a positive impression of the organization. 16.9 38.8 0 20 40 60 80 100 Percent Rating: NYP's was Worse NYP's was Better The largest differences between NYP s hiring process and other organizations were in NYP s ability to provide information about the job and organization and leaving a positive impression of the organization with job candidates. NYP s hiring process was rated lower in only one area hiring managers taking time before the interview to help applicants feel more comfortable and relaxed. These ratings may be related to the fact that the TS interviewing process is meant to be more structured and objective. However, it is still important for hiring managers to help applicants feel comfortable and relaxed so that they are able to provide adequate information during the interviewing process without being nervous. Hiring managers were also asked to compare hiring processes, albeit a different type of comparison. They were asked to compare the current hiring process (including the NCB and TS interview) to the previous hiring process, before the NCB and TS were implemented. Figure 12 presents the results of this comparison in critical aspects of effective hiring processes. 10
Figure 12. Percent of Hiring Managers Rating the System as High Efficiency of hiring system Guidance and support provided by the organization for evaluating candidates' qualifications Ability of the hiring process to address job fit Confidence in the legal defensibility of hiring decisions Overall ability to make good hiring decisions Objectivity of the hiring system Ability of the hiring process to address organizational fit Quality of candidates referred for interviews Ability of the hiring process to identify people with the right behavioral experiences and background 8.7 11.5 7.7 8.3 12.5 8.0 12.5 20.8 34.6 34.8 32.0 44.4 52.0 55.6 57.7 57.7 65.4 68.0 0 10 20 30 40 50 60 70 80 Previous System Current System The current NYP hiring process was rated higher in every critical aspect compared to the previous hiring process. Major improvements were seen in the process ability to assess both job and organizational fit. Ensuring that candidates have good job and organizational fit will help NYP retain top talent after they are hired. In fact, hiring managers suggest that fewer new hires leave the organization within the first 60 days using the current hiring process compared to the previous hiring process. Moreover, the most substantial improvement was seen in the current system s ability to identify people with the right behavioral experiences and background. In terms of helping to make good hiring decisions, the most critical aspect of any hiring process, the percent of hiring managers that rated the system as high went from 13 to 56 after the NCB and TS were implemented, suggesting that the current hiring process is more than four times better. When asked if they were satisfied with the current hiring process for nurses, the majority of hiring managers (54 percent) agreed or strongly agreed. 11 Development Dimensions International, Inc., MMVII. All rights reserved.
Return on Investment Based on the data that hiring managers provided in the Hiring Checkpoint SM survey, several estimates of return on investment were calculated. Specifically, savings-related reductions in the numbers of interviews conducted and amount of time to fill positions were calculated based on the salaries of hiring managers and new hires. To estimate the return on investment related to reductions in candidate interviews conducted, hiring managers were asked to estimate the number of candidates interviewed before finding a good hire both before and after the NCB and TS were implemented. Hiring managers indicated, on average, they interviewed 3.3 candidates per new nurse hire before changes were made to the hiring process. After changes were made, hiring managers indicated, on average, they interviewed 3.0 candidates per new nurse hire. Although the difference, 0.3 interviews, may not seem like much, substantial savings in hiring managers time were realized. Between November 2010, when the NCB and TS were implemented, and July 2012, when these estimates were gathered from hiring managers, 1,046 nurse positions had been filled. In total, a reduction of 0.3 interviews per nurse hire resulted in a reduction of 523 hours of hiring managers time to fill 1,046 nurse positions, considering that the average interview takes approximately 90 minutes (15 minutes to prepare, 60 minutes to conduct, and 15 minutes to review and provide ratings). Based on annual salary, a hiring managers hour is worth approximately $65. Therefore, the reduction in interviews conducted for 1,046 positions resulted in an estimated savings of $33,995 worth of hiring managers time. Table 1 presents the data related to the calculated savings in hiring managers time. Table 1: Return on Investment for Reduction in Number of Candidates Interviewed Estimated number of candidates interviewed per new hire Before System Changes After System Changes 3.3 3.0 Difference - 0.3 interviews * The average interview takes approximately 90 minutes. Number of interviewing hours needed per nurse hire 5 hours 4.5 hours - 0.5 hours * Since the system changes were in place (November 2012 to July 2012) 1,046 nurse positions had been filled. Total number of hours needed to conduct nurse interviews (for 1,046 positions) 5,230 hours 4,707 hours - 523 hours * Each hour of hiring managers time is worth approximately $65 based on annual salary. Total cost of hiring managers time conducting interviews $339,950 $305,955 - $33,995 12
To estimate the return on investment related to filling positions more quickly, hiring managers were asked to estimate the number of weeks a nurse position was left vacant both before and after the NCB and TS were implemented. Hiring managers estimated that positions were filled approximately 1.6 weeks sooner after the NCB and TS were implemented in October of 2010. Between November 2010 and July 2012 (when the estimates were gathered from hiring managers), 1,046 nurse positions had been filled. Filling those 1,046 positions 1.6 weeks sooner resulted in a reduction of overtime needed by 1,674 weeks (or 61,101 hours, as the average workweek for a nurse at NYP is 36.5 hours). To cover the time those positions were left vacant, current nurses were paid time and a half (an extra $22.15 per hour) to work overtime. If those positions were filled, NYP would have saved $22.15 per hour by not paying at the overtime rate. The reduction in weeks a position was left vacant resulted in 61,101 hours of overtime pay, resulting in an estimated savings of $1,353,387. Table 2 presents the data related to the calculated savings in overtime paid. Table 2: Return on Investment for Reduction in Overtime Paid for Vacant Positions Before System Changes After System Changes Difference Estimated number of weeks a position was left vacant 8.4 6.8-1.6 weeks * Since the system changes were in place (November 2012 to July 2012) 1,046 nurse positions had been filled. Total Number of weeks for position vacancies (for 1,046 positions) 8,786 7,112-1,674 weeks * The average workweek for an NYP nurse (and for vacant positions) is 36.5 hours. Total number of hours needed to be covered for vacancies (for 1,046 positions) 320,689 259,588-61,101 hours * Nurses are paid 1.5 times their normal hourly pay (an extra $22.15 per hour) when working overtime to cover vacancies. Cost for overtime coverage of vacant positions (extra $22.15 for 1,046 positions) $7,103,261 $5,749,874 - $1,353,387 13
Figure 13. Hiring Process Measurement Model Results CONTACT INFORMATION WORLD HEADQUARTERS 412.257.0600 E-MAIL INFO@DDIWORLD.COM WWW.DDIWORLD.COM/LOCATIONS This document was developed by DDI s Center for Applied Behavioral Research (CABER). For more information, contact CABER@ddiworld.com. CONCLUSIONS Overall, this study demonstrates that the introduction of the Nurse Career Battery and Targeted Selection into the hiring process for nurses has had a positive impact on the hiring process, individuals, and organization. Using the Hiring Process Measurement Model created, NYP was able to show the competencies identified in the Success Profile TM and targeted in the NCB and TS were critical to job success and the tools were implemented successfully, via the criterion-related validity study, TS training course reactions, and Hiring Checkpoint SM survey. New hires, via the Hiring Checkpoint SM survey, reported the hiring process left a positive impression and rated the NYP process higher than experiences with other organizations. Additionally, the Hiring Checkpoint SM survey showed the quality of hires more than doubled as a result of the NCB with regard to increased engagement, decreased turnover, and increased productivity. Hiring managers indicated the efficiency of the hiring process substantially increased. Specifically, hiring managers reported decreases in the number of applicants interviewed, resulting in an estimated savings of almost $40,000 in the amount of hiring manager time needed to conduct interviews since the NCB and TS were implemented. Moreover, hiring managers reported a decrease in the number of weeks it takes to fill nurse positions. This resulted in an estimated savings of nearly $1,400,000 in overtime that needed to be paid to cover vacancies. Overall, the results of this research prove that NYP did it right from start to finish. Specifically, creating Success Profiles TM and implementing the NCB and TS into the NYP nurse hiring process positively impacted the bottom line for the business according to NYP s recent hires, current nurses, and hiring managers. 14 TSRR14-0213