Evaluating Interview Data

Transcription

1 Evaluating Interview Data After you and the other interviewers have completed the interviews, the interviewers attention shifts to organizing and evaluating the collected data. At this point, you will see how the effort you put into collecting specific STARs will make the decision-making process consistent, manageable, and accurate. The next step in the selection process consists of two parts: evaluate the data and integrate data. Evaluate the Data Working independently of the other interviewers, look at the STARs you ve collected for each target to make sure they are complete and in the right target. Next, determine whether each STAR is effective or ineffective, and then determine the weight of each STAR by considering its similarity to the job for which the person is being considered, its impact, and how recently it occurred (that is, its recency). You then rate each target. Integrate the Data In a data integration meeting attended by all the interviewers, each interviewer posts his or her target ratings on a Rating Grid. Then, everyone discusses and exchanges data to reach consensus ratings for each target. Based on the consensus ratings, the group can then make a decision to retain or reject the candidate. How to Evaluate the Data As you sort through the data you ve collected, check other places besides the Planned Behavioral Questions section for appropriate STARs for particular targets, such as: Key Background Review section of the interview guide. STARs often are found in the comments a candidate makes early in the interview as he or she reviews significant experiences. Interview Close. Any additional questions you ask, questions the candidate asks about the position and organization, and information the candidate volunteers at the end of the interview can be excellent sources of STARs. Development Dimensions International, Inc.,

2 The candidate s application, resume, curriculum vitae, or other sources. Look for information that was not covered in the interview but could provide behavioral information. Application forms often call for information about certifications, licenses, training, and educational qualifications. Chances are good that the application form asks for job-specific information that will make a relevant STAR or two. (Although these topics usually should be followed up during the interview, time constraints might have prevented this.) Ask Yourself: Is the STAR Effective or Ineffective? As you review the interview guide and the supplementary sources, indicate the information that makes up a complete STAR. Although you won t discard partial STARs, complete STARs are the best predictors of future performance. As you review each STAR, ensure that it is in the appropriate target by comparing it to the target definitions and the key actions. In the case of Motivational Fit, compare the STAR to the definition and facets. If you find that a STAR is not in the right target, make a note to indicate that the STAR should be included in a more appropriate target. Note: You might not be able to categorize every example. Some STARs won t fit any of the targets related to the job. You may disregard any irrelevant STARs. Complete STARs can demonstrate either positive or negative behaviors. Use the target s key actions and the Result part of the STAR to determine whether it s an effective or ineffective example. Your task is to decide if the candidate s use (or lack of use) of the key actions was appropriate or inappropriate, given the situation. The Result part of the STAR usually will indicate the effectiveness of the candidate s actions. In most instances, the connection between an effective action and a positive result is obvious; at other times, it might not be so clear. It s possible for a candidate to take effective action that has a negative outcome. For example, a candidate might do a good job of selling something gathering information or demonstrating product capability but be ineffective at building productive customer relationships over the long term. In this case, the candidate demonstrated effective Sales Ability/Persuasiveness but was ineffective in Building Customer Loyalty. Also, what other people do might contribute to a negative result. Although the candidate did everything right, other people s incorrect actions could cause a negative result. Development Dimensions International, Inc.,

3 To evaluate STARs within the Motivational Fit target, compare the STAR to the facets. For instance: If the STAR illustrates a facet that there are many opportunities for in the job or organization, and the person s example indicates that he or she was satisfied with this facet in past jobs, this is a positive example of motivational fit. If the STAR illustrates a facet that there are many opportunities for in the job or organization, and the example indicates that he or she was dissatisfied in this situation in past jobs, this is a negative example of motivational fit. If the STAR illustrates a facet that there are few or no opportunities for in the job or organization, and the example indicates the person is highly motivated by the facet, there is a mismatch. Weigh the Significance of Each STAR Not all STARs are equal; some are more significant than others. For instance, I saved three people from drowning when I was a lifeguard has more weight for someone applying to be a scuba instructor than for someone applying for an accounting job. Likewise, a STAR based on an event 11 years ago, no matter how outstanding, is not as good a predictor of future events as a STAR describing a work-related activity that took place two months ago. The following factors give a STAR its total weight that is, its ability to accurately predict future performance: Similarity to the Job The more similar the STARs are to the job, the better they will predict how a candidate will perform in the position. Example: A candidate for a team leader position gives a STAR describing excellent leadership skills when she was the leader of a club. In another STAR the candidate describes poor leadership when leading a team in her job. The team leader STAR should be given more weight because it is more similar to the job. Example: A candidate for a position that requires attention to detail describes a situation in which she missed important details and admits that she doesn t like sweating the details. This example should be given considerable weight in determining the candidate s motivation for the job. Development Dimensions International, Inc.,

4 Don t disqualify a STAR if it s not a perfect match. Sometimes you need to look beyond the surface dissimilarities to what the situation (or action) in the STAR has in common with the job. People with little workplace experience or with experience in a different field can still provide relevant STARs. At the same time, you need to give more weight to a good match than you do to a poor one. Impact of the Behavior To determine the impact of the behavior, consider both the situation and the result. Behaviors that make a big difference in meaningful situations have more weight than behaviors that make a small difference in trivial situations. Example: In one STAR a candidate describes her decision to discontinue a product line; in another STAR the candidate describes her decision to stop sending out memos on brightly colored paper. The first STAR should be given more weight in determining a target rating because it had more impact. Example: A candidate for a position that requires attention to detail describes having some inaccurate cost information in a presentation he made to senior leaders at a company sales conference. He also describes typos in an he sent to his team. The first STAR has greater impact. Recency of the Behavior Recent behavior is a better predictor of future behavior than behavior that occurred in the distant past. Example: A candidate provides examples of ineffective behavior in a job she held 10 years ago and examples of effective behavior for the same target in recent jobs. The more recent STARs should be given more weight. Example: A candidate describes not liking business travel several years ago when he was working and attending college at the same time. He describes enjoying travel more now. The recent behavior should be given more weight. Development Dimensions International, Inc.,

5 Rate Each Target Your goal when rating targets is to use the data you ve collected to predict the candidate s future performance in the job. For each of your assigned targets, assess the extent to which the candidate demonstrated effective use of key actions. In your assessment consider both the quality and quantity of the key actions demonstrated. To rate Motivational Fit, consider the intensity of the candidate s likes or dislikes related to the facets. For example, consider the level of annoyance (i.e., from minor displeasure to major irritant). Most people will tolerate a minor annoyance if other indicators of good job fit or organization fit offset it significantly. Then ask yourself: Will this candidate enjoy the tasks associated with the job? The environment associated with the organization? Rating Scale Rating scales can vary, and your organization might have a predefined scale. Targeted Selection uses a five-point numerical rating scale to represent the different levels of candidate qualifications. These ratings, the common language of evaluation in the Targeted Selection system, are: 5 Much More Than Acceptable Significantly exceeds criteria for successful job performance/motivational fit. 4 More Than Acceptable Exceeds criteria for successful job performance/ motivational fit. 3 Acceptable Meets criteria for successful job performance/motivational fit. 2 Less Than Acceptable Generally does not meet criteria for successful job performance/motivational fit. 1 Much Less Than Acceptable Significantly below criteria required for successful job performance/motivational fit. When rating targets, a 3 (Acceptable) rating is the standard for successful job performance/motivational fit. It does not mean average. Rating a candidate Acceptable in every target predicts effective performance in the job. Development Dimensions International, Inc.,

6 Guidelines for Rating Targets Do s Carefully review each target to ensure that you understand its definition and key actions. For Motivational Fit, review each facet s definition. Compare the STARs you ve categorized under a target to its key actions or, for Motivational Fit, to its facets. Individual STARs might include examples for more than one key action or facet. Emphasize the STARs to which you ve assigned greater weight (similarity to the job, impact, and recency of the behavior) when determining the target s rating. Review each STAR and determine the extent to which the behavioral information predicts the candidate s future behavior in this target in the job. Some interviewers find it useful to ask themselves a final outcome question. For example: For Planning and Organizing, ask yourself, Will the person like to plan but not carry through on the project or not follow up on any progress? For Motivational Fit, ask yourself, Will the person be happy with a job that requires high attention to detail? or Will the person be comfortable working in an organization with a strong emphasis on results and the bottom line? Also consider the intensity of the candidate s likes or dislikes related to the facet. Compare relationships among key actions to determine if negative examples for one key action nullify positive examples for others. These relationships can indicate that a candidate lacks the required skills to achieve an acceptable rating in the target. For example, a candidate who always acts positively about change and new situations, but takes no steps to adapt his or her behavior in response to the change, might not merit an acceptable rating in Adaptability. Don ts Do not determine ratings for targets based on whether you collected examples for all the key actions or facets. Time limitations in the interview might prevent you from obtaining examples for every key action or facet in every target. For instance, if you have examples for only two key actions in one target and for five key actions in another target, review the quality of the examples. It s possible that the two examples in the first target are more meaningful in relation to the job than the five examples in the second target. Do not base your ratings on how many examples you ve collected. Use the quality and weight of the examples you have to determine ratings. Development Dimensions International, Inc.,

7 Clarifying Ratings You might find these additional notations useful in describing behavior: N No opportunity to observe or make an assessment An N rating indicates you did not collect data on this target because: You didn t have time. The candidate didn t have an opportunity to demonstrate behavior in this target. I Incomplete data The I rating, which is always coupled with a number rating (e.g., 2I, 3I, 4I), is used to indicate a tentative judgment when: You didn t get enough STARs to fully support your rating. You want to indicate your caution that the rating is based on less behavioral information than you would like. 5H Candidate is too high in this target A 5H rating indicates that, for a particular job: The candidate is too strong in a target; that is, he or she exhibits so much of a characteristic that it could cause ineffective performance or other problems. What might be a strength in other circumstances might be an area for improvement for the job. For example, a candidate who often missed deadlines because he insisted on rechecking details for a fourth or fifth time might be rated 5H in the Quality Orientation target. In this case his Quality Orientation was too much of a good thing. The 5H rating is not used for Motivational Fit. If a candidate has strong motivation for facets that are not present in the job or the organization, you would indicate this by giving a Less Than Acceptable rating in Motivational Fit. Development Dimensions International, Inc.,