U.S. ARMY FUTURE FORCE SELECTION AND CLASSIFICATION RESEARCH

Transcription

1 U.S. ARMY FUTURE FORCE SELECTION AND CLASSIFICATION RESEARCH Kimberly S. Owens, Richard R. Hoffman III U.S. Army Research Institute for the Behavioral and Social Sciences Arlington, VA, USA Deirdre J. Knapp Human Resources Research Organization Alexandria, VA, USA The U.S. Army currently is undertaking fundamental changes to transform into the Future Force to face 21 st century challenges. This transformation to Brigade Combat Teams (BCTs) requires personnel with a different mix of knowledges, skills, and attributes (KSAs). Some jobs or military occupational specialties (MOS) are in high demand, while others are being de-emphasized, and still others are undefined. The reorganization places a greater emphasis on having the right Soldier in the right job. Therefore, the Army Research Institute (ARI) is in the process of a 10-year selection and classification research program, composed of two successive 5-year projects. The primary objective of this program is to develop and validate tools to assist the Army in selecting and classifying enlisted Soldiers. Initial analyses have been completed for the concurrent validation of a selection research project (known as Select21). 1 The follow -on classification research project (known as Army Classification) is underway with a 1-year concurrent validation to be followed by a 4-year longitudinal validation. This research will be described, including a high level summary of some of the results from Select21, as well as the research plan for the Army Classification project. Finally, potential applications of the research for the Army will be discussed. A FUTURE-ORIENTED APPROACH Since the Army is in the process of its transformation effort, an important objective of the Select21 project was to develop new tools to help ensure that new Soldiers have the KSAs needed to perform the duty requirements that emerge as part of the transformation. Special care has been taken to ensure that the predictor measures and performance requirements generated from this research would apply not only to the Current Force, but also to the Future Force as well. In order to meet these objectives, we determined that we needed to conduct an extensive analysis of the changes forecasted for the Army. The first step in our future-oriented approach was a comprehensive review of relevant literature including Army doctrine, Future Force planning documents, and prior 1 This paper describes a research project sponsored by ARI and contracted to HumRRO. A large team of researchers led by Trueman Tremble, ARI and Deirdre Knapp, HumRRO contributed to the effort. The views, opinions, and findings contained in this paper are solely those of the authors and should not be construed as an official U. S. Department of the Army or U.S. Department of Defense position, policy, or decision unless so designated by other documentation.

2 research. Once integrated, this information provided descriptions of forecasted changes to doctrine, missions, technologies, training environments and battlefields. The integrated review provided a meaningful starting point for forecasting future performance requirements of all first-term Soldiers. The research team projected future entry-level job requirements in three complementary ways: (a) conditions under which future Soldiers will work (e.g., future Soldiers are expected to take greater responsibility in for their own training and development) (b) broad dimensions potentially descriptive of job performance of Soldiers in any future job or MOS (e.g., adapts to changing situations, demonstrates teamwork) and (c) common technical task requirements (Sager, Russell, Campbell & Ford, 2005). These performance requirements served two major purposes for the Select21 research program. First, they served as a basis for inferring the relevant pre-enlistment KSAs that the Army should look for when considering applicants for enlistment. Second, researchers used the performance requirements to develop job performance measures that were used as criteria to evaluate (or validate ) the experimental pre-enlistment tests. This development work yielded the predictor and criterion measures used in Select21 and Army Classification. PREDICTOR MEASURES A fundamental goal of the selection and classification initiative is to develop measures that predict entry-level Soldier performance and add incremental validity over the current system as embodied by the Armed Services Vocational Aptitude Battery (ASVAB). Prior research has shown that the ASVAB is a psychometrically strong measure of cognitive aptitude and is an effective predictor of job performance (Campbell & Knapp, 2001). It is expected that the experimental non-cognitive predictors developed for this research will predict the more motivational aspects of performance and turnover and therefore, provide incremental validity over the ASVAB. In Select21, we developed eleven predictor measures. For the purposes of this discussion, we will focus on three predictor measures that highlight the motivation, temperament, and person-environment fit aspect of the research. See Table 1 for a brief description of the measures. Table 1. Predictor Measures Title Description Rational Biodata Inventory (RBI) Work Preferences Survey (WPS) Work Values Inventory (WVI) Assesses motivational and temperament characteristics important for entrylevel Soldiers by asking about past behaviors and reactions to previous life events. The RBI generates several sub-scores for various constructs such as Achievement Motivation, Cognitive Flexibility, Peer Leadership, Hostility to Authority, and Cultural Tolerance. A person-environment fit measure that assesses respondents work related interests based on Holland s (1985) occupation type model. A sorting task in which respondents rank order work characteristics (e.g., autonomy, physical activity) that are important to them. The characteristics in the WVI are derived from multiple research sources examining personal values (e.g., Dawis & Lofquist, 1984; Sackett & Mavor, 2002).

3 Particularly when trying to implement non-cognitive self-report measures in an operational setting, a concern is individuals faking or otherwise distorting (whether intentionally or unintentionally) their responses to present themselves in a more positive manner. This problem is one plausible explanation as to why predictor measures sometimes predict performance better in a research setting than they do in a operational setting (Knapp, Waters, & Heggestad, 2001). Therefore, in the Select21 research, we sought to minimize this problem. For example, the RBI contains a response distortion scale. This scale can be employed to identify respondents whose scores on other RBI scales appear to be distorted; it also can be used during item development to identify particular items that seem prone to faking. CRITERION MEASURES An important aspect of the Select21 project was inclusion of measures of both job performance and attitudinal criteria. We drew on prior research about job performance and attitudes to build a set of criterion measures that would tap both domains using a variety of measurement methods including attitude surveys, peer and supervisor ratings, a job knowledge test, a criterion situational judgment test, and personnel records. This was a significant step toward more complete coverage of criteria that are important to the Army. Performance Criteria We conducted modeling exercises using scores on the performance criterion measures and identified the five job performance factors presented in Table 2. These performance factors are quite similar to those found in Project A (Campbell & Knapp, 2001). All of the performance composites demonstrated adequate discriminant validity, and most appeared to be reasonably reliable. Table 2. Performance Factors Title General Technical Proficiency Achievement and Effort Physical Fitness Teamwork Future Expected Performance Description Based on the Army-Wide Job Knowledge Test (AWJKT) score, the Weapons Qualification score, and peer and supervisor ratings of Common Task Performance, MOS-Specific Task Performance, Communication, Information Management, Problem Solving, and Adaptation Based on prior military education and disciplinary actions, the Criterion Situational Judgment Test (CSJT) score, and peer and supervisor ratings of Effort and Initiative, Professionalism/Personal Discipline, and Personal/Professional Development. Based on the Army Physical Fitness Test (APFT) score and peer and supervisor ratings of Physical Fitness Based on peer and supervisor ratings of Supports Peers and Exhibits Tolerance rating scales Based on peer and supervisor ratings of expected performance in four different anticipated future conditions: Individual Pace and Intensity, Learning Environment, Disciplined Initiate, and Communication Method and Frequency.

4 Attitudinal Criteria There were a large number of scale scores yielded by the Army Life Survey (ALS) and Future Army Life Survey (FALS). Empirical approaches did not prove useful for reducing the attitudinal criterion space. Accordingly, we used a rational approach to select a subset of the scales for predictor validation analyses. We chose scales to meet two primary objectives: (a) representation of current and future-oriented constructs, (b) balance in terms of proximity of the chosen scales to the Select21 predictors and actual attrition and re-enlistment behavior. Toward those ends, we selected five attitudinal scales on which to focus for the validation effort. Descriptions of the scales are presented in Table 3. Table 3. Attitudinal Criterion Measures Title Satisfaction with the Army Perceived Army Fit Attrition Cognitions Career Intentions Future Army Affect Description a 10-item scale from the Army Life Survey (ALS) that gets at the Soldiers satisfaction with Army life in general a 6-item scale from the ALS that assesses how well Soldiers perceive themselves in fitting in the Army in general a 3-item scale from the ALS assessing the degree to which Soldiers have thought of leaving the Army a 5-item scale from the ALS assessing Soldiers intentions to re-enlist and make the Army a career a 5-item scale from the Future Army Life Survey (FALS) assessing the extent to which Soldiers have positive feelings about expected future Army conditions All attitudinal scales exhibited sufficient levels of variance and had acceptable levels of internal consistency. Correlations among scales were moderate, suggesting that scales were not overly redundant with one another. CONCURRENT AND LONGITUDINAL VALIDATION RESEARCH In addition to developing predictor and criterion measures, one of the main objectives of the research program is to evaluate the measures with validation samples of new recruits and first-term Soldiers. To date, we have completed the Select21 concurrent validation and the Army Classification concurrent validation is in process. Next year, we will begin a longitudinal validation for the Army Classification project. In the following sections, we provide descriptions of the three validation projects. Select21 Concurrent Validation The Select21 concurrent validation data collection took place in Participants consisted of 812 first-term enlisted Soldiers who had been in the Army between 12 to 36 months. Soldiers completed all measures in a single day with some measures completed via the computer and others completed via pencil and paper. In addition these Soldiers Supervisors were asked to participate and provide Soldier performance ratings. A total of 388 Supervisors participated.

5 Validity analyses have been completed recently. In general, the RBI, WPS, and WVI showed promise as selection instruments with significant relationships with both performance and attitudinal criteria. Validity coefficient ranges are presented in parentheses. Many RBI subscales predicted performance including General Technical Proficiency (.10 to.22), Achievement and Effort (.09 to.29) and Physical Fitness (.10 to.33). Several RBI subscales also predicted attitudes such as Satisfaction in the Army (.08 to.32), Perceptions of Fit in the Army (.20 to.40), Career Intentions (.08 to.25), and Attrition Cognitions (.10 to.22). While the WPS and WVI both predicted the performance composites Achievement and Effort (WPS =.14, WVI =.16) and Physical Fitness (WPS =.11, WVI =.15), they had stronger relationships with the attitudinal criteria (WPS =.19 to.29, WVI =.20 to.42). These predictors also demonstrated statistically significant increments in validity when compared to the ASVAB, especially with attitudinal measures. In particular, the RBI, WVI, and WPS yielded significant incremental validities of.20 or more for predicting Satisfaction in the Army and Perceptions of Fit in the Army. With regard to performance measures, the RBI, WVI, and WPS all added incremental validity to predicting Achievement and Effort with the R ranging from.15 for the RBI to.04 for the WPS. Thus, the initial results offer support for the hypothesis that non-cognitive predictors can supplement the ASVAB for attitudinal criterion measures and to a lesser extent, performance criterion measures. Army Classification Project The focus of the Army Classification research program is job classification, which is the assignment of new recruits into a particular MOS. Entry-level Soldiers must be placed in jobs that best emphasize their existing KSAs, interests, and potential. More comprehensive assessment of new recruits may improve classification into Army positions and result in valued Army outcomes (e.g. improved performance, increased satisfaction, and increased retention). Hence, the Army is interested in conducting research to develop and validate assessment tools to assist with new recruit job classification. There are two large data collections planned to support this classification research effort a concurrent validation and a longitudinal validation. Each of the validation projects are described below: Army Classification Concurrent Validation Concurrent validation data collection began in April 2006 and continues through November. The sample includes primarily E2-E3 Privates from four MOS. The instruments included in the predictor test battery are: the Work Suitability Inventory, the Work Values Inventory, the Work Preferences Survey, the Rational Biodata Inventory, and the Predictor Situational Judgment Test. All of the measures are computer-based and are administered on laptop computers. The total administration time for the predictor and criterion measures is about 4 hours. Once the data collection is completed, we will conduct basic psychometric analyses for each measure and validity analyses to assess the classification usefulness of the measures. Analysis of results should be completed early in The concurrent validation data analysis will inform decisions about any changes needed to the instruments before the longitudinal validation.

6 Army Classification Longitudinal Validation The timeline for the longitudinal validation is about 4 years. We will begin with predictor administration in 2007 and about 2 years later, we will collect criterion data for Soldiers who participated in the predictor data collection. In the final year of the project, we will conduct the longitudinal criterion-related validation analyses. In the first phase of the longitudinal validation, we will collect predictor data. We plan to test several thousand applicants or new recruits entering the Army during a 9-12 month data collection period. We are exploring the possibility of administering the predictor measures to applicants at Military Entrance Processing Stations (MEPS), the centers at which the services test and screen candidates for eligibility in service. If it is not viable to collect data at the MEPS, the data collection will occur with new recruits at reception battalions. The classification predictor measures will encompass a range of constructs (e.g., cognitive abilities, work values, temperament, vocational interests, preenlistment training and experience). The predictor test battery may include the Work Values Inventory, the Work Preferences Survey, the Rational Biodata Inventory, and other non-cognitive predictor measures depending upon the results of the concurrent validation as well as the parameter data collections. Data analyses will include psychometric analyses of the predictor measures. In the second phase of the validation criterion data will be collected from the Soldiers who participated in the predictor data collection. The data collection will commence about 2.5 years after the predictor data collection has ended. The criterion measures may include: job knowledge tests, performance ratings, the Army Life survey, and the Criterion Situational Judgment Test. We will conduct psychometric analyses for each criterion measure and classification validation analyses of the predictor and criterion data. Analysis of the results should be completed by the end of RESEARCH ISSUES One on-going issue ARI faces in selection and classification research is that the Army has approximately 175 different job types or MOS. Such a large number makes it difficult from a time and cost perspective to determine job requirements and performance dimensions for each job type as well as to generalize research findings. Therefore, as part of the job analysis performed in Select21, an attempt was made to develop MOS clusters so that similar MOS could be grouped together for analysis purposes. The expectation was that developing a broader unit of analysis would make it easier to generalize job tasks, job requirements, and research findings across similar MOS. However, using the clusters proved to be untenable because the heterogeneity of job requirements within clusters proved to be as great as the heterogeneity across MOS clusters. The large number of MOS continues to be a challenge in ARI s selection and classification research because it is difficult to get the detailed evidence needed to develop the selection rules (e.g., how to weight scores from different pre-enlistment measures) specific to each MOS. Currently, ARI is working with a scientific review panel as to how to address this issue. Another on-going issue is participant attrition during longitudinal research. It is not unusual for longitudinal data collections to have problems finding and recruiting participants after the first data collection. However, for longitudinal validation efforts to work this problem needs to be solved or at least managed. One of our strategies for

7 handling participant attrition is to include a much larger number of participants during the first data collection so that the loss across data collection events does not drive the sample size below a predetermined cutoff. Additionally, we will seek to encourage participants commitment by maintaining contact and gathering additional data from them at regular intervals. Another concern is the mobility of the participants and how we will keep track of them. Because our participants are new Soldiers, they will be particularly mobile. We will need to remain in contact with them from the reception battalion through initial entry training and subsequently through their first tour job assignment at their unit. Additionally, Soldiers will be widely spread out among units. We will work with the Army G-1 Personnel department, the reception battalions, and the schools to identify ways to track Soldiers; however, at this point no specific solutions have been identified. POTENTIAL APPLICATIONS The primary goal of this research is develop non-cognitive measures that can be used operationally in the selection of qualified entry-level Soldiers and their subsequent classification into appropriate MOS. However, several measures used in the research project have other potential applications for use in the Army as well. For example, the RBI already has been used in an operational setting in the Special Forces selection process. Job knowledge tests could play an important role as the Army continues its Future Force Transformation. Because of this transformation, many Soldiers will be reclassified into a different MOS, and these knowledge tests could help the Army determine in what areas re-classified Soldiers have sufficient knowledge and in what areas they would need additional training. Currently, research is being conducted as to the feasibility and content of re-classification job knowledge tests. Using non-cognitive predictors coupled with the ASVAB as a means to tailor training for entry level Soldiers is another potential operational use, and there are plans to conduct research to assess this possibility. Finally, with regard to the performance rating scales, several officers and NCOs in the field have requested copies and have shown interest in employing the rating scales at the unit level. REFERENCES Campbell, J. P., & Knapp, D.J. (Eds.) (2001). Exploring the limits in personnel selection Classification. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Dawis, R.V., & Lofquist, L. H. (1984). A psychological theory of work adjustment. Minneapolis: University of Minnesota Press. Knapp, D.J. & Trueman, T.R. (Eds.) (2006). Concurrent Validation of Experimental Army Enlisted Personnel Selection and Classification Measures. Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. Holland, J.L. (1985). Making vocational choices: A theory of vocational personalities and work environments (2 nd ed.). Upper Saddle River, NJ: Prentice Hall.

8 Sackett, P.R., & Mavor, A. (Eds.) (2002). Attitudes, aptitudes, and aspirations of American Youth: Implications for military recruitment. Washington, D.C.: National Academies Press. Sager, C.E., Russell, T.L, Campbell, R.C. & Ford, L.A. (2005). Future Soldiers: Analysis of Entry-Level Performance Requirements and Their Predictors. (Technical Report 1169). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.