Job Analysis 101 Pamela Ing Stemmer, Ph.D. April 11, 2016 The certification program must have a job analysis that defines and analyzes domains and tasks related to the purpose of the credential, and a summary of the study must be published. (Institute for Credentialing Excellence, 2014) What is a job analysis? Why is it necessary? Who conducts a job analysis? When is one needed? What is involved in the job analysis process? What issues need to be considered? What? A job analysis (sometimes called a practice analysis or role delineation) is a scientific inquiry conducted in order to identify the critical tasks performed in a profession and the competencies (knowledge areas, skills, abilities, and other characteristics) required for performing the job successfully. When completed, the job analysis process yields test specifications that are supported by external validation evidence and that reflect the scope of practice, allowing for the development of a fair, accurate, and realistic assessment of an individual s competency in a particular profession (Figure 1). Job Analysis Test Specifica3ons Item Development Test Assembly Figure 1. The results from a job analysis are used to develop test specifications, which subsequently guide the development of item content and test assembly for an examination. Why? The purpose of a licensing or certification program is to demonstrate to the public that an individual with a credential has the requisite knowledge, skills, and abilities to perform competently in a given profession. To achieve this goal, a credentialing examination must establish a relationship with the occupation for which the credential may be earned (Raymond & Neustel, 2006). The results of a job analysis provide critical validity evidence for a certification or licensure examination. A job analysis is the primary mechanism by which a certifying body or regulatory board can confirm the accuracy and defensibility of an exam. It serves as the foundation of the licensure or certification examination and is critical to the success of the entire test development life cycle (Figure 2). 1
1. Job Analysis 9. Technical Report 2. Test Specifica3ons 8. Scoring and Repor3ng 3. Item Development 7. Test Administra3on 4. Test Assembly 6. Equa3ng 5. Standard Se@ng Figure 2. During the first stage of the test development life cycle, a job analysis is conducted. The results from the job analysis are used in the development of test specifications and provide the foundation for the rest of the test development life cycle. Who? A test sponsor generally hires a psychometrician to perform a job analysis for its testing program. The psychometrician will consult with the test sponsor to determine the most appropriate method for conducting the job analysis. The test sponsor is usually responsible for recruiting job incumbents to participate on a Job Analysis Advisory Panel. These subject matter experts (SMEs) are selected to represent a wide variety of job characteristics (such as years of experience, work setting, geographical location, and area of specialty), in order to develop a scope of practice that is reflective of the functions and responsibilities of the job and is relatively free from bias. When? A job analysis is typically performed every five years, in order to update the test blueprint to represent the current scope of practice. However, test sponsors need to exercise judgment and be prepared to conduct job analyses more frequently, when certain circumstances arise. For example, professions in which developments in research and/or technology can rapidly change the scope of practice may be better served by fewer years 2
between job analyses. The length of time between job analyses may also be reduced when, for example, new regulations or guidelines are published that directly affect the practice of the profession. Job Analysis Example While there are dozens of methods for conducting a job analysis, task inventory approaches are frequently implemented and provide appropriate structure for assessing a wide variety of professions (Raymond & Neustel, 2006). The following steps and associated deliverables are typically included in a job task analysis study: 1. Interviews with SMEs The client selects subject matter experts (SMEs), who are job incumbents, and who represent a variety of work- related characteristics and other demographics within the population, such as geographical location, years of experience, work setting, and area of expertise. The psychometrician conducts structured interviews with each of the SMEs to discuss tasks and competencies associated with the job, in order to define the scope of practice. 2. Development of Draft Task and Knowledge Statements The psychometrician aggregates and analyzes interviewee responses to develop a set of common themes, tasks, and knowledge areas. The tasks and knowledge areas are organized and classified into content domains. Task and knowledge statements are drafted for the job tasks and knowledge areas identified by the SMEs during the structured interviews. 3. Pre- Survey Meeting of Job Analysis Advisory Panel The client identifies SMEs to serve on a Job Analysis Advisory Panel. The psychometrician conducts an in- person focus group with the Panel to discuss and define the full scope of practice, write operational definitions for the content domains, refine the set of task and knowledge statements, and approve the set of task and knowledge statements to appear on the job analysis survey. 4. National Survey of Job Incumbents The psychometrician develops, administers, and monitors a national online survey questionnaire open to all job incumbents. The survey asks respondents to assess the criticality of the tasks and knowledge statements developed during the job analysis workshop. The survey collects relevant demographic and background information, as well as respondents ratings on the importance of each task and how often each task is performed. 5. Analysis of Survey Responses The psychometrician analyzes the responses from the national survey of job incumbents to determine the required proportion of examination items to be allocated to the tasks identified for each content domain. 6. Post- Survey Meeting of Job Analysis Advisory Panel The psychometrician conducts a virtual or in- person, post- survey meeting with members of the Job Analysis Advisory Panel (the same SMEs from Step 3). During the post- survey meeting, the psychometrician presents the results of the national survey to the Panel and facilitates a discussion involving the Panel s expert review and recommendations for the test blueprint and content outline. 3
7. Final Job Analysis Technical Report The psychometrician drafts a full technical report of the job analysis method, results, and resulting test specifications, including a test blueprint and content outline, with associated task and knowledge statements. Issues to Consider SME Recruitment. For national credentialing programs, when enlisting SMEs, it is important to consider the current composition of the population of practitioners. Generally, it is desirable to solicit participation from experienced job incumbents, but to include some newer incumbents as well. SMEs should represent a variety of work- related characteristics within the population, such as geographical location, work setting, and area of expertise. Demographic characteristics may also be relevant, particularly if a group has been historically underrepresented in a profession. In addition to diversity, it is imperative to select SMEs who do not have any conflicts of interest that would prevent them from participating in job analysis activities. SME Compensation. Will compensation be provided to SMEs? If so, what type of compensation will be provided? For example, continuing education units (CEUs) may be awarded to SMEs who serve on a job analysis advisory panel. Survey Recruitment. Elements of survey recruitment requiring consideration include target audience (respondents), methods of soliciting job incumbent participation, representativeness, diversity, incentives, media, etc. It is also important to establish exclusion criteria, to ensure that data collected from inappropriate respondents (e.g., not working in the appropriate state/country, do not have a specified minimum amount of experience in the position, are not current job incumbents) are not included in the main analysis of the job analysis survey responses. Survey Deployment. There are several issues related to survey deployment to consider, including: launch and end dates; administration time (i.e., how long will it take to complete the survey); possibility and length of extensions to survey response collection; and coordination with other relevant dates, such as conferences, holidays, etc. Survey Design. It is important to assess a consultant's expertise with survey design, paying particular attention to content and rating scales. Discuss the structure and format of the survey with the consultant, in order to ensure that the appropriate methods will be implemented. In other words, decide what type of results you want to have (e.g., criticality ratings, risk of adverse events) and ensure that the design of the survey allows for the appropriate data to be collected. Additionally, don't waste survey space on information that is not informative (e.g., omit knowledge statements if all are determined by SMEs to be equally and highly important). 4
Testing Standards Endorsement Use of a job analysis as a method of defining examination content is clearly endorsed by the National Commission for Certifying Agencies (NCCA) 2016 Standards for the Accreditation of Certification Programs (Institute for Credentialing Excellence, 2014): Standard 13 The certification program must use panels of qualified subject- matter experts (SMEs) to provide insight and guidance and to participate in job analysis, standard setting, and other examination development activities. Standard 14 The certification program must have a job analysis that defines and analyzes domains and tasks related to the purpose of the credential, and a summary of the study must be published. and the 2014 Joint Standards for Educational and Psychological Testing (AERA/APA/NCME, 2014): Standards 4.12 Test developers should document the extent to which the content domain of a test represents the domain defined in the test specifications. Standards 11.03 When test content is a primary source of validity evidence in support of the interpretation for the use of a test for employment decisions or credentialing, a close link between test content and the job or professional/occupational requirements should be demonstrated. Standards 11.13 The content domain to be covered by a credentialing test should be defined clearly and justified in terms of the importance of the content for credential- worthy performance in an occupation or profession. A rationale and evidence should be provided to support the claim that the knowledge or skills being assessed are required for credential- worthy performance in that occupation and are consistent with the purpose for which the credentialing program was instituted. A job analysis is the primary method by which a certification or regulatory board can establish the accuracy and defensibility of an examination. It serves as the foundation for a certification or licensure examination and is critical to the success of the entire examination development process. It is essential to keep a test blueprint updated and representative of a profession s current scope of practice and thus, to perform a job analysis every five years, or more frequently, if necessary. If you would like a free one- hour consultation on a job analysis or another component of your testing program, please use the following link to contact Comira s Psychometric Team. 5
References American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Downing, S. M. (2006). Twelve steps for effective test development. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 3-25). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Institute for Credentialing Excellence. (2014). National Commission for Certifying Agencies Standards for the Accreditation of Certification Programs. Washington, DC: Author. Raymond, M. R. & Neustel, S. (2006). Determining the content of credentialing examinations. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 181-223). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. 6