How to Develop and Administer a Job Analysis Survey. Presented by Dr. Jim Higgins Executive Director, BCGi

Transcription

1 How to Develop and Administer a Job Analysis Survey Presented by Dr. Jim Higgins Executive Director, BCGi

2 Biddle Consulting Group Institute for Workforce Development BCGi Memberships (free): ~7000+ members / 15,000 HRCI credits to-date Online community Monthly webinars on EEO compliance topics EEO Insight Journal (e-copy) BCGi Platinum Membership (paid) Includes validation/compensation analysis books EEO Tools including those needed to conduct AI analyses EEO Insight Journal (e-copy and hardcopy) Access to the BCGi library of webinars, training materials, and much more

3 A few housekeeping items: Webinar slides have been sent out, if you didn t receive them in your please follow up with us at bcgi@biddle.com HRCI & SHRM credits are no longer offered for free through BCGi, to get the credits you must register through both the BCGi website and GoToWebinar. The cost is $99 and you receive both credits, if you meet their requirements. Webinars are still offered for free! This webinar, along with all of our previously recorded webinars, is accessible to BCGi Platinum Members on our website.

4 How to Develop and Administer a Job Analysis Survey Dr. Jim Higgins, Ed.D. Executive Director BCGi

5 Definition of Job Analysis A detailed statement of work behaviors and other information relevant to the job (Uniform Guidelines on Employee Selection) A purposeful, systematic process for collecting information on the important work-related aspects of the job (Gatewood & Field, 2001)

6 Why Conduct a Job Analysis? Meet legal and professional standards for employment Select the best qualified candidates Use Job Analysis to make employment and management decisions

7 The Important Uses for a Job Analysis Testing Recruiting Screening Job Analysis Promoting Appeals Evaluating

8 Legal Background Title VII, Equal Employment Opportunity, of the Civil Rights Act of 1964 states in part: SEC (a) It shall be an unlawful employment practice for an employer: o to exercise any employment-related decisions resulting in discrimination due to an individual s or group s race, color, religion, sex, or national origin and to use a selection instrument that results in adverse impact unless the employer can demonstrate the jobrelatedness of the procedure

9 Case Law Griggs vs. Duke Power Company (1971) Ruled that selection instruments had to be demonstrably related to job performance (i.e., job-related) Found that the absence of the intent to discriminate does not justify use of selection procedures that resulted in discrimination Established that companies could not use artificial, arbitrary, and unnecessary obstacles to employment that serve as discriminatory barriers

10 Case Law Albemarle Paper Company vs. Moody (1975) Ruled that employment tests must be predictive of or significantly correlated with important elements of work behavior which comprise or are relevant to the job or jobs which candidates are being evaluated

11 Case Law Kirkland vs. New York State Department of Correctional Services (1983) Established that identifying critical tasks and skills as well as the competency required to fulfill the various aspects of the position is an essential part of job analyses Ruled that the foundation of a content valid examination is the job analysis

12 Uniform Guidelines on Employee Selection Procedures Requirements (1978) Adopted by the Department of Labor, the Equal Employment Opportunity Commission, the Civil Service Commission, and the Department of Justice Promotes a uniform set of principles to help employers comply with Federal laws that prohibit discrimination Establishing guidelines for the proper validation and use of selection procedures

13 Standards for Educational and Psychological Testing (1999) Prepared by a joint committee (American Psychological Association, American Educational Research Association and the National Council on Measurement in Education) Provides assessment professionals with guidelines for the evaluation, development and use of testing instruments When evidence of validity based on test content is a primary source of validity evidence in support of the use of a test in selection or promotion, a close link between test content and job content should be demonstrated.

14 Americans with Disabilities Act (ADA) The ADA requires that employers evaluate candidate capabilities in relation to their ability to perform the essential functions of the job and prohibits discrimination against individuals with physical or mental disabilities

15 Validity The extent to which a test measures what it purports to measure Content Validity: extent to which a measurement reflects the specific intended domain of content Job Analysis is a process for establishing Content Validity!

16 Job Analysis Process: 1. Determine the scope of the job analysis 2. Determine the procedures 3. Draft preliminary Task and KSAPC statements based on archival research 4. Identify subject matter experts (SMEs) 5. Conduct job audits 6. Edit previously drafted Task and KSAPC statements

17 Job Analysis Process: 7. Finalize Task and KSAPC statements 8. Organize the Task and KSAPC statements into logical content areas 9. Obtain SME ratings regarding Task and KSAPC statements 10. Link critical Task and KSAPC statements 11. Select employment selection method(s) 12. Create an examination outline 13. Document processes in a job analysis report

18 Job Analysis Process - Overview JOB ANALYSIS The Job TASKS Task/KSA Links KSAs Test Content/KSA Links TEST CONTENT

19 Collecting Job Analysis Information Subject Matter Experts (our best resource) Archival Research Job Audits / Interviews Job Observations Committee-Based Data Collection

20 Definition of Subject Matter Experts Subject Matter Experts (SMEs) are expert-level incumbents and supervisors, knowledgeable about the duties and requirements of the job. SMEs should be top performers in the job, first-level supervisors or hiring managers SMEs have direct knowledge of the job and know what is required to succeed in the job

21 Subject Matter Experts Gather a representative sample: Gender Race Age Length of time at job Work location (e.g., field office) Different divisions or work units Shift work, i.e., morning, swing, evening (if applicable)

22 Archival Research Written information pertaining to the job Job descriptions/class specifications Rules and regulations Training manuals Performance appraisals Previous job analyses Web sites (O*Net, opm.gov, job-analysis.net) Collect and review prior to meeting with SMEs

23 Job Audits / Interviews Most widely used method for data collection! Gather information from SMEs Face-to-face meeting Phone interview Typically conducted at incumbent work location

24 Job Audits / Interviews Ask SME to describe the work that they perform in their own words Have SMEs describe a typical day and then month Have SMEs list major functions and then tasks Probe, probe, probe! Identify duties (key, major responsibilities) and then work down to tasks for each) Begin to prepare tentative list of Tasks and KSAPCs

25 Job Audits / Interviews Interview Questions: Tell me what you do in a typical day? What is important about your work? What tasks are performed most frequently? What must an incumbent be able to perform at entry to the job? Why is the work performed? How do you do this work? What is your role?

26 Tips for Conducting an Audit Brainstorm KSAPC information What knowledge skills, and abilities are needed to perform the critical tasks of this classification? Review tasks with the SME(s), then ask them what they need to know to complete the tasks How do your duties differ from the classification just below yours? Above yours? Technical knowledge Tools and equipment Reading materials Software knowledge Interpersonal and communication

27 Tips for Conducting an Audit Establish rapport Maintain interest (watch non-verbal cues) Review your written information for clarity, completeness and understanding Ask open ended questions Walk through the worksite; look at any equipment used by incumbents

28 Tips for Conducting an Audit More Tips Be courteous Explain what you are doing and why Maintain comfort level If interviewing, use a protocol Remember the incumbent is the expert Keep the incumbent on track Provide feedback

29 When are you finished with your audit? SMEs begin to repeat themselves SMEs begin to identify overly detailed tasks and KSAPCs SMEs begin to identify insignificant tasks and KSAPCs SMEs have no more information to provide WHEN YOU ARE CONFIDENT THAT YOU HAVE MET YOUR OBJECTIVE!

30 Job Observations Observe SMEs who are actually performing their job Serves as an unobtrusive data collection technique Most appropriate for jobs that require: Field work Manual labor Interpersonal communication

31 Job Observations Observation Questions: What are you doing? Why are you doing it? What information and training is required? What equipment is used, and could you demonstrate it? Are there different ways of doing this? What is the consequence of error?

32 Committee-Based Job Analysis What if there are not enough Subject Matter Experts (SMEs) available? What if the objective of the job analysis is to establish a new classification?

33 Committee-Based Job Analyses Committee-based Job Analyses are performed when a lack of staff or SMEs prevents a typical job analysis from being performed A panel of employees from all levels of the organization is assembled Expectations of the job are developed The panel develops a list of tasks and KSAPCs, then rates each statement in terms of position criticality

34 So which data-gathering method should you use? Its not necessary to use all the data-gathering methods for every job analysis Simply choose the best method for getting the data necessary to achieve your desired goal

35 Job Analysis Process - Develop Task Statements JOB ANALYSIS The Job TASKS Task/KSA Links KSAs Test Content/KSA Links TEST CONTENT

36 Task Statements A discrete sentence containing one action verb that concisely describes a single observable activity Address important aspects of the job Task statements are drawn from the information you collect in archival research, interviews, observations, etc.

37 Task Statements Five elements 1. Action verb 2. Object of the verb 3. Rationale or expected output of the action 4. Materials, tools, procedures, or equipment used 5. Under what direction or guidelines

38 Example Task Statement Task Components 1. Action verb 2. Object of the verb 3. Expected output (purpose) or work product 4. Materials, tools, procedures, or equipment used 5. Under what direction or guidelines Transcribe board meetings discussions to maintain Record of Minutes using transcription equipment at the request of the Manager. 4 5

39 More Task Statement Examples Enter employee training requests into online personnel tracking 1 2 system using database software in order to keep up-to-date 4 3 employee development records in accordance with departmental 5 training delivery policies.

40 More Task Statement Examples Perform routine tractor maintenance 1 2 by checking fluids (e.g. coolant, oil, power steering etc.), examining seals, 4 pistons, and chambers in order to identify safety issues and/or replace 3 necessary equipment in accordance with the XYZ maintenance repair and 5 safety checklist.

41 Task Statements You may find it difficult to include all five elements into every task statement within a job analysis What s important: Observable behaviors and work products

42 Task Statements Task statements should be written at the same level of specificity. Avoid being overly broad or overly specific What is insufficient information in a task statement... Poor Fair Better Answer office correspondence. Write replies to letters asking for job information using the computer. Compose written responses to inquiries regarding job openings using word processing software to provide application information within departmental eligibility guidelines.

43 Task Statements and what is too much information in a task statement? Poor Better Prepare travel form #625(a) in triplicate giving airline flight numbers and hotel contact information to obtain travel advances and distribute one copy each to supervisor, manager, and file twelve days prior to departure date using #2 lead pencil at the request of staff. Complete travel forms in a timely manner to obtain travel advances following departmental guidelines at the request of staff.

44 Task Statements Essentially, Tasks should: Start with an action verb Describe work done by one person only Give mention to an identifiable product or outcome Be specific enough to derive associated KSAPCs

45 Assessment Development Process - Develop KSAPC Statements JOB ANALYSIS The Job TASKS Task/KSA Links KSAs Test Content/KSA Links TEST CONTENT

46 KSAPC Statements KSAPC: Knowledge, Skills, Abilities, and Personal Characteristics Derived from essential job tasks Address a single related knowledge, skill, ability, or personal characteristic A single task statement may generate several KSAPCs, and each KSAPC statement may apply to more than one task Every task should have a minimum of one KSAPC Note: KSAPC statements must be operationally defined, meaning they provide enough information to be consistently understood by any reader, with minimal ambiguity or confusion.

47 KSAPC Statements Three elements 1. The knowledge, skill, ability, or personal characteristic required. 2. Degree of proficiency or level used 3. How/why the KSAPC is used or under what circumstances (context). Example Knowledge of multivariate statistics (factor analysis, discriminant 1 2 analysis, logistic regression) to identify population characteristics 3 and make relevant predictions.

48 KSAPC Statements Knowledge: A body of information applied directly to the performance of a function Facts, procedures, etc. Skill: A present observable competence to perform a learned psychomotor act Manual manipulation with precision Ability: Present competence to perform an observable behavior or a behavior that results in an observable product Describe capacity to perform activities, typically cognitive

49 KSAPC Statements Knowledge of basic spreadsheet software functions (e.g., data entry, arithmetic 1 2 formulas, logic tests) to generate monthly expense reports. 3 Skill to mince various food products such as garlic, onion and ginger using 1 2 standard kitchen cutlery to assist in food preparation. 3 Ability to interpret and explain policies, procedures, and regulations to non- 1 technical individuals (e.g. departmental employees, the public, vendors, other 2 State agencies) to ensure they are well-informed. 3

50 KSAPC Statements Personal Characteristics Other elements/aspects relevant to the position may include: o Unique working condition o Unusual job locations o Physical demands o Personality traits

51 KSAPC Statements Example of Personal Characteristics: Willingness to work nights, weekends, and holidays Conscientiousness: Take responsibility and initiative for completing work in a timely manner with an appropriate focus on details Honesty: Refrains from making statements or taking actions that are knowingly false, deceptive, or fraudulent

52 KSAPC Statements KSAPCs should: Address only one worker trait or human characteristic Be operationally defined Be related to one or more task or job dimension Be observable Be easy to read and understand

53 Assessment Development Process - Develop Job Analysis Survey JOB ANALYSIS The Job TASKS Task/KSA Links KSAs Test Content/KSA Links TEST CONTENT

54 Job Analysis Finalization Meeting SMEs must review and finalize all task and KSAPC statements developed through archival research and job audits etc. prior to survey (i.e. JAS) administration Edit task and KSAPC statements, add statements, delete statements, refine language, clarify, etc. This meeting should contain between SMEs

55 Job Analysis Finalization Meeting As with all parts of the Job Analysis process, use a demographically representational sample of SMEs: Gender Length of time at job Race Work location (e.g., field office) Age Different divisions or work units Shift work, i.e., morning, swing, evening (if applicable)

56 Job Analysis Survey (JAS) Purpose SMEs evaluate task and KSAPC statements in terms of their relative importance and other relevant data Essential to the job analysis process, as it determines the appropriateness of using specific Tasks and KSAPCs for selection

57 Job Analysis Survey Things to include: Letter of explanation Contact information (in case of questions). Clear return instructions (paper-form and electronic) A reasonable deadline (in a prominent place) Demographic questions o o Job relevant questions EEO information (Optional response item)

58 Job Analysis Survey Things to include (continued): Demographic Questions o Job relevant questions Current job classification Relevant expertise (incumbent, supervisor, trainer..) Classification filling out the survey for (if the survey is evaluating more than one job as in a classification series job analysis) Program area/division Office location Work location (Field, Headquarters ) o EEO information (Optional response item) Race Gender Age

59 Job Analysis Survey The number of SMEs needed to complete the JAS varies depending on how many incumbents are currently in the job. Number of Incumbents Job Analysis Sample Number of Incumbents Job Analysis Sample % (70 max)

60 Job Analysis Survey Tasks and KSAPCs are separated into two lists because they are evaluated using different ratings scales. TASKS Importance KSAs Importance Frequency Expected at Entry Relationship to Performance

61 Task - Importance IMPORTANCE: How important is this task to successful job performance? (0) Does Not Apply (1) Moderately Important (2) Important (3) Very Important (4) Critical

62 Task - Frequency FREQUENCY: How often is this job task performed? (0) Does Not Apply This task is not an essential function of the job, or this task is not performed on the job. (1) Less than Once a Month This task is performed at least one time a year, but less than once a month (includes quarterly). (2) Monthly This task is typically performed at least one time a month but less than once in a typical week. (3) Weekly This task is typically performed once to several times each week but less than once a day. (4) Daily This task is performed one or more times a day.

63 KSAPC - Importance IMPORTANCE: How important is this KSAPC to successful job performance? (0) Does Not Apply (1) Moderately Important (2) Important (3) Very Important (4) Critical

64 KSAPC Expected at Entry EXPECTED AT ENTRY: When is a person expected to have this KSAPC? Is it required before being hired or do they learn it on the job? (0) Not Needed This KSAPC is not needed on the first day of the job. It is either learned on the job or through training, or is not an essential component of the job. (1) Needed Possession of this KSAPC is needed and expected on the first day of the job. Possession of MOST of this KSAPC is needed and expected upon entry to the job. (2) Essential Possession of this KSAPC at entry to the job is essential. Possession of ALL of this KSAPC is expected upon entry to the job.

65 KSAPC Relationship to Job Performance RELATIONSHIP TO JOB PERFORMANCE: Does possession of more of this KSAPC beyond minimum requirements lead to better job performance? (0) No Observable Relationship This KSAPC is not required to perform the job, or possession of more of this KSAPC (beyond the minimal level required) does not result in better job performance. (1) Observable Relationship Possession of more of this KSAPC (beyond the minimal level required) does result in better job performance.

66 Survey Administration As usual, strive for a representative sample: Gender Length of time at job Race Work location (e.g., field office) Age Different divisions or work units Shift work, i.e., morning, swing, evening (if applicable)

67

68 Linking Task and KSAPC Statements The Linking process is used to identify which KSAPCs are required to perform which tasks Linking evidence should be provided in the final Job Analysis report If a KSAPC does not link back to a specific task it is dropped from the final job analysis

69 Linking Task and KSAPC Statements KSAPC Ability to ascend a twelve-foot ladder to perform rooftop maintenance. Task Repair shingles, gutters, and drainage systems of government owned facilities in compliance with applicable building codes by replacing missing and damaged components in a timely manner. KSAPC Expert Knowledge of rooftop drainage systems to complete building maintenance projects. KSAPC Skill to operate electric powered drills to repair damaged or broken building components.

70 Linking Task and KSAPC Statements SMEs are presented with the edited list of Tasks and KSAPCs. They are instructed, for each KSAPC, to read through every task and identify those which require or utilize the specific KSAPC. With this process, one KSAPC may be linked to several tasks. Example:

71 Linking # KSAPC Statement Linking Task # Scientific Knowledge and Research 1 2 General knowledge and understanding of scientific research principles and investigatory methods and statistical analysis as applied to integrated waste management to achieve program goals. 6,7,9,32,33 1,2 Ability to extract relevant facts and information from written documents in order to perform effective project planning, solve 18,19,20,21,22,28 1,2,4 problems, or summarize information. Classification Specification # 3 Knowledge of chemical, biological, physical, and environmental sciences as applied to waste management. 9,20,21,24,34 5,6 4 Ability to identify the information needed to answer research questions. 6,7,9,10,13,28,34 1,2,3 Data Interpretation and Computation 5 Ability to interpret graphical data to extract or identify the key points or issues that are presented. 6,7,9,10,13,20,21,22, Ability to interpret quantitative or statistical data to extract key information and make valid inferences. 6,9,10,13, Ability to perform basic computations using addition, subtraction, multiplication and division to summarize data. 6,9,10,13,20,21,22,23 8,10

72 Linking Task and KSAPC Statements Review linkage data for agreement Agreement is generally found if two or more SMEs endorse the linkage If a group setting is used, the SMEs establish consensus on which KSAPCs link to which tasks KSAPCs that do not link are dropped from the final job analysis Note: Linked KSAPCs become the basis of the examination plan and may be used to select testing methods.

73 Task/KSA Linking Requirements Technically You really only need to link those KSAPCs for which you are testing. Therefore, it is not a requirement that you link every KSAPC. o especially if linking will be arduous, or interfere with examination timelines.

74 Online Survey Tools to use with Job Analyses! Making your life just a little bit easier. 74

75 Different Online Survey Options Survey Monkey Zoomerang EZquestionnaire KeySurvey

76 Online Surveys 1. High levels of Customization allowed o o o Color Logos Size and Placement 2. Collects demographic and job information from SMEs

77 Online Surveys

78 Benefits of Online Surveys 3. Collects SME ratings of Tasks and KSAPCs Many different scale option and designs available Forced choice (No more missing data )

79 Online Surveys

80 Benefits of Online Surveys 4. Can disseminate the JAS to hundreds of SMEs simultaneously o o Can set up a link in an Customized message can be created and sent to a preestablished list of SMEs Can have Survey Monkey resend the JAS to only those who have not finished the survey

81 Benefits of Online Surveys 5. Reduce mailing costs Faster responses Higher response rate

82 Benefits of Online Surveys 6. Reduces Data Entry o Can cut and paste Tasks and KSAPCs straight from a Word or Excel document into the JAS o Avoid wrist fatigue and injuries!

83 Benefits of Online Surveys 7. Improves Quality of Data o o Reduces human error Easier to fill out than a Scantron

84 Benefits of Online Surveys 8. Analyzes Data Summarizes data automatically Responses can be downloaded into a variety of formats o including an Excel spreadsheet where results can be copied and pasted right into a job analysis report

85 Summary Job analysis surveys are a critical part of many HR responsibilities particularly in selection There is no one single best way but you should strive for objective quantitative data Information from job analysis can come from many sources such as duty statements, class specifications, previous job analyses, O*NET, job audits and observations

86 Summary Rely heavily on SMEs when developing or finalizing task and KSA statements Job analyses can be administered in groups, via paper-and-pencil surveys or online Online surveys have many advantages

87 Jim Higgins, Executive Director BCGi

88 Upcoming Webinars Visit bcginstitute.org to register! June 21 Ask a Consultant July 12 The Role of Talent Acquisition in Affirmative Action Planning July 19 VETS 4214 July 24 How to Create Statistically Valid Samples Using Microsoft Excel