BOARD ON HUMAN-SYSTEMS INTEGRATION

Transcription

1 BOARD ON HUMAN-SYSTEMS INTEGRATION DIVISION OF BEHAVIORAL AND SOCIAL SCIENCES AND EDUCATION Good Measurement: What Employee Selection, Training, Performance, Satisfaction and Turnover Should Have in Common Frederick Oswald Rice University 4/20/16 BOARD ON HUMAN-SYSTEMS INTEGRATION

2 Preliminary Point: Measurement is Critical! The point is not that adequate measurement is nice. It is necessary, crucial, etc. Without it we have nothing. (Korman, 1974, p. 194) to have confidence in the findings of our studies, we must first have confidence in the quality of our measures (Noar, 2003, pp ).

3 Context of Measurement General Contexts Academic vs. organizational IRB vs. organizational climate for surveying General perceptions of fairness, relevance (from all stakeholders, including those of the test-taker) Legal concerns

4 Context of Measurement Goal-driven measurement Research (e.g., compensate, promote, transfer terminate Managerial decision-making (e.g., compensate, promote, transfer terminate) Selection (e.g., HR staffing, team composition) Developmental (e.g., training evaluation)

5 Context of Measurement Content General Specific/contextualized Multiple Single measures Strong Weak or subtle indicators Forms Many items Few items Self-report Other report Traditional Innovative

6 Context of Measurement Content Big data leanings General Specific/contextualized Multiple Single measures Strong Weak or subtle indicators Forms Many items Few items Self-report Other report Traditional Innovative

7 Context of Measurement? Limited Testing Time Problem: Too many constructs and not enough time, resources or test-taker patience Reasons: Theories become more complex; greater demand for answers to practical organizational questions Solutions: Big data? Reduce constructs to essential ones? Abandon use of multiple scales for a construct (as can be used in SEM)? Shorten measures?

8 Given context Develop a New Measure? No do not toss an ad hoc measure into a study Use an existing measure when there is a strong theoretical basis past empirical research demonstrates reliability and validity you are not interested in a measuredevelopment study etc.

9 Given context Develop a New Measure? Yes develop a new measure when access to existing measures is limited (expensive, proprietary) test security is of concern ( freshening item pools, previous test compromises) there is important room for improvement (improved theory, aligning the measure with the intended purpose, increased sensitivity to the test-taker perspective, updating language) testing time is limited (don t just cut the measure) etc.

10 Item Development Good measure development and therefore good results requires sound investment.* Expertise (substantive researchers, SMEs, psychometricians, sensitivity review) Development (item generation, refinement, translation/backtranslation) Research/evidence (reliability, validity, low adverse impact, generalizability)

11 Item Development Item content can be evaluated for relevancy, deficiency and contamination; However, these three characteristics can also be psychological phenomena (e.g., did the test-taker forget or get confused by the item content?).

12 Item Development Appropriate content sampling from a construct domain is a necessary condition for obtaining interpretable reliability evidence for a set of items. High reliability coefficient does not ensure adequate content sampling: item relationships are affected by shared contaminants and unmodeled deficiencies.

13 Item Development e.g., Conscientiousness Identified facets (aspects) under its construct umbrella: Competence, Order, Dutifulness, Achievement Striving, Deliberation Controlled heterogeneity: create items reflecting each facet Given enough items, a facet can become a reliable scale on its own (e.g., high alpha, strong single factor).

14 Item Development Who generates items? SMEs: domain-specific experts/theorists; job analysts; job incumbents; researchers themselves relying on past theories and measures How many people are needed to generate items? Often fewer than one thinks (it depends, but 5-8 for many constructs). Generate items (from SMEs, research literature) until categories/themes (and possibly specific content) are redundant.

15 Item Development What items are appropriate given the measurement goals? e.g., knowledge: easier items (minimal competence); difficult items (certify professionals); full range (accurately measure people s knowledge across the range) personality: screen out extremes (e.g., OCD) vs. assess normal personality (e.g., conscientiousness)

16 Measure Format Instructions Often, instructions are written at too high of a grade level (5 grade levels too high for patient discharge interview; Spandorfer et al., 1993). Very few people read them (Novick & Ward, 2006), though novices who read them will improve (Catrambone, 1990). Detailed instructions about providing objective information (BARS, time spans, frequencies) often do not change the subjective response process. General suggestion: Assume that test-takers will ignore (or skim) instructions and proceed accordingly! Novel formats requiring instructions demand pilot testing to ensure comprehension and quality responding.

17 Measure Format Usability Grammar and syntax matter, not just for understandability and better data but for credibility. Clear, readable font. Intuitive method of responding. For web-based measures, check browser types and screen resolutions. Minimize drudgery; maximize simplicity and readability. Keep all cognitive burdens to a necessary minimum (see Dillman s Tailored Design Method; Dillman, 2006)

18 Measure Format Randomize items? Cluster items within a construct; cycle across constructs; randomize items? Randomizing does not seem to make a difference. It might increase honesty (by not knowing the construct being assessed), though it may also heighten confusion (by shifting gears across constructs). General suggestion: Group items within construct, especially for longer batteries of measures.

19 Measure Format Negatively worded items? Responses based on misreading even a small number of negatively worded items can distort the reliability, and therefore the validity, of a measure (Schmitt & Stults, 1985). Double negatives are especially bad. General advice: Keeping items in the direction of the construct and avoiding negation outweighs balancing positively and negatively worded items.

20 Measure Format Pilot test a new measure before collecting data. Have people typical of your future sample(s) take the measure and provide feedback about the items (e.g., relevant, noninvasive, culturally sensitive, no typos, easy to take?).

21 Measure Format Are single-item measures useful? Perhaps in some cases: Research using single item measures of each of the five JDI job satisfaction facets and found correlations between.60 and.72 to the full length versions of the JDI scales (Nagy, 2002) Review of single-item graphical representation scales; so called faces scales (Patrician, 2004) Single item graphic scale for organizational identification (Shamir & Kark, 2004) Single item measures work best on homogeneous constructs (Loo, 2002) Question: Why not administer a single-item knowledge test (hey, it would be quick to administer, easy to grade)?

22 Measure Format Make items very similar, almost parallel? ( I attend to every detail. vs. I am very detail-oriented ) 1.a way to ensure respondents are paying attention to a particular aspect of the construct (are you sure that you attend to every detail?), vs. 2.a sneaky way to ensure that alpha is high but this strategy may capitalize on item specificity

23 Factors Affecting Reliability Assuming that items reflect a single dimension: α =.70 is usually not high enough (Lance, Butts, & Michels, 2006, ORM); e.g., to make selection decisions higher item intercorrelations = higher alpha (add similar items) additional items = higher alpha (add more items)

24 Factors Affecting Validity Examples: Incumbent scores, hired via top-down selection procedures and low selection ratios (= direct and incidental range restriction); Minimal training competencies tested on a highly able sample (= floor effect); or the converse, such as a pre-test prior to training (= ceiling effect) Sampling error variance (small random samples), creates variability in validity that is statistical, not substantive All of the above also affect item-total correlations and item-level validity (e.g., everyone gets an item right).

25 Further Reading General Guidelines DeVellis, R. F. (2012). Scale development: Theory and applications. Thousand Oaks, CA: Sage. Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press. Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, Shortening measures Credé, M., Harms, P.D., Niehorster, S., & Gaye-Valentine, A. (2012). An evaluation of the consequences of using short measures of the Big Five personality traits. Journal of Personality and Social Psychology, 102, Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The Mini- IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment, 18, Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55,

26 Thank you! Fred Oswald Rice University 6100 Main St MS25 Houston, TX (713)