An Evaluation of the Utility of Three Ergonomics Checklists for Predicting Health Outcomes in a Car Manufacturing Environment

Transcription

1 An Evaluation of the Utility of Three Ergonomics Checklists for Predicting Health Outcomes in a Car Manufacturing Environment DAVID BRODIE AND RICHARD WELLS Department of Kinesiology, University of Waterloo Waterloo, Ontario, N2L 3G1 The need for a quick and easy observational tool for analysis of jobs has prompted the development of many ergonomic checklists. If the results of an analysis using a simple tool are to be considered valuable, the accuracy and reliability of measurements with these tools must be determined. This paper presents a preliminary testing of the reliability and accuracy of three previously developed checklists: Rapid Upper Limb Assessment (RULA); Occupational Safety and Health Administration (OSHA) draft risk factor checklist; and the Posture and Upper Extremity checklists developed by Keyserling et al. The results illustrate that caution should be used before checklists are adopted as a component of an ergonomic program. 1. Introduction This project is part of an Ergonomics Initiative (EI) involving a partnership between the University of Waterloo, and three private corporations involved in car assembly and parts manufacturing. One of the key goals of the EI is the development of tools or methods for analysing jobs. Due to the large number of jobs within each of these companies, a quick method of analysis was desired that could give a good preliminary indication of risk. An ergonomic checklist was proposed as a tool that could provide quick observation and characterization of the levels of risk factors present (Hagberg et al., 1995). This study was designed to assist the EI in the development of an ergonomic checklist that could be used to analyse the jobs within these industries to help prioritize jobs for intervention. An ergonomics analysis of a job may use tools that vary from simple, observational methods to more complex multi-dimensional techniques. A basic checklist, where the demands of a job are simply recorded (Holzmann, 1982; Karhu, Kansi, & Kuorinka, 1977; Corlett, Madeley, & Manenica, 1979; Keyserling, Brouwer, & Silverstein, 1992; Keyserling, Stetson, Silverstein, & Brouwer, 1993; Stetson, Keyserling, Silverstein, & Leonard, 1991), would be considered the simplest form of analysis. Increased sophistication is achieved when the individual task variables are combined in some arithmetic fashion to determine their interaction (McAtamney & Corlett, 1993; Moore & Garg, 1995). Further sophistication of analysis can be obtained through the use of biomechanical or mathematical models, video analysis, electromyography, or such devices as the lumbar motion monitor (Marras, Lavender, Leurgans, Rajulu, Allread, Fathallah, & Ferguson, 1993). Simple analyses focus on the collection of task variables which may give rise to mechanical exposure within the body (Winkel and Mathiassen, 1994). Complex analyses attempt to measure or predict the forces that are acting on or in the body (eg. spinal compression). The measurement of internal exposure increases the precision and sophistication of the analysis, but requires larger resources of time and money. Simple analyses may not have the same level of information as the more complex methods, but they are fast and easy to use by larger numbers of people. A decision must be made, based on the expected utility of the tools, as 45

2 to which method is best suited for the task. A company is often limited in the time and resources available for a more detailed ergonomics analysis. Consequently, the more simple observational tools are the more frequent method of choice. For a tool to be a useful component of an ergonomic analysis, it must be a reliable, valid measure of the desired variable. This is especially important if the development of an intervention is based on the output of the analytical tool. Many checklists and indices have been developed that attempt to achieve the simplicity required for large scale use, while providing a useful output (Holzmann, 1982; Karhu et al., 1977; Keyserling et al., 1992; Keyserling et al., 1993; Lifshitz & Armstrong, 1986; McAtamney & Corlett, 1993; Moore & Garg, 1995; Rose, Ericson, Glimskar, Nordgren, & Ortengren, 1992; Stetson et al., 1991). A problem that has emerged is that more work has been done on creating the tools, and relatively little has been done to determine their utility. The intent of this paper was to evaluate the utility of ergonomic checklists in their current stage of development. Two components of utility were deemed to be important: the first was reliability, where consistency of checklist output was deemed to be a necessary quality for wide spread use within industry; and, the second was the ability of ergonomic checklists to provide outputs that document the presence of risk factors and potentially resulting musculoskeletal disorders and job turnover. 2. Methodology 2.1 Subjects Sixty-seven persons were recruited to perform the job analyses. The subjects were divided into three categories based on their levels of experience in job analyses, analysing human movement, and working within an industrial environment. Group I (N=46) had minimal experience in all three areas; this group had approximately 6 months experience in analysing postural variables, one week of training in ergonomics and no experience working in an industrial environment. Group II (N=12) had intermediate experience working in an industrial environment, and advanced experience in postural and job analyses; the members of this group had worked 1-2 years within an industrial environment, and had 2-3 years experience in postural analysis. Group III (N=9) had minimal experience with postural and job analyses (3-day training session), but had extensive experience working in an industrial environment; the members of this group were mature individuals (28-55 years), with considerable work experience in the health and safety field (3-15 years). 2.2 Jobs and Job Selection Criteria Nine jobs from an automotive assembly operation were chosen to be analysed. The jobs were chosen based on the varying types of movements required, and the variable levels of demands imposed by each job. The jobs were chosen from various locations within an automotive assembly plant, including sections with work at levels from wheel-well to under-hood to overhead. The jobs involved everything from heavy lifting to intensive tool use to fine-finger manipulations. The jobs were videotaped from numerous angles and distances to provide as much detail as possible to reduce the variability caused by on-site versus videotape analysis (Ericson et al., 1991; Douwes & Dul, 1991; Stetson et al., 1991). 2.3 Checklists Three checklists were utilized to perform the job analyses: the Rapid Upper Limb Assessment (RULA) tool (McAtamney & Corlett, 1993); the body region specific checklists produced by Keyserling et 46

3 al. (KEY)- neck, trunk and legs checklist (1992), and the upper extremity checklist (1993); and, the draft OSHA checklist for analysing the stress on the upper limb and lower back. Members of Group I were randomly assigned one checklist to use, resulting in 15 subjects using RULA, 15 using OSHA, and 16 using the KEY checklists. Group II used RULA to analyse 4 jobs, and OSHA to analyse the remaining five. Group III used RULA to analyse 4 jobs only, which were the same four jobs analysed by Group II with RULA. Those subjects using RULA were instructed to analyse the instant of the job that they felt produced the peak stress on the worker, and were asked to provide a description of this instant. 2.4 Analyst Training All three groups received training in the use of the checklists. This training lasted approximately 20 minutes. The subjects were allowed to ask questions regarding the use of the checklist while they were performing the analyses. Group I received additional training dealing with the basics of ergonomics and task analyses. Group III was on the third day of a 3-day ergonomics seminar that was provided by an external occupational health and safety consulting firm. Group II received no additional training outside of explanations of the checklists. 2.5 Job Analysis with Checklists The taped jobs were presented on a 21" video monitor, with no more than 12 individuals watching a monitor at any time. The nine jobs were randomly presented to the analysts. No time restrictions were placed on the analysis of the jobs, and all jobs were analysed in one session. Group I performed an initial analysis of the jobs, and after a two-week break, a second analysis of the same nine jobs was performed. Groups II and III performed only one analysis of the jobs. 2.6 Symptom Questionnaires A symptom questionnaire was utilized to determine the prevalence of musculoskeletal problems within each of the jobs. The questionnaire was modelled after one presented within the OSHA draft regulations, with questions similar to those used in previous NIOSH investigations (NIOSH, 1991; NIOSH, 1993). The questions were designed to determine the frequency (5 point scale), duration (6 point scale), severity (5 point scale), and location of symptoms of musculoskeletal disorders. Symptoms of pain, aching, stiffness, burning, tingling, or numbness were considered indicative of a work related musculoskeletal disorder (NIOSH, 1991). All individuals performing the nine jobs were interviewed. The experimenter administered the questionnaire, and remained present to answer any questions. 2.7 Supervisor Interviews The supervisors were asked to rank the jobs in their area based on a ten point scale, with anchors reflecting multiple characteristics of the job (eg. 0 = very low absenteeism, very low turnover, no complaints, job looks very easy; 10 = very high absenteeism, very high turnover, excessive complaints, job looks very difficult). The supervisors were not asked to provide separate ratings for the upper limb and lower back. 2.8 Video Analysis of Jobs Each job was analysed using the WATbak 2-D video analysis software. Several portions of each job were analysed to determine the peak spinal compression and peak anterior reaction shear forces. Peak spinal compression was chosen based on studies illustrating its link to risk of injury of the lower back (Marras 47

4 et al., 1993). 2.9 Classification of Jobs A classification system was necessary to provide a criterion measure for determination of the accuracy of checklist outputs. The quantitative measures of spinal compression and shear were available for the back, but no comparable measures were available for the upper limb. Additionally, the measures of spinal loading were strictly mechanical in nature, and do not represent all of the variables that could contribute to injury or illness due to work. A three-point ordinal ranking scale was used to rank each job s level of risk of injury for the upper limb and lower back individually. The scale had values of 1, 2, and 3, where 1=low, 2=intermediate, and 3=high levels of risk of injury. Each job was assigned a value for the upper limb and lower back based on: symptom questionnaires; supervisor interviews; spinal loading measures; and, expert opinion Data Analysis A 2 x 2 analysis of variance (ANOVA) was performed for every variable in each checklist and group. The ANOVA provided a measure of the variance contributed by jobs, analysts and by chance. A reliable checklist would have a larger proportion of variance contributed by the jobs versus the analysts. An intra-class correlation coefficient was calculated to determine the inter-analyst reliability. This coefficient was used for all variables, regardless of scale, and has been shown to be equivalent to the kappa coefficient (Fleiss, 1986). The coefficient was evaluated using categories that were proposed for the kappa coefficient (Fleiss, 1986): excellent agreement - r>0.75; fair to good agreement - r= ; poor agreement - r<0.40. An additional 2 x 2 ANOVA was performed on the test, re-test data for Group I; intra-class correlation coefficients were calculated from the results to determine the intra-analyst reliability. A calculation of percent agreement, across all jobs, was performed for variables with low reliability coefficients to clarify the source of the low correlation (Wiktorin et al., 1991). Spearman correlations were performed on all data to determine the relationships between the checklist outputs and the criterion measures. An average output per job (for all groups) for the checklists was used to reduce the effect of the unreliability of the outputs. The scores used, therefore, represent the best score available from each group. An analysis of sensitivity and specificity of the RULA and OSHA checklists was performed to determine how effectively the analysts could determine which jobs were at high levels of risk. The OSHA and RULA checklists, and not the KEY checklists, were used in this portion of the analysis because they provided a distinct cut-off point for low risk/high risk jobs. 3. Results and Discussion Are checklists a viable tool for use in an ergonomics program? The results of this analysis would suggest that improvements should be made. Table 1 provides a general summary of the results of the reliability analysis. A few trends developed in the analysis, which held true for each checklist and analyst group. In general, movements that are hard to define (e.g. twisting, rapid rotation), postures of smaller joints (wrist, elbow), and items that were difficult to see from video (e.g. contact stress, vibration) were poorly analysed. Conversely, movements of larger joints (shoulder, back) and variables that had quantitative measures (e.g. mass, force) were analysed very effectively. Although an attempt was made to find jobs that had a wide range of activities, it became apparent that certain factors did not change across jobs or did not occur within the jobs analysed in this project. As 48

5 such, the variables listed under stable and non-applicable were not adequately analysed for reliability during this project. Only the total score provided by the Manual Materials Handling portion of the OSHA checklist proved to be reliable; the remaining checklists had low to moderate reliability, which simply reflects the poor reliability found in the individual variables that are measured within these checklists. Table 1. Summary of Reliability Analysis Reliability Coefficients Stable Factors Non-Applicable Risk Factors Poor Fair-to-Good Excellent Individual Factors twisting neck posture shoulder posture repetition seated work rapid movements trunk posture lighting prone work wrist posture elbow posture contact stress quantitative measures (eg. mass, force) control of work pace temperature kneeling/ squatting forceful push/pull keyboard work local vibration Total Scores RULA (Grand Score) RULA (Score A & B) OSHA (MMH) KEY (Posture and Upper limb) OSHA (Back and Lower Extremity) OSHA (Upper Extremity) The analysis of the validity of the checklist predictions was performed on several levels. Initially, the ability of the individual analyst to provide a valid measure of the jobs was determined; it was found that individuals varied greatly in their ability, with some being excellent while others were inconsistent with their predictions. The next stage was to take the average prediction of risk across all the analysts. This was necessary to determine how well the checklists could work if the problem of reliability could be resolved. The analysis of the upper limb was found to be less accurate than that of the low back. The Grand Score from RULA seemed to provide the best measure for the upper limb. OSHA provided the most consistent, accurate prediction for the low back. Basically, no one checklist performed well for all types of analyses, but OSHA had the best overall performance. It must be clearly stated that before these levels of accuracy can 49

6 be achieved, the reliability of the checklists must be improved. The final stage of analysis of checklist performance was to determine the trends in the predictions of the analysts. Given that the analysts were providing different answers in many cases, it was necessary to determine it they were placing the jobs in the same risk categories. An analysis of sensitivity and specificity illustrated a strong tendency for analyses with each checklist to overestimate the number of jobs that were hazardous. The sensitivity of the analyses tended to be high, indicating an excellent trend of classifying a job as hazardous when it was truly hazardous. Conversely, the specificity of the analyses tended to be low, which indicates that many jobs were classified as hazardous when they actually were not. Therefore, if a person wanted a checklist that would effectively identify all of the potentially hazardous jobs in a plant, then these checklists perform this function well. 4. Conclusions This paper has shown that many of the questions found within the checklists were not reliable, and the spread in individual scores was large for many questions. Accuracy cannot be attained if reliability is poor, however large differences between jobs can still be detected even with an unreliable instrument. When the average output per group was used as the score for a specific job, then the predictive ability of the checklists was fairly good for the lower back; the performance of the checklists for predicting risk of injury to the upper limb was poor, and improvements in the design of the checklists dedicated to this area of the body is necessary. In using average scores to determine accuracy, this paper illustrates the potential of the checklists to provide useful output if the reliability is improved. This result justifies any work performed to improve the reliability of the checklists. There are many ways in which the reliability of the checklists can be improved, such as improved wording of questions, developing decision criteria and the utilization of site-specific examples, but research must be performed to determine the effectiveness of these recommendations. The results of this project have illustrated that the output of checklists can be very unreliable and inaccurate, and care must be taken in adopting any tool within an ergonomics program, regardless of its cost or complexity. 5. References Corlett, E.N., Madeley, S.J., & Manenica, I. (1979). Posture Targeting: A Technique for Recording Working Postures. Ergonomics, 22(3): Douwes, M., & Dul, J. (1991). Validity and reliability of estimating body angles by direct and indirect observations. In Y. Quéinnec & F. Daniellou (Eds.). Designing for Everyone (pp ). London, New York, Philadelphia: Taylor & Francis. Ericson, M., Kilbom, A., Wiktorin, C., & Winkel, J. (1991). Validity and reliability in the estimation of trunk, arm and neck inclination by observation. In Y. Quéinnec & F. Daniellou (Eds.). Designing for Everyone (pp ). London, New York, Philadelphia: Taylor & Francis. Fleiss, J.L. (1986). The Design and Analysis of Clinical Experiments. John Wiley & Sons, Toronto. Hagberg, M., Silverstein, B., Wells, R., Smith, M.J., Hendrick, H.W., Carayon, P., & Perusse, 50

7 M. (1995). Work Related Musculoskeletal Disorders (WMSDs): A Reference Book for Prevention. Ed. I. Kuorinka & L. Forcier. Taylor & Francis, London. Holzmann, P. (1982). ARBAN: A new method for analysis of ergonomic effort. Applied Ergonomics, 13(2): Karhu, O., Kansi, P., & Kuorinka, I. (1977). Correcting working postures in industry: a practical method for analysis. Applied Ergonomics, 8(4): Keyserling, W.M., Brouwer, M., & Silverstein, B.A. (1992). A checklist for evaluation ergonomic risk factors resulting from awkward postures of the legs, trunk and neck. International Journal of Industrial Ergonomics, 9: Keyserling, W.M., Stetson, D.S., Silverstein, B.A., & Brouwer, M.L. (1993). A checklist for evaluating ergonomic risk factors associated with upper extremity cumulative trauma disorders. Ergonomics, 36: Lifshitz, Y. & Armstrong, J. (1986). A Design Checklist for Control and Prediction of Cumulative Trauma Disorder in Intensive Manual Jobs. Proceedings of the Human Factors Society--30th Annual Meeting, Liker, J.K., Evans, S.M., Ulin, S.S., & Joseph, B.S. (1990). The strengths and limitations of lecture-based training in the acquistion of ergonomics knowledge and skill. International Journal of Industrial Ergonomics, 7: Marras, W.S., Lavender, S.A., Leurgans, S.E., Rajulu, S.L., Allread, W.G., Fathallah, F.A., & Ferguson, S.A. (1993). The Role of Dynamic Three-Dimensional Trunk Motion in Occupationally-Related Low Back Disorders. Spine, 18:5: McAtamney, L., & Corlett, E.N. (1993). RULA: a survey method for the investigation of work-related upper limb disorders. Applied Ergonomics, 24(2): Moore, J.S., & Garg, A. (1995). The strain index: a proposed method to analyse jobs for risk of distal upper extremity disorders. American Industrial Hygienists Association Journal, 56(5): National Institute for Occupational Safety and Health. (1991). Health Hazard Evaluation Report: Los Angeles Times. NIOSH Report No. HETA U. S. Department of Health and Human Services, Public Healthy Services, Centers for Disease Control, National Institute for Occupational Safety and Health, Cincinnatti, OH. National Institute for Occupational Safety and Health. (1993). Comments on the Occupational Safety and Health Administration proposed rule on Ergonomic, Safety and Health Management

8 CFR Part 1910, Docket No. S-777. U. S. Department of Health and Human Services, Public Healthy Services, Centers for Disease Control, National Institute for Occupational Safety and Health, Cincinnatti, Ohio. Rose, L., Ericson, M., Glimskar, B., Nordgren, B., & Ortengren, R. (1992). Ergo-Index: development of a model to determine pause needs after fatigue and pain reaction during work. In M. Mattila & W. Karwowski (Eds.). Computer Applications in Ergonomics, Occupational Safety and Healthy, (pp ). Elsevier Science Publishers B. V. Stetson, D.S., Keyserling, W.M., Silverstein, B.A., & Leonard, J.A. (1991). Observational Analysis of the Hand and Wrist: A Pilot Study. Applied Occupational and Environmental Hygiene, 6(11): Wiktorin, C., Karlqvist, L., Nygard, C-H., & Winkel, J. (1991). Design and reliability of a questionnaire for estimation of physical load in epidemiological studies. In Y. Quéinnec & F. Daniellou (Eds.). Designing for Everyone (pp ). London, New York, Philadelphia: Taylor & Francis. Winkel, J. & Mathiassen, S.E. (1994). Assessment of physical work load in epidemiologic studies: concepts, issues and operational considerations. Ergonomics, 37(6): Acknowledgements The authors wish to acknowledge the support of the Natural Sciences and Engineering Research Council and the Ergonomics Initiative, directed by Dr. R. Norman and funded by G.M. of Canada, A.G. Simpson Ltd., and The Woodbridge Group, for their financial support of this project. We would also like to thank all the workers, supervisors, and the numerous Initiative members whose assistance and cooperation made this project possible. RÉSUMÉ Le besoin d un outil d observation qui serait facile et rapide à utiliser dans l analyse des tâches a suscité la création de nombreuses listes de contrôle de facteurs ergonomiques. Si l on désire considérer comme valable les résultats d une analyse effectuée à l aide d un outil simple, la précision et l efficacité des mesures de cet outil doivent être déterminées. Cet article présente un test préliminaire de l efficacité et de la précision de trois listes de contrôle déjà élaborées : le Rapid Upper Limb Assessment (RULA); la liste préliminaire des facteurs de risque du Occupational Safety and Health Administration (OSHA); et la liste de contrôle Posture and Upper Extremity élaborée par Keyserling et coll. Les résultats démontrent que l on doit être vigilant avant d adopter des listes de contrôle comme composante d un programme ergonomique. 52