Rater-Reliability of a 5S Audit Checklist

Transcription

1 Proceedings of the 2014 Industrial and Systems Engineering Research Conference Y. Guan and H. Liao, eds. Rater-Reliability of a 5S Audit Checklist Lawrence E. Whitman and Michael Jorgensen Department of Industrial and Manufacturing Engineering Wichita, KS 67260, USA Nishant Gorrepati Goodman Manufacturing Dayton, TN 37321, USA Abstract Audit checklists help evaluators understand how a program is performing and what improvements must be made in order to achieve better results. Testing the of the rating method, before developing conclusions regarding performance metrics and areas of improvement, is essential. This paper presents that the existing 5S audit checklist has low. After assessing the content validity, the 5S audit checklist was modified and this paper presents the modification. Keywords Lean manufacturing, 5S, Audit checklist, Rater-, Training techniques 1. Introduction Global markets are continuously changing and demanding customized, low cost and high-quality products and services [1]. Lean manufacturing has been developed as a method to achieve competitive advantage. Lean manufacturing minimizes waste and produces high-quality products that meet customer expectations [2]. Some tools being used in lean manufacturing are: Kaizen, one-piece flow, 5S, cellular manufacturing, inventory management, poka yoke, standardized work, and scrap reduction [3]. 5S is a powerful tool that can be used to reduce waste and achieve a workplace with high productivity and quality through orderly maintenance. To achieve the best results from the 5S program, it has to be sustained for long periods. The best way to sustain the program is through audits, requiring the creation of audit checklists. A checklist is a device that consists of a list of activities, items and criteria used to perform certain tasks [4]. When used for auditing, they are known as audit checklists. Unfortunately, the majority of performance measures are contaminated by rating errors that lead to inaccurate performance evaluation scores [5]. A study was conducted using a virtual reality factory model and a group of students who evaluated the factory using the 5S audit checklist commonly used in aerospace industries. The study hereafter will be referred to as the existing 5S audit checklist and the study results showed low rater- scores. This paper presents the reasons for low rater- and proposes a new 5S checklist. 2. LITERATURE REVIEW Lean manufacturing combines the production techniques of mass production and craft production; as a result, the process benefits from improved product quality, reduction in cycle time, reduction in Work in Process (WIP), reduction in inventories, improved net income etc. [6]. 5S is a lean manufacturing tool that can help maintain a clean and organized shop floor, increasing the efficiency and effectiveness of the system [7]. 5S stands for five Japanese words: Seiri (Sort), Seiton (Set in order), Seiso (Shine), Seiketsu (Standardize) and Shitsuke (Sustain) [8]. During the implementation of 5S, the tools of the workplace are sorted and identified, the tools in use separated from the tools not in use. All the tools that are not in use are removed from the workplace. After sorting and identifying the tools in use, the tools are designated a specific space and order within the workplace. The sorting eliminates the need for operators to waste time searching for tools. Keeping a spotless,

2 shining workplace through regular cleanings is an essential part of 5S. The 5S practices are standardized through visual management and the whole program sustains itself through audits. Within the organizations that implement 5S, a main reason for failure stems from the lack of institutionalization of proscribed 5S activity [9]. A commitment by the management to perform regular audits promotes workplace wide involvement with the 5S system. The audit checklists are a vital part of creating a standardized environment. Audit checklists are developed to help evaluators understand how the current program is performing and what improvements must be made to achieve better results. However, poorly designed checklists can have potentially negative impacts such as checklist fatigue, decreased by adding unnecessary complexity to the evaluation process and delays in the completion of the evaluation process. In addition, an inadequate checklist may lead to evaluation myopia, or the inability of the evaluator to identify side effects or side impacts due to rigid application and non-reflective use of the checklist [4]. In order to review whether the checklist is performing its required job or not, a validity and test of the checklist is done. Validity testing addresses whether an outcome of an instrument measures the properties it was intended to measure. Performing content validity tests will determine whether the questions chosen represent the concepts they are intended to reflect [10]. To test content validity of the checkpoints, the rater should select one of these three options for each checkpoint: 1) Essential 2) Useful but not essential and 3) Not necessary. Later, the marked checkpoints will be analyzed using the formula: CVR =!!!!!!! (1) Where CVR= content validity ratio, ne= number of people indicating the checkpoint essential and N= number of people who marked the checklist [11]. Once that data has been calculated, it will be compared to the minimum value of CVR needed for each checkpoint depending upon the number of raters used and if a checkpoint doesn t meet the minimum value than it will be removed from the checklist. Reliability is defined as the probability that a product or a service will operate properly without any problems or failures for a specific period of time under the designed operating conditions [12]. Evaluators trying to evaluate the of the data use mathematical tools to estimate random or systematic error. Random errors occur by chance and they do not impact the consistency of the data, but systematic errors impact some results and occur due to a problem in the checklist. Evaluation of rater is necessary to identify systematic errors and for this, intrarater and inter-rater should be determined. The degree to which ratings by two or more persons vary is called inter-rater [13]. The degree to which two measurements collected by the same person vary from one rating to the next over a period of time is called intra-rater [14]. 3. Method The 5S checklist used for evaluation in this project has five sub-divisions: sorting, storage, shining, standardize and sustaining. The 5S checklist contains a separate section to list potential improvements for the workplace. Each section has a number of questions to use when evaluating 5S in the workplace. There are 28 questions and each question grades out of 5 points. The score of each section can be determined and necessary measures can be taken if the corresponding section has a low score. The total score of the checklist is out of 140 points. Each question has a possible grade between 0 and 5 points. The description for the scores from 0 to 5 is shown in Table 1. Students who had enrolled in a lean manufacturing class during fall 2007 semester at University evaluated a workplace using the 5S checklist. Forty-five student raters participated in the study for a class assignment. The students who participated in the study were trained in 5S techniques and they had initial knowledge about the benefits of 5S tool in lean manufacturing.

3 2 A 1 = A E (Rosner, 1994) Whitman, Jorgensen, Gorrepati Table 1: Description for scores in checklist (Cessna supplier 5S audit checklist) Score Description 0 Workplace is unacceptable or zero effort is done 1 Activity has started and slight effort has been put 2 Widespread activities have been performed and there are many opportunities for improvement 3 Minimum acceptable level and the system has been sustained for at least one month 4 Best in class and the system has been sustained for three months 5 World class example and the system has been sustained for a minimum of six months The 5S checklists were given to the students. A virtual reality factory model was created using Quest software and was given to the students to observe the workplace. The simulation model had a worker performing normal tasks in the assigned workplace. Students were asked to evaluate the workplace using the 5S checklist. The students were reassured that any information revealing their personal identity would be kept confidential and would be known only to the researchers to avoid the Hawthorne effect [15]. The completed checklists were then collected back from the students. In order to determine the intra-rater which represents how reliable the checklist is within the same user over a period of time, a similar procedure was followed after 6 months. The same procedure was repeated again with a time gap [16]. The data was collected from the students for the second time after 6 months. The data was collected electronically via an to all the student raters who participated in the first trial. The consent form, virtual reality factory model and the checklist was ed to the participating students. Students who were willing to participate in the study completed the checklists and ed them back. A total of 21 students took part in the study for the second trial. The names of the students were used to identify the rater s score before and after the 6 month time interval. Inter-rater can be determined, for an evaluation checklist with multiple scores for each checkpoint, by estimating the within person variance and between person variance. According to Fleiss [17] the inter-rater can be calculated using the formula: (MS between - MS within ) Inter - rater = (MS between + (K - 1)MS within ) (2) Where, K is the degree of freedom, MSbetween is the unbiased estimator of between person variance and MSwithin is the unbiased estimator of within person variance that are obtained as SPSS outputs. Similarly Intraclass correlation coefficient represented as ρ1 can be calculated using the mathematical formula by Rossner [18]: ρ! =!!!!!!!!!! (3) Where, σ2a = between person variance, σ2e = within person variance that are obtained as SPSS outputs.

4 4. Results The existing 5S checklist is not reliable to assess the virtual reality factory model due to low inter-rater and intrarater. A summary of the results is shown in table 2. Table 2: Reliability scores of 5S checklist Subdivisions of the 5S checklist Inter- Result Sort Slight Set in order Slight Shine Poor Standardize Slight Sustain Slight rater- Intrarater Result Fair to good Poor Poor Poor Poor The following guidelines were used to verify the results: Table 3: Guidelines for inter-rater and intra-rater [19, 20] Guidelines for inter-rater Guidelines for intra-rater Kappa scores < 0 Reliability of the rating Almost perfect Substantial Moderate Fair Slight Poor Intraclass correlation coefficient ( ρ ) 1 < >0.75 Reproducibility Poor Fair to good Excellent In general, the reasons for low scores of checklists are the following: checkpoint complexity, lack of specificity to focus and the length of the checklist [15]. In some companies, differences in the culture, norms, policies or organizational climate among workers can produce low scores [21]. Other factors can play a critical role in determining a score: the ability of the rater to rate any given material depending upon the intelligence of the rater [22], the rater s knowledge of the performance dimensions [23] and the training received by the rater [24]. Content validity also plays an important role in determining the [25]. Next, content validity was performed on the existing 5S audit checklist. Five industry experts who had a minimum of five years experience implementing 5S and working with 5S audits were selected. The expert group performed content validity and each individual marked that all the checkpoints of the 5S audit checklist remained essential. In the comments section, however, they wrote that some checkpoints were too complex to understand because they contained too much content. For example, the experts commented that checkpoint 2.2 contains too much content which could confuse rater while performing audits. Checkpoint 2.2 states that WIP, tools, materials, and products are not left to sit directly on the floor. Large items such as tote bins are positioned on the floor in clearly marked

5 areas, delineated by painted lines. By splitting the checkpoint into two separate items, it reduces the chance of ambiguity for the raters as each checkpoint is now clearly defined with less information in it. The modification marked the first checkpoint as specifically about items on the floor WIP, tools, materials, and products are not left to sit directly on the floor and the second checkpoint as concerning certain items location large items such as tote bins are positioned on the floor in clearly marked areas, by painted lines. By taking the experts other comments into consideration, the existing checkpoints were modified and the modified checklist is shown in Appendix A. 6. Conclusion and Future work The objective of this paper was to test the rater of an audit checklist. The paper also proposes a revised checklist. The authors believe that training the raters before performing the 5S audit checklist will improve scores. The significance of this work is seen in the improved 5S checklist that can be repeatable. Future work planned includes validating the modified checklist and the training techniques in a real world shop floor instead of using the virtual reality factory model, as the model does not answer all the checkpoints correctly. The results were obtained with a group of students and not industry professionals. The raters of the control group and the experimental group were assumed to be of the same demographics, but the students were not randomly selected. The raters were asked to maintain notes and refer to it while rating for the second time and the training technique could have inflated the intra-rater score as the workplace in the virtual reality model would not have any changes when rating for the second time. This particular training technique is works well in a real world scenario. References 1. Ablanedo-Rosas, J. H., Alidaee, B., Moreno, J. C., and Urbina, J., 2010, "Quality Improvement Supported by the 5S, An Empirical Case Study of Mexican Organisations," International Journal of Production Research, 48(23), Womack, J. and Jones, D., 1996, Lean Thinking, Banish Waste and Create Wealth in your Corporation, New York: Simon & Schuster. 3. Russell, R. and Taylor, B. W., 2990, "Operations Management: Creating Value Along the Supply Chain, Hoboken," ed: NJ: John Wiley & Sons. 4. Martz, W., "Validating an Evaluation Checklist using a Mixed Method Design," Evaluation and Planning Program, 33(3), Borman, C. W., 1975, "Effects of Instructions to Avoid Halo Error on Reliability and Validity of Performance Evaluation Ratings," Journal of Applied Psychology, 60(5), Pavnaskar, S., J. Gershenson, and Jambekar, A., 2003, "Classification Scheme for Lean Manufacturing Tools," International Journal of Production Research, 41(13), Chapman, C. D., 2005, "Clean House with Lean 5S," Quality Progress, 38(6), Osada, T., 1991, The 5S: Five Keys to a Total Quality Environment. Tokyo: Asian Productivity Organization. 9. Kobak, K. A., Engelhardt, N., and Lipsitz, J. D., 2006, "Enrich Rater Training using Internet Based Technologies: A Comparison to Traditional Rater Training in a Multi-site Depression Trial," Journal of Psychiatric Research, 40(3), Godfrey, J., Hamman, R., Lowenstein, S., Briggs, K. and Kocher, M., 2007, "Reliablity, Validity, and Responsivness of the Simple Shoulder Test: Psychometric Properties by Age and Injury Type," Journal of Shoulder and Elbow Surgery Board of Trustees, 16(3), Lawshe, C. H., 1975, "A Quantitative Approach to Content Validity," Personnel Psychology, 28(4), Elsayed, E. A., 2000, "Invited Paper: Perspectives and Challenges for Research in Quality and Reliability Engineering," International Journal of Production Research, 38(9),

6 13. Bortolotti, T., Danese, P., and Romano, P., 2013, "Assessing the impact of Just-In-Time on Operational Performance at Varying Degrees of Repetitiveness," International Journal of Production Research, 51(4), Thompson, C. B. and Barrett, L. S., 2007, "Intra-rater/Inter-rater Reliability of the Air Transport Minimum Data Set," Air Medical Journal, 26(3), Zenk, S. N., Schulz, A. J., Mentz, G., House, J. S., Gravlee, C. C., Miranda, P. Y., et al., 2007, "Inter-rater and Test-retest Reliability: Methods and Results for the Neighborhood Observational Checklist," Health and Place, 13(2), Halpern, M., Hiebert, R., Nordin, M., Goldsheyder, D., and Crane, M., 2001, "The Test-retest Reliability of a New Occupational Risk Factor Questionnaire for Outcomes Studies of Low Back Pain," Applied Ergonomics, 32, Fleiss, J. L., Levin, B., and Paik, M. C., 2004, "The Measurement of Interrater Agreement," in Statistical Methods for Rates and Proportions, ed: John Wiley & Sons, Inc., pp Rosner, B., Donner A., and Hennekens, C. H., 1977, "Estimation of Interclass Correlations from Familial Data," Applied Statistics, 26(2), Fleiss, J. L., 1986, The Design and Analysis of Clinical Experiments. New York: Wiley. 20. Landis, J. R. and Koch, G. G., 1977, "The Measurement of Observer Agreement for Categorical Data," Biometrics, 33(1), Glisson, C., and Hemmelgarn, A., 1998, "The Effects of Organizational Climate and Interorganizational Cooperation on the Quality and Outcomes of Children's Service System," Child Abuse & Neglect, 22(5), Spool, M. D., 1978, "Training Programs for Observation of Behavior: A Review," Personnel Psychology, 31(4), Ilgen, D. R., 1983, "Gender Issues in Performance Appraisal: A Discussion of O'Leary and Hanson," in Performance Measurement and Theory, , Wherry, R. J., and Bartlett, C. J., 1982, "The Control of Bias in Rating: A Theory of Ratings," Personnel Psychology, 35(3), Aday, L. A., and Cornelius, L. J., 2006, Designing and Conducting Health Surveys: A Comprehensive Guide: Jossey-Bass.

7 0 Unacceptable Zero Effort (5 or more mistakes) 1 Activity Started Slight Effort (4 mistakes) Whitman, Jorgensen, Gorrepati Appendix A: Revised Checklist Table A-1: Modified 5S Audit Checklist 2 Widespread Activity Many Opportunities for Improvement (3 mistakes) 3 Minimum Acceptable Level Sustained for at Least One (1) Month (2 mistakes) 4 Best in Class Results Sustained for Three (3) Months (1 mistake) 5 World Class Example Sustained for at Least Six (6) Months (0 mistakes) DESCRIPTION EVALUATION CRITERIA SCORE 1S. Sorting (Clearing Up) 1.1 First Impression Overall Your general impression should tell you this is the best you have seen for a factory using similar processes. 1.2 Removing Unnecessary Items All items not necessary for performing work are removed from the workplace. 1.3 sorting frequency Sorting exercise is performed in each work area at least on monthly basis 1.4 Tools placement Only tools and products are present at the workstations. 1.5 Bulletin Boards No outdated, torn, or soiled announcements are displayed. All bulletins are arranged in a straight and neat manner. 1.6 Red Tagging Red tagging is performed on the shop floor on regular basis, at least monthly. 2S. Storage (Organizing) 2.1 Shelves, Benches, and Desks Arranged 2.2 Shelves, Toolboxes, Benches, and Desks: Control Items are arranged, divided, and clearly labeled such that it is obvious where things are stored and where they should be returned. These are kept free of unused objects, including files and documents. Tools and fixtures are stored in dedicated shadow boxes or hanging, and are cleaned before being returned to proper position after use. 2.3 Items on Floor WIP, tools, materials, and products are not left to sit directly on the floor. 2.4 items location Large items such as tote bins are positioned on the floor in clearly marked areas, by painted lines. 2.5 General Storage Storage of boxes, containers, and material is always neat. 2.6 Arrangement of items When items are stacked, they are never out of alignment, or in danger of falling over. 2.7 Equipment: Cleanliness Nothing is placed on top of machines, cabinets, and equipment. and Organization Nothing leans against walls or columns. 2.8 Unobstructed Aisles and Aisles are free of material and obstructions. Nothing is ever Access placed on the lines. 2.9 Aisle Markings Aisles and walkways are clearly marked and can be identified at a glance. Lines are straight, with no chipped or soiled paint.

8 2.10 Emergency Access Fire hoses, extinguishers, and other emergency equipment are unobstructed and stored at locations that are clearly marked and highly visible for easy operation Documents: Storage Only documents necessary to do the work are stored at workstations. Documents and binders are stored in a neat and orderly manner Equipment storage Tools, jigs, and fixtures are arranged neatly and stored in a way that they are kept clean and free of any risk of damage Equipment: Maintenance 2.14 Tools and Gases: Arrangement Controls for machines are properly labeled. Critical points for daily maintenance checks are clearly marked (fluid levels, etc). Checklists are neatly displayed and kept clean and updated. Tools are located for easy access for changeovers and set-ups. Gages and measuring equipment used to monitor quality are also arranged as above. 3S. Shining (Cleaning) 3.1 Floor Cleaning All floors are clean and free of debris, oil, and dirt. Cleaning of floors is done regularly, daily at minimum. 3.2 Equipment: Painting All machines and equipment are painted and look clean. There are no places in the plant less than six feet high that are unpainted. 3.3 Equipment: Cleanliness Machines and equipment are constantly kept clean and unsoiled. Routine daily care keeps glass, work surfaces, and the general area clean and polished. Guards and deflectors are used to keep chips and waste from falling on the floor. 3.4 Storage of Cleaning Equipment and Supplies All cleaning equipment is stored in a neat manner. It is obvious where it belongs and is easily available when needed. Hazardous materials and containers are properly labeled. 4S. Standardize 4.1 Visual Control Display boards are present in each production work area and are within sight of all operators in the area. 4.2 Weekly Audits 5S audits are done in each work area at least weekly with the results recorded on the work area s radar chart. 4.3 Status Displays Work area display boards include at least the weekly 5S audit checklist results (radar charts), Kaizen activities performed or identified, layout or chart of 5S responsibilities in the plant. 4.4 Continuous Improvement A document is being published at least once in every month which includes the Kaizen activities being performed in the previous cycle and Kaizen activities identified newly along with the person and department in which it has been identified. 5S. Sustaining (Training and Discipline) 5.1 Maintenance Maintenance and production resources are adequately deployed to keep equipment running properly; an effective preventative maintenance program is in place. 5.2 Documents: Control All documents are labeled clearly as to contents. Responsibility for control and revisions is clear. No unlabeled binders/documents are present. Obsolete or unused documents are taken care of on regular basis. 5.3 Area 5S Responsibility Each work area of the plant should fall under the responsibility of a person on the plant manager s staff as indicated on the plant 5S responsibility layout that shall be prominently displayed throughout the plant.

9 5.4 5S Control and Sustaining Whitman, Jorgensen, Gorrepati There is a disciplined system of control and maintenance to assure that each of the above items is maintained at the highest possible level. It is the responsibility of management to maintain this system. 5.5 Work Area Visits The responsible plant manager staff person should visit each work area at least weekly and initial / date the 5S audit checklist (radar chart) results; additionally, each overdue Kaizen Newspaper action item should be initialed by this person. It is also recommended that the plant manager visit each work area at least monthly and initial / date the radar chart.