New method for assessment of raters agreement based on fuzzy similarity

Transcription

1 New method for assessment of raters agreement based on fuzzy similarity Magdalena Diering 1, Krzysztof Dyczkowski 2, Adam Hamrol 1 1 Chair of Management and Production Engineering Faculty of Mechanical Engineering and Management Poznań University of Technology Pl. M. Sklodowskiej-Curie 5, Poznań, Poland magdalena.diering@put.poznan.pl 2 Department of Imprecise Information Processing Methods Faculty of Mathematics and Computer Science Adam Mickiewicz University Umultowska 87, Poznań, Poland chris@amu.edu.pl Abstract. In measurement systems one of the components affecting its variation is a human factor. Man - as a process operator - measures or rates the product. Since his decisions may have significant impact on the customer satisfaction, his reliability and usefulness for the measuring tasks should be evaluated. This article describes authors proposal for a new measurement system analysis methodology for unmeasurable features. In this method many features of the product are rated during one study, and the value of them can be expressed in a nominal or an ordinal (with imprecise data) measurement scale, and each of the features can be weighted (as less or more important from the customer point of view). The goal of the methodology is to gain information about raters ability to define the value of each feature in relation to the specification and customer requirements. The authors propose to use a similarity measure for rates assignment of product features values. The method is based on the new fuzzy similarity coefficient SC, and the level of agreement of the final decisions of the raters is analyzed based on Gwet s AC 1 coefficient. Keywords: measurement system analysis (MSA), level of agreement, unmeasurable characteristic, similarity coefficient, AC 1 coefficient, fuzzy sets, similarity measure, cardinality of fuzzy sets 1 Introduction Making a decision by operators about the compatibility of manufactured products with customer requirements is executing a primary function of industrial measurement systems for unmeasurable characteristics. Depending on the rank of the feature, the decisions may be more or less important and have significant impact on the effects. Since each operator s decision pulls behind a particular

2 2 M. Diering, K. Dyczkowski, A. Hamrol benefit or loss to the organization, understanding the variation resulting from the interaction of the measurement system components and the evaluation of its usefulness are fundamental in solving the basic problems in the production process [1]. Hence, the industrial measurement systems should be analyzed in order to confirm their adequacy for the measuring tasks and reliability of the data obtained from measuring process. In a study of measurement systems there are applicable statistical methods and mathematical models. In every case, they should be accordingly selected, i.e. with taking into account the specificity of the manufacturing process, and also the measurement techniques and technologies. Analysis of the measurement system should - in the authors opinion - give an answer to at least some of the questions: What is the level of the type I error of each rater (as the rater the authors define a man who measures or rates the product or its features, that is manufacturing process operator, quality controller or expert; type I error means that conforming part is consistent wrongly rejected)? What is the level of the type II error of each rater (type II error - nonconforming part wrongly accepted)? What is the level of contradictory rates/decisions or ambiguous rates/decisions for the each rater? What is the level of agreement of each individual rater with an expert (raterexpert reliability)? What is the level of agreement between operators (inter-rater reliability)? What is the internal agreement of each rater (taking into account the accidental rates which are called chance agreement) that is the repeatability of raters assessments (internal/within-rater reliability)? Which of the features turned out to be the most difficult to rate to? On this basis, corrective, preventive and improving actions should be taken. Next, as a further step, the more important questions should be asked to quality managers [2]: How are we doing - are we getting better or worse? Did the changes we made improve measurement system reliability? What s not working as it should? Which raters (operators and/or experts) need some help? Where are the largest opportunities for improving measuring process? What are the bottlenecks or limiting factors in our measurement system? 2 Research problem In practice, in the case of measurement systems for unmeasurable characteristics, the best known are: the cross-tab method (based on Cohen s Kappa coefficient), and effectiveness study. Usually the companies operate methodology recommended by the AIAG group [3]. The main limitation of these methods

3 New method for assessment of raters agreement based on fuzzy similarity 3 is that they are presented as applicable to be used only for assessing decisions about the characteristics of the nominal scale and precise data. The level of agreement of the final decisions of operators is analyzed without analyzing the similarities of their assessments in relation to the each features of the product. In addition, cross-tables do not include the level of internal agreement of the each raters (repeatability), and effectiveness study ignores the so-called chance of agreement [4]. Meanwhile, in assessing unmeasurable characteristics of the product in practice various measuring scales are used (depending on the situation, the nature of the manufacturing process and customer acceptance criteria). Besides, in decision-making about the state of the product operators rates and assess many of its features, and raters information about their assessment can be soft (incomplete, imprecise or uncertain). For example, the raters opinion during visual inspection of the color of the produced items may be formulated (depending on the specifications of the product) as follow: a) The color is correct or incorrect, that is assessment with a dichotomous nominal scale. In mathematical models one of the categories is assigned a value of 1 (1 - meets the requirements, conforming part), and second - value of 0 (0 - does not meet the requirements, nonconforming). b) The color is very dark, dark, light, very light, that is the assessment with an ordinal scale (in the case of the color most often it is assessed by comparison with the color of a master (standard) part). In mathematical models categories are graduated and linguistic categories are described by membership function values of fuzzy sets i.e values from interval [0, 1], for example: very dark - 0.9, dark - 0.7, light - 0.4, and very light The situation may be even more complicated if, for example, the color of the front side of the product have a greater importance to the client (if it affects the aesthetic qualities of the final product) than the color of the back side of the same element (which may be negligible upon receipt of goods). Importance (weights, priorities) of the different products features affect the way of evaluation by the operators. The weights of the features - according to the authors - should also be taken into account in assessing the reliability of the measurement system, and therefore in the mathematical model as well. Both the cross-tab method and the effectiveness study in many cases do not allow to reflect production conditions to evaluate measurement system in its natural environment. This may have a huge impact on the quality and the effect of the whole study. What is more, changed conditions of the study (in comparison to production conditions) can cause abnormal raters behavior during the study, and the wrong decisions of operators, wrong decisions of executive engineer (expert), discouraging raters to study and others. In summary, these methods do not cover the demand process engineers in the analysis and assessment of the reliability of the systems for unmeasurable features. However, statistical engineering and mathematical sciences provide many different models and approaches to assess the level of rater s agreement with reference value (in practice - with experts decision), but - because of their various limitations - only few of them have been

4 4 M. Diering, K. Dyczkowski, A. Hamrol successfully implemented for engineering applications. They are used in various analysis by analysts and statisticians, but are not in common use as part of measurement system study. Hence, the authors undertook research the aim of which is to develop not only models, but also methodologies for the analysis of industrial measurement systems for unmeasurable characteristics, taking into account the different production conditions. This article describes a proposal for a measurement system assessment methodology in which many features of the product are rated, and the value of these features can be expressed in nominal or ordinal scale, and each of the features can be weighted (as less or more important from the customer point of view). The goal of the authors is to build a useful tool for engineers. Thus, the methodology will be validated with practitioners and quality engineers. 3 Fuzzy method for measure raters agreement 3.1 A characteristics of the approach For industrial measurement system analysis for unmeasurable features the authors propose an original methodology. The goal of the methodology is to take into account not only the final decision about the quality of measured product, but also to gain the information about raters ability to define the value of each feature according to the experts requirements. The authors propose to use a similarity measure for rates assignment of product features values. The new method is based on the similarity coefficient SC (which is described in details in section 3.2 of the paper). The novel authors methodology allows to rate many features of the product, and the value of these features can be expressed in nominal or ordinal scale (optional for each feature). The authors model is applicable to the cases: multiple raters (at least one), multiple unmeasurable features (at least one) in the nominal scale or ordinal measurement scale (and it is possible to express the value of the feature using imprecise data), any number of parts (in practice it is accepted that it is at least 30 elements), multiple series of ratings (1-3). Thus, the study scheme (defined as the selection of the elements participating in the study) is flexible and better - in comparison with commonly used in the practice the cross-tab method - to reflect the conditions in which every day measurements are taken and thus, to obtain accurate information about the reliability of the measurement system used. Besides, the framework of the proposed methodology includes assessing the level of internal agreement of each raters, that is their ability to repeat the same ratings in the following series of assessments (because it would be questionable to analyze the level of agreement between raters if one may not agree with himself). As in previous studies that were not taken into account, in the context of engineering applications, this approach is novel and unique. The characteristics of the model proposed by the authors is described by comparison to other approaches (Table 1). In this comparison the most commonly are presented: cross-tab method, the effectiveness study (often used in parallel with the cross tables), another author s method based on the basic Gwet a agreement coefficient AC 1 (details of which

5 New method for assessment of raters agreement based on fuzzy similarity 5 are discussed in the previous authors work [5]) and the new authors method with similarity coefficient SC. Besides the comparison of the number of raters, or the number of trials (series) of the assessment, the table includes several other criteria, among other: the number of assessed features of the product, a measurement scale adopted for each feature individually, importance of each features (weights, priorities), the level of raters internal agreement and others. Table 1. Methods for measure raters agreement - comparison of chosen approaches No. Comparison criteria Effectiveness Study [3] Cross-Tab Method [3, 6] 1. Number of raters Number of series (trials) for each operator Feature s measurement scale Number of categories (possible values) of single feature assessment Number of categories (possible values) of single decision (final decision about the product) Number of features in the study The authors method based on K win coefficient [5, 4] Any number (no limit) The new authors method based on SC coefficient Any number (no limit) Nominal dichotomous Nominal dichotomous (identical to the categories of single feature assessment) 1 (assessment of a single feature or decision about the final product) 2 (identical to the categories of single feature assessment) 1 (assessment of a single feature or decision about the final product) 7. Nature of the data Precise (crisp) Precise (crisp) Acceptance criteria for the system Taking into account the importance of assessed features (the ability to give weights, priorities) Taking into account the level of internal agreement of the raters % Effectiveness, Miss Rate, False Alarm Rate Kappa [ 1, 1] (in practice [0, 1]) Nominal Any number (no limit) Any number (no limit, but identical to the categories of single feature assessment) 1 (assessment of a single feature or decision about the final product) Precise (crisp), but taking into account the lack of data (incomplete/missing data is possible) K win [0, 1] Nominal, Ordinal Any number (no limit) Any number (no limit; may be different than the categories of single feature assessment) Any number (possible to assess the many features of the product and the decisions about the final product) Precise (crisp) (for nominal scale) and/or imprecise (soft) (for ordinal scale) SC [0, 1] and K win [0, 1] Not applicable Not applicable Not applicable Yes Yes (but without taking into account chanceagreement) Not applicable (in practice it is recommended to provide cross-tab method parallel with the effectiveness study) Yes Yes

6 6 M. Diering, K. Dyczkowski, A. Hamrol 3.2 The model In our model we use fuzzy sets concept to represent ratings. Fuzzy sets theory was introduced by Zadeh in 1965 [7] and since then, has been widely used in many fields of research. There are a lot of literature on theory and application of fuzzy sets (eg. [8, 9, 10]). More information on cardinality and similarity of fuzzy sets used in this paper one can find in [11, 12, 13] Formally as an input data for the problem analysis is as follows: set of raters: R = {r 1,, r l }, where l is a number of raters set of rated products (or parts): P = {p 1,, p k }, where k is a number of rated products set of rated features (or attributes) A = {a 1,, a m }, where m is a number of rated products features set of series of ratings S = {s 1,, s n }, where n is a number of series of ratings set of features weights (which represents importance of each feature): W = {w 1,, w m } Each rater from set R makes an assessment of each product on each attribute in n series. The expert does the same in one series ie. rates all features from set A for all products from set P. The rating of k-th product p k done by expert is represented by fuzzy set E pk : E pk = oe 1 /a 1 + oe 2 /a oe m /a m, where oe i [0, 1] is rating of expert for i-th attribute. The rating of k-th product p k done by r-th rater (taking into account n series of ratings) is represented by fuzzy set R r,pk : R r,pk = a i A s j S or ij /(a i, s j ), where or ij [0, 1] is a rating of r-th rater for i-th attribute in j-th series. A similarity Sim p between expert and r-th rater of rating of p k -th product is defined by equation: Sim p (E pk, R r,pk ) = 1 n m i=1 (w i oe i or ij ) n m i=1 w. (1) i j=1 We compute the similarity Sim p for each product and receive as a result fuzzy set C r which represents agreement between expert and r-th rater on all products: C r = Sim p (E 1, R r,1 )/p 1 + Sim p (E 2, R r,2 )/p Sim p (E k, R r,k )/p k. Fuzzy set C r represents information how much expert agreed with given rater on every product.

7 New method for assessment of raters agreement based on fuzzy similarity 7 As a measure of agreement between expert and given rater we take cardinality card t (C r ) of fuzzy set C r with weighting function f t : card t (C r ) = k f t (C r (p i )). (2) i=1 For more information on theory of fuzzy sets cardinalities and construction of weighting functions see [11, 13, 10]. The role of weighting function f t : [0, 1] [0, 1] is to decide with given threshold point t [0, 1] when to treat agreement level as sufficient to be counted. The authors propose to use the following weighting function: { x if x t, f t (x) = (3) 0 otherwise. We can interpret card(c r ) as a number of products rated similar between expert and given rater (ie. cardinality of fuzzy set of expert and raters similarities). Finally we get rating agreement coefficient SC r for each rater: SC r = card(c r). (4) l Iteratively we compute rating agreement coefficients for all raters. Based on Table 1 and on the basis of formulas 1, 2, 3, 4, it can be concluded that the model proposed by the authors allows - as the only from the wellknown in the literature approaches - to analyze the ratings of the many features (attributes) of the product in one study. The model takes into account the weight of each characteristics and its nature (the model allows to measure the level of agreement with imprecise data, also includes a measuring scale individually adopted for each feature). 3.3 The methodology For the model described in sections 3.1 and 3.2, a novel authors methodology is proposed (in order to engineering applications): 1. To undertake a study at least 30 parts (products, subjects) should be prepared (in the literature can be found different recommendations, usually parts). About 50% of the subjects should be out of the specification and at least 50% of all subjects should be taken from the border of the specification (from the grey area ), ie. should be difficult (hard) to clearly assess. In the study should be parts which - in experts opinion - can be easy to rate (in analogy to measurable characteristics - parts from the middle of tolerance), and those that will be difficult in the assessment (parts close to the limits of acceptance). Analysis of the results of those difficult points to opportunities for improvement and the analysis of these easy - for necessary of corrective and remedial actions in the area of measurement system.

8 8 M. Diering, K. Dyczkowski, A. Hamrol 2. Prior to the study, make sure that the executive expert has his within-rater reliability on the accepted level (to make sure that experts decisions are reliable source of information). Expert rates are treated as reference values and master to the operators ratings. Meanwhile, in practice it turns out that the expert also can make a mistake (and sometimes he does!) in the evaluation of the products. Experts internal-rater reliability can be estimated, for example, based on the coefficient AC 1, but with the note that instead of number of raters there is a number of series of the same operator ratings [4, 5]. 3. As with all known methods of measurement systems analysis, including the methods proposed by the authors, each subject should be marked and easy to identify by executive (but not by the raters!) - conditions of independence results should be assured. 4. The study should be carried out in the place in which the measurement system is used in practice, that is with the standard (daily) environment and measurement conditions. 5. During the study, the executive person should write operators results in the data collection sheet (Table 2). Operators rate each characteristic of each object using nominal or ordinal measurement scales, and then make a decision about the product, classifying it into one of at least two categories. Table 2. A worksheet (with sample data) for measurement system study (for unmeasurable characteristics), according to the model described in section 3.2 (r i - rater, e - expert, a i - feature, w i - fetures weight, s i - series, p i - part). r 1 r 2 r 3 e a 1 a 2 a 3 a 1 a 2 a 3 a 1 a 2 a 3 w 1 w 2 w 3 w 1 1 w 2 1 w w 1 1 w 2 1 w w 1 1 w 2 1 w s 1 s 2 s 3 s 1 s 2 s 3 s 1 s 2 s 3 s 1 s 2 s 3 s 1 s 2 s 3 s 1 s 2 s 3 s 1 s 2 s 3 s 1 s 2 s 3 s 1 s 2 s 3 a 1 a 2 a 3 p p p p The completed worksheet should be computed according to the model described in section Evaluation of the level of raters agreement should be based on the acceptance criteria. Benchmark scale is shown in Table 3. Table 3. Level of raters agreement - acceptance criteria Benchmark scale Level of agreement 0 SC r < 0.50 Poor 0,.50 SC r 0.80 Good 0.80 < SC r 1 Very Good

9 New method for assessment of raters agreement based on fuzzy similarity 9 8. After analyzing the similarities in all parts, the measurement system reliability study for the raters decisions about the product can be performed. Here, the applicable is a method proposed in [5], based on the coefficient AC 1 [4]. If evaluation of the raters ability to assign the proper value to the characteristics (compared to the customer s specification or expert) is at acceptable level (SC is greater than 0.5), then the assessment of their level of decision agreement with the expert may be studied. For this purpose, the value of K win (cf. [5]) is applicable: where: l i=1 K win = AC 1K w l i=1 K, (5) w K win aggregated measure of the level of internal raters agreement (withinrater reliability) and the level of agreement between the raters (inter-rater reliability), AC 1 Gwet s AC 1 coefficient (cf. [4]), K w Kappa type coefficient, which measures level of internal agreement of each rater (within-rater reliability); to calculate K w the AC 1 is used, where the number of raters replaces the number of repeated series of ratings by the same rater. Interpretation of results should allow to answer questions posed in section An example To show the procedure of computation as an input we take sample data form Table 2. In the real case, in accordance to the methodology, the number of elements would be much greater. Computations will be done in three steps: 1. In first step we compute similarity Sim p resulting fuzzy sets of similarities for each rater: C 1 = 0.68/p /p /p /p 4, C 2 = 0.68/p /p /p /p 4, C 3 = 0.87/p /p /p 3 + 1/p Next we compute cardinalities of fuzzy sets C r with threshold t = 0.7: card t (C 1 ) = 2.69, card t (C 2 ) = 3.67, card t (C 3 ) = Finally we get similarity coefficient SC r for all raters: SC 1 = 0.67, SC 2 = 0.92, SC 3 = 0.94.

10 10 M. Diering, K. Dyczkowski, A. Hamrol The level of agreement in assessing each attribute (feature) of the product for each of the raters with the expert is at the acceptable level. It means that raters knowledge (know-how) about features and the product (about its variation) is reliable source of information. Thus, the next step of the measurement system analysis may be assessment of the level of agreement of raters decision with the expert. An example of this computation one can find in [5]. 4 Conclusions Proposed novel model is the basis for the authors to build an expert assessments supporting system based on fuzzy models. In summary, the result of the authors work is presented in the article model which is an original, flexible solution that can be used in the evaluation of any industrial measurement system for unmeasurable characteristics (for the characteristics of the nominal or ordinal measuring scales and imprecise data). In order to engineering applications, for the described model the authors methodology is proposed. References [1] Hamrol, A.: Quality management with examples (in Polish: Zarządzanie jakością z przykładami). Polish Scientific Publishers PWN (2008) [2] Kaydos, W.: Operational performance measurement: increasing total productivity. CRC Press (1998) [3] AIAG Work Group: Measurement Systems Analysis, 4th ed., Reference manual, AIAG Work Group, Daimler Chrysler Corporation, Ford Motor Company, General Motors Corporation. (2010) [4] Gwet, K.L.: Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC (2014) [5] Diering, M., Dyczkowski, K., Hamrol, A.: Estimating the level of assessments agreement in visual inspection - the problems in determining kappa coefficients (in Polish: Szacownaie poziomu zgodności ocen w kontroli wizualnej - problemy w wyznaczaniu współczynników typu kappa). In: Innovation in Management and Production Engineering. (2015) [6] Cohen, J.: A coefficient of agreement for nominal scale. Educational and Psychological Measurement 20(1) (1960) [7] Zadeh, L.A.: Fuzzy sets. Information and control 8(3) (1965) [8] Klir, G., Yuan, B.: Fuzzy Sets and Fuzzy Logic: Theory and Applications. Prentice Hall (1995) [9] Dubois, D., Prade, H.: Fundamentals of fuzzy sets. Volume 7. Kluwer (2000) [10] Wygralak, M.: Intelligent Counting Under Information Imprecision: Applications to Intelligent Systems and Decision Support. Springer (2013) [11] Dyczkowski, K.: A less cumulative algorithm of mining linguistic browsing patterns in the world wide web. In: EUSFLAT Conf. (2007) [12] Stachowiak, A., Żywica, P., Dyczkowski, K., Wójtowicz, A.: An interval-valued fuzzy classifier based on an uncertainty-aware similarity measure. In: Intelligent Systems Springer (2015) [13] Wygralak, M.: Cardinalities of fuzzy sets. Springer (2003)