OUTLIERS AND SOME NON-TRADITIONAL MEASURES OF LOCATION IN ANALYSIS OF WAGES

Transcription

1 Europea Scietific Joural September 0 /SPECIAL/ editio Vol. ISSN: (Prit) e - ISSN OUTLIERS AND SOME NON-TRADITIONAL MEASURES OF LOCATION IN ANALYSIS OF WAGES Mila Terek, Prof., PhD Matúš Tibeský, Mgr. Uiversity of Ecoomics i Bratislava, Departmet of Statistics, Slovakia Abstract The paper deals with a aalysis of how to use certai measures of locatio i aalysis of wages. Oe of traditioal measures of locatio the mea should to offer typical value of variable, represetig all its values by the best way. Sometimes the mea is located i the tail of the distributio ad gives very biased idea about the locatio of the distributio. The removig of outliers, if ay, or usig of differet measures of locatio could be useful i these cases. Outliers are characterized ad some robust methods of their detectig are described i the paper. The the trimmed mea ad M-estimators are characterized. Computig of oe-step M-estimator ad modified oe-step M-estimator of locatio is described. The possibilities of usig these tools are illustrated o the aalysis of the gross yearly wages of employers of oe Slovak firm i the year 03. Keywords: Detectig outliers, trimmed mea, oe-step M-estimator, modified oe-step M- estimator, aalysis of wages Itroductio The distributio of wages is obviously skewed ad outliers are preset. The, the iterpretatio power of the mea is very small 3 It will be show that the removig outliers or alteratively, usig of some o-traditioal measures of locatio could be iterestig. Outliers are defied as uusually large or small values. Sometimes itroductio of outliers ito the ivestigatios ca lead to the loss of iterpretatio power of results, so the methods of their detectio are applied. Sometimes the usig of some o-traditioal measures of locatio is appropriate. Oe from these measures is trimmed mea. Trimmed mea refers to a situatio where a certai proportio of the largest ad smallest values are removed ad from the rest, the mea is calculated. M-estimators provide aother class of measures of locatio that have practical value. Their costructio requires the detectio of outliers. The paper focuses o the aalysis of outliers, descriptio of the trimmed mea ad M-estimators ad o their applicatio i the aalysis of wages. Outliers I almost every series of observatios, some are foud, which differ so much from the others as to idicate some abormal source of error ot cotemplated i the theoretical discussios ad the itroductio of which ito the ivestigatios ca oly serve to perplex or mislead the iquirer (Barett, Lewis, 99). Such observatios are call outliers. We shall defie a outlier i a set of data to be a observatio (or subset of observatios) which appears to be icosistet with the remider of that set of data (Barett, Lewis, 99). 3 More i details i Halley, 00; Terek, 008; Terek, Nguye Dih He, 0. 80

2 Europea Scietific Joural September 0 /SPECIAL/ editio Vol. ISSN: (Prit) e - ISSN What characterizes the outlier is its impact o the observer ot oly it will appear extreme but it will seem, i some sese, surprisigly extreme. Outlyig observatios are ot ecessarily bad or erroeous. There are the situatios i which a outlier ca idicate for example some uexpectedly useful idustrial treatmet. Frequetly, outliers are very useful i the fraud recogitio. I the situatios like these, it may ot be ecessary to adopt either of the extremes: of rejectio (with a risk of the loss geuie iformatio) or iclusio (with the risk of cotamiatio). Sometimes the usig of the robust methods of iferece which employ all the data but miimize the ifluece of ay outliers is useful. The detectio of outliers requires the assessig the itegrity of a set of data. Detectig outliers The first strategy is based o the sample mea ad sample stadard deviatio. If the ormal distributio of the populatio is supposed, it is obvious to cosider as outlier a value which is more tha, stadard deviatios σfrom the mea µ: x µ >, σ Geerally µ ad σ are ot kow but they ca be estimated from the data usig the value of the sample mea x ad of the sample stadard deviatio s. The the followig decisio rule ca be formulated: The value x is declared to be a outlier if x x >, () s where s = ( x j x) j= The described method ca lead to the problem kow as maskig. Outliers iflate both the sample mea ad the sample stadard deviatio, which i tur ca mask their presece whe usig equatio () (Wilcox, 003). The rule for detectig outliers that is ot itself affected by outliers is eeded. Two robust methods of outliers detectio will be describe. The method based o quartile rage This method is based o quartile rage R : where R = 3 x 3 x x is the third quartile, x the first quartile The value is outlier if: is greater or equal as ( x +,5 3 is less or equal as ( x,5 R ). R ), The method based o MAD Firstly a measure of dispersio called media absolute deviatio MAD will be described. To compute it, first compute the value x of the sample media / X, the / compute the absolute values of the differeces: 8

3 Europea Scietific Joural September 0 /SPECIAL/ editio Vol. ISSN: (Prit) e - ISSN x i x / for i =,,... Geerally, MAD does ot estimates σ, but it ca be show that whe samplig from a ormal distributio, MAD = 0,675 estimates σas well (Wilcox, 003). The robust decisio rule for the detectio of outliers is followig: The value x is declared to be a outlier if x x / >, () A trimmed mea The value of the trimmed mea is calculated from the data, from which a certai proportio of the largest ad smallest observatios are removed ad the remaiig observatios are averaged. For example 0% trimmed mea is calculated from the data from which 0% of the largest ad 0% ad smallest observatios were removed. A fudametal issue is decidig how much to trim. Whe addressig a variety of practical goals, 0% trimmig ofte offers a cosiderable advatage over ot trimmig ad the media (Wilcox, 003). M-estimators M-estimators are from aother class of measures of locatio. For example, if for ay values X, X,... X we wat to choose c so that it miimizes the sum of squared errors, ( X i c) It ca be show that it must be the case that ( X i c) (3) = 0. From this last equatio c = X. So, whe we choose a measure of locatio based o miimizig the sum of the squared errors give by (3), this leads to usig the sample mea. But if we measure how close c is to the values usig the sum of absolute differeces, the sample media miimizes this sum (Wilcox, 003). Geerally, there are ifiitely may ways of measurig closeess that lead to reasoable measures of locatio. For example, if we measure the closeess by a X i c, the settig a = leads to the media, ad a = leads to the mea. Let us have ay fuctio Ψ havig the property: Ψ( x) = Ψ(x), we get a reasoable measure of locatio, provided the probability curve is symmetric, if we choose c so that is satisfies Ψ( X c) + Ψ( X c) Ψ( X c) = 0 () Measures of locatio based o () are called M-estimators. The calculatio of the M-estimators requires the detectio of outliers. Oe-step M-estimator Let be the umber of observatios X, for which i 8

4 Europea Scietific Joural September 0 /SPECIAL/ editio Vol. ISSN: (Prit) e - ISSN ( X i ) X / < K ad let be the umber of observatios such that ( X i X / ) >K where typically K =,8 is used. The oe-step M-estimator of locatio (based o Huber` s Ψ) is ˆ K ( )( ) + X ( i) + µ = (5) os where X ( i) is i-th order statistic. The calculatio of the value of M-estimator requires the determiatio of outliers usig the method i.., except that () is replaced by x x / >K (6) Next, remove the values flagged as outliers ad average the values that remai. For techical reasos, the oe-step M-estimator makes a adjustmet based o, a measure of scale plus the umber of outliers above ad below the media (Wilcox, 003). A modified oe-step M-estimator Sometimes a simple modificatio of oe-step M-estimator is used: ˆ + X ( i) µ = (7) mom Here, K =, is used to determie ad (Wilcox, 003). Aalysis of wages The possibilities of applicatio of the described tools will be illustrated o the aalysis of the gross yearly wages of 5 employees of the firm Medirex i Slovak republic i the year 03. Table presets the values of some descriptive measures, computed with aid of the software MS Excel 007 (Tibeský, 0). Tab. Descriptive measures of the gross yearly wage Cout 5 Average 9873,68 Media 9,3 Stadard deviatio 538,09 Miimum 9,36 Maximum 55303,78 Rage 538, Order statistic is determied by its rakig i a o-decreasig arragemet of radom variables. 83

5 Europea Scietific Joural September 0 /SPECIAL/ editio Vol. ISSN: (Prit) e - ISSN The frequecy distributio is i the table (Tibeský, 0). Tab. Frequecy distributio of the gross yearly wage Gross yearly wage Frequecy I figure is the histogram for the gross yearly wages. Figure Histogram for the gross yearly wages It is evidet from the histogram, that the distributio of wages is highly skewed o the right. It seems that the mea is ot the best measure of the typical wage of employee i the populatio. Detectig ad removig outliers The outliers will be detected with aid of the method based o quartile rage. outliers were detected by this method. Table 3 presets the values of some descriptive measures after removig these outliers (Tibeský, 0). Tab. 3 Descriptive measures of the gross yearly wage after removig of outliers Cout 9 Average 9397,36 Media 9,95 Stadard deviatio 98,9795 Miimum 3398,7 Maximum 5783,95 Rage 385,78 I the table 3 ca be see that the rage ad stadard deviatio decreased. The chage of media is oly small, the chage of the mea is importat. The distace betwee media ad mea is much smaller. Now, the mea better characterizes the typical yearly wage. M-estimators i aalysis of wages Firstly, the detectio of outliers with aid of the method based o MAD is ecessary. The values MAD = 606,7 ad = MAD/0,675 = 38,06 were calculated. The, detectig outliers based o coditio (6) was applied ad the values of M-estimators were 8

6 Europea Scietific Joural September 0 /SPECIAL/ editio Vol. ISSN: (Prit) e - ISSN computed. The values used i the computig of oe-step M-estimator are i table (Tibeský, 0). Tab. Values used i computig ofoe-step M-estimator K ˆ µ = Cout 5 38,06 K,8 3 ( )( ) + X ( i ) i +,8 38,06( 3) = , 6 = os 5 3 The values used i the computig of modified M-estimator are i table 5 (Tibeský, 0). Tab. 5 Values used i computig ofmodified oe-step M-estimator X ( i ) 089 6,5 + ˆ µ mom = = 9 369, The the 0% trimmed mea was computed. All results are i table 6. Tab. 6 Descriptive Measures Mea 9873,68 Media 9,3 0% Trimmed mea 9 375,36 µˆ os 9 78,0 µˆ 9 369,7 mom 9 78,0 Coclusio It ca be see that the detectig ad removig outliers were very useful i the aalysis of wages. The mea ad media became much closer, the variability of the data was much less as before as well. The the traditioal measures of locatio mea ad media ca better characterize the data. I our opiio this is the first possibility for improvig the aalysis of wages ad makig the results more cosistet, givig better idea about the typical wage i the populatio. The secod possibility is the calculatio of some o-traditioal measures of locatio. The value of 0% trimmed mea is 9 375,36 EUR, value of oe-step M-estimator is 9 78,0 EUR, value of modified oe-step M-estimator is 9 369,7 EUR ad value of media is 9,3 EUR i the aalysis. The values of measures are oly slightly differet. Each of these certaily better characterizes typical yearly wage of a employee i aalyzed period as mea, equal to 9873,68 EUR, which is evidetly highly iflueced by a small umber uusually high wages. I our opiio, aalysis of outliers followed by calculatig of traditioal measures of locatio or alteratively the calculatig of some o-traditioal measures of locatio like, trimmed meas ad M-estimators are efficiet tools for obtaiig more real ad true view to the typical wage. Cout 5 38,06 K,

7 Europea Scietific Joural September 0 /SPECIAL/ editio Vol. ISSN: (Prit) e - ISSN This paper was elaborated with the support of the grat agecy VEGA i the framework of the project o. /076/. Refereces: Barett, V., Lewis, T.: Outliers i Statistical Data. New York: Wiley ad Sos, 99. Halley, R. M.: Measures of Cetral Tedecy, Locatio, ad Dispersio i Wage Survey Research. I: Compesatio ad Beefits 00/36, 39 (00) p Terek, M.: Aalýza odľahlých údajov. I: Forum Statisticum Slovacum 6 (008) p Terek,M., Nguye Dih He:Možosti využitia iektorých etradičých charakteristík v aalýze miezd. I: Ekoomické rozhľady (0) p Tibeský, M.:Chakteristiky polohy založeé a aalýze odľahlých údajov. Ig. theses, Bratislava: Uiversity of Ecoomics, 0. Wilcox, R. R.: Applyig Cotemporary Statistical Techiques. USA: Academic Press