A Note on the Reliability Tests of Estimates from ARMS Data

Transcription

1 A Note o the Reliability Tests of Estimates from ARMS Data C. S. Kim, C. Hallaha, W. Lidamood, G. Schaible, ad J. Paye USDA uses the cocept of publish-ability rather tha statistical reliability of a estimate for quality validatio of USDA estimates, which is solely based o the sample size ad the coefficiet of variatio (CV). We demostrate coceptually how the reliability of the sample mea ca be tested by estimatig the upper ad lower bouds of the cofidece iterval for a ukow populatio mea usig the CV. However, the reliability test for the sample mea ca be made oly uder the ormality assumptio. USDA multiple-way Agricultural Resource Maagemet Survey (ARMS) estimates are used to illustrate the relative measure of precisio for sample-based estimators. Key Words: ARMS data, coefficiet of variatio, publish-ability, reliability The Natioal Agricultural Statistics Service (NASS) ad Ecoomic Research Service (ERS) of the U.S. Departmet of Agriculture (USDA) are strivig to improve the availability ad the quality of data o crop productio practices as well as farm fiacial maagemet. The Agricultural Resource Maagemet Survey (ARMS) Phase II is USDA s primary source of iformatio o farm crop productio practices for major crops, icludig cor, soybeas, wheat, grai sorghum, barley, oats, ad cotto. This survey provides aual field-level data by crop o irrigatio techology ad water use, utriet use ad utriet maagemet practices, crop residue maagemet practices, pesticide use ad pest maagemet practices, ad crop seed varieties icludig geetically modified seeds. These data summaries, curretly available for soybeas, wheat, ad cotto for the period 1996S2000, ad cor for the period 1996S2001, are ivaluable to decisio makers C. S. Kim is a seior ecoomist, C. Hallaha is a mathematicia, W. Lidamood is a programmer, ad G. Schaible ad J. Paye are agricultural ecoomists, all with the U.S. Departmet of Agriculture, Ecoomic Research Service, Washigto, DC. The authors thak Phil Kott, Chief Statisticia at USDA/NASS, ad Robert Dubma, Data ad Survey Coordiator at USDA/ERS, for their commets ad useful discussios, ad two aoymous referees for their helpful commets. The views expressed here are the sole resposibility of the authors ad do ot ecessarily reflect those of the U.S. Departmet of Agriculture. ad aalysts withi govermet agecies ad the public. 1 Quality validatio of USDA ARMS estimates is based solely o the sample size ad the coefficiet of variatio (CV), which is also called the relative stadard error. Some details ca be foud i Dubma (2000), Kott (1997, 2001), ad i Sommer et al. (19). Accordig to USDA s geeral guidelies for statistical reportig stadards, o estimate should be suppressed simply because it is deemed statistically ureliable. Nevertheless, the presece of such a estimate i a published table should be oted. I particular, a estimator (mea or proportio) i a data summary table of a agecy publicatio should be marked with a asterisk deotig it as potetially ureliable (i a statistical sese) if either the sample size is less tha a fixed umber of idividuals or if the estimate s CV is greater tha some desigated limit (USDA, 1993). The desigated CV ca be set at the agecy s discretio for a estimator based o commoly occurrig evets. For the ARMS Phase II data, each estimator is idetified as havig a CV less tha or equal to 25%, greater tha 25% but less tha or equal to 50%, greater tha 50% but less tha or equal to 100%, or greater tha 100%. 1 Data summaries are available at the USDA-ERS olie website, Agricultural ad Resource Ecoomics Review 33/2 (October 2004): 293S297 Copyright 2004 Northeaster Agricultural ad Resource Ecoomics Associatio

2 294 October 2004 Agricultural ad Resource Ecoomics Review The CV is a ideal measure for comparig variatio across umerous sets of data expressed i differet uits, such as cor price per bushel ad cor yield per acre. However, the CV is ot a very meaigful measure without some assurace that the populatio mea, µ, lies withi a preassiged precisio level. For USDA ARMS measures, use of these estimates for policy aalyses requires a broader statistically determied measuremet of precisio. Therefore, the objectives of this paper are fourfold: first, to iform ARMS data users of the reasos why USDA provides the CV for each estimate; secod, to explai coceptually how the CV ca be used for testig the reliability of a estimator; third, to address the assumptio of ormality applied to our reliability tests; ad fially, to demostrate our reliability tests applied to a subset of 2001 ARMS Phase II estimates. While ARMS Phase II summary data tables ca cotai estimates for both meas ad proportios, we cocetrate o mea estimates i this paper. Reliability versus Publish-ability of a ARMS Estimator The coefficiet of variatio associated with each USDA estimate is published for two primary reasos. First, the statistical iterpretatio of the square root of the expected value of the square of the relative absolute error of a estimator is roughly equal to its CV, stated as follows (Kott, 2003): (1) E *(m & µ)/µ* 2 ½ ' (σ 2 /)/µ 2 ½ ' σ/ /µ. s/ /m ' CV, where E represets the expectatio operator, µ ad σ 2 represet the populatio mea ad variace, respectively, ad m ad s represet the sample mea ad sample stadard deviatio, respectively, of a radom sample of size from a ormally distributed populatio such that E(m) = µ, E(s 2 ) = σ 2, ad m ~ N(µ, σ 2 /). Secod, a estimate may be published based o USDA s geeral guidelies for statistical reportig stadards, which states that o estimate should be suppressed eve though it is deemed statistically ureliable, as log as the estimate is marked if its CV is greater tha some desigated limit. Therefore, cosistet with USDA statistical guidelies, the estimate m may be a publishable rather tha a reliable estimator (Kott, 2003). The CV, [/m], is a fuctio of the ratio of two idepedet radom variables, ad m. I this case, however, the stadard method for derivig the itegratig desity fuctio of the ratio of two idepedet radom variables (e.g., Dwass, 1970; Parze, 1960) fails because the itegral correspodig to the mathematical expectatio of CV = /m is ot solvable. Therefore, while we kow that E[/m] [(σ/ )/µ], we do ot kow the magitude of the bias of the CV. Cosequetly, the curret state-of-affairs with respect to statistical validatio procedures based oly o a CV statistic is somewhat usatisfactory as a appropriate statistical assurace that USDA s ARMS estimates are reliable. The cocept of reliability of a estimate we use is based o the precisio of a estimator as a measure of reliability [see Barlow ad Proscha (1965) for a differet view of reliability over time]. To costruct a (1!α) cofidece iterval for µ whe σ is ukow, the estimator, [(m & µ) /s], assumig a ormal distributio, has a t-distributio with (!1) degrees of freedom. Therefore, the cofidece iterval for µ for radom samples from a ormal populatio with a degree of cofidece of (1! α) is represeted as follows: (2) Pr m & t 1&α/2,&1 # µ # m % t 1&α/2,&1 ' Pr &t 1&α/2,&1 # µ & m # t 1&α/2,&1 ' Pr &t 1&α/2,&1 CV # (µ & m)/m # t 1&α/2,&1 CV ' Pr *(µ & m)/m*#k ' (1 & α), where k ' t 1&α/2,&1 CV. I equatio (2), the reliability of m as a estimate of µ requires that the relative deviatio of m from µ be less tha or equal to a give level of precisio k at a (1! α) cofidece level. For example, if the relative error of m is less tha or equal to the precisio level k at a 95% cofidece level, the: Pr *(µ & m)/m*#k ' 95%. Equatio (2) ca ow be rewritte as:

3 Kim et al. A Note o the Reliability Tests of Estimates from ARMS Data 295 (3) Pr m & t 1&α/2,&1 # µ # m % t 1&α/2,&1 ' Pr m 1 & t 1&α/2,&1 /m # µ # m 1 % t 1&α/2,&1 /m ' Pr m(1& k) # µ # m(1 % k) ' (1 & α), where k ' t 1&α/2,&1 CV. Equatio (3) shows that the lower ad upper bouds of the cofidece iterval for a ukow µ are represeted by m(1! k) ad m(1 + k), respectively. Usig equatios (2) ad (3), ad USDA ARMS data as a example, we ca illustrate a relative measure of precisio of a estimator uder two reliability perspectives. Give that a predetermied umber of sample replicatios for NASS s ARMS data is * = 15 for a delete-a-group jackkife variace, t 0.975, 14 = 2.145, the: (4) k ' 2.145CV. As illustrated by equatio (4), if the relative error of a estimate is preassiged at a precisio level k = k 0, the upper boud of the CV must be equal to 100(k 0 /2.145) percet at the 95% cofidece level. Similarly, if the CV is estimated to be CV = CV 0 at the 95% cofidece level, the precisio level required is estimated by k = 2.145CV 0. The reliability coditio for a estimator, expressed i equatios (2) ad (3), was derived uder the assumptio that samples are draw from a ormal distributio. Therefore, it is also appropriate to address the questio about how the reliability coditio holds up uder a oormal assumptio. We ca evaluate whether a reliability coditio, similar to that expressed i equatios (2) ad (3), ca be established whe there is o kowledge of the distributio, but assumig a radom sample. For example, for ay preassiged error g, Chebyshev s iequality may be used: (5) P *(m & µ)*#g $ 1 & (σ 2 /)/g 2. The absolute deviatio i equatio (5) ca be rewritte i the form of the relative deviatio as preseted i equatio (2): (6) P *(m & µ)/m*#z $ 1 & (σ 2 /)/(m 2 z 2 ), where z = g / m. From equatio (6), oe obtais the followig: (7) P *(m & µ)/m*#z $ 95% if z $ σ/ /m. Sice the populatio variace is ukow, the reliability for a estimator caot be tested with equatio (7). Therefore, this result implies that the reliability test for the sample mea ca be made oly uder the ormality assumptio. The Delete-a-Group versus Delete-a- Sample Jackkife Variace USDA s NASS adopted a multi-phased stratified samplig procedure for its ARMS data, ad therefore the variace of a ARMS estimator for a CV value is estimated with a delete-a-group jackkife method. However, a simple delete-a-sample jackkife method ca be illustrated as follows: (8) m (i) ' 1/( & 1) j x j ' (m & x i )/( & 1), j i where m (i) is the sample mea of the data set deletig the ith sample elemet, x i, ad remais the sample size. The full sample mea estimate is represeted by: (9) m (@) ' j m (i) /. i'1 The delete-a-sample jackkife estimate of variace is the deoted by: (10) var(m jack ) ' ( & 1)/ j m (i) & m 2 (@). It should be oted that i equatios (8)S(10) ormally represets the umber of observatios, based o the literature explaiig a delete-a-sample jackkife method (for istace, see Efro, 12). However, for USDA ARMS data, represets the umber of groups correspodig to a delete-agroup jackkife method. NASS predetermied the umber of group replicatios at = 15. For the case of a delete-a-sample jackkife method, Miller (1964) demostrated that the cofidece iterval for a estimator approaches the cofidece iterval for a estimator from the ormal distributio as the sample size icreases, i.e., as 6 4. However, for a delete-a-group jackkife method, a icrease i sample size does ot chage the umber of replicatios. So, at this time, it remais uclear how much the sample size affects the cofidece iterval for a jackkife estimator. i'1

4 296 October 2004 Agricultural ad Resource Ecoomics Review Table 1. Reliability Tests for Nutriet Use by Tillage Practice ad Irrigatio System Cor (all survey states, 2001) [1] Descriptio [2] % Acres Treated a [3] Sample Size [4] 100(CV [5] 100(k b [6] Lower Boud c [7] Upper Boud d Gravity Irrigatio System: Nitroge Pressure Irrigatio System: Nitroge * Descriptio Pouds per Treated Acre a Sample Size 100(CV 100(k b Lower Boud c Upper Boud d Gravity Irrigatio System: All Acres: Nitroge No-coservatio Tillage: Nitroge Coservatio Tillage: Nitroge Pressure Irrigatio System: All Acres: Nitroge No-coservatio Tillage: Nitroge Coservatio Tillage: Nitroge * * ** a Asterisks (*, **, ***) idicate a CV such that 25% < CV # 50%, 50% < CV # 100%, ad CV > 100%, respectively. b k = 2.145CV. c Lower boud (L) = m(1! k). d Upper boud (U ) = m(1 + k). A Example from USDA-ARMS Estimates To explai the reliability of estimates based o USDA ARMS data, we base our aalysis o a summarized data table from the multiple-way ARMS tables posted o the USDA-ERS website uder the title Nutriet use by tillage system ad irrigatio system associated with cor for all survey states (refer to website give i footote 1). The USDA ARMS iformatio is represeted i the first two colums of table 1, where the secod colum represets the sample meas (percet of acres treated ad pouds per treated acre) for the year 2001, ad each estimator is idetified with its CV. Estimates are marked based o a CV of less tha or equal to 25%, greater tha 25% but less tha or equal to 50%, ad greater tha 50% but less tha or equal to 100% (see table 1 footote). Colum [4] idetifies the CV (i percet). Colum [5] idetifies the precisio level (i percet), 100(k, which is estimated usig equatio (4). For example, the rate of potash applicatio for cor

5 Kim et al. A Note o the Reliability Tests of Estimates from ARMS Data 297 productio i 2001 by o-coservatio tillage practice usig a gravity irrigatio system is estimated to be m = 48.5 pouds per treated acre, ad the associated precisio level ad CV are estimated to be 100(k = 56.2% ad 100(CV = 26.2%, respectively (table 1). The estimated precisio level idicates that at the 95% cofidece level, the deviatio of the estimate m = 48.5 pouds per acre from its ukow µ is less tha or equal to 56.2% of the estimate, or 27.3 pouds per acre (i.e., 0.562(48.5 = 27.3 pouds per acre). The lower ad upper bouds of the 95% cofidece iterval for the estimator s ukow µ are idetified i colums [6] ad [7], respectively. These cofidece bouds are estimated by 48.5 ± 27.3, usig the precisio level k show i colum [5]. Results of the reliability tests by the precisio level reveal that for ARMS 2001 data, for all cor acres, estimates of utriet applicatio rates are relatively more precise for acres irrigated with a pressure irrigatio system tha a gravity system. I additio, similar estimates for pressure-irrigated acres usig coservatio tillage are geerally more precise tha are estimates of applicatio rates for gravity-irrigated acres usig coservatio tillage. Fially, while estimates for gravity-irrigated acres usig o-coservatio tillage are slightly more precise tha the estimates for coservatio tillage o gravity-irrigated acres, the precisio of these estimates is relatively similar for pressure-irrigated acres. Coclusios We have reviewed the quality validatio procedure associated with USDA ARMS estimates based o the assumptio that samples are draw from a ormally distributed populatio. While the cocept of publish-ability of a estimate is used by USDA whe providig the CV statistic for each estimate, we demostrate coceptually how the CV ca be used for testig the reliability of a estimator. Specifically, if the relative bias of a estimate is preassiged at a precisio level k 0, the upper boud of the CV ca be derived from k 0 = 2.145CV at the 95% cofidece level. Similarly, if the CV is estimated to be CV 0, the at a 95% cofidece level, a required upper boud o the correspodig precisio level is estimated by k = 2.145CV 0. Furthermore, we have also show that the upper ad lower bouds of the cofidece iterval for a ukow µ ca be estimated usig the coefficiet of variatio with m(1+ k) ad m(1! k), respectively. Fially, our coceptual aalysis has demostrated that the reliability of a estimate caot be tested whe there is o kowledge of the distributio, because the populatio variace is ukow i.e., the reliability test for the sample mea ca be made oly uder the ormality assumptio. For USDA s 2001 ARMS estimates of utriet applicatio rates by tillage practice ad irrigatio system, the ew reliability tests provide sigificatly improved iformatio from which ARMS data users may judge estimator precisio. This relative test of estimator reliability ca help trasform ARMS estimates from beig just publishable to accommodatig a level of cofidece or reliability whe estimates are used withi policy aalyses. Refereces Barlow, R. E., ad F. Proscha. (1965). Mathematical Theory of Reliability. New York: Joh Wiley ad Sos, Ic. Dubma, R. W. (2000). Variace Estimatio with USDA s Farm Costs ad Retur Surveys ad Agricultural Resource Maagemet Study Surveys. Staff Report No. AGES 00-01, USDA/Ecoomic Research Service, Washigto, DC. Dwass, M. (1970). Probability ad Statistics. New York: W. A. Bejami, Ic. Efro, B. (12). The Jackkife, the Bootstrap, ad Other Resamplig Plas. Society for Idustrial ad Applied Mathematics, Philadelphia, PA. Kott, P. S. (1997). Statistical Aalysis with a Delete-a-Group Jackkife. Upublished paper, USDA/Natioal Agricultural Statistics Service, Washigto, DC.. (2001, July). Usig the Delete-a-Group Jackkife Variace Estimator i NASS Surveys. Revised Research Report No. RD--01, USDA/Natioal Agricultural Statistics Service, Washigto, DC.. (2003). Chief Statisticia, Natioal Agricultural Statistics Service, USDA, Washigto, DC. Persoal commuicatio. Miller, R. G. (1964). A Trustworthy Jackkife. Aals of Mathematical Statistics 39, 1594S1605. Parze, E. (1960). Moder Probability Theory ad Its Applicatios. New York: Joh Wiley ad Sos, Ic. Sommer, J. E., R. A. Hoppe, R. C. Gree, ad P. J. Korb. (19, December). Structural ad Fiacial Characteristics of U.S. Farms, 1995: 20th Aual Family Farm Report to Cogress. Agriculture Iformatio Bulleti No. 746, USDA/Ecoomic Research Service, Washigto, DC. [108 pp.]. Olie. Available at publicatios/aib746/aib746f.pdf. U.S. Departmet of Agriculture. (1993, September 23). Joit Policy o Variace Estimatio ad Statistical Reportig Stadards o NHANES III ad CSFII Reports: HNIS/ NCHS Aalytic Workig Group Recommedatios. Washigto, DC.