Richard Bolstein, George Mason University

Transcription

1 I. INTRODUCTION RANDOM MOMENT SAMPLING TO ESTIMATE ALLOCATION OF WORK EFFORT Ricard Bolstein, George Mason University Te proper determination of federal, state, and local sare of administrative costs of operating a State Social Services Department is a complex and time-consuming task. Since cost saring formulas and funding sources differ by type of service provided, te first step in tis process is to accurately allocate staff time to te various types of ~rograms. Te state of Virginia currently uses a case-count metod along wit 'caseloads standards' wic assign an average number of 'earned ours' to eac type of program and activity. Te total work effort for a period of time (mont or quarter) allocated to a particular program is obtained by multiplying te number of cases for tat program, broken down by 'activity' (initial determination, re-determination, fraud, etc.), by te appropriate standard. Te problem wit tis metod is in keeping te 'standards' updated, a difficult, time-consuming task (even if a sampling approac is used). Most oter states use some type of labor-intensive approac wic requires complete case reviews and/or logbooks kept by te staff. Recently, te U.S. Department of Healt and Human Services (HHS) as urged states to adopt a work-sampling approac to tis problem. Tis metod, known as random moment sampling (RMS), intercepts workers at random 'time moments' and records wat program and activity tey are engaged in at tat moment. Statistical metods can be used not only to estimate te allocation of total work time to eac program, but also to provide error bounds for tese estimates. Te latter cannot be done wit te 'complete-review' metods since te bias inerent in tose metods is not computable. Furtermore, RMS is muc less labor-intensive tan te 'complete-review' metods. As a result, several states ave adopted te RMS metod over te past seven years. Most of tese use a simple random sample of moments as suggested by HHS, altoug te structure of te social service system in some states requires a more complex sampling plan. Micigan uses a two-stage cluster design were te clusters are local welfare agencies wic are cosen wit probability proportional to size. Eac worker in te selected clusters is interviewed at four random moments in te quarter. Sout Carolina stratifies by worker classification, wereas Pennsylvania and Tennessee employ systematic sampling. In devising a plan for te Virginia Department of Social Services (VDSS), te autor was led to stratify te sample according to groups of agencies wic are, to some degree, omogeneous wit regard to te types of programs administered. Te description of tis RMS plan for VDSS is te subject of tis paper. Te literature on random moment sampling is virtually non-existent in statistical journals. As te tecnique was originally applied to te textile and oter manufacturing industries, it made its way into te industrial engineering and industrial psycology literature, usually under names like 'work-sampling' and 'time studies' (see Niebal [4]). One goal of writing tis paper is to call attention to some interesting problems tat arise wit tis tecnique in estimation and non-response adjustment tat may be of interest to statisticians. 2. PROBLEM FORMULATION Welfare programs in Virginia are carried out by 124 local agencies. Tese agencies ave two primary functions: (i) Determine client eligibility for financial assistance programs, and (2) Provide or arrange for social services. Most employees are involved wit just one of tese functions, and are classified as eligibility (or benefit) program workers and service program workers respectively. Tere are also some 'generic' workers wo are involved wit bot functions. Te central problem is to determine te federal, state, and local sare of administrative costs (worker salaries) to carry out tese functions. Te sares differ by te type of program witin eac function, as sown in Table 1 below. Altoug sares are te same for some of te programs sown in te table, separate funding sources require breakout of costs for eac program listed. If workers were dedicated to individual programs, it would be a simple matter to allocate costs since, for example, te total salaries of all ADC eligibility workers would be carged according to Table I. As tis is not te case, te next best approac is to obtain te proportion Pi of total worktime spent on program i. If F i denotes te federal sare of program i in Table 1 and C te total administrative costs for te time period in question, ten it is natural to assign as te federal sare of total costs te quantity (~FiPi)C Te problem is to estimate te unkowns Pi" Costs are computed separately for eligibility and service functions (see [5]). Workers are considered dedicated to tat function wic absorbs at least 80% of teir time. For sampling purposes, generic (i.e. non-dedicated) workers are assigned to tat function wic occupies te majority of teir time. However, a sampled moment is assigned to te function in wic it belongs regardless of te function assigned to te worker. Tis appens infrequently and as an insignificant effect on sample size. Tus, for te function under consideration, Pi will denote te proportion of total work-time for tat function wic is spent on program i, and C will denote te total administrative costs of tat function. 3. SAMPLE FRAME To estimate te Pi, workers are intercepted at random times to find out wat activity tey are engaged in at tose moments. Tis is te tecnique of random moment sampling (RMS). Tus, te units of analysis are worker-moments. Procedural difficulties arise if 'moment' is treated as a continuous variable since it is 671

2 essential tat te worker be intercepted at te precise moment selected. It is customary in RMS to divide te workday into equal time intervals and identify eac interval wit a specific 'moment' suc as its' first minute. Te common lengt of te time intervals sould be at least as sort as te sortest period a worker could spend on any one program (Capanis [3]), wic was judged to be five minutes in VDSS. Te typical workday in VDSS is 8.5 to 9 ours long including an our luncbreak. However, ours are flexible wit regard to start and finis time and lengt of luncbreak and local agencies set teir own requirements for number of ours worked. In order to insure tat all workers ave te same probability of inclusion in te sample, a standard workday of 7:30 A.M. to 6:00 P.M. was initially cosen, but later reduced after a pilot test to 7:30-5:30 for eligibility and 8-5 for services. Hence tere are a total of 120 five-minute intervals per day for eligibility (108 for services) wit te first minute of eac interval designated as te corresponding moment. Te number of eligibility moments M in a quarter is te number of workdays in te quarter (typically 62) multiplied by 120. If N denotes te number of eligibility workers in te quarter, ten te eligibility sample frame consists of MN worker-moments. Te services sample frame is constructed analogously. Of course, not all worker-moments will correspond to program activity since te worker may be on a lunc or oter break, annual or sick leave, not at work due to flex-time, or involved in some non-program specific activity suc as training or oter administrative duty. It is useful to define suc non-program categories as 'leave', 'administration', and 'flex-time', since tey can be used to elp monitor te sampling process. For example, flex-time sould account for I0 to 15% of worker-moments and ence if te sample result differs significantly from tis tere may be an error in te coding or interviewing process.every worker-moment is eiter a program moment or one of te tree non-program moments just defined. Denote by Qi te proportion of te MN population worker-moments tat fall in category i, and let Q denote te proportion of program moments. Ten, Q ~l = Qi were te prime means tat te sum is over all indices i corresponding to programs (from Table I, tere are 12 programs under te eligibility function and 9 under te services function). We also ave Pi = Qi/Q for all program categories i. Lower case letters will be used for te corresponding quantities computed from a sample. Recall tat in a simple random sample of n worker - moments, qi and Pi = qi/q are unbiased estimates of Qi and Pi wit estimated variances qi(l-qi)/n and Pi(l-Pi)/(qn) respectively ([2, C.2]). 4. SAMPLE DESIGN A simple random sample is not appropriate for tis problem for several reasons, including: (i) Local agencies vary in size from as little as tree employees to tree-undred. Tus, many small agencies would not appear in te sample, an important consideration in determining local reimbursement. Additionally, tere are indications tat small agencies ave a different program mix tan large agencies. (2) Rare programs, tose wic account for less tan 1% of te work-effort, may not appear in even a moderately large sample. For example, according to te 'caseloads standards' metod currently used in te State (see Section i), Subsidized Adoption accounts for only one-tent of one percent of te service program workload. Indeed, in a pilot sample of size 2400 no suc moment was encountered. (3) Some programs occur primarily in only a few agencies. For instance, te Refugee programs are almost non-existent outside Nortern Virginia. Most of tese problems can be andled by appropriate stratification. Te agencies in Nortern Virginia involved wit te refugee program make up one stratum, te large agencies of te Tidewater area in te souteast constitute a second, Ricmond togeter wit some oter mid-sized agencies in te central part of te state wic andle muc of te 'rare' subsidized adoption cases form a tird stratum, and te remaining agencies scattered trougout te state, wic are primarily small and rural, make up te fourt and final stratum. Not all of te 'rare' programs can be andled by geograpic stratification. If some worker classifications could be identified wic are more likely to be involved wit suc a program, tat could provide an effective stratification. No suc classifications ave yet been identified in VDSS. However, one metod for dealing wit tis problem is to combine samples from previous quarters for estimation of tese programs, and ten normalize te estimates of te oter programs as computed from te present sample. To summarize, te RMS design for VDSS is a geograpically stratified sample wit four strata and simple random sampling of worker-moments witin strata. 5. DATA COLLECTION An observation form was designed to be filled out by te worker and is or er supervisor at te prescribed moment (and required bot signatures). Te supervisor received a scedule of worker-moments about one week in advance. Workers did not know in advance wen or if tey were to be interviewed. To obtain reliable data wit a minimum of effort, te observation form sould be sort and clear but contain enoug information to be able to assess te accuracy of te responses. In addition, it must provide information to elp determine te causes of inaccuracies and missed observations (non-responses) so tey can be remedied by training. For example, it frequently appened tat te interview did not take place at te prescribed moment but was recalled at a later time (see Section 6). Te observation form not 672

3 only records te function and activity (program or non-program) of te worker, but also documents te lengt of recall (same day, or following day or later) and wo was responsible for te missed observation (worker unavailable due to field work, or supervisor unavailable due to a meeting or just forgetfulness). Te non-program categories were included on te form in some detail since, as mentioned in Section 3, tey also provide a ceck. Tus, 'leave' is decomposed into four parts: annual, sick, lunc/break, oter. Te observation form itself is not sown ere due to space limitations, but see [I]. 6. PILOT TEST A pilot test based on tis design was conducted in te October-December quarter of It included 14 of te State's 124 agencies representing all four strata. Equal sample sizes were cosen for eac stratum since results were to be compared by stratum. Tese were set at 800 for te eligibility sample and 600 for te services sample so tat HHS precision requirements would be met ([5]). Te sample size calculations were made from standard formulas for stratified samples (Cocran [2]) using estimates of proportions from a preliminary pilot study in June-August of A total of 3009 eligibility and 2230 services observations were obtained from te pulled samples of 3200 and 2400 reflecting 94% and 93% response rates respectively. Table 2 below provides a convenient way of assessing te overall percentage of program strikes and timeliness of observations for te eligibility sample. Note tat only alf te samples resulted in program strikes. Part of tis can be explained by te fact tat te percent of program strikes prior to 8 A.M. and after 5 P.M. dropped to about 10%. (Subsequent analysis sowed tat te allocation of tese extremal strikes to te various programs did not deviate significantly from te allocation of te entire sample. Terefore, te 8-5 workday is being used for te current statewide study and program strikes are running about 55%.) Note too tat te lunc/break percentages seem low since one would expect about 1.5 ours out of i0 or 15%. It is quite possible tat some workers are reluctant to report 'being on break'. Late reports pose a problem for te validity of te data. However, it is clear tat not all observations occur on time. Te timeliness of observations was subdivided into tree main categories: tose tat occurred on time, tose tat were recalled later te same day, and tose tat were recalled te following day or later. Fortunately, over 85% of all program observations were completed witin te same day. Note tat altoug non-program observations ad a iger percentage of on-time completion, te proportion completed witin te same day was no better tan te program observations. In order to try and reduce late observations, te reason for lateness was reported. Contrary to expectation, in te eligibility sample te observer was unavailable more frequently tan te worker. In te services sample unavailability was about equally divided. Followup meetings wit local supervisors indicated tat te major problem was simply not remembering to make te observations on time. Te use of alarm clocks and/or te designation of a single observer at a local agency for a day or week was recommended as a possible solution. 7. ADJUSTMENT FOR MISSING WORKERS During te pilot study, a group of workers in one stratum were inadvertently omitted from te sample frame. Omissions can also occur naturally if new workers are ired during te quarter after te sample was drawn. Tus, te witin strata estimates must be adjusted in suc cases before making overall estimates. In tis section we focus on a single stratum in wic tis problem occurred, present a metod of adjustment, and apply it to te pilot test. Let L denote te number of workers omitted from te sample frame and N te number included, so tat N + L is te total number of workers. (Bot N and L may be fractional since not all workers are employed te full quarter.) If qi is te sample estimate of te proportion of moments in te sampled population allocated to category i, ten te adjusted estimate of Qi is qai = [N/(N + L)]qi + [L/(N + L)]R i were R i is te (unknown) proportion of te ML unsampled population moments allocated to category i. Tus, te stratum under consideration is decomposed into two substrata consisting of te sampled and unsampled populations. In general, te unsampled population can be broken down into several substrata to facilitate te estimation (actually, educated guesstimation) of te R i. In te pilot study, all L missing eligibility workers came from one agency and were dedicated to one program (fuel). At tat agency, te normal workday was 8.5 ours including one our for lunc and an estimated alf-our for breaks. As tey were temporary workers, no vacation time was available and sick leave was considered minimal. Since te eligibility sample workday was ten ours long, we assigned te values Rflex =.15, Rleav e =.15, Rfuel =.70 and R i = 0 oterwise. Suc guesstimates will be refined by isolating te observations of fuel workers tat are sampled. Te adjusted estimate of te proportion Pi of program moments tat fall in category i is Pai = qai/(~qaj ) = [Nqi+LRi]/[Nq+LR] were R =~'Rj. 8. ESTIMATION Te formula for an unbiased estimate of te statewide proportion Qi in a stratified sample is (Cocran, [2,C. 5]) qi = i(w) qi 673

4 were qi is te estimate in stratum, and W is te proportion of te population in stratum. If M is te number of moments in te quarter, N te number of workers in stratum, and N te total number of workers in te population, ten W = MN/MN = N/N An unbiased estimate of te variance of qi is V(qi) = ~(W)2 (qi)(l-qi)/n were n is te size of te sample from stratum (tat is, te number of worker-moments sampled from stratum ). Estimation of te Pi is more complex. We present two metods. Metod I. Analogy wit simple random sampling suggests setting were Pi = qi/q q =~'qj J is an estimate of te proportion Q of work moments tat correspond to program activity (we call tese program moments). If we let q() =~'qj J denote te sample proportion of program moments in stratum, ten q =~'~Wqj =~W(~'qj) =~Wq() ' ~ ~ so tat Pi ~ Nqi)/(~Nq ()) is seen to be a combined ratio estimate (Cocran [2, p. 165]). Te variance of Pi can be written approximately as V(Pi) = ~[W2/(Q2n)][Qi(l-Qi) +Pi2Q()(l-Q()) - 2PiQi(l-Q())] were Q() and Qi are respectively te stratum population proportion of work moments tat are program moments and te proportion of work moments tat are program i moments. Tese quantities and Pi are estimated from te sample to obtain an estimate of te variance. In large samples te bias of te ratio estimate is negligible if te coefficient of variation of q(~) is small (Cocran [2]). Metod II. Te separate ratio estimate ([2,p.164]) of Pi can be written in te form Pi = (I/MNQ) =~Q()/Q)WPi qi/q())mnq () wic now as te appearance of a stratified proportion estimate wit unknown weigts (Q(~)/Q)W. If we assume tat te proportion of moments tat are program moments is te same in eac stratum, ten all Q() = Q and te weigts become te familiar W. Tus, it seems reasonable to use te W as estimates of te true weigts, in wic case te estimate of Pi becomes A Pi = ~ WPi Te variance and bias are given by V(~i) = KW2pi(l-Pi)/[nq ()] B i = ~W[l-Q()/Q]Pi Tese quantities can be estimated from te sample. As te survey will be repeated on a continuing basis, improved estimates of te Q(), and ence te weigts can be obtained as a moving average from previous quarters to virtually eliminate te bias. However, even wit te current estimate te bias turns out to be negligible in comparison wit te variance in te pilot study. So ow do te two different estimators compare? We partially illustrate wit some data from te eligibility sample in te pilot study. STRATUM i n W q() l-q()/q qi(adc) Pi(ADC) Te last row is just te fourt row divided by te second row. Te estimate of te proportion of all moments tat are ADC program strikes as computed from te first formula in tis section is qi =.1725 wit variance Also, q =YWq() =.5057 and v(q) =~W2q()(l-q())/n-- = So, Pi = qi/q =.3411 wereas, Pi = WPi =.3393 So te two metods produce nearly identical estimates for te proportion of program moments allocated to ADC. Te estimated variance of te first estimate turns out to be Since te coefficient of variation of q, wic serves as an upper bound for te ratio of te square of te bias to te variance, is.0186, te bias of te composite ratio estimate is negligible ([2, p. 162]. Te estimated variance of te second estimate is and te bias as computed from te table is b = Hence, te square of te bias is.016 times te variance so is negligible in tis case too. Te relative design effect of Metod I to Metod II for ADC is

5 TABLE 2 TIMELINESS OF THE ELIGIBILITY SAMPLE FOR THE DECEMBER 1985 QUARTER: UNWEIGHTED COUNTS AND PERCENTS COUNT SAME DAY SAME DAY NEXT DAY NEXT DAY ROW ROW PCT ON TIME WORKER OBSERVER WORKER OBSERVER TOTAL BENEFITS ANNUAL SICK LEAVE OTHER LEAVE LUNCH-BREAK ADMIN FLEXTIME VACANT i I i.i i COLUMN TOTAL i00.0 indicating confidence intervals will be 3.6% wider under Metod I. Similar results were obtained for te oter program categories. A teoretical comparison of te metods will appear in anoter paper~ but it seems experimentally tat Metod II is to be preferred because of its simplicity. 9. SUMMARY Random Moment Sampling is a complex survey procedure Wic requires advanced metodology Problems addressed include te andling of 'rare subpopulations, assessing and improving te quality of data, adjusting for population elements omitted from te sample frame, and coosing among two estimation metods. Major issues not dealt wit in tis paper, due to time and space limitations, include elimination of zero cells and estimation of proportions for 'rare' programs, and variance estimation of te federal, state, and local sare of costs. REFERENCES [i] R. Bolstein, G. Ballentine, J. Vaugan, "Implementing Random Moment Sampling in te Virginia Department of Social Services: Design and Analysis", Proceedings of te Twenty-Sixt annual Worksop of te National Association for Welfare Researc and Statistics, [2] W. Cocran, "Sampling Tecniques", Jon Wiley & Sons, tird edition, [3] A. Capanis, "Studying Work Activities in Office Systems", IBM Tecnical Report, [4] B.W Niebal, "Mo tion and Time Study", Ricard D. Irwin~ Inc., 1972 [5] "A Guide for State and Local Government Public Assistance Agencies/Departments, Procedures for te Preparation and Submission of Cost Allocation Plans", U.S. Department of Healt and Human Services,