Reliability of Recall in Agricultural Data

Transcription

1 PRELIMINARY DRAFT DO NOT CITE Reliability of Recall in Agricultural Data Kathleen Beegle Calogero Carletto Kristen Himelein Development Research Group, World Bank 1818 H Street NW, Washington DC, 20433, USA Abstract Despite the importance of agriculture to economic development, and a vast accompanying literature on the subject, little research has been done on the quality of the underlying data. Due to survey logistics, agricultural data is usually collected by asking respondents to recall the details of events occurring during past agricultural seasons that took place a number of months prior to the interview. This gap can lead to recall bias in reported data on agricultural activities. The problem is further complicated when interviews are conducted over the course of several months, thus leading to recall of variable length. To test for such recall bias, the number of months between harvest and interview is examined for three African countries with respect to several common agricultural input and harvest measures. The analysis shows little evidence of a bias impacting data quality. There is some indication that more salient events are less subject to recall decay. Overall, the results allay some concerns about the data quality of some types of agricultural data collected through recall over lengthy periods; but, further research in different contexts is needed. Key words: agriculture, measurement error, recall JEL classifications: Q12, O12 All views are those of the authors and do not reflect the views of the World Bank or its member countries. This analysis was supported by the Bill and Melinda Gates Foundation Trust Fund for Improving the Quality and Policy Relevance of Household-Level Data on Agriculture in Sub-Saharan Africa at the World Bank and the Economic and Social Research Council, United Kingdom.

2 1. INTRODUCTION For most of the population of sub-saharan Africa, farming is a main source of both food and household income. Analysis of the Rural Income Generating Activities (RIGA) data suggests that on-farm sources of income tend to be more important for the African countries in the database, where the shares range from 59 percent to 78 percent of total income. While nonfarm activities are increasingly important, the vast majority of rural African households remain heavily dependent on agriculture (Davis et al., forthcoming), and agriculture plays a critical role in the economic development and improvement of living standards in the region. There have been volumes of literature, including the World Bank s 2008 World Development report, Agriculture for Development (World Bank, 2008), focusing on the relationship. With greater demand for household data on both farming decisions and non-farm activities, general household surveys are expanded to include extensive agricultural modules to capture agricultural production. Despite the importance of agriculture, the data on rural household farming, particularly when collected outside the domain of specialized farm surveys, are perceived to be of poor quality. Rarely does the vast literature on rural development focus on issues of underlying data quality and data collection methods. Because of the complexity of farming, where salient actions on the farm take place over several months of a season (plot preparation, input application, plot maintenance, harvest and selling), ideally agricultural information would be collected through multiple visits over a farming season to facilitate accurate recall of events. Specialized farm surveys are often designed to visit the household at multiple times, particularly if based on resident enumerators (e.g. agricultural extension agents, or other Ministry of Agricultural staff). Such specialized surveys, however, offer limited scope to analyze farming linked with non-farm income activities and other socioeconomic outcomes. To meet the demand for integrated data, multi-topic/multi-purpose household surveys (such as a Living Standards Measurement Study [LSMS] survey) are often extended to cover agricultural issues. Yet, cost and logistical considerations usually dictate the data in these surveys be collected during a single visit to the household. In this case, the household/farmer is asked to report information relating to farming by recalling all details of past events for the last completed agricultural season, often including two or more separate harvests. Depending on the timing of the survey, the interview itself will not necessarily be the most propitious timing for a single visit (immediately after the main harvest). Moreover, it is common 1

3 that such multi-purpose surveys will be fielded over 12-months, to account for seasonality in consumption expenditures. In this case, the recall period will vary across households depending on in which month the household is interviewed. In the case of surveys from year-long fieldwork, such as the three examined here, there is variation up to 11 months between the time of the harvest and the data collection, and recall periods for input use are even several months longer. Even if all sample households are surveyed once over a very short period just after harvest, events early in the season will be reported with a recall of several months (e.g. fertilizer application). This, then, raises concerns about the reliability of information reported for earlier events. Substantial recall effects resulting from this gap in time could introduce bias or measurement error into data collected further from events of interest, and the problem may be further aggravated by variable recall periods across sample households. The objective of this paper is to investigate the extent of recall bias, using data from three national household surveys conducted in East and Southern Africa with fieldwork over about 12 months. We use the variation in recall period and random assignment of households to month of interview (and, therefore, recall period) to examine several important agricultural indicators within the three data sets. Specifically, we explore whether we find any differential in reporting of input use and harvest amounts for main crops based on the time elapsed between events and reporting. In general, we find little evidence of any significant recall bias, in terms of under/over reporting or variance. The findings suggest that farmers reports of harvest, crop sales, and input use are robust even when measured up to 15 months from the event or activity. The next section briefly reviews the literature on survey reporting errors, with attention to types of bias most relevant for agricultural household surveys. Section 3 discusses the data and empirical approach. Results are presented in Section 4. Section 5 concludes. 2. Literature Review There is a large body of work in survey literature about reporting errors in survey data; Sudman and Bradburn (1974) summarize the extensive work in this area from the 1950s and 1960s. Retrospective data, in particular, introduces various data quality issues linked to recall errors. Over time, respondents may not be able to accurately recall details of events. Accurate remembrance of events will depend not only on the duration of the recall period, but also on the 2

4 nature of event being reported. The literature generally categorizes three main types of recall error in household survey data: telescoping, heaping, and recall decay. Telescoping refers to inaccurately identifying the date of events, either forward or backward into the recall reference period. Heaping refers to the use of estimation by respondents that places large numbers of responses at particular points, such as an expenditure of about $100 or dating an event about six months ago. Recall decay refers to forgetting details of events. Since the focus of our study is farm input usage and harvest size, rather than the timing of events, we do not focus on telescoping. There are, though, areas within agricultural statistics where telescoping would be relevant, such as dates of fertilizer application or input purchase. It is also difficult to consider the issue of heaping as agricultural data is naturally heaped, as both inputs and products are sold in uniform quantities (50 kg sacks, oxcarts, etc ). Recall decay bias can occur in two ways. First respondents may forget performing certain activities altogether, biasing frequency estimates downward. While there is little empirical work on the subject related specifically to agriculture, the topic is covered in guidelines for marketing and social research questionnaires. This literature describes salience as an important factor as to whether events are likely to be affected by recall decay. Bradburn et al. (2004) identify three factors which determine salience: (1) the unusualness of the event, (2) the economic and social cost or benefits of the event, and (3) the continuing consequences of the event (p. 64). Loftus and Marburger (1983) assess the improvement in accuracy of retrospective accounts when surveys use a highly salient landmark event (the eruption of Mt. St. Helens) to mark the beginning of the reference period. In their study on migration, Smith and Thomas (2003) find that migration events which are of greater salience to the respondent are less likely to be affected by recall decay. Relating specifically to agriculture, Judge and Schechter (2009) test the distribution of the first significant digit of various data to detect abnormalities. Using Benford s Law, they find evidence of the importance of salience in data quality from an agricultural survey in Paraguay, specifically that more accurate information is recorded for crops that represent a larger share of household income. Recall decay can also affect the details of events reported by respondents. Much of the research in this area relates to consumption experiments, which focus on the reliability of reports of consumption or expenditure on food and non-food consumer goods. One exception is Coleman (1983) who addresses recall decay in agriculture through comparisons of the mean number of reported labor hours for each day during the week prior to the interview. Using a 3

5 dataset of 129 households in 12 villages in Benue State, Nigeria from 1979/1980, he takes as a reference point the mean number of hours reported for the day before the survey. With each additional day from the reference point, he finds a consistent and significant over-reporting of labor hours, ranging from 17.5 percent on the day before the reference day to 61.6 percent on the first day of the recall period. His finding with regard to agricultural labor contradicts expectations based on marketing and household consumption data, which would predict increased under-reporting with the passage of time, as respondents forget events. Studies of recall in household consumption data assess the reliability of data by the length of time to which the data refer, such as food expenditures over the last 7 days, 30 days, or even as long as the last year. These studies suggest that longer recall periods are associated with lower aggregate totals of frequent food expenditures. Gibson and Kim (2007) hypothesize that when respondents are asked to recall larger amounts of information, from a greater number of transactions due to larger household size or longer recall period, they will shift from summing the total of individual remembered events to estimating the overall total. In a survey experiment in Papua New Guinea, Gibson (2002) found that the average food expenditure was 26 percent higher with a consumption diary (asking for consumption for each of the 7 days covered) than asking for one reported amount for the entire 7 days. In a survey experiment in Ghana, reported expenditure on frequently purchased items fell by nearly three percent for each day added to the recall period, though leveled off at about 20 percent after two weeks (Scott and Amenuvegbe, 1991). There are also further examples of recall decay for analysis of other topics. Bound et al. (2001) provides a general review of this literature, with examples taken from labor, health care, crime and motor vehicle accident statistics. The authors summarize the overall findings as the greater the length of the recall period, the greater the expected bias due to respondent retrieval and reporting error (p. 3743). Relating these findings to agricultural data and the concept of salience, we would expect that large scale, unusual or expensive events for farm households are less likely to suffer from recall decay than smaller, less-salient events. We would expect to find less recall decay in cash crops rather than staple crops. Cash crops are sold, generally by weight at a fixed price, while at least a portion of staple crops are kept for home consumption. Moreover, cash crops are critical sources of cash income to subsistence farmers. With respect to inputs, we expect to find more 4

6 evidence of decay in labor usage as opposed to fertilizer use, as labor can be used sporadically throughout the growing season while fertilizer purchase and application are generally singular events. We would also expect to find less evidence of recall decay in relation to less common events, such as labor use in places where outside labor is uncommon (such as Rwanda). Finally, when both are collected in a single visit, we expect to find more evidence of decay related to input rather than harvest data, as the recall period is several months longer. 3. Data & Empirical Approach To investigate potential recall bias in agricultural harvest estimates, we make use of three nationally representative multi-topic household surveys from Sub-Saharan Africa, the 2004/2005 Malawi Integrated Household Survey (IHS), the 2005/2006 Kenya Integrated Household Budget Survey (KIHBS), and the 2001 Rwanda Enquête Integrale sur les Conditions de Vie des Menages (EICV). These surveys were chosen because of their 12 month fieldwork calendar and because interview locations were randomized across regions throughout the data collection. The 2004/2005 Malawi Integrated Household Survey (IHS) was collected over 13 months from March 2004 to March 2005, and covered all districts of Malawi (except Likoma island) (Malawi NSO 2005). Field work was conducted concurrently in the three main agricultural regions of the country (north, central and south) to prevent regional averages from being distorted by seasonal bias. The sample was selected using a two-stage stratified sampling design based on a frame from the 1998 census, and was structured to be representative at the district level. The total number of households interviewed was 11,280. Of these households, the analysis was restricted to those households involved in rain-fed agriculture. Households report agricultural information with respect to the most recently completed agricultural seasons (2002/03 or 2003/04). The 2005/2006 Kenya Integrated Household Budget Survey (KIHBS) was collected over 12 months from May 2005 to April 2006, covering all eight provinces of the country concurrently (Kenya NBS 2007). The KIHBS sample was also selected using a two-stage stratified sampling design based on national sample frame developed from the 1999 Population and Housing Census. The sample is designed to be representative within urban and rural stratifications at the national, provincial and at the level of the country s 69 districts. The total number of households 5

7 interviewed was 13,212, and the questionnaire refers to the most recently completed agricultural season. The 2001 Rwanda Enquête Integrale sur les Conditions de Vie des Menages (EICV) was collected over 21 months from October 1999 to June 2001, with the bulk of the data collection being done in the last 12 months (Rwanda Statistics Department, 2002). Data collection was done concurrently in all 12 prefectures of the country. The sample was selected using a twostage stratified sample design and the sample is designed to be representative at the prefecture-level. The total sample size was 6420 households, and refers to the most recently completed agricultural season. In the analysis, we regress information on harvest (in kilograms and in local currency) sales and input use (fertilizer and hired labor) on time elapsed between the harvest and the date of the interview. We explore two dimensions of recall bias: systematic under/over-reporting and greater noise (variance) in reporting by length of recall. For the former issue, we examine whether reporting changes by the number of months of recall, controlling for observed household characteristics. For the noise in reporting, even if the mean of the variable of interest does not change with time elapsed, the variance may increase. This is to say, as time passes, respondent are less accurate in their reporting, but that this inaccuracy does not vary systematically in direction. To test for this possibility, a White Test for heteroskedasticity is performed on non-binary variables and the time elapsed variable. Agriculture is the dominant source of income for the rural households in these surveys. In Kenya, 82 percent of rural households were engaged in agriculture, as opposed to 22 percent who were engaged in non-farm enterprise and 31 percent where at least one member was engaged in wage labor. Differences are even larger for rural households in Malawi and Rwanda. In Malawi, 95 percent of rural households were involved in agriculture, with only 30 percent engaged in non-farm enterprises and 16 percent in wage labor. In Rwanda, 99 percent of households were engaged in agriculture, and only 13 percent in non-farm enterprises and 30 percent in wage labor. Most of the farming households are small holders with plot sizes generally between two and three acres (see Appendix Table 1). Both food staple and cash crops are analyzed for evidence of recall decay. Maize, a main staple crop, is examined in all three countries. Sorghum is also included for Rwanda as it is the most 6

8 widely grown staple in that country. The main cash crops studied are coffee in Kenya and Rwanda and tobacco in Malawi. In addition to harvest quantity, we also examine the value of both cash crop and staple crops sales; between 45 and 50 percent of rural respondents across the three countries sell at least part of the crop they produce. 1 A portion of these sales are cash crops bound for export, such as coffee in Rwanda and Kenya, and tobacco in Malawi, but staple food crops were also sold. In Malawi, about 16 percent of maize-producing households sell at least part of their maize crop, compared with 18 percent in Rwanda and 44 percent in Kenya for the last completed cropping season. In Rwanda, 42 percent of sorghum producers also sell part of their crop. It is also worth noting that, compared to Malawi and Rwanda, in Kenya sales of the staple crop maize are more important to household income, with a median value of 7,000 KSH compared with a median value of 3,800 KSH from coffee sales. 2 In both Rwanda and Malawi, the opposite is true. In Rwanda, the median value of maize sales is 2,000 RWF and sorghum 2,500 RWF, compared with 6,000 RWF for coffee sales. In Malawi, median sales of maize are just 1,800 MKW, compared with 10,000 MKW from tobacco production. With regard to input usage, labor usage varies across both crops and countries. Generally, the use of hired labor is most common in the Kenyan households, with about 30 percent of households, lower in Malawi at just below 20 percent, and lowest in Rwanda, where less than 10 percent engaged outside labor. Also, the incidence of hired labor seems to be slightly higher for cash crops as opposed to staple crops, though this difference is not statistically significant. Fertilizer use is fairly high in both Malawi and Kenya, with usage around 65 percent for staple crops and 85 percent for cash crops (fertilizer usage data is not available for Rwanda). Sample sizes and mean values for other variables of interest are included in Appendix Table 1. In addition to information on harvest and input use, this analysis also requires an estimate of the date of harvest. Since the time elapsed cannot be measured from the interview date to two separate harvest periods, we restrict the analysis to crops that are harvested only once during the agricultural calendar. Coffee and tobacco are both single harvest crops. Maize and sorghum, however, could have multiple harvests during the same year. In Rwanda, where the 1 The statistics cited on agriculture for these three surveys are restricted to the specific samples studied, which in the case of Kenya is region specific as described in the text. 2 Local currencies are Kenya Shillings (KSH), Malawi Kwacha (MK), and Rwanda Francs (RWF). 7

9 month of harvest information is recorded in the questionnaire, we restrict the analysis to households that indicated only one month of harvest, or two consecutive harvest months for maize or sorghum. In Malawi, seasonal rain patterns allow for one main and one secondary (dimba) harvest per year. We include only the main harvest from rain-fed plots. In Kenya, the calculation of the harvest date is complicated by high degrees of variability in agricultural zones within the country. The analysis is therefore limited to areas and crops with only one harvest per year and reliable information on the harvest month for the crop. 3 This includes only the Rift Valley for maize production, and the Eastern, Central, Western and Rift Valley regions for coffee production. The final sample sizes are therefore as follows. In Malawi, we restrict the sample to those households whose production and input use referred to the 2003/2004 agricultural season to limit the effects of annual variation in rainfall and government programs such as input subsidies. The sample size for Malawi is 4,940 households for maize, and 733 households for tobacco. In Kenya, for maize production in the Great Rift Valley province, the sample size is 3,285 households, and for coffee farming in the Eastern, Central, Western, and Great Rift Valley regions, we have 406 households. 4 In Rwanda, once the sample is restricted to those household with a single annual harvest, the sample size is 2,816 for maize, 3,119 for sorghum and 556 for coffee. For the harvest date assigned households, in the Rwanda EICV we have a direst reporting of the harvest month for maize and sorghum. Some respondents indicate multiple harvest months, but, as noted earlier, the analysis is limited to crops and households indicating a one harvest period, either during a single or two consecutive months. The KIHBS and IHS questionnaire do not record the date of the harvest, and the EICV does not include a harvest month for coffee. For cases in which an explicit date is not captured, the harvest date for the staple and cash crops are constructed based on input from local agronomists on the normal month of harvest by region to account for spatial variation in production seasons. The harvest month for maize in Malawi is assumed to be April for the southern region, May for the central region, and July for the more arid northern region. The harvest months for tobacco are assumed to be one month 3 In Kenya the design of the survey instrument makes the exclusion of irrigated plots more difficult. The incidence of irrigation in the Rift Valley, however, is quite low, with less than five percent of farms using irrigation on maize crops. These observations remain in the data set, though their exclusion does not markedly change the results. 4 Unlike in Malawi, we do not restrict the analysis in Kenya to only one growing season because of both sample size considerations and because in the KIHBS questionnaire we do not have a question that explicitly indicates to which growing season the respondent is referring. 8

10 earlier. In Kenya, the harvest date for maize in the Rift Valley is assumed to be November. For coffee, the sample is expanded to include the Eastern, Central and Western provinces, with coffee harvest dates assumed to be December in the Central, Western and Rift Valley provinces, and May in the Eastern province. Harvest month for coffee in Rwanda is assumed to be June. Time elapsed since harvest is the number of whole months between the last month of presumed or reported harvest month and the month of the interview. The same time elapsed variable is used for both input and harvest analysis, despite input use being several months before the harvest. As the planting date would be approximately the same number of months before the harvest date in all cases, using the harvest date should not affect estimations. For the empirical approach, there are two ways in which recall bias may impact the quality of the data. Respondents may be inaccurately remember details, such as the amount of fertilizer used or crops harvested, or they may forget events altogether, such as the use of fertilizer or the hiring of labor. To that end, we examine both the quantity of harvest in kilograms and local currency (assuming that farmers do not forget the harvest all together) and the application and quantity of inputs. We present three specifications for each outcome of interest. The first has minimal controls (land, geographic dummies, and, for harvest regressions, dummy variables for the unit in which the harvest was reported). Despite the geographic randomization across the fieldwork calendar, we include geographic dummies to control for location effects with may be correlated with harvest amounts, input use, and recall decay. The second specification has additional controls for the characteristics of the household head (gender, age and education of the household head and an indicator for head being main decision maker in household). Finally, the third includes an interaction between months and land, as it is possible that recall bias, while on average not statistically significant, varies by landholdings. Larger farmers, for example, may have more recall bias since they have higher production, or less recall bias because they keep better accounting records or have better unobservable entrepreneurial skills. Finally, it is possible that while the mean of the variable of interest does not change with time elapsed, perhaps the variance increases. This is to say, as time passes, respondent are less accurate in their reporting, but that this inaccuracy does not vary systematically in direction. To test for this possibility, a White Test for heteroskedasticity was performed on each non-binary variable of interest and the time-elapsed variable. 9

11 4. Results Starting with the harvest of staple crops, Table 1 shows the regressions of harvest amounts for staple food crops, maize in the three countries and sorghum in Rwanda. The time elapsed since harvest (in months) is not significant in any specification. The reported amount of maize and, for Rwanda only, sorghum harvest is not lower (or higher) for households interviewed further from the harvest date. Other covariates included generally have coefficients in the expected direction. Households with larger land holdings report a larger harvest. Controlling for land and other covariates, female-headship in Rwanda is associated with reporting of smaller harvests; in Kenya and Malawi, this is not the case. Households with better-educated heads, controlling for land holdings, have higher production, with exception of maize output in Rwanda. The third specification includes an interaction term for the time elapsed since harvest and landholdings; the coefficient on this term is negative but not statistically significant for maize production in Kenya and Malawi, and positive but not significant for sorghum production in Rwanda. In these cases, the time elapsed since harvest remains statistically insignificant. For maize production in Rwanda, the interaction term is negative and weakly significant, and the time elapsed variable is positive and weakly significant. This would indicate some evidence of greater recall bias (and under-reporting) in maize harvest associated with larger landholdings in Rwanda. For any given recall period, households with one additional acre report, on average, about 4 percent less maize output at the mean (84 kilograms). Our tests for a change in the variance of reporting by time-elapsed, we find no evidence of heteroskedasticity. Although we find no recall bias in harvest quantity, there is some indication of bias in the reports on crop sales in local current units. In Table 2, we find evidence of recall bias in the case of maize sales in Malawi. Households further from the harvest date, controlling for other covariates, report higher crop sales in local current units. An additional one month lag in interview from harvest results in maize sales reported to be about 6 percent higher relative to mean sales value of maize harvest among households in Malawi. Since we find no impact on kilograms produced (Table 1 results), this could be bias in either the recall of the portion of total harvest sold or the price of maize sold. Regressions using the amount sold, measured in kilograms as opposed to local currency units, show similar results, therefore indicating the source of this bias in the Malawi data is at least partly tied to reports of the kilograms sold 10

12 (results not presented). This significance disappears, however, when the interaction term is added, though the variable itself is not significant. The recall bias is therefore likely associated with larger farmers over-estimating levels of sales. In the cases of Kenya and Rwanda, we do not find any evidence of recall bias in staple crop sales, measured either in local currency units (Table 2) or kilograms sold (not presented). We also do not find any significance on the interaction term for Kenya, nor does the sign change on the time elapsed variable. In the case of maize in Rwanda, the interaction is not significant, though it does change the sign on the time elapsed variable. The interaction term on the sorghum in the Rwanda regression is weakly significant, and changes the sign on the time elapsed variable. Examining the other included covariates, variables tend to not be statistically significant, though there is some indication of weakly significant positive relationships with female headed households and years of education in Malawi. Also in Rwanda there are weakly significant relations with age and a strongly significant negative relationship between sorghum production and female headed households. The results for harvest of cash crops (coffee and tobacco), presented in Table 3, are similar to the staple crop findings presented in Table 1. Again, time elapsed since harvest is not associated with differential harvest reports. Also, the interaction term between land and time elapsed is not significant in any of the three countries. Female headship is associated with lower quantities sold of coffee in Rwanda; headship education is associated with more sales in Kenya and Malawi. Looking at the value of cash crop sales in local currency units, presented in Table 4, we again find that time elapsed is statistically significant only in Malawi. Tobacco sales are higher for households interviewed further from the harvest month. An additional one-month lag in interview from harvest results in tobacco sales reported to be 6 percent higher relative to mean sales. Also similar to the staple crop results, when the interaction term between land and time elapsed, though the variable itself is not significant, the significance on the time elapsed variable disappears. With regard to coffee harvests in Kenya and Rwanda, neither of the time elapsed variables are significant nor are the interaction terms. The test for heteroskedasticity of timeelapsed is marginally significant for maize sales in Malawi. These results are driven by three 11

13 outlier values in the later months. If these values are excluded, the test cannot reject the null hypothesis. The variance, therefore, does not seem to significantly increase over time for sales (or any of the other non-binary outcomes). Turning to inputs, we explore the recall bias in reporting hired labor and fertilizer use for staple and cash crops. For hired labor, presented in Table 5, we find some recall error with respect to reporting any hired labor for maize farming in Kenya and Malawi. In the case of Kenya, an additional one-month lag in interview is associated with about a 4 percentage point increase in probability of reporting hiring labor. This is a difference of 10 percent from the mean. In Malawi, the recall bias is associated with a decrease in probability of reporting of hired labor, about 15 percent decline relative to mean levels. We see no recall bias in Rwanda for hired labor on maize or sorghum plots. Additionally, those households with higher land holdings are more likely to employ outside labor, as well as those households with higher levels of education. For hired labor on cash crop production (Table 6), we find no recall bias in any country though the same relationships with education and landholdings apply. With regard to fertilizer usage, we employ three estimations: a probit model for fertilizer usage, a tobit model for the amount of fertilizer in kilograms, and an OLS model on non-zero values of fertilizer purchases (Tables 7, 8, and 9). Rwanda is excluded since the EICV survey did not collect this information. In both Kenya and Malawi, we find no evidence of recall bias in the overall use of fertilizer for either staple or cash crops (Table 7). We also find similar positive significant relationships with land holding and years of education as were found with labor usage. In the tobit regressions in Table 8, for the amount of fertilizer measured in kilograms used, when we include the set of controls, we find no evidence of bias; again we find the same relationship with land and education. Looking at expenditure for those households that purchased fertilizer, we find no evidence of recall bias. The coefficients on the months since harvest variable are generally positive but not statistically significant (Table 9). Table 10 offers a summary of the findings with respect to the results in Tables 1-9 and statistical significance for the time-elapsed variable. It could also be hypothesized that what we record as recall bias is actually caused by interviewer error. Due to the geographic distribution of field work and the construction of the time elapsed variable, if certain interviewers did not accurately collect harvest or input 12

14 information, this could be correlated with recall bias. To test for interviewer effects, dummy variables were added to the Malawi regressions (results not presented). 5 The coefficient on the main variable of interest, time elapsed since harvest, does not change it remains insignificant. In the case of the sales of maize and tobacco in monetary values, where we had previously seen some evidence of recall bias, the time elapsed variable remained significant even with controls for interviewer effects. It is also possible recall bias does not vary linearly with time. For example, a farmer could accurately remember production details up to a certain point following the event, then begin to forget as time passes. Alternatively the slope could increase then decrease over time, producing no slope on average. In order to allow more flexibility in the relationship between the dependent variable and the time elapsed since the harvest, we also estimate partial linear regression models. 6 The graphs for the harvest amount of staple and cash crop specifications are presented in the Appendix. From the graphs presented (Appendix Figures 1 7), the relationship between months since interview and our outcomes of interest appears to be flat or weakly linear. There is no point after which time the slope changes sharply. Also shown are the graphs related to the two variables where we found recall bias, the value of sales of maize and tobacco in Malawi (Figures 8 and 9). Both of these graphs show an upwards slope, consistent with the positive coefficient from the results above, but again, the relationship is linear. 5. Conclusion Agriculture is critical for development, particularly in sub-saharan Africa, where a large share of households depend on farms as their main source of livelihood. Despite this, data on agriculture remain weak and little research is undertaken on the quality of its statistics. Where data are available which links agriculture with non-farm activities, information on farming activities may be collected as part of a larger multi-purpose survey and based on a single interview. At best, households are interviewed over a short period of time just right after the harvest, which will minimize recall bias for crop production estimates. However, even in this case, input application may have occurred several months before and harvest may have been over several months. Adding to this, surveys data are often collected several months, often 12 or more months. These data potentially suffer from recall bias since both farming decisions and data collection 5 Interviewer identification variables are not available for Kenya or Rwanda. 6 We used the PLREG program for STATA written by Lokshin (2006). 13

15 occur over the course of several months, leading to long and variable recall periods across households. This paper studies the extent of recall bias in agricultural statistics, using data from three national surveys conducted over a year such that the recall period has sufficient random variation across samples households. Using various specifications on key input and crop production measurements, we generally find little evidence of recall bias. We find no evidence in either harvest amounts or fertilizer usage, and only a few cases with regard to crop sales and hired labor. Interestingly, in these cases, the direction of the bias is generally positive, contradicting the initial hypothesis that the forgetting of events as time passes would lead to under-reporting. The magnitude of the bias, however, is small, and is unlikely to markedly affect the quality of the resulting data. Recall bias does not generally seem to increase with time in terms of the variance or noise of reported activities; this is the case even for fertilizer usage with recall periods of up to 15 months. These findings are robust to partial linear regressions, allowing for a more flexible specification, which show flat or slight sloping curves. Finally, we find mixed evidence for the salience hypothesis. In support, some bias is found in hired labor usage, a common and repeated activity often of small relevance, but not in fertilizer usage, a single, more expensive event for farm households. We also find less evidence of decay in labor in Rwanda, where it is a comparatively much rarer event than in Kenya and Malawi. The evidence, however, is far from conclusive and should be interpreted with caution. On the other hand, we do not seem to find any differential recall bias between staple and cash crops, though this might be partially explained by the fact that many households also sell staples. Overall, the results seem to suggest that agricultural data collected through single visits over several months do not suffer from large recall decay, at least for the types of events and in the contacts studied here. Further research, however, is warranted. 14

16 Table 1: OLS Amount (in hundreds of kilos) of Staple Crop Harvest (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Kenya Maize Malawi Maize Rwanda Maize Rwanda Sorghum time elapsed since harvest (in months) * (0.20) (0.21) (0.62) (0.06) (0.06) (0.30) (0.02) (0.01) (0.03) (0.01) (0.01) (0.01) total land (in acres) 4.08*** 4.03*** 5.37** 2.79*** 3.00*** 3.58** 0.14** 0.14** 0.30** 0.12*** 0.10*** 0.07** (1.20) (1.20) (2.10) (0.68) (0.76) (1.57) (0.06) (0.06) (0.15) (0.02) (0.02) (0.03) female headed household ** -0.30** -0.31*** -0.32*** (3.09) (3.11) (0.38) (0.37) (0.14) (0.14) (0.08) (0.08) age *** 0.04*** (0.38) (0.38) (0.04) (0.04) (0.02) (0.02) (0.01) (0.01) age squared * -0.00* -0.00*** -0.00*** (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) years of education 0.91*** 0.83*** 0.16*** 0.16*** *** 0.09*** (0.29) (0.30) (0.04) (0.04) (0.05) (0.05) (0.02) (0.02) household head is main decision maker (2.74) (2.71) (0.51) (0.53) (0.16) (0.15) (0.10) (0.10) interaction land * time elapsed * 0.01 (0.28) (0.17) (0.02) (0.01) Breusch-Pagan p-statistic Number of observations 1,330 1,330 1,330 4,873 4,873 4,873 1,843 1,843 1,843 2,879 2,879 2,879 Adjusted R Notes: *** p<0.01, ** p<0.05, * p<0.1. Samples include households who reported growing each of the crops. Standard errors are in parentheses. Geographic dummies and the original unit in which the harvest was reported are included but not presented. Breusch-Pagan p-statistic is for the outcome and the time elapsed since harvest. 15

17 Table 2: OLS Amount (in local currency units) of Staple Crop Sales time elapsed since harvest (in months) (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Kenya Maize Malawi Maize Rwanda Maize Rwanda Sorghum *** 242.7*** (570.4) (631.9) (1,340.1) (54.8) (59.7) (133.3) (192.2) (297.0) (663.3) (142.6) (143.4) (120.4) total land (in acres) 6,612.9*** 6,752.1*** 5,678.3** 1,158.7*** 1,193.5*** 745.7* 3,109.6** 3,180.2** 5, female headed household (1,659.3) (1,664.5) (2,875.1) (155.8) (162.5) (420.2) (1,394.1) (1,352.7) (3,146.7) (258.8) (263.5) (323.7) -3, , , , ,783.0*** -2,861.3*** (6,811.9) (6,471.1) (348.6) (358.5) (1,898.9) (2,100.5) (845.7) (867.8) age * * 393.9** 403.7** (1,159.0) (1,152.8) (56.2) (55.6) (397.6) (419.8) (194.6) (193.7) age squared ** -4.6** (9.3) (9.3) (0.6) (0.6) (4.1) (4.4) (2.0) (2.0) years of education * household head is main decision maker interaction land * time elapsed (657.4) (679.4) (53.1) (53.0) (455.0) (420.8) (325.5) (326.4) 8, , , ,489.5 (8,183.6) (7,953.8) (643.2) (640.0) (5,184.5) (5,081.7) (922.4) (938.1) * (406.6) (59.2) (423.7) (62.9) Breusch-Pagan p-statistic Number of observations ,199 1,199 1,199 Adjusted R Notes: *** p<0.01, ** p<0.05, * p<0.1. Sample is households who reported any sales of the crop, among the sample in Table 1 growing the crop. Standard errors are in parentheses. Geographic dummies are included but not presented. Breusch-Pagan p-statistic is for the outcome and the time elapsed since harvest. 16

18 Table 3: OLS Amount (in hundreds of kilos) of Cash Crop Harvest (1) (2) (3) (4) (5) (6) (7) (8) (9) Kenya Coffee Malawi Tobacco Rwanda Coffee time elapsed since harvest (in months) * (0.1) (0.1) (0.2) (0.1) (0.1) (0.2) (0.0) (0.0) (0.0) total land (in acres) 0.7*** 0.7** *** 1.3*** 1.7*** 0.0*** 0.0** 0.0 (0.2) (0.3) (0.6) (0.2) (0.2) (0.7) (0.0) (0.0) (0.0) female headed household *** -0.2*** (0.7) (0.8) (0.5) (0.5) (0.0) (0.0) age *** 0.0*** (0.1) (0.1) (0.0) (0.0) (0.0) (0.0) age squared *** -0.0*** (0.0) (0.0) (0.0) (0.0) (0.0) (0.0) years of education 0.3** 0.3** 0.1* 0.1** household head is main decision maker (0.1) (0.1) (0.1) (0.1) (0.0) (0.0) (1.0) (1.0) (0.7) (0.7) (0.1) (0.1) interaction land * time elapsed (0.1) (0.1) (0.0) Breusch-Pagan p-statistic Number of observations Adjusted R Notes: *** p<0.01, ** p<0.05, * p<0.1. Samples include households who reported growing each of the crops. Standard errors are in parentheses. Geographic dummies and the original unit in which the harvest was reported are included but not presented. Breusch-Pagan p-statistic is for the outcome and the time elapsed since harvest. 17

19 Table 4: OLS Amount (in local currency units) of Cash Crop Sales time elapsed since harvest (in months) (1) (2) (3) (4) (5) (6) (7) (8) (9) Kenya Coffee Malawi Tobacco Rwanda Coffee ,161.67** (180.27) (229.47) (333.39) (413.61) (482.43) (1,262.97) (496.77) (392.96) (651.99) total land (in acres) 1,703.02** 1,977.73*** ,110.71*** 6,437.87*** 4, , ,635.05* 2, (664.95) (643.94) (1,038.65) (965.30) (1,024.08) (3,600.51) (940.02) (945.71) (1,621.17) female headed household -2, , ,455.39** -5,352.44** 11, , (1,363.83) (1,399.48) (2,463.63) (2,477.04) (13,733.39) (13,946.51) age (250.35) (245.86) (287.27) (290.23) (1,144.85) (1,150.47) age squared (1.90) (1.94) (2.84) (2.90) (11.79) (11.80) years of education * * household head is main decision maker (244.36) (244.40) (274.42) (270.30) (1,293.37) (1,272.17) 3, , , , ,977.94* 7,104.63* (1,950.19) (1,950.87) (7,505.42) (7,454.04) (3,672.05) (3,817.90) interaction land * time elapsed (168.68) (516.34) (234.86) Breusch-Pagan p-statistic Number of observations Adjusted R Notes: *** p<0.01, ** p<0.05, * p<0.1. Samples include households who reported any sales of the crop, amongeach sample in Table 1 growing the crop. Standard errors are in parentheses. Geographic dummies are included but not presented. Breusch-Pagan p-statistic is for the outcome and the time elapsed since harvest. 18

20 Table 5: Staple Crop Labor Probit (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Kenya Maize Malawi Maize Rwanda Maize Rwanda Sorghum time elapsed since harvest (in months) 0.04*** 0.04*** 0.05*** -0.02** -0.03*** -0.04** (0.01) (0.01) (0.02) (0.01) (0.01) (0.02) (0.02) (0.02) (0.02) (0.01) (0.01) (0.01) total land (in acres) 0.04*** 0.03** 0.06** 0.14*** 0.17*** 0.16*** 0.06*** 0.06*** 0.06*** 0.05*** 0.05*** 0.05*** (0.01) (0.01) (0.03) (0.01) (0.02) (0.05) (0.01) (0.02) (0.02) (0.01) (0.01) (0.02) female headed household (0.11) (0.11) (0.06) (0.06) (0.12) (0.12) (0.08) (0.08) age 0.04** 0.04** (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.01) (0.01) age squared -0.00* -0.00* (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) years of education 0.06*** 0.06*** 0.09*** 0.09*** 0.05** 0.05** 0.11*** 0.11*** (0.01) (0.01) (0.01) (0.01) (0.02) (0.02) (0.01) (0.01) household head is main decision maker (0.16) (0.16) (0.09) (0.09) (0.16) (0.16) (0.13) (0.13) interaction land * time elapsed (0.00) (0.01) (0.00) (0.00) Number of observations 1,330 1,330 1,330 4,827 4,827 4,827 2,314 2,314 2,314 2,884 2,884 2,884 Notes: *** p<0.01, ** p<0.05, * p<0.1sample is households who reported growing the crop. Standard errors are in parentheses. Other controls included in regressions but not presented include geographic dummies. 19

21 Table 6: Cash Crop Labor Probit (1) (2) (3) (4) (5) (6) (7) (8) (9) Kenya Coffee Malawi Tobacco Rwanda Coffee time elapsed since harvest (in months) * (0.02) (0.03) (0.04) (0.02) (0.03) (0.05) (0.02) (0.02) (0.03) total land (in acres) 0.09** 0.08** *** 0.19*** *** 0.06*** (0.03) (0.04) (0.09) (0.03) (0.03) (0.09) (0.02) (0.02) (0.04) female headed household (0.20) (0.20) (0.27) (0.26) (0.22) (0.22) age (0.03) (0.03) (0.02) (0.02) (0.03) (0.03) age squared (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) years of education 0.13*** 0.13*** 0.08*** 0.08*** 0.08** 0.07** (0.03) (0.03) (0.02) (0.02) (0.03) (0.03) household head is main decision maker (0.21) (0.21) (0.52) (0.52) interaction land * time elapsed * (0.01) (0.01) (0.01) Number of observations Notes: *** p<0.01, ** p<0.05, * p<0.1 Sample is households who reported growing the crop. Standard errors are in parentheses. Geographic dummies are also included but not presented. 20

22 Table 7: Fertilizer Use Probit (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Kenya Maize Kenya Coffee Malawi Maize Malawi Tobacco time elapsed since harvest (in months) (0.01) (0.02) (0.02) (0.05) (0.05) (0.06) (0.01) (0.01) (0.02) (0.03) (0.03) (0.06) total land (in acres) 0.02* 0.02* *** 0.18*** 0.23*** 0.14*** 0.18*** 0.10 (0.01) (0.01) (0.02) (0.06) (0.10) (0.16) (0.01) (0.02) (0.05) (0.05) (0.06) (0.13) female headed household (0.12) (0.12) (0.43) (0.42) (0.05) (0.05) (0.23) (0.23) age (0.02) (0.02) (0.07) (0.07) (0.01) (0.01) (0.03) (0.03) age squared (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) years of education 0.06*** 0.06*** *** 0.06*** 0.04** 0.04** (0.01) (0.01) (0.05) (0.05) (0.01) (0.01) (0.02) (0.02) household head is main decision maker (0.17) (0.17) (0.38) (0.38) (0.09) (0.09) (0.52) (0.53) interaction land * time elapsed (0.00) (0.03) (0.01) (0.02) Number of observations 1,209 1,209 1, ,827 4,827 4, Notes: *** p<0.01, ** p<0.05, * p<0.1sample is households who reported growing the crop as primary crop on a plot. Standard errors are in parentheses. Geographic dummies are included but not presented. 21

23 Table 8: Fertilizer Amount Tobit (1) (2) (3) (5) (6) (7) (8) (9) (10) (11) (12) (13) Kenya Maize Kenya Coffee Malawi Maize Malawi Tobacco time elapsed since harvest (in months) * (6.25) (6.93) (36.97) (5.42) (7.35) (35.66) (3.84) (4.20) (6.76) (3.42) (3.67) (8.89) total land (in acres) * *** 48.91*** * 63.31*** 70.12*** 86.96*** 63.77*** 64.61*** 54.02*** (26.20) (29.49) (67.28) (13.27) (12.24) (25.14) (8.74) (10.07) (23.74) (6.61) (6.53) (19.43) female headed household * (42.17) (48.85) (24.43) (4.67) (26.66) (26.65) (27.12) (26.76) age ** 8.33** (7.20) (7.53) (4.68) (0.04) (3.40) (3.39) (2.84) (2.85) age squared (0.07) (0.07) (0.04) (6.85) (0.03) (0.03) (0.03) (0.03) years of education 25.43*** 29.08** * 17.62*** 17.55*** 5.99** 5.83** (9.50) (12.51) (6.82) (42.47) (3.70) (3.68) (2.35) (2.40) household head is main decision maker * (53.12) (55.71) (52.20) (6.48) (37.47) (37.39) (40.34) (41.01) interaction land * time elapsed (16.55) (7.89) (2.87) (2.89) Number of observations 1,226 1,226 1, ,875 4,875 4, Notes: *** p<0.01, ** p<0.05, * p<0.1. Sample is households who reported growing the crop as primary crop on a plot; it includes those with no fertilizer use. Standard errors are in parentheses. Geographic dummies are included but not presented. 22

24 Table 9: Fertilizer Value OLS (in local currency units) (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Kenya Maize Kenya Coffee Malawi Maize Malawi Tobacco time elapsed since harvest (in months) * *** (85.38) (84.92) (196.22) (168.75) (276.32) (264.54) (33.89) (37.19) (137.34) (160.97) (147.42) (242.94) total land (in acres) *** *** 1,116.34* 1,201.37*** 1,277.26*** ,089.88*** 1,110.02*** 1,258.87** 1,642.84*** 1,818.00*** 1,126.48* (212.46) (197.73) (613.94) (426.37) (395.21) (1,251.68) (270.08) (300.37) (588.49) (169.68) (183.92) (588.52) female headed household , , ,223.31* -1, (839.97) (781.70) (786.20) (843.89) (364.64) (366.26) (738.67) (728.94) age ** 65.56** ** ** (97.04) (92.54) (123.45) (123.92) (28.92) (28.82) (78.36) (78.47) age squared * -0.54* -2.31*** -2.35*** (0.84) (0.80) (0.99) (1.02) (0.29) (0.29) (0.84) (0.84) years of education *** *** *** *** (58.56) (58.98) (202.71) (224.20) (30.00) (29.77) (67.46) (67.90) household head is main decision maker , , (546.62) (551.92) (1,236.26) (1,027.67) (266.26) (279.67) (1,059.24) (1,074.62) interaction land * time elapsed (95.45) (222.81) (59.19) (84.34) Breusch-Pagan p-statistic Number of observations ,018 2,018 2, Adjusted R Notes: *** p<0.01, ** p<0.05, * p<0.1 Samples include households who reported growing the crop as primary crop on a plot and do not include those with no fertilizer use. Standard errors are in parentheses. Geographic dummies are included but not presented. Breusch-Pagan p-statistic is for the outcome and the time elapsed since harvest. 23

25 Table 10: Summary of Results Kenya Malawi Rwanda Rwanda Kenya Malawi Rwanda Maize Maize Maize Sorghum Coffee Tobacco Coffee Harvest (no interaction) ns ns ns ns ns ns ns Harvest Sales (no interaction) ns +*** ns ns ns +** ns Harvest (with interaction) ns ns +* ns ns ns ns Harvest Sales (with interaction) ns ns ns ns ns ns ns Labor Use +*** -*** ns ns ns ns ns Fertilizer Use ns ns ns ns Fertilizer Amount ns ns ns ns Fertilizer Value ns ns +*** ns Notes: ns indicates no statistical significance of the time-elapsed coefficient. *** p<0.01, ** p<0.05, * p<0.1. +/- indicates a positive/negative coefficient. 24

26 Appendix Figures: Partial Linear Graphs Staple Crops Figure 1 Figure 2 Figure 3 Figure 4 Cash Crops Figure 5 Figure 6 25