Overview of the Organized Symposium # OS 06-02: Symposia 6 29 th International Conference of Agricultural Economists Milan, Italy, August 9-14, 2015

Transcription

1

2 Overview of the Organized Symposium # OS 06-02: Symposia 6 29 th International Conference of Agricultural Economists Milan, Italy, August 9-14, 2015 Title: Improving the methods of measuring varietal adoption by farmers in developing countries: Recent experience with the use of DNA fingerprinting and implications for tracking adoption and assessing impacts Theme: Adoption of Improved Varieties Using Innovative Methods Session Organizers: Mywish K. Maredia (maredia@msu.edu), Michigan State University, and Byron Reyes (b.reyes@cgiar.org), International Center for Tropical Agriculture (CIAT) Abstract: Identifying and measuring the area under improved varieties and assessing varietal turnover plays a central role in varietal adoption and impact assessments. These studies have mostly relied on farmers responses in household surveys to estimate these indicators. This method of farmer elicitation to estimate varietal adoption can be fairly accurate when the varietal turnover is high and the seed system is well-functioning. However, when the formal seed system is non-existent or ineffective, and farmers mostly rely on harvested grain as the main source of planting material, the reliability of estimating varietal adoption using farmer or expert elicitation method can be challenging. This symposium brings together researchers who have used the DNA-fingerprinting method for varietal identification. It provides a forum for exchange of ideas and sharing new insights on the challenges and potential of using this innovative method for estimating varietal adoption and increasing the accuracy of results of impact assessments. JEL Codes: C81 Methodology for Collecting, Estimating, and Organizing Microeconomic Data, Data Access, C83 Survey Methods, Sampling Methods, O3 Technological Change; Research and Development 1

3 1. Background and Rationale Since the pioneering research by Griliches on assessing the impact of hybrid corn adoption in the U.S. almost six decades ago, the interest in measuring the impacts of adoption of improved technology by farmers has expanded to include a gamut of agricultural technologies in both developed and developing country settings. Among the most widely assessed agricultural technologies in the developing country context is the adoption of improved varieties. These assessments have consistently reported that adoption of improved varieties and rapid varietal turnover increases productivity, income and other measures of welfare of farm households. Central to these assessments is the identification of improved varieties, measuring the area under those varieties, and assessing varietal turnover. Most varietal adoption and impact assessment studies in the past have relied on farmers responses in household level surveys to estimate these indicators. This method of farmer elicitation to estimate varietal adoption can be fairly accurate in a setting where farmers are mostly planting seeds freshly purchased or acquired from the formal seed market as certified or truthfully labeled seed, and the seed system is well-functioning and effective in monitoring the quality and genetic identity of varieties being sold by the seed vendors. However, in settings where the formal seed system is non-existent or ineffective, and farmers mostly rely on harvested grain (either from their own farms or acquired from other farmers or purchased from the market) as the main source of planting material, the reliability of estimating varietal adoption using this method is challenging. By implication, it also makes the results of impact assessments based on those adoption estimates questionable. The challenges stem from several confounding factors. These include farmers inability to identify varieties by names, the inconsistency in the names of the varieties as identified by the farmers and what is in the variety registration list (i.e., varieties may have locally adapted names), and the loss of genetic identity due to cross-pollination when seeds are recycled several seasons. DNA fingerprinting, which is routinely used by plant breeders and is becoming widely available and affordable, offers a reliable method to address these challenges and to accurately identify varieties grown by farmers. The use of this method can thus increase the accuracy and credibility in the interpretation of results of economic analysis based on household surveys that estimate the causal link 2

4 between the adoption of improved varieties and the impact on crop productivity and income. Despite the advantages, there are several issues related to sampling, logistics, and cost-effectiveness of using this innovative method that need to be investigated and addressed before DNA fingerprinting method becomes routine for tracking varietal adoption and assessing the impact at the farm level. This proposed symposium proposes to bring together economists who have recently used the DNA fingerprinting method for varietal identification. The symposium will provide a forum for exchange of ideas and sharing new insights on the challenges and potential of using this innovative method for estimating varietal adoption and varietal turnover, and increasing the accuracy of estimates of impacts of varietal adoption by farmers. 2. Symposium Format The symposium will feature four short paper presentations preceded by introductory remarks by the symposium organizers. The presenters will share their recent experiences of using the DNA fingerprinting methods to estimate adoption rates of rice varieties in Bolivia, wheat and maize varieties in Ethiopia, cassava varieties in Ghana, and bean varieties in Zambia. Approximately one-third of the time will be left for discussion and obtaining feedback from attendees. Given the novelty of this method, it is expected that symposium attendees would engage in discussing the methodology, its cost-effectiveness, and the implications of the results on the extent to which there are estimation errors in existing varietal adoption data. This should provide good feedback that could be used to advance this methodology for tracking varietal adoption and in future technology impact studies that rely on farm household survey data. 3. Presenters and their Bios (in alphabetical order) Ricardo Labarta <r.labarta@cgiar.org> Ricardo Labarta is a Senior Scientist and the Impact Assessment Research Leader at the International Center for Tropical Agriculture (CIAT). He has many years of experience working on impact assessment of agricultural technologies and natural resource management. His economic research has focused in biotechnology, integrated crop 3

5 management, experimental economics, agricultural extension, public-private partnerships and natural resource economics. He has developed his professional career working in many countries of Africa and Latin America and has previously worked for the International Potato Center (CIP) and World Agroforestry Center (ICRAF). Ricardo Labarta holds a PhD in Environmental and Natural Resource Economics, and a MSc in Agricultural Economics, both from Michigan State University. Mywish K. Maredia <maredia@msu.edu> Mywish Maredia has worked extensively in the area of impact assessment and the economics of agricultural science and technology. She is currently a Professor, International Development, in the Department of Agricultural, Food and Resource Economics at Michigan State University. Her research focuses on the evaluation and impact assessment of international development interventions. She has served as the Deputy Director of the USAID funded Dry Grain Pulses CRSP from , and as a member of the Standing Panel on Impact Assessment of the CGIAR s Science Council from She was the recipient of the Outstanding Ph.D. Dissertation Award in 1994 from the American Agricultural Economics Association. Byron Reyes <b.reyes@cgiar.org> Byron Reyes is currently an Agricultural Economist at the International Center for Tropical Agriculture (CIAT) based in Nicaragua. His main research area is impact assessment of agricultural research and development. He has research experience in Latin America (Costa Rica, Ecuador, El Salvador, Guatemala, Honduras and Nicaragua) and Africa (Angola, Burkina Faso, Ghana, Mozambique and Zambia) and has experience in household level survey and impact evaluation design, data collection, complex household survey data analysis and testing alternative methods to estimate adoption of improved varieties. Chilot Yirga Tizale <ctizale@yahoo.com> Chilot Yirga is a socio-economist at the Ethiopian Institute of Agricultural Research, Addis Ababa. He has expertise in Agricultural development, climate change, 4

6 environmental science, and sustainable development. He has led many household survey projects and authored several publications based on his research on assessing the impact of the adoption of agricultural technologies by farmers in Ethiopia. Greg Traxler Greg Traxler is a Senior Lecturer in the Evans School of Public Affairs at the University of Washington. Prior to that he was a Senior Program Officer at the Bill & Melinda Gates Foundation, and Professor in the Department of Agricultural Economics at Auburn University. His teaching and research focuses on research policy and the impacts of agricultural technology in the U.S. and in developing nations. 4. Summaries of Papers to be Presented Paper 1 Using DNA Fingerprinting to Estimate the Bias of Farm Survey Identification of the Diffusion of Improved Crop Varieties in Ethiopia. Presenter and author(s): Chilot Yirga Tizale or Greg Traxler Co-authors: Mariana Kim, and Dawit Alemu All existing evidence on the adoption of improved crop varieties has been collected either by expert elicitation or through farm surveys. The reliability of these approaches has never been verified, leaving the bias and standard errors of existing diffusion estimates unknown. This study will report the results of a nationally representative survey of 2,000 maize and 2,000 wheat farmers in Ethiopia conducted during the 2014/15 growing season. Plant samples taken from farmers fields were compared to breeders seed reference materials using DNA fingerprinting to generate precise estimates of the area under each variety released in the country. The DNA fingerprinting estimates were compared to estimates derived using farmer variety identification. Analysis of the relative productivity of each variety was also conducted using crop cut estimated yields and information on crop management practices. Survey information on seed sources and seed handling was used to assess the impact of seed management on yield and variety performance. 5

7 The study intended to evaluate the merit of using DNA fingerprinting for tracking crop varietal use by smallholder farmers in Ethiopia. The primary objective was to ascertain whether DNA provides a superior alternative to other approaches for tracking varietal diffusion. Specifically, the study aimed to determine whether DNA fingerprinting is accurate, cost-effective, and feasible. The main study questions are: Are the data developed through fingerprinting accurate and specific enough to provide a useful account of varietal diffusion? Can DNA fingerprinting be done at a cost that can be incorporated and scaled nationally? Can the benefits of the study outweigh the required investments in building the operational, logistical, and technological capacity of Ethiopian agencies and actors? Preliminary results confirm widespread use of improved wheat and maize varieties in Ethiopia, but that farmer recall data underestimates the diffusion levels of improved varieties. This phenomenon is more pronounced in the case of wheat than for maize. Preliminary estimates of adoption levels of improved wheat varieties based on the farmer recall information is 62% compared to 96% from the DNA fingerprinting approach. In the case of maize, estimates based on farmer recall data indicate 56% adoption rate for improved varieties compared to 61 % from the DNA finger printing. Genetic fingerprinting appears to be a technically feasible method for tracking varietal diffusion and that generates more precise estimates than farmer recall data. All adoption estimates will be updated in June-July The relatively high levels of mismatch between what the farmers reported and the actual genetic material in their fields warrants further investigation. Only 9% of wheat farmers correctly identified the variety that they were using; in the case of maize, this figure was higher but still only 30%. Starting from the premise that farmers have no incentive to hide or provide misleading information on varieties in their fields, it is plausible to posit that the majority of the respondents reported what they knew; it would be desirable to find out if there are any incentives for farmers to misreport to hide their use of de-listed varieties. Understanding the cause and source of distortion or information gap could be achieved by examining the existing farmer information networks including 6

8 the sources, nature and medium of communicating information on improved technologies to farmers. For Ethiopia s extension and seed delivery systems the results suggest that their primary clients do not have adequate information on maize and wheat varieties in their fields. Likewise, given that the seed demand assessment is based on information gathered by frontline extension staff interviews with farmers, the seed demand projections that inform foundation seed production are inaccurate, leading to a mismatch between seed supply and demand. For the research system, these findings imply that that previously adoption estimates are likely underestimates. There is potential for wider application of the DNA fingerprinting technique in Ethiopia to balance seed supply and demand. DNA analysis could also enhance the capacity of the seed regulatory authorities in Ethiopia by helping to resolve some of the seed quality disputes arising from reported contamination. The identification of one wheat variety with two different names (registered in Variety Release Registry) among the reference materials in the present pilot study confirms the need for a more rigorous check before a variety is released. Generating more accurate estimates of adoption levels using genetic fingerprinting approach requires additional inputs in terms of human capital and operational costs, compared to the traditional farm household surveys and use of secondary data. A typical data household survey data collection costs range between 45 to 60 USD per respondent. DNA fingerprinting is estimated to add 15 to 20 USD per respondent. These are rough estimates based on variable cost components of the pilot as opposed to an estimation of the full research costs. Paper 2 Assessing Impacts of the adoption of modern rice varieties using DNA fingerprinting to identify varieties in farmer fields: A case study in Bolivia Presenter and author: Ricardo A. Labarta Co-authors: Jose M. Martinez, Diana C. Lopera, Carolina Gonzalez, Constanza Quintero, Gerardo Gallego, Juana Viruez and Roger Taboada 7

9 Documenting the adoption of agricultural technologies has a long history in agricultural economics. Perhaps the most studied topic has been the adoption of improved crop varieties which has resulted in large evidence of the level of adoption of different improved crop varieties and about the determinants that favor or constraint the adoption of these varieties. In most of these studies researchers have used survey instruments and relied on farmers self-reported variety names to estimate varietal adoption and the determinants of this adoption. In more recent studies, there has been an attempt to verify whether the varieties reported by farmers are actually the varieties that they are growing on their crop plots. In spite that many adoption studies have included crop experts and used photographs and morphological descriptors to verify the names and the origin of the varieties reported by farmers, many of these recent studies have found difficulties in verifying the crop varieties being grown by farmers and have failed to identify important number of varieties in farmers fields (Walker et al 2014, Larochelle et al 2013). This has created some uncertainties about the level of adoption of some crop varieties and the validity of some studies aiming to estimate the determinants of these varieties. DNA fingerprinting has been proposed as a means that could help to improve crop varietal identification and especially in recent times that is becoming affordable for large samples. This study tests the use of fingerprinting in the identification of rice varieties and compares it with the use of varietal identification self-reported by rice growers in Bolivia. It then provides an estimation of the level of adoption of modern improved varieties and allow to study the determinants of the adoption of these modern varieties. The study then compares the results of the adoption rates and estimation of adoption determinants using farmers self-reported varietal identification This study was designed to first estimate the level of adoption of modern improved rice varieties in Bolivia and then the determinants of the adoption of these modern varieties. It has two components and the first one collected household and plot level data from rice growers of the departments of Santa Cruz, Beni and Cochabamba that represents 97% of the total rice production in Bolivia. This field level data included the registration of all varieties planted by rice growers interviewed and the acreage of each of them. 8

10 The analysis reported in this paper is based on a sample of 298 households located in 56 different communities that were randomly selected. For the 298 farmers that had planting material at the time of the interview, we first ask farmers to identify all rice varieties that they have grown in the last growing season and to estimate the area under each rice variety. We then requested farmers in the same interview to provide a small quantity of seed (10-20 grains) that were immediately inserted in a small paper bag and labelled with information about the household and the location of the collection. All the rice seed samples collected from Bolivia were shipped to CIAT headquarters in Cali, Colombia for DNA extraction and analysis using Single Nucleotide Polymorphisms (SNPs) that is based on the fluidigm genotyping. As a second component of this study, the DNA fingerprinting method served as an alternative varietal identification of all planting material found on the ground. Our Results indicate that using DNA fingerprinting varietal identification the adoption rate of modern varieties is estimated in 44.96% compared with the 41.58% that is estimated when using farmers self-identification of rice varieties. Thus, using only farmers self-identification of varieties may lead to an underestimation of the adoption of modern rice varieties of almost 3.5 percentage points. The difference in estimates for old varieties reaches only 2.25 percentage points. It seems that all unidentified varieties by farmers where later confirmed as modern varieties by the genetic analysis. But behind these relatively modest changes in the estimated adoption rate there are larger mismatches in varieties that were identified by farmers as modern or old and those that were identified by those categories by the DNA fingerprinting analysis. Our analysis found that 8.38% of the varieties that were reported by farmers as old varieties turned out to be modern varieties. Likewise, 3.23% of the varieties that were identified as modern varieties by farmers where really old varieties according to the DNA fingerprinting identification. Regarding the estimation of the determinants of the adoption of modern rice varieties, our estimations found mixed results. On one hand both definitions of modern variety indicates that there is a strong and significant effect of the distance of San Juan the Yapacani (the main center of dissemination of rice technologies in Bolivia) and the effect of being located in the Beni department. While the farther the distance from San 9

11 Juan de Yapacani reduces considerably the probability of adopting modern rice varieties, being located in the Beni department increases significantly the probability of adoption of modern varieties. This can be explained by the large demand of new seed that is being experimented by Beni that is a new frontier of the rice production. On the other hand both definitions of modern varieties also differ in other determinants of the adoption of these modern varieties. While an adoption using farmers self-reported varietal identification shows a significant and positive effect of years of education of household head and farm size, an adoption using DNA fingerprinting results suggests a significant but negative effect of household head age. Our findings confirm the importance of using DNA fingerprinting as an alternative and necessary method to identify varieties. Although rice is a fairly homogenous crop with very few varieties released in Latin America, relying only in farmers (small holders) self-identification of the varieties that they are growing may lead to wrong conclusions. In the case of rice it seems to be an underestimation of the adoption of modern varieties and different results to understand the determinants of this adoption. However, more research would be needed in order to further analyze the implications of a different identification of rice varieties using DNA fingerprinting and the challenges faced in the application of this advance method on adoption studies. Paper 3 Testing the effectiveness of different approaches of collecting variety-specific adoption data against the benchmark of DNA fingerprinting: The case of cassava in Ghana Presenter and author: Byron Reyes Co-authors: Mywish Maredia, Joe Manu, Awere Dankyi, Peter Kulakow, Ismail Rabbi, Elizabeth Parkes, and Tahirou Abdoulaye Assessing technology adoption and its impact has expanded since the pioneering work of Griliches. Among the most widely assessed agricultural technologies in the developing country context is the adoption of improved varieties. Central to these assessments is the identification of improved varieties, measuring the area under those varieties, and assessing varietal turnover. Most varietal adoption and impact assessment studies in the past have relied on farmers responses in household level surveys to 10

12 estimate these indicators (although there are other alternatives-each with its pros and cons- also used such as secondary data, expert opinions, etc). The accuracy of the indicators thus depends on how well the farmer identifies each variety grown, which in turn depends on them knowing the name of the variety which in many cases changes from one source to another (i.e., there are no consistency in the names for the same genetic material or variety). Because of this, it is of utmost importance to estimate adoption of varieties using techniques that can provide more accurate information. A technique that is becoming more popular among economists includes testing DNA from plant tissue to separate individuals with different genetic characteristics, commonly called DNA fingerprinting. Under the Strengthening Impact Assessment in the CGIAR (SIAC) project, several pilot studies were implemented to explore and understand practical challenges of using DNA fingerprinting as a method of varietal identification as part of a farmer survey, test the effectiveness of different methods of varietal identification against a benchmark (i.e., DNA fingerprinting), and to come up with lessons learned and recommendations on methods that can be scaled up. This paper presents the results from the pilot study implemented in 2013 in Ghana for cassava (Manihot esculenta). Cassava was selected for one of the pilot studies based on its importance in the diet and the interest from national and CGIAR centers to collaborate in this study, which required a multi-disciplinary team of researchers. The study was conducted in the Brong Ahafo, Ashanti and Eastern regions, which account for 61% of cassava production. A total of 500 households across 100 villages (five farmers per village) were sampled using a multistage cluster sampling method, where districts, villages and farmers were randomly chosen. However, the realized sample was of 495 households. The survey was implemented in October-November 2013 and was coordinated by a research team led by the cassava breeder from the Ghana Crops Research Institute (CRI) and a socioeconomist from the Agriculture Innovation Consult (AIC). The survey included a household interview, a field visit to one cassava plot (the plot with the most varieties), and obtaining tissue samples from cassava plants for DNA extraction and analysis. Each survey team consisted of three people with different responsibilities: an enumerator responsible for completing the household modules, a 11

13 cassava expert responsible for completing the field module, and a DNA sampling expert responsible for collecting, labeling and storing the plant tissue samples as per an established protocol. A total of 855 samples were collected for DNA fingerprinting. Among the methods for variety identification tested were A) farmers responses about name (A1) and type of variety (A2), B) asking farmers to indicate the morphological characteristics of the plants from observing pictures, information that was later used for identification of varieties, C) cassava experts visiting a cassava field and recording observations on varietal characteristics which was later used for identification (C1) and also identifying the variety based on these observed characteristics (C2), D) taking pictures of characteristics of the plants during the field visit for latter identification by experts, and E) DNA fingerprinting from leaf tissue taken during the field visit. The estimates of adoption rates obtained from methods A-D were compared to varietal identification obtained from DNA fingerprinting (method E), the benchmark. The information collected using methods B and C1 was used to generate a unique code because each of the eleven morphological characteristic evaluated had a value (0-9) that when combined, allowed creating a unique code, which was compared to a similar code generated from known morphological characteristic of the varieties released in the country (contained in a library created for this purpose). Method D required assembling a panel of crop experts familiar with the varieties grown in the study area, including breeders and technicians from different institutions, to look at the pictures to identify varieties. Method E required first establishing a reference library of DNA fingerprints, and then applying the same or a sub-set of markers used to establish the reference library to genotype the samples collected. The results suggest that there may be considerable differences between the estimates of adoption rates obtained by these methods, compared to fingerprinting results. About 180 variety names were reported by farmers, being Debor and Ankra the most common names. When asked the type of varieties grown (method A2), farmers reported that most were local varieties (87%), followed by not knowing the type of variety (7%) and growing improved varieties (6%). However, only 20 of the 51 names given by farmers when reporting growing an improved variety (IV) matched with the name of an IV. Several farmers also (mistakenly) gave names of IVs when reporting growing a local 12

14 variety. During the field visit, cassava experts could not identify the name of a variety for 299 observations but they reported (method C2) that most varieties were local (88%), followed by not knowing the type of variety (7%) and improved varieties (5%). In contrast, experts looking at photos (method D) reported that most varieties were local (70%), followed by improved varieties (16%) and not knowing the type of variety (14%). Results from method B are still being analyzed and are not presented in this paper. The results from method E suggest that some of the released improved varieties are genetically identical, many released varieties are mixtures or hybrids, and that library accessions representing both released and local varieties (or landraces) fall under the same cluster groups. The latter represents a challenge because farmer samples that fall under these cluster groups could be classified either as an IV or a local variety. Due to the ambiguity of the DNA results regarding the cluster groups, two scenarios were analyzed: a liberal scenario where farmer samples that fall in any of these cluster groups were assumed to be IVs, and a conservative scenario where they are assumed to be local varieties. The DNA results suggest that adoption of IVs ranged from 4% (conservative scenario) to 31% (liberal scenario). When comparing methods A1, A2, C1, C2, and D to the benchmark (method E), in the conservative scenario where the DNA results suggest 4% adoption rate, adoption rates estimated from methods A1 and D greatly differ (1% and 15%, respectively) from the truth. However, estimations of adoption rates from methods A2, C1 and C2 are close to the truth, being method C2 the closest. In the liberal scenario where the DNA results suggest 31% adoption rate, adoption rates estimated from all other methods considerably underestimate adoption rates (the highest adoption rate came from method D--15%). These results suggest that no method stands out to be the most effective on all measures, that methods based on farmer elicitation and field observations by experts provide closest estimates under the conservative scenario but with high error rate, no method come close to the truth in adoption estimates in the liberal scenario, identifying cassava varieties accurately by name when hundreds of names exist is a challenge across all methods, and adoption estimates by experts are substantially higher than other methods and have much higher false positives. 13

15 Further, when there is a diversity of names by which farmers call their varieties, the traditional method of farmer elicitation give an underestimation of adoption of improved varieties. Thus, the current method most commonly used (i.e., farmer elicitation) may not be an accurate method for measuring varietal turnover and assessment of type II (new IVs replacing old IVs) benefits of plant breeding research. Also, farmers and experts are better able to give an aggregate assessment of the adoption of improved varieties as a category than by name. Finally, none of the alternative and non-traditional methods tested emerged as most effective and their scalability remains questionable on the grounds of cost-effectiveness. Paper 4 Testing the effectiveness of different approaches of collecting variety-specific adoption data against the benchmark of DNA fingerprinting: The case of beans in Zambia Presenter and author: Mywish Maredia Co-authors: Byron Reyes, Enid Katungi, Petan Hamazakaza, Kennedy Muimui, Bodo Raatz and Clare Mukankusi Rationale and objective Varietal adoption based on household surveys has mostly relied on farmers response to varietal identification. This method can give biased estimates if farmers are unable to give the correct name or give names that do not match with the improved variety list. To tackle these potential problems requires time intensive data collection such as including follow-up questions in the survey instrument, visiting the field to observe plant characteristics, or collecting sample materials (i.e., photos, seeds/plant tissues) from the farmers for later verification by experts. Each of these approaches has implications on the cost of data collection and the accuracy with which they can correctly identify a variety. DNA fingerprinting offers a reliable method to accurately identify varieties grown by farmers. The use of this method can: a) Increase the accuracy and credibility in the interpretation of results of economic analysis based on household surveys that estimate the causal link between the adoption of improved varieties and the impact on crop 14

16 productivity and income; and b) Serve as a benchmark against which to compare the effectiveness of other potential methods for scaling up. However, despite the advantages, DNA fingerprinting has not been used widely for tracking varietal adoption. Questions related to sampling, logistics, and costeffectiveness of using this innovative method remains to be explored. This study reports the results of a pilot study conducted in Zambia to: 1) Explore and understand some of the practical challenges of using DNA fingerprinting as a method of varietal identification as part of a farmer survey; 2) To test the effectiveness of different methods of varietal identification against the benchmark of DNA fingerprinting; and 3) To come up with lessons learned and recommendations on methods / approaches that can be scaled up. Methodology Four methods of tracking varietal adoption using farm household surveys were evaluated against the benchmark of DNA fingerprinting. These methods can be grouped into two types farmer elicitation methods (methods A-B) and expert elicitation methods (methods C-D). These methods include: Method A: As part of the survey instrument, farmers were asked to provide the name(s) and type (improved vs. local) of varieties planted in the current planting season (for cassava) or the last completed season (for beans). This method of eliciting the name of the variety was also implemented with vendors selling dry beans in two local markets in the study region. Method B: This method involved showing the farmer seed samples representing different varieties and asking him/her to identify the seed sample that matched the varieties grown on their farm. Method C: This method consisted of taking photographs of harvested seeds and later using these pictures for varietal identification by a panel of experts. Method D: Consisted for collecting seed samples of varieties grown by the farmer for later identification by a panel of experts. 15

17 For both methods C and D, the expert panel consisted of breeders and extension staff from the study districts. After showing the photos or seed samples, a consensus name and type of variety (improved, local, mix) was recorded for each sample. The study was conducted in Muchinga and Northern provinces of Zambia based on the importance of beans (Phaseolus vulgaris) and prior seed dissemination efforts. A total of seven districts were purposively selected which together represent 59% of the total bean area in Zambia. Data were collected from a sample of about 400 farmers across 67 villages (sample size mostly determined based on the available budget). The survey was implemented between August-September DNA fingerprinting methodology used to establish the benchmark involved, first establishing a reference library of DNA fingerprints, and then collecting samples (plant tissues or seeds) during the farm surveys and genotyping them using the same or a subset of markers used to establish the reference library. Towards this goal, 13 accessions specific to Zambia (including 11 released varieties and two landrace Kabulengeti market classes) and 723 accessions from the East/Southern Africa region (that were genotyped as part of another project by CIAT) were included in the reference library as the background materials to compare the samples collected from farm surveys. The farmer samples were genotyped using 66 assays/markers selected as a sub-set of ~800 SNPs used for the reference library. The 66 SNP assays were made up of 4 groups, each of which has more or less the same power to differentiate released varieties from each other and from the background genotypes. Results About 16% of bean samples collected from farmers were identified as sharing identical DNA fingerprints with four released varieties in Zambia. A majority of these belonged to the variety group kabulengeti, which was an improved landrace released by ZARI in As against this benchmark, effectiveness of different methods was evaluated using four different measures: Aggregate outcome, Accuracy of name and type, type I error (false negative) and type II error (false positive). Results indicate that there is no one method that stands out to be most effective across all measures. In terms of outcome, at the aggregate level, the estimates of data 16

18 points classified as improved varieties by experts either based on seed photos (18%) or seed samples (15%) were closest to the estimates using DNA fingerprinting (16%). The farmer elicitation method A1 (i.e., asking farmers the name of the variety and matching that with the name in the released variety list) gave the lowest estimate of varietal adoption (4%). Identification of varieties based on farmers self-reported assessment of growing an improved variety (13%) was substantially higher than the identification of improved variety based on the self-reported names (4%) and closer to the benchmark of 16% estimated using the DNA fingerprinting method. In Method C, where farmers were shown the seed samples of released varieties and asked to match which one their planted seed resembled, farmers substantially over reported the adoption of varieties (71%). All the methods had low accuracy rate and high type II error rates when the outcome for each data point is compared against the DNA fingerprinting result. Only 9% of data points were correctly matched with the variety name in Method A1 (i.e., farmer elicitation of variety name) and 16% matched by variety type in Method A2. The accuracy rate in expert elicitation methods C and D was 27% and 30%, respectively. In other words, in more than 70% of observations, an improved variety was incorrectly identified by experts as a local variety or by an incorrect name in Methods C and D. this is considered that type II error. In terms of type I error, a local variety was incorrectly identified as an improved variety for 3%, 13%, 67%, 16% and 12% cases under Methods A1, A2, B, C and D, respectively. Implications and need for further research The results of this study have many implications for the methodology of varietal identification used in the past and present. For example, this study has shown that when there is a diversity of names by which farmers call their varieties, the traditional method of farmer elicitation will give an underestimate of adoption of improved varieties by names. Thus the current gold standard of eliciting varietal adoption from farmer surveys may not be an accurate method for measuring varietal turnover and assessments of type II benefits of plant breeding research (i.e., benefits from varietal replacement). On the other hand, showing the seed samples to elicit farmers response on variety specific adoption is prone to overestimate adoption of improved varieties if only improved varieties are 17

19 included. This was a limitation of this study where Method B only included seed samples of improved varieties. More studies are needed to test whether the upward bias of this method can be reduced by including some popular landraces in the visual samples. Results also indicate that farmers and experts are better able to give an aggregate assessment of the adoption of improved varieties as a category than variety specific adoption. However, all the methods evaluated are prone to both type I and type II errors which has implications on the accuracy of any adoption analysis conducted using such data at the farmer level. None of the alternative and non-traditional methods tested emerged as most effective on all measures of effectiveness; although, methods that involved experts opinion and interpretation were generally closer to the benchmark estimates. However, given the time and logistics of implementing these methods, the scalability of some of these methods remains questionable on the grounds of cost-effectiveness and feasibility. This study has shown that molecular markers (SNPs) technology is a useful tool for determining the genetic identity of varieties grown by farmers. It provides a true picture of what is in farmers fields. However, the potential for scaling up this method as part of household surveys will depend on several factors, such as: a) the logistics of collecting, tracking, storing and transporting the samples from farmers fields to a lab facility to get high quality DNA; b) the cost of DNA fingerprinting which includes establishing the reference library, DNA extraction, and genotyping service. In this study the estimated cost per data point was ~$30, which can add substantial costs to the overall cost of doing household surveys in developing countries; and c) capacity to do high volume DNA fingerprinting within the country or easy access to such capacity internationally (i.e., no government restrictions on the shipment of plant tissues or DNA samples to other countries for analysis). Given these challenges, the use of DNA fingerprinting as part of large scale representative household surveys may be long way from becoming routine. Potential ways to reduce the cost and to make the logistics more manageable would be to use DNA fingerprinting as a method of validation on a random sub-sample of households rather than all the households. More studies on different crops and country settings are needed to generate an experience base and derive generalizable conclusions. 18