Improving Methods for Estimating Livestock Production and Productivity Test Stage: Fieldwork Report and Summary Data Analysis

Transcription

1 Improving Methods for Estimating Livestock Production and Productivity Test Stage: Fieldwork Report and Summary Data Analysis November 2016 Working Paper No. 13

2 Global Strategy Working Papers Global Strategy Working Papers present intermediary research outputs (e.g. literature reviews, gap analyses etc.) that contribute to the development of Technical Reports. Technical Reports may contain high-level technical content and consolidate intermediary research products. They are reviewed by the Scientific Advisory Committee (SAC) and by peers prior to publication. As the review process of Technical Reports may take several months, Working Papers are intended to share research results that are in high demand and should be made available at an earlier date and stage. They are reviewed by the Global Office and may undergo additional peer review before or during dedicated expert meetings. The opinions expressed and the arguments employed herein do not necessarily reflect the official views of Global Strategy, but represent the author s view at this intermediate stage. The publication of this document has been authorized by the Global Office. Comments are welcome and may be sent to global.strategy@gsars.org.

3 Improving Methods for Estimating Livestock Production and Productivity Test Stage: Fieldwork Report and Summary Data Analysis

4 Drafted By Michael Coleman Phil Morley Derek Baker Jonathan Moss

5 Table of Contents Executive Summary Introduction Scope of the test phase Goals of the test phase Overview of method Summary of previous work for the project Summary of literature review Summary of GAP analysis Expert meeting Data collection methods and instruments Questionnaires: Existing and Alternative Gold standard data collection Practical aspects of case study country work Sampling Logistics Data handling Tanzania Botswana Indonesia Key results of the analysis and indicators produced Botswana sheep and goats Botswana feed availability Tanzania eggs Tanzania milk Indonesia cattle and goats Indonesia milk Paradata and in-country evaluation of methods employed in the Test phase Data collection Data cleaning and processing Project Resources Usefulness of the alternative questionnaire Usefulness of the gold standard data Relevance of the process to farmers Costs comparisons in evaluating methodological change Introductory comments Cost of design, survey testing and planning for data collection Cost estimates... 76

6 9. Summary and recommendations Influences on herd size and structure, Botswana, sheep and goats Body weight of sheep and goats, Botswana Changes in body weight of sheep and goats, Botswana Feed production and purchase, Botswana Feed usage and pasture degradation, Botswana Egg production,tanzania Influences on egg production,tanzania Milk production, Tanzania Influences on milk production, Tanzania Milk production, Indonesia Weight of cattle and goats, Indonesia References

7 List of Tables Table 6.1. Summary data from existing questionnaire, sheep and goats, Botswana Table 6.2. Summary data from alternative questionnaire, sheep and goats, Botswana Table 6.3. Summary data from gold standard data, goats and sheep, Botswana Table 6.4. Summary of indicators used and data sources, sheep and goats, Botswana Table 6.5. Summary data from existing and alternative questionnaires, feed availability, Botswana Table 6.6. Summary data from gold standard data, feed availability, Botswana Table 6.7. Summary of indicators used and data sources, feed availability, Botswana Table 6.8. Summary data from existing questionnaire, eggs, Tanzania Table 6.9. Summary data from alternative questionnaire, eggs, Tanzania Table Summary data from gold standard data, eggs, Tanzania Table Summary of indicators used and data sources, eggs, Tanzania Table Summary data from existing questionnaire, milk, Tanzania Table Summary data from alternative questionnaire, milk, Tanzania Table Summary data from gold standard data, milk, Tanzania Table Summary of indicators used and data sources, milk, Tanzania Table Summary data from questionnaire, cattle and goats, Indonesia Table Summary data from gold standard data, cattle and goats, Indonesia Table Summary of indicators used and data sources, cattle and goats, Indonesia Table Summary data from questionnaire, milk, Indonesia Table Summary data from gold standard data, milk, Indonesia Table Summary of indicators used and data sources, milk, Indonesia Table 8.1. Assumptions used in cost calculations Table 8.2. Estimates of cost per day of survey work, Existing method Table 8.3. Costs of Alternative methods, by country Table 8.4. Costs of workshop for communal data collection, by country... 81

8 List of Figures Figure 3.1. Basic questions Botswana sheep and goats Figure 3.2. Existing questionnaire Botswana sheep and goats Figure 3.3. Alternative questionnaire Botswana sheep and goats Figure 3.4. Alternative questionnaire Botswana sheep and goats (continued) Figure 3.5. Basic questions Botswana feed Figure 3.6. Existing questionnaire Botswana feed Figure 3.7. Alternative questionnaire Botswana feed Figure 3.8. Basic questions Tanzania eggs Figure 3.9. Existing questionnaire Tanzania eggs Figure Alternative questionnaire Tanzania eggs 28 Figure Basic questions Tanzania milk. 29 Figure Existing questionnaire Tanzania milk. 29 Figure Alternative questionnaire Tanzania milk. 30 Figure Indonesian milk questionnaire 32 Figure Indonesian milk questionnaire (continued).. 33 Figure Indonesian milk questionnaire (continued) Figure Indonesian cattle and goat questionnaire Figure Indonesian cattle and goat questionnaire (continued) Figure Gold standard data collection form Botswana sheep and goats Figure Gold standard data collection form Tanzania egg data collection Figure Gold standard data collection form animal identification elements, Tanzania milk data collection Figure Gold standard data collection form Tanzania milk data collection Figure 4.1. Map of sampling locations, Botswana: Mahalapye, Tshane/Kgalihad, Kweneng/Molopole Figure 4.2. Map of sampling locations, Tanzania: Morogoro and Dodoma Figure 4.3. Map of sampling locations, Indonesia: Lampung Selatan, Malang, and Bima Figure 8.1. Distribution of data collection costs per survey day by country, Existing Method Figure 8.2. Indexed costs of scenarios... 80

9 Executive Summary The test phase of the project Improving Methods for Estimating Livestock Production and Productivity utilised literature review and gap analysis to develop and implement methodological trials in collection of smallholder livestock production and productivity in late In three countries (Botswana, Tanzania, and Indonesia) testing was conducted in a number of locations and carried out for a limited number of farm-level livestock commodities. The findings of the trials are to be applicable across a range of country and commodity contexts. The method centred on a comparison of existing ( E ) data collection methods (constituting mostly questionnaires and data handling in calculation of indicators) with an experimental alternative ( A ). Alongside this test, actual data (Gold Standard or GS ) was also collected in an intensive activity yielding data against which the E and A methods could be gauged for accuracy. E, A and GS data were compiled, and evaluations of the data, the method and the collection process were carried out. Evaluation of methodologies in this report extends to summaries of costs, advantages and lessons learned in the test phase and extensions to utilisation of GS data in formulating proxy measures for A methods. Costs of data collection in each country were compiled and analysed so as to provide a comparison between countries and to project the costs of alternative methods which either followed A methods as tested in the project or new ones associated with the analysis following the test phase. In particular, new datasets were collected for each country which offer opportunities for estimation of relationships between simple and cheaply-available proxy measures and more difficult ones associated with GS measurement. Use of animal girth measurement, as a proxy for liveweight is one example. Total costs of existing smallholder livestock data collection (including training, all manpower, running and incidental costs, but excluding the cost of farmer participation) in the three pilot countries amounts to US$ 2,100 3,300 per day of survey activity. This numeraire was chosen for convenient scalability. The costs of alternative methods were also estimated using the actual costs of additional activities during the test phase. These extended daily costs by 4-50% depending on the extent of additional training, manpower, expertise and transport. Farmer participation in data collection, widely advocated by in- 9

10 country partners during the GAP analysis, was identified as a means of reducing the additional costs of alternative methods while enhancing uptake and use of the data collected. In-country partners evaluated the test phase in terms of the quality of data collected, the efficiency of the collection processes for E and A methods, value for money for each method, and farmers benefits amongst other measures of satisfaction. Overall the alternative methods were evaluated as being clearer and more effective in measuring key variables and indicators, and as being appreciated by farmers. In all cases the A methods are more expensive than the E methods, but despite this it was important to note that in-country partners still generally found A methods to be good value for money in most cases. These conclusions are presented in chapter 9 in tabular form. 10

11 1 Introduction The Improving Methods for Estimating Livestock Production and Productivity project seeks methods to improve the quality of livestock data. Supporting the Global Strategy on Agricultural and Rural Statistics, the project seeks opportunities to improve livestock data collection methods across a range of commodities. Concentrating on production-level livestock, improvements are sought specifically within the measurement of production and productivity at the household level, although the project also addresses the definition of target variables, methods of collection, procedures for benchmarking, and institutional organisation surrounding livestock data collection SCOPE OF THE TEST PHASE The test phase follows on from an initial literature and gap analysis held in early 2015 within the three countries selected for testing of livestock data collection approaches: Botswana, Tanzania, and Indonesia. In each country testing was conducted in a number of locations and carried out for a limited number of farm commodities (two in Botswana and Tanzania and three in Indonesia). In practice, however, the collection methods used would be applicable, with minor modification, to other agricultural commodities. The limited scope of pilot activities reflected the resources available to the project, and the need to successfully demonstrate methods and lead by example in their future wider modification and implementation GOALS OF THE TEST PHASE The goal of the test phase was to determine the usefulness and feasibility of approaches to livestock data gathering through in-field data collection from a limited sample of household livestock holders, residing in selected locations in each of the case study countries. 11

12 1.3. OVERVIEW OF METHOD The field work involved a mixture of questionnaires to estimate, through respondent recall, the production and productivity of various agricultural commodities. In a number of cases different forms of questionnaire were used to test the effectiveness of currently used survey approaches against innovative approaches or questions. In addition to survey results, regular observation was conducted to record and measure commodity characteristics and productivity over a defined period. Data measurement and recording of production quantities represented a gold standard approach to accurate collation of production and productivity data, and provided an opportunity to test the feasibility of more extensive approaches of this nature. Measurement approaches used included observation (e.g. of egg production) through to measurement (e.g. of milk production or livestock weights and physical characteristics). The project also tested approaches to recording and storing data using handheld devices as a comparison to approaches using paper survey or data entry forms, and collation of such data into spreadsheet or database files. As such, the project tested the potential for handheld technology to make livestock data collection more efficient, accurate and secure. Further details on the field work methods used in each case study during the test phase are provided in Sections 3 to 5. 12

13 Summary of Previous Work for the Project 2.1. SUMMARY OF LITERATURE REVIEW 2 A review of the literature was completed in February 2015, to inform the subsequent gap analysis and fieldwork phases of the project. In summary, the literature suggested that timely and accurate data is critically important for the development of food security programs, agricultural development, poverty reduction policies, investment strategies and natural disaster responses. A global assessment of agricultural data found that there has been a decline in both the quantity and quality of agricultural statistics World Bank, FAO and UN. (2010). While it would not be practical, and certainly not economically optimal, for countries to obtain complete information on agricultural systems from surveys, this global assessment found that many developing countries did not have the capacity to collect and disseminate even the rudimentary set of agricultural data required to monitor national trends or to inform the international development discussions. It was noted in this assessment that it is imperative for national agricultural systems to move towards a systematic collection of data to allow reliable statistics to be reported. Given the often heterogeneous nature of agricultural enterprises, there are many recognized difficulties surrounding the collection of statistics for this industry. Collection of data from the livestock section of this industry has additional complexities such as dynamic herd structures and sometimes non-sedentary populations. However, applying an appropriate framework to collect relevant and high priority information while avoiding multiple or non-standardized collection of data by different government agencies has been recognised as an effective method to assist in the design and implementation of policies to promote sustainable livestock sectors (Pica-Ciamarra et al., 2014). 13

14 The most fundamental item of any statistical data collection of a country s livestock is the current number of livestock (FAO, 2005; Nsiima et al., 2013). The World Bank, FAO and UN. (2010) outlined core species for which key indicators should be collected, with the five core livestock species of Cattle, Sheep, Pigs, Goats and Poultry accounting for over 99 per cent of the meat, milk and eggs produced globally (Pica-Ciamarra et al., 2014). Although various factors influence satisfaction with the information collected for these indicators, a systematic approach to the information and its collection is clearly lacking. While there are numerous problems and difficulties in the collection of livestock data, key problems include: capture of data from transhumant populations (of people and animals); inability to identify individual animals; and logistic and technological problems of data recording and transmission within reasonable time frames. These topics refer primarily to national-level data, but a supporting set of difficulties are identified in reference to farm- and enterprise-level data and efforts by various government entities to establish measures of performance and associated benchmarks for use in extension and related work. These include, but are not limited to: reliance on faulty recall; choice of survey respondent within households; lack of training of enumerators; lack of standardisation of methods and formats; and outdated technical coefficients. Since livestock data collection is usually a resource-poor activity, eliminating the widely-reported duplication of collection as part of this framework should be a priority. Further considerations to ensure quality data are collected include timeliness and punctuality, completeness, comparability and coherence, accuracy, relevance and reliability. An ideal statistical system will also provide the framework to store and aggregate livestock data collected, and to disseminate results and indicators on a timely basis SUMMARY OF GAP ANALYSIS Following the literature review, a gap analysis was completed in early 2015 to identify the measures and methods for livestock data collection and analysis for each of the three case study countries. Gap analysis entails the quantification and comparison of desired and existing states. The gap analysis study attempts this for three case study countries (Botswana, Tanzania and Indonesia). It measured functionality gaps by way of direct questioning of stakeholders with regard to the systems delivering the data; canvassed stakeholders ideas about ways in which livestock data may be 14

15 improved so as to fill these gaps; and provided, where possible, commentary regarding capability gaps and likely performance gaps with regard to new and improved livestock data systems. Informed by the project s literature review, the emphasis of the gap analysis was on systematic issues of sampling and collection and approaches to data integration, the quantity of data being collected, and its quality. Commentary and assessment surrounding quality of data followed the Global Strategy s proscribed criteria of relevance, accuracy and reliability, timeliness and punctuality, coherence and comparability, and accessibility and clarity. The GAP analysis study employed a questionnaire which was administered in a workshop format. Workshops were held either in livestock administrative regions or in capital cities, and spanned a range of stakeholder interests, primarily government providers and users of livestock data. After identifying up to six pieces of information that they need in their occupation, participants were asked to provide information on data availability, familiarity, relevance, accessibility, accuracy, timeliness, and other quality measures. Options were also provided for stakeholders to make recommendations on how to improve the quality, collection methods and usefulness of this data. Some 14 workshops were held in the three countries, at which 171 stakeholders provided completed questionnaires. The GAP analysis produced a consistent call for improved logistics and communications for data collection for smallholder livestock systems. In the case of Indonesia, indicators for which no data at all is collected from smallholder livestock systems provided some obvious targets. A significant, and consistent, call from pilot countries was for farmers to be much more involved in data collection and to assume some responsibility for it. This was motivated both as part of productivity measurement per se, and also as part of a broader drive to encourage and improve on-farm record-keeping. Training for farmers in data collection activities was identified as a key need. These results were reinforced by widespread statements that (a) numbers of animals, feed availability and feed intake were key data needs and that (b) even where such information was collected, it was unavailable or unreliable. It is notable that these variables offer scope for collection by farmers or by some form of farmer participation. 15

16 2.3. EXPERT MEETING A meeting of industry experts and representatives from departments of agriculture and statistics from each test country was held in July As a result of the literature review, gap analysis and stakeholder consultation during this meeting, pilot countries target products for methodological improvements were identified as follows: Tanzania: milk production, egg production. Botswana: feed demand, availability and surplus, sheep and goat numbers, sales channels, weights and growth rates. Indonesia: Beef growth rates, sheep and goats numbers, sales channels, weights and growth rates, milk production. The expert meeting also endorsed the rationalisation of Existing and Alternative data collection methods, and the collection of Gold Standard data. Test protocols were then developed for each of the three case study countries with reference to data collection activities already being undertaken in each country. Further details from the protocols for each country are provided in subsequent chapters. 16

17 3 Data Collection Methods and Instruments After examination of the current data collection methods in each of the test countries, project staff developed a set of methods and instruments appropriate to the country. These were discussed with in-country partners and with FAO prior to their use in the TEST phase. In Botswana and Tanzania, two distinct approaches were used with a series of basic survey questions that were common to Existing and Alternative questionnaires, followed by a separate list of questions which differed across Existing and Alternative questionnaires. Most such differences were introduced to better utilise farmer understanding and recall, and to improve consistency and intuitive appeal of questions. They also shifted the targeting of production-related information so as to allow multiple approaches by which key indicators could be calculated and disaggregated. In addition, some non-survey methods were introduced in both countries to reflect the GAP analysis emphasis on (a) the need for non-recall methods, (b) farmer involvement, (c) access to information on feeds and growth rates which had not been attempted before. In all countries, new approaches to measurement of animal numbers and herd/flock dynamics were assigned a high priority. Moreover, relevant variables were addressed in Alternative questionnaires in such a way as to better relate to productivity measures (e.g. milk production, calving interval, and calf suckling all related to numbers of cows being milked; access to common or fenced pasture, seasonality and animal numbers in Botswana). Questionnaire design was substantially informed by the desire for consistency in these matters. The first version was based the current methods and information gained by government authorities, and was designed to create a benchmark on existing indicators. The second questionnaire asked a separate set of questions and 17

18 provided a test of alternative questioning designed to examine other indicators and variables. A gold standard system was established for comparison to survey responses where local enumerators were employed to test and physically measure specific indicators. Respondents were randomly allocated to complete either an existing or alternative survey, and gold standard measurements were taken for all participants. Existing and alternative questionnaires were completed before gold standard data collection commenced, so that the gold standard measurements did not bias the questionnaire response. In Indonesia there is no existing data set based on smallholder livestock production and productivity. Milk production in Indonesia is carried out only within a limited geographical area (Malang) and current surveys consider only the largest producers, who would not be considered smallholders. In this context, a single survey (effectively the Alternative) was therefore used to record information on smallholder production, with actual milk production per animal supervised and measured by enumerators for a Gold Standard comparison. Similarly in examining cattle and goat productivity in three test locations within Indonesia, a single survey (again, the Alternative) was used and a Gold Standard employed enumerators weighing the cattle using a set of transportable scales. In all countries, group scoring and assessment were trialled under Alternative procedures to allow an assessment of bias in existing recall-based individual surveys. In Indonesia and Botswana, animal liveweight and growth rate was measured under the Gold Standard procedure, and accompanied by girth measurement, and body condition score which offer (non-recall or non-survey) alternative measurement systems cheaper and less infrastructure-dependent than weighing. These also contribute to a possible set of farmer roles in livestock data collection. In Botswana, Existing questionnaires on feed and pasture use were tested against new recall-based questions, and non-survey and non-recall questions requiring subjective assessment of pasture condition. These were tested against a Gold Standard that used modern agronomic pasture assessment procedures. Again, these approaches favour farmer involvement, and consistency and flexibility in use for calculation of key indicators. In Tanzania, Alternative questionnaires addressed animal breeds and management systems (particularly egg clutch management and calves consumption of milk) so as to better disaggregate datasets and provide a practical backdrop to farmer involvement that is consistent with extension work. Also in Tanzania, communal data collection was completed to obtain a collective response on monthly production of milk and eggs, using a 18

19 proportional piling technique. Communal data collection is well suited to the establishment of certain qualifying variables, particularly those used in calculating indicators. These may include seasonal timing of events; market channels used; presence of diseases; and transhumant practices. Combinations of communal types of data collection and the completion of individual questionnaires have also been tried (e.g. Baker et al., 2015). The appendices document to this report suggests that the communal method is useful in some cases, showing some correlation with alternative questions addressing these issues QUESTIONNAIRES: EXISTING AND ALTERNATIVE Details on existing and alternative questionnaire forms used are included below BOTSWANA SHEEP AND GOAT QUESTIONNAIRES Basic or common questions were asked in both surveys to enable a household identifier and recognise the position in the household of the respondent. A sample is shown in Figure

20 Figure 3.1. Basic questions Botswana sheep and goats 20

21 The existing (E) survey was conducted by household members of half of the total participant sample and is shown below in Figure 3.2. Figure 3.2. Existing questionnaire Botswana sheep and goats 21

22 The alternative (A) survey (Figures 3.3 and 3.4) was completed with the other half of the householder sample, and asked for further details on seasonal production and animal weight. Figure 3.3. Alternative questionnaire Botswana sheep and goats SHEEP AND GOATS: ALTERNATIVE QUESTIONNAIRE ADMINISTRATIVE DISTRICT CENSUS DISTRICT HOUSEHOLD CODE ENUMERATION AREA cell phone number BLOCK NUMBER ENUMERATOR DWELLING NUMBER HOUSEHOLD NUMBER Date FULL NAME OF HOLDER SEX OF HOLDER NAME OF RESPONDENT What is the age of the holder? years What is the sex of the holder? MALE/FEMALE How many years of school eductaoin does the holder have? years Is the holder a member of a small stock association or any similar group or organisation>? YES/NO How often does the holder meet with extension officers? never about 1 X/year More than 1X/year How far is the holding from a town where livestock can be marketed? km What is the monthly income from non-farm sources? up to 500 PULA/month PULA/month OVER 1000 PULA family non-family How many herdsmen are employed? 1. How many goats and sheep did you have NOW? Other questions will refer to the agricultural season 1 Oct Sept SHEEP Ram Castrated (uncastrate males 1 d) 1 year year and and over over Females 1 year and over Males under 1 year Females under 1 year Total GOATS Buck Castrated (uncastrate males 1 Females 1 d) 1 year year and year and and over over over Males under 1 year Females under 1 year Total NUMBERS NUMBERS 2. What breed are these animals? CODE Ram Castrated (uncastrate males 1 d) 1 year year and and over over Females 1 Males year and under 1 over year Females under 1 year Total CODE Buck Castrated Females 1 Males (uncastrate males 1 year and under 1 d) 1 year year and over year and over over Females under 1 Total year Don't know the breed Don't know the breed 1. Dorper; 2 Karakul; 3. Tswana; 4. Other (Specify) 1. Tswana; 2. Boer Goat; 3. Crosses; 4. Kalahari; 5. Savannah; 6. Sanaan; 7. Other (specify); 8. Boer X Tswana 3. How many animals were born in each of these seasons of the last year? SHEEP Males under 1 year Females under 1 year Is there a main lambing date? GOATS Males under 1 year Females under 1 year Is there a main kidding date? RAINY SEASON October March Yes RAINY SEASON October March Yes DRY SEASON April Sept No DRY SEASON April Sept No 4. How many animals were purchased or obtained in each season of the last year? SHEEP Ram Castrated (uncastrate males 1 d) 1 year year and and over over Females 1 year and over Males under 1 year Females under 1 year Total GOATS Buck Castrated Females 1 (uncastrate males 1 year and d) 1 year year and over and over over Males under 1 year Females under 1 year Total RAINY SEASON October March RAINY SEASON October March DRY SEASON April Sept DRY SEASON April Sept Purchased RAINY SEASON October March Purchased RAINY SEASON October March From trader From trader From other farmer From other farmer From other source From other source Purchased DRY SEASON April Sept Purchased DRY SEASON April Sept From trader From trader From other farmer From other farmer From other source From other source 5. How many animals were sold or exchanged in each season of the last year? Ram Castrated Females 1 Males Females Buck Castrated Females 1 Males Females 22

23 Figure 3.4. Alternative questionnaire Botswana sheep and goats (continued) 5. How many animals were sold or exchanged in each season of the last year? SHEEP Ram Castrated (uncastrate males 1 d) 1 year year and and over over Females 1 year and over Males under 1 year Females under 1 year Total GOATS Buck Castrated Females 1 (uncastrate males 1 year and d) 1 year year and over and over over Males under 1 year Females under 1 year Total RAINY SEASON October March RAINY SEASON October March DRY SEASON April Sept DRY SEASON April Sept Sold RAINY SEASON October March Sold RAINY SEASON October March To trader To trader To other farmers To other farmers To retailer or butcher To retailer or butcher To slaughterhouse To slaughterhouse To other buyers To other buyers Sold DRY SEASON April Sept Sold DRY SEASON April Sept To trader To trader To other farmers To other farmers To retailer or butcher To retailer or butcher To slaughterhouse To slaughterhouse To other buyers To other buyers 6. How many animals died in each season of the last year? SHEEP Ram Castrated (uncastrate males 1 d) 1 year year and and over over Females 1 year and over Males under 1 year Females under 1 year Total GOATS Buck Castrated Females 1 (uncastrate males 1 year and d) 1 year year and over and over over Males under 1 year Females under 1 year Total RAINY SEASON October March RAINY SEASON October March DRY SEASON April Sept DRY SEASON April Sept Causes of death RAINY SEASON October March Causes of death RAINY SEASON October March 1. Diseases 1. Diseases 2. Parasites 2. Parasites 3. Accidents 3. Accidents 4. Predators 4. Predators 5. Drought 5. Drought 6. Other 6. Other Causes of death DRY SEASON April Sept Causes of death DRY SEASON April Sept 1. Diseases 1. Diseases 2. Parasites 2. Parasites 3. Accidents 3. Accidents 4. Predators 4. Predators 5. Drought 5. Drought 6. Other 6. Other 7. How many animals were given away in each season of the last year? SHEEP Ram Castrated (uncastrate males 1 d) 1 year year and and over over Females 1 year and over Males under 1 year Females under 1 year Total GOATS Buck Castrated Females 1 (uncastrate males 1 year and d) 1 year year and over and over over Males under 1 year Females under 1 year Total RAINY SEASON October March RAINY SEASON October March DRY SEASON April Sept DRY SEASON April Sept 8. How many animals were slaughtered for consumption in each season of the last year? SHEEP Ram Castrated (uncastrate males 1 d) 1 year year and and over over Females 1 year and over Males under 1 year Females under 1 year Total GOATS Buck Castrated Females 1 (uncastrate males 1 year and d) 1 year year and over and over over Males under 1 year Females under 1 year Total RAINY SEASON October March RAINY SEASON October March DRY SEASON April Sept DRY SEASON April Sept 9. What is the average weight of your animals at each of the following ages? 3 MONTHS 6 MONTHS 12 MONTHS SHEEP kg kg kg GOATS kg kg kg 23

24 BOTSWANA FEED QUESTIONNAIRES The same set of basic or common household questions were asked for both the existing and alternative feed survey respondents (Figure 3.5). Figure 3.5. Basic questions Botswana feed. 24

25 All householders were asked to complete either the existing (E) or alternative (A) surveys about feed production with sampling chosen randomly. The existing questionnaire is shown in Figure 3.6 and the alternative in Figure 3.7. Figure 3.6. Existing questionnaire Botswana feed. FEEDS: EXISTING QUESTIONNAIRE ADMINISTRATIVE DISTRICT CENSUS DISTRICT ENUMERATION AREA BLOCK NUMBER DWELLING NUMBER HOUSEHOLD NUMBER HOUSEHOLD CODE cell phone number ENUMERATOR Date FULL NAME OF HOLDER SEX OF HOLDER NAME OF RESPONDENT FEEDS Fed to Cattle (tick) Sheep (tick) Goats (tick) Quantities purchased (kg) Lucerne Nitrogen/protein feeds Drought pellets Ram, Lamb and Ewe pellets Moroko Salt Dicalcium Phosphate Lablab Molasses (powder) Molasses (liquid) Molasses (meal) Stover (Lotlhaka) Grasses (fodder) Grains (barley) Mineral block (e.g. rumevite) Hay Feed grade urea Other 4. What area was sown to each of the feed crops in the last agricultural season 1 Oct Sept 2105? Crops grown: Area (ha) 25

26 The alternative (A) questionnaire asked respondents to consider demand for and supply of feed. Figure 3.7. Alternative questionnaire Botswana feed. 2. Which feeds are purchased, and which animals are they fed to? For how many days each year? AREA (ha) DAYS FED TO DAYS FED TO DAYS FED TO COWS? YOUNG CATTLE? CALVES? DAYS FED TO DAYS FED TO DAYS FED TO DAYS FED TO DAYS FED TO YOUNG YOUNG EWES? LAMBS? DOES SHEEP? GOATS? DAYS FED TO KIDS SEASON USED (RAINY OR DRY) Lucerne Nitrogen/protein feeds Drought pellets Ram, Lamb and Ewe pellets Moroko Salt Dicalcium Phosphate Lablab Molasses (powder) Molasses (liquid) Molasses (meal) Stover (Lotlhaka) Grasses (fodder) Grains (barley) Mineral block (e.g. rumevite) Hay Feed grade urea Other 3. How much grazing is available to you? 4. What is the extent of overgrazing in these areas? APPROX TOTAL AREA (ha) DAYS USED DAYS USED DAYS USED FOR DAYS USED FOR FOR OTHER FOR CATTLE SHEEP GOATS ANIMALS? CIRCLE THE RATING OF OVER-GRAZING Fenced grazing Fenced grazing NONE SOME MODERATE SEVERE EXTREME Communal grazing Communal grazing NONE SOME MODERATE SEVERE EXTREME Rented grazing Rented grazing NONE SOME MODERATE SEVERE EXTREME Roadsides and other public areas Roadsides and other public areas NONE SOME MODERATE SEVERE EXTREME Other Other NONE SOME MODERATE SEVERE EXTREME FEEDS: ALTERNATIVE QUESTIONNAIRE ADMINISTRATIVE DISTRICT CENSUS DISTRICT HOUSEHOLD CODE ENUMERATION AREA cell phone number BLOCK NUMBER ENUMERATOR DWELLING NUMBER Date WET SEASON OCT-MARCH 182 days DRY SEASON APRIL-SEPT 183 days HOUSEHOLD NUMBER 365 FULL NAME OF HOLDER SEX OF HOLDER NAME OF RESPONDENT 1. What are the numbers of CATTLE, and OTHER ANIMALS held NOW? (A different questionnaire has recorded the numbers of sheep and goats) Female CATTLE Bulls Oxen Cows Heifers Tollies Male calves Total OTHER ANIMALS Horses calves Donkeys and mules Pigs Poultry Others Others NUMBER NUMBER 2. What areas of crop are grown, and are they or their residues fed to animals? If so, for how many days each year? AREA (ha) NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF NUMBER OF DAYS NUMBER OF DAYS NUMBER OF DAYS DAYS stubble DAYS byproducts fed fed to grazed by fed to products fed to grazed by GOATS? products fed fed to OTHER DAYS grain DAYS stubble DAYS stover DAYS by- NUMBER OF DAYS stubble DAYS by- DAYS stover DAYS grain fed DAYS stover DAYS grain fed DAYS stover grain fed to OTHER stubble grazed by by-products fed to grazed by to CATTLE? fed to CATTLE? to GOATS? fed to GOATS? ANIMALS? OTHER ANIMALS? OTHER ANIMALS? CATTLE? to CATTLE? SHEEP? SHEEP? SHEEP? SHEEP? to GOATS? ANIMALS? Maize Other grains Oilseeds Beans Groundnuts Lablab Lucerne Other crops 26

27 TANZANIA EGG QUESTIONNAIRES In Tanzania the common, basic questions (Figure 3.8) were drawn from the household identification section of the National Panel Survey. Figure 3.8. Basic questions Tanzania eggs. EXISTING QUESTIONNAIRE REGION DISTRICT WARD VILLAGE ENUMERATION AREA KITONGOJI OR MTAA NAME HOUSEHOLD ID (FROM LIST) NAME OF HOUSEHOLD HEAD CELL PHONE NUMBER The existing (E) survey (Figure 3.9) was conducted with half the total sample of householders for this project, and was drawn from the Other Products section of the National Panel Survey. The questionnaire asked respondents to recall the number of months eggs were produced, as well as the amount of production per month. Figure 3.9. Existing questionnaire Tanzania eggs. 1 For how many months were eggs produced in the last year? E1.1 2 How many eggs per month were produced in each of those months? YES/NO No. eggs Sep E2.1 E2.2 Oct E2.3 E2.4 Nov E2.5 E2.6 Dec E2.7 E2.8 Jan E2.9 E2.10 Feb E2.11 E2.12 Mar E2.13 E2.14 Apr E2.15 E2.16 May E2.17 E2.18 Jun E2.19 E2.20 Jul E2.21 E2.22 Aug E2.23 E

28 The alternative (A) survey was conducted with the other half of the total sample of householders. The questionnaire asked respondents to recall the number of active hens in a year, the number of clutches per year per hen, the number of eggs per clutch, and the gap between clutches. A sample of the alternative survey is shown below in Figure Figure Alternative questionnaire Tanzania eggs. ALTERNATIVE QUESTIONNAIRE REGION DISTRICT WARD VILLAGE ENUMERATION AREA KITONGOJI OR MTAA NAME HOUSEHOLD ID (FROM LIST) NAME OF HOUSEHOLD HEAD CELL PHONE NUMBER 1 What strain of hen do you keep? A1.1 strain 2 How many hens laid eggs in the past year? A2.1 hens 3 How many clutches did each hen lay, on average? A3.1 clutches 4 How many eggs per clutch were laid on average? A4.1 eggs 5 What is the usual period from the beginning of a clutch, to the beginning of the next clutch? A5.1 days 6 In your practice, do you transfer eggs from laying hens to brooding hens? A6.1 YES/NO 7 In your practice, do you remove eggs for sale, consumption or to give as gifts? A7.1 YES/NO TANZANIA MILK QUESTIONNAIRES As with the egg questionnaires, basic questions were drawn from the household identification section of the National Panel Survey (Figure 3.11). 28

29 Figure Basic questions Tanzania milk EXISTING QUESTIONNAIRE REGION DISTRICT WARD VILLAGE ENUMERATION AREA KITONGOJI OR MTAA NAME HOUSEHOLD ID (FROM LIST) NAME OF HOUSEHOLD HEAD CELL PHONE NUMBER The existing (E) questionnaire (Figure 3.12) again asked half the householder sample to recall the number of months of milk production, and the amount of production per month. Figure Existing questionnaire Tanzania milk 1 How many cows were milked in the last 12 months? E1.1 cows 2 How many months were cows milked for, on average? E2.1 months 3 What was the average milk production per cow per day? E3.1 l/day 4 In which month was the highest milk production per cow per day? E4.1 l/day in month E4.2 5 In which month was the lowest milk production per cow per day? E5.1 l/day in month E5.2 6 What was the average calf suckling practice? E6.1 NONE LIMITED UNLIMITED 7 What was the average household use of milk? E7.1 l/day The alternative (A) questionnaire (Figure 3.13), conducted with the other half of the householder sample, asked for more details on milk production over lactation periods, time between calvings, calf access practices, and milk usage. 29

30 Figure Alternative questionnaire Tanzania milk ALTERNATIVE QUESTIONNAIRE REGION DISTRICT WARD VILLAGE ENUMERATION AREA KITONGOJI OR MTAA NAME HOUSEHOLD ID (FROM LIST) NAME OF HOUSEHOLD HEAD CELL PHONE NUMBER 1 What is the total number of cows milked in the last 12 months? indigenous cows A1.1 cows improved cows A1.2 cows indigenous cows improved cows 2 What is the average milk production per cow day over whole lactation? A2.1 A2.2 l/cow/day 3 What is the average number of months milked per cow? A3.1 A3.2 months improved indigenous cows cows 4 What is the average milk production per day in these months of lactation? First month A4.1 A4.2 l/day/cow Second month A4.3 A4.4 l/day/cow Third month A4.5 A4.6 l/day/cow After third month A4.7 A4.8 l/day/cow indigenous cows improved cows 5 What is the average number of months between calvings? A5.1 A5.2 months 6 What is your practice for calf access? ANSWER ONE Limited to a few minutes per milking A6.1 YES/NO More than a few minutes but not unlimited YES/NO Unlimited access YES/NO 7 What was the average household use of milk? A7.1 l/day 2 udder quarters left A6.2 YES/NO 1 udder quarters left YES/NO Other YES/NO COMMUNAL QUESTION proportional piling Month of highest milk production for the community Month of lowest milk production for the community Month of greatest pasture availability Month of least pasture availability INDONESIA MILK QUESTIONNAIRES The goals of the Indonesian milk survey were: To establish a reliable benchmark of milk production per cow for Indonesian smallholder farmers. To improve the quality of information on milk production by smallholders. To test the reliability of farmer recall for average milk production per cow. To test the reliability of lactation curves for milk production in Indonesia. At present, there are no existing measures of milk production of small scale farmers in Indonesia. The livestock census asks for household milk production for the year, and the Dairy Cattle Business Report only considers businesses with more than 10 lactacting cows. 30

31 An alternate survey was conducted of a sample of small-scale dairy farmers, shown in Figures 3.14 and Each farmer was asked to estimate the average milk production per animal per day over the past year. At each subsequent visit by the enumerator, follow up questions were asked to record any changes that may have occurred in the time between visits. These questions are shown in Figure Consultation with staff from the Indonesian Ministry of Agriculture and the Central Bureau of Statistics survey led to a variety of questions being asked in the alternative milk questionnaire, relating to market channel information for milk, milk products and cattle. 31

32 Figure Indonesian milk questionnaire. 32

33 Figure Indonesian milk questionnaire (continued). 33

34 Figure Indonesian milk questionnaire (continued). 34

35 INDONESIA CATTLE AND GOAT SURVEY The primary goal of this survey was to examine the growth of cattle and goats on smallholder farms. At present, the weight of animals is recorded sporadically, primarily at slaughterhouses. Growth rates based on weight are not recorded. Weighing animals on each visit also provided an opportunity to test the reliability of girth measurement as a surrogate for weight. As little information exists regarding market channels for smallholder farmers, and no information exists regarding the rates of private animal slaughter, additional goals included an examination of market channel information, slaughter weight and rates of unofficial slaughter. The opportunity presented by this survey and related measurement activities was also used to test the use of handheld devices as survey instruments. As per the milk data survey, a set of separate of questions were asked to record herd dynamics and other changes that may have occurred between enumerator visits. A sample of the questionnaire with possible responses is shown in Figures 3.17 and

36 Figure Indonesian cattle and goat questionnaire. 36

37 Figure Indonesian cattle and goat questionnaire (continued). 37

38 3.2. GOLD STANDARD DATA COLLECTION BOTSWANA SHEEP AND GOAT DATA COLLECTION The gold standard data collection was completed with all households included in the sample and involved weighing, taking measurements, and estimating body condition score. Information was collected for each animal as shown in Figure Figure Gold standard data collection form Botswana sheep and goats. For All Goats and sheep less than 1 year of age ID Tag Number or spray raddle Age Breed Gender ANIMAL CODE 1st Visit 2nd Visit Date Weight kg kg Girth measure (cm) cm at brisket 4 weeks cm at brisket Shoulder height (cm) cm at withers cm at withers Body Condition from scores 1-5 from scores Botswana feed data collection Gold standard data collection was conducted by staff of the Botswana Ministry of Agriculture. This included: (1) Calculation of stocking rate based on local extension office s records of: Communally grazed areas (in ha) Stock held at each site (in hd) (2) Calculation of stock feed intake based on: Physiological requirements of stock Stock held at each site (in hd) The Ministry of Agriculture s staff also conducted transect-sampling of communally grazed pastures to assess the extent, distribution and severity of overgrazing. Transect-sampling was completed in three regions (Central, Kgalagadi and Kweneng). The goal of the sampling was to assess the variety of herbaceous species present. Herbaceous frequency data were collected along 100 metre transects using a wheelpoint method. At the end of each transect, 38

39 nested quadrats were used to clip herbaceous biomass and woody species count (1 x 1 metre and 10 x 10 metre quadrats respectively) TANZANIA EGG COLLECTION Gold standard data collection was completed with all householders included in the sample for the project (Figure 3.20). The goal was to collect data on the number of hens active over the data collection period, the number of clutches, the number of eggs per clutch, and the time between clutches. 39

40 Figure Gold standard data collection form Tanzania egg data collection. 1 Number of adult hens at farm DATE 1 DATE 2 2 Number of hens actively laying For all clutches on farm 3 For each clutch observed HEN TAG Colour HEN TAG Number STRAIN DATE: HOUSEHOLD CODE: Hen 1: Tag #/Colour: Laid: Moved to Brooding Hen: Removed for other reason: Notes: Hen 2: Tag #/Colour: Laid: Moved to Brooding Hen: Removed for other reason: Notes: Hen 3: Tag #/Colour: Laid: Moved to Brooding Hen: Removed for other reason: Notes: Hen 4: Tag #/Colour: Laid: Moved to Brooding Hen: Removed for other reason: Notes: TANZANIA MILK DATA COLLECTION Gold standard data collection was completed with all householders in the sample. Amount of milk produced in morning and evening was recorded on assigned data cards during the data collection period for specific cows within each householder s herd. Figures 3.21 and 3.22 show samples of information recording instruments. 40

41 Figure Gold standard data collection form animal identification elements, Tanzania milk data collection FOR EACH ANIMAL ID Tag Number, or name Age years Breed Date of calving Date of previous calving Pregnant now? YES/NO if YES, when is the next calving? Figure Gold standard data collection form Tanzania milk data collection DATE: HOUSEHOLD CODE: Cow 1: (Name/Tag/Raddle Colour) Milk am Calf access (am) Milk pm Calf access (pm) Total Milk litres (None/Limited/Unlimited) litres (None/Limited/Unlimited) litres Cow 2: (Name/Tag/Raddle Colour) Milk am Calf access (am) Milk pm Calf access (pm) Total Milk litres (None/Limited/Unlimited) litres (None/Limited/Unlimited) litres 41

42 4 Practical Aspects of Case Study Country Work 4.1. SAMPLING BOTSWANA SHEEP AND GOATS Criteria for the sheep and goats study included locations where sheep and goats are commonly kept by households and good access from the town to the countryside. In consultation with Botswana Statistics the areas of Mahalapye, Tshane/Kgalihad, and Kwenengw to respond./molopole were selected as suitable test locations. Figure 4.1. Map of sampling locations, Botswana: Mahalapye, Tshane/Kgalihad, Kweneng/Molopole. 42

43 Within each district, the following procedures were used to select the sample: 1. Randomly select households from lists held at agricultural extension offices in the districts 2. Select 40 households at each location for a total of 120 households. 3. Where a household did NOT offer a respondent, select more households as needed by approaching immediate neighbours 4. Questionnaires administered at extension offices as households were brought in for the day. 5. Animal weighing to take place on 2-3 occasions at households, some 3-4 weeks apart. For all selected households G (Gold Standard) information including animal count numbers and weights would be collected and at each location the 40 households would be randomly split with each half being asked either: E: existing questions on animal numbers and dynamics or A: alternative questions animal numbers and dynamics by season which included households being asked to estimate animal weights at the first visit, prior to weighing. This method allowed for the following comparisons to be facilitated by the data collection: E vs G (same households, sample of 40) A vs G (same households, sample of 40) E vs A (different households, samples of 40 vs 40) Using aggregated data: as above for paired samples of 120 households Effects LARGE vs SMALL vs ALL HOUSEHOLDS FEED The selection criteria for locations for the feed study included those with a predominance of communal grazing as well as good access from the town to the countryside. In consultation with Botswana Statistics the same locations were chosen as for the sheep and goats study. 43

44 The following sampling procedures were followed for the feed study: 1. The same households as for sheep and goats households. 2. GS sampling to address the same communal grazing zones as the two questionnaire-based studies. At each location the sample was similarly split in to households to be given either E: existing questions on feeds or A: alternative questions on animal numbers by season as well as gold Standard for all participants. The following comparisons were possible from the data collected: A vs Min Agr physiological requirements for animal intake A vs Min Agr assessment of overgrazing A vs Min Agr on stocking rate E vs A (different households, samples of 40 vs 40) on feed used by households TANZANIA Sampling for the project was completed in consultation with the Tanzanian Bureau of Statistics. The sampling criteria included: High production areas for eggs and milk Many households producing eggs and milk Good access to Dar Es Salaam Good access from the town to the countryside Compact sampling areas 44

45 Figure 4.2. Map of sampling locations, Tanzania: Morogoro and Dodoma. Based on these criteria, two locations were chosen: Morogoro and Dodoma (Figure 4.2). In each district, approximately equal numbers of households were selected for the egg and milk commodity survey work. Stratified sampling was not used for egg production. Where a household did not produce eggs and/or milk, replacement households for the sample were selected by approaching immediate neighbours. In the case of both commodities, approximately half the sample were asked to complete the existing (E) questionnaire, while the other half were asked to complete the alternative (A) questionnaire. All households in the sample for each commodity participated in the gold standard data collection (recording of daily egg and milk production with each householder). This was designed to facilitate the following comparisons: E vs G (same households, sample of all E respondents) A vs G (same households, sample of all A respondents) E vs A (different households, comparative sample of E and E respondents) 45

46 INDONESIA MILK Nearly all milk production in Indonesia is conducted within the East Java region, and as such this area was the focus of the milk survey work. In particular, higher concentrations of dairy farms, dairy cattle and small scale dairy farmers occur within the area surrounding the city of Malang. From the data gathered, comparisons were to be made between: Smallholder milk production per animal vs Large Dairy from Dairy Cattle Business Report. Accuracy of famer recall on average daily milk production (survey vs measured). Reliability of lactation curves for Indonesian small holder dairy farmers CATTLE AND GOATS Survey and measurement for the cattle and goats survey work occurred in separate areas within three provinces: Lampung Selatan, Malang and Bima. Households with between three and eight cattle and/or goats were chosen for the survey sample. A minimum of sixty households were chosen in each area. Figure 4.3. Map of sampling locations, Indonesia: Lampung Selatan, Malang, and Bima. The collection of data was designed to facilitate comparisons between measured growth rates, and growth rates from literature, as well as various comparisons for girth measurements, weight of cattle and goats, breeds, locations, and feed. 46

47 4.2. LOGISTICS BOTSWANA STAFF AND TRAINING ISSUES Enumerators collected all data, with oversight from project staff. Sampling and design details were approved by representatives of Statistics Botswana. For the sheep and goats and feed survey work, a 1-day training course was held for enumerators, and a half-day orientation held at each site to familiarise extension staff DATA CUSTODY AND CURATION Enumerators recorded interview data on forms, for later entry. Data entry will was completed using Microsoft Excel format recording tables. These electronic records were then transferred to the project team. The records were stored in multiple copies on secure servers TANZANIA ROLES Farmers were used as data collectors, entering data onto data collection templates or notebooks provided. Daily data were collected for eggs and milk. Farmers counted eggs, and while doing so marked each egg counted. Farmers were asked to record any eggs sold, consumed or given away in their notebooks. Each hen was tagged, and farmers entered data into the notebook by tag colour and number. In general, hens were to be tagged when they began a clutch. Where hens were in mid-clutch at the beginning of the test, they were tagged and the day of end of clutch recorded, but eggs from those partial clutches were not counted. Cows to be milked in the trial were selected based on known calving date. Where possible, indigenous AND improved breeds were selected, for a maximum of 5 cows per farmer. Farmers collected data every day on BOTH morning and evening milkings, where relevant. Also where relevant, access of calf to the cow was recorded (limited, unlimited, none). 47

48 Extension officers or enumerators visited farms regularly (although not every day) to check that data were being recorded. Enumerators collected the data recording templates and checked data entry practices. Supervisors oversaw enumerators work at regular intervals TRAINING A 2-day training course was held for enumerators and supervisors in Dar Es Salaam, including preparation for farmer training. A subsequent 1-day training course was held for farmers. Immediately following training, extension officers visited households to ensure proper procedures were being followed. Farmer training courses were also used as an opportunity to interview farmers with the existing (E) questionnaires (approximately half the participants), and the alternative (A) questionnaires (the other half of the participants). Farmer training was completed first in Morogoro, and then immediately afterwards in Dodoma WORK PROGRAMME Immediately after training, extension officers visited households as soon as possible to ensure that the proper procedures were being followed. Enumerators visited households about once every 3 days thereafter, and supervisors visited households with enumerators about once every 10 days DATA CUSTODY AND CURATION Enumerators collected the data entry templates and provided them to the UNE project officers, who entered the data and maintained secure data storage. Data entry was completed using Microsoft Excel. Records were kept on USB drives, backed up, and regularly transmitted to the in-country research partners and UNE INDONESIA STAFF AND DATA COLLECTION Farmers collected milk production for two animals each day with enumerators supervising milking, measurement and data collection for a minimum of one day per fortnight per household. In the case of the cattle and goats survey work, three enumerators in each area worked together to visit households, conduct the 48

49 survey, and weigh animals. Supervisors oversaw (at regular intervals) enumerators work. Extension officers (local government staff) assisted with access to the farm households and residents for the milk survey TRAINING A 2-day training and testing of procedures was held for enumerators working on both the milk and the cattle and goats surveys DATA CUSTODY AND CURATION For the milk survey, farmers and enumerators completed data templates, with data entry completed using Microsoft Excel. These electronic records were kept on USB drives, backed up, and regularly transmitted to the Ministry of Agriculture, PT Myriad and UNE. For the cattle and goats survey, enumerators collected data using an Android tablet, which was automatically uploaded to a secure UNE server. Full access to the secure online data was given to the Ministry of Agriculture and PT Myriad. 49

50 5 Data Handling 5.1. TANZANIA Microsoft Excel was used for entry and collation of the Tanzanian data, and was also used to integrate the data from the two case study locations. For ease of analysis and integration, two files were created for each commodity; one comprising the existing and alternative questionnaire response data, and the other comprising the gold standard daily data. All records in each data set include a unique household identifier code, and each record in the data was also uniquely numbered BOTSWANA The Botswanan research partner used Microsoft Access for data entry of all existing and alternative questionnaires. The data files were exported from Access as.csv files for checking in Microsoft Excel, and later importation into SPSS for data analysis. Gold standard data were collated into a Microsoft Excel workbook, and imported into SPSS for data analysis INDONESIA Microsoft Excel was used by the Indonesian research partner, for data entry of the milk questionnaires and of the fortnightly data collected through observation of particular cattle. A single data entry file was created, integrating the questionnaire and fortnightly observations which pertained to the same response group. All records were coded with a unique identification number. Software was developed using the Open Development Kit (ODK) for the data entry of the Indonesia cattle and goat tests. The software was run on a combination of Samsung Galaxy Tab A and Google Nexus 7 Tablets. The 50

51 server portion of the software operated on a UNE server. At the completion of testing, data was extracted from the server. Separate files were created in the online data entry system for the related gold standard cattle and goat measurement data. These files included initial measurement data from the first farm visits (separate files for cattle and goats) as well as a separate file for subsequent data recording visits, again with separate files for cattle and goats. These files were exported from the online system as.csv files for analysis. Once the survey and measurement data had been integrated and/or cleaned where applicable, each data set was imported into SPSS to facilitate analysis. Missing values analysis was completed, and frequencies of all variables produced, to identify aberrations during data entry, or to identify potential outliers amongst the responses or collected data. Presence of outliers is an important element of statistical procedures, so the project s datasets have retained all outliers. Missing values were recorded as zeros where appropriate, or set as missing in the SPSS files for analysis. Some issues identified during data cleaning are discussed further in Sections 7.2. Basic indicators were identified by project staff, based on the literature review and gap analysis, and in consultation with local stakeholders. These indicators were variously available from data collected from the questionnaire data set/s, or the gold standard data. Analysis was restricted to frequencies, means, and multiple responses (using SPSS and Microsoft Excel) to produce these basic indicators. SPSS output and syntax files were saved to allow analysis to be readily reproduced. SPSS output was copied to Microsoft Excel to produced tables or charts for the report, where appropriate. In some cases, raw data were copied directly to Excel for transformation and analysis where very simple analysis was required. A single Excel data output file was kept for each case study country and commodity. 51

52 6 Key Results of the Analysis and Indicators Produced Full results of the analysis are included in Appendix BOTSWANA - SHEEP AND GOATS CASE STUDY SUMMARY Table 6.1. Summary data from existing questionnaire, sheep and goats, Botswana. Number of respondents, broken down by administrative district (n = 62) Letlhakeng Mahalapye Molepolole Tsabong Number of Male Respondents Number of Female Respondents Respondent Demographics Average Age Male Respondents (Years) Average Age Female Respondents (Years) Animals held by respondents Total Sheep (n = 25) Average Sheep per Respondent (n = 25) Total Goats (n = 57) Average Goats per Respondent (n = 57) Average changes to sheep and goat herds 1/10/14 to 30/9/15 Sheep (n = 20) Goats (n = 57) Additions to Deductions from Additions to herd Deductions from herd herd herd

53 Table 6.2. Summary data from alternative questionnaire, sheep and goats, Botswana. Number of respondents, broken down by administrative district (n = 61) Letlhakeng Mahalapye Molepolole Tsabong Number of Male Respondents Number of Female Respondents Respondent Demographics Average Age Male Respondents (Years) Average Age Female Respondents (Years) Animals held by respondents Total Sheep (n = 33) Average Sheep per Respondent (n = 33) Total Goats (n = 58) Average Goats per Respondent (n = 58) Table 6.3. Summary data from gold standard data, goats and sheep, Botswana. Number of participants, broken down by district Central Kgalagadi South Kweneng Letlhakeng Molepolole Average body weight, goats and sheep < 12 months old (kg) Female Goats (n = 765) Male Goats (n = 825) Female Sheep (n = 380) Male Sheep (n = 300) Average girth measurement, goats and sheep < 12 months old (cm) Female Goats (n = 765) Male Goats (n = 825) Female Sheep (n = 380) Male Sheep (n = 300) Average shoulder height, goats and sheep < 12 months old (cm) Female Goats (n = 765) Male Goats (n = 825) Female Sheep (n = 380) Male Sheep (n = 300) Average body condition score, goats and sheep < 12 months old (1-4 scale) Female Goats (n = 765) Male Goats (n = 825) Female Sheep (n = 380) Male Sheep (n = 300)

54 Table 6.4. Summary of indicators used and data sources, sheep and goats, Botswana. Indicator Changes to herd structure (births, deaths, acquisitions, disposals) over a 12 month period Impact of animal age and time of year/season on changes to herd structure (births, deaths, acquisitions, disposals) over a 12 month period Sources of livestock of different ages and at different times of year Purchasers of livestock of different ages and at different times of year Causes of death of livestock of different ages and at different times of year Average body weight x age category of livestock Average body weight x sex of goats or sheep Average girth measurement x sex of goats or sheep Average shoulder height measurement x sex of goats or sheep Average body condition score x sex of goats or sheep Changes in body weight x sex of goats or sheep Changes in girth measurement x sex of goats or sheep Changes in shoulder height measurement x sex of goats or sheep Proportion of animals with each body condition score x sex of goats or sheep Source Existing Alternative Alternative Alternative Alternative Alternative Gold standard Gold standard Gold standard Gold standard Gold standard Gold standard Gold standard Gold standard 6.2. BOTSWANA FEED AVAILABILITY CASE STUDY SUMMARY Table 6.5. Summary data from existing and alternative questionnaires, feed availability, Botswana. Average area of feed crops grown per household (hectares) Existing Questionnaire (n = 16) Alternative Questionnaire (n = 30) Average grazing availability per household (hectares, alternative questionnaire) Fenced grazing (n = 10) Communal grazing (n = 29) Average quantity of stock feed purchased per household (kg) Existing Questionnaire (n = 43) Alternative Questionnaire (n = 50)

55 Table 6.6. Summary data from gold standard data, feed availability, Botswana. The three most common herbaceous species identified on communal grazing sites, by district Central Kgalagadi Kweneng Aristida congesta Digitaria eriantha Eragrostis rigidor Stipagrostis uniplumis Eragrostis lehmanniana Schmidtia kalihariensis Average herbaceous biomass (g/m 2 ), by district Digitaria eriantha Aristida congesta Schmidtia pappophoroides Central (n = 13) Kgalagadi (n = 38) Kweneng (n = 15) Proportion of survey sites with good grass palatability estimated, by district Central (n = 13) Kgalagadi (n = 38) Kweneng (n = 15) Table 6.7. Summary of indicators used and data sources, feed availability, Botswana. Indicator Total area sown to (unspecified) feed crops over the last agricultural year Average total area sown to specific feed crops each year Quantity of specific livestock feeds purchased annually (kg) Proportion of farmers using various stock feed options to supplement the diet of cattle, sheep and goats Number of days purchased feed used for cattle, sheep and goats, by stock age categories Number of days annually each type of livestock uses particular feed crops and their residue categories Number of days annually different types of grazing area grazed by different types of livestock Rating of pasture degradation on a scale from None to Severe using the contributing factors of presence of Seloka Grass, and bush encroachment Composition and density of herbaceous species across different communal grazing sites Amount and quality of herbaceous biomass across different communal grazing sites Composition and density of woody species across different communal grazing sites Distribution of woody species of different heights across different communal grazing sites Source Existing Alternative Existing, Alternative Existing Alternative Alternative Alternative Alternative Gold standard Gold standard Gold standard Gold standard 55

56 6.3. TANZANIA EGGS CASE STUDY SUMMARY Table 6.8. Summary data from existing questionnaire, eggs, Tanzania. Number of respondents, broken down by location Morogoro Dodoma Egg production during 12 month period Total eggs produced Average eggs produced per respondent Table 6.9. Summary data from alternative questionnaire, eggs, Tanzania. Morogoro Number of respondents, broken down by location Dodoma Number laying Number of hens laying eggs in past 12 months Average number laying per respondent Total eggs produced Egg production during 12 month period Average eggs produced per respondent Clutches Average number of clutches per Average days per respondent from beginning of respondent over 12 months clutch to beginning of next clutch Table Summary data from gold standard data, eggs, Tanzania. Morogoro Number of participants, broken down by location Dodoma Number of hens laying eggs during data collection period Hens 'currently laying' Hens laying one or more eggs Egg production Total eggs produced during data Estimated average eggs per respondent over 12 collection month period

57 Table Summary of indicators used and data sources, eggs, Tanzania. Indicator Eggs produced on farm over 12 month period: Number of eggs Sep + Number of eggs Oct + + Number of eggs Aug Eggs produced on farm over 12 month period: Number of hens laying eggs last 12 months x Average number of clutches per hen x Average number of eggs per clutch Eggs produced per hen over 12 month period: (Number of eggs counted / number of days data collection carried out for this hen) x 365 days Eggs produced per hen over 12 month period: Average number of clutches per hen x Average number of eggs per clutch. Times of year of high and low egg production Influence of hen breed on egg production Source Existing Alternative Gold standard Alternative Existing; Communal Alternative; Gold standard 6.4. TANZANIA MILK CASE STUDY SUMMARY Table Summary data from existing questionnaire, milk, Tanzania. Number of respondents, broken down by location Morogoro Dodoma Number of cows milked in the last 12 months Total (n = 76) Average per respondent Average milk production per cow milked (litres) Daily Annual Daily, highest month of production Daily, lowest month of production Average milk production per farm (litres; n = 76) Daily Annual

58 Table Summary data from alternative questionnaire, milk, Tanzania. Number of respondents, broken down by location Morogoro Dodoma Number of indigenous cows milked in the last 12 months Total (n = 67) Average per respondent Number of improved cows milked in the last 12 months Total (n = 28) Average per respondent Average milk production per lactation per cow milked (litres) Indigenous cows (n = 66) Improved cows (n = 28) Average milk production per lactation per farm (litres) Indigenous cows (n = 66) Improved cows (n = 28) Table Summary data from gold standard data, milk, Tanzania. Number of cows observed, broken down by location Morogoro Dodoma Breed of cows milked during data collection period Indigenous Improved, other, and not specified Average daily milk production per cow milked during data collection (litres) AM (morning) PM (evening) Average daily milk production per breed per cow milked during data collection (litres) Indigenous Improved

59 Table Summary of indicators used and data sources, milk, Tanzania. Indicator Daily milk production per cow per day Annual milk production per cow: Average milk production per cow per day x Average number of months cows milked for x 30 (days/month) Daily milk production per farm: Average milk production per cow per day x Number of cows milked in the last 12 months Annual milk production per farm: Average milk production per cow per day x Number of cows milked in the last 12 months x Average number of months cows milked for x 30 (days/month) Influence of time of day on milk production Influence of time of year on milk production Influence of breed on milk production Quantity of milk produced per lactation per cow (indigenous and improved cows) Source Existing; Alternative; Gold standard Existing Existing Existing Gold standard Existing Gold standard Alternative 6.5. INDONESIA CATTLE AND GOATS CASE STUDY SUMMARY Table Summary data from questionnaire, cattle and goats, Indonesia. Number of respondents, broken down by location Bima Lampung Malang Average number of animals kept per respondent Cattle (n = 219) Goats (n = 178) Average changes to cattle and goat herds since last Eid al-adha Cattle (n = 408) Goats (n = 408) Deductions from Additions to Deductions from Additions to herd herd herd herd Main feed types used for cattle (proportion of response; n = 408) Collected native grasses Stover Grazing Rice bran Main feed types used for goats (proportion of response; n = 408) Collected native grasses Stover Grazing Planted forages

60 Table Summary data from gold standard data, cattle and goats, Indonesia. Number of participants, broken down by location (where location recorded) Bima Lampung Malang Average body weight, cattle (kg) Calves (n = 97) Weaners (n = 59) Young Adults (n = 200) Adults (n = 352) Average girth measurement, cattle (cm) Calves (n = 97) Weaners (n = 59) Young Adults (n = 200) Adults (n = 352) Average body condition score, cattle Calves (n = 98) Weaners (n = 59) Young Adults (n = 200) Adults (n = 352) Average body weight, goats (kg) Young Goats (n = 115) Young Growing Goats (n = 174) Adult Goats (n = 337) Average girth measurement, goats (cm) Young Goats (n = 115) Young Growing Goats (n = 174) Adult Goats (n = 337) Average body condition score, goats Young Goats (n = 115) Young Growing Goats (n = 175) Adult Goats (n = 337)

61 Table Summary of indicators used and data sources, cattle and goats, Indonesia. Indicator Body weight of cattle or goats by age category: Average body weight x age of cattle or goats Girth measurement of cattle or goats by age category: Average girth measurement x age of cattle or goats Body condition score of cattle or goats by age category: Proportion of cattle or goats under each BCS x age of cattle or goats Changes in body weight of cattle or goats by age category: Changes in body weight x age of cattle or goats Changes in girth measurement of cattle or goats by age category: Changes in girth measurement x age of cattle or goats Changes in body condition score of cattle or goats by age category by sex: Changes in body condition score x age x sex of cattle or goats Changes in body weight of cattle or goats by age category by sex: Changes in body weight x age x sex of cattle or goats Changes in girth measurement of cattle or goats by age category by sex: Changes in girth measurement x age x sex of cattle or goats Source Gold standard Gold standard Gold standard Gold standard Gold standard Gold standard Gold standard Gold standard 6.6. INDONESIA MILK CASE STUDY SUMMARY Table Summary data from questionnaire, milk, Indonesia. Number of respondents, broken down by village (all respondents located in Malang) Gading Kembar/Depok Gunung Kunci Kemiri Average number of animals kept per respondent Cattle (n = 60) Goats (n = 13) Chickens (n = 9) Ducks (n = 2) Average changes to cattle and goat herds since last Eid al-adha Additions to herd Deductions from herd Responsibility for cattle husbandry (% of respondents; n = 60) Adult Male Adult Female Child Main feed types used for cattle (proportion of response; n = 60) Purchased concentrates/rice bran Planted forages Stover Sugarcane shoots/ top stem Milk production (n = 60) Total cows milked Average Average daily milk Average annual milk cows milked production per cow production per cow

62 Table Summary data from gold standard data, milk, Indonesia. Number of cows observed, broken down by village Gading Kembar/Depok Gunung Kunci Kemiri Average daily milk production per cow milked during data collection (litres) High Production Cow (n = 60) Low Production Cow (n = 60) AM (morning) PM (evening) AM (morning) PM (evening) Estimated annual average milk production per cow (litres) High Production Cow (n = 60) Low Production Cow (n = 60) Average age of cows (months) High Production Cow (n = 60) Low Production Cow (n = 60) Average girth measurement of cows (cm) High Production Cow (n = 60) Low Production Cow (n = 60) Average body condition score of cows High Production Cow (n = 60) Low Production Cow (n = 60) Table Summary of indicators used and data sources, milk, Indonesia. Indicator Average annual milk production per cow: Average milk production per cow per day x Average number of months cows milked for x 30 (days/month) Average annual milk production for high production, low production and all observed cows: Average milk production per [high production/low production/all observed] cows per day (GS) x Average number of months cows milked for (Q) x 30 (days/month) Average daily milk production per farm: Average milk production per cow per day x Number of cows milked in the last 12 months Average annual milk production per respondent: Average milk production per cow per day x Number of cows milked in the last 12 months x Average number of months cows milked for x 30 (days) Influence of time of day on milk production Influence of productivity of individual cows on daily milk production Influence of time of year on milk production Influence of body weight, girth measurement, and body condition score on daily milk production Influence of calf suckling practice on daily milk production Source Questionnaire Questionnaire; Gold standard Questionnaire Questionnaire Gold standard Gold standard Questionnaire Gold standard Questionnaire 62

63 7 Paradata and In-Country Evaluation of Methods Employed in the Test Phase Research partners from Botswana, Tanzania and Indonesia were asked to evaluate the Test phase after its completion and having seen the resulting data. This feedback centred on the processes of preparation and design, data collection, cleaning and analysis, the amount and type of resources allocated. The relative merits of the alternative and existing questionnaires and aspects of the gold standard data collection were also assessed. Feedback was received from three partner staff in each of Botswana and Tanzania, and two in Indonesia DATA COLLECTION EFFICIENCY AND TIMELINESS Pilot country partners were in general agreement that the collection of the alternative questionnaire data was completed in an efficient and timely manner. Partners reported experiencing some problems with implementing the originally-agreed data collection processes, and reported making adjustments accordingly. Such issues included: Insufficient staff resources dedicated to collecting the data in the allocated time was an issue identified in all three pilot countries. In part this was due to distances the enumerators and survey managers had to travel to interact with farmers. It is notable that this problem was encountered (in both Botswana and Tanzania) despite the project s 63

64 having adopted a cluster-based sampling strategy in order to minimise this problem. Solutions adopted included delays in delivery, and hiring of extra staff. A Botswanan research partner commented that there was insufficient time to weigh animals twice in the field as part of gold standard data collection, while at the same time completing existing and alternative questionnaire interviews with the farmers. A Tanzanian research partner commented that it was difficult to time the collection of poultry laying data with the clutching period of chickens. He suggested that a longer data collection window would have resulted in a greater number of hens available for measurement. A research partner in Tanzania commented that some farmers were deliberately not being truthful about the status of their hen flocks and clutching periods, so as to ensure their participation in the data experiment. A consequence was that fewer hens were enrolled in the survey than planned during sampling. Some difficulty was experienced in Indonesia, with participating farmers being unwilling to participate in subsequent on-farm visits for data collection. The in-country research partners in both Indonesia and Tanzania (where this problem had been anticipated) addressed this issue by providing gifts to the farmers at the time of their final visit, as a way of thanking them for their participation and boosting the participant base. Collection of data by farmers in Tanzania proceeded by way of the fieldwork team checking the farmer s measurements to ensure that the data were accurate. Indonesian research partners noted that it was more difficult to use the database-driven/tablet recording and uploading system in remote rural regions in the Bima district, where internet access was found to be poor in places. Their solution was to improvise a paper-based version of the tablet survey that had been established by the Australian research team members. A separate team of Indonesian staff was then employed to enter the data collected by the fieldwork team into the tablet-based system ALTERNATIVE QUESTIONNAIRE DATA COLLECTION PROCESSES Research partners in all three pilot countries were generally satisfied that the data collection process used for the alternative questionnaire was appropriate. Nonetheless, team members from Tanzania commented that the process 64

65 required continued field testing, with the goal of better matching data collection periods with farm production periods. This indicates an abiding belief that recall and self-evaluation by farmers are affected by the proximity in time to actual events, which was also raised by a research partner in Botswana with respect to recall periods of 12 months, and the fact that farmers did not tend to keep production records. This colleague suggested that farmers have advance warning of data collection activities so that they are aware of the related need to keep records and in general, be more aware of their production data. For Botswana, a role for trained extension officers was proposed, to assist farmers in collecting data at six monthly intervals. The research teams were unanimous in their belief that the process used to collect gold standard data was appropriate. Some improvements were suggested, including that the period of data collection needed to be lengthened, and that (as for the alternative method) it needed to be timed to better coincide with livestock production periods. One research partner, from Tanzania, cautioned that over-reliance on farmers to assist with measurement had the potential to result in inaccurate measurements of production, and therefore unreliable data. In Botswana, the research partners noted that farmers were very interested in seeing their animals weighed and their dimensions measured and recorded. They suggested that generalising/categorising the age of livestock was not best practice, and that a better approach to give a more accurate estimate of livestock age was to take note of the dentition (development of teeth), in measured animals QUALITY CONTROL PROCESSES The research partners were asked to consider ways in which the quality control processes used during data collection may be improved. Suggestions included: Training farmers in data collection processes in advance of a formal data collection period, to ensure they knew what is expected of them. Ensuring that data collectors are well trained and well supervised, with the supervisor becoming directly involved in data collection where this was necessary. A Tanzanian research partner noted that daily involvement with the data collection team may be required to verify their processes continually. In Botswana, the research partner pointed out that that staff turnover amongst the enumerators needed to be minimised. 65

66 Pre-selecting farmers for involvement in data collection based on their willingness to have an extension officer or team of investigators take measurements of production and productivity on their farm. Random and/or targeted data quality checking by supervisors or a quality control team. An Indonesian team member suggested that a random sample of 30 per cent of the data may need to be checked for quality during the collection process, perhaps by a trusted local person who has also observed the data collection staff in operation. An Indonesian partner believed that implementing the technology-assisted data collection approach had been important in strengthening quality control of data collection for the Indonesian Test phase. In Botswana, one of the research team members likewise advocated moving to a computerised data collection system rather than using paper-based forms. Their belief was that this would both improve the quality of the data, and reduce survey time and costs GENERAL IMPROVEMENTS TO DATA COLLECTION PROCESSES A range of improvements were suggested, both directly relevant to this project and relevant to future or expanded data collection exercises of this nature: Increasing the sample size. Determining the role that students may play (for example a livestock training institution in Tanzania) to carry out the field work activities, so as to address both survey costs and the experience of livestock professionals in data collection. Pre-collection visits to farmers, or consultation by other means, to determine whether or not farmers were suited to data collection in terms of readiness (one on hand) and suitability according to sampling criteria (on the other). Collection of data during both the wet and dry seasons, and annual repetition of surveys to build up a comparative data set over time and across varying seasonal conditions. 66

67 Local government involvement in data collection, for example their sending letters to potential respondents to boost the response rate giving the survey more credence and some official weight. Having farmers collect data themselves, under the supervision of extension staff or other relevant support workers. Including a larger selection of regions, to be able to consider a more representative sample. New variables in the alternative questionnaires: including livestock breed, as one example. Provision of gifts to farmers to boost participation. In Indonesia where electronic data collection was used in the pilot, it was recommended that although this data collection process was evaluated as very time efficient, it would be advantageous to employ younger enumerators, who are more likely to be familiar with the technology DATA CLEANING AND PROCESSING Regular data cleaning was carried out by the research partners in each country both during the data entry phase and before the final cleaned data set was forwarded to the Australian team for analysis. In Botswana, data collection managers checked regularly with the enumerators to make sure data were being recorded correctly, and that the questions made sense to the enumerators. Data cleaning included checking that totals added up as they should, and that blank entries were valid. With respect to the sheep and goat data collection, the Botswanan managers checked that animals dying between the first and second data collection periods were recorded properly. Tanzanian data collection managers checked the data for obvious outliers and incorrect data entry from paper forms into computerised data entry files, and corrected appropriately. 67

68 The Indonesian data processing team checked the data quality on a daily basis, and provided feedback to field enumerators where this was necessary. Further data checking and cleaning was carried out once the full data sets had been collected. Research partners were asked to consider data cleaning and processing of the alternative questionnaires used in their country in comparison with the existing questionnaires used. Although not all partners were able to provide a response on these issues, there was general agreement across the three pilot countries that: The alternative questionnaire was easier to understand than the existing questionnaire. The alternative questionnaire data were easier to process than the existing questionnaire data. The alternative questionnaire data were easier to interpret than the existing questionnaire data PROJECT RESOURCES Items cited as requiring greater resource allocation than achieved during the project included: training; supervision; data collection time; data entry; day-to-day planning and implementation; communication activities with farmers, farmer groups and local officials; access to vehicles and fuel for transport; and operating costs (e.g. transport and accommodation) associated with delays due to delays in completion. The research partners in each pilot country were asked to provide their opinion on whether each of the existing questionnaire, alternative questionnaire, and gold standard data collection represented good value for money. 68

69 Across the Botswanan and Tanzanian research partners, there was no agreement that the existing questionnaire was good value for money. This contrasts with the evaluation of the alternative questionnaire which team members in both countries saw as good value for money. One Tanzanian team member still felt the alternative questionnaire was not a valuable return on the investment necessary to construct it. The distinction between existing questionnaire and alternative questionnaire did not apply in Indonesia as no existing questions were identified. Nonetheless, the Indonesian team members reported that the questionnaires used were a productive investment. Gold standard data collection process represented substantially more expense than did the questionnaires. Despite this, the gold standard data sets were considered overall to be the best value for money from the project amongst research partners in Botswana, Tanzania and Indonesia. Further, partner organisations in all countries requested immediate access to the gold standard data for further analysis. Research partners provided feedback on the support offered by the Australian project leadership team. A team member from Botswana suggested that the Australian team needed to give the Botswanan research partners more time to plan for intended project activities and a larger budget. It was suggested that an advance period of 6 months was required to allow local team members to plan properly, liaise effectively with government staff and work with government systems for implementation, and have sufficient time to advise intended participants (farmers) about the project objectives and benefits. An Indonesian team member commended the Australian management team for being flexible in budget management USEFULNESS OF THE ALTERNATIVE QUESTIONNAIRE Generally, the alternative questionnaires addressed more variables, and included more detailed questions than did the existing questionnaires. Research partners were asked to consider the usefulness of the alternative questionnaire, with an overall focus on the FAO criteria for quality of production and productivity data (relevance; accuracy and reliability; timeliness and punctuality; coherence and comparability; and accessibility and clarity). 69

70 There was strong agreement across pilot countries that the alternative questionnaire data were both more useful, and more relevant, than were the existing questionnaire data. Partners in all three countries agreed that the alternative questionnaire data were accurate and reliable. Opinion was divided on whether the alternative questionnaire could be implemented in a similar amount of time to the existing questionnaire. Team members in Botswana were inclined to agree that the time required for collection would be similar for both questionnaires; however those in both Tanzania and Indonesia believed that more time was required for alternative questionnaire data collection. Across all three countries, partners generally agreed that the alternative questionnaire data were coherent, and were comparable with data available from other sources. One Tanzanian team member did not agree that alternative questionnaire data were coherent with data available from other sources. All partners consulted on the methodology believed that the alternative questionnaire data were closer to a gold standard for measurement of livestock production and productivity than the existing questionnaire data. An Indonesian research partner commented that the questionnaire and gold standard collection exercises both illustrated that it was possible to obtain more detailed information from farmers than had previously been attempted USEFULNESS OF THE GOLD STANDARD DATA As with the alternative questionnaire data, the research partners were asked to consider the usefulness of the gold standard data against the FAO criteria for quality control of production and productivity data (relevance; accuracy and reliability; timeliness and punctuality; coherence and comparability; and accessibility and clarity). There was strong agreement in that the gold standard data were more useful and relevant than were the existing questionnaire data. One partner each from Botswana and Indonesia were unsure on this matter. Apart from one Indonesian team member who was uncertain, all the remaining partners concluded that the gold standard data were both accurate and reliable. 70

71 Most partners agreed that the gold standard data warranted the time required for data collection (measurement and recording), although one Indonesian partner disagreed. There was general agreement that the gold standard data were comparable with data sets available from other sources, although one Tanzanian partner strongly disagreed. The partners believed strongly that the gold standard data collection process resulted in a scientifically valid data set, although one Indonesian team member was unsure. A Tanzanian team member commented that the gold standard was more effective than was the questionnaire-based data collection using the alternative and existing methods. Some advantages of the gold standard data over questionnaire-based data were noted by the research partners: The data were considered useful in guiding more appropriate and ultimately more productive decision making amongst farmers. Tanzanian and Indonesian partners commented that it became unnecessary to rely on farmer recall through daily or regular measurement of production and productivity (e.g. measuring milk or egg production, or animal weights and dimensions). A Botswanan partner suggested that the gold standard approach to data collection not only resulted in more detailed data, but had the flexibility to include a variety of measurements to ensure that the most important indicators of livestock production and productivity were included RELEVANCE OF THE PROCESS TO FARMERS The research partners were asked to provide their feedback on how relevant they believed the data collection process was to farmers. The partners all expressed the view that the project was useful in demonstrating to farmers the importance of keeping more detailed production records. A number of partners commented further on this issue, noting that: more detailed record keeping helped farmers to determine the productivity of particular animals 71

72 a much higher degree of accuracy and usefulness of data collection was demonstrated, relative to mere estimation of production or production potential gold standard measurement allowed for instant use of production and productivity factors to farmers, enabling management tasks such as selection and herd culling, and sale and purchase of animals. Research partners considered that the project was an important extension activity, by reinforcing the significance of data collection and record keeping as a way of improving productivity. All partners agreed that the farmers involved in the research were able to understand the questions being asked of them in the existing and alternative questionnaires, and to answer these questions clearly. All partners were in strong agreement that this project demonstrated the potential for farmers to become involved in a more formal and regular way in collection and measurement of data on livestock production and productivity. The data collection process gave farmers an extra opportunity to interact with extension staff (in Botswana), and to discuss issues with each other. A Botswanan team member suggested that local livestock groups could potentially play a role in facilitating data collection and simultaneously boosting their membership amongst farmers, while extension staff had a role in helping farmers to understand the data and to act on it. All partners considered that farmers had learned more about factors influencing their profitability, and on certain items such as the importance of vaccination (in this case, of poultry) and animal health management. An Indonesian partner commented that, in general terms, farmers were more willing to participate in livestock data collection if they had a greater preexisting awareness of the importance of good data. They noted a general trend where farmers in Malang were more willing to participate than those in Bima. A potential solution suggested by this research partner was to undertake farmer education/extension to emphasise to them the benefits of data collection and more accurate and detailed livestock record keeping, prior to implementation of data collection activities. 72

73 8 Costs Comparisons in Evaluating Methodological Change 8.1. INTRODUCTORY COMMENTS In consideration of methodological change, substantial costs are incurred in conceptualisation, consultation, information collection, design, and testing of instruments. Further, and of particular interest in the current project, Gold Standard data collection is expensive due to its repetitive and exacting nature, its duration, the inconvenience to various parties to the process, and the need for vehicles, accommodation and rest periods associated with the use of survey staff COSTS OF CONSULTATIONS UNDER THE PROJECT In addition to discussions surrounding the Literature Review, GAP analysis and the consultative meeting in Ghana, substantial communication was held with incountry partners. These costs are essentially associated with the project so are not considered further in this report. This should however not be interpreted as any devaluation of these interactions, were conducted as a substantial cost to those involved THE TEST PROCEDURE AND ITS RELEVANCE TO COSTS Formulation of the test procedure was an iterative process leading up to, including, and then proceeding from the consultative meeting in Ghana. As above, this engaged many in-country people from various agencies. Staff training, purchase and preparation of materials and logistic design all occurred at significant cost. The implementation of the test procedure, associated 73

74 logistic costs, data handling and on-going management are also costly processes in terms of time and materials. Although largely paid by the project, these costs are considered here to a small degree because: the Gold Standard surveys are on-off events and not encountered as a part of any regular data collection; staff training and many staff preparation and planning costs for the alternative questionnaires are very similar those encountered in the existing surveys; the existing and alternative questionnaires are very small in extent, addressing only the selected indicators and methods, in comparison to full questionnaires as used in full scale surveys; the Existing and Alternative interviews and related survey activities were carried out on the same sample and logistic basis as the Gold Standard work. Hence separation of costs is necessary for insight into costs of Existing and Alternative methods; the cluster approach to sampling in the test procedure resulted in a dense sampling pattern which is a strong influence on the costs of data collection APPROACH USED The cost information provided here is primarily a comparison between Existing and Alternative questionnaires and their implementation, and the subsequent use of their results. Moreover, costs assigned to various methods are expressed in incremental terms, to better identify the cost advantage or disadvantage offered by any alternative method. This approach provides the result that all Alternative methods are more costly than Existing ones, as they involve change COSTS OF DESIGN, SURVEY TESTING AND PLANNING FOR DATA COLLECTION SCALE RELATED INFLUENCES IN COSTS The size of a data collection activity (number of respondents, length of questionnaire, etc.) affects survey costs by a variety of mechanisms. Costs are therefore comparable only when expressed per unit of activity. The approach taken here is to express data collection activities in terms of cost per day of survey. Variations on baseline costs which affect the duration of data collection activities are represented as increases in staff and activity levels, so as to keep survey duration constant in the analysis. 74

75 The purpose of arriving at a cost per survey day, and its extension through scenarios representing Alternative methods, is to provide a measure of costs which is scaleable to national levels by multiplying by the days estimated. Similarly, the basic assumptions and relations can be varied to address national or regional estimates based on other measures of scale or intensity LABOUR Labour included in this analysis: Refers only to existing and alternative questionnaires Includes training time for alternative questionnaires and other data collection procedures under the alternative methods Includes travel time used for alternative questionnaires and other data collection procedures under the alternative methods Includes materials used for alternative questionnaires and other data collection procedures under the alternative methods Includes enumerator and supervisor time, and the shares of each used, time for alternative questionnaires and other data collection procedures under the alternative methods Includes data entry and data cleaning Includes input by specialists where data collection requires this input LOGISTICS Logistic costs included in this analysis: Includes fuel, vehicle maintenance and costs associated with drivers Includes accommodation and food Includes drivers time EQUIPMENT Equipment used include any measurement, recording or communication items used in the Alternative or Existing procedures FARMERS TIME Farmers time is not considered here, but would include: Farmers time used in the surveys 75

76 Time (actually or conceptually) taken by farmers as they were trained in data collection roles for the Existing and Alternative questionnaires SAMPLING RELATED INFLUENCES ON COSTS Consideration is not given here to sampling, although it might be conducted at larger scales than the test phase of the current project and so result in cost savings SYSTEM RELATED INFLUENCES ON COSTS Consideration is not given here to costs associated with applications of methods in a range of production and social systems, nor in any variety various physical locations. 8.3 COSTS ESTIMATES APPROACH TAKEN AND ASSUMPTIONS MAINTAINED For each of the three pilot countries, costs of using the Existing data collection method were calculated, and costs of selected departures associated with Alternative methods are analysed. A set of baseline data was assembled during the TEST phase of the project, presented as Appendix 2. This was processed according to a set of assumptions surrounding the structure of training, staffing and logistics, which are presented in Table 8.1. The baseline then entails a team of 6 enumerators (working in pairs) and 2 supervisors, conducting a data collection activity accessing 350 participants. Derived measures of activity duration are based around 7 household visits or interviews/day (just over 1 hour per interview). Drivers and vehicles, and supporting staff, are proportionately assigned. This size of data collection activity and support structures approximates that used in testing of Existing and Alternative methods in the field, adjusted upward to approximate a feasible sample size and a feasible period of work (4-5 days) for such a team. 76

77 Table 8.1. Assumptions used in cost calculations. Scale of data collection and training activities Sample size 350 respondents Interviews/day/2 enumerators 7 interviews per day Communal event size 80 in communal data collection for one day Training duration 2 days Data entry duration 3 days Staffing Enumerators Supervisors Extension officers or others Drivers Cars Training 6 2 Survey Data entry 3 2 Communal data collection Derived measures Length of interview 1.14 hours Days to conduct survey 4.2 days Days to conduct communal events 4.4 days COSTS OF DATA COLLECTION, EXISTING METHOD Based on these assumptions, the following indicative costs of Existing methods were calculated, expressed as $US/day of survey work (Table 8.2) across activities in each pilot country. 77

78 Table 8.2. Estimates of cost per day of survey work, Existing method. Tanzania Botswana Indonesia Training costs USD/survey day USD/survey day USD/survey day Trainers' time Trainees' time Meals Accommodation Transport Facilities Equipment Stationery Total training costs 1, Survey costs Personnel Supervisors Enumerators Extension officers and local officials per diems Drivers Logistic costs Fuel Car R&M Phone cards Meals Accommodation 1, Data entry Enumerators Supervisors Total survey costs 2,102 2,184 1,605 Total cost of training and survey 3,283 2,901 2,102 The sum of training and survey costs per day spans about $2100-$3300 per survey day. The distribution of costs amongst major categories is shown in Figure 8.1. Meals and accommodation for data collection staff comprise the major cost item, with manpower costs being the main source of variation between countries. 78

79 Figure 8.1. Distribution of data collection costs per survey day by country, Existing method COSTS OF SCENARIOS ASSOCIATED WITH ALTERNATIVE METHODS The cost estimates from the previous section were then used to estimate cost changes associated with elements of the alternative methods tested in the project. The scenarios are constructed by increasing particular training and data collection costs as shown in Table

80 Baseline Expanded detail on egg production and husbandry Expanded detail on milk production and husbandry Expanded detail on sales and purchase channels Expanded detail on seasonal production Expanded detail on herd dynamics Expanded detail on feed measures Expanded detail on animal disease Proxy for animal liveweight Proxy for milk production Proxy for pasture availability Table 8.3. Costs of Alternative methods, by country. Scenario for Alterantive method Training time increase 10% 20% 10% 10% 20% 20% 10% 25% 25% 50% Survey staffing increase 5% 5% 20% 20% 20% 10% 10% 15% 15% 25% Specialist staff Additional vehicle $US/survey day Tanzania 3,283 3,569 3,693 4,073 4,073 4,214 3,867 3,737 4,393 4,393 4,906 Botswana 2,901 3,119 3,194 3,555 3,555 3,641 3,343 3,264 3,852 3,852 4,270 Indonesia 2,102 2,255 2,308 2,566 2,566 2,626 2,414 2,359 2,747 2,747 NA COSTS OF SCENARIOS ASSOCIATED WITH ALTERNATIVE METHODS Figure 8.2. Indexed costs of scenarios. 80

81 In general, Alternative methods using proxy measures of production (e.g. girth for animal weight, use of lactation curve information for milk production) provide for the highest cost increases (up to 50% for proxy measures of pasture productivity). This is because the additional training requirements and the need for specialists to do the training and assist with data collection. Alternative methods involving somewhat more detailed questionnaires, specifically adding more detail on production variables, add around 10-25% to costs per survey day. Where such Alternative methods add large and complex tables to the questionnaires (e.g. seasonal production or herd dynamics), costs increase by around 30% COSTS OF COMMUNAL DATA COLLECTION ACTIVITIES Table 8.4 provides estimates of the costs incurred conducting communal data collection activities. These estimates are based on actual costs incurred in the TEST phase of the project, but extrapolated to the sample size of 350 households used above, and expressed per day of communal workshop. The data in Table 8.4 do not include training costs, which would be added to any further analysis. Table 8.4. Costs of workshop for communal data collection, by country. Tanzania Botswana Indonesia Communal data collection USD/day USD/day USD/day Supervisors Enumerators Extension officers and local officials per diems Drivers Car R&M Fuel Meals Accommodation Transport Other costs Total cost of communal data collection 2,248 3,041 2,030 81

82 9 Summary and Recommendations In the following pages, a series of tables are included to summarise the data collection methodology, skills and resource requirements, role of farmers, and data quality and usefulness of the groups of indicators calculated for this study. A list of indicators calculated in each category are provided. Analysis is covered in more detail in Appendix 1. Summary budgets are included in the tables below, as well as a brief discussion of the implications of each group of indicators for future data collection, and lessons learned from the process. The following indicator groupings have been included in the tables: Botswana Influences on herd size and structure, sheep and goats Body weight of sheep and goats Changes in body weight of sheep and goats Feed production and purchase Feed usage and pasture degradation Tanzania Egg production Influences on egg production Milk production Influences on milk production 82

83 Indonesia Body weight of cattle and goats Changes in body weight of cattle and goats Milk production Factors influencing milk production 83

84 9.1. INFLUENCES ON HERD SIZE AND STRUCTURE, BOTSWANA, SHEEP AND GOATS Criteria Existing Q Alternative Q Conclusions Methodology used Sampling frame and design Face-to-face interview, using questionnaire form and reliant on participant recall. Random selection of households using sample frame of district extension lists for three districts: Mahalapye, Tshane/Kgalihadi, Kweneng/Molopole 40 households selected each location; supplemented with neighbours where response not obtained E = 20 responses from each region (half of sample) Data collection period 7/9/15 15/10/15 7/9/15 11/10/15 Data entry and processing Analysis Staff skills required Managers Enumerators collected information on paper forms. Data entered into computer in-country using Microsoft Access. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using means of variables relating to additions to and deductions from herd. Face-to-face interview, using more extensive questionnaire form and reliant on participant recall. As per Existing Q A = 20 responses from each region (other half of sample) Enumerators collected information on paper forms. Data entered into computer in-country using Microsoft Access. Botswanan team members believed the A data were easier to understand and process than the E data. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using means of variables relating to additions to additions to and deductions from herd during rainy and dry seasons (including more details on causes of death), livestock sources, and buyers. Botswanan team members believed the A data were easier to interpret for calculation of indicators than the E data. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity concepts relating to the survey. Local knowledge and relevant languages. Usage of participant recall more problematic in the case of the A questionnaire as considerably more detail required from participants data richer but more prone to recall bias. Sampling approach identical for both E and A Cluster sampling was used to save on logistic costs, and this introduces likely bias when number of clusters is small. Microsoft Access not recommended due to extra time required to set up data entry forms, and necessity of exporting data in several sections for analysis. Questions asked in A allow for more detailed indicator calculation based on age and sex of animals, seasonal influences on changes to herd structure, causes of death, and parties involved in sale and purchase. The A questionnaire also allows multiple approaches to calculating key indicators. The A questionnaire is considerably more detailed, and requires closer attention to detail and greater understanding of production and productivity factors by managers. 84

85 Enumerators Data entry Farmer availability and willingness to participate Farmer understanding of data requirements Influence on farmer practices Data quality and usefulness* Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Training in interviewing and collection of data. Understanding livestock production concepts including age and sex ( rams, castrated males, ewes ) and causes of livestock death. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production concepts including age and sex ( rams, castrated males, ewes ), causes of livestock death, seasonal influences, and livestock trading options. Local knowledge and relevant languages. Training required in Microsoft Access database creation and/or usage for data entry. Attention to detail and accuracy. Botswanan team members agreed that the questionnaire work demonstrated the potential for farmers to be involved in more formalised and regular livestock data collection. Botswanan team members agreed that farmers understood the questions, and were able to answer them clearly. Botswanan team members were of strong agreement that the questionnaire-based data collection demonstrated the importance and usefulness to farmers of keeping more detailed production records. Gap Analysis scores, Botswana relevance scores: no. of animals 4.14; animal deaths 3.09; no. animals sold Gap Analysis scores, Botswana accuracy scores: no. of animals 2.46; animal deaths 2.09; no. animals sold Gap Analysis scores, Botswana timeliness scores: no. of animals 2.36; animal deaths 1.55; no. animals sold Gap Analysis scores, Botswana coherence scores: no. of animals 2.57; animal deaths 1.91; no. animals sold Gap Analysis scores, Botswana accessibility scores: no. of animals 3.00; animal deaths 1.82; no. animals sold Considered more relevant than E data (2 x strongly agree, 1 x agree ) by Botswanan team members. Considered more accurate and reliable than E data (1 x strongly agree, 2 x agree ) Botswanan team members noted that data could be gathered in a similar amount of time as E data (2 x strongly agree, 1 x unsure ) Agreement that A data more coherent than E data, and more comparable with other sources (1 x strongly agree, 2 x agree ) N/A The A questionnaire is considerably more detailed, and requires closer attention to detail and greater understanding of production and productivity factors by enumerators. Instruction in relevant aspects of the nature and distribution of seasons is necessary. Identical skillset required E and A. Farmers appear willing to take part in questionnaire-based livestock data collection, particularly if they can see the benefits for their own farm s management. By including more detailed information, the A questionnaire in particular made farmers aware of the importance of record keeping to enhance their productivity and profitability. Consultation with Botswanan team members suggests strongly that the A data are of a higher quality than E data, assessing them per the FAO criteria. The team members also agreed that the A questionnaire met the goal of gold standard measurement to a greater degree than the E questionnaire. * Gap analysis scores on a scale of 0 to 5: 0 being not available or not useable, and 5 being perfect. See separate Gap Analysis report for more details. 85

86 Indicator Changes to herd structure (births, deaths, acquisitions, disposals) over a 12 month period Impact of animal age and time of year/season on changes to herd structure (births, deaths, acquisitions, disposals) over a 12 month period Sources of livestock of different ages and at different times of year Purchasers of livestock of different ages and at different times of year Causes of death of livestock of different ages and at different times of year Source Existing Alternative Alternative Alternative Alternative Budget Recommendations Sampling design and estimation procedures Botswanan team members believed that the E questionnaire did not represent good value for money, but that the A questionnaire did. TEST procedure followed a cluster-oriented sampling strategy, which provides no significant cost savings for questionnaire-based data collection. A method includes a large number of disaggregated questions requested by Botswana stakeholders during GAP analysis. A method remains centred on farmer recall. Botswana features a large proportion of producers being absentee owners, which lends itself to self-collection of data by owners.. Challenges encountered during collection of the data sets In-country team reported: insufficient time on-site; large transport distances (despite clustered sample); and difficulty in finding farmers at their household to conduct interviews (often being out in the field). It was suggested that farmers need to be forewarned of a survey being conducted, and a time booked to ensure their availability on the day. 86

87 Lessons learned, including feasibility of tested methods for wider-scale implementation The A Questionnaire is a better option than the E Questionnaire for collecting more detailed data which can be used, for example, to illustrate seasonal and market influences on herd size and structure. A potential issue to consider with this approach may be the impact of farmer recall on the quality/accuracy of the data in practice based on the assumption that farmers do not keep detailed records of livestock herd changes, is it too much to ask of them to provide accurate details of herd structure changes at different types of year, numbers bought and sold with more specific sources, in addition to recalling the causes of death of their animals? One Botswanan team member did note that farmers has difficulty at times recalling details of sheep and goat management due to lack of record keeping. Although appreciated by the in-county team as good value or money, the A method is carried out at higher cost than is the E method. 87

88 9.2. BODY WEIGHT OF SHEEP AND GOATS, BOTSWANA Criteria Alternative Q Gold Standard Conclusions Face-to-face interview, using more extensive Direct measurement and weighing of questionnaire form than E, and reliant on animals owned by households in the survey Methodology used participant recall. Note that this A sample, and estimation of body condition questionnaire uses new questions not covered score. All animals under 1 year of age in the E questionnaire. included. Sampling frame and design Random selection of households using sample frame of district extension lists for three districts: Mahalapye, Tshane/Kgalihadi, Kweneng/Molopole 40 households selected each location; supplemented with neighbours where response not obtained A = 20 responses from each region (half of sample) Data collection period 7/9/15 11/10/15 9/9/15 19/11/15 Data entry and processing Enumerators collected information on paper forms. Data entered into computer in-country using Microsoft Access. Botswanan team members believed the A data were easier to understand and process than the E data. Selection of households from the same sampling frame as used for the E and A questionnaires. Animals at 61 households measured across the three districts (1,590 goats, 680 sheep measured). Enumerators collected information on paper forms. Data entered into computer incountry using Microsoft Excel, with a separate worksheet for each district, and each household entered in a separate section of each worksheet. Usage of participant recall problematic data more prone to recall bias than the GS data. Inclusion of many of the same households in the samples would facilitate potential direct comparison of A and GS data to test the effectiveness of farmer recall in collecting accurate data on livestock weight. However this was not possible due to the data being incomparable in terms of recorded age of animals, and aggregation of estimates for the A questionnaire. Microsoft Access (A questionnaire) not recommended due to extra time required to set up data entry forms, and necessity of multiple data exports for analysis. 88

89 Analysis Staff skills required Managers Enumerators Data entry Farmer availability and willingness to participate Analysis completed using Microsoft Excel and SPSS. Indicators calculated using means of estimated livestock weight for animals 3 months, 6 months and 12 months of age. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production and productivity concepts. Local knowledge and relevant languages. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using means of livestock weight and potential proxy indicators of girth, shoulder height measurements and body condition score for female and male sheep and goats. Knowledge of enumeration, livestock measurement, and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity concepts relating to the data collection. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production and productivity concepts. Training in appropriate measurement of livestock weights and dimensions, and livestock handling. Knowledge of livestock sufficient to estimate body condition score). Local knowledge and relevant languages. Training required in Microsoft Access database creation and/or usage for data entry. Attention to detail and accuracy. Botswanan team members agreed that the data collection demonstrated the potential for farmers to be involved in more formalised and regular livestock data collection. Botswanan team members had difficulty accessing some farmers due to them being off-site or otherwise busy. Nonetheless, they agreed that the project demonstrated the potential for farmers to be involved in more formalised and regular livestock data collection. It would be useful for future alternative questionnaires to expand the data collection to fully grown animals/those over 12 months of age, providing a sample of animals of known age could be identified. Animal breed should also be included to allow differences of estimated weight at different age categories to be given. Low-cost proxy measures can be used by farmers to estimate weights and growth rates. A questionnaire requires closer attention to detail and good understanding of production and productivity factors by managers. A questionnaire requires closer attention to detail and good understanding of production and productivity factors by enumerators GS data require considerably more time on the part of the enumerators, therefore adding significant cost Identical skillset required A and GS (assuming that identical software is used for data entry, e.g. Excel) Farmers appear willing to take part, however it is important to schedule mutually suitable times. Given this difficulty in locating farmers, the participation of farmers offers an improved methodology. 89

90 Farmer understanding of data requirements Influence on farmer practices Data quality and usefulness Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Botswanan team members agreed that farmers understood the questions, and were able to answer them clearly. Botswanan team members were of strong agreement that the questionnaire and data collection demonstrated the importance and usefulness to farmers of keeping more detailed production records. It also provided the farers with a method for estimating indicators of direct commercial significance, such as liveweight and growth rates. Considered more relevant than E data (2 x Considered more relevant than strongly agree, 1 x agree ) by Botswanan questionnaire data (2 x agree, 1 x team members. neutral ). Considered more accurate and reliable than E data (1 x strongly agree, 2 x agree ). Botswanan team members noted that data could be gathered in a similar amount of time as E data (2 x strongly agree, 1 x unsure ). Agreement that A data more coherent than E data, and more comparable with other sources (1 x strongly agree, 2 x neutral ). N/A. Team members consider more accurate and reliable than questionnaire data (2 x strongly agree, 1 x agree ). Team members agreed (x 3) that the GS data warranted the extra time required for collection. Agreement (x 3) that the GS data were coherent and comparable with other data sources. Agreement that GS data was clear and easily accessible for analysis (2 x strongly agree, 1 x agree ). Few farmers had no experience with scales, and farmers weight estimation was not previously calibrated. The A questionnaire made farmers aware of the importance of record keeping to enhance their productivity and profitability. The A data set was considered by Botswanan team members to be of a high quality, though the extent to which farmers are able to recall the weight of livestock of different age categories may be disputed given their presumed lack of access to measuring equipment, and lack of record keeping. Indicator Average body weight x age category of livestock Average body weight x sex of goats or sheep Average girth measurement x sex of goats or sheep Average shoulder height measurement x sex of goats or sheep Average body condition score x sex of goats or sheep Source Alternative Gold standard Gold standard Gold standard Gold standard 90

91 Budget Recommendations Sampling design and estimation procedures Cluster-based sampling offers cost advantages where intensive work is done on numbers at each household. TEST procedure collected the data to provide the basis for use of proxy measures of bodyweight in young stock. No attempt had been made before to measure animal liveweight as a measure of productivity. Based on on-going analysis by Botswana partners, relationships between bodyweight, girth and shoulder height (from Gold Standard) for sheep and goats can be evaluated to enable a proxy measure of liveweight. Despite the high cost of the underlying GS study, Botswana partners view is that the dat provides good value for money. Example correlations of girth measurement and shoulder height (cm) to body weight, GS, goats Challenges encountered during collection of the data sets In-country partners reported overruns in time and difficulties with transport logistics (despite the clustered nature of the samples). Lessons learned, including feasibility of tested methods for wider-scale implementation Given the general lack of experience with animals weights eye-assessment, widespread absence of scales for weighing animals, and paucity of record-keeping by farmers, the establishment and use of methods and facility for the proxy measurement for liveweight marks an important advance despite its high cost. Analysis suggests that girth measurement and shoulder height have the potential to act as reliable proxies for body weight, however body condition score is not a reliable proxy. There is potential to implement data collection processes requiring the farmer and/or enumerator to take girth and shoulder height measurements, and using a formula (which would require development) to provide an estimate of animal weight from these measures (see graphs to the left for goats as examples). This would also provide an advance for farmer record-keeping. Cost estimates currently assume that survey-type enumerators collect the data using proxy measures. Substantial cost savings are available if farmers do this work themselves. 91

92 9.3. CHANGES IN BODY WEIGHT OF SHEEP AND GOATS, BOTSWANA Criteria Gold Standard Conclusions Methodology used Direct measurement and weighing of selected animals owned by households in the survey sample, and estimation of body condition score, at two points in time. Sampling frame and design Data collection period 9/9/15 19/11/15 Data entry and processing Analysis Staff skills required Managers Enumerators Data entry Selection of households from the same sampling frame as used for the E and A questionnaires. Animals at 61 households measured across the three districts (1,590 goats, 680 sheep measured). Enumerators collected information on paper forms. Data entered into computer in-country using Microsoft Excel, with a separate worksheet for each district, and each household entered in a separate section of each worksheet. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using changes in means of livestock weight and potential proxy indicators of girth, shoulder height measurements and body condition score for female and male sheep and goats. Knowledge of enumeration, livestock measurement, and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity concepts relating to the data collection. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production and productivity concepts. Training in appropriate measurement of livestock weights and dimensions, and livestock handling. Knowledge of livestock sufficient to estimate body condition score). Local knowledge and relevant languages. Training required in Microsoft Excel spreadsheet creation and/or usage for data entry. Attention to detail and accuracy. GS data is prone to significant additional time and cost for collection. A and E questionnaires featured just one interview so did not allow an estimate of weight difference over time. Microsoft Excel form can be used more efficiently for data entry by using a single worksheet for all districts and households, and having no separation on the basis of household. This will considerably simplify data import for analysis, without having an impact on data entry. It would be useful for future alternative questionnaires to expand the data collection to fully grown animals/those over 12 months of age. Animal breed should also be included to allow differences of estimated weight at different age categories to be given. Two estimates of liveweight on individual animals, over a specified time interval, would enable a measure of growth rate. Considerable extra time will be required of managers in collecting GS data, including additional training and supervision, and potentially scheduling measurement times convenient to farmers and enumerators. The GS data will require considerably more time on the part of the enumerators, therefore adding significant cost. It will also require a practical skillset (e.g. cattle handling skills), in addition to knowledge of the requirements and importance of accurate measurement. Microsoft Excel skillset adequate. 92

93 Farmer availability and willingness to participate Farmer understanding of data requirements Influence on farmer practices Data quality and usefulness Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Botswanan team members had difficulty accessing some farmers due to them being of-site or otherwise busy. Nonetheless, they agreed that the project demonstrated the potential for farmers to be involved in more formalised and regular livestock data collection. Botswanan team members were of strong agreement that the data collection demonstrated the importance and usefulness to farmers of keeping more detailed production records. Considered more relevant than questionnaire data (2 x agree, 1 x neutral ). Team members consider more accurate and reliable than questionnaire data (2 x strongly agree, 1 x agree ). Team members agreed (x 3) that the GS data warranted the extra time required for collection. Agreement (x 3) that the GS data were coherent and comparable with other data sources. Agreement that GS data was clear and easily accessible for analysis (2 x strongly agree, 1 x agree ). Farmers appear willing to take part, however it is important to schedule mutually suitable times. Bear in mind for the A questionnaire that many farmers may have had no experience with scales, and therefore lack understanding of how to estimate weight with any accuracy. GS data provides an important and often previously untapped source of management information for farmers who do not historically monitor livestock growth rates. GS data provide a highly accurate form of livestock productivity measurement, however these are hampered by considerable costs and impracticalities such as timing, to be implemented at a broad scale. Indicator Changes in body weight x sex of goats or sheep Changes in girth measurement x sex of goats or sheep Changes in shoulder height measurement x sex of goats or sheep Proportion of animals with each body condition score x sex of goats or sheep Source Gold standard Gold standard Gold standard Gold standard 93

94 Budget Recommendations Sampling design and estimation procedures TEST procedure used clustered sampling, which is appropriate for methods involving handling multiple animals at each household. TEST procedure measured weights on two occasions so as to estimate growth rates on an interval. Application of this work is in using proxy measures of liveweight as above, over a time interval. Challenges encountered during collection of the data sets In view of the constraint reported, associated with non-availability of farm staff at the time of a survey interview, the use of self-assessment using proxy measures shows much promise. Lessons learned, including feasibility of tested methods for wider-scale implementation As above, analysis suggests that girth measurement and shoulder height have the potential to act as reliable proxies for body weight, however body condition score is not a reliable proxy. There is potential to implement data collection processes requiring the farmer and/or enumerator to take girth and shoulder height measurements, (analysis is on-going in Botswana) to provide an estimate of animal weight. Cost estimates currently assume that survey-type enumerators collect the data using proxy measures. Substantial cost savings are available if farmers do this work themselves. Recording and understanding growth rate offers substantial insight to farmers, who it is widely acknowledged do not accumulate or make use of management information. 94

95 9.4. FEED PRODUCTION AND PURCHASE, BOTSWANA Criteria Existing Q Alternative Q Conclusions Methodology used Sampling frame and design Face-to-face interview, using questionnaire form and reliant on participant recall. Random selection of households using sample frame of district extension lists for three districts: Mahalapye, Tshane/Kgalihadi, Kweneng/Molopole. Same sample as goats and sheep survey used. 40 households selected each location; supplemented with neighbours where response not obtained E = 20 responses from each region (half of sample) Data collection period 7/9/15 8/11/15 7/9/15 18/10/15 Data entry and processing Analysis Enumerators collected information on paper forms. Data entered into computer in-country using Microsoft Excel, using a separate file for each question. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using means of variables relating to additions to crop plantings and purchase of different feed types. Face-to-face interview, using more extensive questionnaire form and reliant on participant recall. As per Existing Q A = 20 responses from each region (other half of sample) Enumerators collected information on paper forms. Data entered into computer incountry using Microsoft Excel, using a separate file for each question. Botswanan team members believed the A data were easier to understand and process than the E data. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using means of variables relating to additions to specific crop plantings, and purchase of different feed types. Botswanan team members believed the A data were easier to interpret for calculation of indicators than the E data. Usage of participant recall more problematic in the case of the alternative questionnaire as considerably more detail required from participants data richer but more prone to recall bias. Sampling approach identical for both E and A Microsoft Excel form can be used more efficiently for data entry by using a single worksheet for all questions. This will considerably simplify data import for analysis, without having an impact on data entry. E questionnaire lacked information on specific feed crops grown by the households. A questionnaire may be improved further by asking for details on the last 12 months. A questionnaire allowed more detailed indicators using data on specific feed crop types. 95

96 Staff skills required Managers Enumerators Data entry Farmer availability and willingness to participate Farmer understanding of data requirements Influence on farmer practices Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity and livestock feed options and concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production concepts including the different types of purchased/stored feed options available, and area planted to feed crops. Local knowledge and relevant languages. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity and livestock feed options and concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production concepts including the different types of purchased/stored feed options available. Local knowledge and relevant languages. Training required in Microsoft Excel spreadsheet creation and/or usage for data entry. Attention to detail and accuracy. Botswanan team members agreed that the questionnaire work demonstrated the potential for farmers to be involved in more formalised and regular livestock data collection. Botswanan team members agreed that farmers understood the questions, and were able to answer them clearly. Botswanan team members were of strong agreement that the questionnaire-based data collection demonstrated the importance and usefulness to farmers of keeping more detailed production records. The A questionnaire is considerably more detailed in the information it seeks on feed usage quantities, types, and consumption by different types of livestock. It therefore requires closer attention to detail and greater understanding of production and productivity factors by managers. The A questionnaire is considerably more detailed in the information it seeks on feed production and usage quantities, types, and consumption by different types of livestock and number of days of feeding. It therefore requires closer attention to detail and greater understanding of production and productivity factors. Identical skillset required E and A. Farmers appear willing to take part in questionnaire-based livestock data collection, particularly if they can see the benefits for their own farm s management. By including more detailed information, the A questionnaire in particular made farmers aware of the importance of record keeping to enhance their productivity and profitability. 96

97 Data quality and usefulness Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Not assessed. Not assessed. Not assessed. Not assessed. Not assessed. Considered more relevant than E data (2 x strongly agree, 1 x agree ) by Botswanan team members. Considered more accurate and reliable than E data (1 x strongly agree, 2 x agree ) Botswanan team members noted that data could be gathered in a similar amount of time as E data (2 x strongly agree, 1 x unsure ) Agreement that A data more coherent than E data, and more comparable with other sources (1 x strongly agree, 2 x agree ) N/A Consultation with Botswanan team members suggests strongly that the A data are of a higher quality than E data, assessing them per the FAO criteria. The team members also agreed that the A questionnaire met the goal of gold standard measurement to a greater degree than the E questionnaire. Indicator Total area sown to (unspecified) feed crops over the last agricultural year Average total area sown to specific feed crops each year Quantity of specific livestock feeds purchased annually (kg) Source Existing Alternative Existing, Alternative 97

98 Budget Recommendations Sampling design and estimation procedures The main difference between the A and E methods is the detail of questions and some enhanced coherence between them. Farmer recall is retained. Improved data on livestock smallholders feed supplies is suitable for integration with oter data such as that on crops, land use, water management and climate. If integrated with administrative boundaries, this offers opportunities for higher level management surrounding overgrazing. Challenges encountered during collection of the data sets Botswanan team members believed that the E questionnaire did not represent good value for money, but that the A questionnaire did. Note however that the marginal additional time required to collect and enter an A questionnaire, compared to an E questionnaire, multiplies to a more significant overall cost where a large-scale survey is being conducted. The in-country team reported that challenges included: insufficient time onsite; large transport distances; and difficulty in finding farmers at their household to conduct interviews. Greater than planned for time was therefore required in the field. It was suggested that farmers need to be forewarned of a survey being conducted, and a time booked to ensure their availability on the day. Lessons learned, including feasibility of tested methods for wider-scale implementation The Alternative Questionnaire provides more detail on specific crops grown rather than just unspecified feed crops, as addressed by the Existing Questionnaire. If applied at a wider scale, this question would allow regional differences in type/s of feed crops to be identified, as well as those crops most likely to be planted by farmers specialising in different types of livestock. It would be feasible to use this question rather than the existing non-specific question without requiring too much additional time from farmers or enumerators, though the potential for the data to become less reliable due to reliance on farmer recall rather than record keeping needs to be addressed. A method adds significantly (14%) to data collection costs, and although perceived as good value for money, is still reliant on farmer recall. 98

99 9.5. FEED USAGE AND PASTURE DEGRADATION, BOTSWANA Criteria Existing Q Alternative Q Gold Standard Conclusions Methodology used Sampling frame and design Face-to-face interview, using questionnaire form and reliant on participant recall. Random selection of households using sample frame of district extension lists for three districts: Mahalapye, Tshane/Kgalihad, Kweneng/Molopole. Same sample as goats and sheep survey used. 40 households selected each location; supplemented with neighbours where response not obtained. E = 20 responses from each region (half of sample). Face-to-face interview, using more extensive questionnaire form and reliant on participant recall. As per Existing Q A = 20 responses from each region (other half of sample). Transect-sampling to assess extent, distribution and severity of overgrazing of communal pastures. Wheelpoint method used to collect data over 100 mete transects. Nested quadrats at the end of each transect established to measure herbaceous biomass and woody species counts. 244 sites sampled (7 Central; 53 Kgalagadi; 119 Kweneng). Data collection period 7/9/15 8/11/15 7/9/15 18/10/15 November 2015 Enumerators collected information on paper forms. Enumerators collected information on Data entered into computer incountry using Microsoft Excel, paper forms. Data entered into Data entry and processing computer in-country using Microsoft using a separate file for each Excel, using a separate file for each question. Botswanan team question. members believed the A data were easier to understand and process than the E data. Data entered into computer in-country using Microsoft Excel, with a separate worksheet for herbaceous and woody vegetation counts, and herbaceous biomass. Usage of participant recall more problematic for A questionnaire as considerably more detail required from participants data richer but more prone to recall bias. GS data collection time consuming. Sampling approach identical for both E and A Data samples of A and GS not directly comparable given differences in methods and measurements, but A considered a good proxy for GS given farmer experience and knowledge of the pastures they use. Microsoft Excel form can be used more efficiently for data entry by using a single worksheet for all questions. This will considerably simplify data import for analysis, without having an impact on data entry. 99

100 Analysis Staff skills required Managers Enumerators Data entry Analysis completed using Microsoft Excel and SPSS. Indicator calculated as percentage of respondents using different feed types. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity and livestock feed options and concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production concepts including usage of different feed options for different types of livestock. Local knowledge and relevant languages. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using means of variables relating to days different grown and purchased/stored feeds and grazing land used for different livestock. Botswanan team members believed the A data were easier to interpret for calculation of indicators than the E data. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of livestock production and productivity and livestock feed options and concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding livestock production concepts including usage of different types of feed for cattle of various ages. Local knowledge and relevant languages. Analysis completed using Microsoft Excel and SPSS. Indicators calculated using frequencies of herbaceous and woody species present, averages of plants per district, average herbaceous biomass per district, and average woody species height. Knowledge of plant species counting techniques, plant species identification, ability to supervise and advise field team gathering species data. Local knowledge and relevant languages. Considerable knowledge of plant species identification (herbaceous and woody species), accepted techniques for transectsampling, plant species counting and biomass measurement, experiencing in assessing and judging grass palatability at sites. Training required in Microsoft Excel spreadsheet creation and/or usage for data entry. Attention to detail and accuracy. E questionnaire lacked information on usage of communal grazing land and planted feed crops that was included in the A questionnaire for the first time. E and A questionnaires comparable in terms of % of respondents using different purchased feed types. The A questionnaire is considerably more detailed in the information it seeks on feed usage quantities, types, and consumption by different types of livestock. It therefore requires closer attention to detail and greater understanding of production and productivity factors by managers. The A questionnaire is considerably more detailed in the information it seeks on feed production and usage quantities, types, and consumption by different types of livestock and number of days of feeding. Requires closer attention to detail and greater understanding of production and productivity factors. GS data collection requires agronomic expertise; skills not comparable. Identical skillset required E, A and GS. 100

101 Farmer availability and willingness to participate Farmer understanding of data requirements Influence on farmer practices Data quality and usefulness* Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Botswanan team members agreed that the project demonstrated the potential for farmers to be involved in more formalised and regular livestock data collection. Botswanan team members agreed that farmers understood the questions, and were able to answer them clearly. Botswanan team members were of strong agreement that the data collection demonstrated the importance and usefulness to farmers of keeping more detailed production records. Gap Analysis scores, Botswana relevance scores: feed intake 3.75 Gap Analysis scores, Botswana accuracy scores: feed intake 3.00 Gap Analysis scores, Botswana timeliness scores: feed intake 2.75 Gap Analysis scores, Botswana coherence scores: feed intake 3.00 Gap Analysis scores, Botswana accessibility scores: feed intake 3.50 Considered more relevant than E data (2 x strongly agree, 1 x agree ) by Botswanan team members. Considered more accurate and reliable than E data (1 x strongly agree, 2 x agree ) Botswanan team members noted that data could be gathered in a similar amount of time as E data (2 x strongly agree, 1 x unsure ) Agreement that A data more coherent than E data, and more comparable with other sources (1 x strongly agree, 2 x agree ) N/A N/A More relevant than questionnaire data (2 x agree, 1 x neutral ). More accurate and reliable than questionnaire data (2 x strongly agree, 1 x agree ). Team members agreed (x 3) that data warranted the extra time required for collection. Agreement (x 3) that data coherent and comparable with other data sources. Agreement that data clear and easily accessible for analysis (2 x strongly agree, 1 x agree ). Farmers appear willing to take part in questionnaire-based livestock data collection, particularly if they can see the benefits for their own farm s management. Farmers were made aware of the importance of record keeping to enhance productivity/profitability. Consultation with Botswanan team members suggests strongly that the A data are of a higher quality than E data, assessing them per the FAO criteria. The team members also agreed that the A questionnaire met the goal of gold standard measurement to a greater degree than the E questionnaire. A data on pasture degradation relies on farmer opinion/interpretation of pasture degradation on scales from none to severe, and is thus open to interpretation/variation. GS data of a very high quality but requires considerable time to collect, and requires a high level of expertise in plant species identification. * Gap analysis scores on a scale of 0 to 5: 0 being not available or not useable, and 5 being perfect. See separate Gap Analysis report for more details. 101

102 Indicator Proportion of farmers using various stock feed options to supplement the diet of cattle, sheep and goats Number of days purchased feed used for cattle, sheep and goats, by stock age categories Number of days annually each type of livestock uses particular feed crops and their residue categories Number of days annually different types of grazing area grazed by different types of livestock Rating of pasture degradation on a scale from None to Severe using the contributing factors of presence of Seloka Grass, and bush encroachment Composition and density of herbaceous species across different communal grazing sites Amount and quality of herbaceous biomass across different communal grazing sites Composition and density of woody species across different communal grazing sites Distribution of woody species of different heights across different communal grazing sites Source Existing Alternative Alternative Alternative Alternative Gold standard Gold standard Gold standard Gold standard Budget Recommendations Sampling design and estimation procedures Further work (including multi-disciplinary consultation) is necessary to establish an appropriate sampling strategy to establish relations between indicators species at the agronomic level and feed availability and production at the farm and communal grazing system level. Results of the TEST procedure are encouraging in that E method is recognised as being poor value for money and despite high cost of A method, its results are appreciated Challenges encountered during collection of the data sets further input required from Botswanan team? Botswanan team members believed that the E questionnaire did not represent good value for money, but that the A questionnaire did. The in-country team reported challenges such as: insufficient time on-site; large transport distances; and difficulty in finding farmers at their household to conduct interviews. Greater than planned for time was therefore required in the field. Specialist input will be needed to use the proxy measures (currently being developed in Botswana) and this contributes significantly to costs. 102

103 Lessons learned, including feasibility of tested methods for wider-scale implementation The A Questionnaire provides considerably more detail on feed usage than the Existing Questionnaire, including new data on usage of feed crops and grazing pastures, and the number of days these were used for livestock of different types and of different age categories. The Alternative method therefore have much greater potential to reveal more specific livestock feeding patterns. The data set could potentially be supplemented further with data about seasonal variation in livestock grazing and feed usage, though this would be a significant leap in the level of detail required of farmers, even greater attention to detail on the part of enumerators. Once again, the implications of reliance on farmer recall rather than record keeping need to be considered when evaluating the validity of the Alternative Questionnaire data in particular. A method provides substantial insight into the availability and use of communal grazing as a feed resource. As there is currently so little information available on this subject, any advance will be considered worthwhile. Training represents a substantial part of the costs of adoption of proxies of pasture feed measurement. Extending this capacities to farmers represents a significant challenge which is also a high cost activity. 103

104 9.6. EGG PRODUCTION, TANZANIA Criteria Existing Q Alternative Q Gold Standard Conclusions Methodology used Sampling frame and design Face-to-face interview, using questionnaire form and reliant on participant recall. Random selection of households using sample frame of households in two districts: Morogoro and Dodoma. Approximately 40 households chosen in Morogoro, and approximately 30 in Dodoma; supplemented with neighbours where response not obtained. E = 67 responses across 2 regions Face-to-face interview, using more extensive questionnaire form and reliant on participant recall. As per Existing Q A = 68 responses across two regions. Direct measurement and collection of data on hen laying activity for selected households and chickens on a daily basis. Selection of households from the same sampling frame as used for the E and A questionnaires. Animals at 127 households measured across the two districts (354 observed hens laid one or more eggs during the data collection period). Data collection period Approx. 25/8/15 31/8/15 Approx. 25/8/15 31/8/15 25/8/15 9/10/15 Enumerators collected Enumerators information on paper forms. As per Existing Q. One Data entered into computer Tanzanian team member in-country using Microsoft Data entry and processing indicated that the A data Excel, using a separate were easier to understand and worksheet for E and A process than the E data. worksheet questionnaires, separate file for each region. collected information on paper forms. Data entered into computer in-country using Microsoft Excel, with a separate template established for each district. Usage of participant recall problematic for A questionnaire since more detail required from participants data richer but more prone to recall bias. GS data richer still, but expensive to collect. Sampling approach identical for both E and A, and use of same sample for GS data facilitates comparison of egg production indicators across three data collection approaches. Using a separate data entry form for each region resulted in considerable time spent on integration of GS data before analysis, given that each regional team customised their form slightly. 104

105 Analysis Staff skills required Managers Enumerators Data entry Farmer availability and willingness to participate Farmer understanding of data requirements Analysis completed using Microsoft Excel and SPSS. Indicators calculated as an average of the sum of annual egg production per farm. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of poultry production and productivity concepts relating to the survey. Local knowledge and relevant languages. Analysis completed using Microsoft Excel and SPSS. Indicators calculated as an average of the sum of annual egg production per farm and per hen. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of poultry production and productivity concepts relating to the survey. Local knowledge and relevant languages. Analysis completed using Microsoft Excel and SPSS. Indicators calculated as an average of the sum of annual egg production per farm. Knowledge of enumeration, livestock measurement, and data entry processes. Ability to lead and supervise enumerator team. Good understanding of poultry production and productivity concepts relating to the data collection. Local knowledge and relevant languages. Training in interviewing and Training in interviewing and Training in interviewing and collection of data. collection of data. collection of data. Understanding poultry Understanding of poultry Understanding poultry production concepts, production concepts. Local production and management particularly clutches. Local knowledge and relevant concepts. Local knowledge knowledge and relevant languages. and relevant languages. languages. Training required in Microsoft Excel database creation and/or usage for data entry. Attention to detail and accuracy. Tanzanian team members agreed that the questionnaires and GS data activities demonstrated the potential for farmers to be involved in more formalised and regular livestock data collection. Tanzanian team members agreed that farmers understood the questions, and were able to answer them clearly. N/A Both A and GS data allowed for more detailed calculation of estimated production per hen rather than per farm (A from farmer recall, GS from data collected over a selected number of days). Greater scope than E. The A questionnaire is considerably more detailed, and requires closer attention to detail and greater understanding of production and productivity factors by managers. The A questionnaire and GS data collection involve more specific information, requiring closer attention to detail and greater understanding of production and productivity factors by enumerators. Identical skillset required E, A and GS. Farmers appear willing to take part in questionnaire-based poultry data collection, particularly if they can see the benefits for their own flock management. 105

106 Influence on farmer practices Data quality and usefulness* Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Tanzanian team members were of strong agreement that the data collection activities demonstrated the importance and usefulness to farmers of keeping more detailed production records. Specific influences mentioned by the Tanzanian team included: demonstrating the importance of accurate measurement rather than (often inaccurate) estimation; demonstrating the importance of vaccination; providing data for flock selection and improvement; and demonstrating the importance of record keeping. Gap Analysis scores, Tanzania relevance scores: no. eggs produced/collected 3.00 Gap Analysis scores, Tanzania accuracy scores: no. eggs produced/collected 3.00 Gap Analysis scores, Tanzania timeliness scores: no. eggs produced/collected 2.20 Gap Analysis scores, Tanzania coherence scores: no. eggs produced/collected 2.40 Gap Analysis scores, Tanzania accessibility scores: no. eggs produced/collected 2.80 Considered more relevant than E data (1 x strongly agree, 1 x agree ) by Tanzanian team members. Considered more accurate and reliable than E data (1 x agree ). Disagreement that data could be gathered in a similar amount of time as E data (1 x strongly disagree, 1 x disagree ) Ambivalence that A data more coherent than E data, and more comparable with other sources (1 x strongly disagree, 1 x agree ) N/A Strong agreement that GS more relevant than questionnaire data (2 x strongly agree ). Considered more accurate and reliable than questionnaire data (1 x strongly agree, 1 x agree ). Agreement (x 2) that the GS data warranted the extra time required for collection. Strong disagreement (x 2) that the GS data were coherent and comparable with other data sources. Strong agreement that GS data was clear and easily accessible for analysis (2 x strongly agree ). The A questionnaire and GS data made farmers aware of the importance of record keeping to enhance their productivity, profitability and flock management. Consultation with Tanzanian team members suggests strongly that the A data are of a higher quality than E data, assessing them per the FAO criteria. The team members agreed that the A questionnaire met the goal of gold standard measurement to a greater degree than the E questionnaire. It is notable that the team members from Tanzania strongly disagreed that GS data collected for this project was coherent and comparable with other similar sources and data sets, but strongly agreed that the data were accessible for analysis.gs data re also considered the most accurate, because there is no potential to cheat (through difficulty in providing estimation on the basis of recall, vs direct measurement) * Gap analysis scores on a scale of 0 to 5: 0 being not available or not useable, and 5 being perfect. See separate Gap Analysis report for more details. 106

107 Indicator Eggs produced on farm over 12 month period: Number of eggs Sep + Number of eggs Oct + + Number of eggs Aug Eggs produced on farm over 12 month period: Number of hens laying eggs last 12 months x Average number of clutches per hen x Average number of eggs per clutch Eggs produced per hen over 12 month period: (Number of eggs counted / number of days data collection carried out for this hen) x 365 days Eggs produced per hen over 12 month period: Average number of clutches per hen x Average number of eggs per clutch. Source Existing Alternative Gold standard Alternative Budget Recommendations Sampling design and estimation procedures Sampling for the TEST phase included isolated sites which possibly had somewhat specific characteristics, and certainly faced specific climatic and disease threats. This suggests that an unbiased sample would include a large number of sample sites, rather than a large sample size per se. A method offers a more coherent and simple way of measuring egg production variables, and converting them to useful productivity indicators. Costs associated with implementing the A method are small. The Tanzanian team were ambivalent about whether the E and A questionnaires represented good value for money, but agreed strongly that the GS data were good value. Challenges encountered during collection of the data sets Farmers were responsible for collection of the data, possibly introducing respondent bias.some outliers were evident in the A data. These suggests that close attention is required to selection and training of enumerators, ensuring that their knowledge of poultry production is sufficient to identify such issues as data are being collected. Because two teams were in operation across the two pilot study locations, some inconsistency emerged in collation and entry of data. This necessitated considerable extra time in integrating the data into a single set and data cleaning. Poultry disease outbreaks occurred in the sample areas, limiting the numbers of hens continuously enrolled. There were instances of farmers overstating their numbers of hens and/or egg production so as to be sure of being included in the survey. This resulted in smaller than expected samples. 107

108 Lessons learned, including feasibility of tested methods for wider-scale implementation A questionnaire allowed calculation of key indicators by way of several combinations or variables. The Tanzanian team have indicated that the A questionnaire took more time to complete in the field than the E questionnaire (reflected in the cost estimate). The E questionnaire relies on a greater degree than the A questionnaire on respondent recall, given that it includes estimates of egg production for each month rather than annually. The A questionnaire was considered by the Tanzanian team to be clearer and more logical, and the data are also considered more accurate and more relevant. This suggests strongly that they are worth the extra time required to collect. The data collection exercise would be well advised to be confer with specialists familiar with household egg production and flock/clutch management prior to data collection. There is no necessity for specialists to be on hand for data collection, but early training (see reflected in costs) will familiarise enumerators with outcomes reported. 108

109 9.7. INFLUENCES ON EGG PRODUCTION, TANZANIA Criteria Existing Q Alternative Q/Communal Q Gold Standard Conclusions Methodology used Face-to-face interview, using questionnaire form and reliant on participant recall. Face-to-face interview, using more extensive questionnaire form and reliant on participant recall. Group-based questioning; proportional piling technique for C response. Direct measurement and collection of data on hen laying activity for selected households and chickens on a daily basis. Random selection of households Sampling frame and design using sample frame of Selection of households from the households in two districts: As per Existing Q same sampling frame as used for Morogoro and Dodoma. A = 68 responses across two the E and A questionnaires. Approximately 40 households regions. Animals at 127 households chosen in Morogoro, and C question addressed only to the measured across the two districts approximately 30 in Dodoma; Morogoro community. 301 (354 observed hens laid one or supplemented with neighbours instances in the communal more eggs during the data where response not obtained. questioning session. collection period). E = 67 responses across 2 regions Data collection period Approx. 25/8/15 31/8/15 Approx. 25/8/15 31/8/15 25/8/15 9/10/15 Enumerators collected Enumerators collected information on paper forms. As per Existing Q. One information on paper forms. Data entered into computer incountry Tanzanian team member Data entered into computer in- Data entry and processing using Microsoft Excel, indicated that the A data were country using Microsoft Excel, using a separate worksheet for E easier to understand and process with a separate worksheet and A questionnaires, separate than the E data. template established for each file for each region. district. Usage of participant recall problematic for A questionnaire since more detail required from participants data richer but more prone to recall bias. GS data richer still, but expensive to collect. Sampling approach identical for both E and A, and use of same sample for GS data facilitates comparison of egg production indicators across three data collection approaches. Using a separate data entry form for each region resulted in considerable time spent on integration of GS data before analysis, given that each regional team customised their form slightly. 109

110 Analysis Staff skills required Managers Enumerators Data entry Farmer availability and willingness to participate Farmer understanding of data requirements Analysis completed using Microsoft Excel and SPSS. Indicators calculated as percentage of eggs produced at different times of year. Analysis completed using Microsoft Excel and SPSS. Indicators calculated as an average of the sum of annual egg production per farm for different hen breeds. Percentage of eggs produced at different times of year for C question. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of poultry production and productivity concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding of poultry production concepts. Local knowledge and relevant languages. Proportional piling technique used in C question needed to be taught to enumerators during training. Analysis completed using Microsoft Excel and SPSS. Indicators calculated as an average of the sum of annual egg production per farm for different hen breeds. Knowledge of enumeration, livestock measurement, and data entry processes. Ability to lead and supervise enumerator team. Good understanding of poultry production and productivity concepts relating to the data collection. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding poultry production and management concepts. Local knowledge and relevant languages. Training required in Microsoft Excel database creation and/or usage for data entry. Attention to detail and accuracy. Farmers appear willing to take part in questionnaire-based poultry data collection, particularly if they can see the benefits for their own flock management. Tanzanian team members agreed that farmers understood the questions, and were able to answer them clearly. Compared to E, A data provide scope for enhanced analysis of egg production by breed, production per hen and farm, and clutch analysis. The A questionnaire only allowed respondents to list one chicken breed. Multiple breeds would also require multiple response estimates of egg production per breed and may be overly complex, but would allow more detail across respondent populations. The A questionnaire is considerably more detailed, and requires closer attention to detail and greater understanding of production and productivity factors by managers. The A questionnaire and GS data collection involve more specific information, requiring closer attention to detail and greater understanding of production and productivity factors by enumerators. Identical skillset required E, A, C and GS. 110

111 Influence on farmer practices Data quality and usefulness* Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Tanzanian team members were of strong agreement that the data collection activities demonstrated the importance and usefulness to farmers of keeping more detailed production records. Specific influences mentioned by the Tanzanian team included: demonstrating the importance of accurate measurement rather than (often inaccurate) estimation; demonstrating the importance of vaccination; providing data for flock selection and improvement; and demonstrating the importance of record keeping. Gap Analysis scores, Tanzania relevance scores: no. eggs produced/collected 3.00 Gap Analysis scores, Tanzania accuracy scores: no. eggs produced/collected 3.00 Gap Analysis scores, Tanzania timeliness scores: no. eggs produced/collected 2.20 Gap Analysis scores, Tanzania coherence scores: no. eggs produced/collected 2.40 Gap Analysis scores, Tanzania accessibility scores: no. eggs produced/collected 2.80 Considered more relevant than E data (1 x strongly agree, 1 x agree ) by Tanzanian team members. C question accessed indicators affecting the group (e.g. weather) Considered more accurate and reliable than E data (1 x agree ). C method reduces recall error and bias. Disagreement that data could be gathered in a similar amount of time as E data (1 x strongly disagree, 1 x disagree ) Ambivalence that A data more coherent than E data, and more comparable with other sources (1 x strongly disagree, 1 x agree ) N/A Strong agreement that GS more relevant than questionnaire data (2 x strongly agree ). Considered more accurate and reliable than questionnaire data (1 x strongly agree, 1 x agree ). Agreement (x 2) that the GS data warranted the extra time required for collection. Strong disagreement (x 2) that the GS data were coherent and comparable with other data sources. Strong agreement that GS data was clear and easily accessible for analysis (2 x strongly agree ). By including more detailed information, the A questionnaire in particular made farmers aware of the importance of record keeping to enhance their productivity, profitability and flock management. Consultation with Tanzanian team members suggests strongly that the A data are of a higher quality than E data, assessing them per the FAO criteria. The team members agreed that the A questionnaire met the goal of GS measurement to a greater degree than the E questionnaire. It is notable that the team members from Tanzania strongly disagreed that GS data collected for this project was coherent and comparable with other similar sources and data sets, but strongly agreed that the data were accessible for analysis. GS data re also considered the most accurate, because there is no potential to cheat (through difficulty in providing estimation on the basis of recall, vs direct measurement) * Gap analysis scores on a scale of 0 to 5: 0 being not available or not useable, and 5 being perfect. See separate Gap Analysis report for more details. 111

112 Indicator Times of year of high and low egg production Influence of hen breed on egg production Source Existing; Communal Alternative; Gold standard Budget Recommendations Sampling design and estimation procedures The Tanzanian team were ambivalent about whether the E and A questionnaires represented good value for money, but agreed strongly that the GS data were good value. The communal data were not evaluated in our questions asked of in-country research partners. Sampling for the TEST phase included isolated sites which possibly had somewhat specific characteristics, and certainly faced specific climatic and disease threats. This suggests that an unbiased sample would include a large number of sample sites, rather than a large sample size per se. The TEST phase engaged with several of these factors as they were reflected in management practices and breeds used. A method offers a more coherent and simple way of measuring egg production variables, and converting them to useful productivity indicators. Costs associated with implementing the A method are small. Challenges encountered during collection of the data sets Outliers Inconsistency of data recording between two teams in the field, requiring high costs of data cleaning. Lessons learned, including feasibility of tested methods for wider-scale implementation A questionnaire allowed calculation of key indicators by way of several combinations or variables. Moreover, inclusion of a number of flock-related variables enabled disaggregation of productivity measures into useful categories. The Tanzanian team have indicated that the A questionnaire took more time to complete in the field than the E questionnaire (reflected in the cost estimate). The A questionnaire relies to some extent on respondent recall, but is also subject to the checks associate with the coherence of questions. This offers a 112

113 quick check for an enumerator during an interview. As above, the data collection exercise would be well advised to be confer with specialists familiar with household egg production and flock/clutch management prior to data collection. There is no necessity for specialists to be on hand for data collection, but early training (see reflected in costs) will familiarise enumerators with outcomes reportedboth the A questionnaire and the GS data included a hen breed variable, and in both cases the data appeared to illustrate the greater productive capacity of certain breeds. As such, it is important that future poultry surveys of production and productivity include a poultry breed variable, to demonstrate to farmers the potential benefits of engaging in a flock improvement program. There was some correspondence between the E questionnaire and the communal research approach in identifying months of low and high egg production. However, notable differences in these results suggested that communal data are not an effective substitute for questionnaire-based data. 113

114 9.8. MILK PRODUCTION, TANZANIA Criteria Existing Q Alternative Q Gold Standard Conclusions Methodology used Face-to-face interview, using questionnaire form and reliant on participant recall. Face-to-face interview, using more extensive questionnaire form and reliant on participant recall. Direct measurement and collection of data on milk production activity for selected households and cows on a twice daily basis. Random selection of households Sampling frame and design using sample frame of Selection of households from the households in two districts: same sampling frame as used for Morogoro and Dodoma. the E and A questionnaires. Approximately 40 households As per Existing Q Animals at 143 households chosen in Morogoro, and A = 68 responses across two measured across the two districts approximately 30 in Dodoma; regions. (342 observed hens producing supplemented with neighbours milk during the data collection where response not obtained. period). E = 76 responses across 2 regions Data collection period Approx. 25/8/15 31/8/15 Approx. 25/8/15 31/8/15 25/8/15 17/9/15 Enumerators collected Enumerators collected information on paper forms. As per Existing Q. One information on paper forms. Data entered into computer incountry Tanzanian team member Data entered into computer in- Data entry and processing using Microsoft Excel, indicated that the A data were country using Microsoft Excel, using a separate worksheet for E easier to understand and process with a separate worksheet and A questionnaires, separate than the E data. template established for each file for each region. district. Analysis Analysis completed using Microsoft Excel and SPSS. Indicators calculated as average daily/annual milk production per cow or per farm. Analysis completed using Microsoft Excel and SPSS. Indicators calculated as average milk production per cow or per farm for indigenous and improved cows. Analysis completed using Microsoft Excel and SPSS. Indicator calculated as an average of the sum of morning and evening milk production for daily milk produced. Usage of participant recall problematic for A questionnaire since more detail required from participants data richer but more prone to recall bias. GS data richer still, but expensive to collect. Sampling approach identical for both E and A, and use of same sample for GS data facilitates comparison of milk production indicators across three data collection approaches. Using a separate data entry form for each region resulted in considerable time spent on integration of GS data before analysis, given that each regional team customised their form slightly. Compared to E, A data provide scope for enhanced analysis of milk production for indigenous and improved cows, and specifies lactation period in the farmer s recall-based thinking on milk production. 114

115 Staff skills required Managers Enumerators Data entry Farmer availability and willingness to participate Farmer understanding of data requirements Influence on farmer practices Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of milk production and productivity concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding of milk production concepts. Local knowledge and relevant languages. Proportional piling technique used in C question needed to be taught to enumerators during training. Knowledge of enumeration, livestock measurement, and data entry processes. Ability to lead and supervise enumerator team. Good understanding of milk production and productivity concepts relating to the data collection. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding milk production and management concepts. Local knowledge and relevant languages. Training required in Microsoft Excel database creation and/or usage for data entry. Attention to detail and accuracy. Farmers appear willing to take part in questionnaire-based milk data collection, particularly if they can see the benefits for their own herd management. Tanzanian team members agreed that farmers understood the questions, and were able to answer them clearly. Tanzanian team members were of strong agreement that the data collection activities demonstrated the importance and usefulness to farmers of keeping more detailed production records. Specific influences mentioned by the Tanzanian team included: demonstrating the importance of accurate measurement rather than (often inaccurate) estimation; demonstrating the importance of vaccination; providing data for herd selection and improvement; and demonstrating the importance of record keeping. The A questionnaire is considerably more detailed, and requires closer attention to detail and greater understanding of production and productivity factors by managers, including breed identification. The A questionnaire and GS data collection involve more specific information, requiring closer attention to detail and greater understanding of production and productivity factors by enumerators, including breed identification. Identical skillset required E, A, C and GS. By including more detailed information, the A questionnaire in particular made farmers aware of the importance of record keeping to enhance their productivity, profitability and herd management. 115

116 Data quality and usefulness* Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Gap Analysis scores, Tanzania relevance scores: milk production 2.85 Gap Analysis scores, Tanzania accuracy scores: milk production 3.15 Gap Analysis scores, Tanzania timeliness scores: milk production 2.92 Gap Analysis scores, Tanzania coherence scores: milk production 3.00 Gap Analysis scores, Tanzania accessibility scores: milk production 3.31 Considered more relevant than E data (1 x strongly agree, 1 x agree ) by Tanzanian team members. C question accessed indicators affecting the group (e.g. weather) Considered more accurate and reliable than E data (1 x agree ). C method reduces recall error and bias. Disagreement that data could be gathered in a similar amount of time as E data (1 x strongly disagree, 1 x disagree ) Ambivalence that A data more coherent than E data, and more comparable with other sources (1 x strongly disagree, 1 x agree ) N/A Strong agreement that GS more relevant than questionnaire data (2 x strongly agree ). Considered more accurate and reliable than questionnaire data (1 x strongly agree, 1 x agree ). Agreement (x 2) that the GS data warranted the extra time required for collection. Strong disagreement (x 2) that the GS data were coherent and comparable with other data sources. Strong agreement that GS data was clear and easily accessible for analysis (2 x strongly agree ). Consultation with Tanzanian team members suggests strongly that the A data are of a higher quality than E data, assessing them per the FAO criteria. The team members agreed that the A questionnaire met the goal of GS measurement to a greater degree than the E questionnaire. It is notable that the team members from Tanzania strongly disagreed that GS data collected for this project was coherent and comparable with other similar sources and data sets, but strongly agreed that the data were accessible for analysis. GS data re also considered the most accurate, because there is no potential to cheat (through difficulty in providing estimation on the basis of recall, vs direct measurement) * Gap analysis scores on a scale of 0 to 5: 0 being not available or not useable, and 5 being perfect. See separate Gap Analysis report for more details. 116

117 Indicator Daily milk production per cow Annual milk production per cow: Average milk production per cow per day x Average number of months cows milked for x 30 (days/month) Daily milk production per farm: Average milk production per cow per day x Number of cows milked in the last 12 months Annual milk production per farm: Average milk production per cow per day x Number of cows milked in the last 12 months x Average number of months cows milked for x 30 (days/month) Source Existing; Alternative; Gold standard Existing Existing Existing Budget Recommendations Sampling design and estimation procedures Sampling targeted arid and isolated production systems featuring dual beefdairy production. This is one of several dairy systems in Tanzania, so further progress on milk production measurement requires a much-improved sampling approach. It is the case that the GS milk production measures do not provide insight into specialist dairy lowland and coastal systems productivity. Milk production from the sample base encountered in the TEST study was however measured during the TEST phase, providing a valuable database for the development of proxy measures. To this end, on-going work in Tanzania is developing estimates of lactation curves which can be used for proxy measurement of milk production in the future. Challenges encountered during collection of the data sets Farmers were responsible for collection of the data, possibly introducing respondent bias. Outlier values, possibly due to farmer and enumerator inexperience with data norms. Respondent recall is required for calving dates of cows. Procedures for dealing with milk use by suckling calves were systematic, but limited in precision. 117

118 The Tanzanian team were ambivalent about whether the E and A questionnaires represented good value for money, but agreed strongly that the GS data were good value. Lessons learned, including feasibility of tested methods for wider-scale implementation The A questionnaire offers a considerable improvement in scope for analysis and choice of milk production and productivity indicators over the E questionnaire, with only a few additional recall-based questions, and including questions which it could be reasonably expected of farmers to be able to understand and answer. GS data suggested that daily milk production was significantly lower than the figures given in the E and A questionnaires by farmers. Claims were made that the collection period for GS data fell at a time of year reported by farmers to be a period of lower milk productivity, but the sampling strategy provided that cows at various stages of lactation were enrolled. Despite the high cost of changing to a proxy measure of milk production, a proxy-based method will deliver considerable benefits in terms of data quality. Lactation curves would need to be developed for several production systems, with the use of that of the current study being limited to certain production systems. 118

119 9.9. INFLUENCES ON MILK PRODUCTION, TANZANIA Criteria Existing Q Alternative Q/Communal Q Gold Standard Conclusions Methodology used Face-to-face interview, using questionnaire form and reliant on participant recall. Face-to-face interview, using more extensive questionnaire form and reliant on participant recall. Group-based questioning; proportional piling technique for C response. Direct measurement and collection of data on milk production activity for selected households and cows on a twice daily basis. Random selection of households Sampling frame and design using sample frame of Selection of households from the households in two districts: As per Existing Q same sampling frame as used for Morogoro and Dodoma. A = 68 responses across two the E and A questionnaires. Approximately 40 households regions. Animals at 143 households chosen in Morogoro, and C question addressed only to the measured across the two districts approximately 30 in Dodoma; Morogoro community. 300 (342 observed hens producing supplemented with neighbours instances in the communal milk during the data collection where response not obtained. questioning session. period). E = 76 responses across 2 regions Data collection period Approx. 25/8/15 31/8/15 Approx. 25/8/15 31/8/15 25/8/15 17/9/15 Enumerators collected Enumerators collected information on paper forms. As per Existing Q. One information on paper forms. Data entered into computer incountry Tanzanian team member Data entered into computer in- Data entry and processing using Microsoft Excel, indicated that the A data were country using Microsoft Excel, using a separate worksheet for E easier to understand and process with a separate worksheet and A questionnaires, separate than the E data. template established for each file for each region. district. Usage of participant recall problematic for A questionnaire since more detail required from participants data richer but more prone to recall bias. GS data richer still, but expensive to collect. Sampling approach identical for both E and A, and use of same sample for GS data facilitates comparison of milk production indicators across three data collection approaches. Using a separate data entry form for each region resulted in considerable time spent on integration of GS data before analysis, given that each regional team customised their form slightly. 119

120 Analysis Staff skills required Managers Enumerators Data entry Farmer availability and willingness to participate Farmer understanding of data requirements Analysis completed using Microsoft Excel and SPSS. Indicators calculated as percentage of respondents indicating high and low months of milk production on their farm. Analysis completed using Microsoft Excel and SPSS. Indicators calculated as average per lactation milk production per cow for indigenous and improved cows, and average daily production at different stages of the lactation. Percentage of high milk production at different times of year for C question. Knowledge of enumeration and data entry processes. Ability to lead and supervise enumerator team. Good understanding of milk production and productivity concepts relating to the survey. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding of milk production concepts. Local knowledge and relevant languages. Proportional piling technique used in C question needed to be taught to enumerators during training. Analysis completed using Microsoft Excel and SPSS. Indicator calculated as an average of morning and evening milk production for daily milk produced, including differences for indigenous and improved cows. Knowledge of enumeration, livestock measurement, and data entry processes. Ability to lead and supervise enumerator team. Good understanding of milk production and productivity concepts relating to the data collection. Local knowledge and relevant languages. Training in interviewing and collection of data. Understanding milk production and management concepts. Local knowledge and relevant languages. Training required in Microsoft Excel database creation and/or usage for data entry. Attention to detail and accuracy. Farmers appear willing to take part in questionnaire-based milk data collection, particularly if they can see the benefits for their own herd management. Tanzanian team members agreed that farmers understood the questions, and were able to answer clearly. Compared to E, A data provide scope for enhanced analysis of milk production for indigenous and improved cows, length of lactation period, declining production over the lactation period, and identifying months of high and low production through the C question. The A questionnaire is considerably more detailed, and requires closer attention to detail and greater understanding of production and productivity factors by managers, including breed identification and lactation periods. The A questionnaire and GS data collection involve more specific information, requiring closer attention to detail and greater understanding of production and productivity factors by enumerators, including breed ID/lactation. Identical skillset required E, A, C and GS. 120

121 Influence on farmer practices Data quality and usefulness* Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity Tanzanian team members were of strong agreement that the data collection activities demonstrated the importance and usefulness to farmers of keeping more detailed production records. Specific influences mentioned by the Tanzanian team included: demonstrating the importance of accurate measurement rather than (often inaccurate) estimation; demonstrating the importance of vaccination; providing data for herd selection and improvement; and demonstrating the importance of record keeping. Gap Analysis scores, Tanzania relevance scores: milk production 2.85 Gap Analysis scores, Tanzania accuracy scores: milk production 3.15 Gap Analysis scores, Tanzania timeliness scores: milk production 2.92 Gap Analysis scores, Tanzania coherence scores: milk production 3.00 Gap Analysis scores, Tanzania accessibility scores: milk production 3.31 Considered more relevant than E data (1 x strongly agree, 1 x agree ) by Tanzanian team members. C question accessed indicators affecting the group (e.g. weather) Considered more accurate and reliable than E data (1 x agree ). C method reduces recall error and bias. Disagreement that data could be gathered in a similar amount of time as E data (1 x strongly disagree, 1 x disagree ) Ambivalence that A data more coherent than E data, and more comparable with other sources (1 x strongly disagree, 1 x agree ) NOT ASKED OF ALT DATA; TBA FROM OTHER SOURCE? Strong agreement that GS more relevant than questionnaire data (2 x strongly agree ). Considered more accurate and reliable than questionnaire data (1 x strongly agree, 1 x agree ). Agreement (x 2) that the GS data warranted the extra time required for collection. Strong disagreement (x 2) that the GS data were coherent and comparable with other data sources. Strong agreement that GS data was clear and easily accessible for analysis (2 x strongly agree ). By including more detailed information, the A questionnaire in particular made farmers aware of the importance of record keeping to enhance their productivity, profitability and herd management. Consultation with Tanzanian team members suggests strongly that the A data are of a higher quality than E data, assessing them per the FAO criteria. The team members agreed that the A questionnaire met the goal of GS measurement to a greater degree than the E questionnaire. It is notable that the team members from Tanzania strongly disagreed that GS data collected for this project was coherent and comparable with other similar sources and data sets, but strongly agreed that the data were accessible for analysis. GS data re also considered the most accurate, because there is no potential to cheat (through difficulty in providing estimation on the basis of recall, vs direct measurement) * Gap analysis scores on a scale of 0 to 5: 0 being not available or not useable, and 5 being perfect. See separate Gap Analysis report for more details. 121

122 Indicator Influence of time of day on milk production Influence of time of year on milk production Influence of breed on milk production Quantity of milk produced per lactation per cow (indigenous and improved cows) Source Gold standard Existing; Communal Gold standard Alternative Budget Recommendations Sampling design and estimation procedures As above, sampling targeted arid and isolated production systems featuring dual beef-dairy production. This is one of several dairy systems in Tanzania, so further progress on milk production measurement requires a muchimproved sampling approach. It is the case that the GS milk production measures do not provide insight into specialist dairy lowland and coastal systems productivity. Some of these limitations are however addressed by the A method which draws on more coherent and practical information as expressed by farmers, to characterise production in light of the management systems in place. Challenges encountered during collection of the data sets Outlier values. Lessons learned, including feasibility of tested methods for wider-scale implementation The Tanzanian team were ambivalent about whether the E and A questionnaires represented good value for money, but agreed strongly that the GS data were good value. The A questionnaire offers a considerable improvement in scope for analysis and choice of milk production and productivity indicators over the E questionnaire, with only a few additional recall-based questions, and including questions which it could be reasonably expected of farmers to be able to understand and answer. Additional factors include cow breed, the length of lactation, and milk productivity at different stages of the lactation period. The resulting data have been used to demonstrate the influence of these factors on 122