Harmonized Database for Agricultural Incentives: Data Processing

Transcription

1 Harmonized Database for Agricultural Incentives: Data Processing Prepared by IFPRI for the IO Consortium for Measuring the Policy Environment for Agriculture, June

2 Table of Contents Harmonized Database... 3 MAFAP database overview... 3 Summary Information... 4 Data cleaning, harmonization and consolidation... 4 Procedure summary... 4 Detailed steps... 5 MAFAP Definitions... 7 MAFAP Issues... 7 OECD database overview Summary Information Data cleaning, harmonization and consolidation Procedure summary Detailed steps OECD Definitions OECD Issues World Bank database overview Summary Information Data cleaning, harmonization and consolidation Procedure summary Detailed steps World Bank Definitions World Bank Issues Table 1. Database Variables... 3 Table 2. MAPAP Dataset: Summary Indicators - Raw dataset... 4 Table 3. MAFAP Costs... 5 Table 4. Reference Price in MAFAP... 5 Table 5. Issues remaining and actions taken... 8 Table 6. Commodity harmonization for labeling purposes in the Database... 9 Table 7. Summary of data sources in FAOSTAT Table 8. Adjustment to Production for computing consumption Table 9. OECD Dataset: Summary Indicators Raw Dataset Table 10. World Bank Dataset: Summary Indicators - Raw dataset

3 Harmonized Database Harmonized database includes data from 3 sources; OECD, FAO-MAFAP, and World Bank. The main variables included in the database are listed in below Table 1. This report includes detailed explanations of the data processing conducted by IFPRI on these 3 databases. Table 1. Database Variables Variable Name MPS (Market Price Support) PSCT (Producer Single Commodity Transfer) PSE (Producer Support Estimate) NRP (Nominal Rate of Protection) Reference Price at Farm Gate Producer Price at Farm Gate Availability Ag Incentives Website Indicator Catalog Ag Incentives Website Indicator Catalog Ag Incentives Website Indicator Catalog Ag_incentives_NRP.xlsx file, Ag Incentives Website Indicator Catalog Ag_incentives_NRP.xlsx file Ag_incentives_NRP.xlsx file These variables and indicators are either collected or computed for all individual agricultural commodities, for an aggregate other commodity (NONMPS), and for agricultural sector (TOTAL). MAFAP database overview The version of MAFAP database used is MAFAP Prices Incentives Database for IFPRI 2015_02_06.xlsx. This file includes 9 countries and 27 Commodities for the period (some countries only have data for ). The background worksheet used from MAFAP database is Incentives_augmented. The MAFAP database includes agricultural incentives parameters based on prices and costs at different points in the market. Some values are reported as observed and some are reported as adjusted. MAFAP database methodology is described in Barreiro-Hurle, J. and Witwer, M. (2013) 1. Once the data set has been imported and processed, it is used to populate the Ag_incentives_NRP.xlsx file where further checks and computations are performed on the data as detailed below. All commodities have been mapped to an aggregate category of Animal Products, Fruits and Vegetables, Grains, Oilseeds and Products, and Other. Data for NONMPS and for TOTAL (agricultural sector) are also included, from computation based on MAFAP database, as described below. 1 MAFAP Methodological Implementation Guide: Volume I. Analysis of price incentives and disincentives. MAFAP Technical Notes Series, FAO, Rome, Italy. < ines_-_volume_i_-_price_incentives_-_final.pdf> 3

4 Summary Information Table 2 displays summary information regarding the filling rates of the initial dataset. Table 2. MAPAP Dataset: Summary Indicators - Raw dataset Total Period, all rows Total Period, excluding total and unindentified products all rows , excluding total and unindentified products Number of rows in the dataset Number of rows including a NRA computation % 100% 100% 100% Number of rows with exchange rate information % 100% 100% 100% Number of rows with production (quantity) % 99% 99% 99% Number of rows with Producer Price (LCU) % 100% 100% 100% Number of rows with Producer Price (US$) % 0% 0% 0% Number of rows with Border Price (US$) % 87% 87% 87% Number of rows with a least 1 Producer Price % 100% 100% 100% Number of rows with a least 1 Price % 100% 100% 100% After treatment, the final dataset has 581 rows for the period, including 581 rows for specific products. Of course, the final dataset has a 100% filling rate for all variables. Data cleaning, harmonization and consolidation Procedure summary The following iterative procedure was implemented: 1. Code harmonization 2. Reference Price Computation compilation of costs 3. Reference Price Computation exchange rate 4. Reference Price Computation formula 5. Production Quantity 6. Consumption Quantity 7. Trade Status 8. MPS computation 9. NRP computation 10. PSCT Computation 11. Variables for Sector TOTAL are computed 12. Variables for Sector NONMPS are computed 13. Harmonized unit descriptions 4

5

6 In seventh step, we defined Trade Status of a commodity in a country. Trade_Status column shows trade status based on MAFAP database Commodity Trade Status column. It shows up as exports if it was x, imports if it was m, and empty for Total and NONMPS sectors. In eight step, we computed Market Price Support (MPS) = (Producer Price at FG Reference Price at FG)*Production Quantity In ninth step, we computed Nominal Rate of Protection (NRP): (Producer Price at FG/ Reference Price FG)) In tenth step, for PSCT values, although public expenditures are included in an earlier version of MAFAP database MAFAP_Guillaume Pierre General Database (v021)_without non calculated NRAs (received July 10th from ).xlsx, these values are not included for computation of PSCT for commodities since the effective sectoral allocation is unclear. At this point, PSCT for MAFAP data base equals MPS value for individual commodities. In eleventh step, we computed variables for Sector TOTAL: As an aggregate, we define the Producer Price at FG equal to 1. Total production value at producer price (FG) is based on FAOSTAT (total production value in LCU). However several adjustments were made since the FAOSTAT aggregated value does not cover the same sector from one country to another due to missing price information, even when physical production occurs. We have implemented a procedure to fill these gaps and have level of production value comparable from one country to another. In addition, no production value for Uganda was available and the complete series were built based on physical production index and value information from a reference country (i.e. Tanzania). Production Quantity is total production value (since Producer price is equal to 1) The total MPS value is based on the homogeneity principle as assumed in the OECD PSE data base and implies that non covered products, independently of their trade status, are affected by the same price distortions, captured by the MPS, as the covered commodities. So, the MPS value, as a percentage of production value, is the same for the total agricultural sector as for the covered commodities. Reference Price = 1 - [Market Price Support (LCU) / Production Value at Farm Gate (VP)]. The total support (PSE) is computed as the sum of the MPS for the whole sector plus the effective budgetary payments (from national entity or donors, A, B, C and D categories 2 ) addressed to the farmers (MAFAP public expenditures data at the country level). In twelfth step, we computed variables for Sector NONMPS. Producer Price at FG is assumed to be 1. Reference Price equals Reference Price for TOTAL. Production Quantity is Production Value for NONMPS derived as = (MPS / (1-Reference Price). The production and MPS value of these NONMPS commodities are computed as a difference between the total production value, and those of the covered commodities. In thirteenth step, we harmonized unit descriptions. Price units are split into two cells (so FCFA/Tonne now becomes two cells, one for the physical unit (MT) and the other for the currency (XOF)). Physical units for quantities are also included separately. Please note that all prices are now in local currency, i.e. million LCU have been transformed to LCU. The same transformation is also done for physical units, i.e. thousand metric tons are now metric tons. 2 A. Production subsidies based on outputs, B. Input subsidies, C. Income support and D. Other 6

7 MAFAP Definitions Source column shows the main database name, i.e. MAFAP, OECD, or World Bank. Source_File shows the specific workbook from MAFAP database. Timestamp shows when file was downloaded. Country_Code is 3 digit ISO code. Country_Label is full country name. Year is the year of the indicator. Product_Code is FAO numeric code. Product_Label is FAO commodity label. Product_Category is names for aggregate sectors as explained above. Production Quantity is the level of production for each commodity. Physical Unit column reports the basic unit of the data. For example if the original data was in 000 tons, the column will show MT for metric tons. Likewise, it will convert the data associated with it to MT as well. Trade Status is generated based on MAFAP database definition. It shows up as exports if it was x, imports if it was m, and empty for Total and NONMPS sectors. Reference_Price (Farm Gate) LCU column contains sector specific data computed as described above. Producer price (farm gate) LCU column contains sector specific data populated from MAFAP database. Currency Unit column shows local currency using ISO currency alpha code. Physical Unit column shows final physical units. We computed Nominal Rate of Protection (NRP): (Producer Price at Farm Gate/ Reference Price at Farm Gate). Production Value (RP_FG) LCU column shows Reference_Price (Farm Gate) multiplied by Production Quantity. Production Value (PP_FG) LCU column shows Producer_Price (Farm Gate) multiplied by Production Quantity. MAFAP Issues Some computed indicators are different from MAFAP because of errors in the MAFAP dataset or because official exchange rates from WDI are different from official exchange rates in MAFAP. 7

8 Table 5. Issues remaining and actions taken Country Commodities Issues Ghana All commodities Exchange rate unit Changed from GHC/US$ to GHS/US$ Kenya Cassava 2013-blank Added from previous dataset Kenya French Beans Negative other costs from Border to PoC No action Kenya Dry Beans 2013-blank Added from previous dataset Kenya Tea Negative processing margins from PoC to fg for 2011 No action Mali Empty cells for Qt and Ql factor Replaced by 1 Mali Groundnuts Empty cells for Qt and Ql factor Replaced by 1 Malawi Sugar Empty cells for Qt and Ql factor Replaced by 1 No Point of Competition Price No action Cotton No Point of Competition Price No action Groundnuts No Point of Competition Price No action Tea No Point of Competition Price No action Malawi Tobacco Benchmark Price in LCU is the same as Benchmark Price in US$. The exchange rate is not used. Cannot reconcile reference prices and the NRP. Computed the Reference Price as Benchmark Price in US$ times the exchange rate Tanzania Cashew nuts with shell No Point of Competition Price ( ) No action Uganda Negative other costs from Border to PoC No action Cotton lint Negative processing margins from PoC to FG for 2010 No action All countries All commodities Production Source: FAOSTAT All countries All commodities Net trade Source: FAOSTAT All countries All commodities Stock variation Source: FAOSTAT All countries All commodities Consumption Computed 8

9 Table 6. Commodity harmonization for labeling purposes in the Database Commodity labels in countries Countries Common labels used in Database Haricot Beans Ethiopia Beans, dry Dry Beans Kenya Beans, dry French beans Kenya Beans, green Cashew nuts (raw) Mozambique Cashew nuts (with shell) Cashew nuts Tanzania Cashew nuts (with shell) Cashew nuts (processed) Mozambique Cashew nuts (without shell) Cassava GHA, KEN, MOZ, and UGA Cassava (fresh) Coffee ETH, TZA, and UGA Coffee (green) Cotton BFA, KEN, MWI, MLI, MOZ, and UGA Cotton (lint) but production refers to seed cotton Groundnuts MWI, and MLI Groundnut, with shell Milk Mali Cow milk Rice BFA, GHA, KEN, MLI, MOZ, TZA and UGA Rice (paddy) Sesame BFA and ETH Sesame seed Sugar Malawi Sugar cane 9

10 Stock Variation Trade Production Table 7. Summary of data sources in FAOSTAT Indicator Domain Years available Production for crops Code Production of Livestock Primary Code Commodity Balance (BC) Crops primary (depending on the equivalent Code 5510 country) Trade TP Trade and livestock products M Code X Code Trade TP Live animals M Code X Code Commodity Balance (BC) M Code X Code Commodity Balance Crops Primary Equivalent (BC) Code Commodity Balance Livestock and Fish (BC) Code

11 Table 8. Adjustment to Production for computing consumption Country Production commodity Adjusted production commodity Trade commodity Adjustment Consumption commodity BFA Seed cotton Cotton Lint Cotton Lint Cotton Lint KEN Seed cotton Cotton Lint Cotton Lint Cotton Lint MWI Seed cotton Cotton Lint Cotton Lint Cotton Lint MLI Seed cotton Cotton Lint Cotton Lint Cotton Lint MOZ Seed cotton Cotton Lint Cotton Lint Cotton Lint UGA Seed cotton Cotton Lint Cotton Lint Cotton Lint BFA GHA KEN MAL MOZ UGA TZA KEN MAL UGA Rice, paddy Rice, paddy Rice, paddy Rice, paddy Rice, paddy Rice, paddy Rice, paddy Sugar cane Sugar cane Sugar cane Sugar, Raw Equivalent Sugar, Raw Equivalent Sugar, Raw Equivalent Sugar, Raw Equivalent Sugar, Raw Equivalent Sugar, Raw Equivalent 11 FAO standard yield rate of 67% paddy to rice FAO standard yield rate of 67% paddy to rice FAO standard yield rate of 67% paddy to rice FAO standard yield rate of 67% paddy to rice FAO standard yield rate of 67% paddy to rice FAO standard yield rate of 67% paddy to rice FAO standard yield rate of 67% paddy to rice 10% yield rate of cane to raw equivalent 10% yield rate of cane to raw equivalent 10% yield rate of cane to raw equivalent Sugar, Raw Equivalent Sugar, Raw Equivalent Sugar, Raw Equivalent

12 MWI MLI BFA ETH MLI UGA Groundnuts, with shell Groundnuts, with shell groundnuts (shelled eq) groundnuts (shelled eq) groundnuts (shelled eq) groundnuts (shelled eq) groundnuts (shelled eq) groundnuts (shelled eq) 12

13 OECD database overview OECD database was created using the data from specified country files on OECD Producer and Consumer Support Estimates database available on URL < The country files included are Australia, Canada, Chile, European Union, Iceland, Israel, Japan, South Korea, Mexico, New Zealand, Norway, Switzerland, Turkey, United States, Brazil, China, Indonesia, Kazakhstan, Russia, South Africa, and Ukraine. The country files are from September 2014 country PSE data from the OECD website. The SourceFile column for each database shows the source for the data. Each of these country files contains data for various indicators from 1986 to the latest data year. Each indicator also has a description and a formula and unit variable. The database reads in data from 4 types of worksheets from OECD country files: < TOTAL>, < SCT GCT>, <Commodity SCT>, and <Commodity MPS>. Data on yearly exchange rates is read from an exchange rate file at the same URL titled monitoringexchange-rates-2014.xls. This data set contains exchange rates from 1986 to the latest year for OECD and related countries to US$. Data contained are exchange rates, and currency and country acronyms. Once the data sets have been imported and processed, they are merged and used to populate the Ag_incentives_NRP.xlsx file where further checks and computations are performed on the data as detailed below. All commodities have been mapped to an aggregate category of Animal Products, Fruits and Vegetables, Grains, Oilseeds and Products, and Other. Data for NONMPS and for TOTAL (agricultural sector) are also included, either from OECD database or by computation, as described below. Summary Information Table 10 displays summary information regarding the filling rates of the initial dataset. As it appears, the overall dataset, excluding aggregates, has Producer Price (LCU) for 91% of the rows (7284), while MPS has been computed for 91% of the rows (7331). Restricting the dataset to the period, used as the default period covered by the consolidated database, the filling rate is better (99% for Producer Price and 100% for MPS). 13

14 Table 9. OECD Dataset: Summary Indicators Raw Dataset Number of rows in the dataset Number of rows including a MPS computation Number of rows with exchange rate information Number of rows with production (quantity) Number of rows with Producer Price (LCU) Number of rows with Producer Price (US$) Number of rows with Border Price (US$) Number of rows with a least 1 Producer Price Number of rows with a least 1 Price Total Period ( ), all rows Total Period ( ), excluding total and unidentified products Total Period( ), all rows Total Period( ), excluding total and unidentified products % 91% 100% 100% % 100% 100% 100% % 91% 87% 99% % 91% 87% 99% % 0% 0% 0% % 0% 0% 0% % 91% 87% 99% % 91% 87% 99% After treatment, the final dataset AgPolicyIndicators_OECD.xlsx has 5868 rows for the period, including 5126 rows for specific products. This last figure should be compared to the 5126 rows for the similar definition in the initial dataset (see Table 109). This 100% filling rate is achieved by using Value of Production in place of Quantity of Production and setting Producer Price to 1 for certain commodities such as Flowers for EU and Fruits and Vegetables for ISR (detailed below). Of course, the final dataset has a 100% filling rate for all variables. 14

15 Data cleaning, harmonization and consolidation Procedure summary The following iterative procedure was implemented: 1. Data set restriction 2. Trade Status 3. Variables for Sector TOTAL are computed 4. Variables for Sector NONMPS are computed 5. NRP computation 6. Harmonized unit descriptions Detailed steps During the first step, we restricted the dataset to the period. In the second step, Trade_Status is generated based on the difference between Quantity Produced and Quantity Consumed, which if positive shows up as exports, imports if negative, and Non Tradable if neither. For all commodities for which we do not have a 'true' price (TOTAL and NONMPS), we applied specific assumptions. In the third step, for sector Total, Producer Price at FG is assumed to be 1. Production Quantity is Production Value reported in TOTAL worksheet. Reference Price = 1 - [ Market Price Support (LCU) / Production Value at Farm Gate (VP) ]. In the fourth step, for sector NONMPS, Producer Price at FG is assumed to be 1. Production Quantity is (Production Value for Total Sum of Production Value for reported Commodities). Reference Price = 1 - [ Market Price Support (LCU) / Production Value at Farm Gate (VP) ]. For sector NONMPS, PSCT value is set equal to MPS value. In fifth step, we computed Nominal Rate of Protection (NRP): (Producer Price at FG/ RP(FG)) In sixth step, price units are split into two cells (so AUD/t now becomes two cells, one for the physical unit (MT) and the other for the currency (AUD)). Physical units for quantities are also included separately. Please note that all prices are now in local currency, i.e. million LCU have been transformed to LCU. The same transformation is also done for physical units, i.e. thousand metric tons are now metric tons. OECD Definitions Source column shows the main database name, i.e. MAFAP, OECD, or World Bank. Source_File shows the specific workbook downloaded from the OECD PSE database. Timestamp shows when file was downloaded. Country_Code is 3 digit ISO code. Country_Label is full country name. Year is the year of the indicator. Product_Code is commodity code used by OECD. Product_Label is commodity name used by OECD. Product_Category is names for aggregate sectors as explained above. 15

16 Production Quantity is the level of production (QP) reported in the sector specific worksheet for each commodity. Physical Unit column reports the basic unit of the data. For example if the original data was in 000 tons, the column will show MT for metric tons. Likewise, it will convert the data associated with it to MT as well. Trade Status is generated based on the difference between Quantity Produced and Quantity Consumed in OECD worksheets. Formula is Net Trade = Net Exports = Production - Consumption. If this value is positive, trade status shows up as exports, and if negative it shows up as imports if negative, and Non Tradable if neither. Reference_Price (Farm Gate) LCU column contains sector specific data populated from Reference price (at farm gate) (RP) from the workbook (MPS sheets). Producer price (farm gate) LCU column contains sector specific data populated from Producer Price (at farm gate) (PP) from the workbook (MPS sheets). Currency Unit column shows local currency using ISO currency alpha code. Physical Unit column shows final physical units, We computed Nominal Rate of Protection (NRP): (Producer Price at Farm Gate/ Reference Price at Farm Gate) Production Value (RP_FG) LCU column shows Reference_Price (Farm Gate) multiplied by Production Quantity. Production Value (PP_FG) LCU column shows Producer_Price (Farm Gate) multiplied by Production Quantity. OECD Issues Some items that we have noticed are as follows. Methodology from the typology sheet (based on Table 4.1 Border prices and alternative methods used to derive commodity reference prices by country) is missing information for Canada Rapeseed and Ukraine Rye. Consumption price for Mexico Coffee in 1986 is OECD suggested deleting all data for Mexico prior to Australia Cotton Consumption for are 0. It is unclear whether these entries are blank or if consumption is zero. Value of Production for these entries are also 0. Both EU27 Flowers and Israel Fruits and Vegetables have Production Quantity and Producer Price at 0, but have positive Value of Production. For these commodities, we set Production Quantity to Value of Production. We assumed Producer Price at FG to be 1. We computed Reference Price at FG using the formula: Reference Price = 1 - [ Market Price Support (LCU) / Production Value at Farm Gate (VP) ]. For entries where reference price is missing (SP - Japan, MA-South Africa for some years): We computed Reference Price at FG using the formula: Reference Price = 1 - [ Market Price Support (LCU) / Quantity Produced (QP]. 16

17 World Bank database overview The World Bank Estimates of Distortions to Agricultural Incentives covered the period for 81 countries is based on Anderson and Valenzuela, and was updated June 2013 by Nelgen. The file is available on the World Bank website and is titled UpdatedDistortions_to_AgriculturalIncentives_database_0613.xls. It is an Excel workbook containing a main data table with a reference worksheet. This data file was processed first in Excel to clean main problems (country code, duplicated rows, missing ID) and then imported into GAMS to manipulate the data and proceed to different checks and computations. Additional information coming from online sources, namely the World Bank Development Indicators (exchange rate) and FAOSTAT (production value at the country level) were used to fill some gaps. Once the data set has been imported and processed, it is merged and used to populate the Ag_incentives_NRP.xlsx file where further checks and computations are performed on the data as detailed below. All commodities have been mapped to an aggregate category of Animal Products, Fruits and Vegetables, Grains, Oilseeds and Products, and Other. Data for NONMPS and for TOTAL (agricultural sector) are also included, either from World Bank database or by computation, as described below. Summary Information Table 10 displays summary information regarding the filling rates of the initial dataset. As it appears, numerous rows are in reality partially empty or irrelevant (just the country name and the year is filled). For instance, the overall dataset, excluding aggregates, has producer price (either in LCU or US$) for only 60% of the rows (25385) while Nominal Rate of Assistance (NRA) has been computed for 83% of the rows (35075), leading to difficulty to check the computation or even build a dataset focusing on distorted and undistorted prices. Restricting the dataset to the period, used as the default period covered by the consolidated database, the filling rate is better (64%) but this figure includes cells that have appeared to be filled by default using the previous year value ( variable in T = variable in T-1) without any reason or justification. 17

18 Table 10. World Bank Dataset: Summary Indicators - Raw dataset Total Period, all rows Total Period, excluding total and unindentified products , all rows , excluding total and unindentified products Number of rows in the dataset Number of rows including a NRA computation % 83% 90% 97% Number of rows with exchange rate information % 86% 98% 97% Number of rows with production (quantity) % 80% 84% 91% Number of rows with Producer Price (LCU) % 60% 59% 64% Number of rows with Producer Price (US$) % 60% 59% 64% Number of rows with Border Price (US$) % 80% 84% 91% Number of rows with a least 1 Producer Price % 60% 59% 64% Number of rows with a least 1 Price % 80% 84% 91% After treatment, the final dataset has rows for the period, including 8445 rows for specific products, this last figure should be compared to the rows for the similar definition in the initial dataset (see Table 10), so a reduction of 45%. The differences are due to the elimination of rows with too many missing information or aberrant values, but also the fact that we have aggregated for the EU the 26 countries in the initial dataset, into one (EU27). Belgium and Luxembourg were grouped in one country in the initial dataset. This involves a mechanical reduction of 32% of the dataset. Of course, the final dataset has a 100% filling rate for all variables. Data cleaning, harmonization and consolidation Procedure summary The following iterative procedure was implemented: 7. Dataset restriction (selection of relevant period and variables) in Excel 8. Elimination of rows with limited information in Excel 9. Code harmonization in Excel 10. Filling missing information for country level information in Excel 11. Transfer of the dataset to GAMS for data manipulation 12. Exchange rate management and harmonization 13. Quantity and Price filling procedure 14. MPS and PSCT computation in value 15. Management of the different values and indicators for the TOTAL sector 16. Management of the different values and indicators for the NONMPS sector 17. EU aggregate computation 18

19 18. Export to EXCEL and the different dataset format of Ag-Incentives 19. Harmonized unit descriptions Detailed steps During the first step, we restrict the dataset to the period and we keep a subset of variables relevant for the following stage: Country Code, Year, Sectoral code, quantity produced, value of production, value of consumption, exchange rates (x3), trade status, NRA, NRA_BMS, Non Product specific payment, prices at farm gate (LCU and US$), Border price (US$). In step 2, we eliminate rows that does not include at least one price (Border price, Farm Gate), information on production (in quantity or in value). Step 3 involves the harmonization of country and product codes and labels e.g.: the real ISO code is introduced for Malta and Romania (wrong code in the original database), difference in sectors label across countries are eliminated (Oilseed and Oilseeds become Oilseeds for all countries). In step 4, we identify countries for which no information on exchange rate and total production value were available. A country level table information is built combining the WB dataset and additional sources and will be used at a later stage. Step 5 is straightforward and allows us to use the flexibility of an index-based language. In order to use only the official exchange rate to compute prices for the Consortium database, we harmonize the exchange rate dataset used at the product level. Using the 3 exchange rate fields of the initial dataset, and the country level table built in step 4, we produce two exchange rate values at the product level: a product specific one, based on the raw data and combining the different fields (the filling rate of these fields vary from one country to another and something even within a country for a specific year), and an economic wide one (the official exchange rate). The later will be used to convert the undistorted farm gate price in US$ to LCU. For some country/year/product we fix manually the exchange rate when it appears to be clearly an outliner without consistency with the LCU/US$ price information. Step 7 is the most complex one since it involves significant reverse engineering to fill existing gaps. For instance, if produce prices are available in US$ and not in LCU, we use the product specific exchange rate to recompute the original producer price in LCU. If producer price and production value is available, but not the quantity, we compute the quantity, and reciprocally. So, anytime we have a logical link between 3 variables, and 2 are available we use them to complete the missing data. When the most straightforward manipulations have been performed, we add a first assumption: NRA_BMS = Producer Price in LCU / Reference Price in LCU 1. From this equation, we can compute 1 missing price as long as we have 1 of the price item and the NRA_BMS. For some aggregated commodities with missing price (e.g. mixed vegetables), producer prices are fixed to 1, allowing further computations. Consumption values are computed based on the database information. For missing values, we assume consumption equals to production. At this stage, the core dataset is built with information on all remaining rows regarding prices and quantities. 19

20 Step 8 is aimed to compute MPS and PSCT in value (LCU) to allow comparison with other sources (same concept) but also straightforward aggregation (across products). MPS is computed at the product level as : MPS = NRA_BMS x Undistorted production value PSCT is computed at the product level as : PSCT = NRA x Undistorted production value In step 9, we first compute the average MPS for all covered commodities (at producer price) and we apply this coefficient to the total production value (at producer price) following the homogeneity principle. For the TOTAL PSE, we add up the MPS value just computed, the sum of the differences between PSCT and MPS for all covered commodities and the Non Product Specific payments (converted in US$). Producer price for the TOTAL is defined as equal to 1. Production Quantity is equal to the total production value (VP). Reference Price is equal to 1 - [ MPS / VP ]. In step 10, we expand this approach to the NON MPS aggregate. The only difference here is that the PSCT is just equal to the MPS. The later being computed as the difference between total MPS and the sum of MPS over covered commodities. Step 11 is about the computation of the EU27 aggregate. The first action is to convert all LCU amount into EURO or ECU (the new LCU for the EU27 region). Then values are added. For specific commodities, quantities are summed up and the aggregated prices are defined based on value and quantity ratio. For aggregated commodities (TOTAL, non covered MPS commodity), producer price is redefined to 1 and other values and volumes defined accordingly. In Step 19, price units are split into two cells (so AUD/t now becomes two cells, one for the physical unit (MT) and the other for the currency (AUD)). Physical units for quantities are also included separately. Please note that all prices are now in local currency, i.e. million LCU have been transformed to LCU. The same transformation is also done for physical units, i.e. thousand metric tons are now metric tons. World Bank Definitions Source column shows the main database name, i.e. MAFAP, OECD, or World Bank. Source_File shows the specific workbook downloaded from the World Bank database. Timestamp shows when file was downloaded. Country_Code is 3 digit ISO code. Country_Label is full country name. Year is the year of the indicator. Product_Code is commodity code used by World Bank. Product_Label is commodity name used by World Bank. Product_Category is names for aggregate sectors as explained above. Production Quantity is the level of production (QP) reported in the sector specific worksheet for each commodity. Physical Unit column reports the basic unit of the data. For example if the original data was in 000 tons, the column will show MT for metric tons. Likewise, it will convert the data associated with it to MT as well. Trade Status is based on World Bank database. 20

21 Reference_Price (Farm Gate) LCU column contains sector specific data populated f the workbook or computed. Producer price (farm gate) LCU column contains sector specific data populated from the workbook or computed. Currency Unit column shows local currency using ISO currency alpha code. Physical Unit column shows final physical units. We computed Nominal Rate of Protection (NRP): (Producer Price at Farm Gate/ Reference Price at Farm Gate). Production Value (RP_FG) LCU column shows Reference_Price (Farm Gate) multiplied by Production Quantity. Production Value (PP_FG) LCU column shows Producer_Price (Farm Gate) multiplied by Production Quantity. World Bank Issues For this dataset, most of the efforts were devoted to fill the gaps, eliminate most aberrant values and build common definition and methods with the other sources. Specific issues remain: The dataset has partial coverage: some countries are missing for some years. At the country level, only 86% of the potential cells (country x time) are filled; When data appears to have been weakly updated (t and t-1 values are identical without reason e.g. 319 rows for production quantity, but only 14 rows for producer price) in the recent years, update should be performed; Information on consumption volume should be cross checked; Total production value for Kazakhstan has to be improved as well as production value for several African countries in 2005; Apparently inconsistent values remain with PP_FG > RP_FG and NRA <0, and reciprocally; Aberrant values for some prices. 21