Agriculture Information System Building Provincial Capacity for Crop Forecasting and Estimation Crop Information Portal of Pakistan: version 2 Database Administration Antonio Martucci (FAO-DDNS)
Task 1: Login To access the administration area, you need to have an account of administrator Click on the Login button in the MapStore toolbar and insert the account parameters Click on Administrative Page and access the home page
Administration Web Interface: Workflows Section Active Section Menu Sections Section tasks Results area
Administration Web Interface: Entities Section
Administration Web Interface: Entities Section
Administration Web Interface: File Manager
COMPONENTS: Administration Web Interface Tasks manageable through the Administration Web interface: Workflows Preparation and update boundaries Ingestion tabular data Ingestion NDVI images Extraction of NDVI indices from uploaded images User Manager File Manager Administration Configuration Files Entities Create new/edit existing crops Create new/edit existing agromet variables Create new/edit existing units of measure
Current Data Content Crops Area, production and yield data by District (official government source): Wheat (Rabi season) Cotton (Kharif season) Sugarcane (Kharif season) Rice (Kharif season) Maize (Kharif season) Agromet 10 day mean data by District. Minimum Temperature ( C) Maximum Temperature ( C) Precipitation (total mm) Day length (hr)
Current Data Content NDVI Source SPOT-VGT/AVHRR-metop at 10 day Mean indices by District Image composites Other indictors Fertilizers offtake (monthly by District, by Province and National) Irrigation water supply from canal withdrawal (10 day by District and by Province) River water flow at RIM stations (10 day by station) Market Prices (10 day by District/market) Alternative Crop Estimates (by Crop/District)
Crop Data Crops Wheat Rice Sugarcane Maize Cotton Values Area, production and yield Ingestion By District
Agromet Data Source Pakistan Meteorological Department Variables Temp max/min Precipitations Day length Mean NDVI in crop areas Temporal Frequency 10 day Ingestion By District
Fertilizers Source National Fertilizer Development Centre of Islamabad, Pakistan (http://www.nfdc.gov.pk/) Ingestion Unit Tons Temporal Frequency Monthly with on the fly annual aggregation Ingestion By District By Province National
Water Resources Source Indus River System Authority Ingestion Unit 000 Cubic Feet per Second (cusecs) Temporal Frequency 10 day with on the fly monthly aggregation Ingestion By measurement station
Water Resources Source Indus River System Authority Ingestion Unit Million Acres Feet (MAF) Temporal Frequency 10 day with on the fly monthly aggregation Ingestion by District by Province
Market Prices Source Government of Pakistan -Bureau of Statistics (http://www.pbs.gov. pk/cpi) Ingestion Unit Pakistan Rupee Temporal Frequency 10 day with on the fly monthly aggregation Ingestion by District (major market) Price denominator is defined from list during ingestion (1, 40 or 100 kg). Conversion from RP to US$ can be stored at ingestion.
Task 1: Interface Structure Tasks Results area
Task 2: Workflows Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Example split a District 4 files to update. Do not change attributes structure. Zip with the same names (below) and upload in Workflows Upload NDVI images NDVI Statistics by Admin units masked for crop areas Details of steps and GeoServer updates are described in manual
Task 2: Workflows Create or Update Layer (only admin geometry) Consistent table header (no variants allowed) Before any ingestion, the administrator needs to be sure that the CSVs are correctly formatted; there are some mandatory requirements that he needs to be aware of, for example Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas
Task 2: Workflows Consistent District/Province names Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas Inconsistent names between CIP database and CSV will result in ingestion failure for those records. See the log of the ingestion task for last verification.
Task 2: Workflows Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas Consistent values Crops: 000 t/ 000 bales; 000 ha; kg/ha; Agromet variables: NDVI (index); Precipitations (mm); Temp Max and Min ( C); Day Light (hr). Fertilizer: metric ton (t) Market Prices: Pakistan Rupees / 1, 40 or 100 Kg. Irrigation Water Supply (Canal withdrawal): million acre foot (MAF). River Water Inflow rate: 000 cubic feet per second (000 cusecs).
Task 2: Workflows See example of CSV in editor Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas Recap rules: 1. Consistent fields structure and names 2. Consistent district / province names 3. Consistent values
Task 2: Workflows INGESTION RULES Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas 1. Each record is uniquely identified by the combination of all fields except the value field (example below is price). The ingestion can do the following: 1. ADD new data 2. UPDATE existing records 3. DELETE records
Task 2: Workflows INGESTION RULES Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas 1. Each record is uniquely identified by the combination of all fields except id and value field (example below is price). The ingestion can do the following: 1. ADD new data 2. UPDATE existing records 3. DELETE records
Task 2: Workflows INGESTION RULES Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images 1. If the record is NOT FOUND in the database and the value is included (eg 61.00), the record is ADDED 2. If the record IS FOUND in the database and the value is included (eg 61.00), the record is UPDATED 3. If the record IS FOUND in the database and the value is empty, the record is DELETED NDVI Statistics by Admin units masked for crop areas
Practical example: build CSV file for ADD values 1. Go in \Crops\Ex_01\ Wheat 2014 Balochistan Districts 2. Open pak_crops_2000-2012_portal_04-05- 2015_revised.xlsx 3. Select Wheat tab 4. Scroll down to first year with no data, fill up Balochistan with new values and export.. (open the prepared file: wheat_balochistan_2012.xlsx) 5. Export to CSV and look at it in text editor (Notepad++). Check out problems and clean them (open the correct file: wheat_balochistan_2012_ok.xlsx) 6. Upload CSV in Portal and run ingestion task: NEXT
Upload CSV in Portal and run ingestion task Click on Workflow/CSV file ingestion (icon) Select the ingestion type (Cropdata) Upload CSV files into the Portal s file system: Click on Upload Click on Add files Select file and click on Upload (do not consider the FAILED message in IE) Click on Select button and use the file just uploaded (select and click OK)
Upload CSV in Portal and run ingestion task Select the source: Official Click Run
Upload CSV in Portal and run ingestion task Check Log
Upload CSV in Portal and run ingestion task Try DELETE the same values Open the excel file Remove values Save it as CSV Upload it in the Crop Portal and run the task with the same steps for adding values Check the log
Task 2: Workflows Create or Update Layer (only admin geometry) Ingestion other data Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas
Fertilizer Data Ingestion Fertilizers data are stored by District, Province and National separately, at monthly temporal frequency, in metric tons District Province Pakistan
River Water Inflow Rate Data Ingestion River water inflow rate data are stored by measurement station on specific rivers at 10 day temporal frequency. Monthly data are aggregated on the fly Values are in 000 cubic feet per second (cusecs) District
Irrigation Water Supply Data Ingestion Irrigation water supply data are stored by District and Provinces separately at 10 day temporal frequency. Monthly data are aggregated on the fly Values are in million acres feet (MAF) District Province
Market Prices Data Ingestion Market prices data for crop or other commodities are stored by District at 10 day temporal frequency. District Provincial or national aggregations (averages) are calculated on the fly. Also monthly data are averaged on the fly. Denominator: 1/40/100 kg Fixed or variable exchange rate
Task 2: Workflows NDVI 10 day composite images at 1km resolution. Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas Time series: April 1999 to Nov 2015. From April 1999 to April 2014 the source is the Vegetation sensor on SPOT-4 satellite (decommissioned Later dates are covered by the AVHRR sensor on European Metop satellites Format: Geotiff
Task 2: Workflows - Upload NDVI images Download from source (METOP-AVHRR S10 syntheses) and subset images to the extent of Pakistan http://www.vito-eodata.be/pdf/portal/application.html#home
Task 2: Workflows - Upload NDVI images Final result NDVI Image preparation
Task 2: Workflows - Upload NDVI images Rename files according to naming convention Upload file(s) In CIP s File Manager and Ingest NDVI images METOP_AVHRR_20150111_S10_AS_pak.tif to dv15012_pak.tif dv + YY + mm (01/12) + dek (1/3) File Manager
Task 2: Workflows Create or Update Layer (only admin geometry) Ingestion CSV Files: crops, agromet variables, vegetation condition, fertilizers, water resources and market prices Upload NDVI images NDVI Statistics by Admin units masked for crop areas
Task 2: Workflows -NDVI Statistics After running the task, the result will be a.csv file saved in the file system s folder /test_csv/generated/ with a standard name: pak_ndvi_${region}_${crop mask}_${start date}_${end date} e.g. pak_ndvi_district_standard_20130201_20130210.csv
Task 3: User manager Users and their ROLE Users with role ADMIN can manage other users and can access this section. Users with role USER can download tabular data from the Crop Information Portal. By clicking on ADD Userbutton you can create new users. User name, password and role are the only attributes of importance.
Task 4: File Manager Section for managing files in the Crop Information Portal storage space. Files and folders are managed by using the context menu. Files can be downloaded (from Portal to local, or uploaded (from local to Portal). IMPORTANT NOTE: use wisely the renaming/deleting capability. By renaming system folders some tasks could not work anymore. CSVs old uploads are currently stored in \csv_archive\
Task 5: Entities Configuration section for: Crops Agromet variables Units of measure Definition of crop and agromet characteristics Definition of units of measures, exchange rate for currency Definition of id and correspondent label Etc..
Task 5: Entities -CROPS It allows the management/ configuration of new or existing crops. With this task the administrator creates the structure; through the ingestion workflow the administrator fills the database with values. Current list of crops: soybean, chickpea and folder have no data ID: unique (DO NOT EDIT WHEN ALREADY POPULATED WITH DATA) Label: as it appears in the Portal Season: for grouping Default units for production, area and yield (the ones used for data ingestion)
Task 5: Entities -CROPS With new crops (eg soybean), additional configuration in GeoServer is required: Define styles for districts based choropleth maps of area, production and yield Define styles for province based choropleth maps of area, production and yield See manual for details
Task 5: Entities - AGROMET It allows the management/ configuration of new or existing agromet factors. Water resources, fertilizers and market prices are not configured. Their ID and default unit is predefined. Factor: unique, including aggregation and default unit Label: as it appears in the Portal Unit of measure Spatial aggregation
Task 5: Entities -UNITS It allows the management of units and currencies
Task 5: Entities -UNITS Configuration form ID: unique and short Label and Short Name: names appearing in the Portal Description: sometime info on conversion factor Class: type of database to which the unit is applied (area, prod, yield, water resource, denominator, currency); denominator is used only for market prices. Factor: conversion number with reference unit (index 1). The database stores data in default units (eg mm for precipitations, kg/ha for crop yield, etc..). The system uses the conversion factor to show data in another available unit on user request. Filter: It applies only to crops and filters the crop(s) for which the unit is available (eg bales to cotton only). Using the comma, multiple crops can be inserted.
Task 5: Entities -UNITS Let s analyze the Million Cubic Meter unit ID and names: no comments Description: it describes the fact that the default unit for the irrigation water supply database is Million Acre Foot, and that this unit has 1233 conversion factor Class: category of data to which this unit is applicable (irrigation water supply) Factor: as described, the conversion is 1233.48 Filter: N/A
Exercise If you wish to add a new unit of measure, please identify all the parameters required, and from the Entities section, add a new unit and fill the form (see on the right). We will see then if it works in the portal. Eg: add 1 bale as denominator of market price to be used only for cotton (pk/bale). 1 bale = 170 kg Reference denominator (1) is 100 kg
Exercise TASK: add 1 bale as denominator of market price to be used only for cotton (pk/bale). 1 bale = 170 kg Reference denominator (1) is 100 kg Jan 2012 average wheat price in Punjab: 2660.96 pkr / 100 kg 4523.63 pkr / bale
COMPONENTS: GeoNetwork Metadata Catalogue
COMPONENTS: GeoNetwork Metadata Catalogue Standard based spatial information management system Enables access to geo-referenced databases and cartographic products from a variety of data providers Enhance the spatial information exchange and sharing between organizations Opensource and standard based GeoNetwork opensource Community website: http://geonetwork-opensource.org/ Data is described through metadata records The Crop Information Portal platform has implemented an instance of Geonetwork opensource to help Pakistan authorities to describe properly spatial datasets that are published in the Portal.
Thank you Crop Information Portal (CIP) test site, fully working: http://cip-pakistan.geo-solutions.it/mapstore/ Dedicated pages to CIP (not yet completed): Intro: http://dwms.fao.org/~test/croportal/prt_background_en.asp Overall project site: Home: http://dwms.fao.org/~test/home_en.asp Satellite-driven Global Agricultural Monitoring (GLAM) for Pakistan http://pekko.geog.umd.edu/glam/pakistan/ 54