QA4ECV. Quality Assurance for Essential Climate Variables. Project Number

Transcription

1 QA4ECV Quality Assurance for Essential Climate Variables Project Number Deliverable: Responsible Partner: D2.3 Review of Validation Standards and Practices for Terrestrial ECV Products National Physical Laboratory Delivery date: August

2 Review of Land Validation Practices and Standards (Version 1.0) Authors: Tracy Scanlon 1 Joanne Nightingale 1 Other Contributors: Jan-Peter Muller 2 1 National Physical Laboratory, UK 2 University College London, UK - 2 -

3 CONTENTS EXECUTIVE SUMMARY Introduction Quality Assurance for Essential Climate Variables QA Service and System Objective Validation Standards Validation Definition CEOS WGCV LPV GCOS Requirements Core Climax System Maturity Matrix Core Climax Validation Information Validation Case Studies Introduction Albedo Example: GlobAlbedo LAI / FAPAR Example: Copernicus LAI / FAPAR Soil Moisture Example: ESA CCI Soil Moisture Summary Conclusions and Design of QA Service... Error! Bookmark not defined. 5 References

4 TERMS & ACRONYMS ATBD BHR C3S CCI CDR CEOS Core Climax DHR ECV EM EO ESA EU FAPAR FCDR GCOS LAI LPV MISR MODIS MSSL NIR NPL NRT PDF PUG QA QA4ECV QI ROI SDR SMM TCDR UCL WGCV Algorithm Theoretical Basis Document Bi-Hemispherical Reflectance Copernicus Climate Change Service ESA s Climate Change Initiative Climate Data Record Committee on Earth Observation Satellites Coordinating earth observation data validation for re-analysis for climate services (EU FP7 Project) Direct Hemispherical Reflectance Essential Climate Variable Electromagnetic Earth Observation European Space Agency European Union Fraction of Absorbed Photosynthetically Active Radiation Fundamental Climate Data Records Global Climate Observing System Leaf Area Index Land Product Validation Multi-angle Imaging Spectro-Radiometer Moderate-Resolution Imaging Spectroradiometer Mullard Space Science Laboratory Near Infra-Red National Physical Laboratory (UK) Near Real Time Probability Distribution Function Product User Guide Quality Assurance Quality Assurance for Essential Climate Variables (EU FP7 Project) Quality Indicator Region of Interest Surface Directional Reflectance System Maturity Matrix Thematic Climate Data Record University College London Working Group on Calibration and Validation - 4 -

5 EXECUTIVE SUMMARY In support of the European Union s Earth Observation Programme s Copernicus Climate Change Service (C3S), the Quality Assurance for Essential Climate Variables (QA4ECV) project aims to fulfil a current gap in the delivery of satellite derived climate data products. The project will prototype a system for the implementation and evaluation of QA measures for 6 satellite-derived ECV and ECV precursor datasets, thus providing confidence in their application for climate monitoring studies and climate change assessments. The purpose of developing and implementing a QA4ECV system is two-fold: 1. To provide ECV data product producers / science teams with the necessary resources (internationally accepted tools, standards, methodologies) to develop products with embedded QA information that is presented in a clear and common format throughout the Earth Observation (EO) community, and, 2. To provide ECV data users (scientists policy-makers) with robust QA information as a means to quantitatively assess uncertainty and fitness-for-purpose of the data and derived products. A QA service is to be provided under the QA4ECV project. The QA system is being implemented is an interactive web-service and the documentary framework is a series of documentation including procedures, good practice guidance and training which support the QA system. The QA system is split into several Quality Indicator (QI) categories. The main objective of this report is to allow an understanding of the current standards and practices implemented in the land validation community which is one the of the QA system QIs. To achieve this, this report studies the current standards / good practices available within the land validation community including information available from other similar projects. This is followed by the study of two validation reports to determine which validation studies are commonly undertaken and how the results are presented. From the review of current practices, a set of pertinent information about each validation study undertaken for a product is determined and these will be included in the questionnaire for data product providers utilising the QA system. In addition, information on the adherence of the product to Committee on Earth Observation Satellites (CEOS) Working Group on Calibration and Validation (WGCV) Land Product Validation (LPV) standards will be sought from each data provider. The QA system will be designed such that evidence provided in this section will aid in the assessment of the product against both the Global Climate Observing System (GCOS) requirements (i.e. information on stability and accuracy will be sought) as well as in the assessment against the Core Climax System Maturity Matrix (SMM)

6 1 Introduction 1.1 Quality Assurance for Essential Climate Variables Climate change mitigation and adaptation has risen to the top of the agenda for many governments and international organisations. This has led to the establishment of projects and programmes dedicated to the development of long-term global records of Essential Climate Variables (ECVs) using space-borne assets. In support of the European Union s Earth Observation Programme s Copernicus Climate Change Service (C3S) 1, the Quality Assurance for Essential Climate Variables (QA4ECV) ( project aims to fulfil a current gap in the delivery of satellite derived climate data products. The project will prototype a system for the implementation and evaluation of QA measures for 6 satellite-derived ECV and ECV precursor datasets, thus providing confidence in their application for climate monitoring studies and climate change assessments. The purpose of developing and implementing a QA4ECV system is two-fold: 1. To provide ECV data product producers / science teams with the necessary resources (internationally accepted tools, standards, methodologies) to develop products with embedded QA information that is presented in a clear and common format throughout the Earth Observation (EO) community, and, 2. To provide ECV data users (scientists policy-makers) with robust QA information as a means to quantitatively assess uncertainty and fitness-for-purpose of the data and derived products. Provision of such QA information will demonstrate traceability of products and simplify comparisons, including round-robin selection, between the same ECV produced by independent science teams. It will also provide data users with evidence-based confidence in the products and enable judgement on the fitness-for-purpose of various ECV Climate Data Records (CDRs) for their specific applications. 1.2 QA Service and System One of the main aims of the QA system being developed under QA4ECV is to bridge the gap between data users and data producers, i.e. allow the transfer of information between the two in easy-to-use and consistent formats (as far as practicable) 2. Taking this into account, the QA system is split into two main paths : Data providers will follow a series of pages through the QA system to provide evidence relating to their ECV data product. This evidence will be included in a central repository. Data users will be able to search and download quality reports about a range of ECV data products from the central repository for comparison. The QA service includes: The QA system is a physical system implemented is an interactive web-service through which data products will be assessed, and, The documentary framework is a series of documentation including procedures, good practice guidance and training which support the QA system (and are linked to 1 Details of the Copernicus climate change service are available at: 2 Note that several other EU funded projects including EUPORIAS and Core Climax have identified this as a key factor which could improve the overall use of climate data sets. See [5]

7 throughout the QA system). The structure of the QA service is summarised in Figure 1 3 (in green) within the context of ECV production (yellow) and dissemination (blue and purple). The documentary framework which is the associated guidance etc. is within the Tools & Guidance to establish and evaluate QA box (in green). Figure 1: Diagram of system. 1.3 Objective The main objective of this report is to allow an understanding of the current standards and practices implemented in the land validation community; this information will be input to the QA system as it is being developed. To achieve this, this report studies the current standards / good practices available within the land validation community (Section 2) including information available from other similar projects (originally studied in [1]). This is followed (Section 3) by the study of two validation reports to determine which validation studies are commonly undertaken and how the results are presented. 1.4 Validation Definition Validation is the process of assessing, by independent means, the accuracy of the data products derived from system outputs [2]. There are 2 types of validation (further details at [3]): Absolute validation involves comparison against in situ reference data or scaled highresolution reference data Product Comparison is the inter-comparison of products from different sensors offers a simple way to evaluate the temporal and spatial consistency between products [4]. Product intercomparisons, whether between satellite-derived or ecosystem modelderived land products show where parameter estimates both agree within suitable bounds or clearly disagree. This provides a unique opportunity to evaluate underlying model assumptions through sensitivity analyses, the need for additional data collection 3 Note this diagram is also provided at

8 and highlights where perhaps more detailed validation studies are warranted [5]

9 2 Validation Standards 2.1 CEOS WGCV LPV The Committee on Earth Observation Satellites (CEOS) Working Group on Calibration and Validation (WGCV) Land Product Validation (LPV) subgroup is tasked with the coordination of the quantitative validation of satellite derived products with a focus on standardisation, intercomparison and validation across products from different satellites, algorithms and space agencies. LPV is organised into 10 focus areas for different biophysical variables; for each of these, two experts are assigned. The LPV subgroup has defined a validation hierarchy which allows the assessment of products against a standardised set of criteria Table 1: CEOS WGCV LPV Validation Hierarchy Stage Description Product accuracy is assessed from a small (typically < 30) set of locations and time periods by comparison with in-situ or other suitable reference data. Product accuracy is estimated over a significant set of locations and time periods by comparison with reference in situ or other suitable reference data. Spatial and temporal consistency of the product and consistency with similar products has been evaluated over globally representative locations and time periods. Results are published in the peerreviewed literature. Uncertainties in the product and its associated structure are well quantified from comparison with reference in situ or other suitable reference data. Uncertainties are characterized in a statistically rigorous way over multiple locations and time periods representing global conditions. Spatial and temporal consistency of the product and with similar products has been evaluated over globally representative locations and periods. Results are published in the peer-reviewed literature. Validation results for stage 3 are systematically updated when new product versions are released and as the time-series expands. One of the main objects of the LPV is to provide good practice guides on validation. For some of the biophysical variables covered by the group this has been implemented, for others, specific examples are pointed to as validation and for some, there is no best practice guidance available. For the three variables being studied under QA4ECV albedo and FPAR currently has no good practice guidance available; a document has been written for Leaf Area Index (LAI) [6]. 2.2 GCOS Requirements The Global Climate Observing System (GCOS) requirements for satellite derived products are set out in GCOS-154 [7]. For each ECV, there are a set of requirements relating to stability, accuracy etc. (see example for albedo in Figure 2). Whilst the GCOS requirements are not a set of validation criteria, assessment against the requirements may be best housed in a validation report as details of the stability and accuracy will be presented in the validation report in most cases

10 Figure 2: Example of GCOS requirements (for Albedo) as presented in [7]. 2.3 Core Climax System Maturity Matrix The aim of the Core Climax 4 System Maturity Matrix (SMM) (described at [8]) is to evaluate the production of the ECV CDRs to ensure they follow best practices for science, engineering and utilisation; it is not to assess the quality of the data itself. The SMM is divided into 6 areas: software readiness, metadata, user documentation, uncertainties characterisation, public access, feedback and update and usage. Of these, user documentation and uncertainties characterisation both have elements relating to validation. These elements are listed in Table 2. As can be seen from the table below, a high scoring product would be validated against external reference data for spatially and temporally representative locations and times as well as being compared against, ideally multiple, other datasets with updates being made to the product where inconsistencies / differences are identified. This validation information should then be reported in a publically available report, and ideally, in a peer-reviewed article which contains information on the uncertainties associated with the product. 4 Details of the Core Climax project are provided at:

11 Table 2: Core Climax System Maturity Matrix Elements applicable to validation Section User Documentation Uncertainty Characterisation Sub Section Formal Validation Report Formal Product User Guide 1 None Not applicable to 2 validation Report on limited validation available from PI 3 Report on comprehensive validation available from PI; Paper on product validation submitted 4 Report on intercomparison to other CDRs, etc. Available from PI and data Provider; Journal paper on product validation published 5 Score 4 + Report on data assessment results exists 6 Score 5+ Journal papers more comprehensive validation, e.g., error covariance, validation of qualitative uncertainty estimates published Score 4 + regularly updated by data provider with product updates and/or new validation results None Validation Validation using external reference data done for limited locations and times Validation using external reference data done for global and temporal representative locations and times Score 3 + (Inter)comparison against corresponding CDRs (other methods, models, etc.) Score 4 + data provider participated in one inter-national data assessment Score 4 + data provider participated in multiple international data assessment and incorporating feedbacks into the product development cycle Uncertainty Quantification Not applicable to validation Score 5 + comprehensive validation of the quantitative uncertainty estimates and error covariance

12 2.4 Core Climax Validation Information The Core Climax project provides a protocol for verifying, monitoring, calibrating and validating Fundamental CDRs (FCDRs) and Thematic CDRs (TCDRs) [9]. This document provides an architecture for climate monitoring from space which is broken down into detail throughout the report. One of the figures (shown in Figure 3) describes the different validation techniques commonly utilised. Figure 3: Validation strategy for CDRs showing (left) the scaling method and (right) the direct comparison method. Taken from [9]

13 3 Validation Case Studies 3.1 Introduction Satellite-derived terrestrial ECV products are subject to validation; this validation may be presented either as a single report, or as several reports and / or peer-reviewed or conference papers. This section studies three such validation reports, one for the GlobAlbedo product which is a global, multi-decadal albedo product and has much in common with the QA4ECV albedo product, one for the Copernicus land monitoring LAI / Fraction of Absorbed Photosynthetically Active Radiation (FAPAR) data product, which is provided in Near Real Time (NRT) for Europe and one for the global soil moisture data product produced through the European Space Agency (ESA) Climate Change Initiative (CCI) programme. The different properties of these products (i.e. global vs. regional, long-term vs. short-term) help to ensure that the results of this study are representative for different product types 5 and between different organisations. The following sections (Sections 3.2, 3.3 and 3.4) describe the general content of the validation reports including methods used and examples of how results have been presented; this is followed by a summary in which the two products are compared (Section 3.5). 3.2 Albedo Example: GlobAlbedo The GlobAlbedo product was developed by Mullard Space Science Laboratory (MSSL) which is part of University College London (UCL). The product provides albedo (Directional Hemispherical Reflectance (DHR) and Bi-Hemispherical Reflectance (BHR)) from 1998 to 2011 at 8-day or monthly intervals. The product is derived from European satellites using a MODIS-derived climatology as a prior to constrain the solution. The product is available at various spatial scales: 1 km, 0.05 or 0.5 pixels and covers the visible, short wave and near infra-red (NIR) regions of the electromagnetic (EM) spectrum. The algorithm is described in the Algorithm Theoretical Basis Document (ATBD) [10] and details of the QIs are provided in the Product User Guide (PUG) [11]. The product validation report [12] provides details of the validation not only for the final product but for some of the intermediate stages of the product generation process (including cloud detection, radiometric inter-calibration (after narrowband-to-broadband transformation), atmospheric correction and the generation of the Surface Directional Reflectances (SDRs)). For the final product, the broadband albedo, comparisons of the product are undertaken against tower albedos and against Moderate-Resolution Imaging Spectroradiometer (MODIS) and Multi-angle Imaging Spectro-Radiometer (MISR) derived albedo datasets. In addition, known issues with the product are subjected to a qualitative study. The tower albedo data utilised in the validation included data from FLUXNET (which includes EuroFlux and AmeriFlux), SURFRAD and ARM (73 sites in total); these sites are spatially distributed among different continents, temperate zones etc. The comparisons were undertaken for a period of 14 years (where data was available). The validation methods utilised are described in the Validation Plan [13]. The validated spectral range (in this case short wave) is stated, with reasons as to why the other parts of the spectrum cannot be validated given. The results of the tower validation are shown as a graph (see example at Figure 4(a)), with space-borne albedos plotted alongside validation site values. It is stated that for some of the sites there is excellent agreement between the datasets. Where possible, discrepancies between the datasets are explained. In addition, scatterplots are provided to show the 5 Note: The two products have also been considered in the review of Quality Indicators (QIs) as part of the QA4ECV project [18]

14 correlation between some of the datasets (see example at Figure 4(b)). (a) Plot of GlobAlbedo values against validation site and other satellite-derived products. (b) Scatter plot to show correlation between tower and GlobAlbedo values. Figure 4: Plots used in the GlobAlbedo validation report [12] to show the agreement between tower albedo and satellite-derived values. Plots of the difference between the measurements versus the uncertainty associated with the satellite-derived products are provided (see example provided in Figure 5). These plots allow for information about potential issues with the uncertainty values to be discussed. (a) Plot to compare the differences between the GlobAlbedo and validation site values with the uncertainties associated with GlobAlbedo. (b) Scatter plot to show correlation between tower and GlobAlbedo values. Figure 5: Plots used in the GlobAlbedo validation report [12] to explore how the observed differences between GlobAlbedo and the validation sites relate to the uncertainties associated with the GlobAlbedo product. Plots of the differences between tower and satellite derived measurements are also presented by land cover type. This allows for information on the applicability of the product over different land areas to be conveyed to the user. The intercomparison of the GlobAlbedo values with MISR and MODIS is undertaken for the year 2005 (global coverage). This is stated as having good agreement (for MODIS) and less good (for MISR); in both cases, an R squared value is quoted. The final stage of the validation included a triple colocation with the intent of uncovering systematic errors. This has been applied to GlobAlbedo, MODIS and MISR for the temporal

15 range The key parameter taken from this activity is the variance and maps, as well as histograms, are provided (see example provided in Figure 6). A summary of the key points from the validation, including cases potential issues in use are provided in the summary section of the validation document. (a) Variance map for GlobAlbedo. (b) Variance histogram for GlobAlbedo. Figure 6: Plots used in the GlobAlbedo validation report [12] to explore how the observed differences between GlobAlbedo and the validation sites relate to the uncertainties associated with the GlobAlbedo product. 3.3 LAI / FAPAR Example: Copernicus LAI / FAPAR The Copernicus LAI / FAPAR 6 product is produced by INRA, CREAF and VITO. The product has been derived from PROBA-V data and covers the period May 2014 present for Europe. It is a global product provided at a spatial resolution of 1 / 3 km. The product is described within the ATBD [14] and information on the QIs associated with the product are given in the PUG [15]. The validation protocol and results for the Copernicus LAI product are presented in [16]. A summary of the validation methods utilised are provided in Table 3. The requirements for the quality criteria, which are listed fully in [16] have been derived both from defined user requirements and using the GCOS requirements 7 document [7] and are referred to throughout the validation document. Where a satellite product has been utilised in the validation, summary information on the products are provided including sensors utilised, spatial resolution, frequency of observations and retrieval and details of the algorithm employed and included in the validation report. In addition, for in-situ reference products used, sites are listed including the dates utilised, type of land cover etc. 6 Note: The Copernicus product also includes FCover, however, this is not one of the ECVs being developed under Q4AECV and therefore is not considered further. 7 Including the use of threshold, target and optimal requirements which is a method suggested as part of the Core Climax project for assessing the fitness-for purpose of a product to a particular application xxx

16 Table 3: Summary of the validation methods utilised for the Copernicus LAI products. Taken from Table 5 from [16]. Quality Criteria Product Evaluated Reference Product Coverage Completeness Spatial Consistency Temporal Consistency Intra-annual Precision (Smoothness) Statistical Analysis (Discrepancies) Accuracy Assessment (Error) PROBA-V GEOV3 HIS &NRT PROBA-V GEOV1 MODIS C5 Europe 153 EUVAL sites Gap size distribution (average gap maps, temporal variations per biome). Length of gaps. PROBA-V GEOV3 HIS &NRT PROBA-V GEOV1 MODIS C5 Europe 153 EUVAL sites Visual inspection of global maps. Difference maps and histograms of residuals (Europe). PDFs of retrievals and histograms of residuals per biome and region (EUVAL) PROBA-V GEOV3 HIS &NRT PROBA-V GEOV1 SPOT / VGT GEOV1 MODIS C5 153 EUVAL sites Qualitative inspection of temporal variations (153 EURO sites). Histograms of the cross-correlation per biome. PROBA-V GEOV3 HIS &NRT PROBA-V GEOV3 All modes PROBA-V GEOV1 MODIS C5 Histograms of the smoothness. PROBA-V GEOV1 MODIS C5 153 EUVAL sites 153 EUVAL sites Scatter plots (R2, RMSE, bias, scattering) Box-pots of uncertainty metrics (bias and RMSE) per biomes. Temporal evolution of mean bias per biomes. PROBA-V GEOV3 HIS &NRT Ground-based maps In-Situ Products Scatter-plots, pearson s correlation, RMSE, bias, linear fit (offset, slope) The completeness of the data records is assessed within the validation document for the year For spatial completeness, this is achieved through calculating the percentage of missing values for different spatial regions; maps are provided. For temporal completeness, the missing values are shown over the period of 2014 for a specific area and for different biomes (see example at Figure 7)

17 Figure 7: Plot used in the Copernicus LAI / FAPAR validation report [16] to show missing values over a year for different biome types. The spatial consistency of the product is assessed through visual inspection of the data product. This is carried out on specific Regions of Interest (ROIs) which include bare areas and densely vegetated areas; an example map is provided in the validation report. The spatial consistency is also checked quantitatively through comparison against a reference validated product with maps analysed on a monthly and annual basis. Histograms showing the differences between the reference and assessed datasets are provided, as per the example in Figure 8. Reasons for unexpected results are clearly provided. This is also undertaken for each biome type. Figure 8: Plot used in the Copernicus LAI / FAPAR validation report [16] to show the differences between the product and a reference data product for different months over the course of one year. The assessment of temporal consistency is undertaken for a selection of EUVAL sites for different biome types as well as through comparison with other satellite-derived products. For both assessments, temporal profiles are provided for specific sites, as shown in Figure 9. In addition, for assessing the temporal consistency, a cross correlation metric is calculated and the histograms of the metric are evaluated. This is undertaken for the different available satellite products in different biomes

18 Figure 9: Plot used in the Copernicus LAI / FAPAR validation report [16] to show the temporal profile of the product over specific sites throughout the year compared to other satellite derived data products. The internal annual precision (smoothness) is analysed assuming that there is no serial correlation within a season. It is estimated based on each triplet of consecutive observations and the associated histograms are evaluated. Statistical analysis is performed using EUVAL network of sites considering all available dates; the statistics listed in Table 4 are calculated. The EUVAL network is selected to consider the variability of the land surface types over Europe

19 Table 4: List of statistics calculated to compare the Copernicus LAI / FAPAR product to EUVAL sites. Taken from Table 4 from [16]. Gaussian Statistic Scatter plot of ground versus product Number of samples RMSE Mean bias Standard deviation Correlation coefficient Linear fit (slope, offset) p-value Comment Qualitative assessment of agreement. Indicative of the power of the validation. RMSE computed between ground and product values should be compared to the RMSE value of the ground measurements. Outliners should be removed or weighted. Indicates the accuracy. Difference between average values of ground and product. Indicative of accuracy and possible offset. Standard deviation of the pair differences. Indicates precision. Indicates descriptive power of the linear accuracy test. Pearson coefficient was used. Indicates some possible bias. Statistical test on whether the slope is significantly different to 1, which is the null hypothesis here. P value < 0.05 indicates potential bias (slope different of 1). In addition to the generation of the statistics provided in Table 4, the consistency between products is further evaluated using Probability Distribution Functions (PDFs) of product values and their residuals to the ground data as well as through the generation of scatterplots of product pairs for different satellite-derived products (example shown in Figure 10). In addition, box-plots are provided to show the distribution of residuals between satellite derived products (see example shown in Figure 11). Figure 10: Plot used in the Copernicus LAI / FAPAR validation report [16] to show the difference between the product and a different satellite derived product over all of the EUVAL sites

20 Figure 11: Plot used in the Copernicus LAI / FAPAR validation report [16] to show the differences between the product and a different satellite derived product for different biome types. The accuracy assessment of the product is undertaken at two different spatial scales: 20 x 20 km 2 and 3 x 3 km 2 with up scaling being undertaken in accordance with CEOS WGCV LPV procedures [6]. The results are provided as a combination of maps, histograms and scatterplots. The concluding remarks of the validation report stated that the product shows good quality. The conclusions then go on to provide a summary of the main findings of the report and associated warnings for use to the data user. Linkages are made back to the user and GCOS requirements stated at the start of the report where applicable. 3.4 Soil Moisture Example: ESA CCI Soil Moisture The ESA CCI for soil moisture is a 35 year time series developed based solely on merging active and passive remote sensing observations. The product is formally called ECV SMv2.0. The validation of the soil moisture CCI is presented at [17]. The validation procedures have been undertaken by organisations independent from the developers of the data record. Four different validation studies and intercomparisons have been conducted utilising in-situ, model and other satellite derived soil moisture datasets. The in-situ datasets are from the International soil moisture network xxx and the North America soil moisture data base xxx. The other satellite derived dataset utilised is the precursor to the validation target product called ECV SMv0.1. The document is split into four sections, each covering one study. Details of each study are provided including information on the temporal coverage of the study, the sites / locations utilised and the applicability of the studies to the dataset, including questions raised by the studies. The results of each study are presented in different ways, for example, correlations, spearman rank correlations, and unbiased root mean square differences (ubrmsd) are used as statistical measures. In terms of graphical representations, temporal trends are plotted, spatial correlation maps are provided (see Figure 12) and Taylor diagrams (see Figure 13) showing the performance of datasets for different soil classifications are also used

21 Figure 12: Plots used in the ESA CCI Soil Moisture validation report [17] to show the spatial correlation between datasets at a particular geographical location. Figure 13: Taylor plots used in the ESA CCI Soil Moisture validation report [17] to allow the visual assessment of the degree of similarity of surface soil moisture of the products. Includes information on correlation, ubrmsd and standard deviation. 3.5 Summary The previous sections (Sections 3.2 and 3.3) have considered the validation methods utilised and the method of presentation of results for two different ECV products. A summary of this information is presented in Table

22 Table 5: Comparison of validation methods utilised and presentation of results for different ECV data products. GlobAlbedo Copernicus LAI / FAPAR ESA CCI Soil Moisture Standards Statement of following CEOS WGCV LPV good practice - X - Assessment against CEOS WGCV LPV validation hierarchy Presentation of Requirements GCOS requirements stated / assessed against - X - User requirements stated / assessed against - X - Validation Assessments Undertaken Completeness of the data record - X - Evaluated for spatial consistency - X - Temporal Consistency - X - Intra-annual Precision (Smoothness) - X - Statistical Analysis (Discrepancies) - X - Accuracy Assessment (Error) against independent (likely ground based) dataset X X - Triple collocation??? X - - For different biomes / land cover types X X - Uncertainties considered somehow? X - - Information Provided for Each Validation Assessment Undertaken Temporal range for the validation stated X X X Spatial range for the validation stated X X X Spectral range for the validation stated X N/A N/A Presentation of Results Scatterplots of differences between product and reference dataset X X X Histograms of differences between product and reference dataset X X - Box-plots of differences between product and reference dataset Plots of timeseries (for comparison or stability monitoring) X X X Correlation maps - - X Conclusions Provides a summary of the applicability of the validation to the use of the product X X X X Provides warnings to the data user where appropriate X

23 4 Summary This report has summarised the validation standards and requirements currently available to the land community (Section 2) as well as providing a review of the current validation practices employed for three land products (Section 3) from different organisations. This information is utilised in this section to feed into the design of the QA service with regards to validation. From the review of the currently utilised practices, it is clear that, in some cases, product completeness, consistency and other measures are considered part of validation. However, in many cases they are not and therefore they are not to be considered further under the validation QI of the QA system (they will be considered as part of a separate QI called Product Completeness and Consistency ). The aim of the validation section of the QA system is to ensure that all data producers putting their product through the system can demonstrate that their product has been validated and that this information is readily available and allows the users to assess the fitness for purpose of the product for a particular application. To enable this, the data provider will be asked to provide information about the validation studies undertaken for their product. As part of this, they will be asked to distinguish between absolute and relative validation studies as well as providing information on the temporal, spatial and (if relevant) spectral coverage of the validation studies undertaken. They will also be asked to provide a short statement for each validation study summarising how the data could be interpreted in terms of validation of the entire spatial / temporal coverage of the full product. Information on the validation studies undertaken will need to be supported by the upload / pointing to of relevant documentation, i.e. a report or peer-reviewed paper. Guidance and a template on the product of validation reports will also be available to the user of the QA system in this section. Some of the information gathered in this part of the validation QI section will feed into the assessment of the product against the SMM. In addition to the provision of information on individual studies, the data producers will also be asked to provide a statement of the accuracy and stability of the overall product, where possible. These will be utilised directly in the assessment against the GCOS requirements (under the Evaluation against International Standards QI). Data producers will also be asked to make a statement about whether or not recommended guidelines on validation have been followed, i.e. those available from the CEOS LPV website and to assess their product against the LPV validation hierarchy

24 5 References [1] T. Scanlon, S. Douglas, J. Nightingale, S. Compernolle, and J.-C. Lambert, QA4ECV: Linkages between the QA Service of QA4ECV and Other Projects and Initiatives, [2] C. Justice, A. Belward, J. Morisette, P. Lewis, J. Privette, and F. Baret, Developments in the validation of satellite sensor products for the study of the land surface, vol. 21, no. 17, pp , [3] J. Nightingale, J. Nickeson, C. Justice, F. Baret, S. Garrigues, R. Wolfe, and E. Masuoka, Global validation of EOS land products, lessons learned and future challenges : A MODIS case study, in Proceedings of the 33rd International Symposium on Remote Sensing of Environment, [4] S. Garrigues, R. Lacaze, F. Baret, J. T. Morisette, M. Weiss, J. E. Nickeson, R. Fernandes, S. Plummer, N. V. Shabanov, R. B. Myneni, Y. Knyazikhin, and W. Yang, Validation and intercomparison of global Leaf Area Index products derived from remote sensing data, J. Geophys. Res., vol. 113, no. 2, [5] N. C. Coops, C. J. Ferster, R. H. Waring, and J. Nightingale, Comparison of three models for predicting gross primary production across and within forested ecoregions in the contiguous United States, Remote Sens. Environ., vol. 113, no. 3, pp , [6] Global Leaf Area Index Product Validation Good Practices, pp [7] WMO, Systematic Observation Requirements For Satellite-Based Data Products for Climate Update - GCOS-154, [8] J. J. Bates and J. L. Privette, A Maturity Model for Assessing the Completeness of Climate Data Records, Eos --- Trans. Am. Geophys. Union, vol. 93, no. 44, p. 441, [9] Core Climax, Core Climax: Protocol for verifying, monitoring, calibrating and validating FCDRs and TCDRs of the CDRs / ECVs, [10] P. Lewis, G. López, J.-P. Muller, D. Potts, N. Shane, S. Kharbouche, D. Fisher, C. Brockmann, O. Danne, O. Krueger, A. Heckel, P. North, and L. Guanter, GlobAlbedo Algorithm Theoretical Basis Document, [11] O. Danne, M.Zuehlke, and U. Krämer, GlobAlbedo Product User Guide, [12] J.-P. Muller, G. López, N. Shane, O. Danne, C. Brockmann, U. Krämer, M. Zühlke, A. Heckel, P. North, L. Guanter, J. Fischer, Z. Wang, C. Schaaf, G. Schaapman-Strub, and A. Cescatti, GlobAlbedo Final Product Validation Report, [13] J-P. Muller, C. Brockmann, L. Guanter, P. North, N. Shane, and G. López, GlobAlbedo Product Validation Plan, [14] F. Baret and M. Weiss, Algorithm Theoretical Basis Document: LAI, FAPAR and FCover, [15] B. Smets, T. Jacobs, and R. Lacaze, Gio Global Land Component - Lot I Operation of the Global Land Component : Product User Manual - LAI, FAPAR., FCover - Collection 300m, [16] F. Camacho and J. Sanchez, Copernicus Surface Albedo - Version 1 - SPOT /

25 VEGETATION - Quality Assessment Report, [17] H. Mittelback, M. Hirschi, C. Pratola, T. Smolander, W. A. Lahoz, N. Dwyer, K. Rautianinen, and S. I. Senevirantne, ESA CCI: Soil Mositure: Product Validation and Intercomparison Report (PVIR), [18] T. Scanlon, J. Nightingale, S. Compernolle, and J.-C. Lambert, QA4ECV: Review of Product and Pixel Level Quality Indicators Provided with ECV Data Products, no ,