Session 3 Uncertainty Assessment

Transcription

1 Session 3 Uncertainty Assessment Hand-on Training Workshop on National GHG Inventories - IPCC Cross-Cutting Issues November 2015, Ankara Turkey

2 Introduction Perfect accuracy and certainty impossible Understanding and communicating uncertainties is an essential component of the GHG inventorying process. Use to help prioritize inventory improvement efforts

3 Topics Definitions of key terms and common causes of uncertainties Basis for uncertainty analysis and its key benefits Approaches to quantifying uncertainties Methods to combine uncertainties Strategies to reducing uncertainty

4 Definitions Random error Bias or Systematic error Accuracy and Precision Uncertainty Variability Confidence interval Probability density function Note: You do not need to memorize these terms, but you should have a good understanding of what they mean to follow the discussion in this lesson.

5 Definitions Accuracy: Agreement between the true value and the average of repeated measured observations or estimates of a variable. An accurate measurement or prediction lacks bias or, equivalently, systematic error. Precision: Agreement among repeated measurements of the same variable. Better precision means less random error. Precision is independent of accuracy.

6 Definitions Bias or Systematic error: Lack of accuracy. The mean or average of many separate measurements differs in a regular amount and direction from the actual value. It can occur because of failure to capture all relevant processes involved or because the available data are not representative of all real-world situations, or because of instrument error. Random errors: Random variation above or below a mean value. Random error is inversely proportional to precision. Usually, the random error is quantified with respect to a mean value, but the mean could be biased or unbiased. Thus, random error is a distinct concept compared to systematic error.

7 Definitions Variability: Heterogeneity of a variable over time, space or members of a population. Variability may arise, for example, due to differences in design from one emitter to another (inter-plant or spatial variability) and in operating conditions from one time to another at a given emitter (intraplant variability). Variability is an inherent property of the system or of nature. Uncertainty: Lack of knowledge of the true value of a variable that can be described as a probability density function (PDF) characterizing the range and likelihood of possible values. Uncertainty depends on the analyst s state of knowledge, which in turn depends on the quality and quantity of applicable data as well as knowledge of underlying processes and inference methods.

8 Definitions Confidence Interval: The true value of the quantity for which the interval is to be estimated is a fixed but unknown constant, such as the annual total emissions in a given year for a given country. The confidence interval is a range that encloses the true value of this unknown fixed quantity with a specified confidence (probability). Probability density function (PDF): The PDF describes the range and relative likelihood of possible values. The PDF can be used to describe uncertainty in the estimate of a quantity that is a fixed constant whose value is not exactly known, or it can be used to describe inherent variability. Throughout this lesson it is presumed that the PDF is used to estimate uncertainty, and not variability.

9 The Basics The emissions may be: Directly measured Estimated based on proxy data that is believed to accurately approximate emissions

10 Uncertainty types Scientific uncertainty: Arises when the science of the actual emission and/or removal process is not completely understood. Analyzing and quantifying scientific uncertainties is extremely problematic and is likely to be beyond the scope of project proponents and verifiers. Estimation uncertainty: Arises any time GHG emissions are quantified. Therefore all emissions or removal estimates are associated with estimation uncertainty.

11 Estimation Uncertainty Estimation uncertainty can be further classified into two types: Model uncertainty Parameter uncertainty

12 Model Uncertainty Associated with the mathematical equations (i.e., models) that characterize relationships between various parameters and emission processes. Due to the use of an incorrect mathematical model or inappropriate input into the model. Likely to be beyond the scope of GHG inventory team

13 Parameter Uncertainty Associated with quantifying the parameters used as inputs (e.g., activity data and emission factors) to estimation models Can be evaluated through statistical analysis, measurement equipment precision determinations, and expert judgment. Investigating and quantifying parameter uncertainties is the primary focus of inventory team Activity data and emission factors are parameters Types of parameter uncertainty Statistical (random) Systematic (bias)

14 Random! Statistical Uncertainty Where possible, can be identified through repeated sampling & subsequent statistical analysis For many parameters, it may not be possible to repeatedly sample or collect data under the same operating conditions Statistical uncertainty is generally assumed to average out over time Statistical uncertainty is the cause of noise in the data Statistical parameter uncertainty may be approximated by the rated precision of measurement equipment.

15 Statistical Uncertainty True Values Measured Values

16 Systematic Uncertainty (Bias) Measured values consistently greater (or less) than the true value Biases cannot be detected through sampling of data and statistical analysis Biases can be identified through: Data quality investigations and other QA/QC measures Comparison of data with other independent datasets Biases do not average out over time, and therefore present a more serious problem than random uncertainties. Biases may increase (or decrease) over time if data quality falls (rises) and historical data is not able to be revised. Changes in biases over time are particularly troubling because of their impacts on emission trends and estimated emission reductions.

17 Systematic Uncertainty (Bias) True Values Measured Values

18 Causes of Parameter Uncertainties Random errors in measurement devices (parallax error, temperature variation, etc.) Measurement devices can produce systematic biases Imprecise calibration, faulty measurement equipment, environmental factors, operator error, double counting, omission of data, etc. Parameters may also be based on non-representative samples Fuel samples taken monthly, while fuel deliveries occur weekly. Data does not account for process start-up and shut-down conditions, or irregular operating conditions.

19 Reporting and Combining Parameter Uncertainties Qualitative High, Medium, Low Semi-Quantitative Combine uncertainty values using ranking schemes (e.g., Data Attribute Reporting Systems) Quantitative models Error propagation (assumed data distributions are normal & all parameters are independent) Monte Carlo (complete flexibility)

20 Confidence interval Typical to use a 95% confidence interval 95% probability of enclosing the true, but unknown, value Enclosed by the 2.5th and 97.5th percentiles of a probability density function (PDF)

21 Example: Symmetric uncertainty PDF is a normal (symmetric) distribution Mean value is th percentile of uncertainty is th percentile of uncertainty is ±30%, at a 95% probability range.

22 Example: Asymmetric uncertainty Asymmetric about the mean value of th percentile of uncertainty is th percentile of uncertainty is % to +100%.

23 Biases are often unknown Difficult to capture systematic errors (biases) because you are often not aware of them Biases that are known can be corrected for Biases may arise from: imperfections in the design of the GHG inventory (assumptions, methods selected etc.) simplifications and assumptions in calculation models used measurement techniques

24 Good practice for uncertainty analysis Potential causes of uncertainty that are not quantified should be described, particularly with respect to the design of the GHG inventory, models, and data and to make an effort to quantify them in the future

25 Collecting Uncertainty Information Parameter uncertainty information may be collected from: Calibration records for measurement devices Measurement equipment precision values supplied by equipment manufacturers in their manuals or other technical documentation Statistical analysis techniques where it is possible to make repeated measurements Scientific publications Expert elicitation While much focus is given to quantifying uncertainties, the qualitative explanation of the causes of uncertainties are more important! 25

26 Uncertainty Information Increasing uncertainty Census Survey Empirical data Expert judgment Complete data. If well designed, it should have small errors Data based on sampling. Errors should be quoted or determined Should have quoted errors derived from measurements Experts should give range of possible value or mean and uncertainty

27 Expert judgement Elicitation protocol is key to generating uncertainty estimates that are credible and defensible Includes procedures and techniques to motivate and condition the experts The 2006 IPCC Guidelines provide help

28 Key elements of an elicitation protocol Motivating: Establish a rapport with the expert, describe the context of the investigation, explain the most commonly occurring biases. Structuring: Clearly define the quantities for which judgements are to be sought. (e.g., resulting emissions or removals should be for typical conditions averaged over a one-year period). Conditioning: Work with the expert to identify and record all relevant data, models, and theory relating to the formulation of the judgements. Encoding: Request and quantify the expert s judgement on the value of the selected parameters. While doing this request information (quantitative and qualitative) on what they judge the uncertainties in these values. Verification: After you have analyzed input from experts, provide them feedback to confirm that you have properly encoded and interpreted their judgments. This process also gives the experts a chance to consider if they want to add anything to their answers. Document all information provided by experts.

29 Approaches to combining uncertainties Parameter uncertainties combined to provide: Uncertainty estimates for the emission or removal category Uncertainty estimates for the entire inventory in any year and The uncertainty in the overall inventory trend over time IPCC guidelines include two approaches Approach 1: Error propagation Approach 2: Monte Carlo simulation The bulk of your effort and resources should be on improving inventory estimates rather than quantifying uncertainty.

30 Approach 1: Error propagation The same equation to estimate emissions is used to combine uncertainties for each category Assumptions and Requirements All parameter uncertainties are symmetric and in the form of a normal (Gaussian) probability distribution Is best applied where uncertainties are less than ±30%

31 Error propagation (Product Rule)

32 Error propagation (Summation Rule)

33 Error propagation (Summation Rule) Equations for calculating total emissions (E T ) for source category 1 (E 1 ) and 2 (E 2 ) E T = E 1 + E 2 or E T = E 1 -E 2

34 Uncertainty in trend - Sensitivities Emissions/removals estimates for base year and current year Type A sensitivity: the change in the difference in overall emissions between the base year and the current year, expressed as a percentage, resulting from a 1% increase in emissions or removals of a given category and GHG in both the base year and the current year. Primarily relevant for emission factors. Type B sensitivity: the change in the difference in overall emissions between the base year and the current year, expressed as a percentage, resulting from a 1% increase in emissions or removals of a given category and GHG in the current year only. Primarily relevant for activity data. Type A sensitivities: Uncertainties that are fully correlated between years Type B sensitivities: Uncertainties that are not correlated between years

35 Worksheet for uncertainty calculations using Approach 1 Input data

36 Keep in mind Approach 1 (error propagation) appropriate if Uncertainties are small (standard deviation/mean less than 0.3) Symmetric Uncorrelated If uncertainties are larger will tend to underestimate uncertainty

37 Approach 2: Monte Carlo simulation Simulation technique achieved by running multiple trials using randomly sampled inputs from a user defined distribution of values. This distribution for each input variable represents the uncertainty in that variable. Appropriate for estimating uncertainty when: Uncertainties of the individual components are large The distributions are asymmetric or not normal (non-gaussian) Correlations occur between some of the activity data sets, emission factors, or both Uncertainties are different for different years of the inventory.

38 Requirements for Approach 2 Monte Carlo Specify a probability density function, or PDF, for each input variable that reasonably represents the uncertainty in that variable. Obtained using statistical analysis or expert elicitation Based upon representative range of factors such as seasons and geography. Monte Carlo can deal with PDFs of any shape and width, as well correlations (both in time and between source/sink categories). Can deal with more complex models (e.g., the first order decay for CH 4 from landfills).

39 Illustration of steps for Monte Carlo simulation Step 1 Step 2 Select random value for AD Select random value for EF Select random value for AD Select random value for EF Step 3 Estimate emissions Add up all emissions Estimate emissions Step 4 Store values Calculate mean and uncertainty Mean and distribution constant? If Yes Finished If No Go to Step 2

40 Example of Monte Carlo results Results of a Monte Carlo simulation after: 1 iteration 50 iterations 100 iterations 1,000 iterations 10,000 iterations

41 Example of Monte Carlo results (1 iteration) Frequency 1,2 1 0,8 0,6 0,4 0,2 0 0,4 0,45 0,5 0,55 0,6 Emissions (Gg CO 2 eq)

42 Example of Monte Carlo results (50 iterations) Frequency ,4 0,45 0,5 0,55 0,6 Emissions (Gg CO 2 eq)

43 Example of Monte Carlo results (100 iterations) Frequency ,4 0,45 0,5 0,55 0,6 Emissions (Gg CO 2 eq)

44 Example of Monte Carlo results (1,000 iterations) Frequency ,4 0,45 0,5 0,55 0,6 Emissions (Gg CO 2 eq)

45 Example of Monte Carlo results (10,000 iterations) Frequency ,4 0,45 0,5 0,55 0,6 Emissions (Gg CO 2 eq)

46 Choosing an approach Where the conditions for applicability are met, Approach 1 and Approach 2 will give the same results. Approach 1 is less effort to apply. Although once Monte Carlo software is set up, it can be easy to apply Approach 2. But set up can be a great deal of effort. Approach 2 will provide a better results in terms of the thoroughness of the uncertainty estimates.

47 Reporting and documentation Uncertainty documentation should address for each parameter: Causes of uncertainty What uncertainties were quantified Source of any data used as a basis to estimate uncertainty Methods used to combine uncertainties Background information for expert judgment analysis Explanation of any correlation found between inputs Explanation of any special consideration unique to the country/inventory Explanation of differences in results between Approaches 1 and 2

48 Using and interpreting uncertainties

49 Uncertainty assessment in practice In the context of national GHG inventories. Who are the users of uncertainty information? How do you collect uncertainty information? How can and should they use it? How do you communicate the results of your uncertainty assessment work to stakeholders?

50 Users

51 Users of uncertainty information Uncertainty information provided by a Party under the UNFCCC may be used by: Scientific and research community Policy makers and negotiators Public and other stakeholders IPCC and other methodology developers UNFCCC/KP Expert Review Teams Inventory compilers

52 Stakeholders in uncertainty data What do these groups want to use uncertainty information for? How and should these groups use and interpret GHG inventory uncertainty information?

53 Scientific and research community Wants Statistically valid and detailed uncertainty values for all variables May assume uncertainty estimates have origins similar to experimental data Reality Need to understand the role expert judgment and elicitation play Potential for biases in uncertainty data between categories and countries

54 Policy makers and negotiators Wants Simple to understand assessments of data quality. ( Can we trust it? ) To use quantitative uncertainty estimates as an eligibility or enforcement tool Reality Uncertain estimates should in most cases NOT be used for policy making (we will discuss)

55 Public and other stakeholders Wants Enormously diverse Often don t care or understand Reality Need to educate as to the difference in uncertainties at national level versus at a project or facility level.

56 IPCC and other methodology developers Wants Basis for assessing quality of methods (methodological uncertainty) and factors (parameter uncertainty for guidelines and EFDB Reality Need to understand how uncertainty estimates were developed what types of uncertainties included and excluded And what are the causes of the uncertainties

57 UNFCCC / Expert Review Teams Wants Data consistent with reporting requirements Tool to assess rigor of data quality management process Reality Goals realistic, as long as they follow UNFCCC reporting guidelines that clearly indicate that uncertainty data should not be used to compare across Parties.

58 Inventory compilers Wants Data suppliers: feedback on their data quality Inventory managers: quality management metric and tool Reality Wants are often realistic, as along as it is understood that uncertainty estimates may not be comparable

59 Collection of information

60 Parts of the uncertainty assessment process 1. The rigorous investigation of the likely causes of data uncertainty and quality 2. the creation of quantitative uncertainty estimates and parameter correlations for each variable 3. the mathematical combination of estimates using a statistical model (e.g., first-order error propagation or Monte Carlo) 4. the selection of inventory improvement actions in response to results of uncertainty assessment 5. Communication of uncertainty assessment results

61 Where to focus? We tend to focus on parts 2 and 3 (2) creation of uncertainty estimates and (3) their mathematical combination Often little attention is given to parts 1 and 4 (1) the investigation of causes and (4) inventory improvements Part 1 is best thought of as detective or investigatory work, results are largely qualitative Part 5, communication, is often done poorly, given the potential for misuse of results by stakeholders

62 Org chart for the process Overall Inventory lead Uncertainty assessment coordinator QA/QC Officer Source/sink category lead Source/sink category lead Source/sink category lead Source/sink category lead Outside expert

63 Roles and coordination Inventory Lead: overall director responsible for supervising the uncertainty assessment for the entire Inventory and communicating results

64 Roles and coordination Source Category Leads: responsible for making decisions and performing uncertainty assessment on their specific source or sink categories. determining the appropriate level of disaggregation for data collection and uncertainty model development, prioritizing the variables for input data collection efforts and allocating resources to collect uncertainty information, identifying experts for elicitation reviewing results of uncertainty analysis and identifying corrective actions and improvements

65 Roles and coordination The Uncertainty Analysis Coordinator: responsible for directing the assessment of uncertainty for entire Inventory obtaining data inputs eliciting expert judgments with Source Leads developing the uncertainty model developing quantitative uncertainty estimates Ensuring that qualitative information on causes of uncertainty is documented interpreting the results of the uncertainty analysis

66 Roles and Coordination QA/QC Officer directs the overall implementation of QA/QC supervising QA/QC staff overseeing the expert reviews ensuring the full and adequate implementation of QA/QC elements adequate qualifications of source category staff and contractors

67 Roles and Coordination Outside Experts independent individuals who may contribute data to the inventory estimation (i.e., data suppliers ), may be involved in improving / examining inventory methods, data, or report may serve as expert for elicitation purposes

68 What is uncertainty information? We often focus on the number ±9% But uncertainty information has qualitative as well as quantitative components. Although we focus on the quantitative, it is the qualitative information that is often the most useful. Do we care more about the number or about why the uncertainty is there in the first place?

69 Quantitative uncertainty information Types of data Distribution Standard deviation or standard error Their associated confidence intervals Upper and lower bound of variables and their associated cumulative probability levels In many cases, however, little or none of this quantitative uncertainty data is available for a variable.

70 Qualitative uncertainty information Descriptions of the causes or likely causes of uncertainties Understanding of how uncertainties related to data collection process References for uncertainty information Publications Background and qualifications of experts elicited Elicitation protocols

71 Setting up your uncertainty model: identifying variables The uncertainty estimation methodology based on inventory estimation equations Equations may be identical to inventory estimation methodology, or Equations may be a simplified version of the inventory estimation methodology The mathematical models underlying both inventory and the uncertainty methodologies must yield the same emission estimates. Levels of aggregation for variables are likely to be the main difference between the inventory and the uncertainty estimation methodologies.

72 Setting up your uncertainty model: identifying variables Uncertainty estimation methodologies tend to have fewer variable components (i.e., are less disaggregated). The level of variable disaggregation for uncertainty estimation for each variable in every source and sub-source category should be determined by: the availability of uncertainty data inputs, resource availability and the importance of the source/sub-source category (e.g., if it is a key category).

73 Collecting information: variable by variable Uncertainty data collected at the individual variable level (e.g., EF, activity data) The source category lead must work with uncertainty coordinator to define the appropriate level of disaggregation for uncertainty model Example: Cattle manure may be estimated by province. But one equation at a national level is used for the uncertainty model A single uncertainty value is then needed for the manure EF, and Monte Carlo model is simplified

74 Elicitation If published uncertainty data insufficient, then expert elicitation will likely be necessary Types of elicitation Informal interview : eliciting uncertainty information from the inventory experts that are directly involved in the inventory process (e.g., source category leads, contractors, etc.) Expert elicitation : formally eliciting uncertainty information from outside experts

75 Elicitation Informal interview is a modest process of elicitation, involving discussions with inventory experts that are intimately familiar with the inventory source category.

76 Elicitation A less formal elicitation or informal interview may be warranted where: the emission source is not a key inventory source category and so is a small contributor to inventory emissions only a small number of variables for the category have missing data knowledgeable outside experts on the underlying variables are unavailable resources for conducting a formal elicitation are limited

77 Integration with QC Tier 2 source-specific QC checks and investigations into input data quality and uncertainty investigations require contacting the same persons or organizations. QC and uncertainty assessment processes should be integrated! Investigations should be done as one Avoid duplicative questions and multiple points of contact.

78 When to collect? Uncertainty information may not need to be collected every year for every source or variable A plan should be developed specifying frequency Uncertainty and Tier 2 QC investigations should have one timeline Key categories prioritized in terms of immediacy and frequency Some variables may only need to be revisited every few years For each variable, the year in which uncertainty information is collected must be documented.

79 Trend uncertainty Estimate bias may not be constant from year to year It may exhibit a pattern (e.g., growing or falling) For example, if your data supplier cuts his budget for data collection, then the data he provides you may be getting worse over time. Assuming biases in estimates cancel out when estimating trend uncertainty, may be incorrect

80 Using uncertainty information

81 How should you use uncertainty information? National inventory users and stakeholders want to apply uncertainty information in a number of ways Two main types of applications 1. Comparative 2. Quality management Each application requires uncertainty information to have certain characteristics

82 Comparative applications Policy makers (as well as some other stakeholders) want to use uncertainty information to make decisions or support compliance systems. To compare: Countries Source categories Sectors Facilities Projects, etc.

83 Comparative applications For uncertainty information to be used for international compliance applications it must: 1. Be comparable across countries 2. Be relatively objective and subject to review and verification 3. Not be subject to gaming by countries acting in their own self-interest 4. Be administratively feasible to estimate 5. Be of high enough quality to warrant the compliance costs imposed on countries (e.g., through adjustments) 6. Attempt to address all types of uncertainty National GHG inventory uncertainty estimates in most cases do not have these characteristics!

84 Comparative applications Quantitative uncertainty estimates are often based on expert judgments Only through incredibly rigorous elicitation can you avoid significant subjectivity in expert judgment They are unlikely to be sufficiently comparable across countries, categories, parameters, or time, because of differences across the experts.

85 Why can t I compare? Expert judgments do not currently undergo any rigorous review Without a costly and rigorous review process, they will be ripe for manipulation and gaming It is currently impractical to ensure that uncertainty estimates across Parties are done in a consistently objective fashion. UNFCCC inventory review process is already enormously strained

86 Why can t I compare? The uncertainty in uncertainty estimates is typically vastly greater than the uncertainty in the inventory estimate itself.

87 Quality management One goal for inventory compilers is continuous improvement of emission and removal estimates. From a quality management perspective, uncertainty assessment is a structured way to investigate, conceptualize, and track data quality. From this perspective, uncertainty assessment is just part of your QA/QC program

88 Quality management Quantitative uncertainty estimates can be useful But in isolation, they do not provide information needed to isolate the causes of data quality problems so they can be corrected. An investigation focused approach to uncertainty assessment focuses on Parts 1 and 4 of the process (investigation and improvement). This approach also provides more verifiable information and justifications to support quantitative uncertainty estimates.

89 Quality management An investigation-focused approach requires that you work closely with data suppliers to: 1. Exchange information on the inventory s data quality requirements and data collection practices 2. Identify and understand activity data reporting and collection problems 3. Identify situations where there is a lack of empirical data for emission factors or other parameters 4. Identify situations where the variability in an inventory parameter is high 5. Identify situations where there is a lack of scientific consensus of the appropriate estimation method for an inventory parameter or category 6. Identify specific actions that can be taken to correct or mitigate problems

90 Quality management By working jointly with data suppliers you can Educate them on your needs Help them identify specific causes of uncertainty and the magnitude of their effect on data quality Create pressure for investments in data quality improvements (e.g., expanded data collection or more research)

91 Quality management The required characteristics of uncertainty information are less strict if they are only used prioritizing inventory improvements It is less critical that uncertainty estimates be objective and comparable because they do not have compliance implications

92 Summary You do not have to choose between an investigation-focused and Monte Carlo type uncertainty assessment approach. The former will obtain better results for the latter. However, where resources are limited, inventory quality will likely benefit the most if resources are focused on data quality investigations and improvements (Parts 1 and 4), rather combining subjective uncertainty estimates

93 Summary We wrongly assume that uncertainty estimation will automatically lead to inventory quality improvements In reality, we need a process designed to investigate and assess the causes of poor quality And a feedback process that leads to the implementation of measures for improvements.

94 Communication

95 Communicating uncertainty information A GHG inventory is not a purely scientific exercise. We acknowledge this by aggregating estimates of varying uncertainties without regard to rules of significant figures. It also has accounting and legal compliance characteristics Do accountants report their profit and loss statements with a ±35%? Most stakeholders will only use or pay attention to the point estimates in an inventory. When looked at, quantitative uncertainty estimates are, more often that not, misused

96 Communicating uncertainty information When communicating uncertainty information, focus on explaining the causes of uncertainties in key underlying variables, rather than quantified uncertainty estimates. Always include discussion of what measures are being taken to monitor and improve data quality If you must discuss aggregate quantitative uncertainty estimates, focus on the uncertainty in the trend, rather than annual totals

97 - son -

98 Exercise: Approach 1 - Propagation of error You have estimated emissions (in Gg CO 2 -eq) from a number of activities For this example we will consider 4 categories and different GHGs for two years: 2000 and 2010 You have obtained uncertainty values for the activity data and emission factors used Your data are listed in a table IPCC category Gas 2000 emissions 2010 emissions Activity data uncertainty Emission factor uncertainty 1.A.3b Road transportation Gasoline CO ±2% ±2% 1.A.3b Road transportation - Gasoline CH ±1% ±50% 2.1 Nitric acid production N 2 O ±5% ±100% 3.A.2 Manure management N 2 O ±15% ±160%

99 Prepare your data First, calculate the total emissions for both years of the inventory. IPCC category Gas 2000 emissions 2010 emissions Activity data uncertainty Emission factor uncertainty 1.A.3b Road transportation Gasoline CO % 2% 1.A.3b Road transportation - Gasoline CH % 50% 2.1 Nitric acid production N 2 O % 100% 3.A.2 Manure management N 2 O % 160% Total Next, ensure that you use the absolute values of the uncertainties for both the activity data and the emission factors.

100 Step 1: Combine the uncertainties For each category combine uncertainties of the activity data and emissions factors. Column G is the combined uncertainty by category derived from the data in Columns E and F using the error propagation equation (Equation 3.2). The entry in Column G is the square root of the sum of the squares of the entries in Columns E and F. IPCC category G a s emi ssio ns emi ssio ns Activ ity data unce rtaint y Emis sion facto r unce rtain ty Com bine d unce rtain ty A B C D E F G= ( E 2 +F 2 ) 1.A.3b Road transportation Gasoline 1.A.3b Road transportation - Gasoline C O 2 C H % 2% 3% % 50% 50% 2.1 Nitric acid N % 100% 100%

101 Step 2: Calculate the percentage uncertainty for the latest year 1. For each category, calculate the uncertainty in Column G as a percentage of total emissions in the latest year (2010) (also referred to as contribution to the variance ) and record the value in Column H. The entry in each row of Column H is the square of the entry in Column G multiplied by the square of the entry in Column D, divided by the square of total at the foot of Column D. 2. Calculate the total contribution by summing the entries in Column H. For our example, the total is ΣH= Take the square root of the sum to estimate the uncertainty in the total inventory in the latest year. For our example, the total % uncertainty for the year 2010 is 19.7%. IPCC category Gas 2000 emission s 2010 emission s Activity data uncertainty Emission factor uncertaint y Combined uncertainty A B C D E F G= (E 2 +F 2 ) Contributio n to the variance H= (G D) 2 /(ΣD) 2 1.A.3b Road transportation Gasoline 1.A.3b Road transportation - Gasoline CO % 2% 3% CH % 50% 50% Nitric acid production N 2 O % 100% 100% A.2 Manure management N 2 O % 160% 161% Total

102 Step 3: Estimate the trend uncertainty To determine uncertainties in the trend involves the following intermediate steps: 1. Estimate the Type A sensitivity 2. Estimate the Type B sensitivity 3. Estimate trend uncertainty due to EF uncertainty 4. Estimate trend uncertainty due to AD uncertainty 5. Combine the results in steps 3. and 4. above to estimate total trend uncertainty Notes: The Type A and Type B sensitivities are intermediate variables that simplify the calculation procedure. The results of steps 3. and 4. by themselves do not provide uncertainty information for specific categories. The results of these steps combined (step 5.) provide an indication of the trend uncertainties.

103 Step 3.1: Estimate the Type A sensitivity We first calculate the Type A sensitivity for each category as the percentage difference in emissions between the base year and the current year in response to a 1% increase in category emissions/removals in both the base year and the current year. Type A shows the sensitivity of the trend in emissions to a systematic uncertainty in the estimate. The equation for this is: Absolute value of: 0.01 D x + ΣD i (0.01 C x + ΣC i ) 100 (0.01 C x + ΣC i ) ΣD i ΣC i ΣC i 100 Where: Cx, Dx = column C or D value in category x ΣCi, ΣDi = Sum of column C or D values over all categories (rows) of the inventory For example, for Category 1.A.3b the equation Type A sensitivity value is calculated as follows: ( ) ( ) =

104 Step 3.1 (cont): Estimate the Type A sensitivity Performing the same calculation for all categories, you will get the following values for Type A sensitivity. IPCC category Gas 2000 emissions 1.A.3b Road transportation Gasoline 1.A.3b Road transportation - Gasoline 2010 emissions Activity data uncertainty Emission factor uncertainty Type A sensitivity A B C D E F I CO % 2% CH % 50% Nitric acid production N 2 O % 100% A.2 Manure management N 2 O % 160% Total

105 Step 3.2: Estimate the Type B sensitivity In Column J of the worksheet you can calculate the the Type B sensitivity as the percentage difference in emissions between the base year (2000) and the latest year (2010) in response to a 1% increase in category emissions/removals in the 2010 only. This shows the sensitivity of the trend in emissions to random error in the estimate. IPCC category Gas 2000 emissions 2010 emissions Activity data uncertainty Emission factor uncertainty Type A sensitivity Type B sensitivity A B C D E F I J= Abs(D/ΣC) 1.A.3b Road transportation Gasoline 1.A.3b Road transportation - Gasoline CO % 2% CH % 50% Nitric acid production N 2 O % 100% A.2 Manure management N 2 O % 160% Total

106 Step 3.3: Estimate the trend uncertainty due to emission factor uncertainty In Column K of the worksheet you multiply together the values in Columns I and F to estimate the uncertainty introduced into the trend in emissions by emission factor uncertainty, under the assumption that uncertainty in emission factors is correlated between years. If you determine that the emission factor uncertainties are not correlated between years then the Type B sensitivity (Column J) should be used in place of that in Column I and the result multiplied by the square root of 2. IPCC category Gas 2000 emissions 2010 emissions Activity data uncertainty Emission factor uncertainty Type A sensitivity Trend uncertainty due to EF uncertainty A B C D E F I K= I F 1.A.3b Road transportation Gasoline 1.A.3b Road transportation - Gasoline CO % 2% % CH % 50% % 2.1 Nitric acid production N 2 O % 100% % 3.A.2 Manure management N 2 O % 160% %

107 Step 3.4: Estimate the trend uncertainty due to activity data uncertainty In Column L of the worksheet you multiply together the information in Columns J and E to estimate the uncertainty introduced into the trend in emissions by activity data uncertainty, under the assumption that uncertainty in activity data is not correlated between years. If you determine that the activity data uncertainties are correlated between years then the Type A sensitivity value (Column I) should be used in place of that in Column J and the square root of 2 factor does not then apply. IPCC category Gas 2000 emissions 1.A.3b Road transportation Gasoline 1.A.3b Road transportation - Gasoline 2010 emissions Activity data uncertainty Emission factor uncertainty Type B sensitivity A B C D E F J= Abs(D/ΣC) Trend uncertainty due to AD uncertainty L= J E 2 CO % 2% % CH % 50% % 2.1 Nitric acid production N 2 O % 100% % 3.A.2 Manure management N 2 O % 160% % Total

108 Step 3.5: Estimate the total trend uncertainty Estimating the total trend uncertainty involves the following: 1. In Column M you calculate the uncertainty introduced into the trend in total emissions by the category in question. This is derived from the values in Columns K and L using Equation 3.2. The entry in Column M is the sum of the squares of the entries in Columns K and L. 2. The estimate of the total uncertainty in the trend is then calculated from the entries in Column M by using the error propagation equation. This total is obtained by summing the entries in Column M (ΣM=0.87%) and then taking the square root of the sum (9.32% for our example). The uncertainty in the trend is a percentage point range relative to the inventory trend. In our example, the 2010 emissions are 18% percent greater than the 2000 emissions (from 6739 in 2000 to 7936 in 2010), and since we have estimated the uncertainty as 9.32%, then the trend is 18% ±9.32% (or from about 9% to 27% increase) for the 2010 emissions relative to the 2000 emissions. IPCC category Gas 2000 emissions 2010 emissions Activity data Emission factor uncertainty Trend uncertainty Trend uncertainty Total trend uncertainty