Expert Judgment about Uncertainty in PM 2.5 -Mortality: What Have We Learned?

Size: px

Start display at page:

Download "Expert Judgment about Uncertainty in PM 2.5 -Mortality: What Have We Learned?"

Andrew Briggs
6 years ago
Views:

1 Expert Judgment about Uncertainty in PM 2.5 -Mortality: What Have We Learned? Katherine D. Walker, Henry Roman Industrial Economics, Inc. Patrick Kinney Columbia University Lisa Conner, Harvey Richmond, Bryan Hubbell, Bob Hetes, Mary Ross, Zach Pekar US Environmental Protection Agency

2 Overview EPA PM 2.5 Mortality Expert Judgment Elicitation Project Promises and Pitfalls Challenges

3 USEPA Pilot Expert Elicitation of PM 2.5 -Mortality Relationship: Impetus for Study Premature deaths avoided by reduction of PM 2.5 constitute % of monetized benefits $93 billion in reduced mortality (U.S. EPA Clean Air Interstate Rule) National Academy of Sciences (2002). Estimating the Public Health Benefits of Proposed Air Pollution Regulations OMB Circular A-4A

4 USEPA s Primary Benefit Analysis Uncertainty in the mortality estimate is characterized by the confidence interval from the standard error of one epidemiological study, Pope et al. (2002). Why might we need anything else?

5 Uncertainties in PM 2.5-Mortality Relationship How strong is the likelihood of a causal relationship? What is the true shape of the dose-response relationship? Threshold? What is the impact of confounders and effect modifiers? How do potential errors in measuring exposure influence results? What is the impact of relative toxicity of PM components or sources?

6 Overview of PM Expert Elicitation Project Elements Elements Elicitation team (Walker, Kinney) Selection of experts Structured Protocol Pilot testing of protocol Pre-elicitation elicitation workshop Elicitation and verification of individual judgments External Peer Review Pilot Study 5 Full Study 12

7 Protocol Structure Preview & Assumptions Conditioning Questions: Issues and Evidence Quantify C-R Function Examine quantitative question Review Assumptions Mechanisms Conceptual framework for short- and longterm mortality Causality* Threshold Shape Re-visit causality Re-visit threshold Confounding Epidemiology evidence Effect Modification Other issues Exposure Assessment Specify percentiles of C-R function uncertainty distribution *Red items involve quantitative responses Re-visit

8 Pilot Elicitation Results: Comparison to Studies Used in EPA Analyses 2 % Increase in Mortality per 1 ug/m3 PM A B* C--8 ug/m3 C--10 ug/m3 C--15 ug/m3 Expert C--20 ug/m3 D E Pope et al., 2002 Primary Estimate Dockery et al., 1993; Krewski, 2000 Sensitivity Analysis

9 Promises Intellectual cross-fertilization in uncertainty analysis Well-defined questions Individual expert opinions Structured consideration of the evidence Comprehensive and explicit characterization of uncertainties typically left unquantified

10 Challenges Practical Methodological Political

11 Practical Cost This is not a low cost solution! Logistics involving experts Overuse/Conflicts Limits on numbers of experts imposed by federal regulations

12 Methodological Expert selection No one size fits all methodologies Who is an expert for complex multi-disciplinary questions? Assuring elicitation of informed but independent judgments Design of protocol and elicitation method Role and influence of the elicitors Identifying and eliminating motivational biases (real or perceived) Role of pre- or post-elicitation workshops How good are experts judgments about uncertainty?

13 Policy Challenges: We ve Characterized Uncertainty. Now what? Non-road Diesel Rule Annual Change in Mortality Incidence in ,000 Reduction in Premature Mortality in ,000 30,000 25,000 20,000 15,000 10,000 Note: Distributions labeled Expert A - Expert E are based on individual expert responses. The distribution labeled Combined Experts is based on the averaged distributions of reduced incidence of premature mortality across the set of experts. The distribution labeled Pope et al. (2002) Statistical Error is based on the mean and standard error of the C-R function from the study. 11,300 9,600 18,900 9,900 13,800 5,000 6,200 3,900 0 Expert A Expert B Expert C Expert D Expert E Combined Experts Pope et al (2002) Statistical Error

14 Recommendations Near term: Develop internal policies for dealing with quantitative measures of uncertainty Longer term: Applied research on expert judgment methodology on complex problems

15 Disclaimer The opinions, findings, and conclusions expressed are those of the authors and do not necessarily represent those of the U.S. EPA

Characterizing the long-term PM mortality response function: Comparing the strengths and weaknesses of research synthesis approaches

Characterizing the long-term PM 2.5 - mortality response function: Comparing the strengths and weaknesses of research synthesis approaches Neal Fann*, Elisabeth Gilmore & Katherine Walker* 1 * Usual institutional