Characterizing the long-term PM mortality response function: Comparing the strengths and weaknesses of research synthesis approaches

Transcription

1 Characterizing the long-term PM mortality response function: Comparing the strengths and weaknesses of research synthesis approaches Neal Fann*, Elisabeth Gilmore & Katherine Walker* 1 * Usual institutional disclaimers

2 Why study PM 2.5? Large estimates of benefits and costs of regulations total ann. ( ) in billions $ 2 Source:

3 Benefits are primarily from reduced mortality Mostly from reductions in particulate matter, PM2.5 3 Measuring Clean Air Act Progress: Costs vs. Benefits (Section 812 Prospective Analysis)

4 Hot issue: What is the PM 2.5 -mortality concentration response relationship? Magnitude of relationship between exposure and response (e.g. relative risk) Shape of the function (e.g. linear, nonlinear, threshold effects, etc ) Level of confidence or uncertainty Likelihood that relationship is causal. 4

5 Background: Risk analyses have relied heavily on a few key epidemiologic cohort studies All other Epidemiologic studies Harvard 6-City Study (Dockery 1993) ACS study ( Pope et al. 1995) ACS study ( Pope et al. 2002) Number of papers in Pub Med 5 0 Year Published

6 Background: Publications on air pollution and particulate matter have exploded over time All other Epidemiologic studies Harvard 6-City Study (Dockery 1993) ACS study ( Pope et al. 1995) ACS study ( Pope et al. 2002) Harvard 6-City Study (LePeule, 2012) 600 HEI reanalysis 6C &ACS study (Krewski, 2009) Number of Published papers in Pub Med 6 0 Year Published

7 Where are we on the continuum from ignorance to perfect information? No information Perfect information More Judgment More Knowledge 7

8 Compare 4 research synthesis approaches 8 Approach Systematic reviews Formal elicitation of expert judgments Meta-analysis Integrated exposure response assessment Examples EPA s Integrated Science Assessments (ISA) for Particulate Matter EPA (2004) pilot PM 2.5 elicitation of 5 experts EPA (2006) extended PM 2.5 elicitation of 12 experts Illustrative analyses using, preand post-2006 cohort studies Shin et al. (in this series)

9 Systematic Reviews 9 Application: EPA lntegrated Science Assessments (ISA) for Particulate Matter 5-year reviews under the Clean Air Act Used for qualitative judgments about likelihood of a causal relationship Informs CASAC decisions on National Ambient Air Quality Standards (NAAQS) Basis for identification of studies most appropriate for quantitative risk analyses Primary analysis - American Cancer Society (ACS) study Sensitivity analysis Harvard Six City (H6C) study

10 Expert Judgment Elicitation 10 Application: EPA pilot (2004) and extended (2006) PM 2.5 elicitation Multi-disciplinary expert selection through stratified peer nomination process Briefing materials and detailed interview protocol Full-day interviews to elicit quantitative assessments: Likelihood of a causal relationship, threshold % change in all-cause mortality per 1 µg/m 3 decrease in PM 2.5 under a specific scenario Credible intervals (5 th, 25 th, 50 th, 75 th, 95 th )

11 Illustrative meta-analyses Pooled hazard ratios % change in risk per 10 ug/m 3 Studies available pre (available to expert elicitation) 11 Updates to pre studies and new studies available up to mid-2013

12 Evaluation approach #1 -Criteria 12 Broad Questions How valid are the methods and results? How transparent are the methods and results? How suited is are the data, methods, and results to the policy problem? How suited is the method to the user needs and resource constraints? Attributes Appropriate disciplines involved Completeness of data and analysis Analytical methods appropriate to data Verification/Validation of methods Rigorous peer review How analysts are selected and represented How cognitive biases are dealt with Clarity of hypotheses, models, assumptions Exploration of variability and uncertainty in inputs and results Relevance External validity Transferability/Transportability Ability to use existing data Value of output commensurate with costs Repeatability/Updateability Communicability

13 #2 - Illustrative Policy Application: EPA Mercury and Air Toxics Rule (2011) Applied the C-R functions from the different methods to the benefits analysis of the avoided PM 2.5 -related premature deaths due to air quality improvements Used air quality model results from the Regulatory Impact Assessment (RIA) for the Mercury and Air Toxics Rules 13 Combined with the population and incidence values from the environmental Benefits Mapping and Analysis Program Community Edition (BenMAP-CE) v0.63 (USEPA, 2013)

14 Observations: PM ISA Strengths Essential starting point for any research synthesis method Systematic selection of studies virtually all published studies Over time, has improved structured consideration of evidence Multi-disciplinary teams Challenges 14 Qualitative assessment Hard to know how disparate evidence is ultimately weighed and integrated Methods for structured evaluation of evidence evolving (e.g. Cochrane reviews, PRISMA, National Tox. Program - Office of Health Assessment and Translation (OHAT), EPA Next Gen Chemical Risk Assessment)

15 Observations: Expert Judgment Elicitation 15 Strengths Structured, explicit characterization of experts judgments about what they know and don t know Extent of and basis for inter-expert agreement or disagreement Quantitative estimates of the quantity of interest (e.g. causal likelihood; existence of thresholds, and more complete assessment of uncertainty) Uses available evidence, even limited Multidisciplinary Independent, structured process for selection of experts Challenges Assessing the quality of judgments: How well do experts estimate the truth and how well they know it? Dealing with strategic judgments in highly political debates Capturing/communicating transparently the basis for judgments Whether and how best to combine experts Harder to update easily

16 Observations: Meta-Analyses Strengths Structured, explicit approach for selection and combination of evidence Draws strength from multiple studies Easy to test sensitivity to existing studies and updatable with new information Simpler to communicate Transparent Challenges 16 Pooling estimates from studies can be problematic Can only reflect the existing studies, their biases and uncertainties (e.g. standard errors)

17 Observations: Integrated Exposure Response Strengths Structured, explicit approach for selecting and combining evidence Draws from multiple studies and types of exposures Quantitative estimates of the shape of the CR function over broader concentration range and more complete assessment of uncertainty Has been tested against real-world results (e.g. China) Easier to conduct sensitivity analyses and to update with new information Challenges 17 Requires a lot of compatible evidence from different exposures It is still a model Evidence for low level population exposures still limited Would benefit from replication/repetition/validation by others with alternative assumptions and choices of data

18 Comparison of Avoided Premature Deaths Using Different C-R functions for the Mercury and Air Toxics Rule * 18 * Results for IER analysis not yet available

19 Summary Systematic, structured, critical review is essential to all these research synthesis methods Growing number of guidelines in many disciplines Quantitative exploration and analyses needs to be suited to the data and question of interest or decision The PM 2.5 example shows that more data supports more comprehensive and sophisticated analysis The wealth of data we have for PM 2.5 is rare All of these methods require the considerable judgment of scientists, individually and collectively 19 Challenge is to recognize when analyses may be sensitive to differences in judgment and might benefit from more structured examination