Cotter Delivery October 14, 2016 Assessment (DSTA) Submittal Date / Rev February 9, 2017 Rev. 4

Transcription

1 Type Final Agency Comments Document in Review Proposed Scope and Approach to Completing the Data Summary Technical Cotter Delivery October 14, 2016 Assessment (DSTA) Submittal Date / Rev February 9, 2017 Rev. 4 GENERAL COMMENTS Cotter submitted (10/14/2016) a proposed approach for development of the Data Summary Technical Assessment (DSTA) per the AOC/SOW. Cotter s proposed approach limits the amount of data for DSTA review based on relevance to current site conditions, planning work scope for the Remedial Investigation/Feasibility Study (RI/FS), the newly created Quality Assurance Project Plan (QAPP), setting certain limits for data age, and opting to collect newer site data in exchange for more detailed historical reviews. The following Agency (EPA and CDPHE) comments and examples for the DSTA proposed approach are provided. Cotter proposes using current data, more specifically to initially look at data gathered in the last two or five years, depending on the type of media. EPA and CDPHE agree to the concept of using more recent numerical data where feasible with the following caveats: - The DSTA needs to include a minimum of the most recent five years of numerical data for evaluation for all media sources. - EPA and CDPHE also concur with the use of the additional data sources listed in Table 2 and 3 of the Cotter s proposal. - Where past investigations were conducted prior to the most recent five years, but were important or unique in defining site constituents, those data must be included for example, groundwater investigations conducted in 2011 for TCE. - Datasets that identify site Constituents of Potential Concern (COPCs) relevant to site constituents need to be retained and evaluated in the DSTA. We note that this five year window applies only and specifically to numeric data and does not apply to documents, maps, etc. The agencies will need and/or use this older documentation to assist in completing their comprehensive review and overseeing the Remedial Investigation, Feasibility Study, and Risk Assessment. Additionally, the CD/RAP requires specific sampling or remedial activities for various areas outside the mill or the Lincoln Park area, including Willow Lakes (RAP Section 26), Ephemeral Streams of Willow Creek (RAP Section 27), Fremont Ditch (RAP Section 27), Perennial Streams (RAP Section 28), site adjacent soils and run-off contamination to ephemeral drainages (Sand Creek, Willow Creek, Forked Gulch, Wolf Park, and Chandler Creek (RAP Section 24, 29), Arkansas River (RAP Section 30), and Minneque and Pueblo Reservoirs (RAP Section 31). Some of these areas are located outside the OU1 and OU2 boundaries and could be potential OU3 areas. Each of these have been investigated and sampled for groundwater, surface water, soil, sediment, etc. between 1986 and 2000 (except for the Arkansas River which has been and is currently sampled on a quarterly basis). Since most of these historical data will not be used in the proposed DSTA approach, these areas should be included in the data gap analysis and the RI. Any other historical data that are not evaluated in the DSTA but could provide valuable information on previous site conditions and contaminations should also be included in the data gap analysis and the RI. Cotter s proposed methodology for the DSTA utilizes the general assessment factors (GAFs) identified in Table 2 of the EPA Guidance on Systematic Planning Using the Data Quality Objectives Process EPA QA/G-4 (EPA 2006). However, the Agencies believe the proposed methodology needs to be enhanced for objectivity, utility, and integrity by implementing the Elements of Systematic Planning in Table 3 of Page 1 of 15 FINAL

2 the referenced guidance to meet the intent of the GAFs. The following summary table shows the commonalities between systematic planning elements and GAFs from Table 3 of the referenced guidance, including definitions of the systematic planning elements, general assessment factors, data utility and integrity, and an example of applicable evaluation and exclusion criteria for consideration. These are provided as examples, but are not intended to be all inclusive or to limit the finalized DSTA. The following process chart (Page 10) provides a general approach to evaluation of data for the DSTA process with examples and sample criteria for how numerical data and analytical documentation would be processed and dispositioned. The Agencies expect Cotter to provide further details/clarifications in development of what questions and exclusion criteria Cotter will utilize to evaluate data and its applicability for future use, utilizing Table 3 and the process chart in your amended proposal. These questions can be drawn from the examples provided below and/or additional questions Cotter formulates, establishes, and/or uses in the DSTA process. OBJECTIVES DSTA / QAPP REFERENCES QAPP, Section Data Summary Technical Assessment page 19 of 100. Task 7 of the SOW requires that a DSTA shall be developed summarizing existing data in a manner designed to help the agencies determine which data meet the acceptance requirements developed in this QAPP and can be used in the RI and risk assessment. The DSTA will include a complete set of existing data for all media, a recommendation regarding which data meet QAPP requirements and a brief explanation of how the data meet the data acceptance criteria established in this QAPP (See Section 2.4). QAPP, Assessment of Existing Data, Page 29 of 100. These data have largely been compiled into a SCRIBE database. The SCRIBE database is hosted on the internet by the EPA. As additional routine environmental monitoring data are developed, these data will be provided to the agencies as updates to the SCRIBE database. Existing data are assessed for the RI/FS process to optimize RI/FS data collection by minimizing the time and resources allocated for collection of new data where sufficient data may already exist. The goal of assessing the existing data is to determine which data are sufficient to support assessment of COPC nature and extent, assess current human and ecological risks, or screening of potential remedial action alternatives. Therefore, Cotter must determine which of the existing available data are appropriate to support assessment of the specific study topics. QAPP Assessment of Existing Data, Page 35 of 100. The scale of decision-making for the existing data will generally be in batches based on their respective sources. For example, data from a given previous report or study will be evaluated considering the general assessment factors ( GAF ) identified in Table 2 of the EPA QA/G-4 guidance for the DQO process (EPA, 2006a) as well as the available QA/QC data. Cotter does not expect to frequently evaluate individual data values against the GAF. Page 2 of 15 FINAL

3 EPA QA/G-4 Guidance on Systematic Planning Using the Data Quality Objectives Process Elements of Systematic Planning Table 3. Commonalities Between EPA s GAFs for Evaluating the Quality of Scientific and Technical Information and the Elements of Systematic Planning Soundness Applicability and Utility General Assessment Factors Clarity and Completeness Organization Project Goal Schedule Uncertainty and Variability Evaluation and Review Data Needs Criteria Data Collection Quality Assurance Analysis REFERENCES EPA/240/B Feb 2006 Page 3 of 15 FINAL

4 GENERAL ASSESSMENT FACTORS Definitions and Considerations Soundness The extent to which the scientific and technical procedures, measures, methods or models employed to generate the information are reasonable for, and consistent with, the intended application. 1. Is the purpose of the study reasonable and consistent with its design? 2. To what extent are the procedures, measures, methods, or models employed to develop the information reasonable and consistent with sound scientific theory or accepted approaches? 3. How do the study s design and results compare with existing scientific or economic theory and practice? Are the assumptions, governing equations and mathematical descriptions employed scientifically and technically justified? Is the study based on sound scientific or econometric principles? 4. In the case of a survey, have the questionnaires and other survey instruments been validated (e.g., compared with direct measurement data)? 5. Were checks for potential errors made during the interview process? 6. How internally consistent are the study s conclusions with the data and results presented? Applicability and Utility The extent to which the information is relevant for the Agency s intended use. 1. How useful or applicable is the scientific or economic theory applied in the study to the Agency s intended use of the analysis? 2. How relevant are the study s purpose, design, outcome measures and results to the Agency s intended use of the analysis (e.g., for a chemical hazard characterization)? 3. Are the domains (e.g., duration, species, exposure) where the model or results are valid useful to the Agency s application? 4. How relevant is the study to current conditions of interest? For example, in the case of a survey, are conditions likely to have changed since the survey was completed (i.e., is the information still relevant)? Is the sampled population relevant to the Agency s current application? How well does the sample take into account sensitive subpopulations? The degree of clarity and completeness with which the data, assumptions, methods, quality assurance, sponsoring organizations and analyses employed to generate the information are documented. 1. To what extent does the documentation clearly and completely describe the underlying scientific or economic theory and the analytic methods used? 2. To what extent have key assumptions, parameter values, measures, domains and limitations been described and characterized? 3. To what extent are the results clearly and completely documented as a basis for comparing them to results from other similar tests? 4. If novel or alternative theories or approaches are used, how clearly are they explained and the differences with accepted theories or approaches highlighted? 5. Is the complete data set accessible, including metadata, data-dictionaries and embedded definitions (e.g., codes for missing values, data quality flags and questionnaire responses)? Are there confidentiality issues that may limit accessibility to the complete data set? Page 4 of 15 FINAL

5 6. In the case of a modeling exercise, have the definitions and units of model parameters been provided? To what extent have the procedures for applying the model been clearly and completely documented? How available and adequate is the information necessary to run the model computer code? 7. To what extent are the descriptions of the study or survey design clear, complete and sufficient to enable the study or survey to be reproduced? 8. Have the sponsoring organization(s) for the study/information product and the author(s) affiliation(s) been documented? 9. To what extent are the procedures for quality assurance and quality control of the data documented and accessible? Uncertainty and Variability The extent to which the variability and uncertainty (quantitative and qualitative) in the information or in the procedures, measures, methods or models are evaluated and characterized. 1. To what extent have appropriate statistical techniques been employed to evaluate variability and uncertainty? To what extent have the sensitive parameters of models been identified and characterized? 2. To what extent do the uncertainty and variability impact the conclusions that can be inferred from the data and the utility of the study? What are the potential sources and effects of error and bias in the study design? 3. Did the study identify potential uncertainties such as those due to inherent variability in environmental and exposure-related parameters or possible measurement errors? Evaluation and Review The extent of independent verification, validation and peer review of the information or of the procedures, measures, methods or models. 1. To what extent has there been independent verification or validation of the study method and results? What were the conclusions of these independent efforts, and are they consistent? 2. To what extent has independent peer review been conducted of the study method and results, and how were the conclusions of this review taken into account? 3. Has the procedure, method or model been used in similar, peer reviewed studies? Are the results consistent with other relevant studies? 4. In the case of model-based information, to what extent has independent evaluation and testing of the model code been performed and documented? Page 5 of 15 FINAL

6 ELEMENTS OF SYSTEMATIC PLANNING Definitions and Considerations Organization Identification and involvement of the project manager, sponsoring organization and responsible official, project personnel, stakeholders, and experts, etc. (e.g., all customers and suppliers). This element ensures that the study will be designed to address the needs of all vested parties (for example, data users, data generators, data analysts and other stakeholders). Consulting cross-disciplinary experts familiar with the different technical aspects of the problem ensures that important details of the study are not overlooked or ignored and technical challenges will be addressed appropriately. It is also important to assign responsibilities for the project so that conflicts can be resolved and progress is tracked. For some projects, it may be most effective to create a formal "planning team," while for others, one individual may be responsible for the project and involve other individuals when necessary. Project Goal Description of the project goals, objectives, and questions and issues to be addressed. This element ensures that the participants formulate a clear statement of the project's goals and objectives and therefore understand the purpose of the project and expected results. The objectives reflect a general statement of the intent of a project and how that project is linked to addressing the environmental problem (or contributing to the field of science). The project's questions will define what data or information is needed to address the project's goals and objectives. The transition from the project goals, to statement of objectives, to specific and appropriate questions are some of the most important steps in systematic planning. Schedule Identification of project schedule, resources (including budget), milestones, and any applicable requirements (e.g. regulatory requirements, contractual requirements). Identifying the available resources and deadlines at the beginning of a project helps ensure the project is feasible and timely. A clear statement of the project's resources, constraints, and deadlines helps prevent potential issues and/or conflicts by determining practical bounds on the project as early as possible. Regulatory, statutory, contractual and other constraints should be considered that might affect the project schedule. Data Needs Identification of the type of data needed and how the data will be used to support the project's objectives. This element focuses on identifying the specific type of data or information needed to complete the project. Types of, sources for, and how to obtain information needed to address the study questions should be listed. Sources may include literature, existing databases and/or new data collection. By developing a list of the information needed to address the project questions, the project requirements will Page 6 of 15 FINAL

7 be clearly defined. In addition, the list may identify other information that will be helpful, or that can be economically collected to facilitate the use of the project results for other purposes. Criteria Determination of the quantity of data needed and specification of performance criteria for measuring quality. This element focuses on establishing criteria to ensure that the information and products generated meet the objectives of the project. These quality specifications are established at both the product level and at the level of components of that product, such as the quality of individual measurements. Examples of product-level criteria include EPA's information quality guidelines components -- objectivity, utility, integrity and reproducibility. Examples of component-level criteria are quality criteria for individual measurements, for example: criteria for precision bias accuracy representativeness comparability completeness sensitivity criteria for decisions or estimates [for example, a stated desired confidence that results will fall within a specified window such as Type I and Type II error rates (false rejection and acceptance error rates), uncertainty intervals, etc.] After the information, data or product is generated, these criteria are used to determine if they met the project's objectives. Data Collection Description of how and where the data will be obtained (including existing data) and identification of any constraints on data collection. This element focuses on how to amass the data or information needed for a project by collecting new data, using existing data, citing information from other resources, etc. When collecting new data or information, consider where to collect samples (sampling design), when, how to best acquire physical specimens of an adequate size and dimension (sample support) to represent the variable of interest within the sampling unit, questionnaires and survey instruments, sampling technologies, analytical methods, representativeness, etc. When existing data or information (i.e., from models, databases, literature, etc.) is used, consider sources and methods for assembling it. Also consider how the data will be inspected to ensure compatibility with the project's goals and the handling of information/data either through physical custody of samples or the entering of specific information into a database or spreadsheet. Quality Assurance Specification of QA and QC activities to assess the quality performance criteria (e.g., QC samples for both the field and laboratory, audits, technical assessments, performance evaluations etc.). It is often necessary to plan ahead for QA and QC activities to ensure that a process, item, or service is of the type and quality needed and expected by the customer. QA and QC activities measure the attributes Page 7 of 15 FINAL

8 and performance of a process, item or service against defined standards to verify that it meets the stated requirements. Examples of these activities include: assessments/audits of field sampling and laboratory activities QC samples (blanks, duplicates, etc) project reports and inspections/testing/maintenance of equipment, supplies and consumables Analysis Description of how the acquired data will be analyzed (either in the field or the laboratory), evaluated (i.e., QA review, verification, validation), and assessed against its intended use and the quality performance criteria. This element focuses on the reviews of both the information (such as verification and validation) and the project (peer reviews, clearance procedures, etc.). It is important to determine up front how data and information will be summarized, displayed and communicated; how uncertainty in the information will be determined and accounted for in the final product; and how the information will be used to achieve the project's goals. Page 8 of 15 FINAL

9 DATA SUMMAY TECHNICAL ASSESSMENT PROCESS CHART Definitions and Considerations The process flow chart is provided as a generalized process to support the DSTA approach at Cotter. After appropriate general assessment factors and elements of systematic planning have been identified an evaluation of data sets can follow this process. Generally, there are six definition categories included in the process flow chart. The categories are shown in the process diagram along the left column and outline several process steps for the DSTA flow effort. 1. Defining Criteria 2. Exclusion Criteria 3. Selection of General Assessment Factors (GAFs) 4. Elements of Systematic Planning 5. Data Gaps 6. RI/FS Processes Details are generally formulated by site specific process knowledge. Examples are provided based on guidance and then additionally site criteria, exclusions and inclusion questions can be formulated based on the commonalities of Table 3 from EPA QA/G4. Examples follow. DEFINING CRITERIA - EXAMPLES 1. What is the information/data type (field, laboratory, other)? 2. What is the media (groundwater, surface water, air, soil/sediments)? EXCLUSION CRITERIA EXAMPLES What Would Cause the Rejection of Samples - Based on data type and media? Field Data Groundwater 1. Are wells under Site control and installation performed according to industry standards at the time and documented? 2. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or 3. Were field instruments properly calibrated? Air 1. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or 2. Were field instruments properly calibrated? Surface Water 1. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or 2. Were field instruments properly calibrated? Soil 1. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or 2. Were field instruments properly calibrated? Page 9 of 15 FINAL

10 Data Summary Technical Assessment (DSTA) Process Chart Lincoln Park Superfund Site / Cotter Mill DEFINING CRITERA DATA TYPE Field Lab - Other PARAMETERS Measurement Types MEDIA GW SW Air Soil/Seds EXCLUSION CRITERIA DATA PACKAGE TEST Integrity / Traceability EPA/260R Dec 2002 YES EXCLUSION QUESTIONS Pre-screened Categories & Questions Facilitates Initial Data Selection for RI/FS Usage YES SELECT GAFs NO GENERAL ASSESSMENT FACTORS Applicable per site documents / data sets Select GAFs per Table 3 of EPA QA/G4 Guidance NO ELEMENTS OF SYSTEMATIC PLANNING CATEGORICAL QUESTION SETS Developed in the defined in planning process of DQOs per commonalities EPA QA/G4 Guidance Table 3 DATA GAPS DO NOT USE Archives ONLY Include in Data Gaps (SOW 8.3) RANKED USEFULNESS Develop for Sorting Criteria RI/FS PROCESS FINAL ASSESSMENT RI Based on Existing Data (SOW 8.2) Page 10 of 15 FINAL

11 Analytical Data Groundwater 1. Are wells under Site control and installation performed according to industry standards at the time and documented? 2. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or 3. Was field chain-of-custody information available and complete? Or is other documentation available to establish traceability? 4. Was receipt of samples acknowledged by the laboratory? 5. Did the sample, as received by the analytical laboratory, meet pertinent and published guidance (for example; temperature criteria, adequate sample volume, sample container type, appropriate methods of preservation)? 6. Were samples kept in a secure area until all analyses were complete? 7. Did the laboratory receiving the samples check the integrity of the shipping container seal upon receipt? If the seal was broken, did the laboratory staff receiving the container indicate the broken seal on the COC with the samples? Air 1. Were particulate samplers operated at the typical flow rate of m3 /min (40-60 ft3 /min)? 2. Were filters properly conditioned after collection and before analysis? 3. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or 4. Was field chain-of-custody information available and complete? Or is other documentation available to establish traceability? 5. Was receipt of samples acknowledged by the laboratory? 6. Did the sample, as received by the analytical laboratory, meet pertinent and published guidance (for example; temperature criteria, adequate sample volume, sample container type, appropriate methods of preservation)? 7. Were samples kept in a secure area until all analyses were complete? 8. Did the laboratory receiving the samples check the integrity of the shipping container seal upon receipt? If the seal was broken, did the laboratory staff receiving the container indicate the broken seal on the COC with the samples? Surface Water 1. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or 2. Was field chain-of-custody information available and complete? Or is other documentation available to establish traceability? 3. Was receipt of samples acknowledged by the laboratory? 4. Did the sample, as received by the analytical laboratory, meet pertinent and published guidance (for example; temperature criteria, adequate sample volume, sample container type, appropriate methods of preservation)? 5. Were samples kept in a secure area until all analyses were complete? 6. Did the laboratory receiving the samples check the integrity of the shipping container seal upon receipt? If the seal was broken, did the laboratory staff receiving the container indicate the broken seal on the COC with the samples? Soil 1. Were samples collected following an approved SOP, FSP, QAPP or other document based on EPA or Page 11 of 15 FINAL

12 2. Was field chain-of-custody information available and complete? Or is other documentation available to establish traceability? 3. Was receipt of samples acknowledged by the laboratory? 4. Did the sample, as received by the analytical laboratory, meet pertinent and published guidance (for example; temperature criteria, adequate sample volume, sample container type, appropriate methods of preservation)? 5. Were samples kept in a secure area until all analyses were complete? 6. Did the laboratory receiving the samples check the integrity of the shipping container seal upon receipt? If the seal was broken, did the laboratory staff receiving the container indicate the broken seal on the COC with the samples? Data Package Test - Suggested Add-On Integrity/Traceability/Security Provides a simplified process for sorting data groups quickly based on simple data qualifiers for RI/FS usage. Examples related to protection of information from unauthorized access or revision, ensure that information is not compromised through corruption or falsification. In a legal context refers to the chronological documentation or paper trail, showing the collection, custody, control, transfer, analysis, and disposition of samples. EVALUTION CRITERIA -EXAMPLES Data remaining for further evaluation after application of the exclusionary criteria can then be evaluated using the commonalities of the General Assessment Factors and the Elements of Systematic Planning. Example evaluation questions for those commonalities are provided below. Organization Was the study designed to address the needs of all vested parties (for example, data users, data generators, data analysts and other stakeholders)? Were responsibilities assigned for the project so that conflicts could be resolved? Project Goal Soundness Were goals of precision, accuracy, representativeness, comparability and completeness set for the project? Applicability and Utility Does the dataset support the goals and objectives of the RI/FS? Were project objectives documented in a QAPP, SOP, FSP or other agreement? Schedule Applicability and Utility Does the dataset support the relevant timeframe of the goals of the RI/FS? Page 12 of 15 FINAL

13 Data Needs Applicability and Utility Does the data set meet the needs of the end-users? Did sampling and analytical supplies/equipment undergo inspection prior to use? Evaluation and Review Were field, laboratory and management assessments planned? Criteria Was the appropriate number of environmental samples collected? Uncertainty and Variability Was the appropriate number of field QC samples collected? Data Collection Soundness Was an approved SOP, QAPP, FSP or other governing document in place prior to sample collection? Were samples/qc samples collected following the steps/guidance of the approved SOP, QAPP, FSP or other governing document? Were field deviations properly documented? Uncertainty and Variability Were sample locations and time and date of collection documented? Evaluation and Review Was the field logbook and associated documentation signed and dated by the field technician? Did the quality systems employed during sample collection comply with the requirements of the approved QAPP and the currently accepted standards and procedures? Quality Assurance Soundness Was the SOP, QAPP, FSP or other governing document signed by the Quality Assurance Manager? Were QC samples collected and analyzed as specified? Uncertainty and Variability Did QC samples undergo statistical analysis? Evaluation and Review Were field audits performed periodically and found acceptable? Page 13 of 15 FINAL

14 Analysis Soundness Were the analytical procedures or methods and detection limits identified and do they represent the accepted industry standards at the time the samples were collected? Applicability and Utility Was site document provided in work plans, engineering documents, construction completions, status reports, deliverables to agencies, and/or site licenses? Were all required method quality control within limits? Were method detection limits equal to or less than the action limits for the COPC? Was all method required laboratory QC performed and meet method/project criteria? Was analytical data flagged appropriately? Uncertainty and Variability Was the total analytical uncertainty 32% or less? Were the laboratory results from proficiency testing or round robin programs acceptable? Evaluation and Review Did the data package undergo secondary management/quality assurance review? Did the quality systems employed during analysis comply with the requirements of the approved QAPP and the currently accepted standards and procedures? Did the laboratory participate in a proficiency testing or round robin program in the past 12 months? Did the laboratory receiving the samples check the integrity of the shipping container seal upon receipt? If the seal was broken, did the laboratory staff receiving the container indicate the broken seal on the COC with the samples? FINAL ASSESSMENT The answers to the questions developed can then lead to final disposition as to whether the data supports RI/FS decisions or identifies data gaps that would need to be addressed in the RI/FS work planning stages. As stated on page 2, the Agencies expect Cotter to provide further details/clarifications in development of what questions and exclusion criteria Cotter will utilize to evaluate data and its applicability for future use. Steps and/or process for the Ranked Usefulness should be reviewed by the Agencies prior to Cotter s implementation of this step. Page 14 of 15 FINAL

15 REFERENCES EPA A Summary of General Assessment Factors for Evaluating the Quality of Scientific and Technical Information. EPA 100/B-03/001. June. EPA 2002a. Guidance on Environmental Data Verification and Data Validation EPA/240/R-02/004. EPA QA/G-8. November. EPA EPA Quality Manual for Environmental Programs. CIO 2105-P May. EPA EPA Guidance on Systematic Planning Using the Data Quality Objectives Process. EPA QA/G-4. February. EPA Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency. EPA/260R October. Page 15 of 15 FINAL