STATE INNOVATION GRANT PROJECT WORK PLAN Revised April 3, 2006 Project Summary Information

Transcription

1 STATE INNOVATION GRANT PROJECT WORK PLAN Revised April 3, 2006 Project Summary Information Project Title: Promoting Implementation of Innovative Environmental Compliance Strategies in the Northeast and Use of Common Business Sector Performance Measures Project Lead State/Applicant: Massachusetts Department of Environmental Protection (MassDEP) in partnership with the Northeast Waste Management Officials Association (NEWMOA) 1 Participating States: Connecticut Department of Environmental Protection, New Hampshire Department of Environmental Services, Rhode Island Department of Environmental Management, Maine Department of Environmental Protection, Vermont Department of Environmental Conservation Learning States: New Jersey Department of Environmental Protection, New York State Department of Environmental Conservation, Colorado Department of Health and Environment, California Environmental Protection Agency Contact Person: Steven A. DeGabriele, Director, Business Compliance Division, Bureau of Waste Prevention, , FAX steven.degabriele@state.ma.us Focus Areas: Promoting Business Sector Performance Measurement on a Multi-State Basis; Improving Environmental Performance Using Environmental Results Program (ERP) Tools and Methods Total Funds Requested from EPA: $255,000 In-Kind MassDEP Resources: $20,000 Total Grant: $275,000 Funded or Executed with Other Federal Programs: Not applicable Project Period: 3 years Special Federal Flexibility needed to implement the project: None Statements of Support for the Project: See attached letters from project state agencies 1 NEWMOA is a non-profit, non-partisan interstate governmental association whose membership is composed of state environmental agency directors of the pollution prevention, hazardous and solid waste, and waste site cleanup programs in CT, MA, NH, NJ, NY, RI, VT, and ME. NEWMOA s mission is to develop and sustain an effective partnership of states to explore, develop, promote, and implement environmentally sound solutions for the reduction and management of materials and waste, and for the remediation of contaminated sites, in order to achieve a clean and healthy environment. For more information visit: 1

2 Project Narrative Current Situation and Need For the past few years, a number of the Northeast states have begun to actively employ a wide variety of traditional and innovative approaches to environmental compliance, enforcement, and assistance. These initiatives involve experimenting with various combinations of regulatory and non-regulatory tools to drive environmental performance improvements within identified regulated sectors and groups. Although state agencies currently collect information about various aspects of their activities and the general performance of certain sectors, there is seldom an accurate basis for drawing group performance conclusions and limited ability to compare the differences in group performance levels between two or more states working on a common sector or group. Adopting a common approach to measuring the environmental performance status of specific business sectors and groups holds great potential for enabling states to compare the effectiveness and efficiency of differing state strategies for improving the compliance and environmental performance of that group. Objectives and Benefits The Massachusetts Department of Environmental Protection (MassDEP) and the other member state agencies of the Northeast Waste Management Officials Association (NEWMOA) are seeking EPA assistance to support the development and use of shared core performance measures for one or more business sector or regulated group. The project will rely on the models of innovative compliance strategies, including the Environmental Results Program (ERP) and other statistically valid compliance/performance rate approaches already in use. By developing and using shared measures, the participating states can not only evaluate the effectiveness of their own programs, but can also evaluate the relative effectiveness of the other states programs and decide to adopt the more successful compliance assurance approaches used throughout the region. MassDEP is the lead state and grant applicant. Project goals: Improve the ability of state environmental agencies to develop, implement, and analyze innovative performance measures for targeted business sectors Improve the ability of the state environmental agencies to develop and implement innovative compliance strategies, including ERP Project objectives: Develop and implement a common, core set of performance measures for business sectors on a multi-state basis Promote the implementation of innovative compliance strategies, including ERP, in the states in the Northeast that have not yet begun these initiatives, including New York and New Jersey Combine and present the environmental outcome data that is collected from the states using the core performance measures for at least one business sector and analyze and present the results Anticipated project results: Illuminate the nature, scope and seriousness of problems within each selected sector Quantify environmental performance in the selected sectors Record group environmental performance status and changes over time 2

3 Facilitate general comparisons of the effectiveness and efficiency of different state compliance assurance strategies within the same sector. The project will also enable participating states to better focus their limited resources on specific problem areas and to use those strategies that have been demonstrated to achieve the greatest environmental results improvements in the future. Of eight Northeast states, six will be involved in this project as full participants: Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, and Vermont. These six participating states have already initiated an ERP or innovative compliance program and are interested in coordinating on a set of core performance measures. Massachusetts, Rhode Island, Maine, New Hampshire and Vermont are implementing ERP initiatives directed at several business sectors, including printers, dry cleaners, photo processors, small boilers, industrial wastewater holding tanks, dental practices handling mercury, auto salvage yards, auto body shops, and facilities with underground storage tanks. Connecticut and New Hampshire have developed innovative strategies for promoting and improving compliance among RCRA generators by developing methods of evaluating statistically valid compliance rates combined with improved compliance assistance directed toward areas of high non-compliance. New York, New Jersey, Colorado, California and possibly other states will be involved in the project as learning states, since they are beginning to consider developing ERPs or similar programs and are eager to learn and benefit from the experience in the other states. This project would be a vehicle for them to more quickly and efficiently develop their initiatives. The six participating states will evaluate and use, as appropriate, statistical performance measurement methods to enhance confidence in the reliability and accuracy of collected performance data. These statistical methods will enable the participants to more accurately evaluate performance data for selected sectors without requiring every facility within groups to be inspected or otherwise evaluated. The statistical methods would also enable the participating states to draw more accurate inferences as to each group s compliance status (with respect to selected indicators) and overall performance level. Connection to EPA s Strategic Goals EPA s Strategic Goal 5, Objectives 5.1 and 5.2 commit EPA to increase compliance and improve environmental performance through pollution prevention and innovation. This project addresses these objectives by supporting the development of a multi-state approach to implementing and analyzing strategies that effectively combine multi-media compliance improvement and pollution prevention on a sector basis. The ERP approach promotes innovation in permitting by enabling states to implement streamlined programs that emphasize sector-wide performance without the need for site-specific individual permits for large numbers of relatively small entities. It promotes improved compliance by providing the necessary tools for sectors to achieve results and is based on a rigorous, statistically valid method for tracking compliance and beyond compliance practices. The compliance assistance and performance measures that states plan to develop and implement through this project would be multimedia and compliance-based, with an emphasis on encouraging pollution prevention and best management practices. 3

4 Summary Work Plan Key elements: * 1. Identify and select up to three business sectors or regulated groups for performance measurement 2. Select group environmental performance indicators 3. Decide on use of statistical methodologies 4. Develop a data management strategy 5. Collect, evaluate, and report data to NEWMOA for aggregation 6. Submit interim progress and final results to EPA. Participating states will also make performance data collected after the end of the project period available to EPA and others as requested *For ease of presentation, the elements are presented in sequential fashion. However, all project participants recognize that a number of these activities will operate in parallel. For instance, it could be difficult to finalize a performance indicator without participating states first having carefully considered and decided upon statistical approaches, data management strategies and other quality procedures/criteria necessary to support analysis of performance indicators. Element 1: Identify and select business sectors and groups for performance measurement In FY2004, NEWMOA formed an Innovative Compliance Strategies Workgroup including representatives from each of the NEWMOA state environmental agencies both participating and learning states. Members of the Workgroup are actively involved with managing and staffing their states innovative compliance strategies projects, including ERP. This Workgroup will play a key role in overseeing and managing NEWMOA efforts on this project. The state project leads will all be members of this Workgroup. NEWMOA will survey members of the Workgroup to identify which sectors they are currently targeting (or plan to target) with ERP and other compliance assurance strategies, as a basis for selecting the business sectors for the focus of this project. NEWMOA will also catalog all of the performance indicators that the states have already developed and are using in their innovative compliance strategies projects. States will also be asked to summarize how the data are collected and the quality assurance processes that they use with these performance indicators. An understanding of current data collection/quality assurance procedures will help the Workgroup better understand the extent to which participating States' procedures will need to converge and/or improve for a common indicators approach to be effective. Once this survey is completed, NEWMOA will convene a face-to-face meeting of the Workgroup to review results, make plans for the project, and select up to three sectors for regional performance measurement. The initial business sectors that have already been discussed as possibilities are RCRA generators, automotive facilities (including auto body, repair and salvage shops), and facilities with underground storage tanks. In an initial survey of NEWMOA states, these business sectors appeared to have the highest degree of common interest. If selected, the RCRA generators sector would focus primarily on hazardous waste. However, the other sectors would be focused on improving multi-media environmental performance. 4

5 The learning states New Jersey, New York, Colorado, California and potentially others will participate in conference calls, selected meetings and related project activities to learn more about measurement tools and uses to increase overall state capability to develop and implement performance measurement-based programs. In the future, the learning states may join the participating states work and use similar measurement approaches. After the initial meeting, NEWMOA will compile and summarize for the eight states and EPA all of the available checklists, guidance documents, data collection procedures, quality assurance documentation and other project materials related to state ERP and innovative performance-based compliance projects in the selected sectors. Based on the business sectors or regulated groups that are selected, the Workgroup will form subgroups of key staff working on each business sector for more in-depth information sharing, coordination and development of key performance measures as well as relevant quality criteria. Throughout the rest of the first year of the project, NEWMOA will convene Workgroup and sub-group conference calls every two to three months or more frequently, as needed. Element 2: Select group environmental performance indicators and establish quality objectives Once NEWMOA has compiled and shared all of the performance measures that states have developed for the selected business sectors or regulated groups in Step 1, the association will review them to identify common characteristics, potential gaps and quality issues that may need to be addressed. Each of the business sector/regulated group sub-groups will convene at least two conference calls to review in detail the list of sector performance measures and decide upon a final core list for the full Workgroup s consideration and adoption. Along with performance measures, each sub-group will recommend a set of quality criteria that data must meet in order to be included in the regional-level analysis. The kinds of quality issues the sub-groups will consider in developing these criteria will include (but not be limited to): the types of statistical sampling and analysis approaches that will be considered acceptable; whether a standard confidence level should be used by states; if data should be independently obtained, or whether data volunteered by the regulated community or from other sources are acceptable; the need for normalizing data, if any; whether all datasets should have a common time frame (and the implications of not having common time frames); and procedures necessary for any physical/environmental samples that are to be taken. The proposed measures and associated quality criteria will also be shared with and commented on by the NEWMOA Board of Directors during one of its regular meetings. The Workgroup may modify the sector lists, measures and/or quality criteria based on NEWMOA Director feedback. Element 3: Decide on use of statistical methodologies and train and assist states concerning related Quality Assurance Issues A contractor retained by MassDEP and supported by EPA OECA funds has recently completed development of four ERP analysis automation tools: inspection checklist data management and analysis, environmental performance results presentation, environmental performance statistical analysis, and environmental outcome analysis. MassDEP will share these software tools with the Workgroup. 5

6 NEWMOA will convene a training session for the Workgroup and other interested state staff to learn statistical techniques, software tools (which may not be limited to the MassDEP tools) and quality objectives, criteria and procedures/steps. NEWMOA will procure the contractor(s) services to help with this training and to assist individual states as they implement statistical methods. MassDEP and NEWMOA staff will also be available to assist states with software tools, statistical approaches and quality issues as part of this training. If deemed necessary, MassDEP and NEWMOA will seek additional contractor assistance to ensure that proper statistical methods are used throughout the project. Wherever appropriate the project will rely upon the methodologies that were developed and peer reviewed through previous ERP work. Once the states have become familiar with the basics, the Workgroup will review the various options for which statistical approaches are the most appropriate for measuring sector performance levels. If needed, the contactor will provide assistance to states for specific issues and questions that may arise as they implement the statistical methods and use the new tools. The contractor will also advise and assist in the development of the QAPP amendment. Element 4: Develop a data collection, management, and analysis strategy The Workgroup will provide a forum for states to share information and techniques for data collection and management. Several states, including Massachusetts and Rhode Island, have devised strategies for data collection and management under their ERPs. Connecticut recently hired a contractor to assist its efforts to evaluate compliance trends among RCRA generators in the states. These and other experienced states will share their methods with partner states and address any questions or concerns during face-to-face meetings of the Workgroup. As part of this element, NEWMOA will also develop a methodology for analyzing data to be submitted by states, incorporating workgroup decisions on indicators and quality criteria. This methodology will be reviewed by the participating states and by the NEWMOA Board of Directors. Once adopted, the methodology, data collection/management strategy, indicators and quality criteria will be submitted as part of an amended QAPP for EPA approval. Element 5: States report data to NEWMOA NEWMOA will coordinate with the states on implementing this method for at least one of the targeted business sectors, with the assistance of a contractor as necessary. States will have approximately one year to gather and analyze supporting data, and report indicator data to NEWMOA, and will meet together to discuss progress and share lessons learned at approximately the midway point. Data may be aggregated on a secure portion of NEWMOA s Web site and will be reported to EPA. Element 6: Reports to EPA MassDEP and NEWMOA will submit routine progress reports to EPA detailing workplans, schedules, progress, unanticipated obstacles, and expenditures. 6

7 It is anticipated that the EPA Technical Project Liaison will participate actively in the project and be aware of progress. At the end of the project, MassDEP and NEWMOA will submit a project report for EPA that summarizes results of the project, including: Performance measures selected for each sector, Statistical or other methodologies that the states are implementing to gather and analyze the data Data aggregation, analysis and quality assurance methods used by NEWMOA, and Results of the aggregated data. Descriptions of the participating states compliance assurance strategies Lessons learned Drafts of the final report will be shared with the Workgroup and NEWMOA Directors for review and comment prior to submission to EPA. In addition, the project participants will make performance data collected after the end of the grant available to interested parties, to the extent they continue to collect such data. The Massachusetts DEP Project Manager will be responsible for the following activities: Overall management and contracting with NEWMOA and others for support services Managing outreach and training to participating state agencies (stakeholders) Developing consensus concerning o up to three business sectors o environmental performance indicators o the use of statistical and other methodologies o data quality objectives for the QAPP amendment Developing audit procedures in consultation with stakeholders Sharing ERP information tools and resources with stakeholders Managing the developing of a data management strategy in consultation with stakeholders Managing the collection, evaluation and reporting of data to NEWMOA for aggregation Reporting results to EPA Amending the QAPP, as necessary Issuing quarterly and annual reports to EPA The QA Officer will be responsible for the following activities: Maintaining the QAPP Distributing the QAPP and maintaining the distribution list Conducting readiness reviews Note: The QA Officer will receive support, as needed, from the MassDEP Office of Research and Standards in carrying out her responsibilities under the QAAP. The Office of Research and Standards (ORS) provides scientific expertise to MassDEP in environmental health, toxicology, standard setting, ecological and human health risk assessment, chemistry and statistics. ORS provides information and 7

8 guidance on public health issues for the agency. Using available information and methods, scientists in ORS formulate exposure guidelines for toxics in air, water, soil and wastes, interpret existing state and federal guidelines and lead agency efforts to assess and reduce key environmental pollutants. ORS also develops new methods and conducts environmental research on priority pollutants. Contractor(s) to be determined will provide training in the use of ERP Statistical Methods Protocol and review of project audit procedures concerning: Statistical sample design and analysis, and assistance to state agencies concerning application of proper methods and data management systems and procedures Advice and assistance concerning Quality Objectives and Criteria for Performance Measures to be submitted in a QAPP amendment. NEWMOA will provide support services including: organizing and facilitating conference calls and meetings Serving as a clearinghouse for project information including performance results from participating states Developing and refining proposals for selecting sectors and groups, setting group performance goals and indicators, and compiling project results Organizing measurement and statistical methodology training Assisting MassDEP with QAPP Amendment preparation Providing direct support to individual states Develop and housing the project database Collecting and analyzing aggregated performance data from multiple states Assisting in preparing project reports EPA will be an active participant in the project, providing support needed to assure the overall success of project. This support and participation may include: Assistance in coordination with the participating states Assistance steering the project and ensuring that it remains on track Assistance with statistical analysis including, if deemed necessary, 3 rd party review Participation in the MassDEP Beyond ERP Implementation Team meetings will provide overall guidance, advice and strategic planning for the project. MassDEP Bureau of Waste Prevention Beyond ERP Implementation Team is comprised of key Bureau of Waste Prevention Division Directors, Enforcement, Regulatory Development, and Program Evaluation managers, regional management and staff One of its responsibilities is to champion the implementation of performance measurement throughout the Bureau. Augmented by representatives from NEWMOA and EPA it will provide overall strategic direction and guidance for the project. Participating and Learning States The participating states will: Provide up-to-date sector information, performance measures and results Provide training, support, and assistance to other states collectively and individually Actively participate in Workgroup conference calls, meetings, and training sessions 8

9 Implement the performance measures in at least one sector and share the results through NEWMOA, as applicable Participate in the development of quality objectives and criteria for the QAPP amendment The learning states will not directly participate in developing and implementing performance measurement strategies, but will participate in conference calls, meetings, and related project activities to learn more about the measurement tools and their uses to increase overall state capabilities. In the future, these states may join participating states project work and use the measurement approaches. Key Outputs List of selected groups List of performance measures for each selected group Quarterly and final project status reports Aggregated performance measurement data Project Reporting & Measurement: MassDEP and NEWMOA will provide quarterly project status reports to EPA. The quarterly reports will include the status of the project relative to the elements of the project narrative above, the key outputs as completed and budget status updates, with expenditures for the period and the amount left in the grant. In the final report, MassDEP and NEWMOA will present aggregated performance measurement data for at least one of the selected groups. Major Project Tasks & Milestones (Provided that project is funded by March 1, 2006) Task Description Start Date End Date Collect existing state materials and performance measures March 06 April-06 Hold project organization and kick-off meeting April-06 Finalize sector group selection April-06 Workshop on statistical methodologies & QA May-06 Share statistical methods and data collection information May-06 August-06 Meeting and calls to select draft measurement indicators and June-06 August-06 quality criteria for selected sectors Develop regional aggregation methodology May-06 September-06 Review by NEWMOA Board of Directors Sept-06 Submit Amended QAAP to EPA Sept-06 States implement performance measures October-06 March-07 Workgroup face-to-face meeting to discuss the status of the April-07 project Continue implementation of performance measures January-07 September-07 Collect data from the states using regional aggregation July-07 December-07 methodology Collect information about each state s performance measurement strategies July-07 December -07 Summarize and present results of data aggregation November-07 April-08 9

10 Task Description Start Date End Date Develop final report summarizing group environmental May-08 performance results Draft final report distributed for comment by participating states July-08 Hold project wrap-up meeting including other interested states September-08 Final progress report including summary of results to date, lessons learned and recommendations for future work March-09 Key Short, Intermediate and Long-Term Outcomes Short-Term Outcomes Project states agreement on groups and common measures Project states knowledge of measurement options, statistical approaches and data collection issues is increased Intermediate-Term Outcomes Project states will gather and report environmental performance data for the selected groups General comparisons of the effectiveness of differing states performance improvement strategies Transfer of performance measurement approaches and benefits within project state agencies Long-Term Outcomes Increased states and EPA acceptance of the benefits or ERP-type measurement approaches Data to support adoption of more effective and efficient state compliance/performance improvement strategies 10

11 11

12 ATTACHMENT 1 STATE INNOVATION GRANT WORKSHOP--logic model worksheet In preparation for the Performance Measurement Component of the September 19 and 20 State Innovation Grant (SIG) workshop, please review the information below and use the attached worksheet to complete the information about your program or project s resources, activities, outputs, and outcomes. Please complete your worksheet prior to attending the workshop. This homework assignment is needed so that we can maximize the time available for developing logic models and performance measures for your SIG project during the workshop. If you have any questions, regarding the worksheet, please contact Yvonne Watson at (202) or watson.yvonne@epa.gov Many programs and projects often run into trouble because they lack a well-articulated road map describing the logic of the program or project. A logic model is a diagram and text that describes and illustrates the logical (causal) relationships among program elements and the problem to be solved, thus defining measurements of success. A logic model synthesizes the key activities intended to achieve the goals of the program/project into a picture linking inputs to activities and to expected outputs and outcomes. Your task at this stage of the process is to first identify the major facets or components of your program/project. This is made less difficult if you determine the major functions of the program and then aggregate similar functions into program components. After each major function area or component is identified, it should be described in terms of the resources (inputs) needed to conduct the activities, (staff, time, finances, information, equipment, facilities, etc.), activities (processes) that will be accomplished to achieve your objectives, outputs, and outcomes. STEP 1: Clarifying Program/Project Theory: On a separate piece of paper, please describe the items listed below. 1. Problem or Issue Statement: Describe the problem (s) your program/project is attempting to solve or the issue (s) your program/project will address. 2. Community Needs/Assets: Specify the needs and/or assets of your office that led your organization to design a program that addresses the problem. 3. Desired Results (Outputs, Outcomes and Impacts): Identify desired results, or vision of the future, by describing what you expect to achieve near and long-term. 4. Influential Factors: Factors that are outside of your control (positive or negative) that may influence the outcome and impact of your program/project. 5. Assumptions: State the assumptions behind how and why the change strategies will work in your affected community. 12

13 COMMON MEASURES PROJECT Clarifying Program/Project Theory 1. Issue Statement: The participating states will develop and agreed upon set of performance measures for several regulated groups and will establish baseline group performance levels and will report group performance changes over time. 2. Community Needs/Assets: Desire to illuminate nature, scope and seriousness of problems within each selected group; quantify environmental improvements or changes within the selected groups; record group environmental performance status and changes over time; and, facilitate general comparisons of effectiveness and efficiency of different state performance enhancement strategies within a sector. 3. Desired Results: Short-term results, same as 2. above. Long-term results, continued cooperation among seven states in using common measures for many more groups leading to more efficient and effective government oversight of regulated sources. Expanding the use of common measures by addition states and programs. 4. Influence Factors: Degree of trust and cooperation achieved by project participants and their managers and organizations. Availability of critical support resources such as statistical measurement and information systems expertise. Competing priorities for project participants. General issues associated with acceptance of alternative approaches within and outside the project. 5. Assumptions: General acceptance that ERP-type measurement is valuable. STEP 2: Demonstrating Your Program s Progress: Please Use Worksheet I to describe your program resources, activities, outputs and outcomes. An Example Worksheet is also included as a reference. 1. Resources/Inputs: Describe the programmatic investments available to support the program ($$, staff, etc) 2. Activities: Describe the things you do activities you conduct in your program. 3. Outputs: Identify the product or service delivery/implementation targets you aim to produce. 4. Customer: Describe the user of the products/services the target audience the program is designed to reach. 5. Outcomes: Identify the changes or benefits resulting from activities and outputs. a. Short-term Changes in learning, knowledge, attitude, skills b. Intermediate Changes in behavior, practice or decisions c. Long-term Changes in condition 6. External Influences: Factors that are outside of your control (positive or negative) that may influence the outcome and impact of your program/project. 13

14 Program/Project Description Worksheet Program/Project Mission or Goal: Use of Common Business Sector Measures Outcomes Resources & Partners Activities Outputs Customer Reached Short Term Intermediate Term Long-Term - 1 lead staff person for each state - NEWMOA support - Contractor support - Other resources to be defined - Select measurement groups - Select performance indicators - Decide on use of statistical methods - Develop data collection approach - Collect, evaluate and report data to NEWMOA - Report results to EPA - List of selected groups List of measures for each group Reports to EPA - OPEI Participating states - NEMOA - Other states and EPA offices - Agreement on groups and common measures - Knowledge of measurement options, statistical approaches and data collection issues increased - Reporting and evaluation of group results by each state - General comparison of effectiveness of differing state strategies possible - Increased states and EPA acceptance of ERP-type measurement Data to support adoption of more effective and efficient state compliance/performance improvement strategies External Influences: 14