Initiating and Managing Risk Assessments within a Risk Analysis Framework: FDA/CFSAN S Practical Approach

Transcription

1 2058 Journal of Food Protection, Vol. 67, No. 9, 2004, Pages Copyright, International Association for Food Protection Initiating and Managing Risk Assessments within a Risk Analysis Framework: FDA/CFSAN S Practical Approach ROBERT L. BUCHANAN,* SHERRI DENNIS, AND MARIANNE MILIOTIS U.S. Food and Drug Administration, Center for Food Safety and Applied Nutrition, HFS-06, 5100 Paint Branch Parkway 2B64, College Park, Maryland , USA MS : Received 25 August 2003/Accepted 12 March 2004 ABSTRACT Management of risk analysis involves the integration and coordination of activities associated with risk assessment, risk management, and risk communication. Risk analysis is used to guide regulatory decision making, including trade decisions at national and international levels. The U.S. Food and Drug Administration Center for Food Safety and Applied Nutrition (CFSAN) formed a working group to evaluate and improve the quality and consistency of major risk assessments conducted by the Center. Drawing on risk analysis experiences, CFSAN developed a practical framework for initiating and managing risk assessments, including addressing issues related to (i) commissioning a risk assessment, (ii) interactions between risk managers and risk assessors, and (iii) peer review. Management of risk analysis involves the integration and coordination of activities associated with risk assessment, risk management, and risk communication. Risk analysis is used to guide regulatory decision making, including trade decisions at national and international levels. Risk assessment provides information for weighing risk management and regulatory options and is becoming an integral tool for decision makers at the U.S. Food and Drug Administration (FDA). The consensus principles of risk analysis as applied to microbiological food safety concerns have been well described by national organizations such as the National Advisory Committee for the Microbiological Criteria for Foods (NACMCF) (4) and international organizations such as the Codex Alimentarius Commission (2) and the Joint Food and Agriculture Organization of the United Nations and World Health Organization Expert Meetings on Microbiological Risk Assessment (3). In 1993, the White House Office of Management and Budget (OMB) issued Executive Order 12866, which required federal agencies to conduct assessments, including the underlying analysis of cost and benefits of potentially effective and reasonably feasible alternatives to the planned regulations, before publishing new regulations and guidance (9). The OMB later required that risk assessments conducted by federal agencies meet certain data quality guidelines (10), where data quality was defined in terms of objectivity, utility, and integrity. FDA data quality guidelines also require that the information provided must be accurate, reliable, clear, complete, unbiased, and useful (5). The FDA Center for Food Safety and Applied Nutrition (CFSAN) must follow a documented process for conducting and managing risk assessments that will meet these requirements. To assist with continued institutionalization * Author for correspondence. Tel: ; Fax: ; robert.buchanan@cfsan.fda.gov. of the principles of risk analysis, CFSAN developed practical procedures for initiating and managing food safety risk assessments within a risk analysis framework (8). These procedures are the outcome of an introspective evaluation of two major microbial risk assessments. The CFSAN approach to initiating and managing risk assessments is summarized in Table 1 and described in detail here in the hope that others may benefit from our experience. LESSONS LEARNED In early 2001, the FDA published two major microbial risk assessments in draft form for public comment: Listeria monocytogenes in ready-to-eat foods (6) and Vibrio parahaemolyticus in raw oysters (7). About this same time, CFSAN formed a working group to evaluate and improve the quality and consistency of their major risk assessments. The working group was facilitated by a consultant and included senior managers, risk communicators, risk assessors, and risk managers. The key experiences identified and ultimately used to develop a practical framework for initiating and managing risk assessments include issues related to (i) commissioning a risk assessment, (ii) interactions between risk managers and risk assessors, and (iii) peer review. Commissioning a risk assessment. In commissioning a risk assessment, it is critical to establish the reason for the risk assessment, the questions that the risk assessment will address, the key assumptions, the scope, and the required resources. Formally commissioning the risk assessment and clearly identifying its scope are important so that expectations are clear regarding the type of evaluation needed, its intended use in regulatory decision making, and the timeframe for completion. The commitment of staff time to develop and manage the risk assessments has been an ongoing issue for CFSAN

2 J. Food Prot., Vol. 67, No. 9 FDA/CFSAN S RISK ASSESSMENT FRAMEWORK 2059 TABLE 1. Process for selecting and conducting a risk assessment Activity Description Responsible party Identification and selection Concept generation Problem formulation Data feasibility Disposition Collect ideas and maintain list of potential risk management questions for which a risk assessment would assist with policy decisions Review candidate risk assessment and supporting information (justification) determine whether the assessment meets CFSAN regulatory needs Collect and review information to determine availability of data needed to answer risk assessment question(s) Use results of data feasibility determination as an aid for selection of risk assessment(s) to be conducted Program offices and risk assessment coordination staff CFSAN leadership and management teams Risk assessment coordination staff and short-term detailees CFSAN management team Conduct Planning Assemble teams; develop assessment charge, Risk management team statement of resources, and schedule Performing Review and refine (as needed) scope of assessment; Risk assessment team collect data and information; develop and validate model Review Review risk assessment results Risk managers, subject-matter experts, other federal risk assessors, stakeholders Publish Publish technical and interpretive summary documents Risk communication team, risk assessment project manager, risk analysis coordinator because of the resource-intensive nature of the these assessments. The staffing problem is compounded by the fact that it is difficult to assess the actual resources needed until the risk assessment teams have started working and because CFSAN does not have a permanent staff committed to major risk assessments; teams are formed of staff from multiple program offices for part or all of the risk assessment. These factors led to the realization that a coordination of priorities and available resources among the various CFSAN program offices was needed, including a formal mechanism for establishing and periodically reevaluating resource commitments spanning the duration of the project. CFSAN concluded that a full-time project coordinator or manager would improve the process of obtaining the necessary resources (as the needs are identified) to successfully complete the assessment within the specified timeframe. Interactions between risk assessors and risk managers. The 1999 Codex Alimentarius draft document on principles and guidelines for the conduct of microbial risk assessments (3) states that there should be a functional separation between risk assessment and risk management to maintain the scientific integrity of the risk assessment process and to assure that the process is unbiased. However, our experiences suggest that this approach is not ideal for CFSAN. CFSAN found that frequent exchange of information and ideas within and among the risk assessment, risk management, and risk communication teams is necessary. This dialog was particularly necessary at the beginning of the risk assessment process to ensure clarification of the problem being investigated and at the end when the implications of the results are being interpreted. The development of risk assessment questions should be an iterative process, with the risk assessors working with the risk managers to define the questions based on available information to provide meaningful and practical answers. During regular meetings, issues related to assumptions, methods, and models should be discussed, and decisions should generally be reached by consensus. Effective communication with and guidance of the risk assessment team by the risk managers may be hampered by a general lack of understanding of the limitations of risk assessment. One such area of misunderstanding was that the more detailed the risk assessment model, the more complex and uncertain is the result. Others have expressed concern that risk managers may attempt to influence the outcome of the risk assessment to meet preconceived ideas. CFSAN acts to prevent actual or perceived biases by identifying roles and responsibilities and assigning a science advisor. The roles and responsibilities of both groups (risk assessors and risk managers) are clearly agreed upon before initiation of the risk assessment. Risk managers should be responsible for making key decisions related to assumptions or data sets to be used, but the risk assessors are responsible for depicting in the risk assessment the impact these decisions have on the reliability of the results. Risk assessors need to explain the quantitative results of the risk assessment in a manner understandable to the risk management team. Risk assessors must also explain both the impact of any uncertainty and variability identified by the risk assessment and the impact of the various parameters and assumptions used in the model. In case of any misconception between the risk assessment team and risk

3 2060 BUCHANAN ET AL. J. Food Prot., Vol. 67, No. 9 managers, the science advisor will ensure that the science of the assessment is not compromised by the policy needs of the risk management team and will be responsible for resolving any science versus policy issues. The science advisor also is responsible for ensuring that preexisting biases of both the risk assessors and risk managers do not inappropriately affect the assumptions, data, conclusions, or interpretations of the risk assessment. Peer review. A thorough peer review process is needed to ensure scientific integrity of risk assessment documents and models used for regulatory decision making. The purpose of a review is to obtain input from stakeholders, scientific experts, colleagues and agency clearance on scientific information, interpretation, assumptions, and modeling approaches. A consultative peer review may occur at various points during the conduct of the assessment. For example, the design and underlying assumptions of the risk assessment model could be presented initially to an advisory body, such as the NACMCF. The second level of review may occur once the data have been compiled and a specific approach to conducting the risk assessment has been developed. Once the risk assessment has been conducted, a draft risk assessment document should be issued for review both internally (within government agencies) and externally (advisory committees, academic reviewers, and the public). The risk assessment process must be transparent at all levels, including activities associated with initiating, performing, and finalizing the risk assessment. In the two microbial risk assessments evaluated, CFSAN invited public comment on the assessments in the planning stages and encouraged stakeholders to submit scientific data and information to support the assessments. The results of a risk assessment should be presented in such a way that they are clear and can be interpreted as intended. All assumptions, data, and decisions that impact the risk assessment conclusions, the uncertainty in the results, the data gaps, and future research needs should be clearly documented and shared with interested parties. In addition, all resources and information should be clearly documented and available for review. Our review process is consistent with the general guidelines for review of risk assessments described in FDA s Data Information Guidelines (5) and with the OMB guidelines (10). Other lessons. These key lessons are a small proportion of those learned during the conduct of the two risk assessments. Many others have also been reported (1). One of these is the need for higher levels of participation by stakeholders, industry, public advocacy groups, and other interested parties early in the process, especially during the data-gathering stage, so that input from these groups can be incorporated into the model during the initial stages of the risk assessment rather than at the end. As a result of these experiences, specific processes for implementing this framework were proposed: (i) a decisionbased approach for selecting risk assessments and (ii) a step-by-step procedure for conducting the risk assessment. IDENTIFICATION AND SELECTION OF MAJOR RISK ASSESSMENTS The identification and selection process is sequential and deliberate to ensure that the selected risk assessment project will meet CFSAN s regulatory needs and that sufficient data and adequate resources are available to complete the assessment. A major risk assessment requires a substantial commitment of CFSAN resources. Thus, risk assessment is not appropriate when CFSAN s risk managers do not need this level of detail to make a decision. If the science is complex, there is no consensus among stakeholders, and there are multiple ways of managing the risk, then a quantitative risk assessment should be considered. If the science is straightforward, there is general consensus among stakeholders, and there is only one risk management option, a risk assessment is not necessary. The identification and selection process is conducted in four phases. At each phase, a decision is made of whether the proposed project should be further considered. Phase 1: concept generation. During this first phase, ideas for candidate risk assessments are solicited from a number of sources, including stakeholders (consumer advocates, trade organizations, industry), public health institutions, regulatory or management staff, researchers, or other government agencies. Ideas can also be generated from follow-up of a previously completed risk assessment. The candidate ideas are reviewed by program office directors, who forward projects for further consideration based on the significance of the problem and CFSAN s need to develop controls for this issue. Phase 2: problem formulation. During the second phase, CFSAN s senior management team reviews the justification for each candidate risk assessment and recommends an action for each. The following are some example recommendations. 1. The candidate risk assessment should be conducted. 2. The candidate risk assessment should not be conducted because it is not relevant to CFSAN s mission or it is not needed to make a risk management decision. 3. The need for this risk assessment could not be determined based on the information available. If additional or new data are needed, research could be commissioned and obtained before a decision to move forward with a risk assessment is made. These recommendations should account for technical merit, resource availability, and other factors (such as public perception). Phase 3: feasibility determination. In the third phase, published and unpublished research is reviewed to determine the availability of data that will address the risk management question(s) and the corresponding risk assessment question(s). To direct and focus the literature search, it may be helpful to prepare a conceptual model, outlining the risk assessment inputs and the types of outputs needed. The outcome of the feasibility evaluation may include requesting a specific risk assessment, selecting a hazard for which ad-

4 J. Food Prot., Vol. 67, No. 9 FDA/CFSAN S RISK ASSESSMENT FRAMEWORK 2061 ditional research is needed, determining whether a risk assessment should be reconsidered at a later date (i.e., when additional research is available or changes in priorities permit), and conducting a more comprehensive or follow-up risk assessment. Phase 4: disposition (selection). Using the results of the feasibility determination as an aid, senior management selects the risk assessment and commits resources to the project. CONDUCTING THE RISK ASSESSMENT Once a risk assessment is selected, it is formally commissioned. All of the many activities that surround the conduct of a risk assessment within a risk analysis framework have been grouped into a four-step process: planning, performing, reviewing, and publishing. Step 1: planning. Planning is a dynamic, iterative process that requires ongoing interaction between the risk assessment team and the risk managers. During the planning step, the risk assessment, risk management, and risk communication teams are formed. The risk management team develops a charge for the assessment, including the key assumptions that define the scope of the project. The charge should include the specific risk management problem or question and the scope (assumptions and objectives), the risk assessment question(s), the type of risk assessment to be conducted (e.g., risk ranking, product pathway analysis), CFSAN s goal and how the assessment information and conclusions will be used by risk managers, and appropriate timelines. The risk assessment team would ideally consist of a project manager, technical team leader, technical writer and editor, administrative assistant, modeler(s), and subject matter experts. Step 2: performing the risk assessment. The ultimate goal of the assessment is to answer the risk management questions. During the performance step, there is regular interaction between the risk management and risk assessment teams to refine the scope of the assessment, inform the risk management team of progress in completing the assessment, and eventually provide the results and a draft report for review. As the risk assessment is being conducted, the risk communication team may conduct an ongoing interaction with stakeholders to determine their concerns, perceptions, and information needs. Step 3: reviewing the risk assessment. Draft technical and summary documents must be reviewed. The review should include the scope of the assessment, data quality, use of data, assumptions, models, and results. The risk assessments should also be reviewed for reasonableness, general trends, whether the question(s) posed by the risk management team was answered, and whether sensitivity and/ or uncertainty analyses were performed. The risk assessment team should consult with the risk management and risk communication teams throughout the the risk assessment process. Subject matter experts in government agencies and special government employees are helpful in the consultative peer review. Advisory committees such as NACMCF are another means of soliciting expert opinion and consultative peer review. Stakeholder input is a critical component of the process and is necessary for meeting CFSAN s requirement for transparency. Workshops and/or public meetings may be held to provide clarity and obtain input from stakeholders regarding the scope of the assessment, assumptions made, and results. CFSAN has established a process for clearing and approving documents that is designed to ensure that materials released by CFSAN are scientifically and technically accurate. Step 4: publishing the risk assessment. Once the risk assessment is completed and cleared, a plan is needed to coordinate the activities of the risk management and risk communication teams with the issuing of the risk assessment report. This plan is referred to as a rollout strategy, and it includes ways of making the risk assessment results available to the public, such as a Federal Register notice, a press contact list, and a public meeting to present, discuss, and clarify the findings of the risk assessment. Documents should be issued in draft to allow stakeholder review and comment. A discussion of the comments submitted and the ways in which CFSAN has addressed those comments should be included in the revised document. RECOMMENDATIONS The CFSAN Risk Analysis Working Group has made the following recommendations. 1. Develop criteria to evaluate the quality of data used for risk assessments and specify what information is needed to scientifically evaluate the usefulness of a study or data set used for risk assessment. 2. Develop guidelines to evaluate risk assessments and supporting data developed by stakeholders and submitted to CFSAN. 3. Formalize a peer review process that will encourage critical review and evaluation of CFSAN s risk assessments by government and nongovernment experts in a manner that improves the science and acceptance of complex risk assessments. 4. Build capacity to conduct complex risk assessments by providing training opportunities for current staff, hiring new staff or using contractors (as needed), and acquiring additional resources such as computers, software, and dedicated workspace. Since the events of 11 September 2001, additional considerations were included to provide procedures when circumstances require that the availability of information must be restricted when risk assessment is used for food security activities. The progress that has been made since the development of the CFSAN Risk Analysis framework includes (i) adoption of a decision-based approach to identify and select risk assessments conducted by CFSAN, particularly those considered major (complex and impacting or involving multiple offices); (ii) consideration of available resources, regulatory needs, and public health concerns in the selection

5 2062 BUCHANAN ET AL. J. Food Prot., Vol. 67, No. 9 of risk assessments; and (iii) establishment of a procedure for the conduct of risk assessment within a risk analysis framework that identifies the boundaries and responsibilities of key participants in the risk analysis process. REFERENCES 1. Buchanan, R. L., and S. Dennis Conduct of quantitative microbial risk assessment by the U.S. Food and Drug Administration s Center for Food Safety and Applied Nutrition: lessons learned. Background paper for Joint FAO/WHO Consultation, Kiel, Germany. Available at: 2. Codex Alimentarius Commission The application of risk analysis principles in CODEX. Report of the 22nd session of the Codex Alimentarius Commission. Food and Agriculture Organization and World Health Organization, Geneva. 3. Codex Alimentarius Commission Draft principles and guidelines for the conduct of microbiological risk assessment. Joint FAO/ WHO Food Standards Programme. Report of the 23rd session of the Codex Alimentarius Commission, Rome. Available at: ftp://ftp.fao. org/codex/alinorm99/al9913ae.pdf. 4. National Advisory Committee on Microbiological Criteria for Foods Principles of risk assessment for illness caused by foodborne biological agents. J. Food Prot. 61: U.S. Department of Health and Human Services Guidelines for ensuring the quality of information disseminated to the public. U.S. Food and Drug Administration. Available at: gov/infoquality/fda.html#i. 6. U.S. Department of Health and Human Services and U.S. Department of Agriculture Draft assessment of the relative risk to public health from foodborne Listeria monocytogenes among selected categories of ready-to-eat foods. Available at: foodsafety.gov/ dms/lmrisk.html. 7. U.S. Food and Drug Administration Draft risk assessment on the public health impact of Vibrio parahaemolyticus in molluscan shellfish. Available at: dms/vprisk.html. 8. U.S. Food and Drug Administration Center for Food Safety and Applied Nutrition Initiation and conduct of all major risk assessments within a risk analysis framework. Available at: dms/rafw-toc.html. 9. White House Office of Management and Budget (OMB) Executive order regulatory planning and review. OMB, Washington, D.C. 10. White House Office of Management and Budget (OMB) Guidelines for ensuring and maximizing the quality, objectivity, utility, and integrity of information disseminated by federal agencies. OMB, Washington, D.C.