REPORT OF THE QUALITY MANAGER Development of Standard Operating Procedures as foundation of EFSA s Quality Management System

Transcription

1 QUALITY MANAGEMENT REPORT OF THE QUALITY MANAGER Development of Standard Operating Procedures as foundation of EFSA s Quality Management System In line with EFSA s Quality Policy (March 2008), EFSA continued in 2009 the work started a year earlier by identifying and documenting the procedures needed for its Quality Management System and key processes critical for the realization of its mission, strategy and policies. In addition, EFSA continues to implement actions necessary to achieve the planned results and continual improvement of the processes and procedures in the course of Since its inception, EFSA s Quality Management has been steering the improvement process, involving staff, measuring progress over time, ensuring consistency, identifying clear roles and responsibilities and achieving best practices. The activities in Quality Management are coordinated and managed by EFSA s Quality Manager, and championed and supported by EFSA s Directors and members of task forces and other groups and teams. For instance, at least 2 Units in the Directorates of Risk Assessment (RA) and Scientific Assistance and Cooperation (SCA) are involved in the drafting of the text of a Standard Operating Procedure (SOP). Upon initial agreement of the draft, the SOP is then consequently being discussed in one or more scientific meetings and being consulted by other relevant Units and/or Directorates. This approach is considered to be efficient and effective since the staff members dealing with the content of the SOP feel empowered and own the SOP. To successfully apply its Quality Management System, EFSA has been: Identifying and documenting the processes needed for the Quality Management System and key processes critical for the realization of EFSA s mission, strategy and policies Determining quality objectives and requirements from regulations, partners, stakeholders and others for the output of EFSA s services and deliverables Determining, documenting, analysing and evaluating the risks related to these processes and introduce and implement control measure to determine or mitigate the risks 1/16

2 Determining and documenting criteria and methods needed to ensure that both the operation and the control of these processes are effective Ensuring that the internal controls cover the policies and procedures established, documented, implemented and maintained by EFSA s management and that there is documented evidence that the output of the service or the deliverable meet the requirements Determining required verification, validation, monitoring, quality control, compliance and test activities specific to the output and the criteria for output acceptance (e.g. deliverables agreed in a framework contract) Ensuring the allocation of appropriate resources and the availability of information necessary to support the operation and monitoring of the processes, allocate available resources according to the prioritization resulting from the risk analysis, and document and communicate the risks related to the unavailability of adequate resources Implementing actions necessary to achieve the planned results and continual improvement of the processes. In 2009, the main focus in relation to SOPs was on harmonizing and documenting best practice in the areas of Organising scientific meetings : targeted to staff members in EFSA s Science Directores and Units. Enabling the organisation (from the decision to the followup) of a scientific (Committee, Panel, Network or Working Groups) meeting. Science internal project : enables the encoding, recording and updating of EFSA science internal projects in the Risk Assessment Workflow system Conducting a survey for data collection : targeted to those staff members who are involved in (1) the drafting of and/or analysis of a survey, (2) the drafting of and/or analysis of a call for data, and (3) a survey organized by a third party. Archiving : this SOP gives an overview of the archiving process from the time the decision has been taken to archive a record up to storage and destruction of the record. The documentation also includes EFSA s Policy on Archiving and guidelines on what, who, when, etc. to archive. Incoming mail : this SOP enables all EFSA staff to register and process nonconfidential incoming mail in a harmonised and consistent manner across all Directorates and Units. Press releases and webstories : the procedure steps in this SOP facilitates the drafting, approval, pre-notification, publication and dissemination of corporate and scientific press releases and web stories. Correcting published scientific outputs : targeting all members of staff in EFSA s RA and the Scientific Committee and Advisory Forum Unit (SCAF) dealing with correcting published scientific documents, this SOP has been developed to enable the correction of any relevant already published EFSA scientific output in a consistent and transparent way by the use of the procedural steps. Planning Communications activities : the aim of this SOP is to explain the cycle of the calendar for communication activities and how it interacts with the different planning processes at EFSA. 2/16

3 Preparing a scientific output for publication : this SOP applies to all staff in EFSA s Communications, RA, SCA and SCAF who deal with preparing and publishing an EFSA scientific output. Developing and approving a procedure and its review : One SOP enables the preparation, development, presentation and approval of a SOP at EFSA in a consistent way. Another SOP aims to harmonize and standardize the process to review an EFSA SOP in a timely manner. Following the Plan-Do-Check-Act approach, EFSA aims at continuous improvement and long-term goals whereby its processes are regularly evaluated and improved in the light of their efficiency, effectiveness and flexibility. EFSA also focuses on satisfying staff & stakeholders/beneficiaries and manages by processes and facts whereby the related activities are systematically measured and monitored. 3/16

4 2. EFSA s review process 2.1 Introduction Based on a proposal from the Scientific Committee (Annex 1), EFSA decided to develop and implement a proposal for a review system for the quality and consistency of EFSA s scientific activities 1. The INEX review system consists of : 1. Self-review: During the development of each opinion or other scientific document the compliance with best scientific practice is checked by; 2. Internal review: After adoption by the Scientific Panel(s) or Committee, EFSA network or working group a sample of EFSA draft opinions or other scientific documents is reviewed by senior scientific staff not involved in the preparation or adoption of the opinion; 3. External review: A number of EFSA adopted scientific outputs are reviewed by independent scientists. These reviews are intended to contribute to improving the quality, clarity and consistency of the scientific advice provided by EFSA in its scientific outputs and ensuring that best scientific practice is followed. The self review and internal review were developed and launched in 2008, with improvements being made in The first external review was carried out in A Standard Operating Procedure on the self and internal review was implemented in April 2008 in order to ensure harmonization across all scientific Units. In addition, an SOP on the external review is being developed. 2.2 Self review Methodology The questionnaire with 30 questions (Annex 2) covers the entire process from developing a scientific document to the delivery and follow-up. The questions relate, amongst others, to: clarity of the Terms of References (ToR), (renegotiated) deadline, identification of conflicts of interests and actions taken, composition of Working Groups, 1 Application opinion/ Generic opinion /Data Collection Report/External Technical and Scientific Report/ Internal Technical and Scientific Report/Guidance Document / Statement 4/16

5 collection, completeness and compliance of data, assessment of studies, addressing of data uncertainties, minority opinions, quality of the summary, actions for follow-up Results Out of 16 units submitting the self-review reports, 1 found none or did not report any problems identified during the process of producing a scientific outputs. Among those reports where areas of improvement were identified, the negotiations of deadlines (n=296), the delays in deadlines (n=75) as well as the clarification of the ToRs (n=33) were most often mentioned (15 out of 16 Units identified at least one of these issues as an area for improvement). As the above mentioned trend also occurred in 2008, it is suggested that EFSA should continue to pay special attention to deadline (re)negotiations, taking into account the complexity of the dossier, the workload of the Panel, Network or Working Group. In order to improve this, EFSA has already initiated the formalisation agreements on any new deadlines for output delivery with the EC, and has established a short-term roadmap with the expected workload, including the timetable. In analogy, EFSA is also setting up a mid-term planning. However, for the applications, legal provisions determine the deadlines. Since (re)- negotiations are therefore only possible in certain instances, good planning and prioritization of the overall work is critical. Good collaboration with the requestors is also needed during the process so that the work can focus on the most essential parts of the question. 2.3 Internal review Methodology Randomly selected scientific outputs were reviewed by EFSA, i.e. 2 scientific internal reviewers and 1 communications specialist anonymously and independently addressing a set number of questions. The Head of Unit formulated a reply to the comments of the 3 reviewers in line with SOP 0002 Self and Internal Review Results In general, the comments of internal reviewers were positive and the scientific outputs were seen to be up to date, concise, well focused with sources of evidence and uncertainties well described. The objectives were clear, the text well written and English in general good. More specifically, the share of the negative comments (C and D scores) out of all 540 scores received for individual areas evaluated, was quite low (9%). The three questions for which these comments were made most often were questions 1, 5 and 8, i.e. if the 5/16

6 ToRs were adequately addressed (13%), if the information/data was adequately presented (24%) and if the conclusions were substantiated in the main body of the text (13%) (total of 23 of all the 540 comments which equals 4,2%). The 3 questions which received higher scores (A and B scores) are 2, 9 and 12, i.e. respecting the boundaries between risk assessment and risk management (100%), coherence between the conclusions and the main body of the text (91%), and a summary which is adequately informative and clear for the risk managers (93%). These questions received a total of 23,7% (128 out of 540 comments) A and B scores. During the internal review of 2008, question 9 was considered as one of the main areas for improvement which shows that the efforts being done in the past year, resulted in this positive trend. The questions for which not applicable was replied, was for question 10 on minority opinions (100% since none of the selected outputs for internal review were minority outputs), and questions 4 and 7, i.e. appropriate applied statistical and other analyses methods and a clear indication of further research/information and a timeline for followup in the case of any provisional/interim conclusion, as the internal reviewers did not feel competent to evaluate these questions. 2.4 External review Methodology Following a call for expressions of interest and a selection process, the External Review Working Group (ERWG) was established. Comprised of 23 experts, the ERWG met in Parma in October 2009 to agree on, and identify areas for improvement, operating procedures and timelines for the review of the areas of activity: i) Chemical risk assessment and connected fields (2 groups) ii) Nutrition and novel foods iii) Biological risk assessment and zoonoses data collection iv) Animal health and welfare v) Plant health vi) Genetically modified organisms vii) Risk assessment methodologies and emerging risks. A total of 35 scientific outputs selected by EFSA, from 16 out of its 17 Units were reviewed. These were subdivided into the above-mentioned 7 areas of activity which led to 8 2 External Review Subgroups (ERS) with each output being assigned a rapporteur and chaired by one expert. Each output was reviewed by a rapporteur using the predetermined template. These were then used by the ERS Chair to generate the Unit Report for each of the 16 Units. During the second meeting of the ERWG in Brussels (November 2 One ERS for each of the 7 area of activity, except for Chemical risk assessment and connected fields which consisted of 2 ERSs. 6/16

7 2009) in depth discussions were held and reported to the EFSA Secretariat and Heads of Unit. The assembled ERWG jointly prepared the final key sections of the review report, including the recommendations and discussion. The external review was applied to a sample of EFSA scientific outputs, adopted and published between 1 July 2008 and 30 June The selection of outputs by unit was made by EFSA before the first meeting of the ERWG. The sample of selected scientific outputs to be externally evaluated, consisted of 2 outputs per Unit, except for EMRISK which had one. All outputs to be externally reviewed had also been self-reviewed and some had also been internally reviewed. Those ERS with outputs assigned only from one Unit, were provided a third output to evaluate. This output was considered to be sensitive and was also self-reviewed. Self and internal evaluations were made available to the reviewers but were not taken into account by external reviewers except in three cases. The Units were given the opportunity to share the overall report and the specific results for their Unit with the Scientific Committee/Panel/Network Results The major outcome of the review was that a high proportion of the outputs were well constructed, transparent and easily understood Recommendations for further development of the quality of EFSA s scientific outputs In some cases (5 out of the 35 outputs externally reviewed) the ToRs are unclear which leads to complications throughout the output. In some outputs, the terms of reference were not adequately addressed. EFSA should: i) where possible exercise their judgement in the adoption of ToRs and/the compilation of ToRs in a output, ii) be explicit about clarity issues regarding the ToRs, if necessary by a commentary within the output, iii) cross-reference answers/chapters within the output to specific ToRs (eg by numbering). In the event that the ToRs are unclear, the quality of the output may be compromised as there is a relationship between the quality of the ToRs and the final output. As very clear ToRs generally led to clear outputs, it is recommended that EFSA be more empowered to have a proactive role in the refinement of ToRs and their adoption considering how ToRs are ascribed to particular outputs. Outputs should be structured in a standard way that is appropriate for that type of output to enhance harmonisation and transparency. It was noted that structures of some outputs varied greatly in their layout from accepted norms. For example, risk assessments should always adhere to a recognised risk assessment framework and if not, clarity should be given for why not. In addition phraseology regarding the magnitude of the risk 7/16

8 should be defined. This will improve the value, readability and consistency of opinions. The ERWG recommends that the pathway from each ToR through methods/results to the conclusion(s) is clarified in order to enhance transparency and validity of the conclusions within each output. This would be reached by: o Following guidance provided in the Transparency Guidance, especially regarding the data sources, selection and incorporation which are used to form the conclusions. o Drafting a summary of the evidences before the discussion or conclusions. Where all evidence is not presented or embodied, justification should be given for the selection process. o The output should be self contained regarding key evidence in support of the conclusions. Where necessary electronic links to key supporting articles should be given in line with EFSA guidance. o The provision of evidence should apply equally to published articles, panel discussions, expert opinions (including cross-unit discussions) and anecdotal evidence. In some cases (11 out of 35 outputs), a more in depth consideration of uncertainties and limitations should be provided, both at the level of the parameter estimates and integrated final risk estimate as outlined in EFSA guidance. The outputs should be subject to more rigorous proofreading and/or review by an individual with appropriate experience of editing complex technical outputs. Glossaries should be introduced at the beginning of the scientific output on a systematic basis. Following the Scientific Committee s proposal, developing an approach to assess the feedback of EFSA s scientific work by the intended users. 2.5 Recommendations for further development of EFSA s INEX process provided by the ERWG and EFSA Given that the self review and the internal review processes were in a trial phase in 2008, EFSA implemented further clarifications, amendments and improvements in the course of There is a need to review the questions and templates for the self-review and the internal and external reviews so as to make sure they each cover the main areas of interest and this in a consistent manner. More specifically, there is a need for EFSA to : 8/16

9 o o o Introduce templates/formats for different sorts of scientific outputs and clear guidance/training on how to use Revise the questionnaire for both the self and internal review Shorter timeframe in completing the self review questionnaires Based on the experience in 2008 and 2009, the following areas were identified for improvement of the INEX process in EFSA: o Implementing a web-based questionnaire for self-review and internal review in 3-5 years-time. This IT tool would create an electronic database for reviews and could also be accompanied by automatic reminders sent to the reviewers, automatic calculations of working days between eg start and adoption date, total days of internal review, link to outputs via output number of Risk Assessment Workflow (RAW) register etc. The basic data concerning the evaluated output should be filled in by the Unit. In the meantime, the INEX TF would work together with SCA in order to create a custom-made form in MS Excel which would also enable to create reports for collecting and analysing the information. o Self review : Developing different Self review templates for different sorts of scientific outputs. This will allow exclusion of the fields that are never relevant for a specific output. The scientific officers completing the self review are encouraged to complete all boxes in the questionnaire. If a question is not applicable to that particular output, it should say so rather than leaving it empty. For scientific outputs which are highly standardised, generated in large numbers through a welldocumented procedure, the self-review might be further simplified considerably. Working on the definitions of the outputs and provide the Units with appropriate information on coding and handling the number of adopted outputs vs the number of self-reviews. An analysis is needed of what needs completion for different types of outputs. Fields that are not relevant for a particular type of of output can then be dropped. 9/16

10 o Internal review : The internal review questionnaire could benefit from further updating : o External review: Since the mandate and ToR are written by the requestor, the evaluator assessing question 1, should also look if the output has a chapter Interpretation of ToR Question 10 refers to minority opinions. However if there is no minority opinion, this question is not applicable, and therefore, N/A should be added to the A, B, C, D scoring. For evaluating the opinions on applications, guidances for applicants should be given to the evaluators so that it is clear which part is due to guidance Increasing the awareness among the internal reviewers that the target audience are risk managers or scientistis and not laymen. Development of different internal review templates for different sorts of scientific outputs Organise the external review on the basis of a three-year cycle Increasing the timeframe in which the external reviewers need to finalise the exercise Developing different External review templates for different sorts of scientific outputs Question 10 refers to minority opinions. However if there is no minority opinion, this question is not applicable, and therefore, N/A should be added to the A, B, C, D scoring. Re-considering the composition of, currently consisting of 3 experts, and expertise needed in the ERS. 10/16

11 3. Activities planned for 2010 and beyond 3.1 Mapping and developing the Quality Management System 3.2 INEX The Quality Management section will continue to build EFSA s Quality Management System and to develop and promote the implementation of any remaining SOPs which were identified as a priority in the area of: Processing and approving declarations of interest Reporting performance indicators Preparing and launching public consultations Sharing scientific data between EC and EU agencies Security of confidential information in the area of international agreements Business Continuity and emergency response plans Appointing and replacing experts to the Scientific Committee and Panels Finalising, endorsing and publishing other scientific outputs of EFSA Preparing and conducting an external review Conducting ad-hoc data collection to support EFSA s opinions In addition, Quality Management will also continue to make an inventory and map existing processes in the remaining areas of EFSA and will introduce Work Instructions (WI) for those areas where a specific task needs to be described, such as processing of MRL and data collection. Furthermore, EFSA will be taken the next step and start mapping the road to certification, based on QM principles such as customer focus, leadership, involvement of people, process approach, system approach to management, continual improvement, factual approach to decision making and mutually beneficial supplier relationships. Following an analysis of the self and internal review in 2008 and 2009, and the external review in 2009, Quality Management, in cooperation with the Science Directorates, will identify and implement the necessary corrective actions in order to further improve the process. The next external review cycle will take place in the first half of 2011 for scientific outputs adopted between 1 January and 31 December A call for expression of interest to become a member of the ERWG will be prepared and launched in September /16

12 3.3 Assessment of the feedback of EFSA s scientific work Following the Scientific Committee s proposal, EFSA will also start to develop an approach to assess the appreciation of EFSA s scientific work by the intended users. ANNEXES Annex 1 Summary of the Scientific Committee s proposal for a review system Annex 2 - Questionnaire for self-review Annex 3 Questionnaire for internal and external review 12/16

13 Annex 1 Summary of the Scientific Committee s proposal for a review system 13/16

14 Annex 2 Questionnaire for self-review 1. Is there a common understanding between the experts and the originator of the request of what is needed and expected (clarity of terms of reference)? 2. Has a realistic time frame been discussed with the originator of the request with respect to the priority and complexity of the question? 3., when necessary, appropriate external experts have been involved for addressing the particular question (e.g. composition of working groups)? 4. Have the Declarations of Interests of Scientific Panel and Committee members, working groups or other external experts been obtained and checked by the secretariat for potential conflicts of interest before their involvement in the Scientific Panel or Committee and/or its working groups? 5. Where potential conflicts of interest have been identified, have correct procedures for dealing with them, as set out in EFSA guidance, been correctly followed? 6. Have any cross-cutting issues been identified and, if so, have other relevant Panels or internal and external expert groups been appropriately involved? 7. Has any need for external consultation been considered prior to the adoption of the document (especially for guidance documents)? 8. Where necessary, have risk communicators been alerted by the secretariat sufficiently early in the process to ensure appropriate and timely communication? 9. Where data have been provided by outside parties, have the dossiers been checked for completeness and compliance with any relevant guidelines for data submission (e.g. adequacy for addressing the question, compliance with GLP, up-to-date, inclusion of unpublished data and comprehensive literature search)? 10. Where primary and/or additional data have been collected by the SC/P or their WGs (e.g. literature searches, consultation of external experts or national authorities, identification of relevant unpublished data, identification of any previous risk assessments on the same or similar topic by EFSA or other expert bodies) how was it ensured that the dataset was comprehensive enough and up-to-date? 11. Have the data been properly described in the document? 12. Are the sources of information used properly described in the document? 13. Has a check been made that all scientific data have been properly referenced (within the text and in the reference list)? 14. Where relevant, have key studies for the assessment been properly identified and were reasons given for their selection as key studies? 15. Have the implications of potentially contradictory/conflicting data been discussed? 16. Have opinions of other expert bodies been discussed and, if relevant, the reasons for any diverging views been explained? 17. Have significant data gaps been identified and discussed? 18. Have the underlying assumptions, limitations and uncertainties in the data been discussed?3 19. Have existing relevant EFSA guidance documents on risk assessment practices/methodologies been followed? 20. Is the approach followed broadly in accordance with internationally agreed guidance on risk assessment procedures? 14/16

15 21. Does the assessment adequately address the terms of reference? 22. Are the conclusions coherent with the main content of the opinion? 23. Are any assumptions, limitations and uncertainties adequately addressed? 24. In the case of a minority opinion, has it been clearly expressed and reported? 25. In the case of a provisional/interim conclusion, is there a clear indication of what further research or information may be needed and is there a timeline for follow-up? 26. Does the assessment respect the boundaries between risk assessment and risk management? 27. Is the summary adequately informative and clear for the target audience? 28. Does the summary adequately reflect the context of the question and the content of the opinion, including the conclusions and any recommendations? 29. Has the document been delivered on time? If not, please specify. 30. What means are in place for follow-up of the documents, if relevant? 15/16

16 Annex 3 Questionnaire for internal and external review 1. Depending on the type of scientific output : a. Does the assessment adequately address the terms of reference? or b. Are the objectives of the report/analyses clearly specified and stated? 2. Does the assessment respect the boundaries between risk assessment and risk management? 3. Are any assumptions, limitations and uncertainties adequately addressed and reported? 4. Are the applied statistical and other analyses methods appropriate and fit for purpose? 5. Is the information (data) adequately presented? 6. Is there a right balance between major and minor findings? 7. In the case of any provisional/interim conclusion, is there a clear indication of what further research or information may be needed and a timeline for follow-up? 8. Are all conclusions substantiated in the main body of the text? 9. Are the conclusions coherent with and supported by the main content of the scientific output? 10. In the case of any minority output, has it been clearly expressed and reported? 11. Does the summary adequately reflect the context of the question and the content of the output, including the conclusions and, if any, recommendations? 12. Is the summary adequately informative and clear for the risk managers? 16/16