Metrics in Microbiology Monitoring Practices

Transcription

1 Metrics in Microbiology Monitoring Practices Crystal Booth, M.M. Masters of Microbiology from North Carolina State University Former Associate Director of Microbiology at Novartis

2 Metrics in Microbiology Monitoring Practices An Overview of Microbial Metrics Learn why monitoring metrics in Microbiology is important Review some available guidance on microbial metrics Review some different types of metrics monitored in Microbiology Review some regulatory observations regarding microbial metrics Developing and Maintaining a Microbial Metric System Establish a metric monitoring program Learn how to interpret the results of the metrics Setting acceptance criteria Program Maintenance Interactive Exercise Using an interactive game, participants review information pertaining to monitoring microbial metrics.

3 An Overview of Microbial Metrics

4 What is a metric? According to the Merriam-Webster Dictionary, a metric is a standard of measurement. 16 Synonyms include: Benchmark Criterion Grade Mark Measure Standard ch&sa=x&ved=0ahukewjhk5onwltmahvdrsykha_vaneq_auibigb&dpr=1#imgrc=zszhratd-f3yam%3a

5

6 Why is Monitoring Metrics in Microbiology Important? Utilizing effective metrics help steer the way to success 9 Drive Improvements Focus Resources Assist with Decisions Metrics tell a story 9 Where its been? Where is it heading? Is something going wrong? Has the target been reached?

7 Why is Monitoring Metrics in Microbiology Important? Metrics are essential to quality management. 4 Regulators may view some metrics during inspections Environmental Monitoring Trends Utility System Trends Annual Product Reviews Deviations Out of Specifications to name a few

8

9 Some Available Guidance on Microbial Metrics Current Guidance Documents Allude to the Importance of Metrics. 21CFR Good Manufacturing Practices 1 Acceptance criteria for the sampling and testing conducted by the quality control unit shall be adequate to assure that batches of drug products meet each appropriate specification and appropriate statistical quality control criteria as a condition of their approval and release. The statistical quality control criteria shall include appropriate acceptance levels and/or appropriate rejection levels

10 Some Available Guidance on Microbial Metrics 21CFR Quality System Requirements 2 (medical devices) Quality Policy- Management with executive responsibility shall ensure that the quality policy is understood, implemented, and maintained at all levels of the organization Resources- Each manufacturer shall provide adequate resources, including the assignment of trained personnel to meet the requirements of this part Management Review- Management with executive responsibility shall review the suitability and effectiveness of the quality system at defined intervals and with sufficient frequency according to established procedures to ensure that the quality system satisfies the requirements

11 Some Available Guidance on Microbial Metrics Guidance for Industry: Investigating Out-of- Specifications (OOS) Test Results for Pharmaceutical Production 7 Laboratory error should be relatively rare. Frequent errors suggest a problem that might be due to inadequate training of analysts, poorly maintained or improperly calibrated equipment, or careless work. An assay result that is low, but within specifications, should also raise concern. o This can move across to bioburden testing trends and other microbial data trending such as endotoxin or environmental monitoring.

12 Some Available Guidance on Microbial Metrics USP <1117> Microbiological Best Laboratory Practices 18 A measure of laboratory performance is the number of investigations performed on tests conducted by the laboratory The period of time between sample submission and initiation of the testing should be tracked, as well as the period of time between end of test and report release. o Significant delays in these measures are indicators of under-resourced laboratory staff.

13 Some Available Guidance on Microbial Metrics ISO 9001:2015- Quality Management System Requirements 11 Process are controlled and managed for continual improvements. This International Standard employs the process approach, which incorporates the Plan-Do-Check-Act (PDCA) cycle and risk-based thinking. o The process approach enables an organization to plan its processes and their interactions. o The PDCA cycle enables an organization to ensure that its processes are adequately resourced and managed, and that opportunities for improvement are determined and acted on. o Risk-based thinking enables an organization to determine the factors that could cause its processes and its quality management system to deviate from the planned results, to put in place preventive controls to minimize negative effects and to make maximum use of opportunities as they arise (see Clause A.4).

14

15 Some Different Types of Metrics Monitored in Microbiology Metrics vary from company to company, but ultimately they all tell a story. Metrics can be: Workload Related- Does the lab have the capacity to test the upcoming workload? Inventory Related 15 - Does the supply order need to be increased? Actionable 15 - The contamination rate in the ISO 5 was 3%.

16 Some Different Types of Metrics Monitored in Microbiology Quantitative 15 -A result is calculated and compared to a specification (e.g. AET, Microbial Enumeration Tests, Bacterial Endotoxin Testing) Qualitative 15 - Binary results- either positive or negative (e.g. sterility, gel clot test, test for specified microorganisms) Leading Indicators 15 - Predictive and allow time to fix a problem before a failure occurs (e.g. out of trend) Lagging Indicators 15 -Problem has already happened. Corrective actions are needed to prevent recurrence of the failure (e.g. out of specifications)

17 A Few Regulatory Observations Regarding Microbial Metrics The following slides discuss some regulatory observations that allude to quality metrics.

18 A Few Regulatory Observations Regarding Microbial Metrics FDA Warning Letter Dated 20Feb13: Failure to establish a quality plan which defines the quality practices, resources, and activities relevant to devices that are designed and manufactured, as required by 21 CFR (d). For example, when asked by the investigator,(b)(4), your firm s Regulatory Consultant, indicated that your firm has not established a quality plan. 6 Effective metrics would help plan resources and would establish a routine way to review the suitability and effectiveness of the quality system as required by the regulations.

19 A Few Regulatory Observations Regarding Microbial Metrics FDA Warning Letter Dated 17Dec14: Failure of management with executive responsibility to establish its policy and objectives for, and commitment to, quality, as required by 21 CFR (a). For example, when asked by the investigator, you replied that your firm has not defined, documented, and implemented a quality policy and quality objectives. 6 FDA Warning Letter Dated 17Dec14: Failure to establish procedures for management review, as required by 21 CFR (c). For example, when asked by the investigator, you replied that your firm has not established procedures for management reviews and you did not maintain documentation that any management reviews were conducted. 6 Effective metrics would establish a routine way to review the suitability and effectiveness of the quality system as required by the regulations. Metrics are an important component to the quality system.

20 A Few Regulatory Observations Regarding Microbial Metrics FDA Warning Letter Dated 23Feb16: Our investigators also determined that your monitoring, inspecting, and repair of the RO water system was inadequate in ensuring that it was maintained in a validated state. Beyond the failure to maintain your RO system from January 8, 2014, through October 8, 2014, microbiological test results from water sampled at the RO (b)(4) were TNTC on (b)(4) occasions In your response to this letter, please provide trending results of (b)(4) microbial testing at all water purification system ports to demonstrate the effectiveness of your replaced gaskets, including counts and microbial identification; Effective metrics and monitoring on this occasion would have raised concern for the out of limit samples and negative trends.

21 A Few Regulatory Observations Regarding Microbial Metrics FDA Warning Letter Dated 18Dec15: Your firm failed to establish and adequately staff a quality control unit capable of meeting the responsibilities outlined in 21 CFR This is a repeat observation from your 2007, 2008, and 2010 FDA inspections. In addition, your firm failed to have written procedures covering significant responsibilities of the Quality Unit, as required by (d). Effective metrics and monitoring on this occasion would have provided data to executive management that the laboratory was under staffed.

22 A Few Regulatory Observations Regarding Microbial Metrics FDA Warning Letter Dated 20Feb15: Failure to establish and maintain procedures for implementing corrective and preventive action, as required by 21 CFR (a). For example: Your firm s procedure, Corrective, Preventive and Improvement Actions, XXX does not require an appropriate statistical methodology, in Sections XXX, to detect recurring quality problems. Effective metrics and monitoring for CAPAs was required by the FDA.

23 Developing and Maintaining a Microbial Metric System

24

25 Establish a metric monitoring program Good Metrics are: Consistent 15 Well defined 15 Objective 15 Goal-driven (assist with collection, organization, and analysis) 15 Easy to collect and interpret Set metrics carefully or they could lead to incorrect or false leading information. Metrics should not encourage employees to take negative actions in order to achieve good metric ratings 9

26 Establish a metric monitoring program Keep metrics simple. Metrics should be limited to most important metrics that are linked to driving a successful organization 4 Establish a balanced set of metrics focused on past, present, and future in order to evaluate overall performance4, 17 Metrics should align with company goals, reflect priorities, and provide a window on performance Metrics should be challenging, worthwhile, and attainable. Metric should assist with continual quality improvements

27 Establish a metric monitoring program Effective metrics can be set-up using 5 steps 9 1. Define the metrics- metrics should be clearly defined. 9 To keep metrics understandable, use SMART Metrics 9 Specific Measurable Achievable Relevant to the business Time-bound 2. Secure buy-in from senior management and employees 9 Culture change has to be lead from the top. High-level endorsement and open communication is needed to get everyone on board.

28 Establish a metric monitoring program 3. Understand the data and how to collect it 9 Metrics need to be standardized so the same answer is achieved every time. 4. Measure and share the results 9 Better to have 5 meaningful metrics that the company will actual use. Share the results and learn from others. 5. Don t forget the continual part of improvement 9 Metrics will need to be revised from time to time. Businesses evolve overtime and metrics need measure the data they are intended to measure.

29 Establish a metric monitoring program The plan do check act cycle is another tool to utilize in a metric monitoring program. The PDCA cycle is a four step model for carrying out change and can be useful when planning data collection and analysis in order to verify and prioritize issues. There are 4 steps to a PDCA cycle that can be repeated repeatedly until the desire outcome or improvement is reached. 1. Plan the change- what is the current condition and what is the desire target condition? What are the obstacles getting in the way of reaching the target condition? 2. Do- Make a minor change to overcome one obstacle and test the change with a small-scale study. 3. Check- Review the result of the study. What has been learned? 4. Act- Take action based on what was learned in the study step: If the change did not work, go through the cycle again with a different plan and target a different obstacle. If the study was successful, the learnings can be incorporated into improved target conditions or metrics.

30 Establish a metric monitoring program Some popular metrics used in Microbiology in the industry include the following Tests Completed within Target Percent of Out of Specification Lead Times Samples Percent Deviations Closed on Utility Trends Time Environmental Monitoring Trends Number of Open Deviations Percent Contamination Rate for Number of Overdue Deviations Grade A Percent of Repetitive Deviations Percent Contamination Rate for Percent CAPAs Closed on Time Grade B Number of Open CAPAs Number of Failed Media Fills Number of Overdue CAPAs Percent Human Error Percent Change Controls Closed Percent Method Error on Time Percent Product Failure Number of Open Change Percent Raw Material Failure Controls Percent Instrument Error Number of Overdue Change Bioburden Trend Data Controls Endotoxin Trend Data Percent Right First Time Number of Sterility Test Failures

31 Establish a metric monitoring program Metrics can change based from company to company and what is important for a particular company to track for success. The next few slides show several sets of metrics and benchmarks four different companies and the metrics they utilized in their microbiology laboratories.

32 Establish a metric monitoring program Metric Company Company Company Company A B C D Annual Product Reviews X X X X 6S Monthly Audit Score X 6S Weekly Audits X Any New Environmental Monitoring Predominate Microorganisms X Any New Utility Predominate Microorganisms X Bioburden Cycle Time X Bioburden Trend Data X X X Daily Huddles X Endotoxin Trend Data X X X Environmental Monitoring Number of Action Limit Excursions X Environmental Monitoring Number of Alert Level Excursions X Environmental Monitoring Trends X X X Number of assigned scheduled roles X Number of CAPAs Past 2 Extensions X

33 Establish a metric monitoring program Metric Company A Company B Company C Number of Change Controls Open Greater than 90 Days X Number of Change Controls Past 3 Extensions X Number of Deviations >60 Days Old X Number of Failed Media Fills X X X Number of Open CAPAs X X X Number of Open Change Controls X X X Number of Open Deviations X X X Number of Operators Disqualified due to X X X Gowning Failure Number of Overdue CAPAs X X X Number of Overdue Change Controls X X X Number of Overdue Deviations X X X Number of roles run below designed spec or greater than levelled demand X Number of Sterility Test Failures X X X Number Unplanned roles or test sessions run X Percent Aborted Assay X Company D

34 Establish a metric monitoring program Metric Company Company Company Company A B C D Percent CAPAs Closed on Time Monthly X Percent CAPAs Closed on Time Year to Date X Percent Change Controls Closed on Time Monthly X Percent Change Controls Closed on Time Year to Date X Percent Contamination Rate for Grade A X X X Percent Contamination Rate for Grade B X X X Percent Deviations Closed on Time Monthly X Percent Deviations Closed on Time Year to Date X X X Percent Human Error X Percent Instrument Error X Percent Invalid Assay X Percent Lab Error X X Percent Method Error X Percent of assigned roles completed on time X Percent of Out of Specification Samples X X X

35 Establish a metric monitoring program Metric Company Company Company Company A B C D Percent of Repetitive CAPAs X Percent of Repetitive Deviations X Percent Product Failure X Percent Raw Material Failure X Percent Right First Time X X X X Percent System Suitability Failure X Tests Completed within Target Lead Times X Utilities Number of Action Limit Excursions X Utilities Number of Alert Level Excursions X Utility Trends X X X Weekly board elements in use & up to date X Weekly Huddles X

36 Establish a metric monitoring program FDA Metric Initiative Under Title VII of the FDA Safety and Innovation Act (FDASIA) Public Law No , FDA may require the submission of any records or other information that FDA may inspect under section 704 of the FD&C Act, in advance or in lieu of an inspection, by requesting the records or information from a person that owns or operates an establishment that is engaged in the manufacture, preparation, propagation, compounding, or processing of a drug. 8 The FDA, PDA, and ISPE have been working to obtain a set of industry defined metrics. The FDA intends to use quality metrics data, in context with other sources of quality data, to address risk-based inspection scheduling, predict potential drug shortages, and possibly mitigate potential drug shortages. 8

37 Establish a metric monitoring program FDA Metric Initiative As of July 2015, the draft guidance to the industry says the FDA intends to calculate the following 8 Lot Acceptance Rate Product Quality Compliant Rate Invalidated Out-of-Specification (OOS) Rate Annual Product Review (APA or Product Quality Review (PQR) on Time Rate These are subject to change as the program matures and as the industry provided information to the draft document gets reviewed. Microbiology labs will contribute to the metrics generated by the Quality Unit, but will probably be directly responsible for calculating their own Invalidated OOS Rates.

38 Establish a metric monitoring program FDA Metric Initiative The draft document also says that the following data will be aggregated and reported in a readily accessible format 8 The number of lots attempted of the product The number of specification-related rejected lots of the product, rejected during or after manufacturing The number of attempted lots pending disposition for more than 30 days The number of OOS results for the product, including stability testing The number of lot release and stability tests conducted for the product The number of OOS results for product release and stability tests for the product which are invalidated due to lab error The number of product quality complaints received for the product The number of lots attempted which are released for distribution or for the next stage of manufacturing the product If the associated APRs and PQRs were completed within 30 days of annual due date for the product The number of APRs or PQRs required for the product

39

40 Days Interpreting the Metric Results The data always tells a story and it can be interrupted differently depending on how the data is sorted Assay Turn Around Time It is beneficial to have a limit or goal to visually track the progress of the metric Metrics need to be meaningful. Reporting numbers will be meaningless unless they are compared to a benchmark to assist with continual improvement initiatives /30/2016 5/2/2016 5/4/2016 5/6/2016 5/8/2016 5/10/2016 Date 5/12/2016 5/14/2016 5/16/2016 5/18/2016 5/20/2016

41 Number of OOS % of Total OOS Interpreting the Metric Results Routine graphs and bar charts may not give enough detail to capture a comprehensive story Pareto Chart Example Pareto charts can be useful in interrupting data. 15 Pareto charts contain both lines and bars. Values are represented in decreasing order by the bars and the line represent the cumulative total. The left y axis usually shows the frequency of occurrence or unit of measure while the right y axis shows the cumulative percentage of the frequency of occurrence or unit of measure In-Process Bioburden Growth Promotion Microbial Limits Endotoxin Sterility Test Type Count Cumulative Percentage

42 Number of OOS % of Total OOS Interpreting the Metric Results More Pareto charts can be used to get more granular to focus on specifics. 15 For example, In-Process Bioburden accounted for 54% of the OOSs in Microbiology. Pareto can be used to break that down into product type, investigations, actions, etc. Pareto Chart Example Test Type Count Cumulative Percentage

43

44 Setting Acceptance Criteria As metrics can change based from company to company and what is important for a particular company to track for success, the acceptance criteria should change as well. Metrics should align with company goals, reflect priorities, be challenging, worthwhile, and provide a window on performance Metric should assist with continual quality improvements An acceptable starting point to utilize the history of past performance and set the acceptance criteria to a 95% confidence interval. That is to say, 95% of data points will be predicted to fall within the acceptance criteria.

45 Setting Acceptance Criteria For example- Theoretically, it takes 8 days to release a bioburden sample. Historical data demonstrates that several data points take up to 30 days to release. Setting the acceptance criteria to 8 days initially may cause the project to fail. Metrics should be set carefully. Instead, research the data and set a SMART (Specific, Measurable, Achievable, Relevant, and Time-bound) Metric. The 95% confidence interval calculation shows that 95% of the samples could have been released within 15 days. The initial acceptance criteria is set for 15 days. The data is researched further using Pareto charts and it is discovered that the samples taking longer than 15 days to release are samples that are sent off site for identification or have been sitting in a peer review queue.

46 Setting Acceptance Criteria PDCA cycles (plan, do, check, act), identify an obstacle of peer reviewers not knowing when data is ready for peer review. An experiment is attempted where peer reviewers run a query for data ready for review every Tuesday and Thursday. After two weeks, the data is trended and all available data has been peer reviewed within 10 days, even the data requiring off site identifications. The experiment is changed into routine practice as a peer review wheel and becomes part of the routine workload. The new acceptance criteria is changed to 10 days. This particular assay is not rate limiting, the metric is specific, measurable, achievable, relevant, and time-bound.

47 Setting Acceptance Criteria 95% of the samples are expected to meet a 10 day TAT. Exceptions would be investigations (e.g. OOSs, deviations, etc.), or backlogs at the contract identification laboratory. Further enhancements could be bring identifications in-house or to invest in a rapid microbial system.

48

49 Program Maintenance Once metrics are established, they should be continually monitored and adjusted when needed. Metrics should assist with continual improvements Metrics should be aligned with company goals, reflect priorities, assist with continual quality improvements, be attainable, be worthwhile, and goal-driven. As the companies goals change, it may be necessary to reevaluate the metrics being collected in the laboratory to realign with the company goals. New regulations may also drive new or improved microbial metrics.

50

51 Summary Metrics are essential to quality management, they drive improvements, focus resources, assist with decisions, and tell a story. Some guidance documents allude to the need of monitoring metrics (21CFR211, 21CFR280, Guidance for Industry on OOSs, USP <1117>, and ISO 9001:2015). Different types of metrics vary from company to company. They can be workload related, inventory related, actionable, quantitative, qualitative, leading indicators, or lagging indicators. Some metrics are currently reviewed by regulators during inspections.

52 Summary Metrics should be kept simple and limited to the most important goals that drive success. Metrics should align with company goals, reflect priorities, be challenging, worthwhile, attainable, provide a window on performance, and assist with quality improvements. Effective metrics can be established by 1. Clearly defining the metric 2. Obtaining stake-holder buy in 3. Understanding and collecting data 4. Measuring and sharing metric results 5. Focus on continual improvements

53 Summary The FDA intends to use quality metrics and other quality data to help establish risk-based inspections, predict potential drug shortages, and possibly mitigate potential drug shortages Data always tells a story and can be interrupted differently depending on how the data is sorted. Pareto charts can be useful when interrupting data. Just as metrics change from company to company, so do acceptance criteria. A useful starting point to set acceptance criteria is to utilize historical performance and set the criteria to a 95% confidence interval. PDCA cycles can be used to fine tune laboratory practices and overcome obstacles.

54 Summary Metrics should be continually monitored and adjusted as needed. Metrics should be reevaluated and realigned as the company goals or regulatory expectations change.

55 Using an interactive game, participants review information pertaining to monitoring microbial metrics.

56 BINGO 1. Metrics are essential to management. Quality 2. Metrics can vary from company to. Company 3. A leading indicator is predictive and allows time to fix a problem a failure occurs. Before 4. A metric that is calculated and compared to a numerical specification (e.g. microbial enumeration test) is a metric. Quantitative 5. Good metrics are easy to collect and. Interpret

57 BINGO 6. Metrics should be kept. Simple 7. Metrics help drive improvements. Continual 8. The Plan-Do-Check-Act cycle is also known as. PDCA 9. Specific, Measurable, Achievable, Relevant, and Timebound is also known as. SMART 10. The FDA intends to use quality metric data and other quality data to establish a inspection schedule. Risk-based

58 BINGO 11. Data can be interpreted differently depending on how it was. Sorted 12. charts can be useful when interpreting data. Pareto 13. When setting acceptance criteria, performance should be considered. Historical % interval means that 95% of data points are predicted to fall within the acceptance criteria. Confidence 15. Good metrics are well. Defined

59 BINGO 16. Set metrics carefully or they could lead to information. Incorrect 17. Establish a balanced set of metrics based on, present, and future data. Past 18. Metrics should with company goals. Align 19. Metrics should not encourage to take negative actions. Employees 20. Measure and the results. Share

60 BINGO 21. Secure buy-in from management. Senior 22. Understand the and how to collect it. Data 23. New may drive new or improved microbiology metrics. Regulations 24. Metrics tell a. Story

61

62