Handling OOS, OOT and Unexpected Results. Karen Ginsbury PCI Pharmaceutical Consulting Israel Ltd for IFF, October 2017

Transcription

1 Handling OOS, OOT and Unexpected Results Karen Ginsbury PCI Pharmaceutical Consulting Israel Ltd for IFF, October 2017

2 What is an OOS

3 What is an OOT

4 What is an Unusual, Questionable, Atypical Result

5 Objective: of this course and BE objective when assessing OOS, OOT Understand what an OOS, OOT, Unusual result is Understand why Data Integrity is a BURNING hot issue Understand what you can and can t do and WHY There is no place for emotions put them aside 5

6 Write Down Keywords relating to OOS

7 Review and Comment

8 KG Review

9 Warning Letter 19 October, 2016 China

10 You do NOT want to go there

11 You do NOT want to go there

12 You do NOT want to go there

13 Unofficial Testing b. Your firm frequently performs unofficial testing of samples, disregards the results, and reports results from additional tests. For example, during stability testing, your firm tested a batch sample six times and subsequently deleted this data. You performed trial sample HPLC analyses prior to acquiring the official analyses. The trial results were subsequently discarded. Trial HPLC analyses were apparently run as part of the 12-month long-term stability studies on batch #15069 for related substances. Your employee ran an HPLC analysis sequence and subsequently deleted the raw data files. Your quality control staff named the samples using the last three digits of the batch numbers to link the "trial" injections with the official assay analyses. Your Senior Quality Control (QC) Officer confirmed these were analyses of batch samples. Furthermore, we found that this batch was analyzed for unknown impurities and results reported as within specifications. However, the data showed that the "trial" injection for this batch failed the unknown impurities specification in several test runs. Your Senior QC Officer confirmed that QC laboratory employees had frequently practiced the use of trial injections at your facility. Significantly, in addition to the example above, our inspection found 5,301 deleted chromatograms on a computer used to operate two HLPC instruments in your QC laboratory. Many of these files were trial injections of batches.

14 Important Disclaimer Karen Ginsbury is a consultant Your company has approved SOPs and a quality system Anything that Karen says is for consideration to stimulate thought and discussion and may result in review or revision of SOPs and procedures. AS LONG AS SOPs ARE NOT UPDATED under change control, you and your company PERSONNEL FOLLOW THE APPROVED PROCEDURE Karen said is NOT an excuse! 14

15 US CGMP s 21 CFR Any unexplained discrepancy of the failure of a batch or any of its contents to meet any of its specifications shall be thoroughly investigated, whether or not the batch has already been distributed. The investigation shall extend to other batches of the same drug product and other drug products that may have been associated with the specific failure or discrepancy. A written record of the investigation shall be made and shall include the conclusions and follow-up. 15

16 EU GMP s Chapter 6: Quality Control Laboratory Documentation should include: A procedure for the investigation of Out of Specification and Out Of Trend results; 6.9 Some kinds of data (e.g. tests results, yields, environmental controls) should be recorded in a manner permitting trend evaluation. Any out of trend or out of specification data should be addressed and subject to investigation. 16

17 Guidance MHRA OOS Guidance FDA

18 DI Guidance MHRA GMP WHO FDA Q&A MHRA GxP PIC/s EMA Q&A January 2015 September 2015 April 2016 July, 2016 August 10, 2016 August 11, 2016

19 STRATEGY DEFINE, EDUCATE, COMMUNICATE ALCOA+ Accurate Legible Contemporaneous (real time) Original Attributable Accurate Complete Consistent Secure 19

20 Controversial Topics BE TRANSPARENT: NEVER delete, alter or hide data Invalidation of results because of laboratory error: under what conditions Reintegration Remeasurement, reinjection Repreparation Retest Resample Reportable result 20

21 Controversial Topics Use of averaging? Definition of reportable values? Number of retests? Second analyst? Use of outlier testing? Defining testing into compliance? 21

22 FDA Guidance 22

23 The Barr Court Case From the New York Times February 6, 1993, Saturday (AP); Financial Desk COMPANY NEWS; Judge Rules On Barr Labs A generic drug manufacturer must recall batches of some of its medicines and stop distributing others until the company completes studies of its manufacturing process, a Federal judge ruled on Thursday. But United States District Judge Alfred M. Wolin refused a request by Federal pharmaceutical regulators to order a complete shutdown 23

24 Barr and OOS Faced with potential closure, the company took FDA to court The judge went into great details as to the meaning and implications of OOS results The outcome: FDA draft guidance: 1998 FDA final guidance:

25 Why the Guide? FDA s current thinking on how to evaluate out-of-specification (OOS) test results The term OOS results includes all test results that fall outside the specifications or acceptance criteria established in drug applications, drug master files (DMFs), official compendia, or by the manufacturer The term also applies to all in-process laboratory tests that are outside of established specifications 25

26 Applicability of Guide Applies to chemistry-based laboratory testing of drugs regulated by CDER Directed toward traditional drug testing and release methods Can we use it for: Microbiological Biological Physical Qualitative tests? 26

27 The Guide Laboratory tests are performed on active pharmaceutical ingredients, excipients and other components, in-process materials, and finished drug products to the extent that current good manufacturing practice (CGMP) regulations (21 CFR parts 210 and 211) and the Federal Food, Drug, and Cosmetic Act (the Act) (section 501(a)(2)(B)) apply Laboratory testing, required by CGMP regulations ( and ) is necessary to confirm that components, containers and closures, in-process materials and finished products conform to specifications, including stability specifications Testing also supports analytical and process validation efforts 27

28 Why does your company perform Tests? What is the purpose of testing? Can you assure a correct result? What is a correct result? 28

29 Why Does your company Perform Tests? To determine if a critical quality attribute conforms or does not conform with its specification You cannot assure (at a 100% confidence level) that you will obtain a correct result The correct result is that which is identical to the true result but You never know the true result because if you did you wouldn t need to test 29

30 Uncertainty Since you cannot assure a correct result there is always some level of UNCERTAINTY associated with any reported laboratory test result Uncertainty represents risk What is the risk associated with reporting an incorrect test result? 30

31 The risk Declare that a result conforms with the specification when the TRUE result actually does NOT conform Declare that a result does not conform with the specification when the TRUE result actually DOES conform (False positive / False negative) 31

32 Some Definitions ReMeasurement: ReTest ReSample ABORTED TEST INVALID TEST INVALIDATED TEST RESULT Reinject same sample preparation Additional test(s) on additional aliquot of same original sample New sample You stop the test e.g. spillage Test acceptance criteria fail (tells you nothing about the results) Found conclusive evidence of laboratory error: method, reagents, performance of test, equipment etc. 32/

33 Reportable Result The Guide The term reportable result as used in this document means a final analytical result This result is appropriately defined in the written approved test method and derived from one full execution of that method, starting from the original sample 33

34 USP Definition of a Reportable Value A reportable value is often a summary value for several individual determinations It is the end result of a completed measurement method, as documented It is the value compared with the acceptance criteria In most cases, the reportable value is used as documentation for internal and external users 34

35 What is an OOS Results A reportable result (obtained after applying the entire method as described and performing any calculations and rounding off the significant figures) which when compared to the specification is outside the limit / range or declared value 35

36 Ever heard this. The Lab don t know how to test Give it to Sheila he knows how to do that test Bruce always gets bad results! 36

37 What Could be the Cause of An OOS Which is most likely? 37

38 How is Testing Performed Collect a sample from the batch Take part of sample (an aliquot) and test e.g. for a liquid: measure out a portion; inject into HPLC or For a powder, take part of the sample (weigh it); dissolve in diluent and inject to HPLC or For a tablet: crush the tablet to make a powder; add diluent to dissolve the powder; inject 38

39 How is Testing Performed Can take one aliquot and prepare for testing Can take duplicate aliquots (two weighings) and prepare for testing Can make duplicate injections (two injections) 39

40 How is Testing Performed In which case, the method will describe the reportable result as Either the average result of the two weighings and two injections with a standard deviation or The average result of each individual preparation (i.e. each preparation / weighing) and not more than a certain RSD between the two (or more) results 40

41 Simplest Scenario (1): One measurement, one preparation, one injection 41

42 Scenario (2): 3 preps; 3 injections, ONE reportable result 42

43 Scenario (3): 3 preps, 3 injections; one reinjection of preparation; repreparation from weighed / crushed powder? 43

44 Apparent Implications The individual determinations do not have to meet the criteria of the reportable result Determinations are not generally reported out of the QC laboratory The variability of determinations is similar to a system suitability issue Set a limit on the standard deviation 44

45 Apparent Implications All reportable results must be documented Do not average OOS with in spec to get an in spec results to release the batch Do not average reportable results for QA to make a batch release decision QA must see all reportable results obtained during testing If after QA makes a decision, and a value is needed for a COA, then average them 45

46 OOS Guide The need to provide all reportable results to the QCU is reinforced In addition, when investigation by a contract laboratory does not determine an assignable cause, all test results should be reported to the customer on the certificate of analysis 46

47 Before Barr Current Practice Prior to 1993 and the court decision it was COMMON practice to retest once or in exceptionally good companies twice and to release the batch if the retest result was within the specification Companies had not really thought about the practice But then nor had the regulators 47

48 Barr: What happened in court The judge heard experts on behalf of FDA and Barr regarding the practice of retesting FDA wanted retesting to be banned under all circumstances After a long hearing at which five industry experts, an FDA investigator, and several company employees testified, Judge Alfred M. Wolin, U.S. District Judge for the District of New Jersey, issued a 79-page opinion 48

49 The Barr Court Case 1993 Data Integrity? Reported problems include misplaced records test data recorded on scrap paper failure to control manufacturing steps such as those governing products' physical properties release of products not meeting their specifications inadequate investigation of failed products 49

50 Barr: Testing into Compliance Barr had numerous failures Performed retests with no investigations no regard for process and product history Tested until results met specifications Then irrespective of previous OOS results for the batch, released product reporting only the passing results Q: How do you report passing OOS s on COA? 50

51 Reading the Judgment The Barr court judgment is like reading FDA s guidance Judge Wolin preferred "out-of-specification" (OOS) laboratory results rather than "product failure" used by FDA's investigators Ruled an OOS result identified as laboratory error by investigation or an outlier test, or overcome by retesting is not a product failure BUT Limited situations where laboratory error could be used 51

52 Guide to Inspection: QC Labs Issued July 1993 (must have been working on it while the court case was ongoing) Addresses OOS results; tells inspectors to be alert Evaluate the company's system to investigate laboratory test failures These investigations represent a key issue in deciding whether a product may be released or rejected and form the basis for retesting, and resampling 52

53 Guide to Inspection: QC Labs OOS results fall into three categories: laboratory error non-process related or operator error process related or manufacturing process error Evaluate the company's retesting SOP for compliance with scientifically sound and appropriate procedures 53

54 Guide to Inspection: QC Labs A very important ruling in one recent court decision sets forth a procedure to govern the retesting program This district court ruling provides an excellent guide to use in evaluating some aspects of a pharmaceutical laboratory, but should not be considered as law, regulation or binding legal precedent The court ruled that a firm should have predetermined testing procedure and should consider a point where testing ends and product is evaluated. If results are not satisfactory, product is rejected 54

55 Collect Data Be Objective It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts. Sir Arthur Conan Doyle ( ) Author of the Sherlock Holmes detective series 55

56 The FDA Guidance III. IDENTIFYING AND ASSESSING OOS TEST RESULTS PHASE I: LABORATORY INVESTIGATION The source of the OOS result should be identified either as an aberration of the measurement process or an aberration of the manufacturing process Even if a batch is rejected based on an OOS result, the investigation is necessary to determine if the result is associated with other batches of the same drug product or other products Batch rejection does not negate the need to perform the investigation The regulations require that a written record of the investigation be made, including the conclusions and follow-up ( ) 56

57 Investigation Hypothesis Testing Retesting If you have a hypothesis, you can test it That is NOT what is generally called retesting and usually should not be performed to obtain a passing result but to recreate the FAILING result

58 The FDA Guidance III. IDENTIFYING AND ASSESSING OOS TEST RESULTS PHASE I: LABORATORY INVESTIGATION To be meaningful, the investigation should be thorough, timely, unbiased, well-documented, and scientifically sound The first phase of such an investigation should include an initial assessment of the accuracy of the laboratory's data Whenever possible, this should be done before test preparations (including the composite or the homogenous source of the aliquot tested) are discarded. This way, hypotheses regarding laboratory error or instrument malfunctions can be tested using the same test preparations. 58

59 Initial Assessment of OOS Result Bear in mind prior: Product history Process history Test history Reliability of equipment Reliability of the analyst Precision of the test (validation) 59

60 The Guide The first responsibility for achieving accurate laboratory testing results lies with the analyst who is performing the test The analyst should be aware of potential problems that could occur during the testing process and should watch for problems that could create inaccurate results In accordance with the CGMP regulations in (b)(4), the analyst should ensure that only those instruments meeting established performance specifications are used and that all instruments are properly calibrated 60

61 Responsibility of Analyst To follow test procedure as written To be alert to errors and STOP test BEFORE obtaining the result if error is suspected, recording what happened e.g. spill Analyst responsible for ensuring that instruments meet performance specifications and are properly calibrated [KSG: maintained?] Once an OOS result is obtained to review all records relative to the test to identify possible laboratory error 61

62 Supervisory Role: USE a checklist Visit the crime scene Supervisors / team leaders / laboratory head: Should be experienced analysts Frequently audit while tests ARE BEING performed in order to be able to objectively investigate OOS results 62

63 Responsibility of Supervisor Objective assessment without preconceived assumptions as to cause of OOS Immediate assessment may include: Re-examination of: actual solutions Test units Glassware used in the original measurements and preparations This could provide more credibility for laboratory error hypotheses 63

64 Supervisor (The Guide) 1. Discuss the test method with the analyst; confirm analyst knowledge of and performance of the correct procedure. 2. Examine the raw data obtained in the analysis, including chromatograms and spectra, and identify anomalous or suspect information. 3. Verify that the calculations used to convert raw data values into a final test result are scientifically sound, appropriate, and correct; also determine if unauthorized or unvalidated changes have been made to automated calculation methods. 4. Confirm the performance of the instruments. 5. Determine that appropriate reference standards, solvents, reagents, and other solutions were used and that they met quality control specifications. 6. Evaluate the performance of the test method to ensure that it is performing according to the standard expected based on method validation data and historical data. 7. Fully document and preserve records of this laboratory assessment. 64

65 Responsibility of Supervisor To INVESTIGATE: Review notebook / worksheet with analyst : Was method was followed: with a copy of the method in your hand, have the analyst describe exactly how they performed each step: confirm that the method was understood & followed Review raw data: Perform calculations again including checking dilution schemes Unauthorised changes to automated calculations Examine reagents, (reference) standards, solutions Examine glassware Performance of instruments 65

66 The FDA Guidance III. IDENTIFYING AND ASSESSING OOS TEST RESULTS PHASE I: LABORATORY INVESTIGATION If this initial assessment indicates that no meaningful errors were made in the analytical method used to arrive at the data, a full-scale OOS investigation should be conducted 66

67 OOS Flow Chart OOS result Laboratory Investigation Report to QA (copy in batch file?) Convincing evidence of Laboratory Error Inconclusive Invalidate result Perform new test on same sample QA Investigation Production Error Sampling Error Inconclusive No evidence of production error Correct if possible or Reject Batch Resample Double sample Revise sampling procedures Retest??? further aliquots from original sample All within specifications One result OOS QA decision regarding batch disposition Reject Batch 67

68 Out of Specification Results If the laboratory investigation is conclusive Document findings INVALIDATE original test Perform NEW test on same sample Report original result with investigation as well as new result in batch record for QA review prior to release COA carries new result only If the laboratory investigation is NOT conclusive inform QA (or customer for contract lab) 68

69 Out of Trend Results If the laboratory investigation is conclusive or inconclusive, consult with QA In most cases, DO NOT perform any additional testing or sampling Make product disposition judgment based on: Original result Product history (e.g. stability data statistical analyses of particular use here) Batch history (e.g. review indicates that there were no processing errors / there were errors) Other investigational findings 69

70 Investigation Use a pre-defined procedure Production/ process review and / or additional laboratory work Identify root cause and implement CAPA QA / QCU responsibility, includes CMOs if used Documented in the batch record Involves all aspects of manufacture, quality control and sampling Describes corrective actions and endpoint Is performed PRIOR to ANY retesting 70

71 Investigation If cause of OOS is identified, batch is rejected In this case need CAPA on process / product May not identify cause and may need additional lab testing: Retest additional portion of original sample Resample 71

72 FDA: Retesting and Resampling Use another analyst? Where possible The maximum number of retests should be specified in advance in an SOP May, on rare occasions, deviate from SOP but with documented rationale and protocol The number may vary depending upon the variability of the test method and NOT depending on the results obtained Resampling raises questions as to sampling procedure validity 72

73 OOS Guidance Is there a fundamental process flaw? Product or process redesign is addressed with a new paragraph. OOS results may indicate a flaw in product or process design. In such cases, it is essential that redesign of the product or process be undertaken to ensure reproducible product quality. A horrible question but needs to be asked AND answered honestly 73

74 Retesting Retesting has an additional requirement. The maximum number of retests to be performed on a sample should be specified in advance in a written standard operating procedure (SOP) Any deviation from this SOP should be rare In such cases, before starting additional retesting, a protocol should be prepared that describes the additional testing to be performed and specifies the scientific and/or technical handling of the data 74

75 Retesting How many repeats? While the guidance does not give recommendations for the sample size for retesting, the example given uses seven retests Seven was the suggestion in a footnote in the Barr Case judgment The sample size question is still unresolved 75

76 Resampling The original sample from a batch should be sufficiently large to accommodate additional testing in the event an OOS result is obtained In some situations, however, it may be appropriate to collect a new sample from the batch. Be very cautious about collecting a new sample 76

77 How to Identify OOT Results Out of Trend or unusual results are generally results that: MEET the product specification (may be borderline) ARE outside the control limits of the process, where control charts are used imoh requiring this routinely or ARE different to results usually obtained e.g. spec: usual results: OOT result:

78 FDA: Concluding the Investigation Evaluate results and determine batch quality Release decision by QCU / QA An initial OOS does not necessarily mean that the batch fails and must be rejected. Where the suspect result is invalidated, the result should not be used to evaluate the quality of the batch or lot For inconclusive results give full consideration to the OOS result 78/87

79 FDA: Concluding the Investigation Example given shows seven retest results which gives the only indication in the guide regarding numbers of retests The example given is also extreme: 89.5% OOS 99.0, 98.9, 99.0, 99.1, 98.8, 99.1, 99.0% Consider method precision and validation data in making release / reject decision 79/87

80 FDA: Cautions Where a series of assay results (to produce a single reportable result) have some individual results OOS and some in spec and all within the known method variability, passing results no more likely to represent the true value than the OOS result. In this case the company must err on the side of caution and reject the batch A result that is low but in specification should also be a cause of concern 80/87

81 Cautions The first paragraph continues a discussion of reportable results stating that a firm should err on the side of caution a low assay result should raise concern and One cause of the result could be that the batch was not formulated properly. Batches must be formulated with the intent to provide not less than 100 percent of the labeled or established amount of active ingredient 81

82 Take Away Information Companies must have an up-to-date SOP Write the SOP at the 8 th grade level Use simple declarative sentences. Use the wording directly out of the guidance where possible; copyright free! Require the operational definition of the reportable result to be in the SOP, analytical test method, retest protocol and reports 82

83 Quality Metrics: are you ready? report number of invalidations for lab error Revisit your investigations With an open mind Ask: Why did we get an OOS result in the first place Could it have been laboratory error? YES Could it have been production error? YES Did we ELIMINATE the possibility of production error? Did we CONCLUSIVELY demonstrate lab error? 83

84 Thank you for your Attention Questions? 84/

85 OOS Keywords Data governance Data integrity ALCOA + OOS OOT Questionable / unusual / atypical result Reportable result Retest New test Invalid test Aborted test New test Resample Batch release Investigation Laboratory error Production error Sampling error Data integrity Data quality Averaging Training, qualification, education COA Reporting Results