Validation of a Rapid Microbial Method Approach to Validation Case Studies: Equivalence Verification of an Alternate Assay for Microbial Limits Screening and Sterility Testing of Pharmaceutical Products Erin Patton, MS Senior Product Specialist Charles River Labs, Microbial Solutions
Overview Two-tier approach to validating a rapid microbial method Provides a pathway to validation that streamlines the validation and regulatory process Consistent with industry guidance Has been effectively utilized to obtain regulatory approvals both in the US and EU Two Case Studies that utilize approach Study I Comparison of 24-hour Rapid Alternative Bioburden assay and Compendial Pour Plate Method for Micro Limits Testing Study II Comparison of 5-day Rapid Alternative Sterility assay and Compendial Sterility Test using Membrane Filtration
Regulatory Overview Regulators do not approve RMM technologies User responsible for seeking approval for use with their products and/or processes A growing number of companies have received approval for the use of RMMs for finished product testing using various submission strategies Guidance for the Validation of Rapid Microbial Methods PDA TR33 USP<1223> EP 5.1.6 Establish a multifunctional team: Microbiologist, Technician, QA/Regulatory Specialist, Statistician
Validation Approach / Qualitative Method PDA TR33 Guidance PDA TR33 Validation Criteria Equivalence / Comparative Testing Qualitative Method User or Supplier Demonstrated Limit of Detection Yes User Method Suitability Yes User Specificity Yes User Recommended Approach Yes User Equivalency Protocol Generated from Equivalency Protocol Typical Sample Effects Typical Spiking studies Ruggedness Yes Supplier Drug Master File Robustness Yes Supplier Drug Master File
Two Tier Approach to Validation Tier One: Equivalency and Limit of Detection Performed on one product Steps Time to Detect Study (for sterility only) Equivalency Study (Equivalency, Limit of Detection) Tier Two: Method Suitability and Specificity Performed on all other products or product groups Steps Sample Effects (Method Suitability) Spiking Study (Specificity) Use of vendor data to support Robustness and Ruggedness claim including DMF reference
Approach to Equivalence Verification Study Objectives The primary goal is to demonstrate the equivalence of two microbiological test methods for the qualitative screening of a pharmaceutical product for microbial contamination. The ability of the alternate method to detect contamination in a product matrix is compared to that of the compendial method in a series of side-by-side assays conducted on identical samples. Equivalence is tested by measuring the relative rates of agreement and disagreement between the two test methods.
Approach to Equivalence Verification Study Design Demonstrate the comparability of the two methods in a series of side by side assays, on multiple lots of product, with different inoculum levels. Relative rates of agreement and disagreement between the methods are used to assess the accuracy, precision, specificity of the alternate method using a receiver operating characteristics table (ROC). The hypothesis that the two methods are equivalent will be tested using an acceptance criteria of 70% for specificity, accuracy and precision. The hypothesis of superiority will be tested using McNemar s test. Superiority will be defined as a statistically significant difference in the limits of detection between the methods using McNemar s test. The microorganisms chosen for comparison were those recommended by USP/EP/JP Pharmacopoeias
Approach to Equivalence Verification Study Design Fifteen + pairs of assessments are conducted for each organism. Five tests at each concentration of inoculum Three inoculum levels Six plus different organisms McNemar s test is used to test the superiority of the new method, with respect to the ability to detect a contaminated sample When demonstrating equivalence of two methods, USP <1223> requires an examination of Accuracy, Precision, and Specificity High values (>70%) are expected if the methods are equivalent. If the new method is superior, the values for these parameters may be low
STATISTICAL ANALYSIS Approach to Equivalence Verification Study Design Alternate Method Inoculum Load Compendial Test A. brasiliensis C. albicans P. aeruginosa E. coli S. aureus B. subtilis C. sporogenes P. acnes T1 T2 T3 T4 T5 (cfu/test) 10 T1 T2 T3 T4 T5 A. brasiliensis C. albicans P. aeruginosa E. coli S. aureus B. subtilis C. sporogenes P. acnes A. brasiliensis C. albicans P. aeruginosa E. coli S. aureus B. subtilis C. sporogenes P. acnes 1 0.1 *Appropriate organism panel TBD
Alternate - + Approach to Equivalence Verification Statistical Analysis; Receiver Operator Characteristics Table The values in the table represent the number of paired samples having positive/negative results for the alternate test and the compendial test ROC Table Agreement Positive Compendial + A + + C - + - B + - D - - A D B C Agreement Negative Disagreement + / - Disagreement - / +
Alternate - Alternate - - + + Alternate + Approach to Equivalence Verification Interpretation of Hypothetical Results The ROC table tests for equivalence between an alternate method and a gold standard. It does not take into account the possibility that the alternate method may be superior to the gold standard. Inferior Equivalent Superior Compendial + - Compendial + - Compendial + - 18 18 19 8 27 27 18 0 8 19 0 0 Accuracy Precision Specificity Sensitivity 0.00 0.50 0.33 0.50 0.70 0.70 0.70 0.70 0.00 0.50 0.50 1.00 McNemar s test is used to test the hypothesis that the alternate method is superior to the compendial method in its ability to detect a contaminated inoculated sample.
Approach to Equivalence Verification Statistical Analysis; Receiver Operator Characteristics Table Taking into account that the referenced method is the compendial one, the accuracy, precision and specificity of the alternative method are expressed in terms of relative rates of false positive and false negative results. Accuracy, calculated as (A+D) / (A + B + C + D), gives an indication on the equivalency/closeness of both methods. Precision, calculated as A / (A + B), gives a degree of agreement between both methods on the positive results obtained with repeated tests. Specificity, calculated as D / (D+B), gives information on the degree of interference due to external events during the analysis. Sensitivity, calculated as A / (A+C), gives a degree of agreement between the methods taking into account the number of discordant negative results with the alternate method.
Approach to Equivalence Verification Analysis If the rate of positives are not found to be significantly different using McNemar s test, the hypothesis of equivalence is tested using 70% as the acceptance criteria for accuracy, precision, specificity and sensitivity. This specification is based on expected results for each concentration, taking into account the probability of growth of the microorganisms at different concentrations of inoculum. This specification also takes into account the variability linked to very low inoculation levels.
Approach to Equivalence Verification Statistical Analysis; Limit of Detection LOD 1 cfu LOGISTIC REGRESSION
Case Studies Results of two studies designed to demonstrate equivalence of Amplified ATP Bioluminescence assay to compendial methods for detecting microbial contamination in pharmaceutical products Study I Micro Limits Testing of non-sterile, non-filterable betamethasone suspension Direct inoculation protocol Commissioned by Celsis and performed by an independent lab using a validated method and product provided by a commercial client. Study II Sterility Testing of sterile, filterable Saline solution Membrane filtration protocol Commissioned by a pharmaceutical client and performed by an independent lab using a validated method and product provided by the client.
Case Study One: Micro Limits Testing Study Overview / Product Betamethasone Suspension Product Betamethasone Suspension (Betamethasone acetate/betamethasone sodium phosphate 4mg/3mg per ml) Non preserved, non-filterable solution Organisms and Culture Conditions USP compendial plus E. coli organisms were obtained as lyophilized Bioballs Bioballs were dissolved in saline and serially diluted to the final concentration. Compendial Method parameters Microbiological quality was evaluated through plate count method» 1 ml of product» SDA plates incubated at 20-25 C for 5 days» TSA plates incubated for 3 days at 30-35 C.
Case Study One: Micro Limits Testing Study Overview Alternate Method parameters Microbial quality evaluated using AMPiScreen enhanced Bioluminescence assay» 1 ml of Product into 25 ml of TSB.» Incubation for 24 hours at 30-35 C with shaking.» After incubation, samples were agitated on a linear shaker for 30 min. with glass beads» Two 50 µl aliquots of the sample were transferred into duplicate cuvettes and placed into the Advance Luminometer.» Bioluminescence was measured and results were recorded as relative light units (RLUs) Samples with mean RLU values 3X the baseline control were scored as positive
AMPiScreen - + Case Study One: Micro Limits Testing Results A. brasiliensis C. albicans P. aeruginosa E. coli S. aureus B. subtilis AMPiScreen (Broth) Inoculum Load (cfu/test) 10 5 Plate Count (Agar) 6 7 7 3 9 13 7 5 12 9 4 7 6 6 4 10 10 11 5 7 6 11 3 5 8 15 10 10 7 Mean 5.8 9.2 6.4 8 6.4 10 Plate Count + - 42 12 A. brasiliensis 1 2 1 1 C. albicans P. aeruginosa E. coli S. aureus B. subtilis 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 8 28 A. brasiliensis C. albicans P. aeruginosa E. coli S. aureus B. subtilis 0.1 1 1 1
AMPiScreen - + Case Study One: Micro Limits Testing Analysis / Superiority or Equivalency ALL CONCENTRATIONS Plate Count + 42-12 McNemar s Test for Superiority Prob(+ ATP+) = (42+ 12) / 90 = 60.0% Prob(+ PC) = (42+ 8) / 90= 55.6% Statistic (S) = 1.565 P-value (one sided) =.10502 The McNemar s test concluded that the AmpiScreen method was not significantly better at detecting a contaminated sample than the plate count method. 8 28 Receiver Operator Characteristics Accuracy (42+28) / (42+12+8+28) * 100 = 77.8% Precision (42/ (42+ 12)) * 100 = 77.8% Specificity (28 / (28+ 12)) * 100 = 70.0% Sensitivity (42/ (42+ 8)) * 100 = 84.0% Each of the values computed for Accuracy, Precision, Specificity, and Sensitivity satisfied the 70% acceptance criteria. As such, the AMPiScreen method is equivalent to the compendial plate count method for detecting a contaminated sample.
Case Study One: Micro Limits Testing Analysis / Probability of Detection Organism Concentration (cfu/ml of sample) Probability of Detection / Inoculum 10 1 0.1 All AMPiScreen (ATP+) 0.93 0.67 0.20 0.60 Plate Count (PC) 1.00 0.57 0.10 0.56 Difference -0.07 0.10 0.10 0.04
Case Study One: Micro Limits Testing Statistical Analysis; Limit of Detection Logistic Regression Curve for Probability of Detection for AMPiScreen Method Log 10 value of the LOD was 1.85, or 0.014 CFU/mL. Computed LOD 1 CFU Logistic Regression Curve for Probability of Detection for Plate Count Method Log 10 value of the LOD was 1.1, or 0.079 CFU/mL. Computed LOD 1 CFU The probability of detecting contamination at 1 cfu/ml was identical.
Case Study Two: Sterility Testing Study Overview / Product Sterile Saline Product Sterile Saline Filterable solution Organisms and Culture Conditions USP compendial plus P. acnes organisms were obtained as lyophilized Bioballs from biomerieux where available Bioballs were dissolved in saline and serially diluted to the final concentration. Compendial Method parameters Microbiological quality was evaluated through membrane filtration method 250 ml of product was filtered and either TSB or FTM broth added to incubation vessels TSB broth samples were incubated at 20-25 C for 14 days FTM broth samples were incubated at 30-35 C for 14 days
Case Study Two: Sterility Testing Study Overview / Product Sterile Saline Alternate Method parameters Microbial quality evaluated using AMPiScreen enhanced Bioluminescence assay 250 ml of product was filtered and either TSB or FTM broth added to incubation vessels TSB broth samples were incubated at 20-25 C for 5 days FTM broth samples were incubated at 30-35 C for 5 days After incubation, samples were vortexed for 30 seconds Two 50 µl aliquots of the sample were transferred into duplicate cuvettes and placed into the Advance Luminometer. Samples with mean RLU values 3X the baseline control were scored as positive
STATISTICAL ANALYSIS Case Study Two: Sterility Testing AMPiScreen Inoculum Load Sterility Test Results A. brasiliensis C. albicans P. aeruginosa S. aureus B. subtilis C. sporogenes P. acnes T1 T2 T3 T4 T5 (cfu/test) 10 T1 T2 T3 T4 T5 A. brasiliensis C. albicans P. aeruginosa S. aureus B. subtilis C. sporogenes P. acnes 1 A. brasiliensis C. albicans P. aeruginosa S. aureus B. subtilis C. sporogenes P. acnes 0.1
AMPiScreen - + Case Study Two: Sterility Testing Analysis / Superiority or Equivalency ALL CONCENTRATIONS Compendial + 42-11 McNemar s Test for Superiority Prob(+ ATP+) = (42+ 11) / 105 = 50.5% Prob(+ PC) = (42+10) / 105 = 49.5% x 2 Statistic (S) =0.476 The McNemar s test concluded the the Celsis AMPiScreen method was not significantly better at detecting a contaminated sample than the plate count method. 10 42 Receiver Operator Characteristics Accuracy (42+42) / (42+11+10+42) * 100 = 80% Precision 42 / (42+11) * 100 = 79% Specificity (42 / (11+42) * 100 = 79% Sensitivity (42 / (42+ 10) * 100 = 81% Each of the values computed for Accuracy, Precision, Specificity, and Sensitivity satisfied the 70% acceptance criteria. As such, the Celsis AMPiScreen method is equivalent to the compendial plate count method for detecting a contaminated sample.
Case Study Two: Sterility Testing Statistical Analysis; Limit of Detection Logistic Regression Curve for Probability of Detection for AMPiScreen Method Log 10 value of the LOD was 1.00, or 0.1 CFU/mL. Computed LOD 1 CFU Logistic Regression Curve for Probability of Detection for Sterility Test Method Log 10 value of the LOD was 0.65, or 0.224 CFU/mL. Computed LOD 1 CFU The probability of detecting contamination at 1 cfu/ml was identical.
Conclusions Approach to Demonstrating Equivalency to Compendial Methods Experimental Design Adaptive design that places both methods on equal footing» Reveals subtle differences with respect for organisms and sensitivity Equivalency Results demonstrate the systems are equivalent for both micro limits testing and sterility testing Validity of Approach Has been used to successfully obtain regulatory approval
Thank you! Erin Patton erin.patton@crl.com