A Verification Study for Implementing the Revised CLSI Breakpoints Jean B. Patel, PhD, D(ABMM) Deputy Director, Office of Antimicrobial Resistance National Center for Emerging and Zoonotic Infectious Disease The findings and conclusions in this presentation have not been formally disseminated by the Centers for Disease Control and Prevention and should not be construed to represent any agency determination or policy Summary Why a study is needed Study design Data analysis 2 AST Device Glossary A commercial antimicrobial susceptibility testing method Examples: MicroScan, BD Phoenix, Vitek2, Etest Reference AST method A standardized, non proprietary antimicrobial susceptibility test method Examples: CLSI frozen broth microdilution, disk diffusion 3
Glossary Federal laboratory regulation The regulation: CLIA The enforcers: CAP and Joint Commission FDA The federal agency which regulates antimicrobials & the AST devices Information is in the product label or package insert 4 CLSI Glossary A not for profit laboratory standards organization Describes AST reference methods & AST interpretive criteria AST interpretive criteria Criteria for categorizing a bacterial isolate as susceptible, intermediate, or resistant to an antimicrobial Also known as breakpoints 5 Verification Glossary The documentation of either commercial or laboratory developed tests to determine or confirm test performance characteristics before the test system is used for patient testing Validation The documentation that a test which has already been verified is repeatedly giving the expected results as the test is performed over a period of time. 6
Why are there Different Breakpoints for the Same Drug? CLSI FDA Different types of organization Different processes for setting breakpoints 7 Breakpoint Differences Cephalosporin Breakpoints for Enterobacteriaceae CLSI & FDA ceftriaxone breakpoints revised in 2010 8 Breakpoint Differences Carbapenem Breakpoints for Enterobacteriaceae 9
Updating FDA Breakpoints Ceftriaxone breakpoints revised 10 Which Breakpoints Can My Lab Use? Lab regulatory agencies Joint Commission & CAP allow labs to use either CLSI or FDA breakpoints Lab regulatory agencies also require that commercial antimicrobial susceptibility testing (AST) devices Either meet FDA performance standards or Be verified for acceptable performance by the lab using the test 11 The Issue AST device manufacturers are required by law to use FDA breakpoints Panels can be updated with new drug dilutions for reporting lower MICs if the manufacturer has previous clearance for those drug dilutions Panels cannot be updated to provide new interpretations based upon non FDA breakpoints 12
Consult with stakeholders If using disk diffusion you can implement revised breakpoints without a validation study If using a device, a validation study is needed 13 IDSA notified members of revised CLSI breakpoints http://www.idsociety.org/content.aspx?id=17429 See Appendix A Validation Suggestion Why is a Study Needed? To meet CLIA Requirement 493.1253(b)(2): Prior to reporting patient test results, the laboratory is responsible for establishing the performance specifications for each modified FDA-cleared or approved test system, each test system not subject to FDA clearance or approval, and each test system for which the manufacturer does not provide performance specifications. The establishment of method performance specifications should provide evidence that the accuracy, precision, analytical sensitivity, and analytical specificity of the procedure is adequate to meet the clients' needs as determined by the laboratory director and clinical consultant. 15
The Purpose of the Study To evaluate whether an AST device meets the FDA required category agreement FDA requirements for AST devices 90% essential agreement 90% category agreement This requirement has already been met for all dilutions reported by an AST device http://www.fda.gov/medicaldevices/deviceregulationa ndguidance/guidancedocuments/ucm080564.htm 16 What is Essential Agreement? MIC results from a reference method and a device method are within a single doubling dilution Example: Klebsiella pneumoniae meropenem MIC (µg/ml) Reference broth microdilution 1 Device 1 0.5 Device 2 4 Device 3 2 No essential agreement 17 What is Category Agreement? Category agreement for Susceptible, Intermediate, and Resistant Example: Zone diameter vs. MIC 18
Category Agreement Antimicrobial with S, I, R breakpoints of 1/2/4 Test Method MIC (µg/ml) Reference MIC (µg/ml) 0.5 1 2 4 0.5 3 1 15 9 2 2 4 1 Example: MIC vs. MIC 19 If essential agreement has already been established, then why does category agreement need to be evaluated? Answer: It is possible to have essential agreement without category agreement Reference MICs Category Agreement Without Essential Agreement Hypothetical experiment: 30 isolates of Enterobacteriaceae were tested for meropenem susceptibility by reference MIC method and an AST device. New S/I/R breakpoints: 1/2/4 Test MICs 0.125 0.125 0.25 0.5 1 2 4 0.25 6 10 0.5 4 8 1 2 2 4 Essential agreement = 66% Category agreement = 100% 21
Essential Agreement Without Category Agreement Hypothetical example: 30 isolates of Enterobacteriaceae were tested for meropenem susceptibility by reference MIC method and an AST device. Test MICs New breakpoints: 1/2/4 0.5 1 2 4 Essential agreement = 100% Category agreement = 63% Reference MICs 0.5 3 1 15 9 2 2 4 1 22 Interpretation for Reference Method Category Errors Interpretation for Test Method Type of Error Susceptible Intermediate Minor error Susceptible Resistant Major error Intermediate Susceptible Minor error Intermediate Resistant Minor error Resistant Intermediate Minor error Resistant Susceptible Very major error 23 Acceptable Results No standards for a study like this Must establish before starting the study Suggestion: Category agreement 90% No more than 1 major error No very major errors 24
Calculation Category agreement = # of isolates with the same interpretive result as the reference method # of isolates tested X100 25 A Study to Implement the Revised CLSI Cephalosporin and Carbapenem Breakpoints Availability of Concentrations of Antimicrobial Agents Needed to Implement Revised CLSI Breakpoints on Commercial AST Panels Antimicrobial Agent MicroScan Phoenix Sensititre Vitek2 Etest Aztreonam x 1 x x x x Cefazolin (2011 or later) 2 (2011 or later) x (Fall 2011) - (2011 Cefotaxime (2011 or later) or later) x x x (2011 or Ceftriaxone x later) x x x Ceftazidime x x x x x Doripenem x (2011 or later) x x x (2011 or (2012 or Ertapenem (2011 or later) later) x later) x Imipenem x x x x x Meropenem x x x x x 1 x = Available on one or more routine panels/cards as of 1/5/11 2 Projected availability
Possible Isolate Selection Revised Isolates Breakpoints to Test Carbapenems 5 carbapenemase (+) 5 Modified Hodge Test (-)( ) and elevated carbapenem MICs 20 other Enterobacteriaceae Cephalosporins 5 ESBL (+) and Aztreonam 5 ESBL (-)( ) and ESBL screen positive 20 other Enterobacteriaceae (preferably with MICs within S range using old BPs) Labs may consider testing more isolates Reference Method for the Validation Study CLSI reference broth microdilution frozen CLSI agar dilution CLSI disk diffusion The most accessible method 29 Procedures Test all isolate by the test AST method & the reference AST method Test isolates by both methods on the same day Prepare each inoculum from the same subculture plate If possible, use the same inoculum suspension (e.g. 0.5 McFarland equivalent) Use methods that you will use routinely Don t forget about quality control 30
Example: Data Collection Isolate ID Test AST Microscan Panel MIC (µg/ml) Reference AST Disk Diffusion Zone Diameter (mm) Isolate ID Test AST Microscan Panel MIC (µg/ml) Reference AST Disk Diffusion Zone Diameter (mm) BVS1 0.5 24 BVS2 0.5 23 BVS3 1 19 BVS4 0.5 25 BVS5 2 20 BVS6 1 24 BVS7 4 18 BVS8 0.5 23 BVS9 4 17 BVS10 1 21 BVS11 0.5 24 BVS12 0.5 23 BVS13 0.5 25 BVS14 1 23 BVS15 1 25 BVS16 1 23 BVS17 0.5 24 BVS18 2 21 BVS19 0.5 24 BVS20 0.5 25 BVS21 1 24 BVS22 2 19 BVS23 0.5 24 BVS24 2 24 BVS25 1 24 BVS26 8 18 BVS27 1 24 BVS28 0.5 25 BVS29 2 21 BVS30 0.25 24 31 Example: Scattergram 16 8 1 MIC (ug/ml) 4 1 1 2 1 1 2 1 1 1 1 2 4 1 0.5 3 5 4 0.25 1 16 17 18 19 20 21 22 23 24 25 26 27 Zone diameter (mm) 32 Example: Identify Errors 16 8 1 4 1 1 2 1 1 2 1 1 1 1 2 4 1 0.5 3 5 4 0.25 1 16 17 18 19 20 21 22 23 24 25 26 27 Minor errors = 2, Major errors = 1, Very major errors = 0 Category agreement = 27 30 = 0.90 or 90% 33
Documentation The study is needed to fulfill CLIA regulation If it is not documented, you didn t do it Documentation includes: Study procedure Results: data table and/or scattergram Calculations Conclusion statement Validation results must be approved by laboratory director 34 What if the results are unacceptable? 1-2 isolates with discrepant results could indicated random error >2 isolates with discrepant results may indicate systematic error caused by : Technical problems with the methods used in the study Discordance between test method and reference method 35 Unacceptable Results cont. Assessing error type Random errors unlikely to occur upon repeat testing Systemic technical errors will not repeat if the technical problem(s) is addressed Errors caused by a discordance between methods will repeat 36
Unacceptable Results cont. Review the procedures looking for technical errors Repeat testing can be done, but avoid repeating only isolates with discrepant results Decide on a systematic approach to repeat testing (e.g., repeat 1 to 2 isolates with nondiscrepant results for every one isolate with discrepant results) 37 Unacceptable Results cont. Calculate category agreement and error rates A) Using original data B) Using repeat data If the repeat study results meet the acceptability requirement, then Document the original results Document the repeat results Describe likely source of error 38 Unacceptable Results cont. Discordance between methods cannot be fixed by troubleshooting Consult manufacturer 39
The AST Manufacturer Cannot: Provide software or alter devices to provide non-fda breakpoints Can: Provide general instructions for how to customize the device expert system to apply the revised breakpoints Alternatively, labs can implement the revised breakpoint in their laboratory information 40 Applying New Breakpoints to Etest Applying revised breakpoints to Etest results is a modification of a FDA-cleared device A validation study is needed 41 Qs & As Q: What if some of the carbapenemaseproducing isolates test carbapenemsusceptible or the ESBL-producing isolates test cephalosporin-susceptible A: The purpose of this study is to determine if the category agreement between the test and reference AST methods are within acceptable limits regardless of underlying resistance mechanisms in the challenge isolates. 42
Qs & As Q: If we aren t scoring results by resistance mechanism, why do I need to test isolates with different resistance mechanisms? A: Using isolates with different resistance mechanisms allows for a more robust challenge of your AST method. 43 Qs & As Q: Why is it important to implement the CLSI revised carbapenem breakpoints? A: The carbapenem breakpoints are: Based upon new PK/PD data Can be used to guide treatment regardless of underlying resistance mechanism(s) More likely to identify carbapenemaseproducing isolates as nonsusceptible (treatment & infection control implications) Avoids using a subjective and sometimes inaccurate phenotypic test to guide treatment decisions 44 Qs & As Q: Why is it important to implement the CLSI revised cephalosporin breakpoints? A: The cephalosporin breakpoints are: Based upon new PK/PD data Can be used to guide treatment regardless of underlying resistance mechanism(s) Avoids using a limited phenotypic test to guide treatment decisions False-positives associated with multiple cephalosporinases Does not detect all cephalosporinases 45
Sources for Isolates CDC is sending isolates to: College of American Pathologists ATCC Isolates that you may have on hand Clinical isolates ATCC QC isolates Isolates from previous proficiency testing challenges 46 What are your plans? Our lab plans to implement the CLSI revised cephalosporin breakpoints this year Yes No Our lab plans to implement the CLSI revised carbapenem breakpoints this year Yes No 47 Acknowledgements Janet Hindler, Jim Jorgensen, Barb Zimmer Kamile Rasheed, David Lonsway, Brandi Limbago IDSA Antimicrobial Resistance Working Group CLSI The Joint Commission CAP FDA 48
Thank You jpatel1@cdc.gov