ANTIMICROBIAL SUSCEPTIBILITY TESTING: ADVANCED Romney Humphries, Ph.D., DABMM Section Chief, Clinical Microbiology University of California Los Angeles Lost Angeles, CA Susan E. Sharp, Ph.D., DABMM, FAAM Director, Airport Way Regional Laboratory Director, Regional Microbiology Kaiser Permanente Portland, OR
OBJECTIVES 1. Examine approached to verification studies for antimicrobial susceptibility test systems. 2. Evaluate approaches to cascade reporting. 3. Discuss the development of an antibiogram. 4. Review the principles of pharmacokinetics and pharmacodynamics (PK/PD) in relation to how these apply to MIC interpretive criteria.
STRATEGIES FOR VERIFYING ANTIMICROBIAL SUSCEPTIBILITY TESTS AND TEST SYSTEMS Romney Humphries UCLA Clinical Microbiology Los Angeles CA
WHAT WE WILL COVER: List antimicrobial susceptibility tests (AST) and test systems that are available for use, including their FDA status. Describe strategies for verifying antimicrobial susceptibility tests and test systems. Discuss criteria used to determine if verification is successful and options for documenting the verification process.
AST METHODS/SYSTEMS Disk diffusion (Kirby Bauer) CLSI reference method Automated zone reader (biomic) Broth microdilution MIC CLSI reference method Commercial systems MicroScan (Siemens) Sensititre (Trek)
AST METHODS/SYSTEMS (CON T) Etest Automated / semi-automated systems MicroScan (Siemens) Phoenix (Becton Dickinson) Sensititre (Trek) Vitek (biomerieux)
WHY DO AST DEVICES REQUIRE FDA CLEARANCE? To prove they are safe and effective and pose minimum risks to patients Identified Risk Administration of an inappropriate antimicrobial agent to a patient
WHAT DOES IT MEAN FOR A COMMERCIAL AST DEVICE TO BE FDA CLEARED? Manufacturer submitted 510k to FDA Manufacturer demonstrated that results with their AST device are comparable to those generated from a CLSI REFERENCE method. FDA specifies the data that must be submitted and defines criteria for acceptability.
WHAT DATA ARE REQUIRED BY FDA? No. sites (including 1 in-house) 3 Organisms Fresh or stock (clinical) 100/site Challenge * Reproducibility Interpretive criteria / standards Stability (3 lots) 75/one site 25/site or 10x3x3/site FDA Real time QC CLSI strains 20 results/site Manufacturer s strains Optional On-scale At least 1 Inoculum density check QC/reproducibility/fresh CLSI reference method * MICs near R breakpoint MIC http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocu ments/ucm080564.htm
WHAT CRITERIA ARE USED TO DETERMINE IF AST DEVICE RESULTS ARE ACCEPTABLE? Criteria Defined as Acceptable limits Essential Agreement (EA) MIC within +/- 1 doubling dilution of the REF MIC >89.9 % Category Agreement (CA) S, I, and R results agree >89.9 %
WHAT CRITERIA ARE USED TO DETERMINE IF AST DEVICE RESULTS ARE ACCEPTABLE? (CON T) Error REF Results AST device Acceptable Error Rate Very major R 1 S Based on # of R orgs tested Major S 2 R 3 % Minor S I R I I I S R Not specified 1 only isolates R by REF included in calculating error rate 2 only isolates S by REF included in calculating error rate
CALCULATING EA / CA Essential agreement (EA) = # within +/- 1 two-fold dilution of REF MIC Total # isolates tested X 100 Category agreement (CA) = # with same S, I, or R result as REF MIC Total # isolates tested X 100
CALCULATING % ERRORS Very major error (VME) = No. with VME (false R) Total # R isolates tested Major error (ME) = No. with ME (false S) Total # S isolates tested Minor error (me) = No. with me Total # isolates tested X 100 X 100 X 100 Careful! Data in literature sometimes uses total # tested
EXAMPLE: CEFAZOLIN Breakpoints (µg/ml) S I R 8 16 32 Isolate REF TEST EA CA Error? 1 8 S 8 S yes yes no 2 8 S 2 S no yes no 3 16 I 8 S yes no minor 4 8 S 32 R no no major 5 32 R 8 S no no very major
CALCULATING EA / CA EXAMPLE: CEFAZOLIN Organism N EA CA # % # % E. coli 100 91 91 94 94 C. freundii 94 89 94.6 84 89.4 1 1unacceptable re: FDA criteria
CALCULATING % ERRORS EXAMPLE: CEFAZOLIN Organism N REF Result Very major major minor # S # R # % # % # % E. coli 100 60 30 1 3.3 2 3.3 3 3 C. freundii 94 49 45 0 0 1 2.0 9 9.5
PACKAGE INSERT OR LABELING SECTIONS FOR AST DEVICE Intended Use Reagents Reporting of results Performance characteristics QC Limitations Other
RUO AND IUO AST DEVICE Research use only: For Research Use Only. Not for use in diagnostic procedures must be in labeling Devices are in research phase of development Investigational use only: For Investigational Use Only. The performance characteristics of this product have not been established must be in labeling Products being tested or evaluated prior to regular marketing Reporting: Report should indicate that test is not FDA-cleared (RUO)
SCENARIO: You acquire a NEW automated instrument for AST What should you do to make certain the AST system will produce accurate and reproducible results?
Verification initial assessment of NEW AST system CLIA 493.1253
CLIA 493.1253 ESTABLISHMENT AND VERIFICATION OF PERFORMANCE SPECIFICATIONS Each lab that introduces an unmodified, FDA-cleared or approved test system must do the following before reporting patient test results: Demonstrate that it can obtain performance specifications comparable to those established by the manufacturer for the following performance characteristics: Accuracy Precision (reproducibility) Reportable range of test results for the test system. Verify that the manufacturer's reference intervals (normal values) are appropriate for the laboratory's patient population.
USER MUST DEMONSTRATE THESE PERFORMANCE SPECIFICATIONS WHEN USING AN FDA-CLEARED TEST BEFORE USE FOR PATIENT TESTING Accuracy Precision (reproducibility) CLIA 493.1253
CLIA 493.1253 ESTABLISHMENT AND VERIFICATION OF PERFORMANCE SPECIFICATIONS Each lab that modifies an FDA-cleared or approved test system, or introduces a test system not subject to FDA clearance or approval (including methods developed inhouse), or uses a test system in which performance specifications are not provided by the manufacturer must, before reporting patient test results: Establish for each test system the performance specifications for the following performance characteristics, as applicable: Accuracy Precision Analytical sensitivity Analytical specificity to include interfering substances Reportable range of test results for the test system. Reference intervals (normal values). Any other performance characteristic required for test performance.
USER MUST DEMONSTRATE THESE PERFORMANCE SPECIFICATIONS WHEN USING A TEST THAT IS NOT FDA-CLEARED OR USING A MODIFICATION OF AN FDA - CLEARED TEST BEFORE USE FOR PATIENT TESTING Accuracy Precision (reproducibility).plus Analytical sensitivity (AST, generally ability to detect R ) Analytical specificity (AST, generally ability not to call false R ) Reportable range of patient test results Reference range(s) Any characteristic required for test performance or interpretation of results..,..some do not apply to AST! CLIA 493.1253
USAGE OF TERMS Initial in-house assessment of NEW test CLIA Verification Validation Ongoing evaluation of IN-USE test CAP Verification Ongoing evaluation JCAHO Validation Verification Terms used in this talk Verification Ongoing evaluation
Verification (initial assessment) Accuracy Reproducibility(P recision) Compare NEW w/ REF method Run to run - ATCC strains Within run - 5 bugs x 3 x 3d Manufacturer s information In-house studies Literature Anecdotal information Test minimum of 100 isolates FDA-cleared AST Prior to patient testing
QC e.g., CLSI ATCC Competency Ongoing evaluation PT Equipment PM Correlation with clinical findings Other FDA-cleared AST...During patient testing
WHAT DO WE HAVE TO CONSIDER WHEN ESTABLISHING ACCURACY FOR USE OF A NEW AST DEVICE IN OUR LABORATORY? What method should we use as REF? How many bugs? What kind of bugs? resistance profiles? What is acceptable? What if results are not acceptable? How should we document?
WHAT METHODS SHOULD WE USE AS REF? CLSI broth microdilution or agar dilution reference method Commercial FDA-cleared broth microdilution method (historically acceptable as REF) OLD method (that was previously verified) Accuracy Studies
HOW MANY ORGANISMS SHOULD WE TEST? Minimum 30; 100 preferred, more if possible Selection criteria Represent clinical isolates tested Variety of susceptibility profiles Some around breakpoints Use organism mix on next few slides as guide Note: some bugs fit multiple R criteria Accuracy Studies
WHERE DO WE OBTAIN ISOLATES? Routine QC strains (CLSI) Clinical isolates isolated in-house or from other laboratories Fresh Stock Resistant and susceptible isolates for each antimicrobial agent Old PT samples Organisms provided by manufacturer Accuracy Studies
EXAMPLE: ORGANISM MIX (N=75) STAPH/ENTEROCOCCUS PANEL (CON T) No. Species R characteristics R (pos) S (neg) 5 E. faecalis vancomycin 2 3 4 E. faecalis HLAR (Gm) 2 2 4 E. faecalis HLAR (Str) 2 2 6 E. faecium vancomycin 3 3 4 E. faecium HLAR (Gm) 2 2 4 E. faecium HLAR (Str) 2 2 4 Motile enterococci low level vancomycin 2 2
EXAMPLE: ORGANISM MIX (N=75) STAPH/ENTEROCOCCUS PANEL (CON T) No. Species R characteristics R (pos) S (neg) 20 S. aureus meca 10 10 2 S. aureus β-lactamase - 2 6 S. aureus inducible clindamycin 3 3 10 CoNS meca 5 5 6 Other drug dependent* *e.g., if rifampin reported on S. aureus, test at least 1 R and 1 S also Daptomycin-NS MRSA and E. faecalis Linezolid-R MRSA, CoNS, enterococcus
EXAMPLE: ORGANISM MIX (N=126) ENTEROBACTERIACEAE PANEL No. Species R characteristics R (pos) S (neg) 8 E. coli ampicillin 4 4 8 E. coli ESBL 4 4 4 E. coli ESBL scrn + confirm- 8 E. coli gentamicin 4 4 8 E. coli fluoroquinolone 4 4 8 E. coli nitrofurantoin 4 4 4
EXAMPLE: ORGANISM MIX (N=126) ENTEROBACTERIACEAE PANEL No. Species R characteristics R (pos) S (neg) 8 C. freundii broad spec β-lac 4 4 8 E. aerogenes broad spec β-lac 4 4 8 E. cloacae broad spec β-lac 4 4 8 E. cloacae gentamicin 4 4 8 M. morganii broad spec β-lac 4 4 8 Providencia spp. broad spec β-lac 4 4 8 S. marcescens broad spec β-lac 4 4 Broad spec β-lacs = one or more from cefotaxime, ceftriaxone, ceftazidime, cefepime, piperacillin-tazo, carbapenem
EXAMPLE: ORGANISM MIX (N=126) ENTEROBACTERIACEAE PANEL (CON T) No. Species R characteristics R (pos) S (neg) 1 K. pneumoniae KPC 1 8 Klebsiella spp. ESBL 4 4 8 P. mirabilis fluoroquinolone 4 4 1 P. mirabilis ESBL 1 4 Salmonella spp fluoroquinolone 1 3 4 Shigella any also additional CRE Low level FQ R Salmonella (if can be tested on panel) Tigecycline-R Enterobacteriaceae
EXAMPLE: ORGANISM MIX (N=83) NON-FERMENTER PANEL No. Species R characteristic R (pos) S (neg) 10 P. aeruginosa R- Gm, Tob (S- Amk) 5 5 5 P. aeruginosa R- Gm, Tob, Amk 5 10 P. aeruginosa imipenem 5 5 6 P. aeruginosa piperacillin-tazobactam 3 3 5 P. aeruginosa fluoroquinolone 3 2 10 P. aeruginosa (mucoid) 10 Pseudomonas sp. (not aeruginosa) if CF patients are seen at facility any
EXAMPLE: ORGANISM MIX (N=78) NON-FERMENTER PANEL (CON T) No. Species R characteristic R (pos) S (neg) 5 Steno. maltophilia trimethoprim-sulfa 1 4 10 Acin. baumannii imipenem = R 5 5 4 Achrom. xylosoxidans any 4 Burkholderia sp. any 4 Aeromonas sp. any
VALIDATION DATA - ACCURACY
WHAT DO WE HAVE TO CONSIDER WHEN ESTABLISHING REPRODUCIBILIT Y FOR USE OF A NEW AST DEVICE IN OUR LABORATORY? How many bugs? What kind of bugs? resistance profiles? Routine QC strains? How many replicates? test days? What is acceptable?? What if results are not acceptable?? How should we document?? Reproducibility Studies
PRACTICAL STRATEGY FOR REPRODUCIBILITY STUDIES Routine CLSI ATCC QC strains Data from routine testing (run to run reproducibility) Select 5 clinical isolates (fresh or stock) that meet as many of the following as possible On-scale endpoints Some resistance Mix of high and low MICs for a given drug Good growth characteristics Test in triplicate on 3 days (within run and run to run reproducibility) Acceptable = 95% within +/- 1 dilution Reproducibility Studies
SCENARIO #1: YOUR NEW AST DEVICE* GENERATES THE FOLLOWING FOR E. COLI AND GENTAMICIN. WHAT WILL YOU DO? N REF Result Very major major minor # S #I #R EA CA # % # % # % 30 23 2 5 83 86. 6 3 60 0 0 1 3.3 *OLD system = commercial broth microdilution
Repeat NEW and REF No VME Stop Test more R bugs? 1 VME persists Send to reference lab (CLSI MIC method) Agree with NEW 1 VME persists Stop Consider: Could isolates be clonal? Any unique growth characteristics? NOT validated
SCENARIO #2: YOUR NEW AST DEVICE* GENERATES THE FOLLOWING FOR P. AERUGINOSA AND PIPERACILLIN-TAZOBACTAM. WHAT WILL YOU DO? N REF Result Very major major minor # S #I #R EA CA # % # % # % 25 15-10 96 1 80 1 10 4 26.6 NA NA *OLD system = commercial broth microdilution 1 MIC for only 1 isolate >1 dilution from REF MIC
SCENARIO #3: YOUR NEW DEVICE* GENERATES THE FOLLOWING FOR S. AUREUS AND OXACILLIN. WILL YOU USE A CEFOXITIN BACKUP TEST? N REF Result Very major major minor # S #I #R EA CA # % # % # % 30 12-18 93.3 96.6 0 0 1 8.3 NA NA *OLD system = cefoxitin disk diffusion
SCENARIO #4: YOU WANT TO TEST PASTEURELLA SPP. WITH YOUR COMMERCIAL BROTH MICRODILUTION SYSTEM (PANELS WITH 2.5-5% LYSED HORSE BLOOD) THAT IS NOT FDA-CLEARED FOR TESTING THIS ORGANISM. WHAT SHOULD YOU DO? Most widely accepted practice: Use validation data for panel with cleared organisms Qualify results
SPECIMEN: BLOOD DIAGNOSIS: LEUKEMIA PASTEURELLA MULTOCIDA MIC (µg/ml) amoxicillin ceftriaxone doxycycline levofloxacin penicillin 0.5 S 0.12 S 0.5 S 0.06 S 0.5 S Susceptibility testing performed per Dr. Jones request; Infectious Diseases consult suggested; testing performed using a system FDA cleared for other similar organisms but NOT for P. multocida
SCENARIO #5: CLIA SURVEYOR ASKED LAB FOR SENSITIVITY/ SPECIFICITY DATA FOR D ZONE TEST No performance specifications from the manufacturer Lab questioned request and contacted AST experts Experts agreed not practical / necessary One strategy for verifying D zone test: 1. Test CLSI QA strains (S. aureus ATCC BAA-976, BAA- 977) 2. Test 5 staphylococci with known inducible clindamycin resistance and 5 without 3. Place disks at various distances to ensure placement doesn t cause false S or false R - use ATCC strains in 1)
WHAT ERROR RATE IS ACCEPTABLE? (N 30) Error Type Acceptable Error Rate Comment Very major 1 0 Major < 5 % Major + Minor 2 10 % Combined major and minor Essential agreement 90 % Categorical agreement 90 % Reference system not used as comparator Cumitech 31A. ASM Press, 2009.
WHAT ERROR RATE IS ACCEPTABLE? (N 100) Error Type Acceptable Error Rate Comment Very major 1 3 % Minimum 35 R isolates Major 3 % Major + Minor 2 7 % Essential agreement 90 % Categorical agreement 90 % Combined major and minor Arbitrate discrepancies with a Reference system 50% should exhibit some type of R If significant numbers of isolates near breakpoint, CA my be <90% Cumitech 31A. ASM Press, 2009.
But, do not interpret recommendations rigidly; consider degree of difficulty involved in detecting resistance in some organisms.
EXAMPLES OF ERRORS AND STRATEGIES FOR DEALING WITH THEM.. Error 5/25 (20%) VME ceftazidime / P. aeruginosa 5/25 (20%) ME nitrofurantoin / C. freundii 16/20 (80%) EA cefazolin / K. pneumoniae (all errors in S range erring on higher MIC with NEW test) Optional Strategy Unacceptable use alternate method Use alternate method for OP urine C. freundii R isolates accept 4/16 (25%) ME pip-tazo / P. aeruginosa Accept...only S and R (no I ) breakpoints; educate ID/pharm about MICs close to breakpoints 8/10 (80%) CA for tobramycin / P. aeruginosa Test additional isolates
EXAMPLES OF ERRORS AND STRATEGIES FOR DEALING WITH THEM.. Error Optional Strategy 8/16 (50%) ME clindamycin / CoNS Do not report offer test as MD request only..use alternate method 3/10 (30%) ME for ESBL confirmatory test for K. pneumoniae 2/14 (14.3 %) VME for HLAR gentamicin and E. faecalis 2/16 (12.5%) VME for cefazolin and E. cloacae Use alternate method; check if isolates might be clonal; check with other users Use alternate method (sterile sites; not generally needed from other sites); check other Enterococcus spp. carefully Override all cefazolin-s or I results to R
EXAMPLE: ENTEROBACTERIACEAE PANEL Drug Amikacin Ampicillin Cefazolin Ceftriaxone Ciprofloxacin Gentamicin Imipenem Nitrofurantoin Piperacillin-tazo Tobramycin Trimethoprim-sulfa Post Validation Comments Failed verification for Serratia spp. Use alternate test when needed Edit E. cloacae to R OK OK OK OK Only 1 R isolate in validation; verify all R patient isolates by alternate method; test more R Failed validation for C. freundii (false R); use alternate test for R urine isolates OK OK OK
QUESTIONS RE: VERIFICATION What if??? Company rep does validation We get a loaner instrument for 2 weeks We relocate instrument Verify performance by. Showing this correlates with performance by lab staff. Testing QC strains (daily) and 25 or so PT samples and/or previously tested patient specimens Following manufacturer s package insert re: critical requirements (setup, environment, etc.) Use common sense and document, document, document!!!
ONGOING EVALUATION Documentation that a test which has been validated is repeatedly giving the expected results Most commonly involves following CLSI QC procedures Specified by CLIA, CAP, JCAHO Other components.
QC e.g., CLSI ATCC Competency Ongoing evaluation PT Equipment PM Correlation with clinical findings Other FDA-cleared AST During patient testing
WHERE DO WE FIND INFORMATION ON LEGAL REQUIREMENTS FOR VERIFICATION AND ONGOING EVALUATION? CLIA sections 493.1253 CAP - MIC.21040, GEN.42020 GEN.42160 Joint Commission Quality control sections QC.1.20 QC.1.150
WHAT DOCUMENTS/RECORDS SHOULD WE MAINTAIN FOR THE VERIFICATION? Validation Parameter Protocol Accuracy Documents Written protocol for process used to verify NEW AST Data NEW AST vs. REFERENCE, summary, conclusions Reproducibility Data - QC with CLSI ATCC and additional QC strains, summary, conclusions Putting Test IN USE All those documenting how test will be introduced and protocols for ongoing evaluation (e.g., QC, competency, PT, equipment PM, correlation with clinical findings, etc.) *Note: good laboratory practice would dictate that records should be maintained as long as test is in use
ENSURING QUALITY OF AST 10% 10% Validation 10% 10% 10% 10% 10% 10% Ongoing QC Proficiency tests Instrument maintenance Competency Relevant testing strategies Verify unusual results Sup review of result Clinical correlation 10% 10% Keep current with changes
Also CAP s BIT (breakpoint implementation tool) Verification of ASTs
.but think about impact to the patient.. we must not rationalize that an error is OK nor become obsessed about errors that are unavoidable Thank you!