Introduction to Software for Analysis of Susceptibility Experiments

Transcription

1 1 / 31 Introduction to Software for Analysis of Susceptibility Experiments With Open Questions and Discussion Glen DePalma gdepalma@purdue.edu Bruce A. Craig bacraig@purdue.edu Clinical and Laboratory Standards Institute Meeting 01/13/2013

2 2 / 31 Goal To develop an easy to use software program utilizing model-based techniques to estimate the diffusion test (DIA) breakpoints for pathogen susceptibility tests. Created an R package called dbets diffusion Breakpoint Estimation Testing Software Working on an experimental GUI interface. Continue to welcome suggestions for improvement and new features.

3 Example of Model 3 / 31

4 4 / 31 Software Features Data Visualization Model - Based Approaches Error Rate Bounded Method The following is an example demonstrating the various tools.

5 Simulated Model-Based Data Example 5 / 31

6 Alternate Visualization We prefer this plot since the MIC range is to while the DIA is only postive. Various S shaped functions (e.g., four-parameter logistic function) can be considered for the model-based approaches. 6 / 31

7 7 / 31 Start with Error Rate Bounded (ERB) Features We implemented an algorithm to find the optimal DIA breakpoints based on two user-specified sets of discrepancy percentage limits. Set one for MIC values within one dilution of the intermediate range Set two for MIC values outside one dilution of the intermediate range The default percentages match current standards: Within One: Very Major = 10%, Major = 10%, minor = 40% Outside One: Very Major = 2%, Major = 2%, minor = 5% Limits are converted into penalty weights and the function searches for DIA breakpoints which minimize the overall sum of penalties. index = w VM1 (#VM1) + w M1 (#M1) + w m1 (#m1) + w VM2 (#VM2) + w M2 (#M2) + w m2 (#m2) where w represents the weight associated with a specific discrepancy and #* represents the count for that discrepancy. A grid search is performed for all possible DIA breakpoints that minimizes the index within a specified range.

8 8 / 31 Optimal DIA Breakpoints Find the optimal DIA breakpoints using the Error Rate Bounded method for a given set of weights (default weights used here). Optimal DIA Breakpoints for ERB: Number of Isolates: 500 Number Observed Outside One of Intermediate Range: 246 Number Observed Within One of Intermediate Range: 254 Index Score = Count (%) Range of Observed Agree Very Major Major Minor Within One 189 (74.41) 0 (0) 1 (0.39) 64 (25.2) Outside One 244 (99.19) 0 (0) 0 (0) 2 (0.81)

9 Resulting plot 9 / 31

10 10 / 31 ERB Method for Given DIA Breakpoints Produce discrepancy information for a given set of DIA breakpoints. Classification for DIA Breakpoints for ERB: Number of Isolates: 500 Number Observed Outside One of Intermediate Range: 246 Number Observed Within One of Intermediate Range: 254 Index Score = Count (%) Range of Observed Agree Very Major Major Minor Within One 184 (72.44) 0 (0) 0 (0) 70 (27.56) Outside One 244 (99.19) 0 (0) 0 (0) 2 (0.81)

11 Resulting plot 11 / 31

12 12 / 31 Assess Uncertainty in ERB Breakpoints Assesses the variability in the DIA breakpoints for the ERB method by using the bootstrap method DIA Breakpoints by Confidence DIABrkptL DIABrkptU Percent Cumulative_Percent

13 13 / 31 Model-Based Approaches Model-based approaches account for the variability in the data through statistical models. They model the probability of discrepancies between the two tests rather than minimizing the observed discrepancies. Aim is to select the DIA breakpoints that makes the DIA test at least as powerful as the MIC test (minimize when it s worse then MIC). Current L = min (0, pdia(g(u)) pmic(u))2 w(u) du w(u) is a weight function This results in increased precision. Note: our approach can be changed to a different aim such as minimizing the probability of discrepancies between the two tests. May be more aligned with the ERB approach. Open question number 3 to discuss later.

14 Example of Model 14 / 31

15 Probability of Correct Classification 15 / 31

16 Logistic Model Plot 16 / 31

17 17 / 31 Logistic Model Results Fits a Logistic curve to the data and estimates DIA breakpoints DIA Breakpoints by Probability DIABrkptL DIABrkptU Percent Cumulative_Percent

18 Spline Model Plot 18 / 31

19 19 / 31 Spline Model Results Fits a Spline-model to the data and estimates DIA breakpoints DIA Breakpoints by Probability DIABrkptL DIABrkptU Percent Cumulative_Percent

20 20 / 31 Compare Two Model Fits Displays the Logistic and Spline fits to the data.

21 21 / 31 Compare Probabilities of Correct DIA Classification Compares the probability of correct DIA classification given sets of DIA breakpoints. DIA Breakpoints Set 1: 34, 39 Set 2: 31, 40 Probability of DIA Classification Set 1 Agree Set 2 Agree Weighted by MIC Density Probability of DIA Classification Set 1 Agree Set 2 Agree Used spline model for calculations.

22 Resulting Plot ERB breakpoints poorly classifies in the intermediate range but is slightly better in the susceptible and resistant regions 22 / 31

23 23 / 31 Open Questions True MIC breakpoints vs Test MIC breakpoints (model-based) Lab-to-Lab variability Best index for estimating DIA breakpoints (model-based) Minimize probability of discrepancies, weight by type (?) Minimize difference in classification curves Weight by underlying distribution

24 24 / 31 True vs Test MIC Breakpoints (model-based) With model-based approach we distinguish between obs MIC and true MIC. Due to rounding of MIC test, do we need separate breakpoints? Currently use given MIC breakpoints for both. Is this appropriate? Top graph only uses test breakpoints. The bottom graph takes into account what the true breakpoints might be. The resulting classification curve is now symmetric, similar to the DIA classification curve.

25 25 / 31 Inter-Lab Variability Quality Control Study - MIC test: 10 Labs, 50 reps/lab E. coli ATCC MIC LAB All Labs Mean StDev S. Aureus ATCC MIC LAB All Labs Mean StDev Inter-Laboratory Variation accounts for around 50% of the total variance.

26 Inter-Lab Variability Effect (simulation) 26 / 31

27 27 / 31 Best Approach ERB: 34, 41 ERB: 33, 41 ERB: 33, 40 Method Test Breakpoints True Breakpoints Minimize P(Discrepancies) 30, 39 32, 41 DIA at Least as Powerful 31, 40 28, 40 Similar Tests 34, 40 28, 41 Estimated DIA Breakpoints

28 28 / 31 Additional Examples -1 (a) Fits (b) ERB

29 29 / 31 Additional Examples - 1 (c) Logistic (d) Nonparametric

30 30 / 31 Additional Examples - 2 (e) Fits (f) ERB

31 31 / 31 Additional Examples - 2 (g) Logistic (h) Nonparametric