In house Validation/Evaluation of Microbiology Methods: When is it Required?

Transcription

1 In house Validation/Evaluation of Microbiology Methods: When is it Required?

2 In-house Evaluation: When Required? Common Situations 1. Introducing a new method to test for an organism not previously handled. 2. Replacing an existing method with a new method 3. Modifying technical content of an existing method (if performance could be affected) 4. Extending the scope of an existing methods application to new/different sample matrices. 5. Changes in legislation 6. Updates to source method

3 In-house Method Evaluation: Preparation and Planning

4 Pre-Evaluation Preparation: Literature Review Obtain background info on method: Review published/scientific literature Obtain validation data from manufacturer (if applicable) Check status of method (ISO, BS, AOAC, AFNOR) Check scope of validations, identify potential limitations in relation to lab requirements Helps gauge potential performance of method Can identify strengths/weakness of method or limitations and can influence type of evaluation and criteria used

5 Pre Evaluation Preparation: Practical Issues Obtain new media/equipment/cultures etc Familiarise/train staff in conduct of method Calibrate equipment, if applicable Review published work / check method status

6 Planning Evaluation Work Define purpose (why is the evaluation required) Define scope (sample type(s) and range of microorganism(s)) Define your criteria to appraise results/accept method as suitable for use Identify practical approach(s) to generate performance data Draw up a programme/timetable to do practical work Regularly review results and change design if necessary With good planning much of practical work could be accommodated into normal lab work

7 Criteria to Appraise Results/Method Performance

8 Criteria To Assess Methods Generally assessment will result in a decision on use, or continued use, of a method for a defined scope Should be set before evaluation, but frequently not considered, or defined after work done (to fit data obtained!) For decision to be fair, objective and justifiable important to define criteria against which method will be judged before evaluation assessment begins

9 Criteria to Assess Methods S M A R T specific measureable achievable realistic timed

10 Criteria to Assess Methods Fitness for purpose: should reflect purpose for which method will be used. Testable: at end of assessment it should be clear whether criteria have/have not been met (or if there is inadequate data to judge) generally quantifiable criteria (eg accuracy, precision, limit of detection)

11 Criteria to Assess Method Subjective: often related to administrative considerations associated with conduct of method (eg difficulty/ease) Practical: (e.g. cost, speed) Multiple: Some will/must relate to technical performance of method. However, may also be equally important subjective criteria. (Multiple criteria often used)

12 No 2 methods will give identical results. A method does not have to give an identical result to a reference/ano method to be useful depends upon what lab/user wants Evaluations are designed to look at what the differences are. And decide if differences are/are not significant to the requirements of the laboratory. ALWAYS REMEMBER FITNESS FOR REQUIRED PURPOSE

13 Technical Criteria To Appraise Methods Criteria to appraise results should NOT require NO Difference BUT Should state the MAX Difference acceptable to lab Generally criteria for technical performance of method will be acceptable differences between its results and the true result however that is defined. Multiple/Combination of criteria often set Enumeration and detection methods require different criteria

14 Technical Criteria To Appraise Methods Avoid criteria in terms of statistical significance Statistical significance does not imply practical significance. A difference may be statistically significant, but so small as to be of no practical interest/relevance at all. Conversely a practically substantial difference may not result in a statistical significance if the data or analysis is inadequate. Your criteria should not require no difference, but should state the maximum acceptable difference.

15 Technical Criteria To Appraise Method Avoid Criteria of zero or 100% Method A must give the same results as Method B Method A must never underestimate Method A must give comparable results as Method B Generally impossible to prove absence of a difference Such criterion should be expressed in terms of the acceptable extent of under/over estimation

16 Examples of Testable Criteria: Enumeration Methods Regression coefficient to be between 0.96 and 1.04 The difference in counts must not exceed 0.5log for 97% of samples tested. The difference in counts must not exceed +1log for 2% of samples tested. The difference in counts must not exceed -1log for 1% of samples tested. The bias of the new method should be within 0.3 log

17 Examples of Testable Criteria : Detection Methods False positive rate for new method to be less than 3% False negative rate of new method to not exceed that of reference method by more than 1% Method to be able to recovery 10-50cfu/25g on 96% of occasion

18 Distinguish between Average and Individual Differences Avoid looking only at average of results and average differences. Method 1 Method 2 Difference (2-1) Average No average bias, but individual differences big.

19 Distinguish between Average and Individual Differences Much better to look at individual differences Method 1 Method 2 Difference Average Individual differences small, but differences in same direction (bias of +0.65) is this significant? lab must decide based on required use/application

20 Approaches for Validation/Evaluation

21 Approaches to Validation/Evaluation Many common factors, but approaches will be unique to each situation. Examples: Compare against another method (if applicable) Appraise recovery from typical samples (natural/artificially contaminated) Duplicate Samples PT Samples/Reference Materials (if appropriate/available) Pure cultures (target, competitor, similar organisms (if applicable) Inter laboratory ring trials (if applicable/available) Confirmation of positive/negative results (if applicable) Experimental controls

22 Introducing a Method for An Organism New to Laboratory Possible Approaches No existing method in place for comparison. Different approach therefore required. Common Approaches: Analysis of naturally contaminated and spiked samples. Analysis of EQA samples. Use of pure cultures (if useful). Analysis of duplicate samples by ANO laboratory (if available/applicable to sample type).

23 Replacing an Existing Method Possible Approaches? Compare new method against outgoing method over a period of time to establish performance of new method. Where possible comparison should be against recognised/standard method for traceability. Non standard method not recommended unless previously been assessed against a standard protocol. Use natural/artificially contaminated samples. Inclusion of EQA samples helpful. Use of pure cultures (if useful/applicable).

24 Modifying An Existing Method Only required if modification likely to affect performance of method Possible Approaches: Compare the 2 versions of method. Use naturally/artificially contaminated samples. Include EQA samples if available. Use pure cultures (if applicable).

25 Extending Scope of an Existing Methods Application Possible Approaches Appraise impact of different food matrices/different background flora on performance. Run new sample matrices through method. Use natural and/or artificially contaminated samples. Consider different sample preparation protocols (if applicable).

26 Compare Against Another Method Benefits Allows comparative/relative performance of methods to be appraised. Useful when: replacing existing method with new method adding an alternative method for same organism/test modifying an existing method Tips! Comparator method to be a reference, standard or recognised method (unless method has been previously validated against reference method)

27 Choice of Samples How many samples to include? The MILLION $ QUESTION!! No Definitive Answer!! Enough to be analysed so the confidence interval lies inside or outside the defined criterion Depends upon: Criteria set by lab for acceptance of method Experimental design(s) chosen (paired or unpaired) Range of samples routinely tested Type of method under evaluation Reason for Testing

28 Choice of Samples Samples to represent type/range routinely handled + scope of method application Use naturally contaminated samples (if possible/ available) Use Spiked samples Consider replicate/duplicate samples/analyses Tips Include good number of positive samples (to give sufficient data/points to appraise results/method performance) Include a few negative samples (if available) Don t forget Environmental Samples/Swabs if within scope

29 Choice of Organisms Consider source/origin/identity (traceability) Strains/serotypes to use (if relevant) Use typical strains Consider atypical strains (if relevant) Sub-lethal injured/stressed cells (if relevant) Tips! Include cultures traceable to recognised culture collections Use food derived isolates, not clinical isolates (if possible) Consider wild strains isolated by lab (if available) Consider competitor organisms that mimic target organism (if relevant) Caution: Type Cultures not always typical of genus

30 Spiked Samples Levels should: Be known + appropriate. Reflect anticipated contamination levels/client specifications/legislation. Challenge/establish detection limit of method. Demonstrate performance/linearity over a range of levels relevant to lab (e.g. enumeration methods). Tips! Include low levels to check detection limit /sensitivity Detection Methods: normally emphasis on low levels + odd medium (high normally irrelevant) Enumeration Methods: include low, medium and high levels as appropriate to organism, customer specs/materials, legislation.

31 External Proficiency Samples Useful to include (if available/applicable). Provides data to demonstrate lab accuracy of enumeration/detection compared with peers Should not be sole source of validation/evaluation data Pure Cultures Not frequently included, but can be useful in certain situations: Developing a method entirely in-house method. When evaluating a confirmation step

32 Reference Materials Useful to consider if available at suitable contamination levels. Can be expensive, but can be useful option to avoid use of broth cultures with high levels of contamination. Examples: BioBall, HPA lenticules, TSC quanti disc, Oxoid Quanti loop.

33 Confirmation of Results Hello I m here! Honest! Presence of false results accounted for? Are positive results true positives? Are negative results true negatives? Tips! Purify and confirm isolates by recognised protocol When required: Isolates from naturally contaminated samples; Novel confirmation tests; If results based on growth on selective media without further confirmation.

34 Confirmation of Results A method may not routinely require a confirmation step, BUT for evaluation/performance studies it may be necessary to show positives are true positives and are not competitor organisms (i.e. False positives). If method under assessment is, or uses novel conformation test(s) it may be necessary to show the confirmation gives true positive not false positive results.

35 Experimental controls Shows spiked samples were not naturally contaminated Establish/calculate spike levels used. Tips! Run un-inoculated samples in parallel with spiked samples Perform viable count on inoculum + calculate number added to samples to allow recovery to be established

36 Preparation of Spiked Samples Practical Considerations

37 Enumeration v presence/absence Levels required? Background flora Realistic sample types Organism stress

38 Inoculation Approaches In-house cultures of approx known number Commercial preparations of bacteria Related samples: Ring trial/comparison trial samples (contaminated by organiser) Proficiency samples

39 Spike Levels Choose levels appropriate to samples and organism. Detection methods: focus on low, very low and a few medium levels. Enumeration methods: cover range from low, medium & high (include ranges relevant to client specs/legislation limits) Levels should investigate/determine limit of method detection. Consider what may already be present in sample... Consider possible antimicrobial effects

40 Traditional Approach Spike sample with cultures grown in suitable liquid medium: Dilute to required level. Inoculate without affecting sample (i.e. keep inoc volume small) Purity/suitability checks?

41 Traditional Approach How to Determine Spike Level? Overnight cultures grown in none selective broth at optimum conditions generally will give: Bacteria ~ 10^8 cfu/ml Yeasts ~10^5 10^6 cfu/ml Remember some organisms need >24 hours or are weak growing

42 Traditional Approach: Dilution Dilution of Broth Culture: Do not over dilute Take sample volume into account Add to bag or add to set aliquot Check the maths and establishj actual numbers added to sample - plate out inoculum Carry out test without delay

43 Things to Consider Food is not sterile! Background flora in the sample Un-inoculated controls

44 Things to Consider Organism stress Chill inoculated food? Freeze inoculated food? Assess loss of viability Chill for up to 72 hours, freeze for a number of days. Difficult to obtain consistent population of stressed cells

45 Things to Consider Dry products Inoculate a small portion of the product Dry overnight and then add to bulk (e.g. peppercorns with Salmonella)

46 Commercial Preparations Can be useful to spike samples Advantages: Stable Easy to use Does not involve culturing high levels of pathogens in lab Disadvantages: Expensive May not have required organisms available Levels may not always be suitable may need to dilute if too high.

47 Commercial Preparations of Bacteria Quantitative Preparations (ie of known cfu) Bioballs HPA lenticules Quanti-loops Quanti discs Qualitative Preparations (numbers present not assigned) Culti-loops Selectrol Discs

48 Heat-processed foods Heat culture at 50 C for 10 min to achieve 50 80% injury of the inoculum The degree of injury is calculated as follows: 1 (nselect/nnonselect) x 100 where nselect = mean number of colonies on selective agar nnonselect = mean number of colonies on non -selective agar

49 Liquid products Add diluted organism to sample and mix, stir, etc Examples: liquid egg, juices, liquid milk

50 Solid/moist foods Spot inoculate diluted organism over food sample/test portion Homogenise as appropriate Examples: meats, fish, vegetables, fruit

51 Solid/moist foods Chocolate: add diluted organism to melted chocolate, mix by stirring and allow to reharden Frozen foods: add diluted organism to thawed product, mix and refreeze

52 Dry foods Inoculate a small portion of the food to be tested Dry (e.g. overnight) and then add to food sample and mix: roll, shake, etc Examples: peppercorns, milk powder, dry pet food

53 Reporting and Presenting Validation/ Evaluation Work

54 Reporting Validation/In-House Evaluations Report to Include: Background/reason for evaluation Supporting literature/published data/approval status Criteria used to appraise results/data The experimental approach(s) used The method(s) used (method under review and comparator if applic)

55 Reporting Validation/In-House Evaluations Report to incl (Cont): The identity of samples used Identity of organisms and source/origin Spike levels ALL results/data generated Explain/justify any data excluded from appraisal Analysis of data Conclusions regarding method suitability for lab requirements

56 Successful Integration of Methods into Laboratory Operations

57 Integration into Routine Use Select method, bearing in mind origin/history Document method Train staff Calibrate equipment (if applicable) Obtain cultures (if applicable) Set Criteria to appraise data/method (seek advice if needed) Plan evaluation work (seek help/advice if needed)

58 Integration into Routine Use Collate data to demonstrate fitness for use + labs competence Analyse data (seek help if required) Draw conclusions Write up evaluation report Retain report for due diligence /accreditation/clients

59 Conclusions and The Future of Validation/Evaluation

60 In House Evaluation/Validation: Summary of Factors to Consider Review of literature/independent data Compare against another method (if applicable) Appraise recovery from typical samples (natural/artificially contaminated) Include External Proficiency samples Use pure cultures (if applicable) Inter laboratory ring trials (if applicable/available) Confirm positive/negative results (if applicable) Experimental controls Whilst many common factors, approaches used will be unique to each situation.

61 Validation of Methods: The Future The future is ISO Tech Committee 34 Sub Committee 9 working group 3 1. Validation Terminology - glossary of terms 2. Validation of Proprietary Methods - update present Intermediate Validation - for specialised tests where only v few labs available to test 4. Method Verification - in house evaluation of methods 5. In-House Method Validation - development of in-house methods with/without a reference method 6. Tech Requirements for establishment revision of Standardised Methods - protocol for approving/validating ISO methods