Yes, there is going to be some math (but not much) STATISTICAL APPROACH TO MEDICAL DEVICE VERIFICATION AND VALIDATION

Transcription

1 Yes, there is going to be some math (but not much) STATISTICAL APPROACH TO MEDICAL DEVICE VERIFICATION AND VALIDATION

2 Medical Device Verification and Validation In the medical device world, verification and validation are the backbone of design control This is how you prove you have produced what you said you would and what the customer wants! It s also required

3 What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec Definitions Verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled Design verification. Each manufacturer shall establish and maintain procedures for verifying the device design. Design verification shall confirm that the design output meets the design input requirements. The results of the design verification, including identification of the design, method(s), the date, and the individual(s) performing the verification, shall be documented in the DHF

4 What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec Definitions Validation means confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled Design validation means establishing by objective evidence that device specifications conform with user needs and intended use(s)

5 What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec Definitions Design validation. Each manufacturer shall establish and maintain procedures for validating the device design. Design validation shall be performed under defined operating conditions on initial production units, lots, or batches, or their equivalents. Design validation shall ensure that devices conform to defined user needs and intended uses and shall include testing of production units under actual or simulated use conditions. Design validation shall include software validation and risk analysis, where appropriate. The results of the design validation, including identification of the design, method(s), the date, and the individual(s) performing the validation, shall be documented in the DHF This is NOT the same as Test Method or Process Validation

6 Where Does V&V Fit Into the Design Control Picture?

7 A Few Quick Points Design V&V studies are NOT experiments! Y0u should know the outcome before you start. Require predefined acceptance criteria Which is why they are not experiments Before you start V&V, you should have enough characterization data to understand your system (product)

8 Ok Enough Introduction But before we get into the nerdy stuff a few hints to make it easier: Ensure you have clearly defined and measurable product requirements Use standards when available (more in a minute) Hire a good statistician!!

9 Standards and Guidelines There are several good guidelines and standards published to help CLSI publishes a variety of guidelines for statistical analysis useable for verification studies Normally protocols based on CLSI guidance is accepted by the FDA Common Technical Specifications (CTS) for in vitro diagnostics medical devices These CTS requirements are associated with the IVD Directive 2009/108/EC of the European Parliament and the Council of 3rd February 2009 on in vitro diagnostic medical devices

10 Standards and Guidelines A few words on CLSI EP Protocols Provide estimates for individual parameters (limit of detection, precision, bias ) CLSI does not have a standard to combine parameters CLSI standards in many cases proscribe sample sizes and outliers routines

11 Two Requirements for CLSI EP Protocols Doable Don t need a Nobel prize to figure out how to do it Right As in right enough. If CLSI method deviates from a more complicated gold standard method after the 7th decimal place, that s ok An EP rule The least burdensome of several alternatives will always be used if the minimum sample size is 40, 40 will be the only sample size used

12 Speaking of Sample Size Three ways to get sample size From a standard This is the easiest Math Good enough Most common questions I get (how many???)

13 Number of Replicates Math for Replicate Number: nn zz 11 αα/22 + zz 11 ββ/22 SSSS/dddddddddd 22 SD is the standard deviation estimate based on characterization delta is the allowable error limit n is the number of samples α is the significance level, usually β is the power of the test z 1-α/2 and z 1-β/2 are the z values corresponding to the probability of 1-α/2 and 1-β/2 respectively on a normal distribution table (z table)

14 Number of Replicates Good Enough for Replicate Number: You will find that unless you have unusual circumstances, the calculation on the last slide will normally give you a small number of replicates Three (3) is the minimum number you need to see a possible outlier and get a good mean Power calculations will tend to give you an answer which is more than you have time or money for Unless y0ur system has a lot of rep to rep variability, a smaller number of reps will work Variability can be determined through good characterization and used as justification for number of reps.

15 Speaking of Sample Size The same type of logic applies for how many samples are needed: From a standard Again, this is the easiest Math Good enough

16 Speaking of Sample Size Math: Some definitions: Margin of Error (Confidence Interval) No sample will be perfect, so you need to decide how much error to allow. The confidence interval determines how much higher or lower than the population mean you are willing to let your sample mean fall Confidence Level How confident do you want to be that the actual mean falls within your confidence interval? The most common confidence intervals are 90% confident, 95% confident, and 99% confident

17 Speaking of Sample Size Math: Confidence Level Your confidence level corresponds to a Z-score. This is a constant value needed for this equation. Here are the z- scores for the most common confidence levels 90% Z Score = % Z Score = % Z Score = nn = zz22 SSSS (11 SSSS) MMMMMMMMMMMM oooo EEEEEEEEEE 22 You will find this will give you pretty large sample numbers You also have to know the SD

18 Speaking of Sample Size Good Enough Large vs. Small Sample Numbers Larger sample numbers are more likely to approach a closer model to the total population than smaller samples numbers Larger sample numbers are more likely to approach a normal distribution than small sample numbers A rule of thumb is to have 16 degrees of freedom (D.F.) for any average standard deviation used in building a model "Understanding Industrial Designed Experiments" (Schmidt & Launsby) This would be 17 samples

19 Speaking of Sample Size "The closeness of the approximate probability 1- alpha to the exact probability depends on both the underlying distribution and the sample size...as the underlying distribution becomes 'less normal' (i.e., badly skewed or discrete), a larger sample size might be required to keep a reasonably accurate approximation. But, in all cases, an n of at least 30 is usually quite adequate. Probability And Statistical Inference 2nd Edition" (Hogg & Tanis)

20 Speaking of Sample Size As a rule of thumb, you can consider n=30 observations or more to constitute a large' sample So If n=30 is a large sample, and n-17 is a reasonable number to get enough degrees of freedom Therefore, by using n=20 there is a compromise in dropping the theoretical value (n=30) to a value that is more practical to use in industrial settings

21 Speaking of Sample Size A few final notes on sample size If SD is known, then use z distribution. If SD is not known: If n is large, then use z distribution. If n is small, then use t-distribution

22 Design Verification Examples There are so many different types of product requirements and potential ways to verify You really don t want to be here all night, do you? Here some suggestions of product requirement verifications and what elements you should have in your protocols and reports

23 Precision and Accuracy These are sometimes confused You need to be sure you are measuring the correct parameter And that parameter meets you needs and requirements

24 Evaluation of Imprecision and Inaccuracy Imprecision Refers to random analytic error Lack of repeatability and/or reproducibility Inaccuracy Refers to systematic analytic error Lack of trueness Constant Proportional Total Error Combined error for a single result

25 Precision and Accuracy

26 How Much Error is Ok?? Performance is acceptable when Observed error < allowable error Performance is not acceptable when Observed error > allowable error How much error is acceptable? Based on medical and or clinical need User needs traced to product requirements which are what you verify/validate

27 Protocols Should contain (at a minimum) Purpose including design input, design output being verified Rationale for the verification design Measurable acceptance criteria. Materials required Verification Method/Procedure. Include procedure or reference for evaluation of unexpected test results (e.g. no-tests, invalids) Boundary values, must be assessed and included Data Analysis including validity criteria, if applicable Statistical methods with references to published methods, if applicable Include method for handling exceptions and outliers.

28 Reports Should contain (at a minimum) Materials used (including part number, lot number, expiration date, storage condition, as applicable) Equipment used and calibration status Revisions of any reference procedures Date and location of testing Individual(s) performing the verification Results including or referencing raw data Summary of procedural errors and investigation of failures or unexpected results, if any Summary of excluded data, if any Disposition, i.e., whether or not the acceptance criteria was met. If acceptance criteria are not met, a discrepancy report should be initiated

29 Design Validation I did not spend a lot of time (or any) on design validation In many cases, the number of sites, samples and statistics required are dictated by Regulatory agencies Cost of external studies Availability of clinical samples Time (mainly cost and management driven) You still have to justify what you do and have the data analyzed statistically

30 In Summary You always need to have some sort of statistical justification for your studies You need to be able to defend your justification There are many guidelines and standards available to reference HIRE A GOOD STATISTICIAN

31 Almost Last Slide

32 That s All Folks! My Contact Information: Alan Golden Manager QA, Operations Support Abbott Molecular Alan.golden@abbott.com