Solving Statistical Mysteries What Does the FDA Want?

Size: px

Start display at page:

Download "Solving Statistical Mysteries What Does the FDA Want?"

Augustine Warren
6 years ago
Views:

3 7 25Temperature(degC) 40 3.4 6.6 2.6 3.6 75 Humidity % 1.

1 3 7 25Temperature(degC) Humidity % No Base Present Yes Solving Statistical Mysteries What Does the FDA Want? Ronald D. Snee, PhD IVT Statistics in Validation Conference Philadelphia, PA June 14-16,

About the Speaker.. He is also an Adjunct Professor in the pharmaceutical programs at Temple and Rutgers Universities. He worked at DuPont for 24 years prior to starting his consulting career.

2 About the Speaker.. He is also an Adjunct Professor in the pharmaceutical programs at Temple and Rutgers Universities. He worked at DuPont for 24 years prior to starting his consulting career. Ron Snee, PhD is Founder of Snee Associates, LLC, a firm dedicated to the successful implementation of process and organizational improvement initiatives. He provides guidance to pharmaceutical and biotech senior executives in their pursuit of improved business performance using Quality by Design, Lean Six Sigma and other improvement approaches that produce bottom line results. He has authored several articles on how to successfully implement QbD, coauthored 2 books on QbD tools and speaks regularly at pharmaceutical and biotech conferences. Ron received his BA from Washington and Jefferson College and MS and PhD degrees from Rutgers University. He is an academician in the International Academy for Quality and Fellow of the American Society of Quality, American Statistical Association, and American Association for the Advancement of Science. He has been awarded ASQ s Shewhart, Grant and Distinguished Service Medals, and ASA s Deming Lecture and Dixon Consulting Awards. He is a frequent speaker and has published 5 books and more than 280 papers in the fields of quality, performance improvement, management, and statistics. He recently received the Institute of Validation Technology s Speaker of the Year Award. 2

3 Abstract Use of statistics has been part of the FDA s guidances and regulations for many years. Use of statistics is an integral part of the Jan 2011 FDA Process Validation Guidance. It can be argued that the FDA is looking for the pharma and biotech industries to make even greater use of statistical thinking and methods. Even with this long history, the use of statistics seems to be a source of uncertainty and anxiety for many. This presentation provides an overview of what it appears the FDA is looking for in the use statistics including examples and recommended approaches. 3

4 Agenda Today s Reality: FDA Guidance Regarding: Quality by Design Continued Process Verification Test Method Development FDA Guidance s Examples of Recommended use of Statistical Thinking and Methods Tools, Methods and Personnel What is the Appropriate Documentation? What s the FDA Looking for? Right Process Measurement, Data, Tools and Methods and Interpretation Useful Standards Understanding of tool usage and interpretation of results Tips, Traps and Recommendations 4

Full implementation of QbD and modernization of manufacturing by the pharmaceutical industry in development through manufacturing is expected to provide lasting benefits to industry,

5 Full implementation of QbD and modernization of manufacturing by the pharmaceutical industry in development through manufacturing is expected to provide lasting benefits to industry, regulators and patients. Janet Woodcock, Director FDA Center for Drug Evaluation and Research Before the U.S. House of Representatives Subcommittee on Energy Policy, Health Care and Entitlements 5

6 Quality by Design Quality by Design An Effective Approach Systematic approach to product and process development Begins with predefined objectives Emphasizes product and process understanding and process control Based on sound science and quality risk management From ICH Q8(R1) Step 2 Quality by Design About Building Quality into Products and Processes Based on Effective Collection and Analysis of Data 6

7 FDA Process Validation Guidance Importance of Variation A successful validation program depends upon information and knowledge from product and process development. This knowledge and understanding is the basis for establishing an approach to control of the manufacturing process that results in products with the desired quality attributes. Manufacturers should: Understand the sources of variation Detect the presence and degree of variation Understand the impact of variation on the process and ultimately on product attributes Control the variation in a manner commensurate with the risk it represents to the process and product Each manufacturer should judge whether it has gained sufficient understanding to provide a high degree of assurance in its manufacturing process to justify commercial distribution of the product. 7

8 It s About Variation All processes--human and non human--exhibit variability. This variability is measurable. Joseph M. Juran If I had to reduce my message for management to just a few words, I d say it all had to do with reducing variation. W. Edwards Deming State of statistical control is not a natural state for a manufacturing process. It is instead an achievement, arrived at by elimination, one by one, by determined effort, of special causes of excess variation. W. Edwards Deming So Why Statistics? Statistical Thinking and Methods Enable Us to Deal Effectively with Process and Product Variation 8

9 Variation Drives Risk, Quality, Cost and Customer Satisfaction Risk Quality Variation Variation Costs ($$) Customer Satisfaction Variation Variation Understanding and Reducing Variation is a Good Thing 9

10 Variation Also Drives Process Understanding Process Understanding Variation Increased Process Understanding Enhanced Process Control and Improvement It Starts with Understanding Variation: You Can t Successfully Control, Improve and Transfer a Process that You Don t Understand 10

11 Process Validation Guidance Stage 3 Continued Process Verification An ongoing program to collect and analyze product and process data that relate to product quality must be established ( (e)). The data collected should include relevant process trends and quality of incoming materials or components, in-process material, and finished products. The data should be statistically trended and reviewed by trained personnel. The information collected should verify that the quality attributes are being appropriately controlled throughout the process. 11

12 FDA Wants Statistics to be Used We recommend an integrated team approach to process validation that includes expertise from a variety of disciplines (e.g., process engineering, industrial pharmacy, analytical chemistry, microbiology, statistics, manufacturing, and quality assurance). Project plans, along with the full support of senior management, are essential elements for success.. We recommend that a statistician or person with adequate training in statistical process control techniques develop the data collection plan and statistical methods and procedures used in measuring and evaluating process stability and process capability. FDA Process Validation Guidance, Jan

13 FDA Guidance for Industry Analytical Procedures and Methods Validation for Drugs and Biologics VII. STATISTICAL ANALYSIS AND MODELS A. Statistics Statistical analysis of validation data can be used to evaluate validation characteristics against predetermined acceptance criteria. All statistical procedures and parameters used in the analysis of the data should be based on sound principles and appropriate for the intended evaluation. Several statistical methods are useful for assessing validation characteristics, for example, an analysis of variance (ANOVA) to assess regression analysis R (correlation coefficient) and R squared (coefficient of determination) or linear regression to measure linearity. Many statistical methods References ASTM E1488 Statistical Procedures to use in Developing and Applying Test Methods ASTM E2782 Standard Guide for Measurement Systems Analysis (MSA), 13

14 Code of Federal Regulation Title 21 Part 820 Quality System Regulation Subpart J Corrective and Preventive Action Sec Corrective and Preventive Action. (a) Each manufacturer shall establish and maintain procedures for implementing corrective and preventive action. The procedures shall include requirements for: (1) Analyzing processes, work operations, concessions, quality audit reports, quality records, service records, complaints, returned product, and other sources of quality data to identify existing and potential causes of nonconforming product, or other quality problems. Appropriate statistical methodology shall be employed where necessary to detect recurring quality problems; 14

15 Code of Federal Regulations Title 21 Part Quality System Regulation Subpart O--Statistical Techniques Sec Statistical techniques. (a) Where appropriate, each manufacturer shall establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capability and product characteristics. (b) Sampling plans, when used, shall be written and based on a valid statistical rationale. Each manufacturer shall establish and maintain procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed. These activities shall be documented. 15

16 Guidance for Industry and FDA Staff 30-Day Notices, 135-Day Premarket Approval (PMA) Supplements and 75- Day Humanitarian Device Exemption (HDE) Supplements for Manufacturing Method or Process Changes (April 2011) Summary of the data or information supporting the change, should include: a summary of the procedures established for the identification, documentation, validation, review, and approval of the manufacturing changes submitted in the 30-day notice; the statistical rationale for the sampling method, if you plan to verify the changed processes by routine sampling and independent measurement; a description of how you will monitor and control any manufacturing process you intend to change; a summary of the completed validation study that demonstrates that the manufacturing change can be made without significantly changing the operation of the final device. This summary should be include: a description of the acceptance criteria; information on how, using valid statistical methods, you analyzed the test data; information that describes the statistical rationale for sample sizes; a list of any deviations that occurred; and a determination of the impact of the deviation on the results; an explanation of how change control procedures were implemented,.. 16

17 Today s Realities FDA insists on the use of data and statistics FDA Process Validation Guidance, January 2011 FDA s Rick Friedman and Karthik Iyer explain why some manufacturers need to get a better grip on GMP-related statistics. PharmaManufacturing, June 5, 2012 FDA emphasizes that the appropriate statistical tools must be employed when it is necessary to utilize statistical methodology..fda has seen far too often the misuse of statistics by manufacturers in an effort to minimize instead of address the problem. Such misuse of statistics would be a violation of this section Fed. Reg. at , Comment 159. From K. Lewandowski-Walker. FDA Office of Regulatory Affairs (2012) Data abound.. Data, Data, Data everywhere Big Data 17

18 Agenda Today s Reality: FDA Guidance Regarding: Quality by Design Continued Process Verification Test Method Development FDA Guidance s Examples of Recommended use of Statistical Thinking and Methods Tools, Methods and Personnel What is the Appropriate Documentation? What s the FDA Looking for? Right Process Measurement, Data, Tools and Methods and Interpretation Useful Standards Understanding of tool usage and interpretation of results Tips, Traps and Recommendations 18

19 Current Situation FDA guidances are not prescriptive. The reviewers/inspectors are not harmonized! Different reviewers may make different comments on a given situation FDA Guidances Are not Prescriptive. Now What? Strategy Collect some data on current state Observe FDA practices Get input from professionals in the field Develop a model for what FDA wants industry to do 19

20 1. Warning Letter Sampling Plans Firm using sampling plans incorrectly Pooled X vials, used only 1 reportable value, but used n=x in sampling plan..based your lot or batch acceptance/rejection criteria on a single reportable value averaged from a pooled sample. For.., you are collecting 3 pooled samples (each pool = 10 vials). This equates to a lot disposition action on 3 reportable values with corresponding AQL of X% and LQ of X% respectively. This is not equivalent to an X or X plan as claimed in your SOP. Source: Karthik Iyer, Senior Policy Advisor, CDER/OC/OMPQ March 9 th, 2012, AOAC Conference 20

21 1. Warning Letter Sampling Plans Response to 483 indicated firm did not know how to use and interpret sampling plans correctly. Firm did not understand concepts of Acceptable Quality Level (AQL) and Limiting Quality (LQ) and Operating Characteristic Curve (OC) of a specific sampling plan. Source: Karthik Iyer, Senior Policy Advisor, CDER/OC/OMPQ March 9 th, 2012, AOAC Conference 21

22 FDA Response Case A Questions Use of Valid Statistical Rationale And Sample Sizes Failure to ensure that sampling methods are based on a valid statistical rationale, are adequate for their intended use, and that the sampling plans are reviewed when changes occur, as required by 21 CFR (b). In terms of the statistical sample, your response cited CAPA; however, our review of this document did not reveal any actions regarding sample size vs. lot size. The other document referenced in your response is your Statistical Techniques procedures. You have not provided any documentation regarding the statistical validity of your new sample size. 22

23 FDA Response Case B Questions Rationale for Statistical Methods Used an And Procedures for Use of Selected Methods Failure to establish and maintain adequate procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of a process capability and product characteristics, as required by 21 CFR (a). For example, when the investigator requested the statistical rationale for selecting verification testing, your firm said that there was no rationale. 23

24 FDA Response Case C Lack of Appropriate Trending Procedures and Appropriate Statistical Methods Your firm's response to Observation 3.a is inadequate because it fails to address appropriately trending the 66,390 complaints received between 1/1/2013 and 7/15/2014 Your firm failed to analyze service reports following appropriate statistical methods, as required by 21 CFR (b). Your firm failed to adequately establish procedures for receiving, reviewing, and evaluating complaints by a formally designated unit, as required by 21 CFR (a). 24

25 FDA Response Case D Failure to Use Appropriate Sampling Methods and Provide Statistical Rationale Failure to ensure that sampling methods are based on a valid statistical rationale, are adequate for their intended use, and that the sampling plans are reviewed when changes occur, as required by 21 CFR (b). The adequacy of your firm s response cannot be determined at this time. You have indicated you will be making revisions to the sampling schedule and revising the procedure. These are planned actions that we cannot evaluate at this time. 25

26 FDA Response Case E Failure to Establish and Maintain Procedures for Identifying Valid Statistical Techniques Trend Root Causes Not Considered Failure to establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capability and product characteristics, as required under 21 CFR (a). The trending of the data utilized by your QIT to review and monitor the effectiveness of the corrective and preventive actions is inadequate since the variable factors that might affect the true outcome of the trend were not considered and thus product characteristics were not established, controlled, and verified. 26

27 Summary: Some Problems Found by FDA Acceptance Sampling Plans not understood or used properly Valid Statistical Rationale And Sample Sizes not Used Valid Statistical Rationale And Company Procedures Regarding Use of Statistics Lack of Appropriate Trending Procedures Lack of Appropriate Statistical Methods Failure to Establish and Maintain Procedures for Identifying Valid Statistical Techniques Critical sources of trend (root causes) not considered Establish cause and Effect: Y = f(x) 27

28 Agenda Today s Reality: FDA Guidance Regarding: Quality by Design Continued Process Verification Test Method Development FDA Guidance s Examples of Recommended use of Statistical Thinking and Methods Tools, Methods and Personnel What is the Appropriate Documentation? What s the FDA Looking for? Right Process Measurement, Data, Tools and Methods and Interpretation Useful Standards Understanding of tool usage and interpretation of results Tips, Traps and Recommendations 28

29 What is FDA Looking For? Good rationale for statistical methods used Documentation for statistical methods used Records that show: Proposed methods were actually used Those using the methods were trained on the use of the methods Documentation of validation of statistical software used purchased and internally developed Use of Standards ASTM, ANSI,.. 29

30 Make Use of Standards ASTM Standard E2810 E2782 E2709 E2587 E2281 E2234 E1488 E122 Standard Practice Demonstrating Capability to Comply with the Test for Uniformity of Dosage Units Standard Guide for Measurement Systems Analysis (MSA) Demonstrating Capability to Comply with an Acceptance Procedure Use of Control Charts in Statistical Process Control Process and Measurement Capabilities Sampling a Stream of Product by Attributes Indexed by AQL Statistical Procedures to use in Developing and Applying Test Methods Calculating Sample Size to Estimate, With Specified Precision, the Average for a Characteristic of a Lot or Process 30

31 What is FDA Looking For? Factor in risk/confidence everywhere. Capability: Consider the lower confidence interval of capability indices. If your sample size is too small, you may find that Ppk is 1.4 but the lower CI for Ppk is below 1.0. Sample sizes: Use appropriately calculated sample sizes. Make statements about power, confidence, and risk. Stability: When extrapolating, always use confidence lines and see where they intersect the spec limit. Use statistically designed experiments Collect good data Randomize, replicate, block when appropriate, execute according to a protocol, analyze and interpret appropriately. 31

32 Comments on Specific Methods Perform risk assessments How can the process fail Use Failure modes and Effects Analysis (FMEA) Control charts use the correct chart Average and Standard Deviation Individuals and Moving Range Percent Defective Defects per Unit Demonstrate process understanding. Identify critical process parameters and critical raw material attributes. Establish Cause and effect: Y = f(x) 32

33 Statistical Significance is Not the Same as Practical Significance Statistical Significance: Observed difference is larger than can be attributed to random chance variation Process variation is small: Difference may be statistically significant but be of no practical importance p-value measures statistical significance, not practical significance Statistical significance must be present before it is meaningful to assess practical significance Practical Significance: Determined by size and direction of effect: Effect (difference) is large enough to: Have real meaning in context of study and objectives Justify behavior change; do something in a different way Have an effect on decisions to be made Influence the behavior of professionals in the field Practical Significance is a function of factors such as cost, objectives, priorities, requirements, subject matter knowledge, etc. 33

34 Appropriate Actions Regarding Statistical Significance and Practical Significance Practically Significant? YES NO YES NO Take Appropriate Action Take No Action Keep Effect in Mind Effect May Later Become Practically Significant Statistically Significant? Increase Sample Size Estimate Effect Size More Precisely Action Not Appropriate 34

35 So What Should I do? Use good statistical practice Develop rationale for procedures Planning: Use selected procedures according to protocol Document procedures Train appropriate personnel on the procedures Provide appropriate validated software Demonstrate process understanding Establish Cause and Effect: Y = f(x) Use appropriate standards Show that management supports all procedures used Focus on practical significance of trends and shifts Statistical Thinking and Methods Enable Us to Deal Effectively with Process and Product Variation 35

36 For Further Information, Please Contact: Ronald D. Snee, PhD Newark, DE (610) Please visit our website at: 36

37 Problem Solving Approach Identifies the Right Tools Not the Reverse! What s the problem? Context, process, data pedigree, goals and objectives Problem Solving Approach and Statistical Tools to be used? Statistical tools to aid the problem solving process? Results What Results? Desired Format? Not Understanding the Problem to Be Solved Can Result in a Statistical Tool being Used as A Hammer Looking for a Nail 37

38 Problem Solving Approach Identifies the Right Tools Not the Reverse! Statistical Tools Results ($$) Select Tools Perform Analysis Run Chart? ANOVA? Practical Data DOE? T-Test? Regression? Scatter Plot? Graphical Analytical Collect Other Tools??? Identify Solution and Action 38

39 Problem Example Compare Two Formulations Want to compare two formulations regarding dissolution Data: Dissolution (%) for 10 tablets for each formulation Approach and Statistical Tools Draw a plot of the data to assess the difference between the average amount of Dissolution of the two formulations and the variation of the two formulations Statistical Tools Result Student s t-test to compare the averages of the two formulations Levine's test to compare the variances of the two formulations Formulation averages are significantly different (p=0.028) No significant differences between the Formulation variances (p=0.150) 39

Overview of Statistics used in QbD Throughout the Product Lifecycle

Overview of Statistics used in QbD Throughout the Product Lifecycle August 2014 The Windshire Group, LLC Comprehensive CMC Consulting Presentation format and purpose Method name What it is used for and/or