Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems

Transcription

1 WHITE PAPER Analyst Software Validation Service Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems S. D. Nelson, Blair James, Patrick Quinn-Paquet, Justyna Sekula, Dave Abramowitz Abstract This whitepaper discusses two unique additions to the GAMP 5 Risk Assessment for GLP Computerized Systems: detectability as a third risk assessment vector and Fibonacci sequences for prioritizing risks. Introduction ICH Q9 Quality Risk Management defines risk as the combination of the probability of occurrence of harm (threat likelihood) and the severity of that harm (impact). Using this definition, members of the International Society for Pharmaceutical Engineering (ISPE) ascertained that the level of effort, formality, and documentation of the risk management process should be commensurate with the level of risk associated with the system being managed. The heart of this risk management process is the risk assessment. Currently, risk assessment for GLP laboratory software validation is accomplished in accordance with ISPE 2008 GAMP 5, A Risk-Based Approach to Compliant GxP Computerized Systems (GAMP 5 ). GAMP 5 relies on two important risk assessment vectors: threat likelihood and impact. Leveraging industry best practices the Compliance Services team at AB SCIEX felt that adding the third risk vector of detectability was appropriate for GLP purposes in an LCMS system software validation. Detectability was added because if a risk cannot be detected, it cannot be mitigated. GAMP 5 recommends ranking risks into three categories: high priority, medium priority, and low priority. However, there is no guidance for prioritizing risks within each ranking. For example, during risk assessment of the AB Sciex Analyst software, 73 risks were identified. Forty-one (41) risks were classified as high priority. If there are 41 high priority risks, which one is most important? Which must be mitigated first? The AB Sciex team devised a novel approach to answer this critical question by using partial Fibonacci sequences to rank all the risks into an ordered list. Then, in order to meet GAMP 5, software validation experience was used to divide the fully ranked list of risks into the three GAMP 5 categories. This approach provides the GLP laboratory management with a much clearer picture of the prioritization of all risks so mitigations may be applied to manage higher priority risks first. Dealing with the most critical risks first is the crux of GAMP 5 and thus this innovative approach using detectability as a third risk assessment vector along with Fibonacci sequences to rank all risks not only meets but also exceeds the rigor required for GAMP 5. Risk Assessment This section describes how these two important additions are incorporated into the GAMP 5 risk assessment process. For clear understanding an example is used. This example is the AB Sciex Software Validation Risk Assessment (SVRA) conducted for the Analyst v1.6.x Liquid Chromatography/Mass Spectrometry (LC/MS) system. For GAMP purposes, this system is categorized as a vendor-supplied Category 4 configurable software package used for GLP laboratory instruments with software capable of controlling liquid chromatograph with a mass spectrometer.

2 Communicate Risks 5. Cont rol Monitoring and Risk Reivew Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems 2 Performing Risk Assessment GAMP 5 describes five steps to performing a quality risk assessment: 1. Perform initial risk assessment and determine system impact 2. Identify functions with impact on product quality and data integrity 3. Perform functional risk assessments and identify controls 4. Implement and verify appropriate controls 5. Review risks and monitor controls This whitepaper addresses the first three steps. Software validation addresses number four and Standard Operating Procedures (SOP) and ongoing review cover number five. Figure 1 below shows the GAMP 5 risk assessment process. 1. Initial Assessment of System Impact 2. Identify Risks - Product Quality & Data Integrity 3. Evaluate Risk & Devise Controls Risk Decision #1 (Assessment satisfactory?) No Yes 4. Implement Controls Risk Decision #2 (Controls satisfactory?) No Yes Risk Acceptance End of first or subsequent iterations Figure 1: GAMP 5 Risk Management Process adapted from ISO/IEC Risk Management Process (ISO/IEC 27005, 2012) Identifying Risks Risks are identified by considering business processes, systems, and user and functional requirements. During the initial risk assessment the AB Sciex team investigated the business processes of the GLP laboratory. The LC/MS

3 Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems 3 system is often used for generating data in support of GxP-regulated studies and thus considered to be a critical business process. Additionally, since the LC/MS system will create, modify, distribute, and maintain electronic records in support of FDA-regulated studies, U.S. FDA regulation 21 CFR Part 11: Electronic Records; Electronic Signatures ( Part 11 ), also applies. Based upon the applicability of the aforementioned regulations and the potential impact of the system to directly affect the integrity and reliability of pivotal data included in regulatory submissions, the overall system risk is deemed to be critical for the GLP laboratory. Thus the risk assessment and associated controls merit careful scrutiny. Risks were based on regulatory concerns, GLP laboratory best practices, as well as, the Analyst 1.6.x User and Functional Requirements Specification (UFRS). Seventy-three (73) risks were identified. A few examples of these risks are included in Appendix A Risk Assessment with Proposed Mitigations. Evaluating Risks After risks are identified, each risk must be evaluated. In the example, risks were analyzed to understand possible hazards for the LC/MS system. Three risk assessment vectors were used: Threat Likelihood, Impact and Detectability. Each risk assessment vector was assigned a number from this partial Fibonacci sequence (0,1,2,3,5,8,13,21,34,55). Threat Likelihood is the probability of occurrence of harm. The Threat Likelihood was assessed by assigning one number from the partial Fibonacci sequence where zero indicates no likelihood and 55 indicates high likelihood. Impact is the effect of an adverse event in terms of short and long term duration on the business and regulatory compliance. Impact was assessed by assigning one number from the partial Fibonacci sequence where zero indicates no impact and 55 indicates high impact. Detectability is the ability to discover an adverse event. Detectability was assessed by assigning one number from the partial Fibonacci sequence where zero indicates unable to detect and 55 indicates easy to detect. Prioritizing Risks Finally, risks are prioritized by calculating the relationship between Threat Likelihood, Impact and Detectability. This relationship can be expressed by applying this formula: Likelihood + Impact Detectability = Priority Once risks are prioritized then they are reviewed and categorized as High Priority, Medium Priority, or Low Priority per GAMP 5. Developing Controls Risk mitigations must be devised for each risk. At least one control must be implemented for each risk mitigation. Controls necessary to implement risk mitigations are either technical controls or procedural controls. In the example, technical controls are provided by Analyst 1.6.x software and implemented during software validation. Procedural controls are implemented via Standard Operating Procedures (SOP) after software validation. Implementing and Verifying Controls Once the risk assessment is finished, controls are implemented. In the example, after the SVRA was approved, software validation efforts were planned to focus on mitigating risks of high criticality first. And SOPs were written to mitigate other risks.

4 Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems 4 Figure 2 illustrates the concept of planning validation and selecting controls based on assessed risk. In general, functions with high risk will require more testing than functions with low risk. This figure is from GAMP 5 Quality Risk Management Approach, Kevin C. Martin and Dr. Arthur (Randy) Perez, Pharmaceutical Engineering, May/June 2008 Vol. 28 No.3. Figure 2: Relationship of Risk, Impact and Control Reviewing Risks and Monitoring Controls In order to ensure that risk mitigations and controls are working, they must be reviewed and monitored. After gathering risks and developing mitigations, the AB Sciex reviewed the list of risks to ensure the list was complete. Afterwards they determined if previously identified hazards were still present and ascertained if the estimated risk associated with said hazard was acceptable. Finally, they evaluated whether all existing controls were effective and still necessary. They developed a plan for ongoing risk assessment and monitoring of controls. Risk Summary This section summarizes the risks identified in the Risk Assessment for Analyst 1.6.x software. As per GAMP 5 recommendations, risks have been divided into high priority, medium priority, and low priority risks. Each risk type is detailed and described and summarized below. High Priority Risks In the AB Sciex LC/MS system example, forty-one (41) high priority risks were identified as shown in Table 1: High Priority Risks. High priority risks were considered any serious and impacting risk with a ranking of 16 or higher. These high priority risks include concerns about data that could be lost including project data, audit trail data and quantitation data. They also include the possibility of security configuration changes and unauthorized access to Windows or Analyst software. When risks have the same priority, their ranking is determined by AB Sciex team evaluation. Table 1: High Priority Risks

5 Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems 5 Risk ID Risk Scenario Priority K.15 The integrity of acquisition data files is not verified 68 K.14 Electronic records can be deleted 68 K.72 Sample vial or container is mislabeled 59 K.41 Project data is lost because project folders are overwritten 58 K.12 Existing quantitation files can be changed (overwritten) by users other than Administrator 56 K.51 Re-acquired samples can overwrite the original sample 56 K.42 Project data is lost because there is no standard folder or sub-folders for storing project data 55 K.11 Security configuration can be changed without authorization 55 K.16 Analyst Software can overwrite raw data files during data acquisition 55 K.66 Quantitation results differ by greater than 2 decimal places (e.g..01) 54 K.13 All Analyst Software files are not properly installed 52 K.8 Audit trail map is changed so that GLP events are not properly audited 51 K.9 Quantitation result tables can be changed (overwritten) by users other than the Administrator 51 K.10 Audit trail can be cleared 51 K.6 Acquisition Method files can only be overwritten by Administrator and not by other users 51 K.1 Unauthorized access to Windows 50 K.2 Unauthorized access to Analyst Software 50 K.17 Data files can be changed by outside means without detection 42 K.48 Data files do not reference the corresponding acquisition method in human-readable format 42 K.53 The acquisition date, time and username are not recorded for acquired samples 42 K.73 Sample is loaded in incorrect injection position 40 K.27 Audit trail does not contain the modification of files 35 K.69 Incorrect connection between LC and MS 29 K.32 Audit trail does not reference manual integration of peaks 29 K.7 Project data lost because Analyst root directory is changed 28 K.22 Audit trail does not contain the date on which the event occurred 24 K.23 Audit trail does not contain the time when the event occurred 24 K.39 Audit trail does not record failed logon attempts 24 K.18 System clock settings can be changed without authorization 24 K.70 Unsuccessful restoration of backup 24 K.37 Audit trail does not reference re-acquisition of a sample 23 K.34 Audit trail does not reference removal of samples to a quantitation results table 23 K.36 Changes to quantitation results sets are not recorded in the quantitation audit trail 23 K.64 Users can set a threshold that prevents manual integrations 19 K.26 Audit trail does not reference the creation of files 18 K.28 Audit trail does not contain the addition of a sample to a raw data file 18 K.33 Audit trail does not contain addition of samples to a quantitation results table 18 K.35 Quantitation audit trail does not include saving the results of quantitation 18 K.29 Audit trail does not include changes to the quantitation processing algorithm 17 K.38 Audit trail does not record users logging onto the system 16 K.40 Audit trail does not record changes to security configuration 16

6 Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems 6 Medium Priority Risks Twenty-two medium priority risks listed were identified as listed in Table 2: Medium Priority Risks. Medium priority risks have a ranking between -2 and 15. These risks include concerns regarding misconfigured default values, changes to system clock settings, inability to manually tune the mass spectrometer, unclear privileges and inability to conduct quantitation. When risks have the same priority, their ranking is determined by AB Sciex team evaluation. Table 2: Medium Priority Risks Risk ID Risk Scenario Priority K.30 Changes to internal settings of the quantitation method are not recorded in the audit trail 16 K.25 Audit trail records recording changes do not reference the original and changed value 13 K.68 User selects incorrect tune masses 13 K.4 Administrator privileges are unclear or not specified 11 K.3 Lab workers are assigned to multiple roles making their actual privileges unclear 8 K.20 Unused functionality is not disabled 5 K.45 Manual tuning cannot be accomplished 3 K.44 Instrument performance cannot be verified 3 K.5 User privileges are unclear or not specified 3 K.47 Lab workers cannot create files necessary to acquire data 2 K.50 Analyst cannot control the instrument (e.g. Equilibrate) 2 K.57 Quantitation method cannot be created 2 K.59 Peak area ratios cannot be calculated 2 K.60 Peak height ratios cannot be calculated 2 K.61 Chromatograms cannot be reviewed for correct integration 2 K.62 Quantitation results cannot be changed and recalculated 2 K.58 Quantitation results cannot be created 2 K.21 Audit trail does not reference the user initiating the event 2 K.55 The instrument, user and acquisition information are not embedded in the raw data file 2 Computer system failure resulting in loss of operating system and Analyst Software configuration and K.71 calibration information 1 K.24 Audit trails are not in human-readable format 1 K.31 Audit trail does not reference reversion of manually integrated peaks -2 Low Priority Risks Ten low priority risks were identified as shown in Table 3: Low Priority Risks. Low priority risks are any risk with a ranking less than -2. Risks include unlikely concerns regarding the lack of human-readable audit trails, metadata for acquisition data and report templates. When risks have the same priority, their ranking is determined by AB Sciex team evaluation. Table 3: Low Priority Risks Risk ID Risk Scenario Priority K.52 Default values for the sample queue are misconfigured -6 K.67 QA Review is not recorded in the quantitation audit trail -6

7 Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems 7 Risk ID Risk Scenario Priority K.19 There is no screensaver or user lock out to prevent access to data by an authorized party -13 K.63 Lab workers cannot choose the type of quantitation algorithm to use -18 K.46 A large batch of samples cannot be acquired -18 Analyst software does not allow proper control sample acquisition including, but not limited to, starting, K.49 pausing and stopping data acquisition -19 K.43 Compound optimization cannot be performed -39 K.54 Live updates of chromatograms cannot be viewed -39 K.56 Background subtraction cannot be performed -45 K.65 Custom report templates cannot be created -45 Summary In total, 73 risks were identified and 226 risk mitigations were devised. Two hundred and twenty (220) of these risk mitigations were technical controls to be established, verified and validated during this software validation process. Six of these risk mitigations involve developing and using standard operating procedures and user training. The 73 risks were prioritized using a partial Fibonacci sequence (0, 1, 2, 3, 5, 8, 13, 21, 34, 55). If there was a tie in priority for more than one risk then the AB Sciex software validation team evaluated the risk and ranked according to experience with the LC/MS system. After each risk was prioritized, all 73 risks were listed from highest priority to lowest priority. In order for these risks to be classified into the three categories recommended by GAMP 5.0, it was necessary to review the priority list and make a determination of the cut off for each category: high, medium and low. The main concern for categorizing risks as high priority was control of the audit trail. Thus any risk with priority greater than 16 was deemed a high priority risk. The break between medium and low priority had to do with detectability. Easily detectable risks were considered low priority. Thus, risks with priorities between -2 and 15 were deemed medium priority and risks with priority lower than -2 were considered low priority risks. Conclusion The potential impact of the LC/MS system to directly affect the integrity and reliability of pivotal data included in regulatory submissions is deemed to be critical based on GxP regulations and 21 CFR Part 11. Audit trail issues, missing critical functionality, data corruption, and security concerns pose the most risk. Organizations need to ensure that the assessment of the risks that may have an impact to their business and laboratory processes be evaluated properly, with careful consideration of the impacts at all touch-points. Detailed documentation of those decisions and the rationale behind them, should made available to interested parties, including regulators.

8 Fibonacci Sequences Improve GAMP 5 Risk Assessment for GLP Computerized Systems 8 Appendix A Example Risk Assessment with Proposed Risk Mitigation Table 4 (below) contains an example of the risk assessment conducted for Analyst 1.6.x software for the LC/MS system. The first two columns (Risk ID and Risk Scenario) identify the risk. The next four columns compute the priority of the risk as explained in this document. The last two columns identify the proposed risk mitigation. The Risk Mitigation Type is either the UFRS number (i.e. R.49) or the SOP reference. The Risk Mitigation column contains a description of the risk mitigation corresponding to the UFRS number or SOP reference. Sometimes a single risk has more than one risk mitigation. When this occurs the mitigations are listed under the risk and first six columns are blank. Table 4: Example of Risk Assessment Partial Fibonacci sequence including zero: (0, 1, 2, 3, 5, 8, 13, 21, 34, 55) Risk ID Risk Scenario Likelihood Impact Detectability Priority Risk Mitigation Type (UFRS # or SOP) Risk Mitigation (UFRS or SOP) K.15 K.14 The integrity of acquisition data files is not verified Electronic records can be deleted R R.45 R.46 The Analyst Software must be configured to record a checksum in raw data files. Users must not be permitted to delete electronic records (i.e., files) from the Analyst root directory. Users must not be permitted to delete subfolders from the Analyst root directory. R.47 Users must not be permitted to change permissions in the Analyst root directory. R.48 Users must not be permitted to take ownership of electronic records (i.e., files) in the Analyst network root directory AB SCIEX. The trademarks mentioned herein are the property of AB Sciex Pte. Ltd. or their respective owners. AB SCIEX is being used under license. IN NO EVENT SHALL ABSCIEX BE LIABLE, WHETHER IN CONTRACT, TORT, WARRANTY, OR UNDER ANY STATUTE OR ON ANY OTHER BASIS FOR SPECIAL, INCIDENTAL, INDIRECT, PUNITIVE, MULTIPLE OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH OR ARISING FROM ABSCIEX PRODUCTS AND SERVICES OR THE USE OF THIS DOCUMENT. Publication number Headquarters 353 Hatch Drive Foster City CA USA Phone International Sales For our office locations please call the division headquarters or refer to our website at