VALIDATION OF MANUFACTURING EXECUTION SYSTEMS (MES)

Transcription

1 VALIDATION OF MANUFACTURING EXECUTION SYSTEMS (MES) A SYSTEMS DEVELOPMENT LIFECYCLE (SDLC) APPROACH 1 R AUL S OTO, MSC, CQE IVT ORLANDO 2017 The contents of this presentation represent the opinion of the speaker; and not necessarily that of his present or past employers. 2

2 ABOUT THE AUTHOR 20+ years of experience in the medical devices, pharmaceutical, biotechnology, and consumer electronic industries MS Biotechnology, emphasis in Biomedical Engineering BS Mechanical Engineering ASQ Certified Quality Engineer I have led validation / qualification efforts in many scenarios: High-speed, high-volume manufacturing and packaging equipment Enterprise resource planning applications (i.e. SAP) IT network & infrastructure Quality Systems Software (CAPA / NCs; pharmacovigilance; edms; elms; etc.) Mobile applications Laboratory : information systems and equipment / computerized analytical instruments Product improvements, material changes, vendor changes Contact information: Raul Soto rsoto21@gmail.com 3 4

3 OUTLINE Introduction What is an MES Benefits ANSI / ISA 95 edhr Scope: What should be included in the Validation project SDLC Approach Description and deliverables Validation Deliverables Planning and Strategy Requirements, Design, Trace Matrix Testing Reports, Procedures, and Governance Interfaces Going Live Governance Hypercare 5 INTRODUCTION MANUFACTURING EXECUTION SYSTEMS 6

4 WHAT IS AN MES Manufacturing Execution System Takes in vast quantities of data coming from the PLCs and SCADA/HMI systems Converts that into useful information about production ops Scheduling Materials handling Quality samples 7 ERP MES CONTROLS 8

5 ANSI / ISA 95 CONTROL HIERARCHY LEVELS Level 0: Physical production process Level 1: Sensing/manipulation of production process Level 2: Monitoring, supervisory control and automated control of production process (i.e. SCADA HMI systems) Level 3: Workflow / recipe control to produce desired products, maintain records and optimize production process. Timeframe: shifts, hours, minutes, seconds (i.e. MES, LIMS) Level 4: Establish basic plant schedule production, material use, delivery, shipping. Timeframe: month, weeks, days, shifts (i.e. ERP) 9 ANSI / ISA 95 CONTROL HIERARCHY LEVELS IVT December (c) 2017 Raul Soto 10

6 BENEFITS OF MES Provide complete history of manufactured lots/ batches/ units Electronic Device History Record / Batch Record Electronic approvals and audit trails Full traceability of product to raw materials and manufacturing process parameters at every stage Reduce use of paper 11 BENEFITS OF MES MES system is not just a piece of fancy technology The BUSINESS VALUE of an MES comes from the integration of data from multiple information sources: production data, quality data, business data, controls data, etc. Materials consumed, equipment used, parametric data, exceptions, date / time stamps Enforces as-designed process Ensures the right procedures, operators, equipment materials are used Enforces right sequence of operations Can track equipment usage and communicate with ecmms 12

7 EDHR edhr: Electronic Device History Records paperless, electronic systems within MES enforce production processes capture all information associated with as-built production records. provides the error-proofing and real-time visibility necessary to produce consistent product quality each and every time. 13 EDHR If product quality issues arise, edhr can be used to contain suspect product, either in process or in the field, to take action to address the issue. The electronic records stored in edhr yields information that can also help MD&D companies optimize production processes. 14

8 15 SCOPE : WHAT SHOULD BE INCLUDED IN THE VALIDATION PROJECT MANUFACTURING EXECUTION SYSTEMS 16

9 SOFTWARE VALIDATION validation, software. (NBS) Determination of the correctness of the final program or software produced from a development project with respect to the user needs and requirements. Validation is usually accomplished by verifying each stage of the software development life cycle. See: verification, software. Source: FDA Glossary of Computer System Software Development Terminology 17 SCOPE: SOFTWARE MES Application Workflows ETL tools SQL scripts Databases / Data warehouse (i.e. Oracle) Middleware (i.e. Archestra) Interfaces (more on this later!) Reporting tools not part of the application Reports created with tools like Tableau, Cognos, BusinessObjects Other ancillary / support software 18

10 SCOPE: HARDWARE Servers (for all environments) Sandbox / Development / QA-Test / Staging / Production Scanners Printers Bar code readers RFID systems others 19 SCOPE One or multiple sites? One or multiple manufacturing suites / lines (within each site)? Are you also decommissioning a legacy MES system? Data migration and /or archival GAMP 5: COTS vs Configuration vs Customization Medical Devices : edhr 21 CFR DEVICE HISTORY RECORD 20

11 SCOPE Define your project scope early on, document all changes Map your AS-IS business process workflows (pre-mes), vs your TO-BE Include in scope any configuration and/or customizations required (i.e. reports, interfaces) Recommendation: Minimize custom-coding, maximize the use of configuration alternatives provided by software. 21 PROJECT PLANNING Involve key stakeholders and end users from the start Testing, validation, user training and documentation require time and resources. Make sure you factor that into your project scope from the start. 22

12 SDLC APPROACH MANUFACTURING EXECUTION SYSTEMS 23 WHAT IS AN SDLC SDLC : Systems Development Life Cycle Series of steps / phases that provide a model for development and lifecycle management of an application or piece of software. Software validation is NOT a one-shot deal. Validated software and its documentation require continuous updates and improvements for the life of the system End-of-life has to be managed Data migration or archival Hardware decommissioning 24

13 SDLC APPROACH IVT December (c) 2017 Raul Soto 25 TYPICAL VALIDATION DELIVERABLES Validation Plan Development PQ / UAT Assessments SAT / FAT Traceability Matrix Requirements IQ Procedures Design OQ / System Testing Validation Report 26

14 PURCHASING REQUIREMENTS NOT the same as your validation requirements Develop high level requirements to be used as part of the purchasing decision Vendor assessment should include compliance with these requirements Make sure you have a good idea of how much configuration and/or customization is needed to meet your main requirements MINIMIZE the amount of customization / custom code These requirements will also be the basis for FAT / SAT SLA : Service Level Agreement 27 DEVELOPMENT / CONFIGURATION Development testing occurs before validation COTS functionality + configuration Vendor or hired consultants Unit testing, test configuration, integration testing If custom coding is required, development testing should be more comprehensive Use bug tracking software (i.e. FogBugz, Redmine, Debbugs) to keep track of bug resolution Does not need to be formally documented, but IT HELPS Have Coding standards and follow them! 28

15 VALIDATION DELIVERABLES MANUFACTURING EXECUTION SYSTEMS 29 TYPICAL LIST OF VALIDATION DELIVERABLES Assessments (Safety, 21CFR Part 11, Risk, Compliance, Vendor etc.) Validation Plan Requirements (URS, FRS) Design (DS, FDS, TDS, DDS) Traceability Matrix Testing protocols and scripts Development * Integration * FAT / SAT * Installation OQ / PQ or System Testing / UAT Testing Reports Training Procedures Operational Administration Change Control Validation Report * Do not fall strictly under validation IVT December (c) 2017 Raul Soto 30

16 31 ASSESSMENTS Vendor Assessment Compliance Assessment (i.e. GxP, SOX, etc.) 21 CFR Part 11 ER/ES (especially on Medical Devices) GAMP category Information Security Safety / EHS Risk Analysis / FMEA others 32

17 VALIDATION PLAN & REPORT IVT December (c) 2017 Raul Soto 33 VALIDATION PLAN This is where you establish WHAT needs to be validated, and HOW it will be validated. Get everyone s buy-in and commitment to the Validation Plan from the beginning Update as needed, following your organization s formal change control process for documentation Define clear expectations for all roles Definitions of deliverables, author and approvers for each 34

18 VALIDATION PLAN System Overview Scope of Validation Roles and Responsibilities Assessments Required Validation Deliverables, author and approvers Validation and Testing Strategy Acceptance Criteria How to deal with Testing defects and deviations changes in scope and strategy 35 VALIDATION REPORT Final scope List of Actual Deliverables Document Numbers Date of Completion any additions or deviations from planned Summary of testing results per protocol; any additions or deviations Summary of defects and discrepancies, and resolution status Statement declaring the validated state of the system Release to Production 36

19 RESOURCES You will need resources from multiple disciplines (i.e. QA, IT, Engineering, Operations, technical writers, etc.) Plan which resources you need, when will you need them, and for how long This will help you budget for consultants, and borrow internal resources. I need two validation resources, 25-30% of their time, during August and September is a different conversation than I need to borrow some of your people for the MES project PM tools -such as MS Project- help 37 REQUIREMENTS WHAT do you want your MES system to do? Types: Classify by type and criticality User Requirements Functional Requirements Regulatory Requirements Performance Requirements Security Requirements Low criticality : nice to have Medium criticality: important but not critical High criticality: must have All regulatory requirements should be classified as HIGH criticality 38

20 REQUIREMENTS Requirements should be Clear Complete Correct Consistent Concise Prioritized Relevant Feasible Verifiable Modifiable Traceable Unique 39 REQUIREMENTS Written definition of the MES software functions: Proposed Workflows and Use Cases How users will interact with the system Functions that the MES software will perform Expected system inputs and outputs Reports the system is expected to generate, contents Interfaces (user, external) Required alarms and checks Performance: data throughput, reliability, timing Coding standards 40

21 REQUIREMENTS Required response times Intended operating environment (hardware platform, software operating system, middleware, database software, etc.) All ranges, limits, defaults, specific values that the MES software requires Mirroring & Replication of main and transactional databases Backup, restore, disaster recovery User Roles that need to be defined in the system Any safety-related specifications Part 11: Electronic Records & Signatures requirements Dates (Manufacturing, Expiration): based on local time or UTC? 41 REQUIREMENTS: UDI Unique Device Identification US Initiative to assign a unique identifier to all medical devices Signed into law on 2007, mandatory since 2013 FDA Final Rule Label of a medical device must have a unique identifier Unique identifier should ID device through distribution and use Unique identifier must include lot / serial number Typical elements: Graphics and symbols Regular and 2D barcodes Lot / batch / serial number, expiration date, date of manufacture 42

22 43 DESIGN HOW does your MES as installed and configured, meet the Requirements? Design Specifications describe the actual software solution: Out-of-box / configured / custom-coded Address all requirements, individually or in groups Design documentation may include one or more of the following: Design Specification Detailed Design Specification Technical Design Specification Database Design Specification Software Design Specification Hardware Design Specification Architecture Design Security Design Interfaces 44

23 DESIGN Design documentation should include, at least: Actual workflows and use cases, as configured (or coded) System architecture Modules Screens Formulas, algorithms, and logic used Data structures, data flow diagrams Supporting software that is required for MES operation (i.e. Archestra) Hardware required Parameters that are measured or recorded 45 DESIGN Definition of control and data variables, where used Messages: Errors, alarms, warnings User roles, as configured Physical security Information security Actual reports as configured Interfaces, as configured 46

24 REQUIREMENTS TRACEABILITY MATRIX The RTM should enable us to trace: From individual Requirements to their specific Design Element(s) where the requirement is addressed From Design Elements to the specific Test Script(s) where they are being challenged. We should be able to trace back and forward Requirement Design elements Testing 47 REQUIREMENTS TRACEABILITY MATRIX Requirements Design documents Test scripts => start RTM draft => update RTM draft => finalize RTM Final RTM should be approved before the system is released for production RTM should be updated as part of system change control every time requirements, design, or test scripts are changed, added, removed. 48

25 LIVING DOCUMENTS The following documents should be treated as living documents and maintained up to date throughout the life of the system Requirements Design Traceability Matrix Risk Assessments (i.e. FMEA) 49 TESTING MANUFACTURING EXECUTION SYSTEMS 50

26 TESTING Protocols, Test Scripts, Reports Pre- and post- approved Consistency in roles pre- and post- approving is important Minimum: System Owner, Business Process Owner, QA Avoid conflicts of interest: Approvers cannot execute, testers can t approve Dependencies and order of testing must be clear, evident, obvious Enforce Good Documentation Practices (GDP) Enforce the use of standard templates Keep evidence (printouts, screenshots, labels, etc.) 51 TESTING ENVIRONMENTS Development / Sandbox For DEV testing Qualification not required Test / Staging For your formal validation (OQ, System Testing, UAT) Qualified Functionally equivalent to your PROD environment Production Environment Live system Performance testing Qualified 52

27 INSTALLATION QUALIFICATION Installation Qualification Document / verify the correct installation and configuration of all software and hardware components, as per the Design Specification List actual software components and objects installed: name, version, location List actual hardware installed: name, model, quantity, S/N, location Turn-key test to ensure the system is ready for OQ / System testing 53 OPERATIONAL QUALIFICATION Operational Qualification Ensure system can perform under controlled conditions, nonsaleable product Document that system is installed and configured as per Design Specifications AND complies with requirements as per Requirements Specifications Testing will challenge all requirements, interfaces, reports, etc. Can test each subsystem individually Positive vs Negative testing Formal change control 54

28 OPERATIONAL QUALIFICATION System Testing Test and verify that the system as integrated is functionally complete Challenge compliance with functional and non-functional specifications Integration of MES with interfaces You can test individual functionalities, then end-end process workflows 55 OPERATIONAL QUALIFICATION Risk-based testing: 1. COTS elements: positive testing of all workflows, and alternate flows 2. Configured elements: Positive testing Negative testing Boundary testing 3. Custom: Same as Configured elements, PLUS Code reviews against coding standards IVT December (c) 2017 Raul Soto 56

29 OPERATIONAL QUALIFICATION Tests functional requirements, configuration, security, ER/ES, compliance May include positive testing, negative testing, boundary testing, interface testing Positive Testing: Ensures that system performs as intended, using normally expected inputs Negative Testing: Ensures that system doesn t accept invalid inputs Boundary Testing: Challenges that performance is as expected when specific variables are set to their max / min values. Interface Testing: Tests that system components can pass data correctly to one another 57 PERFORMANCE QUALIFICATION Performance Qualification / Production Qualification Test that system is able to function under normal manufacturing conditions End-to-End challenge of the complete process flow Normal mix of lots, products, operators, shifts, etc. Performance testing / load testing SOPs must be approved and effective Can use saleable product if company procedures allow for it 58

30 PERFORMANCE QUALIFICATION User Acceptance Testing Testing focused on user-related requirements Challenge that the MES system is capable of supporting your normal business process Test cases should address ease of use from the standpoint of operators, QC technicians, etc. 59 PERFORMANCE QUALIFICATION Tests functional requirements, configuration, security, ER/ES, compliance May include positive & negative testing, business process testing, end-to-end testing, stress testing, performance testing Business Process Testing: Verifies that the system works as intended following business process flows End to End Testing: Verifies that the system is capable of supporting the intended process flows, from beginning to end; ensure data integrity and that the correct data passes between components and interfaces. Performance Testing: Verifies system stability, resource usage, and responsiveness under specific workloads. Stress Testing: Tests system s performance beyond the limits established in the specified requirements 60

31 TESTING EXECUTION Ensure test sets and scripts are executed in correct order GDP (Good Documentation Practices) are IMPORTANT Include evidence: screenshots, printouts, labels, etc. Changes to approved protocols / test scripts must follow formal documentation management (version up, re-approve, etc) Coordinate execution dates / times with owners of systems MES interfaces with Follow process for handling testing defects / deviations If you execute PQ / UAT with real product, QA must provide disposition of such product 61 TESTING TOOLS There are electronic testing tools that can be used in substitution of paper test scripts Example: HP Applications Life Management (ALM), Valgenesys MUST validate these tools BEFORE you use them Pros: reduce GDP errors, standardize testing process, enforce use of correct templates, enforce approvers rules, manage test defects, can search for documents electronically Cons: administration, training, cost, less flexibility, maintenance, fixes / upgrades Tools become your official repository of validation documentation 21 CFR Part 11 full compliance may require additional wrap-around software solution 62

32 PROJECT CHANGE MANAGEMENT You must have a formal change control process while in the project, not just after go-live. Design freeze date enforce it! Any design changes requested after this date => after Go Live Except for critical (regulatory, business) changes 63 CHANGE CONTROL All changes should be assessed, documented, pre-approved, tested, post-approved Assessment Nature and scope of the change Risk assessment Documentation that needs to be updated (Requirements, Design, RTM, FMEA, etc.) Testing Functional tests Regression tests 64

33 INTERFACES MANUFACTURING EXECUTION SYSTEMS 65 INTERFACES Do NOT underestimate interfaces, especially to LIMS and your ERP. ALL interfaces must be part of the validation effort Define and document all your interfaces at the Design stage What data passes MES External system Test your interfaces configuration in Development testing Test your interfaces functionality explicitly in OQ / System Testing 66

34 SAMPLE INTERFACES DIAGRAM Global Planning ERP Lot production quantity and status Raw Materials Tracking SCADA / HMI MES Distribution Control Data Lot Info Audit Data PLCs LIMS Change Control 67 POSSIBLE INTERFACES ERP (i.e. SAP, JD Edwards) Master production schedule Lot quantity and status Bill of materials LIMS (i.e. Labware, NuGenesis) QC Audit data Manufacturing lines, SCADA / HMI (i.e. Wonderware, RSView) Manufacturing orders Raw Materials Tracking software Data exchange of r.m. lots used Documentation edms (i.e. Documentum) Change control and procedures data Product hold information Distribution Control system QC release data Lot data Bill of materials Lot data 68

35 GOING LIVE MANUFACTURING EXECUTION SYSTEMS 69 GOVERNANCE Procedures System Administration Operation Change Control Maintenance Training Security and Access Control Configuration Management Backup / Restore / Disaster Recovery others 70

36 GOVERNANCE Hypercare SLA with vendors Change Control Support structure Periodic Review (and revalidation) 71 REFERENCES Information/Guidances/ucm htm 72

37 REFERENCES IVT December (c) 2017 Raul Soto 73 REFERENCES General Principles of Software Validation; Final Guidance for Industry and FDA Staff, Jan ISPE GAMP5 Guide: Manufacturing Execution Systems Understanding Manufacturing Execution Systems (MES) Unique Device Identification (UDI) Global Unique Device Identification Database (GUDID) Guidance for Industry and FDA Staff, June pdf 74

38 REFERENCES US 21 CFR 820 Quality System Regulation US 21 CFR 11 Electronic Records and Signatures Annex 11 Computerised Systems. Jan 2011 update EudraLex, Rules Governing Medicinal Products in the European Union Vol. 4 Good Manufacturing Practice, Medicinal Products for Human and Veterinary Use ISO Medical devices -- Quality management systems -- Requirements for regulatory purposes ANSI/ISA 95 and ANSI/ISA 88 Standards 75 QUESTIONS 76