Whitepaper BMS and Process Historian Validation Strategy I N D U S T R Y I N S I G H T S Norman A. Goldschmidt Principal, VP Engineering www.geieng.com
I N D U S T R Y I N S I G H T S Genesis periodically publishes white papers and reports about topics of special interest to the industries we serve. As veteran advisors for major corporate infrastructure, energy management, facilities, technology, manufacturing and building systems of every type, our leaders share their perspectives to help both clients and the public at large make high value decisions by having the best available information. All information contained herein is copyrighted and cannot be reproduced without permission. For academic uses, please contact us. Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 2
Whitepaper BMS Validation Strategy Introduction Whenever room environmental parameters are critical to product quality Building Management System (BMS) designers face a choice regarding the control and recording of this GMP critical information... What to validate, and how? Regulatory and Guidance Background Regulators have been focused on the keeping of environmental records via electronic systems for over a decade. The table below indicates citations from as long ago as 1999 indicating the importance of environmental records (reprinted from Pharmaceutical Engineering 2005). The ISPE baseline guides have always stressed the need for validated systems to record critical environmental parameter for GMP use. However, they have also promoted a diversity of approaches to recording this data. The OSD guide suggests: "Instrumentation should be provided to monitor critical room parameters and alarms. It is possible to alarm with portable or other instrumentation, which is not part of the BMS system" Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 3
The Sterile guide goes further towards separation of control from monitoring: "It is the monitoring and documenting system that provide "GMP Critical Parameter" Data to production staff, hence these systems are direct impact and require qualification studies... It may be preferable that the monitoring and documenting of these "GMP Critical Parameters" should be isolated from any HVAC BMS control systems, to avoid qualification complications." A number of issues need to be considered when making the choice of approach to collecting and retaining GMP environmental data: Critical Environmental Parameters require validated monitoring Validated monitoring systems are expected to be compliant with 21CFR part 11, Annex 11, PIC/S, etc. Validated systems must be managed under change control to assure continued compliance (a state of control) Non-critical parameters in BMS systems typically outnumber the GMP Critical Parameters Frequent changes to non-critical parameters are needed to keep utility systems operating at peak efficiency Change control, applied to non-critical parameters result in inefficient operation and distract from critical issues It is impractical and cost prohibitive to qualify an entire large ( 1,000 + point) BMS Qualifying whole systems may complicate alarm management, which represents a regulatory risk. The approaches to address these issues are as numerous and diverse as the companies (and even the sites) that produce pharmaceutical products. Each of the potential approaches has its supporters and detractors, but in our professional opinion some of these approaches are superior due to their ease of implementation and robustness in maintaining a state of control. We can summarize the design approaches to these challenges into 3 basic categories, each with a couple of variations: 1. Single System with Validated Monitoring and Control (2 flavors) a. BMS b. Process Control System (PCS) 2. Partitioned Systems with Monitoring and Control, one Validated a. Physically Separate Systems b. Logically Separated (Firewalled) validated and un-validated 3. Partitioned Systems, Validated Monitoring (EMS) un-validated control (BMS) In the following sections we will describe these approaches and discuss some of the pros and cons of each. Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 4
Validated BMS Configuration Options 1. All BMS Validated HVAC Critical Data a. BMS Validated This solution is not preferred due to the level of resources required for simple maintenance changes, this approach may have been partially responsible for some noted 483's. This solution was pushed by BMS vendors in the late 1990's and early 2000's as they developed some expertise in validation. The vendor driven validations are notoriously weak in their linkage between critical process parameters and validation. This approach can choke the site change control system with unnecessary paperwork for non-critical changes, slow down maintenance response and increase the cost of ownership. The part 11 compliance is straightforward though voluminous. b. Process Controls for all BMS Validated This solution has many of the same flaws as the all BMS validated scenario. This solution has been promoted by Process Automation vendors in the 2000's as they developed some experience in HVAC control. The vendor driven validations are generally strong and the part 11 compliance well understood. Using this approach does sacrifice some of the base functionality of BMS systems. Process control vendors generally have less experience in the control of compressible fluids at low pressure (Airflow and room pressurization) and are not used to working with HVAC grade equipment. In addition one generally sacrifices the functionality that comes standard in a BMS such as: Night setback, optimum start/stop, Temperature Reset, Static Pressure Reset, Lighting Control, etc. Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 5
2. Partitioned HVAC CRITICAL DATA a. Partitioned / Separate BMS Systems This solution is superior to the all validated BMS as it employs risk assessment to segregate all critical BMS loops into a single system under quality change control, with a separate system under engineering change control only. This simplifies day to day maintenance, eases stress on the quality system (caused by excessive change control) and focuses the team on critical parameters. Part 11 compliance is fairly straightforward as the systems are entirely separate. There can be some additional stress on maintenance due to the different SOP's used to approach the validated BMS vs. the unvalidated BMS. b. Firewalled Parallel BMS Systems The parallel systems approach can be achieved by installing a firewall between sections of the BMS system, dedicating some controllers to GMP use, providing separate security control of access to that section and limiting data flow across the firewall. Part 11 compliance must be proved for any network hardware serving the validated portion of the system. Engineering change control for the system must consider the implication of changes across the firewall as part of any programming or hardware change. The Part 11 compliance is more complex as the validated and non-validated systems are in contact. Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 6
3. Separate Control From Monitoring Partitioned BMS/EMS Systems The third Parallel approach is becoming the most common, with two systems separating the control of all points from the monitoring of critical points. This approach has, arguably, the longest history of all approaches - first coming into use in the late 1980's when PLC's or Data loggers were used to record critical environmental data because BMS systems could not be validated. This approach creates a robust monitoring system to collect GMP critical data (this system can be part of the process control system as well) but leaves the actual control of all HVAC to the un- validated BMS. Part 11 compliance is fairly straightforward as the systems are entirely separate. There is little additional stress on maintenance due to the different types of systems requiring different SOP's. One complexity can be the choice of using dual redundant instruments (which can raise questions when they don't agree) or using signal repeaters (or dual output transmitters) which introduce an additional calibration, potential error and added cost (though much less expensive than additional instruments). Another alternative is allowing the EMS to repeat the signal, which introduces delay and adds risk to the communication network. Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 7
Selecting an Approach As early as 2005 the ISPE GAMP guidance suggested applying a risk approach to determine the best course for assuring that GMP environmental monitoring systems and records are managed and maintained. The flow chart and table below suggest the GAMP method for determining appropriate system configuration based on risk (reprinted from Pharmaceutical Engineering) Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 8
While this method is useful during a project, we believe it can be summarized when setting an approach for an organization or a site: 1. Are control loops mostly Critical - Validate the whole BMS 2. Are the GMP measurements few and all in one area - Physical Partition or BMS/EMS 3. Are the GMP measurements many and spread out? a. Is the quality of BMS system and maintenance high? - Logical Partition or BMS/EMS b. Is the quality of BMS system or maintenance standard HVAC quality? - EMS/BMS 4. Are there many BMS users or users not under owner control? - Physical Partition or BMS/EMS 5. Is the BMS accessible from outside the site? - BMS/EMS 6. Does the BMS vendor issue frequent updates or patches? - BMS/EMS Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 9
Conclusions Genesis sees the development of partitioned solutions, especially "Parallel BMS/EMS Systems" as the most prevalent design solution in our industry, due to the broad applicability. The relative simplicity of implementation and documentation "Parallel BMS/EMS Systems" and the ease of deploying the highest quality equipment for the most critical parameters makes this approach very attractive to our clients. The separation of monitoring from control assures independence of the readings, as an independent check on the BMS system, much like the way laboratory testing verifies the results of manufacturing operations to assure that a state of control is being maintained. Separation of the systems also provides the greatest protection from accidental unanticipated impact on validated records from seemingly benign changes to non-validated portions of the BMS system. This is particularly important when applying the frequent software updates and patches commonly provided by BMS system vendors. If these updates need to be applied to a validated system each of them must be assessed for impact to the validated state of the system. This additional testing work can lead to delays in applying updates and patches that will negatively impact un-validated systems. We see the implementation of independent parallel systems as being an expeditious and straightforward route to compliance in HVAC critical environmental record keeping, with independent BMS/EMS as a very viable choice for many pharmaceutical clients. Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 10
Things to Remember... Limit system access to authorized individuals and assure a hierarchy of authority Perform operational system checks Perform authority checks Perform device checks Assure that persons who develop, maintain, or use electronic systems have the education, training, and experience to perform their assigned tasks Assure establishment of, and adherence to, written policies that hold individuals accountable for actions initiated under their electronic signatures Assure appropriate control over systems documentation Even though if you print and review the data on a periodic basis, you are not within the scope of 21 CFR 11: 1. Under the narrow interpretation of the scope of part 11, with respect to records required to be maintained under predicate rules or submitted to FDA, when persons choose to use records in electronic format in place of paper format, part 11 would apply. On the other hand, when persons use computers to generate paper printouts of electronic records, and those paper records meet all the requirements of the applicable predicate rules and persons rely on the paper records to perform their regulated activities, FDA would generally not consider persons to be "using electronic records in lieu of paper records" under 11.2(a) and 11.2(b). In these instances, the use of computer systems in the generation of paper records would not trigger part 11. Copyright Genesis Engineers 2011 - All rights reserved - Do not reproduce without written permission. 11