IPC2000-254 DEVELOPMENT AND USE OF SIMULATION TRAINERS FOR PIPELINE CONTROLLERS D.M. Scott Enbridge Pipe Line Company Edmonton, Alberta, Canada ABSTRACT Enbridge is in the forefront of development and application of computer simulation based training systems for Pipeline Controllers. Since 1985, the Pipeline Dynamics section of Enbridge Engineering has been building ever more sophisticated models of the 16 pipelines that Enbridge owns and operates. Enbridge has also utilized its staffs expertise to develop simulation trainer applications for clients in other international pipeline companies. Enbridge uses this simulation trainer with a formalized, module based, training course for Pipeline Controllers. To develop a useful simulation trainer it must be possible to mimic the operational characteristics of a pipeline through use of software, or put otherwise, to build a pipeline in a computer. This type of modeling requires a highly skilled engineer and close cooperation with the most experienced Pipeline Controller and with the SCADA system developers. The simulation software must also allow the application of control logic which is different for each pipeline and it must be tunable so the nuances of each pipeline can be incorporated. Running this type of application requires a high capacity computer and the ability to interface with the pipeline company's SCADA. In simple applications, it is possible to develop a generic pipeline and thus provide a more economical trainer. The simulation trainer's primary application is Pipeline Controller training, however it is Enbridge's experience that a simulation can also be used to test changes to a SCADA system before they are implemented and to operate a new pipeline before it goes into operation. The information gained from the pipeline operation prior to its construction can benefit the engineering design. This paper outlines the Enbridge experience with development and use of pipeline simulation trainers. The paper also discusses how the simulation trainer fits with formal training, testing and retraining of Pipeline Controllers, how it has been used for SCADA system testing and for operating a new pipeline before it's construction. The paper outlines what is required for the development of generic and specific simulations that are interfaced to SCADA, the use of modeling standards in the development, and the process of tuning a model so it 'looks and feels' like a real pipeline. 1. INTRODUCTION Enbridge Pipelines has developed and is using simulation trainers for enhancing the skills of the company's Pipeline Controllers. Enbridge's first simulation trainer system was developed in 1985. Since then, Enbridge has been developing and implementing ever more sophisticated simulation trainers. In the past few years, the use of the simulation trainer has been combined with use of written and multimedia learning modules. This approach puts Enbridge in the forefront of Pipeline Controller training. Enbridge has already self complied with the needs of the recent US-DOT Pipeline Controller qualification requirement. Enbridge owns and operates 16 distinct liquids pipelines, which are controlled by four Control Centres with 49 Copyright 2000 by ASME
Pipeline Controllers. The Enbridge Control Centre management structure includes the use of a Senior Pipeline Controller who provides supervision and guidance in the Control Centre A simulation trainer is optimal if it can mimic the unique operational characteristics of any pipeline. It can essentially be a 'pipeline in a computer' through use the of a software model and an interface to a full SCADA system. Although it is possible to create a generic Model Computer SCADA Figure 1: Simulation Trainer Control Console pipeline trainer that can demonstrate certain operational aspects of pipelines, this has not been the Enbridge approach. At Enbridge it is our practice to create exact models of all our pipelines and thus to allow an entry level Pipeline Controller to operate a pipeline that is as close as possible to the actual pipeline. Additionally, this provides the experienced Pipeline Controller with the most useful training experience on new and existing pipelines. The development of simulation trainers requires a cooperative effort amongst three distinct areas of the company: Pipeline Dynamics (an Engineering area); SCADA Services (an I.T. area) and the Control Centre (an Operations area). Enbridge has also utilized its simulation trainer development expertise to provide simulation trainers for clients at other international pipeline companies. At present, there is in development a natural gas pipeline trainer for use by another of the Enbridge companies. Although Enbridge does not use generic pipeline trainers, Enbridge staff have developed generic trainers for others to be used for basic level Pipeline Controller trainer or as a demonstrator of pipeline hydraulics. The following sections of this paper provide an outline of what constitutes a pipeline simulation trainer, how it is built, how it is used in our training program and some of the difficulties that may be encountered or 'learning opportunities' in developing and implementing simulation trainers. 2. WHAT IS A PIPELINE SIMULATION TRAINER? The simulation trainer is an interactive hydraulic model that is linked to a SCADA system. The basic components of a pipeline simulation trainer are: a hydraulic model of the pipeline; a SCADA system; interfacing software from the SCADA to the hydraulic model; and the hardware that is necessary to run the simulation trainer. These components are shown in Figure 1 and described in the following sections. 2.1 The pipeline hydraulic model. An interactive hydraulic model for a simulation trainer can be purchased from a software vendor then configured by the customer or the vendor to meet the pipeline company's simulation trainer needs. In an interactive hydraulic model, the ability to model active components and control commands and to 'poke' or turn on and off components or vary set points is the primary difference between a simulation trainer model and an engineering model. Enbridge has developed the expertise to 'build' trainers so all our work is done 'in-house'. We did not consider attempting to develop the modeling software itself, as we recognized that the implementation work by itself was a large enough burden without trying to develop the basic hydraulic model. 2.2 Trainer SCADA. For a trainer representing a real pipeline, it is necessary to have an actual copy of the SCADA system (the same SCADA that is used to control that pipeline). All the commands and data points of the real SCADA will then need to be in the model, unless modifications are made to simplify the trainer SCADA. Simplifications, if used, must be well thought out or the trainer will not be representative of the real pipeline. 2.3 Interfacing files. Shared files are used to allow the SCADA to 'speak' to the model. Because SCADA is designed to accept communication data from the field, the data must be presented to SCADA as if it were coming from the actual pipeline. The interfacing routines handle these functions. 2.3 Hardware and Facilities. The hardware needed for a simulation trainer includes the computer(s) to run the software (model, SCADA and interfacing files), SCADA output displays. The facility needed is a Controller training area. The selected software and the trainer complexity will determine the computing power and operating system required. Enbridge uses a UNIX
operating system running on workstations. Since simulation trainers are not critical applications, some companies are finding that the Windows based products running on desktop computers are a satisfactory alternative. The training area for a simulation trainer that mimics a real pipeline needs all the screens and input devices of a real control room, but does not need to have full Control Centre consoles. For a generic trainer the facility requirements will be less. In some case a complete generic trainer may run entirely on a laptop computer. 3. BUILDING A SIMULATION TRAINER A simulation trainer must be able to mimic, as closely as possible, the operational characteristics of a specific pipeline. For initial 'in-house' development, the engineer will require extensive training by the software vendor in model building. A pipeline company that wants to develop trainers for only a few pipelines may find the building of a simulation trainer an overwhelming task. It may be more effective to consider using a 'canned' trainer (generic model) or outsourcing model development The software vendor may be able to do this. The pipeline company's input would then be limited to providing specifics of the pipelines being modeled. The following sections outline the trainer implementation tasks, which are: defining and building the system; testing and tuning the trainer system; building the desired training scenarios; and managing the implementation process. This section's discussion will focus on developing simulation trainers for specific pipelines. A similar but less complex process would be used for implementing generic simulation trainers. 3.1 Defining and Building the trainer system. Enbridge's hydraulic models are built using a well known software product that provides one of the most sophisticated hydraulic simulations available. All significant pipeline details are necessary as input details for this pipeline specific simulation trainer. These include the physical components of the pipeline (e.g. line description, pumps, valves), the types and important characteristics of fluids shipped (e.g. name, density, viscosity), and pipeline operations logic (e.g. for pumps, block valves and pressure control valves). Defining the pipeline model requires substantial data gathering. Data sources include pipeline schematics, valve and pump characteristics, start-up and changes control logic sequences and the range of operating values (e.g. pressure and flows). Specific details of each physical component are then required. For example, the line description for a specific pipe includes the pipe diameter, wall thickness, material type, and elevation of the pipe along the right-of-way. We were able to use line description information that was common to our engineering simulator. Also, many of the basic details were already available electronically. This information is entered into the hydraulic model according to software defined formats. Ali physical model components are linked together at connecting "nodes". The software will diagnose any data entry and design errors such as unlinked nodes. The model does not need actual instrumented data as these are the normal model calculated outputs. However, the location of instrumentation is needed so the model output values can be displayed at appropriate locations. If it is intended to build a number of trainer models then the use of modeling standards is highly recommended. Standards can be used to define common approaches for model methods and nomenclature. These are especially valuable for future model maintenance and support. It is also necessary to build a number of model output displays for Controller information and operation. Even though a trainee uses the existing SCADA screens, it may be necessary to have additional displays comparing modeled to SCADA values. This is to confirm that actual SCADA and modeled values are consistent. The SCADA system used in the simulation trainer is identical to the SCADA system that operates the real pipeline. All of the displays and functionality of the production SCADA system can be utilized in the trainer simulation. The outputs from the model should provide the same inputs as the SCADA system would receive from the pipeline. Unless the production SCADA system is substantially altered, it may not be necessary to upgrade the trainer SCADA. Annual review and changes may be adequate. Maintaining simulator trainers is an ongoing requirement and expense. The interfacing files are necessary to allow the hydraulic model to 'talk' to the SCADA system. These two files are a 'switch" file and a "cross-reference" file. The switch file associates a SCADA data point name with a model name. The building of the switch file is straightforward and simple. It involves setting the model datapoint identifier name to the name expected by the SCADA system. For example a SCADA name for a pressure transmitter may be KB_1_PT_1S (a suction pressure transmitter at a station with call letters KB) but the switch file would relate that to a model name that would be KB P SU PT1S.
The cross-reference file provides data from the model to the SCADA in the order that a production system expects to receive it. Creating this cross-reference file is more complex than creating the switch file. The trainer simulator model outputs must be modified to fit the SCADA needs. RTU reports are used to define the way SCADA needs its data delivered. The hardware and facilities includes the computers monitors and the Controller training area. The trainer simulator may require special consideration depending upon the company's' computing environment. For example, if the SCADA runs on UNIX and the simulator model on Windows in NT, it will be necessary to have network access between the computers. Enbridge has determined that a dual processor workstation or two separate computers are necessary for its simulation trainer application. In addition, Enbridge has an additional separate computer system in the training area so the Training Coordinator can provide real-time monitoring of the training session. For a simple SCADA and model, one computer with multi-tasking capability may suffice. The simulation trainer is now fully functional but is missing the next two key steps for making it an effective training tool. 3.2 Systems Testing and Tuning. This step requires the model developer to work closely with a senior Pipeline Controller (or the Training Coordinator) who knows the modeled pipeline. This senior Pipeline Controller will provide initial values for the trainer that are appropriate for the particular pipeline (e.g. initial pressure set points). The Pipeline Controller can then run the new simulation trainer and can determine if it is truly representative of the real pipeline. He or she will help to determine how accurately the simulated operation follows the expected pipeline behaviour. Typically this will focus on the performance of active components such as pump starts and stops and valve changes. This phase or work requires that the trainer will allow changes to control logic and is "tunable" to the unique patterns of each pipeline. Changing timing variables is frequently the tuning technique that works best. The Pipeline Controller may also feel that the simulation is running too slowly compared to the expected behaviour. In that case, the model may need to be simplified to reduce non critical calculations and run faster. Finally the Pipeline Controller may request additional features to enhance the training capabilities of the simulator. The simulation trainer is now tested and tuned to the real pipeline but is missing the last step, the training scenarios. 3.3 Developing Training Scenarios. This is work that must be implemented by the Training Coordinator or by the Modeling Engineer under the direction of the Pipeline Controller. Training scenarios are the situations that the trainee will experience while he operates the trainer. The types of scenarios are outlined below. The scenarios are standardized so all trainee's experience the same events. These help with evaluation of the entry level Pipeline Controller's ability to meet any performance metrics that have been established. 3.4 Resources Required. Three staff roles are required for the simulation trainer implementation: a Modeling Engineer; a SCADA development expert; and a Training Coordinator. In Enbridge's implementation, a Senior Pipeline Controller will often act in the Training Coordinator role. Enbridge is fortunate in that we have developed a number of simulation trainers and have a number of years experience in this work. This allows us the capability for rapid and efficient trainer modifications. As an example of resources and timing needs, the following was required for a new six station 500 mile pipeline, the work required: - 2 person-months of modeling time (including the SCADA installation) - 2 person-weeks of rough tuning and testing - 3 weeks for two staff (6 person weeks total) of work (Training Coordinator and Modeling Engineer) for scenario development. In addition, the model software vendor usually provides one major software upgrade per year. Installation of the upgrade and related work take about two to four personweeks effort per pipeline. An annual SCADA install will take about one person-week effort. If the new software version of the model has bugs that affect the application, some additional work may be necessary. Other staffing costs include model verification before each use. This is estimated to be at about one person-day per pipeline modeled.
ELEVATIONS PUMPs\\ VALVES FLUIDS The remainder of this section will discuss details only for Pipeline Controller training. SCADA SYSTEM INTERFACING FILES TUNING & TESTING v => Figure 3: Simulation Trainer Inputs SIMULATION TRAINER OK, so you've got a trainer built and ready to go. can it be used?? 4. APPLICATION OF THE TRAINER How Enbridge has determined four general uses for simulation trainers: - for training new and retraining existing Pipeline Controllers. - to allow the Pipeline Controllers to train on a new pipeline before it becomes operational. - to allow the Engineering pipeline designers to try proposed changes or new approaches or equipment before a design is finalized. The information gained from the pipeline operation prior to its construction can benefit the engineering design - to allow SCADA system developers to interact with the simulated (modeled) pipeline to test what effects proposed new control schemes will have on the operation of a pipeline. Before proceeding with trainer use, it is necessary that the trainee have pipeline fundamentals knowledge that is at a level where he or she can benefit from hands-on training. The use of self-study or formal classroom training is necessary to bring a new employee up to that required knowledge level. Enbridge has developed 80 self-study modules that cover a complete Pipeline Controller training program. For the Enbridge training program the Basic and Core knowledge modules are used. These self study modules are administered in eight phases with the titles: Introduction to Fluid Behaviour; Pipeline Equipment; Graphical System Representation; Pipeline Control Systems; Advanced Fluid Behaviour; Batchtracking and Scheduling; Basis Operating Techniques; and Advanced Operating Techniques. A new trainee would not be operating the pipeline until they have completed a number of the phases. However, the simulation trainer may be used to demonstrate some of the information of the modules such as fluid behaviour in a pipeline.
Also, at least for the first few sessions, it will be necessary to have a designated Training Coordinator to instruct the trainee in getting started and in running the simulation. At Enbridge, we have a full time Training Coordinator for all training sessions at the major Control Centre and a designated trainer contact person at others. This provides a single point of contact for the developers and for the entry level Pipeline Controllers. Enbridge has a separate room for the simulation trainer facility. This room also has a set-up for the Trainer Coordinator so he or she can monitor the training session and provide help as necessary. The Enbridge applications for Pipeline Controller training are: 4.1 Training New Pipeline Controllers. As mentioned above, the entry level Pipeline Controller must have sufficient pipeline knowledge to get value from operating a simulation trainer. The best use of the trainer is in conjunction with formal training or self study training, interspersing simulation training with 'book learning' All trainees have to be able to perform the same tasks operating through the standardized scenarios and are tested on how well they are able to handle the work. There are seven standard scenarios: - pipeline start-up - pipeline shut down - a full stream delivery - a full stream injection - a side stream delivery - a side stream injection - a simultaneous delivery and injection. - operations when data loss occurs - operation when a leak occurs - alarms during operation an emergency shutdown a line obstructions (for example closed valves) various transient states (e.g. column separation, station lock out) other circumstances that may be pertinent to the particular line 4.2 Testing Skills/Upgrading Existing Pipeline Controllers. Fully qualified Pipeline Controllers must also use the Enbridge simulation trainer. They are required to annually re-qualify on the items outlined above. Also, if an experienced Pipeline Controller is to be transferred to operating a pipeline he or she has never before operated, the simulation trainer can be used to familiarize the Pipeline Controller with the pipeline before that person operates the line. The trainer can also be used to ensure Pipeline Controllers are familiar with unusually encountered types of pipeline operations or operating patterns. An example is operating a looped pipeline in pure mode when set up for pigging. There may be particular pipeline circumstances that occur infrequently and are not experienced by all staff. The simulation trainer provides an opportunity to maintain skills for those circumstances. 4.3 Other Uses. One new use of the simulation trainer is to teach Pipeline Controllers new operating methods that may be more energy efficient. This too allows an opportunity to standardize operation. All training scenarios are standardized so all Pipeline Controllers encounter a similar level of difficulty. TRAINING MODULES TRAINEE SIMULATION TRAINER Figure 4: Training Process Then there are a number of non-routine situations also modeled: 5. 'LEARNING ISSUES
As with any work, there can be a few surprises during the development and use of a simulation trainer. The intent of this section is to provide useful information for those who decide to undertake the development and implementation of a simulation trainer. 5.1 To achieve the "look and feel" of the operation of the real pipeline it may be necessary to model logic differently than it is in the real world. For most pipelines this is 'cut and try' work and will be finalized with the senior Pipeline Controller during the testing and tuning phase of the development. This is particularly true for items such as variable speed pumps using electric VFD drives or for pumps that are driven by a prime mover. The start-up and shut down logic of a prime mover driven pump can be especially complex and difficult to model. There may even be some technical issues in the model's ability to mimic real control, so the simulation trainer may have to settle for the "best fit" state to the real pipeline. 5.2 Infrequently used trainers may need to be tested each time before they are used. Reasons for this include the following: the computer upon which the simulation trainer is installed may be used for other purposes when the trainer is not in use; the users may need a brief refresher in start-up and operation of the programs; and new features or upgrades that have been developed may need to be installed. So it is necessary for the users to provide some advance warning of the simulation trainer use so developers can make sure the simulation trainer is fully functional before the training sessions. 5.3 SCADA changes or upgrades that are implemented for the control of the pipeline will have to be eventually implemented in the simulation trainer. Certainly any significant SCADA changes will soon need to be in the simulation trainer so the entry level Pipeline Controller is learning on the same pipeline as he or she will be operating. In any case, the pipeline company should plan staffing and budget resources to accommodate the necessary work. It may not be advisable to try to maintain a separate simulation trainer version of the SCADA. This could be costly and confusing. 5.4 If a pipeline is to undergo a change of service or new construction, the simulation trainer will need to be modified. The trainer developers should be included when changes are considered so the model modifications can be accommodated to the simulation trainers. In the case of a new pipeline or if there are significant changes planned on an existing pipeline, it may be necessary very early to have the trainer available for training or operations evaluation. This creates a challenge for SCADA as the new SCADA system will need to be ready well before it is needed for the real pipeline. In addition, the pre-developed SCADA system may need modification for last minute construction changes. 5.5 A simulation trainer license is separate from any other licenses for hydraulic models (such as a engineering simulator or CPM system). So a separate license agreement will need to be negotiated with the software vendors, and maybe also for the SCADA system software. It may be possible to extract savings from the modeling software vendor if the trainer license is incremental to other applications. It may also be possible to have a no additional cost copy of SCADA that is used strictly for training and development. 5.6 If it is proposed to continue simulation trainer development while training is occurring in another area, it will be necessary to be able to use two models and SCADA license applications at the same time. Will this require an additional license, or permission to utilize two applications on one license for periods of time? 5.7 The cost of the simulation trainer (model, SCADA and computers) is only one aspect of the overall training cost. If staff from various geographical areas are to be trained, cost optimization will dictate the need for additional simulation trainer systems. In addition, it is necessary that entry level Pipeline Controllers be at a sufficient level of basic pipeline and hydraulics knowledge before the simulation trainer has full value. Training material such as the Enbridge type of training modules may be needed. 5.8 The pipeline that is the most difficult to operate is probably the one that will be most difficult for simulation trainer development. In our case a 'clean products' pipeline with multiple pipe sizes, both fixed speed and two types of variable speed electric pumps, uneven pump station spacing, multiple deliveries and injections, various valve timing, and 'quirky' equipment was the most difficult one to model and to tune. It is also the pipeline that offers the greatest operating challenges to the Pipeline Controllers. 5.9 As with many applications, it is best to oversize the computing CPU power so later enhancements will not suffer with insufficient processing capability. 5.10 It may be expensive to depend totally upon the vendor(s) to build the simulation trainer. A knowledgeable pipeline company staff member, who can undertake parts of the work, can speed up the
development, save money and help ensure the product is as required. Also the training scenarios will have to be developed to meet the pipeline company needs, so input from the pipeline company will be needed. 5.11 If simulation trainer changes are necessary due to any number of reasons, the first step is to archive the existing working models so they can be resurrected if the new systems do not cooperate with the developer. Next, it is advisable to change only one trainer system at a time and get it fully working before making changes to other models. Coordinate all trainer development work with the customer or users. 6. SUMMING UP. Enbridge has recognized the value of using a simulation trainer to enhance safety and efficiency of pipeline operation. The work of developing the simulation trainer is detailed and time consuming, but the benefits have been demonstrated. This paper is intended to highlight only some of the issues of building and using a simulation trainer. Enbridge and the author welcomes questions or comments on the contents of this paper and the Enbridge training program