The implementation of a test centre for traffic applications in the Netherlands dr. W.J.W. Geurts ing. E.R. van der Ster ing. A.M.M. van Lent Dutch Ministry of Transport, Public Works and Water Management Directorate-General of Public Works and Water Management Transport Research Centre (AVV) Boompjes 200; P.O. Box 1031; 3000 BA Rotterdam; The Netherlands Tel: +31 (0)10 282 5654 - Fax: +31 (0)10 282 5990 - E-mail: Wouter.Geurts@cmg.nl Figure 1: A test environment (right) has been implemented to test traffic management systems in a situation that is functionally similar to an operational traffic management centre (left). ABSTRACT Managing increased (road) traffic has the focus of the Dutch Ministry of Transport, Public Works and Water Management for years. The measures taken to control traffic flow and to minimise the number of traffic jams more and more rely on advanced monitoring and control systems (traffic management systems). These systems traditionally were specified by the Ministry and designed and realised by industrial parties. For years this has been possible because the needed traffic management systems were small, local and independent. Nowadays (and certainly in the future) traffic problems are not local phenomena and the control will need country wide infrastructural decisions, guided by a country-wide control strategy 1. Such a strategy will have to be supported by an integral, uniform set of traffic management tools. To be able to control the tender for a uniform set of tools, a test centre for traffic management systems has been founded. This test centre will contain all the relevant systems in a normal traffic control centre (TCC) and data networks, such that new applications can be added en tested within this operational environment. We describe the organisational and technical details of the implementation of this test centre. Within 18 months we realised a working environment, that acts as an effective filter between development projects and IT-service management in operational traffic control centres.. 1 See the article Organisational Impact of Quality Assurance on Traffic Control Systems by E. Troost et al. (ITS conference Torino 2000).
MOTIVATION TO BUILD A TEST CENTRE The Ministry is responsible for the capacity and safety of the roads. The high ways and high way junctions have been expanded with the growth of motorway traffic. Parallel to this expansion of the capacity of the high ways a set of traffic management systems has been installed to guide the traffic locally and to yield a better throughput. The variety of systems grew over the years with every pilot project to fight the congestion. With the growth of the traffic a country-wide policy on such traffic management systems is needed. For efficiency reasons these systems should use a shared communication/telematics infrastructure and for ergonomic reasons the systems should have a uniform user interface. Moreover the traffic management systems have become more and more mission critical since the capacity of the roads depend strongly on them. Therefore a strict policy on quality control is needed. The first step in quality assurance is the installation of solid configuration management for hardware and software. For this reason the developed systems are maintained by a dedicated branch of the Ministry: the IT-service management department. This department is organised in processes based on ITIL with as main task to control the analysis and repair of outages of systems both in the traffic management centre and at the road-side. The IT-service management dictates quality checks, but until recently these checks still occurred for a large part in the operational environment. New releases of software were installed in operational traffic management centres and then tested on interoperability with their surroundings. This was due to the fact that the supplier of a single application is not responsible for the final integration test software and the Ministry lacked a facility to try out the new (or modified) traffic control system. The chosen way to tackle this problem is the foundation of a test centre for traffic management systems. Experience in testing shows that testing costs are usually higher than necessary due to the fact that testability is not designed into the system. Therefore, testing should from the start be an integral part of the project. Besides a facility (hardware, operational environment and software tools together with a skilled testing crew) the test centre has to provide a consultancy branch that is involved in the development of new systems already in the requirements phase. IMPLEMENTATION OF THE TEST CENTRE The TORO project (TestOmgeving/ReferentieOmgeving translated Test Environment Reference Environment) has been responsible for the implementation of the test centre. The TORO project consisted of three work packages. The first work package consisted of the realisation of the reference environment, i.e. the tender of computer hardware, (flexible) infrastructure (multi-segment ethernet) together with practical issues like housing, power supply, airco for computer rooms, The second work package was the realisation of the test environment and had to guard the stated technical goals, building a consistent, workable test process. In this work package the tender of test tools to support this process had been done. The third work package was the test centre organisation, implementing the test process with people and to position the test centre 2 organisation within the standing organisation. These sub-projects are closely connected through the goals. A good mixture of technical and organisational solutions had to be given. 2 See footnote 1
After the definition of the test process, this new test process was used to define the needed procedures together with the test environment (tools/instruments) to support these procedures. The two criteria for such an environment are (a) it should facilitate the migration to the new test process and (b) it should at least be able to reproduce the acceptance tests of the running systems. Originally these tests were Factory Acceptance Test (FAT) executed at the developer s site. In other words the new environment should be able to perform regression test on the running systems. Furthermore, this new environment should be designed to be highly adaptable to new test needs (think of stress or load tests), such that the regression tests can be expanded to more complete test suites. In the near future the presence of the test environment will contribute to the technical specifications of new systems. The test process In industrial software development projects the test process is tailored to fit the realisation part the of the project. The development projects of the Ministry involve tendering both the realisation and the test activities. Nevertheless we were able to map this tender process to a common software development process. Once such a mapping exists, the IEEE-1012 ( Standard for Software Verification and Validation Plans - 1986 ) can be used to define a suitable Verification and Validation (V&V) process (as a collection of activities). To guide the specification and design of the test centre (organisational as well as technical), the V&V process has been presented in a vision paper (the test philosophy ). This first step already revealed that the needed process reached beyond the scope of the test centre organisation itself. It turned out that if the test centre has to perform tests efficiently and effectively the input to the test centre should be uniform and high quality specifications or design documentation of the system under test. As a first step to such an input specification the organisation adopted the IEEE/EIA J-STD-016 ( Standard for Information Technology Software Life Cycle Processes Software Development Acquirer- Supplier Agreement ) as organisation wide standard. This standard has been chosen as a baseline to migrate to the IEEE standard 12207 (IEEE/EIA 12207.0/1/2-1996 Standard for Information Technology - Software life cycle processes). The key characteristics of the new test are efficiency (less cost than original method/reduction of tests in the operational situation), effectiveness (possibility of load/stress tests) and reduction of problems in the field (by active feedback: learning organisation). Starting from the test philosophy both an organisation plan and requirements for supporting tooling and infrastructure were derived in a well-defined design process. Organisational impact Within the organisation of the Ministry the test centre will play a significant role in quality assurance. This means that the test centre should be an independent professional organisation, strongly linked tot the existing ITIL processes like software development, configuration management, change management. The test centre should also be equipped with processes like service level agreement management. The test centre for traffic control systems will be a central specialistic entity within the Ministry that will capture test experience. A (not unimportant) side effect of such an entity is that testing will be an integral part already in the specification, design and build processes. People become more aware of
verification and validation needs and techniques in earlier phases of a systems life cycle. Already this awareness will cause process improvement on the development side, calling for the need of improvement of the test centre processes. The test centre should therefore be a learning organisation requiring a dynamic quality officer and a dedicated/inspiring general manager. To be able to tender systems in a controllable way the EIA/IEEE J-STD-016-1995 standard will have to be tailored to obtain a really uniform specification structure. Restrictions on the used processes (to introduce standard checkpoints: Preliminary (Design) Reviews (PR/PDR) and Critical (Design) Reviews (CR/CDR). The IEEE-1012-1986 standard is used as a starting point for a well defined process within the test centre organisation. Starting with the most urgent problems, a subset of the prescribed activities of IEEE-1012 has been used. On a regular basis the service catalog of the test centre will be adapted to the needs of the clients.using this regular (about yearly) update the 1998 version of IEEE-1012 will be incorporated. user interfaces system under test datacom modules to road side generic testtool system under test adapter Figuur 2a (left) and b: the system decomposition of a traffic management centre (left); testing one module in isolation (right) using dedicated adapters. Technical impact The test centre contains a hardware reference-target environment that depends on the test purpose of the individual tests. The scope of the test centre is to test application software common to all traffic management centres in the Netherlands, therefore the reference-target environment should be functionally comparable to the HW environments of all these centres. Specific characteristics of the controlled area of the individual TCC s pose requirements on the indivual HW environments of these TCC s (think of performance or communication abilities that are dependent on the number of sensors or actuators that are present in the controlled area). Because the reference hardware will never be completely uniform over all TCC s, the test centre will never be able to ban testing from the operational environment.
test system Keyboard presentation unit PCO: point of control and observation Data storage Database server Communication server Road side system (sensors/actuators) Figuur 3: Once a (technical) structure is known, the test can be defined in terms of points of control and observation PCO's (see text). These points are the interaction points of the test-tools with the system under test. Once the reference-target environment of the applications is known, a set of points of control and observation (PCO s) can be concretised (figure 3). In fact PCO s can be defined at every level in the specification or design, where there is a description of a complete structure of the system. This methodology is borrowed from the ISO-standard ISO-9646 (ISO/IEC 9646, Information Technology - Open Systems Interconnection- Conformance Testing Methodology and Framework). The test centre is not set up completely conform this standard since the test centre is not a conformance test laboratory in the sense of ISO-9646 (traffic management systems lack standardizing bodies, like there are for interconnecting telecom systems). Nevertheless at least the following concepts of the ISO9646 are present: abstract test methods (ATM), abstract test suites (ATS), points of control and observation (PCO), abstract service primitives (ASP; which is quite popular in the implementation as action word, function script or, here, as commands : messages to and from the traffic management application), Implementation extra Information for Testing (IXIT) and means of testing (MOT) which has a resemblence in the test adapter (see fig. 2). To keep solutions with a large Commercial Off The Shelf (COTS) component open, the use of TTCN as test description language had not been required. Architecture CHARACTERISTICS OF THE OPERATIONAL TEST CENTRE The architecture of the testing facility is guided by requirements on configuration management. Since there is one reference target environment, while there are five operational traffic control centres. In principle all operational situations have to be tested. A certain version of a System Under Test requires a certain version of the (executable) test suite, which requires in turn a certain version of the
testing environment. Furthermore the testing environment should be expandable to deal with the systems of the (near) future. Besides configuration management an important fact is that several parts of an operational traffic management centre (see fig. 2a) are delivered by different parties (eventually at different moments in time). In order to be able to assess the quality of these parts before they are integrated with the other systems, stand-alone tests have to be performed. Therefore an architecture has been chosen that consists of a general testing environment which is expandable with adapters (see fig. 2b). The general testing environment will not change (significantly) with the release of a new or modified system. Used Tools The general test tool is implemented using TestDirector, WinRunner (for Windows NT) and XRunner (for X-windows environments under several Unix-variants). These tools are COTS components provided by Mercury Interactive. These tools (WinRunner and XRunner) are well able to provide interaction to the Graphical User Interfaces based on MS-Windows and X-Windows (Motif) but had to be expanded (using a dynamically linked library) to fit specific needs for our case. Specific interactions with the systems under test have been implemented in C (both on NT and on Unix). Well-defined test processes The tool set (COTS and custom) has been implemented for a well defined test configuration. Expandability had to be proven explicitly. Therefore, an extensive user guide was delivered (describing coding standards best practices for the Mercury Interactive test script language TSL, and a procedure concept Statement of Work for developing additional adapters. The connection with the IT-service management organisation is implemented on an organisational level. The test centre organisation will perform tests that are (partly) inspired by the problems in the maintenance phase of the systems (or comparable systems) life cycle. CONCLUSIONS The implementation of the test centre has been supported by a change in the mind set about procurement processes 3. We found that already the decision itself to define and install a test centre has increased the awareness and professionalism of the organisation. The technique of introducing firm quality checks to improve the products worked well for the organisation. Introduction of ITIL-processes improved the administration, configuration management and change controls. Introduction of the test centre added a validation and verification mechanism of changes. After the test centre has matured one could think of introducing the possibility of prototypes to facilitate the research on new means of traffic management. 3 See footnote 1