Instrument Control System Project Management

Transcription

1 Project Documentation Document PMCS-0021 Revision A Instrument Control System Project Management Bret Goodrich Software September 2015 Advanced Technology Solar Telescope 950 N. Cherry Avenue Tucson, AZ Phone dkist@nso.edu Fax

2 1. Date: Aug 5, 2011 Revision: DRAFT Changes: Created 2. Date: Sep 1, 2015 Revision: Rev A Changes: Updated for Baseline. REVISION SUMMARY: PMCS-0021, Revision A Page ii

3 Table of Contents 1. INTRODUCTION CONSTRUCTION FABRICATION PURCHASE INTEGRATION, TESTING, AND COMMISSIONING QUALITY CONTROL VERIFICATION Requirements Verification Design Verification As-Built Verification Documentation Verification QUALITY ASSURANCE TASKS Unit Testing Integration Testing User Acceptance (Verification) Testing VERIFICATION TEST PLAN Unit Tests Integration Tests RISK RISK REGISTER RISK MITIGATION COST AND SCHEDULE ESTIMATES COST ESTIMATE Construction Labor Construction Non-labor Contingency SCHEDULE PMCS-0021, Revision A Page iii

4 1. INTRODUCTION This document discusses the project management of the DKIST Instrument Control System (ICS). It is concerned with construction and development plans, quality control and assurance, risks, costs, and schedule. Each of these areas is covered in sections of this document. PMCS-0021, Revision A Page 1 of 14

5 2. CONSTRUCTION Construction phase planning involves the fabrication, purchase, and integration of the ICS. Planning for each of these areas is described here. 2.1 FABRICATION Fabrication of the ICS will take place at NSO/Tucson. Fabrication involves the development of the ICS software and documentation. All ICS equipment resides in the Tucson NSO Computer Room in a computer rack with independent power and network management. The ICS software development effort will develop three separate packages for DKIST. The ICS software controls the execution of observations on the instruments by issuing commands and data to the instrument in a synchronized and coordinated fashion. The Standard Instrument Framework (SIF) provides a framework for the instrument developers to build their applications. The Base package provides a set of standard components for software and hardware. All fabrication activities continue through August At that time the ICS begins IT&C and maintenance support. 2.2 PURCHASE The ICS software development requires the purchase of two development computers and a set of hardware components. The computers are commodity rack mount with available slots for hardware boards. The hardware purchased includes: PTP grand master clocks; PTP time base cards; Power management controller; Power supply controller; Delta Tau motion control Network switches 2.3 INTEGRATION, TESTING, AND COMMISSIONING ICS integration begins with the development of the instrument systems in their respective locations. The SIF and Base packages are delivered to the instrument developers during 2014 and are maintained through scheduled releases and patches for the duration of construction. SIF development continues through August ICS integration and testing at the summit begins with the first instrument delivery in The ICS package is installed with the OCS in a computer room server. The Test Architecture Framework (TAF) is also installed on the summit, allowing the regular regression testing of PROC-0017 to be implemented here. At this point, the OCS, ICS, DHS, and TCS should be installed at the site. Shortly after the DHS is installed, the Visible Broadband Interferometer (the first light DKIST instrument) should be installed. At this time the DKIST should be ready for the first instrument verification tests, per the VBI IT&C plan. PMCS-0021, Revision A Page 2 of 14

6 3. QUALITY CONTROL Quality assurance and quality control are essential elements in the DKIST construction plan. Many aspects of the QA/QC plan are based upon past history of telescope development, along with modern systems engineering process control. The following definitions apply to the DKIST QA/QC. Quality control, also known as verification, is a process used to evaluate whether or not a product, service, or system complies with regulations, specs, or conditions imposed at the start of development phase. Another way to think of this process is Are you building it right? Quality assurance, also known as validation, is a process used to establish evidence that provides a high level of confidence that the product, service, or system accomplishes its intended requirements. Another way to think of this process is Are you building the right thing? Change Control Board (CCB) is the name of the DKIST group that meets regularly to approve or reject requested changes in the DKIST specifications and interfaces. Specification and Interface Control Documents define the specifications for each Work Breakdown Structure element and the public interfaces between each element. Both sets of documents are held under change control and changes must be approved by the CCB. Design Document is the living document that defines the baseline design of the WBS element. This document is updated through the conceptual, preliminary, and final design phases to reflect the current design approach. During construction this document is updated to reflect the changes in scope or interfaces approved by the CCB. At the completion of construction the document should be a completed theory of operation manual for IT&C and operations. 3.1 VERIFICATION The following tasks will be performed as part of the QA process. The work package manager and the DKIST QC/QA personnel will perform auditing of these tasks Requirements Verification All new and changed requirements of the system will be verified to ensure the following: They are directly related to an approved item from the CCB; They are consistent, feasible, and testable; They have been appropriately allocated to the correct mechanical, hardware, software, and operational processes; and Those that are related to safety, security, and criticality have been verified by the rigorous processes governing those areas. The requirements verification process will be conducted by formal review. It will require participation and sign-off by the following team members: Work package primary investigator; Work package manager; and Work package engineer responsible for the change Design Verification All new and changed design elements will be verified to ensure the following: Design is traceable to requirements; PMCS-0021, Revision A Page 3 of 14

7 Design provides details describing how requirements will be met; and Design implements safety, security, and other critical requirements correctly as shown by suitably rigorous methods. The design verification process will be conducted by formal review and requires participation of the following team members: Work package manager; Work package engineer responsible for the change; and At least one engineering representative from a different DKIST work package As-Built Verification All new and changed components will be verified once construction is complete to ensure the following: Applicable standards are being followed; Applicable best practices standards are being followed; DKIST software coding and commenting standards are met; and DKIST software best practices are met per SPEC The as-built verification process will be conducted by an informal review process such as . The review process must be performed and signed off by at least one engineer from a different DKIST work package Documentation Verification In conjunction with new or changed component being released the following documentation must be provided/updated: Detailed design documentation, specifications, ICDs, etc.; Performance benchmarks (for performance critical modules); Test documentation (unit, component, integration, and user acceptance); and Operations document. The documentation verification process will be conducted by informal review, such as . The review process must be performed and signed off by the following team members: DKIST QA/QC representative; and DKIST release manager. 3.2 QUALITY ASSURANCE TASKS The following tasks shall be performed as part of the quality assurance process. The results of these tasks will be documented and reviewed as part of the Document Verification task of the QC process Unit Testing Unit testing involves the testing of individual units of work to ensure they are fit for use. A new or changed component shall be unit tested. This testing shall be performed before proceeding to component level testing. The tasks that must be completed as part of unit testing are as follows: Prepare new and/or changed unit tests and related documentation; Ensure traceability of new and/or changed tests to requirements; PMCS-0021, Revision A Page 4 of 14

8 Execute new, changed, and existing unit tests upon build of component; and Document unit test results. Unit testing will be performed by the work package engineer responsible for the change or his/her designee Integration Testing Integration testing involves testing an intended release to ensure it integrates correctly with all other DKIST systems for which it interfaces. Integration testing shall be performed in a qualified DKIST test environment that uses mechanical, hardware, and software systems equivalent to the production systems. Integration testing shall be performed before proceeding to user acceptance level testing. The tasks that must be completed as part of integration testing are as follows: Prepare new and/or changed integration tests and related documentation; Ensure traceability of new and/or changed tests to requirements; Coordinate integration test schedule with test engineer of interfacing systems; Execute new, changed, and existing integration tests; and Document integration test results. The work package engineer responsible for the change and the designated test engineer for each interfacing DKIST system will perform integration testing. Results shall be reviewed and signed off by the following team members before proceeding to user acceptance testing: Work package manager; Work package manager(s) for all systems that interface with the released component; Work package engineer responsible for release; and Test engineer for all systems that interface with the released component Software Specific Tasks The following tasks are specific to integration testing for software releases. Any software defects (bugs) identified in testing will be logged in JIRA tracking system; All test cases impacted by the defect must be re-tested once the defect is resolved; Any un-resolved software defects must be approved by the review team before proceeding to User Acceptance Testing; and Upon successful completion of integration testing the software source code will be tagged in CVS to indicate it is part of a release. Test documentation will include reference to this release number User Acceptance (Verification) Testing User acceptance testing involves performing tests for which the user will validate the output of the system to determine pass/fail status. User acceptance testing shall be performed in a production environment, or a qualified test environment that is approved by the user. User acceptance testing shall be performed before a release can be made operational for production use. The tasks that must be completed as part of user acceptance testing are as follows: User to prepare new and/or changed integration tests and related documentation; Ensure traceability of new and/or changed tests to requirements; Coordinate user acceptance test schedule with production or test environments and systems; Execute new, changed, and existing user acceptance tests; and PMCS-0021, Revision A Page 5 of 14

9 Document test results. User acceptance testing will be performed by the user, with the support of the work package engineer responsible for the release, and test engineers from other interfacing systems. Before proceeding to production, the release must be approved by the following team members: Work package user (i.e., owner or primary investigator); Work package manager(s) for all systems that interface with the released component; Work package engineer responsible for release; Test engineer for all systems that interface with the released component; and DKIST release manager Software Specific Tasks The following tasks are specific to user acceptance testing for software releases. Only software source code from the CVS tag that matches the release number identified in the integration test documentation may be used for user acceptance testing; Any software defects identified during testing will be logged in JIRA tracking system; All test cases impacted by the defect must be re-tested once the defect is resolved; Any un-resolved software defects must be approved by the review team before proceeding to production release; and Test documentation should include reference to the CVS tag for this release. 3.3 VERIFICATION TEST PLAN Unit Tests Unit tests will be written with the Test Automation Framework (TAF), as described in PROC All unit tests will test against the ICS compliance matrix (CMX-0006). The major OCS system tests are described below ICS Observation Execution Test procedure: A number of observations are created in the OCS and issued to the ICS. Simulation instruments are used to assure that the observations are executed at the correct times and with the required start and stop synchronizations. Test results: The tests should show that the required execution times are achieved and that the observations complete correctly. The ICS should manage the observation queue are required ICS Observation Cancel and Abort Test procedure: Ongoing and queued observations sent from the OCS are cancelled and aborted. Test results: The ICS should correctly dequeue, cancel, or abort the observations and leave the system ready to execute the next set of observations SIF tests Test procedure: All SIF tests defined by the ICS DRD are implemented as TAF tests. The SIF tests retrieve instrument programs from the database, execute their scripts, and return the results of the execution. The tests use a simple instrument simulator for the execution of the script. Test results: All tests should return the correct results code. PMCS-0021, Revision A Page 6 of 14

10 SIF Instrument tests Test procedure: The SIF is tested as part of the each instrument s test procedures. The Test results: Instrument test that fail are analyzed for the effects of the SIF. Tests where the SIF fail will be rewritten to run as part of the simulator testing Base Test procedure: The Base package is tested as part of the TAF tests run for the Common Services Framework. Base requirements from TN-0088 are implemented as individual test procedures Integration Tests Integration tests are performed both with the Tucson end-to-end test bed and at the summit Camera Software Systems Test procedure: The camera is configured to operate at various rates and sizes that conform to the requirements. The ICS configures the camera according to the instrument program input. Test results: Camera output should meet requirements. Multiple cameras and data streams should not impact the performance of each other Observatory Control System Test procedure: This test must be performed with the cooperation of the OCS and the DKIST End-to-End test bed. The OCS is used to construct a simple experiment using an instruments display plug-in 'view'. A simple SIF simulator is then installed in the DKIST End-to-End test bed and the experiment is executed. The SIF simulator displays the configuration(s) is it given during execution. Test result: The displayed configurations should contain the expected attributes and their values Visible Broadband Imager Test procedure: This test should support the VBI integration tests as defined in SPEC-0107, VBI Critical Design Document, section Test results: The VBI requirements for instrument control should be met. PMCS-0021, Revision A Page 7 of 14

11 4. RISK The DKIST Risk Management Plan defines the project approach to risk analysis and mitigation. The project maintains a risk register containing all identified risks at the beginning of the project and throughout the life of the project. Plans for mitigating each high-level risk and the subsequent results of the actions taken are also given. The risk register exists to accomplish the following risk management goals: provide a tool for managing and reducing the risks identified before and during the project; document risk mitigation strategies being pursued in response to the identified risks and their grading in terms of likelihood and seriousness; provide project management with a document from which status can be reported to oversight; ensure the communication of risk management issues to key stakeholders; provide a mechanism for seeking and acting on feedback to encourage the involvement of the key stakeholders; and identify the mitigation actions required for implementation of the risk management plan and associated costs. 4.1 RISK REGISTER Risk Item Probability Non-labor Cost Labor Cost HLSC-073 Scope refinements, specification documents in transition 80% $216K $173K 4.2 RISK MITIGATION General risk mitigation strategies are applied to all risks in the ICS. These strategies include: Early Development: The ICS will finish construction in The IT&C period will be used to address issues arising from later instrument development or unplanned integration needs. End-to-end test bed: The end-to-end test bed will be used to test the interface to the CSS and OCS. As subassembly element test beds are added (e.g., mount, enclosure, VBI) the end-to-end test bed will be able to test full operational support. In addition, DKIST scientists can begin testing experiments and observations using the test bed. The following risk mitigation strategies are envisioned for each of the major risks defined in the risk register HLSC-073: Scope refinement; specification documents in transition. There may be additional work or delays in development due to formal specification changes. The ICS Design Requirements Document (DRD, SPEC-0163) has not been finalized. PMCS-0021, Revision A Page 8 of 14

12 Mitigation: Complete the specification review and update the specification document. Update the DRD. Work with operations and management to determine the best ICS solutions for any new requirements using the existing budget and/or contingency. Mitigation: Enforce current requirements. Initial operations at DKIST will need to be within the ICS specification and OCD use cases. The DKIST Change Control Board will need to deny scope increases. PMCS-0021, Revision A Page 9 of 14

13 5. COST AND SCHEDULE ESTIMATES The DKIST budget for the ICS is $771,567. This escalated cost number was developed by a bottom-up cost analysis performed in August 2011, and assumed hardware technology available at that time. At that time, the definition of the DKIST instrument suite was not complete; it was uncertain how many cameras would be in operation and what their final data rates would be. Because of this and other technology factors, project contingency of 18% was set aside. 5.1 COST ESTIMATE The August 2011 DKIST budget has created separate work packages for labor and materials of each ICS development area. New quotes were obtained for all materials required by the ICS design. Work Package Amount S-WANS3-170 ICS Management and Support (ARRA) $138,772 S-WANS3-171 ICS Final Design Staffing and Training (ARRA) $13,211 S-WANS3-172 ICS Base Components (ARRA) $142,692 S-WANS3-173 ICS Instrument Sequencer (ARRA) $63,771 S-WANS3-174 ICS ICS Observation Management (ARRA) $93,121 S-WANS3-175 ICS Simple Instrument (ARRA) $122,950 S- WANS3-190 ICS Instrument Control System Development (ARRA) $73,215 S-WMNS3-190 ICS Instrument Control System Development $123,835 Total $771, Construction Labor Labor costs for the ICS were recalculated in August 2011 for the individual schedule items. Beginning in FY2012, the ICS provides 15% of its labor for instrument support; this includes assisting instrument developers with the SIF, instrument program development, and base hardware usage. Resource Work FY2011 SW.1 Instrument Control 100% 85% 85% 85% 85% 85% Instrument Support 15% 15% 15% 15% 15% FY2012 FY2013 FY014 FY2015 FY2016 Construction labor for the DHS ceases in August 2016, although instrument support activities continue through the end of construction Integration, Testing, and Commissioning Labor After construction, the ICS labor remains in the project to perform IT&C activities. The IT&C planning for these activities is currently under development. The activities include: Integration of the telescope subsystems; Construction and integration of the summit computer room; Integration and testing of the OCS and ICS; Integration and testing of the VBI; Integration and testing of the remaining instruments. PMCS-0021, Revision A Page 10 of 14

14 IT&C labor begins September 2016 and completes in July Construction Non-labor Hardware costs for the ICS were recalculated in July 2011 for the ICS design. All computers are commodity items that are expected to be purchase shortly before integration. Therefore the configuration and prices are expected to change over the course of the next three to four years. In some areas, such as network switches and disk drives, new technology may supplant our baseline hardware. In all cases except the real-time processing nodes, the current technology meets the ICS requirements. Total cost for all materials is $142,692. This is comprised of both the computers and the test hardware Contingency DKIST contingency is held by the project management and is not a part of any specific work package. However, to determine the project contingency, the ICS non-labor activities were assessed of possible risk factors and assigned a contingency percentage based upon the criteria of the DKIST Contingency Management Plan. The calculated ICS contingency was 18% of non-labor activities. PMCS-0021, Revision A Page 11 of 14

15 6. SCHEDULE The ICS schedule started construction in FY2011 with the development of the instrument specifications, designs, and interfaces. During FY2012, the ICS developed the Base package software for the majority of standard software and hardware controllers. During FY2013, the ICS began interactions with the instrument developers to determine the specific implementation details of the instrument program execution and to help develop the Facility Instrument Definition, a document that defines the requirements for a facility-class instrument. The ICS began development of the Standard Instrument Framework in 2013 based upon the results of the instrument definition. This advance work also assisted the instrument schedule by providing critical software in a timely fashion. In 2014, the ICS began development of the ICS package, including the instrument program database, sequencing, and SIF interface. This work, along with the improvements to the SIF and Base packages continues through the end of ICS fabrication in August Integration of the ICS with the OCS and instruments is a part of the IT&C process and not covered by the ICS fabrication. PMCS-0021, Revision A Page 12 of 14

16 PMCS-0021, Revision A Page 13 of 14

17 7. CONFIGURATION ITEM DATA LIST The Configuration Item Data List (CIDL) describes the status of all major deliverable software and documentation items. At present, the CIDL for the DHS contains the following: Source Code Packages ICS Final Release 1.0 (zsh) Standard Instrument Framework Release (Canary 10) CSF Base Release (Canary 10) Documentation SPEC-0023, ICS Specification Document SPEC-0163, ICS Design Requirement Document TN-0102, ICS Critical Design Document CMX-0006, ICS Compliance Matrix MAN-0002, ICS Operations Manual PROC-0017, HLS Software Test Plan SPEC-0113, Facility Instrument Interface Specification TN-0088, DKIST Base Application Support Environment ICD-31.4/4.2, ICS to OCS Interface ICS Factory Acceptance Test Plan PMCS-0021, Revision A Page 14 of 14