SOCCI - Towards a Common Software Engineering Environment for Science Operations Vicente Navarro, 1 Kaarel Hanson, 2 Kaarel Lumi, 2 Ranpal Gill, 1 Jose Marcos, 1 Maria Garcia Reinaldos, 1 Juan Carlos Segovia, 1 Monica Fernandez, 1 Ruben Alvarez, 1 1 European Space Agency - ESAC, Villanueva de la Cañada, Madrid, Spain 2 CGI, Karvaamokuja, Helsinki, Finland Abstract. The European Space Astronomy Centre (ESAC) has been ESAâĂŹs Science Operations Centre (SOC) since 2008. For each science mission ESAC hosts a number of operational systems typically related to mission planning, instrument handling, data processing, services and tools to users, and data archiving. As part of this role, ESAC is responsible for the management and implementation of Science Operations Systems throughout the software development lifecycle. Therefore high quality software engineering flow-down from Mission Requirements to Science Operations represents a critical success factor for ESAC in particular, and any space project in general. Although standards and principles for good software engineering have been established for quite some time, over recent years more and better support tools have become available. These tools represent key enablers for increased efficiency, contributing to deliver higher quality software systems with less effort and time. The main goal of this activity is to define a common software engineering environment where a core set of tools and procedures are shared across teams. This environment is to be used for all upcoming missions while current missions, with software engineering environments often characterised by a heterogeneous approach, will migrate depending on their specific circumstances. In order to maximize productivity the project has followed an evolutive / Agile approach, focusing on those areas with higher potential for commonality. Other areas like requirements engineering or test management, were tackled as part of the project needs in a generic way, defining a potential solution for other projects. Moreover the design of the new environment has been done in cooperation with a multi-disciplinary team from different missions, leveraging on existing practices and tools already in place. This paper presents the approach, architecture design and implementation of the future Science Operations Configuration Control Infrastructure at ESAC. 1. Introduction Three different organizational units at ESAC are involved in configuration control activities for science systems in SCI-O. Over the years, these units have utilised several systems to support development and operations of science systems throughout their lifecycles. Evolution of working practices and different project needs have resulted in the coexistence of multiple tools in order to fulfill configuration management requirements. The intention of this project is to analyse current working practices and systems 1
2 Navarro, et al. in use in order to migrate to a single framework that meets all requirements. While each of the current tools has several deficiencies in the context of how it is used, the primary motivation is to simplify the CM processes and reduce the knowledge base required by Science Operations teams to perform its work. The project adopts ECSS-E-ST-40C1 Figure 1. SCI-O Organisational Context standard for software development as the reference process framework to validate adequacy of the configuration management framework to current practices. Nevertheless the outcome of this project is not part of a space system. In order to maximise alignment of project results to actual needs an Evolutive/Agile software development lifecycle has been adopted. It is important to mention that the project itself uses its own outputs. Such approach simplifies the validation of the process and the tools providing early feedback. The figure below illustrates a summary of key software processes with green bubbles highlighting the focus of the project. Figure 2. Project Scope
2. Objectives Science Operations Configuration Control Infrastructure 3 The purpose of the Science Operations Configuration Control Infrastructure (SOCCI) is to support the software development and maintenance processes of all SCI-O units at ESA s European Space Astronomy Center (ESAC). As described in the previous section the requirements identified during the activity and the selection of the associated tools has been organized around the following areas: Problem and change management; Documents management and version control; Source code version control Continuous integration. Requirements management Test management As SOCCI has used SOCCI other topics have been "kept in mind" for potential integration points with the topics above and experimented/prototyped as part of the project. A key objective of SOCCI is to base it on existing state of the art tools which are then configured and enhanced as necessary, limiting the level of ad-hoc development to the absolute minimum. Nevertheless the trade-off assessment of the tools has taken into account past and present experiences in order to define an evolutionary adoption path for SOCCI. The final objective is to deploy SOCCI on ESAC s data centre using conventional hardware, network and operating system. 3. Engineering Strategy SOCCI engineering strategy has been based on the following major drivers: Consolidated of configuration management procedures: a set of interviews with stakeholders from multiple projects were carried out in order to gather a homogeneous set of configuration management procedures and requirements Definition of the System Concept: using the previous set of procedures a preliminary design and technology stack with the candidate tools was defined Implementation iterations: the system concept is further elaborated in a set of iteration were key areas are priositised based on project needs, stakeholder interests and implementation feasibility. Adaptability The nature of the projects The integration nature of the project has led to adopt a requirements engineering approach where standard features usually available in these tools are excluded. In cases where the feature was not clearly covered the requirement has been included in the
4 Navarro, et al. baseline. For instance, the function to commit (check-in) source code to version control systems was considered standard. However, the function to define custom workflows for issue tracking was included in the requirement baseline. The selection of the technology stack was based on existing tools like JIRA, Confluence, Git, etc. The verification effort checks whether the tools can be used in a consistent, consolidated and integrated way at ESAC, increasing the efficiency and quality of the software development process. Besides, the selection of the tools has been based on current practices at ESAC, reducing the number of candidates. Tools have been used in a standard way both technically and procedurally as much as possible. For instance, JIRA standard issue types and corresponding workflows have been by default. Deviations have been mainly triggered by more stringent requirements derived from the application of ECSS processes. Ad-hoc development of plugins to extend or integrate products has been carried out along the same lines. From a conceptual point of view SOCCI has specified key elements to support the processes in its scope. For requirements management one issue type "Requirement" has been defined Figure 3. Key Process Elements in Jira. For each system, subsystem or specific category a separate JIRA project is used. For example user requirements are managed in a separate project with. If a system has several small scale subsystems there is no point to define a project for each subsystem. On that case all requirements should be listed under one project and Domain identifier should be used to distinguish requirements of different subsystems. For project management four issue types have been defined in Jira: Activity - denotes activities in a work package. Sub-Activity - sub-task type issue of Activity. Sub-Activity refers to smaller task(s) of activity, mainly required for time tracking of a single task. Action - reused issue type from Jira
Science Operations Configuration Control Infrastructure 5 Task - used for tracking tasks that do not belong to an Activity For problem and change management, corresponding JIRA issue types for Software Change Request and Software Problem Report are defined with the attributes and workflows for being the same. For test management three key issue types were created The Test Case issue acts as a template for Test Cases. It contains information about a test like test steps and pass/fail criteria. A Test Case can be used many times with different Test Records for testing different versions, environments etc. The Test Record issue is created when a test case is run. It contains the information about the test like version, environment etc. It is linked to the Test Case issue and the test steps are copied from the Test Case to Test Record, because the Test Case can change in time, but the Test Record then has the snapshot of the test case. Test Records are also linked to Requirements to show if the tests regarding those requirements are passed for the last version or not. Any issues found during testing are also linked to Test Records The Test Design issue groups the Test Cases by similar functionality or some other parameter. Test Design is cloned to create Test Records from all the related Test Cases 4. Architecture Functions supported by SOCCI are quite standard. Therefore it makes use of a number of existing products to deliver most of the functionality. These products are integrated and tailored according to the requirements. Where necessary, the products are enhanced using publicly available (commercial) add-ons or custom built add-ons and features. The following diagram gives an overview of SOCCIs architectural design. The top- Figure 4. SOCCI architectural design
6 Navarro, et al. level components/products of SOCCI are: JIRA2 - an enterprise system for change and problem management. The system has very powerful tools for issue tracking, monitoring and management, and allows different kinds of configurations and setups for workflows and issues. Confluence3 - an enterprise system for documentation version control. Confluence provides online concurrent document management functionality for better collaboration within a team/project. Git4 - an open-source source code version control system, providing functionality for central source code repository. Gerrit5 - an open-source team source code collaboration and review application. Gerrit acts as an intermediary between the developers local repository and Git repository. Jenkins6 - an open-source continuous integration services providing system. Enables triggering software build jobs automatically or on-demand. SonarQube7 - an open-source continuous inspection of code quality platform. Provides fully automated analyses of software build jobs, in order to maintain quality in the source code repository. Nexus8 - an open-source artifact repository manager. Chef8/Puppet8 - an open-source configuration management utility. Allows automating administrative tasks such as deploying new versions of software to several servers. These tasks can be achieved mostly declaratively (Puppet) or semi programmatically (Chef). The tool for this purpose has not yet been decided. Livelink - an enterprise content server, providing file-system based sharing functionality. Each of the top-level components listed above provides one or more functions of SOCCI - requirements management, problem and change management. The general relation between a logical component and a product is many-to-many - one product can be used for more than one function and for one function more than one product can be used. For instance, JIRA is used for requirements management and problem and change whereas JIRA and Confluence both are used to provide requirements management. It shall be also noted that there are a number of supporting components (products) that SOCCI recommends to use in software development process and which allow to reuse them in continuous delivery process. SOCCI advises to use these supporting components because of the observable benefits that they provide (easy, supported integration): Ant - which is a build tool. Ivy - which is a dependency management tool. Maven - which is a build and dependency management tool combined, as opposed to Ant and Ivy.
Science Operations Configuration Control Infrastructure 7 JUnit - which is a unit testing framework. Eclipse - which is an integrated development environment that has among other development supporting features a plugin for Git and Gerrit that simplifies cooperation with Gerrit. The logical decomposition of the software follows the functional groups and topics of the requirements, presented mainly from end-user point of view. Each of the toplevel components can be further broken down into sub-components. The figure below depicts the final mapping of the functional elements and technologies. Figure 5. SOCCI products vs functions As previously mentioned, there are a number of plug-ins used in SOCCI to achieve a smooth software development process. Multiple plug-ins have been evaluated as part of the project. Additionally following custom plugins have been developed in order to close feature gaps: Document Map - Allows easier navigation for long Confluence pages by displaying headings in a sidebar when viewing or editing Confluence pages. Clicking on a heading in the sidebar scrolls the content of the Confluence page to the respective heading. Clicking on any text in the Confluence page highlights the respective heading in the sidebar. The document map sidebar provides buttons for hiding/unhiding itself and reloading itself, which is useful in edit mode. Baseline - Allows selecting a list of documents from a document space, copying these pages to a baseline space, making the contents of these pages static, copying the attachments, exporting these pages to Word format with Scroll Office and creating a baseline page which links to all of the above. This is useful for releasing and reviewing static snapshots of documents rather than dynamic live versions. Space Copy - Allows specifying parameter values when copying a Confluence space. Useful when a new Confluence space needs to be created from a Space Template. It replaces placeholder values in the Confluence pages with user-specified values.
8 Navarro, et al. Clone Test Issues - Allows cloning Test Specification, Test Design and Test Case type issues to Test Record type issues and creating all the necessary links with only a few clicks. It supports Test Management as per ECSS standards. 5. Results and Conclusions Currently SOCCI is offered as a service for ESAC missions to take advantage of the tools, processes and guidelines defined by the project. The utilization of SOCCI s project deliverables by SOCCI as part of its development has contributed to achieve a fully functional system well before project completion. The system is now deployed at ESAC where several projects have started to adopt it for some of their processes. In particular, the quick delivery and internal validation of Problem and Change Management processes has enabled other projects to start using SOCCI for these tasks. In the meantime other processes are being finalized while feedback from early adopters is retro-fitted into SOCCI. The figure below shows how the different environments in SOCCI work in an integrated way for an example where documents generated from Jira in Confluence are reviewed online with change requests traced back to Jira. Figure 6. SOCCI s Integrated Environment The main conclusions for SOCCI can be summarized on the following list: It provides an out-of-the-box platform for System Engineering Processes. Its design is based on SciOps / industry best practices and tools. It is driven by compliance with achieve ECSS-E40 standard. It is focused on usability and process guidance.
Science Operations Configuration Control Infrastructure 9 It maximises efficiency via tool integration synergies. It is a flexible platform with a full baseline where projects opt-in or opt-out depending on their needs. Self-validation has contributed to speed up development and adoption by other projects.