AppHub Requirements and Architecture, and Development Plan

Transcription

1 The European Open Source Market Place ICT Project Deliverable D2.1 AppHub Requirements and Architecture, and Development Plan This project has received funding from the European Union s Horizon 2020 research and innovation programme

2 Project Number : Project Title : AppHub Deliverable Number : D2.1 Title of Deliverable : AppHub requirements and architecture, and development plan Nature of Deliverable : Report Dissemination level : Public Licence : Creative Commons Attribution 3.0 License Version : 4 Contractual Delivery Date : March 31, 2015 Actual Delivery Date : July 1, 2015 Contributing WP : WP2 Editor(s) : Peter Deussen (Fraunhofer) Author(s) : Peter Deussen, Majid Salehi Ghmasari (Fraunhofer), Alexandre Lefebvre (UShareSoft) Reviewer(s) : Alexandre Lefebvre (UshareSoft), Olivier Bouzereau (OW2) Abstract The purpose of the requirements and architecture, and development plan is to define the architecture and development guidelines to be followed by the project for the implementation of the project platform. The document provides partners with a point of reference and a common understanding of the platform which will be implemented by the project. Keyword list Architecture, requirements, roles, stakeholders, concept mapping, integration, software components, API, development

3 Document History Version Changes Author(s) 1 Outline 2015/02/18 2 Draft 2015/02/22 3 Final draft 2015/02/28 4 Approved for submission 2015/03/01 Document Review Review Date Ver. Reviewers Comments Outline 2015/02/18 1 N/A Accept Draft 2015/02/22 2 Alexandre Lefebvre (UshareSoft) QA 2015/02/28 3 Alexandre Lefebvre (UshareSoft) Final 2015/03/01 4 Alexandre Lefebvre (UshareSoft), Olivier Bouzereau (OW2) Accept Accept Accept

4 Glossary, acronyms & abbreviations Item API CKAN IaaS JeOS OCD OCEAN OCIF OIF OS REST SaaS SDK VM Description Application Programmer Interface Comprehensive Knowledge Archive Network Infrastructure as a Service Just enough Operating System Open Cloud Open Cloud for Europe, Japan, and Beyond Open Cloud Interoperability Framework Open Interoperability Framework Operation System Representational state transfer Software as a Service Software Development Kit Virtual Machine

5 Table of Contents 1. Introduction AppHub Platform Overview Stakeholders Platform Components AppHub AppHub Factory AppHub Use Cases and Requirements Consumer Perspective Browse AppHub Deploy asset Producer Perspective Register Project and Assets Deploy to Delete Asset from Delete Asset from Delete Project Analysis and Implementation Options Account Management Workflow Transfer Information Update Architecture System Components Concept map Initial Population of Data API Documentation Development Plan Conclusions and Next Steps Annex: AppHub Open Interoperability Framework...11 Annex: Pre-existing Software and Services...11 Open Cloud UForge AppCenter UForge Table of Figures Figure 1: AppHub component overview...1 Figure 2: AppHub directory overview...2 Table of Tables Table 1: Top level concept map...7

6 1. Introduction This document details the different components of the AppHub platform, outlines its architecture, and defines the points of integration between its components. It defines the project stakeholders, their respective roles and usage scenarios (use cases), leading to a mapping of respective concepts used in the AppHub platform components. Finally, it proposes a development plan to be followed in order to deliver the AppHub platform. 2. AppHub Platform Overview Figure 1 provides an overview over the AppHub platform architecture. The platform comprises of three main parts that interact with the various stakeholders, AppHub, AppHub Factory, and AppHub. AppHub identifies three main groups of stakeholders: 2.1. Stakeholders Producers are developers of open source assets, that are products (software, documentation, specifications, use cases, etc.) under open source license (AppHub does not stipulate a particular type of license). While any type of open source asset is a potential addition to the AppHub, we concentrate on software type assets. Consumers are parties that use open source assets (mainly: software) as part of their own IT system landscape. A minimal usage scenario is just trying out some open source software. AppHub however encourages the use of open source software in a production context. Providers of Infrastructure-as-a-Service (IaaS) clouds. Infrastructure services are used by the consumers to deploy open source software within a virtual machine type container in a cloud. In opposite to the other two types of stakeholders, due to self-service functionalities, providers assume a passive role (i. e., they are not involved as actors in any use case). Figure 1: AppHub component overview. AppHub 1

7 2.2. Platform Components AppHub The allows the producer to register open source assets, and to describe them by means of an asset taxonomy (the Open Interoperability Framework, compare Annex: AppHub Open Interoperability Framework). This classification allows the consumer to understand the functionalities a specific asset provides, which activities is supports, and which cross-issues are addressed by it. Moreover, the provides functions to compare assets, search them, identify assets that potentially can be in combination, etc. Also the AppHub has social networking features, consumers can follow projects, rate and comment on assets.the AppHub is an extension of the Open Cloud developed in the OCEAN project AppHub Factory Open source assets of type software can be further accessed from the AppHub Factory and the AppHub. The goal is to provide an environment in which open source software can be directly deployed within a cloud infrastructure (using virtual machines as containers). Open source software asset producers can use the Factory to build their project template (including additional software components, installation procedures, etc.), with the option of generating a machine image for any virtual, cloud or physical (ISO) format, in order to publish it to the AppHub AppHub The AppHub is the place where producers share their project results in terms of templates and/or machine images. In order to do so, they package their results with the AppHub Factory and submit them for approval to the AppHub administrators (see below). Producers can also manage different versions of their products. It is also the place where Consumers will be able to: find the images or templates published by Producers download the images which may have been made available by Producers publish such images to the cloud of their choice in the case of templates, import them into their private library of templates in the AppHub Factory (see previous section). The administrators of the, i.e. AppHub project staff are responsible for approving products submitted by producers to ensure compliance to technical requirements of the AppHub platform, but also sufficient software quality. 3. Use Cases and Requirements This section summarises the main use cases for the AppHub platform from the perspectives of the two main actors, the consumer and the producer. Since the AppHub project (and hence, this deliverable) focuses on the integration of various existing software components to provide an integrated service, we concentrate on use cases that exercise functions not yet presented in these components (and thus require further implementation work). Functions already implemented within these components are described briefly only Consumer Perspective Browse AppHub Activities Required functionality Component Available Consumers can browse the without logging in. They can view the projects and their assets. They can compare the activities they support and the functions that Browsing, search, comparison functions. AppHub 2 Yes

8 they provided Some assets are only in the (e.g. pdf files, html documents, ), while others (software) are "deployable". Workflow transfer from the to the, Continued: Deploy asset Deploy asset Activities Required functionality Component Available Then, the consumer can (depending on the type, image or template): Download the image Downloading Yes Decide to download the image interaction Yes If it is a UForge template, import the template to his/her Factory account, and possibly customize them, and generate before deploying to a cloud account Precondition: In this scenario, the consumer must login to the for either downloading or publishing or importing a template. Factory/ interaction Cloud deployment Unified account management Factory, All Yes 3.2. Producer Perspective Register Project and Assets Activities Required functionality Component Available Producer logs in into the Unified account management Producer registers project Project registry Yes Producer registers asset Asset registry Yes Producer describes asset within the OIF Asset taxonomy Yes Asset is made available also in the Mark asset as deployable Create initial information on the asset in the Transfer of work flow to the Continued: Deploy to., Deploy to Activities Required functionality Component Available AppHub 3

9 Precondition: Producer is logged in into the Market Place Precondition: Asset is registered in the directory Producer navigates to the description of an asset (automated if performing 3.2.1) Producer edits asset description, creates template or provides image download link Producer initiates approval process of the asset by the AppHub administrator Unified account management See Register Project and Assets Market Place / Factory interaction All Market Place, Factory Yes tification service Yes Delete Asset from Activities Required functionality Component Available Producer logs in into the Producer navigates to the asset and performs delete operation Deletion of the asset from the (if present) is performed manually using the delete function. Unified account management Project registry: Delete operation / interaction, Delete Asset from Activities Required functionality Component Available Precondition: Producer is logged into Market Place Producer navigates to the asset and performs delete operation Update of the asset in the is performed automatically: The reference to the item is removed Unified account management Project registry: Delete operation / interaction, Delete Project Activities Required functionality Component Available Precondition: Producer is logged into Precondition: Project defines no assets Producer navigates to the project and performs delete operation Deletion of associated data in the is performed automatically Unified account management Delete Asset from Project registry: Delete operation / interaction 3.3. Analysis and Implementation Options, From the use cases listed in the previous sections it becomes clear that three main problems have to be solved when integrating the three parts of the AppHub platform (where the integration of the Factory and has been already done by UShareSoft): AppHub 4 Yes Yes Yes Yes

10 How to provide a unified user management? How to transfer the workflow from one component to the other? How to update information within one system from the other? Account Management As a general principle, information regarding users, projects, assets, etc. should not be stored at multiple places to minimize the need for synchronisation. Hence, working with different user registries for the and the /Factory is not appropriate. Due to the modular design of the software used for the /Factory, it is possible to use it in connection with external user registries. We therefore decided to use the user registry as centralized user management component, and to extend it by interfaces to be used from the and the Factory. The use cases we have discussed so far indicate that the following scenarios for user registration and log in exist: From To Has account Logged in User Action Interaction User registers at Yes User enters login/password to log in ad creates user entry in the user registry (API to be implemented) validates credentials at user registry Yes ne transmits API key to the. This API key is used to validate user identity at user registry. User registers at directory Workflow Transfer Yes User logs in at creates user entry at user registry (default API) validates credentials at user registry Yes ne transmits API key to the. This API key is used to validate user identity at user registry. Since all three components of the AppHub platform utilize Web based user interfaces, workflow transfer is achieved simply by providing a link to the appropriate page/position of the component where the workflow continues. Hence, the main issue here is to determine this target position based on the information that is available at the starting page. Storing static URIs is not an option since Web resources might not be available (as in the case of asset creation). Resource locators however are composed from a base URI, and project/asset information. Hence, a mapping between the data structures of the and the Factory/ can be used to determine dynamically continuation points of workflows Information Update Closely related to the question how to achieve workflow transfer is the issue of information synchronization which cannot be avoided in all cases. For instance, an asset in the needs to have a counterpart in the (hence, basis information on this asset is shared between these components). Therefore, a minimum set of interfaces need to be available to synchronize information between components. AppHub 5

11 4. Architecture As already mentioned, the AppHub platform is based on two pre-existing products, namely the Open Cloud (which in turn is an extension of CKAN) as technology basis for the, and the UForge AppCenter/ bundle as basis for the Factory/ System Components User management functions are delegated to the. Instead of utilizing its own user data base, the Factory/ is extended with a user authentication connector in order to interact with the user management of CKAN. On the /CKAN side, the implementation of a REST interface that allows to access and to manipulate user data is required. Since CKAN does not provide this interface, an extension (user registry interface) will be implemented. The amount of data on projects and assets that need to be synchronized between the two system parts is kept to a minimum. The available REST interfaces of both CKAN and UForge provide sufficient functionality for this, hence no additional implementation work is required. AppHub GUI Customization AppHub Visualisation and analysis AppHub data CKAN Core User registry User reg. IF Standard REST IF User auth. connector Market Place Data User registry and authentication requests Standard REST IF Factory / Market Place Core Factory / Market Place Project/asset data sync Figure 2: AppHub platform architecture. Figure 2 provides an overview over the components of the AppHub platform and shows the main interaction relationships Concept map Let us now discuss which information on projects and assets need to be synchronized between the two subsystems. Clearly, the data structures (database schemas) of the subsystems are not the same, however, structural similarities can be found or defined, respectively. The following Table 1 discusses the mapping of the top level concepts of the subsystems that is employed for synchronization purposes. On a pragmatic level, these top level concepts do not have much in common. The concept trees are however structurally very similar, hence by hiding the /Factory concepts behind the ones defined in the allows to use both systems in an integrated way. AppHub 6

12 /Factory (UForge) Mapping Project Vendor UForge do not employ the concept of a project, while the does not define vendors. Both concepts are however on the top level of the data structure, thus correspond structurally Asset Product A project comprises several assets, while a UForge vendor groups several products. Asset Resource Product item Each Asset Resource will become a Product Item of the corresponding Product Redaktor Member CKANs redaktor role allows introducing new projects and assets. On the other hand, each UForge member allows to modify vendor and project data. Hence, both roles are in good semantic correspondence Table 1: Top level concept map. The complete concept map is depicted in Figure 3. Each arrow describes a piece of information that resides both in the AppHub and the AppHub /Factory. As described before, synchronization of these information in performed by standard interfaces of both system parts, hence no additional implementation work is required other than calling these interfaces. AppHub 7

13 Figure 3: Detailed concept map Initial Population of Data When a producer wishes to create an item in the, a connector will pre-create the necessary minimum information from the asset data in the AppHub. This will be done by developing a project creation module on the AppHub side, calling the AppHub APIs in order to create the corresponding data inside the. This project creation module is be responsible for: The initial population of the AppHub from pre-existing data inherited from Open Cloud AppHub 8

14 Subsequent creations of projects and assets in the AppHub API Documentation Details on messages will not be described in this deliverable because sufficient documentation is available from the following resources. CKAN API Documentation UForge API Documentation Development Plan The planning for the AppHub system development and integration activities is shown in the table below: Release Description Date Development phase 0 (I) Milestone: Development versions of the AppHub, AppHub Factory, and AppHub available End March Preparation 1 (I) 1 st AppHub developer workshop: Architecture February Concept 2 nd AppHub developer workshop: Interfaces April Milestone: Synchronization support between AppHub and AppHub Factory/ available End June mapping and functional extension 2 (I) AppHub taxonomy workshop June 9 Data extension, 3 rd AppHub developer workshop: Integration Mid July testing and quality testing assurance Milestone: AppHub content extension End August (taxonomy, data sets) functional, system integration completed and tested 3 (E) Milestone: AppHub beta roll-out complete End September Beta roll-out 4 (E) 4 th AppHub developer workshop: Quality assessment Milestone: Beta testing complete; AppHub production version roll-out complete Table 2: Development plan Mid vember End December Beta test period The plan comprises five phases, each of which is accompanied by and internal (I) or external (E) release of the AppHub system in various stages of integration and adaptation. These phases are prepared by a number of workshops (1 2 dates). Phase 0 (Preparation) provides an initial developer installation of the various parts of the AppHub system software. Phase 1 (Concept mapping and functional extension) provides the finalized system architecture, including the conceptual mapping from the AppHub to the AppHub Factory and AppHub, and the definition and implementation interfaces for data synchronization and workflow control. Phase 2 (Data extension, testing and quality assurance) improves the taxonomy of the directory to capture all relevant types of open source assets, and is devoted to system and AppHub 9

15 integration testing. In particular, this phase addresses the extension of the OCEAN taxonomy (OCIF) required to capture the characteristics of the larger set of software assets the AppHub project addressed. Phase 3 (Beta roll-out) deploys the beta version of the AppHub system in preparation of the beta test phase. Phase 4 (Beta test phase) is devoted to a limited operation of the system in its production environment, featuring a number of open source projects and the assets they provide. The purpose is to gain first experience on the operation of the AppHub system, identify software problems and issues not encountered during the test activities, and to prepare the AppHub system for the operational phase (2 nd project year of AppHub). 6. Conclusions and Next Steps In this document, we have detailed the requirements and stakeholder, presented the architecture of the AppHub platform and its components, identified the necessary integration developments to be done in order to satisfy the requirements, and proposed a development plan to reach the project expected deadlines accordingly. AppHub 10

16 Annex: AppHub Open Interoperability Framework At the time of writing this deliverable, the details of the OIF are under investigation. During the preparation of the AppHub proposal is was planned to reuse the OCEAN Open Cloud Interoperabilty Framework (OCIF) as it is to define the taxonomy and associated functions. Initial discussion with various parties (European Commission, project managers, OW2 taxonomists) however made clear that this approach is not appropriate when taking into account asssets that are not directly related to cloud computing (as it was the case in the OCEAN project). Hence, activities have been initiated to extend the OCIF to cater the extended requirements of the AppHub project (c. f. Section 5). The Open Interoperability Framework will be discussed as part of the user documentation of the AppHub system (Deliverable 2.4). Experiences from the OCEAN project however indicate that a general structure comprising four building blocks is suitable as IT system architecture (Figure 4). Figure 4: AppHub directory overview. 1. Each reference architecture has to identify those parties which are engaged in using a software system. These actors perform a number of activities employing the various technical components of the system according to the tasks they are expected to perform, and the responsibilities they have. Grouping the activities an actor can perform defines the role that this actor assumes when interacting with the software system. 2. Each software system is composed of a number of technical components that provides various functions to the actors interacting with the system. These components are usually grouped into layers or in case of components that provide supportive functions multilayers (for instance, security support usually addresses all system layers). 3. Components are implemented by technologies, and in some cases support standards. At this level, interoperability between components, or between systems, becomes and issue. 4. Finally, cross-cutting aspects are those that addresses various roles or layers/multi-layers and manifest themselves not on technical level only, but also concern issues related to organisation, society, regulations, etc. Annex: Pre-existing Software and Services Open Cloud Thee Open Cloud ( is a free online catalogue and repository collecting information about open licensed outcomes of European, Japanese and other Cloud Computing R&D projects to define the existing Open Cloud landscape with special focus of positioning available components, supported standards and potentials for integration as an Open AppHub 11

17 Cloud Interoperability Framework. This is done by mapping the outcomes of Open cloud projects registered in the Open Cloud to a cloud computing reference architecture and open standards. UForge AppCenter UForge AppCenter (previously called UForge Factory) allows users (both end users and publishers) to build and maintain full software stacks, from the OS up to the application, for any cloud. The AppCenter offers the following features: Software templating: Model full software stacks, from OS to application, including DevOps agents and bootscripts to automate deployments Collaboration: Workspaces let users securely share software stacks throughout design and delivery OS updates & rollback: UForge AppCenter maintains a repository of all native Linux OS packages, updated daily Just enough Operating System (JeOS): Automatically optimize the Linux OS footprint to include only those packages required for boot-up UForge AppCenter supports Red Hat Enterprise Linux (note that RHEL will not be exposed in AppHub for licensing reasons, RHEL not being freely redistributable), Fedora, CentOS, Ubuntu, Debian, OpenSuSE, Scientific Linux and Windows Server. It also supports all leading cloud technologies, including Windows Azure, VMware, Apache CloudStack and Citrix CloudPlatform, OpenStack, Eucalyptus, Abiquo, Nimbula (Oracle), Flexiant, Google Compute and Amazon, for hybrid cloud. In addition, the Migration function of AppCenter can be used to: Scan live systems: UForge Migration deep scans live servers and reports back all components, packages and licenses Instant migration: Simply replicate an identical server complete with the necessary drivers etc. for your cloud Incremental scans: Schedule incremental scans to see at a glance what changes have been made Import the scan into the AppCenter as a template for customization UForge UShareSoft's product is a dynamic, white-labelled infrastructure platform for building a cloud-ready software hub. It can be used to deliver any software component customizable templates (built with UForge AppCenter), blueprints (work in progress), existing VMs, binary components and SaaS Services, to any cloud or on-premise datacenter. The goal of UShareSoft's is to connect application designers and consumers, while staying in overall control of the applications. The offers the following features: Browse and search: end users (consumers) can search by name, OS, category, and publisher, or view by date, rating and popularity. Publish templates: end users (consumers) can publish VMs on specific cloud infrastructures using publication connectors and their cloud credentials to such clouds. Social networking: Users can rate, review and follow individual items in the. Publisher rights: Engineering teams and ISVs can upload and delete their software (either in the form of binaries, templates, VMs or SaaS Services), and respond to user reviews. Publishers can define re-usable fine grain marketing, pricing and support plans which are attached to their software and provide the consumers with detailed information. Administrator rights: administrators have control on the publications (approval workflow), define the taxonomy of product categories and product types. They can also define AppHub 12

18 badges (labels such as supported, qualified, ) and decide which badge to attach to which product. UForge (like UForge AppCenter) is a 100% restful product. This means that all operations that are exposed through the web UI are available as API calls. UForge products offer two kinds of APIs: the native REST API, as well as a Java SDK. For the AppCenter product, a subset of operations is available through an open source command-line tool called Hammr. AppHub 13