SKYSCAPE HADOOP IN THE CLOUD

Transcription

1 SERVICE DEFINITION SKYSCAPE HADOOP IN THE CLOUD G-CLOUD 8 Classification: Open

2 Classification: Open ii MDS Technologies Ltd Other than for the sole purpose of evaluating this Response, no part of this material may be reproduced or transmitted in any form, or by any means, electronic, mechanical, photocopied, recorded or otherwise or stored in any retrieval system of any nature without the written permission of MDS Technologies Ltd. MDS Technologies Ltd, 2 Methuen Park, Chippenham, Wiltshire, SN14 0GX Telephone: , Fax: CONTENTS WHY MDS?... 3 SUMMARY OF SERVICE FEATURES... 3 MDS AND SKYSCAPE THE PERFECT PARTNERSHIP... 4 Service assessment guide... 4 Key characteristics of our service... 6 Pricing and packaging... 8 An SLA you can trust Supporting documents and resources Try before you buy The small print Appendix... 14

3 Classification: Open 3 WHY MDS? A privately owned, UK sovereign company Connected to Internet, JANET, N3, PSN, RLI Public, community and private cloud available Security Cleared technical and customer service staff We are Agile, Flexible, Open, Honest and Transparent We deliver cost effective solutions on time and within budget We are your One-Stop-Shop for secure assured Cloud services A fully managed platform using our ITIL-aligned 24/7 Service Desk Experienced at delivering small, large and complex Cloud solutions We are an SME - large enough to deliver, small enough to care PROFESSIONAL, PERSONALISED SOLUTIONS SUMMARY OF SERVICE FEATURES Platform for building your data science services Rapid provision and high capacity to support data-intensive tasks Choice of Hadoop distributions Cloudera Enterprise or Hortonworks HDP Uncontended resources from a Shared multi-tenanted environment Secure and highly scalable Connected to PSN, JANET, N3, Internet Optimised for OFFICIAL and fully aligned with the 14 CESG Cloud Security Principles

4 Classification: Open 4 MDS AND SKYSCAPE THE PERFECT PARTNERSHIP MDS technologies Limited (MDS) offers a cloud platform provided by the award winning Skyscape Cloud Services (Skyscape). We have a strong, committed relationship with Skyscape, the market leader in the Digital Marketplace for G-Cloud Infrastructure as a Service, through a long-standing relationship underpinned by a formal partnering agreement. We have been a Skyscape Partner since their creation in 2011, and have resold services since the launch of the G-Cloud framework in February MDS is a reseller, at cost, of Skyscape s cloud services and is currently Skyscape s largest partner and second largest consumer of cloud services. This status enables us to provide a high level of service to our customers and gain enhanced support from Skyscape. As a Skyscape partner, MDS offers true National Institute of Standards and Technology (NIST) cloud services, which are offered as part of a community cloud service only available to the UK Public Sector and secured and managed in line with UK government security classifications. Skyscape and MDS both utilise pricing models based on transparent charges and pay for what you use on an hourly and/or monthly basis. Service assessment guide What the service provides Hadoop in the Cloud is Skyscape s big data platform built on a dedicated and optimised infrastructure within Skyscape s secure, assured environments (blue components in Figure 1). Data Architecture Application Architecture Tuning & Optimisation Data Ingest Tool Selection (Hive, HAWQ, Storm) Analytics ( MapReduce) Visualisation (Data Science) Customers /Partners Hadoop in the Cloud Skyscape service Hosted in Secure UK Data Centres Powered by OFFICIAL classified Infrastructure HSCIC, MoJ and PASF Independently audited Well connected to Internet, PSN, N3, Janet and RLI Figure 1. Solution elements This service is targeted at data scientists as a data platform service, rather than at end users who usually expect a complete SaaS analytics solution. Data scientists can use the Skyscape service to build higher-layered services for end users. They can rapidly provision capacity optimised for Hadoop to support data-intensive tasks such as indexing, mining, analytics and so on (orange components in Figure 1).

5 Classification: Open 5 What the service can help you achieve Reduce the capital expenditure risk typically associated with exploring big data by using the flexibility of a cloud service Reduce storage cost and complexity by consolidating disparate data stores into a central location, while facilitating holistic analytics across the widest possible dataset Enable analytics across multiple organisations datasets by creating a shared data repository Create a data store capable of handling high-volume, high-variety and high-velocity data (for example, social media sentiment analysis, IoT sensor data, large machine generated/sensor data) that exceeds the capabilities of traditional database technologies Create a cost-effective and future-proof environment to meet the ever-growing storage demands of modern analytic requirements Key service capabilities Choice of the most popular 100% open-source Hadoop distributions Cloudera Enterprise or Hortonworks HDP A true on-demand OPEX-based service which eliminates traditional CAPEX challenges A fully configured deployment which helps reduce the level of expertise needed to get started with the fundamentals of Hadoop Removal of the capacity management burden traditionally associated with Hadoop An extensible solution which enables you to use any Hadoop-compatible business intelligence, analytics and reporting applications Uncontended resources reduce any adverse performance issues as a result of noisy neighbours Proximity of Hadoop data to compute resources increased processing performance and removes data transfer charges between compute and storage An unrivalled big data and analytics partner ecosystem Why Skyscape UK-based telephone service desk and NOC function, with 24/7 support for high-priority incidents, including access to Skyscape s technical subject matter experts Platform optimised for OFFICIAL and fully aligned with the 14 CESG Cloud Security Principles Multiple secure Tier 3 UK data centres separated by more than 100km and securely connected by high-bandwidth, low-latency dedicated connectivity Independently audited and verified by trusted public sector organisations, including HSCIC (NHS Digital), MoJ and the Police (Police Approved Secure Facility PASF), for workloads requiring a higher assurance for sensitive data (for example, MoD Defence-as-a-Platform) UK sovereignty assured cloud platform delivered by a UK-based company with UK government security-cleared staff Platform that hosts workloads exclusively for the UK public sector, creating a known and trusted community of neighbours

6 Classification: Open 6 Cloud characteristics delivered Elasticity scale your big data solution using the Skyscape Portal or our fully documented API Self-service rapid availability on both our Assured and Elevated domains means zero delay to your project Measured usage from just 45p per useable TiB per month for HDFS storage Broad networking connect via DDoS-protected internet, a government community network, or HybridConnect using your own dedicated circuits Resource pooling a neutral platform, shared with a trusted community of UK public sector users to facilitate interconnected datasets Key characteristics of our service Our Hadoop in the Cloud service offers a distinctive mix of multi- and single-tenant elements in its infrastructure to deliver the highest levels of performance and security, without sacrificing the commercial benefits or flexibility of the cloud. Each cluster is deployed on Hadoop-dedicated and -optimised multi-tenant infrastructure, with the following single-tenant elements: Single-tenant cluster. Each customer is provisioned with their own isolated Hadoop environment virtual master nodes, virtual slave nodes and virtual cluster managers aren't shared with any other tenants. Dedicated 1:1 mapping of slave nodes to directly attached physical drives helps increase overall HDFS performance and security. Skyscape uses either VMware vsphere or OpenStack for the underlying IaaS service, with customers choosing Cloudera Enterprise or Hortonworks HDP deployed on top as the enabling Hadoop distribution. The table gives an overview of each distribution. Cloudera Enterprise Hortonworks HDP Based on Apache TM Hadoop 100% open source Based on Apache TM Hadoop 100% open source Cloudera Impala industry-leading massively parallel (MPP) SQL engine built for Hadoop Cloudera Search advanced query and data-browsing capabilities Free-to-use enterprise distribution and features (without support) Support subscriptions available directly from Hortonworks Certified on Skyscape s Platform Support included The platform consists of the following modules and associated supporting services: Hadoop Common: the common utilities that support the other Hadoop modules Hadoop Distributed File System (HDFS ): a distributed file system that provides highthroughput access to application data Hadoop YARN: a framework for job scheduling and cluster resource management Hadoop MapReduce v2: a YARN-based system for parallel processing of large datasets Hadoop Zookeeper a distributed transaction coordinator The Hadoop core platform facilitates data ingress and egress along with native MapReduce v2 applications. In addition, we provide access to Hadoop cluster management via API or GUI

7 Classification: Open 7 (depending on the chosen distribution Cloudera Manager or Ambari), for you to use to manage and maintain your entire Hadoop cluster. We monitor all VMs and services related to the embedded Hadoop core platform as defined above, and provide 'reasonable endeavours' support for all associated services. We provide a secondary virtual data centre (VDC) alongside the Hadoop cluster to enable customers to deploy their own analytics, business intelligence and reporting applications. We charge for and support any VMs within this secondary VDC in line with our Enterprise Compute Cloud or Cloud Native Infrastructure service definitions. However, Skyscape isn't responsible for managing any image, software or service deployed within these VMs. Figure 2. Hadoop in the Cloud overview This service is extensible, as shown in Figure 2, allowing additional analytic applications to be deployed to facilitate functions beyond data ingest, egress and MapReduce v2 jobs. These depend on the choice of distribution and may include: Cloudera Impala a massively parallel processing (MPP) SQL query engine Cloudera Search for non-technical users to search and explore data Apache TM Spark an in-memory data store Apache TM Falcon for data management and pipeline processing Apache TM Kafka a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system Management and maintenance of any additional applications deployed by the customer, beyond the initial deployment, remain the responsibility of the customer or third party.

8 Classification: Open 8 Pricing and packaging To maximise cost flexibility against user requirements, we offer three Hadoop cluster types for you to choose from, based on your initial data requirements, coupled with your projected rate of future data ingest. See the table below. Cluster type Use case Master node spec Slave node spec Low Velocity Medium Velocity High Velocity Proof of concept / small data projects Departmental data lake, Hadoop applications requiring more RAM Enterprise data lake, highly mixed parallel processing applications 9 vcpu 60GiB RAM 500GiB disk 9 vcpu 60GiB RAM 500GiB disk 18 vcpu 120GiB RAM 1000GiB disk 9 vcpu 60GiB RAM 200GiB shared disk 5 * 2TiB local disk (HDFS) 18 vcpu 120GiB RAM 200GiB shared disk 10 * 2TiB local disk (HDFS) 36 vcpu 240GiB RAM 200GiB disk 20 * 2TiB local disk (HDFS) Backup/recovery and disaster recovery The Hadoop storage engine, HDFS, is engineered with infrastructure failure in mind, so localised component failures are tolerated within the infrastructure via data replication, eliminating single points of failure (including physical server failure or disk failure). Hadoop v allows for the manual creation of snapshots of HDFS, which can be stored offline using Skyscape s Cloud Storage. Pricing Skyscape s Hadoop in the Cloud lets customers scale up HDFS storage resources as required, yet retain elasticity around analytics requirements. Each cluster will initially be provisioned with a Starter Pack (defined below), after which customers are free to scale their cluster by adding additional slave nodes. Cluster type TiB capacity (usable ) Starter Pack Additional slave node (above Starter Pack) Cost ( ) per hour TiB capacity Cost ( ) per hour Assured Elevated (usable ) Assured Elevated Low Velocity Medium Velocity High Velocity Notes: Usable capacity reflects HDFS storage including the default data replication factor of three (raw capacity is three times greater) Licensing is not a mandatory Skyscape requirement, so pricing does not include Cloudera or Hortonworks licensing (where applicable) or support subscriptions. You can get prices for these services through a direct arrangement with either vendor via the Digital Marketplace

9 Classification: Open 9 It is currently not possible to mix and match cluster node types within a single cluster (for example, a Low Velocity cluster can only scale out with Low Velocity slave nodes) Additional slave nodes can be expanded in single units as shown above All pricing is charged per hour or part thereof The pricing for the following items is given in the Skyscape Pricing Guide: Enterprise Compute Cloud Cloud Native Infrastructure (Powered by OpenStack) Mass Transfer Facility data ingestion and extraction Connectivity options Pricing example The following table provides an indicative illustration of the costs of operating Hadoop in the Cloud on our Assured platform, based upon a single Starter Pack plus the required number of slave nodes, at various HDFS data volumes across the different velocity cluster types: TiB of storage (HDFS) Low Velocity Medium Velocity High Velocity / hr / TiB / hr / GiB / month / hr / TiB / hr / GiB / month / hr / TiB / hr / GiB / month Comparative Starter Packs To ensure the correct resources are initially deployed to provide a stable Hadoop cluster, we will provision the following virtual nodes, configure the networking and deploy your choice of Cloudera Enterprise or Hortonworks HDP. A Starter Pack consists of the following: 3 x master nodes (Name Nodes, Zookeeper Master, additional master services) 3 x slave nodes (HDFS storage and applications) The specification for each master/slave node will be in line with the cluster type you select, as defined above. Support subscriptions We manage the resources required to ensure the availability of our Hadoop core platform, but we don't manage the analytics or software stacks that sit above it. Unless your organisation has embedded Hadoop expertise and experience, we strongly suggest you purchase a support

10 Classification: Open 10 subscription for your selected distribution to help you with support and usage issues beyond our Hadoop core platform. Additional Hadoop distributions You can deploy on Skyscape s IaaS and manage your cluster directly or via a partner (for example, you may deploy PivotalHD TM Enterprise edition, MapR or IBM BigInsights on Skyscape IaaS and get support directly from Pivotal ). Selecting a partner (optional) Customers who have (or are developing) in-house skills and experience with Hadoop should be able to benefit from using this service directly. However, customers requiring a service that provides business intelligence or data analytics will need to engage a specialist partner to build those layers on top of Skyscape s Hadoop in the Cloud. Skyscape can introduce you to partners in our Partner Programme who offer appropriate specialist consulting services through G-Cloud. You may also choose to work with a specialist partner for enablement services; or to use software services that need licences or support subscriptions. For clarity, we don't mandate the use of partners or the requirement to procure licences/subscriptions. Connectivity options Skyscape provides one of the best connected cloud platforms for the UK public sector. We offer a range of flexible connectivity options that enables access to our secure platform by virtually any community of government users and systems. The variety of government, public and private networks is shown in the following diagram:

11 Classification: Open 11 Discount and purchase schemes You can choose between two discount or purchase schemes see below. You can find full details in our pricing guide. An SLA you can trust We understand that UK public sector organisations need a dependable service that demonstrates value to shareholders, which is why Skyscape offers one of the best SLAs on G-Cloud. What s more, we back our SLA with Service Credits. If we fall short of our agreement, we ll compensate you with Service Credits, so that you can be confident we have a financial incentive to deliver on our promises. The table below outlines the SLA and Service Credit details. For more information, see the terms and conditions. Service level agreement 99.90% Portal level agreement 99.90% Availability calculation Measurement of SLA Key exclusions Availability is calculated based on the number of hours in the billing month (for example, 744 hours for months with 31 days), excluding any planned and emergency maintenance. Skyscape will monitor the services and resources that underpin the Hadoop core platform, and ensure that the availability of the service in a given calendar month equals the above SLA. The following are examples of what is not covered by the SLA: Faults within your control (such as powering down cluster or nodes, or user deletion of cluster or nodes) Faults that lie with external connectivity providers (internet, PSN) and components co-located at Skyscape Service Credit 3% of monthly spend per affected cluster. 1% of monthly spend per 1% below service level target or part thereof for the Skyscape API and Portal. Supporting documents and resources This service definition aims to give you enough information about the service, options and pricing to support your G-Cloud procurement process. The following documents, available on request or from the Skyscape Portal, should provide any additional information you need: FAQs for each service aim to answer the most common questions Datasheets provide a high-level description of each service

12 Classification: Open 12 How to guides provide step-by-step instructions for specific actions and tasks on the platform, including the use of the Portal and interacting with Skyscape support teams In addition, we have: Cloud Architects who can help you understand how this service can address your requirements Over 200 partners who can provide professional or managed services Try before you buy We appreciate you might have various concerns about the suitability of our service to run your specific workloads. As we're confident that our service will exceed your expectations, we offer a 30- day free trial so that you can test and evaluate our service without commitment. Your trial provides you with a live environment on the Skyscape platform to test our services and verify whether they are suited to your needs. Before your trial, a Cloud Architect will engage with you to identify the goals you re working towards, to help ensure that your trial meets those goals. Throughout the trial period, a Customer Success Manager (CSM) will provide support for any issues you may encounter or questions you may have. At the end of your free trial, you can seamlessly move to a billed service, leveraging any of the work you have already completed in the live trial environment. Free trials are: Available to both new and existing customers Available for 30 days from first use; any longer will be at our discretion Available only for products that you haven't previously purchased or trialled Not available for any products and services you are already consuming, unless: Not available for any products and services you are already consuming, unless: You re testing a significantly different use case You re a partner and you re testing a solution for a different customer Excluded from any SLA credits Requesting a trial Request a trial via the Skyscape website ( and accept the trial terms and conditions Your environment will be set up Your trial will be limited to a single Low Velocity Starter Pack During your trial Use your credits for any eligible service within your environment within 30 days Contact your CSM with any issues or questions

13 Classification: Open 13 Ending your trial Your Cloud Architect or CSM will contact you when your trial is due to end, either because you ve used up your credits or because you ve reached 30 days (whichever is sooner) At the end of your trial, you can transition to production or cease the trial The small print The appendix to this service definition provides a summary of the service terms. For full terms and conditions, refer to the Terms and Conditions document, available from the Digital Marketplace.

14 Classification: Open 14 Appendix Ordering and invoicing The service can be ordered via the G-Cloud Framework and must be supported by a valid purchase order. Skyscape will issue invoices as follows: At point of order for upfront fees and service options Annually in advance for pre-payment fees Monthly in arrears for monthly fees Payment can be made by direct bank transfer (BACS/CHAPS). Onboarding Within the lead times stated below, Skyscape will create the customer s Primary Administrator account and send a Welcome Pack which includes the URL for the Skyscape Customer Portal, and the Hadoop guides. Up to 100TiB HDFS Over 200TiB HDFS Cluster expansion Initial deployment 10 days 20 days 20% uplift of HDFS storage Immediate 21% to 30% uplift of HDFS storage 10 days Over 30% uplift of HDFS storage 20 days Lead times are in working days and commence from order acceptance. Shorter deployment times are typically achieved and can be prioritised on request. The customer s Administrator can set policies, create additional user accounts and allocate roles and privileges for users within their project or organisation. Each user can then simply log on and begin using the service. Skyscape has created a number of videos, help guides, manuals and FAQs to help train and instruct users so that they are up and running quickly and easily. These are available within the Knowledge Centre, accessed via the Skyscape Portal. In addition, you will be assigned a Customer Success Manager (CSM) to provide any assistance required during the first 90 days of the service. Skyscape also has a large ecosystem of partners who can deliver additional services such as training, support and professional services. Skyscape would be pleased to introduce you to the right partner to suit your needs. Data migration In many circumstances, Skyscape can help facilitate a bulk migration to the platform using local data import. This is priced on a time-and-materials basis from the Skyscape SFIA rate card. Skyscape can also help facilitate a bulk migration to the platform using offline data ingest and extraction please ask Skyscape for details. Service management As a true cloud service aligned to the NIST definition of PaaS, the service is designed to be self-managed via the Skyscape API and secure online Portal which provide common service management functionality and address standard requirements. Skyscape will allocate a Technical Account Manager (TAM) to provide you with an assigned point of contact. The TAM will provide additional assistance with reporting and incident escalation, at all times following Skyscape s ISO certified ITIL-based process framework. For organisations that require a managed service, Skyscape has a mature and active partner ecosystem that can provide value-added services such as consultancy, training and ongoing custom managed services. Skyscape will be pleased to make an introduction on request. Service constraints Skyscape Hadoop in the Cloud is engineered to be scalable and has been tested up to 850TiB HDFS cluster storage. We advise consulting with Skyscape if you need to scale beyond this capacity Skyscape will adhere to the following in terms of maintenance windows: Planned Maintenance means any pre-planned maintenance to any of the infrastructure relating to the service. Planned Maintenance activity may result in periods of degradation or loss of availability depending on the nature of the activity required. In such cases, Skyscape shall provide affected customers with at least fourteen (14) days' advance notice of the Planned Maintenance. If during Planned Maintenance there is a loss of availability outside the scope described in the planned maintenance notification to the service, an SLA event will be triggered, but excluded from Service Credits. Emergency Maintenance means any urgent maintenance required to prevent or mitigate against any event compromising the infrastructure relating to the service. Whenever possible, Skyscape shall: a) provide affected customers with at least six (6) hours advance notice and b) carry out the emergency maintenance between the hours of 00:00 and 06:00 (UK local time) Monday to Sunday or between the hours of 08:00 and 20:00 (UK local time) on Saturday, Sunday and bank holidays

15 Classification: Open 15 unless there is an identified and demonstrable immediate risk to customer environment(s). Emergency Maintenance may result in periods of degradation or loss of availability depending on the nature of the activity required. If during Emergency Maintenance there is a loss of availability to the service, an SLA event will be triggered. This time will be excluded from the availability calculation but will be included in monthly service reporting related to the service. Technical requirements Customers will require appropriate network connectivity such as internet access or accredited connectivity such as a government secure network to the Skyscape cloud platforms. Connectivity via the internet, a government secure network (PSN, JANET or N3) or private leased line is available but may incur additional charges if the hosting of CPE routers is required see the pricing section for more details. Where they are required, customers are responsible for procuring and managing appropriate devices or software to meet the requirement for data security over the various forms of connectivity. At the point of termination, all customer data, accounts and access will be permanently deleted, and will not be able to be subsequently recovered or restored. Costs There are no termination costs for this service. Customers are responsible for extracting their own data from the Hadoop platform if required. Skyscape may make an additional charge for transferring data out of the service. Offboarding Customers are expected to maintain a master copy or backup copy of the data used in the Hadoop in the Cloud service. So there is no requirement for the customer to transfer their data out of the solution at the end of the contract. Rather, the customer simply has to securely delete the data prior to the termination of the contract Customer responsibilities The control and management of access and responsibilities for end users including appropriate connectivity, security and accreditation if required. If access is required over government secure networks such as N3, JANET, RLI or PSN (including legacy networks), the customer is responsible for adhering to the Code of Connection. Management configuration and administration of layers above the PaaS Hadoop core platform (for example data ingest, analytics jobs, visualisation, tool selection). Customers must be aware of the variable nature of the billing based on usage. Customers are expected to maintain a master copy or backup copy of the data used in the Hadoop in the Cloud service. The customer is also responsible for ensuring only appropriate data (for example OFFICIAL) is stored and processed by applications on this environment and that they comply with the Skyscape Security Operating Procedures (SyOps) and other information assurance requirements as specified in the Skyscape System Interconnect and Security Policy (SISP) and associated accreditation documentation sets Termination Terms Customers can terminate this service by providing Skyscape with not less than 30 days' advance notice in writing.