Deep Learning Acceleration with MATRIX: A Technical White Paper

Transcription

1 The Promise of AI The Challenges of the AI Lifecycle and Infrastructure MATRIX Solution Architecture Bitfusion Core Container Management Smart Resourcing Data Volumes Multitenancy Interactive Workspaces Next Steps The Promise of AI Deep learning and other AI development techniques are disrupting every industry in an incredible way self-driving cars, drones, virtual assistants, more accurate medical diagnosis, automatic lead generation, better customer service, cybersecurity, and much more. By leveraging AI solutions, businesses are discovering ways to create exciting new products, cut costs, and boost profits, differentiating significantly from their competitors, and maximizing the investments they ve made in big data infrastructure and data science talent. The Challenges of the AI Lifecycle and Infrastructure However, these benefits do not come without their challenges. Deep Learning and AI is not one-size-fits-all, and in order to derive the maximum benefit, it is crucial for companies to train on their own data sets and modify models and code to fit their unique use cases. While some API-based products and SaaS (software as a service) solutions will thrive, the companies and products that stand to benefit the most are the ones where they have taken a proactive approach with their data and are doing their own model training to create tailored code and defensible intellectual property that achieves specific, unique goals. Typically, the deep learning development process is very lengthy and results in many redundant iterations of model training. Many months are spent on devops and infrastructure tasks, iterating in a low visibility, guess-and-test oriented environment, and not maximizing the full potential scalability. 1

2 10X Faster Development, Greater Efficiency, Unlimited Scalability The MATRIX was designed as an end to end DL-in-a-Box solution to fast track AI development and depoyment. As a comprehensive line of fully-validated plug & play appliances optimized for Deep Learning, the MATRIX eliminates hardware compatibility and setup overhead so users can quickly achieve time to productivity. Fully integrated with the Bitfusion Flex suite, the MATRIX provides an iteration environment with intuitive user interface, high visibility and control over resources and workflow, and a compute virtualization capability that enables easy scaling for massive data sets across large numbers of GPU resources. Figure 1: Deep Learning Do-It-Yourself vs. MATRIX 2

3 MATRIX Solution Architecture For fastest time to productivity, the MATRIX software, powered by Bitfusion Flex, is optimally deployed on MATRIX appliances, ranging from high-performance workstations and servers, to rackscale appliances featuring advanced cooling, monitoring and efficiency features. MATRIX appliances have been fully-validated and integrated with MATRIX software for a true plug-and-play experience. Machine Learning [SMART]Rack MATRIX 200 MATRIX 400 The software is also compatible with all major hardware and operating environments to bring a heterogenous resource pool into the MATRIX on-premise cloud. The MATRIX provides a total solution for managing underlying CPU and GPU compute resources and enabling self-service development and deployment of Deep Learning workloads. Figure 2: MATRIX Technology Architecture Hardware MATRIX 3

4 Figure 3: Nodes View By leveraging MATRIX software, developers may begin their development on a CPU machine to conserve GPU resources for other users and more critical workloads. When the code is ready to be tested on a GPU, an elastic network-attached GPU can be allocated to the CPU workspace. Then when it s time to scale up the training to many GPUs, the workload can be run using one or many local and remote GPUs for rapid model training while maintaining a simple programming paradigm. At a minimum, it s recommended to use a dual 10GbE data/compute fabric for workstations and dual 25GbE with ROCE for servers to provide sufficient network bandwidth between host and client servers. For larger deployments requiring scale up performance across nodes, we strongly recommend EDR Infiniband (100Gb). Bitfusion Core The secret to how the MATRIX can allow such incredible flexibility is Bitfusion Core, our compute virtualization engine. Bitfusion Core intercepts API calls between applications and the underlying GPU compute, allowing for elastic GPU attachment across nodes over the network (network-attached GPUs), partial GPUs (memory partitioning), GPU scale out of up to 64 GPUs attached to a single work process or container, and overall optimizations for advanced performance. 4

5 Figure 4: Bitfusion Core Technology Architecture Because Bitfusion Core uses an interceptor approach rather than hypervisor or other methods of virtualization, there are no changes required to the underlying hardware or virtual machine environment, as well as no changes required to the application code itself. This means that AI developers and data scientists can leverage the benefits of GPU virtualization seamlessly with minimal overhead and integration requirements. Container Management The MATRIX builds on top of its powerful compute platform and GPU virtualization capabilities with a comprehensive container management solution. By running applications and workloads on a Docker-compliant framework, users gain a huge amount of flexibility, built-in versioning capability, all with minimal performance impact. In fact, most organizations that are highly mature in their Deep Learning and AI workflows leverage a containerized approach as the heart of their workload management engine. However in order to get something off the shelf (i.e. Docker Swarm, Kubernetes, etc.) to work, these companies have invested hundreds of thousands of man-hours customizing these frameworks to work performantly with GPU drivers, Infiniband, GPUs as a fungible resource, hiding complexity from basic users, data management, etc. The MATRIX has incorporated all these requirements into our core product so that there is zero customization required by customers -- you can take advantage of our Docker-based, purpose-built container management solution out of the box for effective Deep Learning and AI from the first minute you log in. 5

6 Figure 5: Example Workspaces With MATRIX Pre-Built Environments Our container management layer includes a built-in repository for managing pre-built containers that are included with every deployment, while providing the option to create your own user-generated containers: MATRIX Pre-Built Environments: The MATRIX containers are kept up-to-date with the latest deep learning frameworks and data science libraries so that you can always leverage the latest and greatest new features being developed by TensorFlow, Caffe, Torch, and other communities. The administrator can run a single command to pull down all the latest containers (environments) into the cluster without any disruption to running workloads. User-Generated Environments: Users can leverage workspace snapshots or bring-your-own-container as ways to modify and save container environments for use by others. Workspace snapshot leverages a docker save workflow to duplicate an environment and add it to the repository with changes saved. The bring-your-own-container approach leverages a minimalistic base container that the MATRIX provides with just the operating system and minimum library and driver requirements to ensure it will work seamlessly. Users can modify this environment, and when ready -- load it into the MATRIX for use as a standard environment. Container Export: Containers can be exported for inference or other production deployment requirements. 6

7 Smart Resourcing Typically an AI developer or data scientist knows how many GPUs or GPU memory they would like to leverage, but does not know (or does not want to know) the underlying resource consumption state at any given time. Take the following scenario: When 4 GPUs are needed, if there are 4 local GPUs (all on the same server) available, then allocate that to my workload. If there are only 2 local GPUs and 2 remote GPUs available, then let me know that is all you have available and allow me to proceed or not. The MATRIX makes this experience automatic and transparent, optimizing ideal GPU topologies ahead of less-ideal GPU topologies -- to strike an important balance between resource efficiency, developer productivity, and ease of use. Figure 6: Workspace Creation Smart Resourcing Slider Data Volumes The data required for Deep Learning and AI workloads often comes from many different sources -- online and offline, external and internal, bulk files and file systems, etc. The MATRIX tries to make this as simple as possible by allowing administrators to define one or many network-attached data locations to be mapped into containers. As long as the underlying host machines have access to the data location, your containers can have access as well. This makes it incredibly simple for AI developers and data scientists to access all the data they need in a direct and seamless way. In addition to allowing for unlimited data volume mapping flexibility, the MATRIX also supports a local NFS file system on each of the nodes. This default option provides a standard location that is workload location-agnostic, so that no matter where a deep learning workload is being run (including if it is running across multiple servers), the processes have fast, local access to the data required to get the job done. 7

8 Multitenancy: Building an On-Premise GPU Cloud Given the incredible promise of Deep Learning and AI, businesses are looking to rapidly enable an on-premise GPU Cloud or GPU over Fabrics as a Service for various business groups, departments, and geographies. The MATRIX includes comprehensive group-based user management for enabling broad, self-service access to the compute and container services for running deep learning and AI workloads, while also maintaining a strong administrator - regular user separation. Depending on your business policies, processes, and industry standards, our flexible approach to user management can be customized to fit your unique situation to enable your ideal balance between centralized control and decentralized self-service. Interactive Workspaces Interactive Workspaces, powered by Jupyter ipython notebooks, provide the fastest way as an AI developer or data scientist to get started using Deep Learning and AI. The MATRIX does not limit the number of users you can have using a MATRIX cluster -- we base our pricing and scaling around infrastructure footprint. This is because we believe strongly that a Deep Learning and AI infrastructure investment should be a service that is shared broadly across the enterprise for maximum impact and return on investment -- and should not be taxed on a per-developer or per-data-scientist basis. Figure 7: Workspace Details View 8

9 Containers are loaded in a matter of seconds with all the Deep Learning frameworks, data science libraries, and GPU drivers pre-installed. Jupyter is just a click away. The exact resourcing you desire, whether CPU-only, a partial GPU, one GPU, or many GPUs, is just a simple slider. More GPU resourcing? Slide right. Less GPU resourcing? Slide left. Smart Resourcing ensures you have the ideal GPU topology, and the underlying Container Management and Bitfusion Core compute virtualization take care of the rest. Figure 8: Jupyter ipython Notebook Figure 9: Jupyter Terminal 9

10 Next Steps Schedule a Demo: Reach out to us for a one-on-one consultation and software demonstration so that we can learn more about your unique challenges and initiatives and show how AMAX can help. Virtualization White Paper: Interested in more information on performance and compute virtualization? Contact us above and inquire about our Virtualization White Paper which goes into more detail on the performance and scaling characteristics of the MATRIX solution. 10