The Optimization Bottleneck sigopt.com 1

Transcription

1 WHITEPAPER The Optimization Bottleneck sigopt.com 1

2 OPPORTUNITY Fill the optimization gap in your ML and AI processes As investment in artificial intelligence (AI) and machine learning (ML) continues to grow, expectations for its impact on business outcomes soar. Over 80% of executives consider AI and ML critical to their company s success 1. And analysts expect trillions in value to be created 2. Contrary to these weighty expectations, most investment in AI and ML has yet to deliver their promised returns. Unless this changes quickly, these companies will continue to be exposed to disruption from AI-enabled competition - which is cited by 79% of data executives as their biggest risk 3. One of the biggest contributors to this problem is too few quality models in production. There are not enough machine learning engineers and these engineers are limited by tools that make it tough to prototype model development. As a result, less than 50% of models make it to production 4. Automated hyperparameter tuning holds the potential to improve team productivity, drive more efficient model prototyping, and sustain model performance. How do teams address this challenge? Optimization. Automated hyperparameter tuning holds the potential to improve team productivity, drive more efficient model prototyping, and sustain model performance. Teams who prefer to outsource their model development typically use solutions that embed optimization - data in, model out. But most teams who want to leverage their unique expertise develop their own models either build an optimizer for a specific model type or require experts manually tune models. Neither option is a sustainable solution for any team who wants to scale development. Any custom-built solution will draw from scarce expert resources and not perform across a variety, volume, or complexity of models. And tuning is a task that does not lend itself to domain expertise. This is the optimization gap that SigOpt fills. The Model Optimization Gap Data Collection & Preparation Feature Engineering & Model Selection Training & Tuning Data Engineer ML Engineer DevOps? Deployment & Production 1. Source: Fortune CEO Survey, See, for example: Accenture, Why AI is the Future of Growth, Source: NVP Big Data Executive Survey Source: Gartner Survey, How to Operationalize Machine Learning and Data Science Projects, SigOpt Whitepaper: The Optimization Bottlneck

3 PROBLEM Tuning cost grows with model variety, volume, complexity Tuning is expensive, inefficient, and difficult to scale. Some methods require so much compute that they make tuning too expensive in time and money. Others require too much valuable expert time that is better spent on problems that lend themselves to expertise. Almost all methods are impossible to scale across the variety, volume, and complexity of models needed to address all of the business problems that any team faces. Each of these problems, however, is symptomatic of the fact that most traditional optimization methods are too limited. Random search is inefficient. Grid search is expensive and infeasible for even moderately complex models. Evolutionary algorithms are so computationally expensive that they are out of the question for most teams. Most Bayesian methods are efficient but rely on open source standards that are brittle in production and come with heavy investment in time and energy to scale. Tuners bundled with workbenches require teams hand over control of their models to a consolidated stack, risking vendor lock-in that does not evolve with the team s needs over time. And homegrown optimizers built for a single model type do not have similar impact for model types two and three - there is no free lunch. As a result, teams find that they are left with a few bad options. In some cases, their experts spend time tuning models, wasting valuable time that should be spent on innovation. In others, they avoid tuning altogether for some models, and leave it to the last mile for others. In most cases, models put in production are rarely re-tuned. And most processes created for any model type fail to scale with others. In all cases, teams are failing to realize the full value of their investment in machine learning or artificial intelligence. This is the trap that teams fall into that is represented in the diagram below. Tuning is expensive, inefficient, and difficult to scale. Some methods require so much compute that they make tuning too expensive in time and money. Others require too much valuable expert time that is better spent on problems that lend themselves to expertise. Data Collection & Preparation Feature Engineering & Analysis Model Analysis & Selection Model Training & Tuning Production & Deployment Experts waste too much time on tasks that do not benefit from their expertise. Process fails to scale. Last mile tuning leads to fewer models in production. Models drift in performance. sigopt.com 3

4 SOLUTION Automate tuning for any model Hyperparameter tuning is the perfect problem to automate in the machine learning workflow. It does not benefit from data scientist expertise. It can be incredibly complex, and its complexity scales exponentially with a model s dimensionality. If applied liberally in the development process, it holds the potential to transform the way teams make decisions regarding every component of a model - including features, architecture, models, and the hyperparameters themselves. And if applied frequently, it holds the potential to deliver and sustained heightened model performance. SigOpt was founded to empower the world s experts by building solutions that amplify and accelerate their modeling impact. As a result, hyperparameter optimization is the ideal problem for us to solve. Our Optimization Solution applies an ensemble of Bayesian and global black-box optimization techniques to automate the hyperparameter tuning process for any variety, volume, or complexity of model. Black-box optimization only requires access to inputs and outputs, tuning any model without ever touching the underlying data or model. This is more secure and repeatable for different model types. Customers access this solution through a simple API integration within their preferred programming language and scale it seamlessly across any model with the support of our robust enterprise platform. Finally, this solution includes an intuitive dashboard that captures all versions of models with insights on the most influential hyperparameters to encourage reproducibility. As a result, teams tune more frequently, less expensively, and with greater impact on performance. OPTIMIZE RETUNE Data Collection & Preparation Feature Engineering & Analysis Model Analysis & Selection Model Training & Tuning Production & Deployment Experts spend less time tuning, more time innovating. Transform model development into a rapid prototyping process. Sustain model performance. Scale across all models. 4 SigOpt Whitepaper: The Optimization Bottlneck

5 HOW IT WORKS Adaptive tuning & model insights with a few lines of code SigOpt was founded to empower the world s experts by building software solutions that accelerate and amplify the impact of their machine learning, deep learning, and simulation models. SigOpt s Optimization Solution is specifically designed to automate a task that typically requires significant expert time and does not benefit from their expertise: hyperparameter optimization (HPO). This solution combines Experiment Insights and a black-box Optimization Engine with an Enterprise Platform that scales with the volume, variety, and complexity of experiments over time. SigOpt-empowered teams prototype & produce better models more efficiently This optimization solution reduces data scientist time wasted on tuning, increases the frequency with which models are tuned, increases throughput to production, and scales across any variety, volume, or complexity of models in development without accessing your models or data. Experiment Insights: Collaborate across teams, track experiment results for reproducibility, and analyze performance through an intuitive web experience Optimization Engine: Automate model tuning with an ensemble of methods capable of optimizing any model Enterprise Platform: Scale model tuning for any variety, volume, or complexity of models with a black-box approach that does not access your proprietary data or models sigopt.com 5

6 IMPACT Maximize the return on machine learning investments Teams who automate tuning are more easily able to rapidly prototype models. As data scientists spend more time applying their expertise to innovation and less time wasting it on tasks like tuning, As data scientists spend more time applying their expertise to innovation and less time wasting it on tasks like tuning, the rate of high-performing models in production grows. the rate of high-performing models in production grows. As for these in-production models, automated tuning helps sustain their performance over time. And with happier, more productive data scientists, your team is less likely to have a retention problem, and far less susceptible to the downside of the quant crunch. This is the virtuous machine learning cycle that maximizes the impact of data-driven modeling on a business. There are at least five benefits of automated tuning that contribute to this cycle: 1. Expert productivity: SigOpt automates tuning, a time- consuming task that does not benefit from expertise 2. Time-to-market: SigOpt s adaptive tuning methods require far less time to tune most complex models 3. Compute efficiency: SigOpt achieves performance with less compute required than any other method 4. Reproducibility: SigOpt can be used to tune any model, and captures every training cycle in a simple dashboard 5. Scalability: SigOpt tunes any model and scales with the variety, complexity, and volume of models over time As this cycle ramps up for any business, prediction becomes cheap, so it is applied to a multitude of problems. And the models behind these predictions become accurate enough to automate entire business processes. In some cases, businesses will end up changing their entire business model as a result. These are the cases where the return on any AI or ML investment far exceeds the cost. Whether in finance, technology, insurance, health, life sciences, or robotics, there are a variety of bottom-line benefits that teams are currently leaving on the table - simply by failing to automate a task like tuning that is gobbling up expert time. Teams who fail to address this challenge risk falling behind their peers, and missing this significant opportunity. 6 SigOpt Whitepaper: The Optimization Bottlneck

7 Resources FOR ENTERPRISE Request a free trial: sigopt.com/pricing Learn how we help: sigopt.com/industries/banking FOR PRACTITIONERS See demo: sigopt.com/getstarted Get docs: sigopt.com/docs FOR PARTNERS Contact us to partner: sigopt.com/contact Learn about our partners: sigopt.com/partners FOR ACADEMICS Peer-reviewed publications: sigopt.com/research Get SigOpt for free: sigopt.com/edu SELECT RESEARCH Evaluation System for a Bayesian Optimization Service Interactive Preference Learning of Utility Functions for Multi-Objective Optimization Preemptive Termination of Suggestions During Sequential Kriging Optimization of a Brain Activity Reconstruction Simulation SELECT PARTNERS SELECT CUSTOMERS SigOpt has an easy integration and useful web UI, but, most importantly, we saw performance improvements to the models optimized by SigOpt. Director of Data Science, Hotwire We can keep our experts focused on the tasks core to our business, and entrust the SigOpt platform to find the optimal hyperparameter configurations for our models, irrespective of the data type and model type. Deep Learning Engineer, *gramlabs SigOt has helped VSA solve an optimization problem that was too challenging for traditional approaches. SigOpt has powered a VSA marketing allocation simulator in a way that has given both VSA and our client a competitive advantage. Director of Data Science, VSA sigopt.com 7

8 8 SigOpt Whitepaper: The Optimization Bottlneck