Jack Weast Principal Engineer, Chief Systems Engineer Automated Driving Group, Intel
From the Intel Newsroom 2
Levels of Automated Driving Courtesy SAE International Ref: J3061 3
Simplified End-to-End Automated Vehicle Architecture In-Vehicle Data Center Automated Driving Functions Vehicle Endpoint Management Sensor Processing Sensor Capture Vehicle Simulation & Validation Captured Sensor Data Analytics Data 4
What Makes an Automated Vehicle Work? Sensing Sensor Processing Sensor Fusion Environment Modeling HD-Map Correlation Maneuvering Decision Making Path Planning Trajectory Enumeration Localization
What is Happening in the Data Center? Development of Automated Vehicles *starts* in the Data Center There you store Data and Lots of it!!! The first step towards developing your algorithms for autonomy Example Data: Raw Sensor Data Vehicle Bus Traffic Environmental Conditions Scenario Under Capture Driver Name / Date / Time More Data
How Does the Data get There? Test Fleet Vehicles produce an incredible amount of data PetaBytes per hour! Sneaker Net! Physical Ingestion is really the only option Commercially Deployed Vehicles produce all the same amount of data But it is used inside the vehicle Intelligent Anomaly Detection The trick is how to know when to capture interesting data for later cloud processing
What else is Happening in the Data Center? All of that uploaded data is used to train our Deep Neural Networks DNN s support key use cases like Pedestrian Detection Traffic Sign Detection Visual Object Detection And, emerging use cases like Driving Policy Model Training Multi-node / IA-optimized Frameworks Captured Sensor Data Analytics Big Data Analytics Statistical Trends Neural Network Design for Target Hardware, and usage (Vision, Data Driven, etc.) Simulation >than Real Time Model Simulation & Verification Intel Architecture (IA)
What is else is Happening in the Data Center? All of that uploaded data is also used for big data style statistical analysis Statistical Analysis can provide: Longitudinal usage patterns Insight into vehicle performance Personalization The real value here is the intersection of Big Data and Deep Learning Model Training Multi-node / IA-optimized Frameworks Captured Sensor Data Analytics Big Data Analytics Statistical Trends Neural Network Design for Target Hardware, and usage (Vision, Data Driven, etc.) Simulation >than Real Time Model Simulation & Verification Intel Architecture (IA)
What else is Happening in the Data Center? But how do I know if the algorithm works? You need to test it on new, labeled data that was not previously used for training Simulation >> faster than Real Time mechanism to test the proposed vehicle implementation Crucial to do before deploying to the real world! Captured Sensor Data Analytics Big Data Analytics Model Training Multi-node / IA-optimized Frameworks Statistical Trends Neural Network Design for Target Hardware, and usage (Vision, Data Driven, etc.) Simulation >than Real Time Model Simulation & Verification Intel Architecture (IA)
Introducing Intel Deep Learning SDK Intel DL Deployment Tool Intel DL Training Tool configure_nn(fpga/cve, ) allocate_buffer( ) fpga_conv(input,output); fpga_conv( ); mkl_softmax( ); mkl_softmax( ); IMPORT Trained Model (trained on Intel or 3 rd Party HW) COMPRESS Model for Scoring on Target Intel HW GENERATE Scoring HW-Specific Code (OpenCL *, C/C++) INTEGRATE with System SW / Application Stack & TUNE EVALUATE Results and ITERATE INSTALL / SELECT IA-Optimized Frameworks PREPARE / CREATE Dataset with Ground-truth DESIGN / TRAIN Model(s) with IA-Opt. Hyper-Parameters MONITOR Training Progress across Candidate Models EVALUATE Results and ITERATE Optimized libraries & run-times (MKL-DNN, OpenVX, OpenCL) Data acquisition (sensors) and acceleration HW Target Scoring Hardware Platform (physical or simulated) MKL-DNN Optimized Machine Learning Frameworks Intel Xeon Workstation or Cluster (local or cloud) https://software.intel.com/en-us/deep-learning-sdk Intel Architecture (IA)
Endpoint Management & Deployment Connected Vehicles represent a significant Endpoint Management Challenge! Vehicle Endpoint Management Our Goal is to deploy early and often, over the air, updated models and other driving policy preferences Model Compression is the process of reducing the size (layers) of the network while retaining accuracy Compression Other methods could include some amount of re-training within the vehicle
Putting it All together: An End-to-End Architecture In-Vehicle Automated Driving Functions Trajectory Enumeration, Path Planning, Selection & Maneuvering Driving Policy, Path Selection Real Time Environment Modeling Localization Sensor Processing and Fusion Object ID & Classification Anomaly Detection Compressed DL Model Real Time HD Map Updates 5G OTA SW /FW Updates Data Formatting, Annotation Annotation Tools Data Storage Dataset Management and Traceability Data Center Endpoint Management Geographical Tracking, OTA Updates Neural Network / Algorithm Design Model Training Multi-node optimized Frameworks Big Data and Statistical Analytics Simulation >than Real Time Model Simulation & Verification Captured Sensor Data 13 Sneaker Net Test Fleets
Introducing Intel GO Automated Driving Solution
But what does all this mean for the In-Vehicle System Architecture? 15
Key Challenges in Enabling Autonomous Vehicles CHALLENGES Computational Performance and Power Efficiency Data Center Compute in a limited power envelope Real-time processing for vehicle control / collision avoidance Safety and Security for increased levels of automation Flexible & Scalable Platform that scales across vehicle models from standard to luxury (price & performance)
The ROI Conundrum for Developing Custom Logic Revenue ROI Required to maintain R&D costs at 20% of Revenue IC Development Costs*: IC Hardware Software Validation $ Millions $1,605 $782 $262 $175 $35 $52 $88 $439 $156 $321 17 1 2 3 4 5 6 7 8 9 10 65nm 40nm 28nm 20nm 14nm * Development Cost Source: IBS
Automotive Radio Data Center FPGAs are Multi-function Accelerator Platforms Pre-distortion CPU Hardware accelerators provide breakthrough compute performance Sensor Fusion Pedestrian Detection FPGA Accelerator Accelerator Re-configurable to adapt to a wide variety of workloads Performance-per-Watt gains with highly paralleled architecture 18 Machine Learning
FPGA: Automated Driving Flexible Accelerator AD Heterogeneous Architecture The Missing Puzzle Piece FPGA Workload Deep Learning Grid Fusion Object Classification Path Planning Sensor Fusion FPGA Attributes Hard Floating Point Deterministic Architecture Embedded Memory & DSP Security Architecture FPGA Value Performance / Watt Real-Time Processing Safety and Security Flexible Accelerator
The Only Timeline That Matters Is When People Are Ready for Automated Driving
Without Trust, Automated Vehicle Adoption Won t Happen Trust means we feel: Safe Comfortable Confident in control
How Do AVs Gain Our Trust? Good listening Open dialogue Prompt action
Dr. Nicholas Epley The Mind in the Machine To the Car: A Name, Gender, and Voice.
So what are the key system elements of trust?
SENSING Visual, auditory, and motion sensors help the system listen actively both inside and outside the vehicle.
COMMUNICATING Touchscreens, voice controls, communication screens, and lighting give the vehicle multiple ways to communicate in multiple directions with passengers.
RESPONDING TO CHANGES Automated Vehicles must react quickly and effectively in response to driving conditions and routing requests.
We Are Building a Trustworthy Platform. Key interactions and capabilities Technical implications Understanding what to optimize
Summary Automated Driving is an End to End Architecture and Intel GO Automated Driving Solutions has you covered Key Technical Challenges in Automated Vehicles can be solved through use of FPGAs as Flexible HW Accelerators for evolving workloads Human afforded Trust is what will decide when we as humans are ready for automated vehicles
Incredibly scalable in-vehicle computing Wide range of reliable, available, and secure compute Collaboration with Wind River and others to deliver functionally safe OS, software, tools Layered security from chip to cloud with features rooted in hardware and support for secure over-the-air updates Ideal combination of sequential and parallel computing Powerful and efficient Intel Atom and Intel Xeon processors for sequential computing Arria FPGAs for a powerful, cost-effective, scalable design platform Hardware acceleration technology for computer vision and machine/deep learning
Suite of tools for automated driving software application developers Support for in-vehicle code and cloud development Machine learning/deep learning, computer vision algorithm development on Intel architecture and accelerators Acceleration libraries, compilers, debuggers and IDE Deep Learning, Computer Vision, and Sensor Fusion Tools Sensor Data Tool [NEW] DL Training and Deployment Tools Intel-optimized DL Frameworks OpenVX* Kernel Library and Graph Builder FPGA & Heterogeneous Programming Tools FPGA OpenCL Full Stack Optimization Tools and Libraries Compiler (ICC), JTAG debugger Intel Vtune Amplifier, Thread Check, Thread Profiler Performance and Threading Libraries (Intel MKL, IPP, TBB) In-Vehicle Platform Tools Yocto* Recipe Board Flashing Utility
Industry s first 5G-ready automotive platform Rapid development and testing of 5G solutions Target use cases: HD map downloads in real time HD content for in-vehicle infotainment Over-the-air updates Sensor uploads from vehicle, for machine learning Safety, smart intersections, cooperative driving
Introducing Intel GO Automated Driving Solutions In-Vehicle Development Platform for Automated Driving Automotive 5G Platform Intel Data Center Solutions Unmatched scalability and performance per power Available 1H 17 Automotive Software Development Kit (SDK) 1st 5G ready platform for automotive development Available Feb 2017 >97% of servers deployed form machine learning workloads powered by Intel