COGNITIVE QA: LEVERAGE AI AND ANALYTICS FOR GREATER SPEED AND QUALITY
ARTIFICIAL INTELLIGENCE vs. COGNITIVE COMPUTING Build a system that can generally perform any intellectual task so called Strong AI Build a system that perform can an intellectual task in smaller domain so called Weak AI Artificial Intelligence Emerging of Deep Learning with neural networks with many layers, new topologies and learning methods Machine Learning Cognitive Computing 1950 1960 1970 1980 1990 2000 2010 Ability to learn and build models, so that a system perform activities like prediction within specific domains Build a system that can learn and interact in natural way with humans 2
CONNECTING DISPARATES TYPES OF DATA WITHIN AND OUTSIDE YOUR WALLS CREATES NEW OPPORTUNITIES FOR UNEXPECTED INSIGHTS Customer records Transactional systems Predictive models Institutional expertise Operational systems Structured and active + Data you possess Data outside your firewall + News Events Social media Weather Geospatial information Data that s coming Internet of Things (IoT) Sensory data Images Video Unstructured and dark
WITH WATSON YOU CAN HARNESS THE POWER OF AUGMENTED INTELLIGENCE Your data + the world s data Unstructured Structured Watson can ingest, cleanse, and comprehend multiple data types. Build off your domain expertise and the largest base of industry offerings in the market. Ingests + transforms Transform Apply a powerful suite of Watson services and APIs to transform your data into valuable insights. Watson is optimized for robust AI workloads and lives on IBM Cloud. Insights + outcomes Insights Watson extracts meaning to provide deep insights and produces powerful outcomes. Train your AI on what s important to your industry gain knowledge and make the most informed, reasoned decisions.
IBM WATSON IS THE AL PLATFORM FOR BUSINESS Watson is the AI platform for business. With Watson you can discover hidden insights, engage in new ways, and decide with greater confidence.
BY APPLYING COGNITVE CAPABILITIES TO YOUR DATA, YOU WILL ENHANCE DIGITAL INTELLIGENCE EXPONENTIALLY UNDERSTAND REASON LEARN INTERACT Cognitive systems understand imagery, language and other unstructured data like humans do. They can reason, grasp underlying concepts, form hypotheses, and infer and extract ideas. With each data point, interaction and outcome, they develop and sharpen their expertise, so they never stop learning. With abilities to see, talk and hear, cognitive systems interact with humans in a natural way.
WHAT CUSTOMERS ARE SAYING How to ensure sufficient quality without slowing down on speed? Which test cases are relevant and which ones are not? How to improve testing with user behavior patterns? How to keep up with increasing complexity of intelligent applications? Intelligent test automation and smart analytics will become essential to support testing as they enable smart decision making, fast validation and automatic adaptation of test suites. 7 World Quality Report 2017-18 findings suggest
KEY QA CHALLENGES TODAY Increasing speed and complexity Insufficient Automation Test environments and test data Validating the increasing complexity of the application landscape is becoming more and more difficult Keeping pace with the demand for increasing speed to market is challenging Test sets are either missing up to 30% of relevant tests and/or containing many irrelevant test scenarios Automation is currently under-exploited in QA & Testing and as a result there is too much reliance on manual activities Traditional automation approaches have delivered only limited benefits Smart applications demand intelligent automation, integrated in continuous delivery Test environments and test data are the most cited challenges for core enterprise applications as well as more fluid IT applications Smart test platforms are required to deal with the growing challenges of test environments, data and virtualization Client focus inside out Usage patterns and user feedback are not exploited for QA purposes Customer experience validation is a challenge Social media and other external data sources are seldom leveraged for QA purposes
COGNITIVE QA VALUE Leveraging self-learning and analytical technologies for INTELLIGENT QA AUTOMATION Predictive QA Dashboards Smart Analytics for QA Intelligent QA Automation and Cognitive QA Platforms Enabling smart quality decision making on factual project data, actual usage patterns and user feedbacks SMART ANALYTICS FOR QA COGNITIVE QA COGNITIVE QA PLATFORMS Delivering quality with speed in a complex connected world at optimized cost. With Cognitive QA organizations will be able to achieve an accelerated and optimized quality by using an > intelligent approach to QA PREDICTIVE QA DASHBOARDS 9
DEVELOP A VISION AND ROADMAP Guidance to the journey towards quality with speed. Predictive QA Dashboards X-application transparency Near real-time insight What-if prediction Smart Analytics for QA Efficiency through focused Testing Testing aligned with actual usage patterns Improved test coverage Improved risk management Intelligent QA Automation Maximized RoI on test automation Focused test automation aligned with actual usage patterns Accelerated, touch-less testing. Cognitive QA Platforms Instant validation. Optimized test coverage with self adapting test suites. Minimized idle time and cost with self-aware and adaptive environment provisioning. Optimized risk management with self-learning capabilities. Future Proof QA Accelerated, intelligent QA leveraging AI and Analytics. Modular. Innovative. Business oriented. 10
PREDICTIVE QA DASHBOARDS Leverage all relevant data from Dev / Test / Ops repositories for: 360 o Visibility get comprehensive and transparent views on application quality on management and operational level. Tailor-made Insights customized for your specific demands to allow better control and steering of quality across all your projects. Near Real-Time Updates - refreshed continuously to enable timely and improved decision making. Zoom In drill down in details to do root cause analysis of real issues and analysis on expected outcomes. Alerts avoid surprises with pro-active notifications to minimize the risk of serious errors. What-if Analysis optimize test strategy in line with available resources and coverage requirements. Predictive Analysis leverage historical usage patterns by applying proven algorithms for defect prediction and test coverage optimization. 11
SMART ANALYTICS FOR QA Proven Analytics Solution Elements delivering Real Business Benefits Automated Test Case Selection and Prioritization gain efficiencies through focused testing effort. Predictive Test Environment Configuration minimize idle time with optimized environment provisioning. Test Automation Prioritization maximize ROI of test automation effort. Test Coverage Optimization make best possible choices based on what if analysis. Test Execution Analysis align test with actual usage patterns. Auto Generation of Test Scripts create optimized test suites while reducing preparation effort. Applying our IP > 40 tried and tested rules Rules of Thumb Common Sense Best Practices Intuitive Judgments Positive / Negative Patterns Implemented step-by-step to solve your Business Challenges Kick-Start Collecting basic information using Cognitive QA Pre-Assessment questionnaire. Checking availability, accessibility and quality of data. Performing Solution Build Workshop to define the scope of POV. Proof of Value Conducting Proof of Value with following steps: Verify analytics readiness. Data collection and preparation. Modeling Cognitive QA rules. Validation and reporting. Define the deployment approach. Deployment Operationalizing rules implemented during PoV. Major activities involved: Implement connectors to pull live data. Implement orchestration engine to automate workflows. Implement portal (responsive web). 12
SOLUTION CONCEPT OF SMART ANALYTICS App. Logs and traces Non-structured Data Data Rules Non-structured Data Test Mgmt System Defect Mgmt System Support engr call notes Usage patterns RTM Systems Operations Repositories structured Data Grouper Software Lifecycle Artifact Quality structured Data Test & Dev Repositories Feedback Source ctrl /Build Mgmt System Project Mgmt System Code Quality Parameters Req. Mgmt system Test Results TestOps Rules Self Learning Recommendation Engine Test Engine Optimized Test list based on real time data
TOOLS AND TECHNOLOGIES IN-HOUSE ASSETS Automatic Regression Test Prioritization and Execution PoV in a box -assets Pre-assessment questionnaire Solution Build workshop template Assessment checklists Cognitive QA rules Report template Customizable Dashboard with Standard Metrics Proof of Value references INDUSTRY TECHNOLOGIES 14
CASE STUDY INDUSTRIAL AUTOMATION OEM CUSTOMER Dynamic Regression Test Selection PoV duration 10 weeks 6 Cognitive QA rules implemented The Total number of Test Cases were 22,289, and regression identified for the current release was 8,773. We recommended 2,288 regression Test Cases. MEDICAL IMAGING SOLUTION PROVIDER Dynamic Regression Test Selection PoVduration 8 weeks 6 Cognitive QA rules implemented The Total number of Test Cases were 13, 721 We recommended regression Test Cases of 1, 664. Selection of Configurations and Tests that matter PoV duration 10 weeks 8 Cognitive QA rules implemented We recommended: 325 high priority & 571 low priority test suites of 896total test suites. We recommended : 44 high priority & 141 low priority configurations among 192 Configurations. KEY BENEFITS KEY BENEFITS KEY BENEFITS ~73% effort saving, shorter release cycle Team gets more time to automate The high risk test cases are executed in the beginning of the test cycle, early detection of regression defects. Fixing the identified tool and process gaps will prepare for better Analytics Readiness. Improved Adaptive Regression Test Selection strategy factoring multiple inputs. More quality builds for special purpose testing like system testing, performance testing. Recommended test list is being automated and made part of regular automated test executions. Fixing the identified tool and process gaps will prepare for better Analytics Readiness. Helped identify 63% test cases that can be deprioritized. Derived critical tests and configurations with high probability of risk. Systematic way of System under Test configuration planning. Better utilization of the hardware/ Cost etc. 15
THANK YOU 16
BACKUP 17
THERE ARE LOTS OF USE CASES Tests that matter! Automated Test Selection & Prioritization Predictive test environment configuration What should we automate? Improve coverage! Optimize and align test sets based on real application usage Test Coverage Optimization Auto Generation of Test Scripts Release Readiness Predict and advice marginal value of additional testing Automated impact Analysis Predict risk level of product components / services / systems (incl. UX) Predictive capacity planning Auto-Heal Throw away the Bad/ obsolete test cases (future proof testware) Isolate Script Failures Throw away the bad code fixes in CI pipeline Quality Dashboard Cost of a bug, Test efficiency, Release Dashboard Predictive quality metrics Resource utilization
RULES SELECTION