QUEST Boston Full Lifecycle Testing of Commercial off the Shelf (COTS) Software. Thursday, April 7 th, :00 AM 12:00 PM

Transcription

1 Thursday, April 7 th, :00 AM 12:00 PM QUEST Boston 2011 Full Lifecycle Testing of Commercial off the Shelf (COTS) Software PRESENTER: Saeid Vakili COMPANY: Ontario Ministry of Education

2 This was page intentionally left blank

3 Full Lifecycle Testing of Commercial-off off-the-shelf (COTS) Software Saeid Vakili, B.Sc., CSQA Business Solutions Services Manager Ontario Ministry of Education Agenda Project Summary Why COTS? COTS vs. In-House Challenges SDLC and Best Practices Testing Solution Strategy Project Examples Conclusion Questions 1

4 Project Summary Employment Ontario On Jan 1, 2007, the transfer of Federal labour market programs to Province of Ontario presented an opportunity to rethink the policy, design and delivery of labour market programs to more effectively meet Ontario s labour market and training objectives. Employment Ontario Information System An integrated information system built to support all employment and training programs and services of the Ontario Ministry of Training, Colleges and Universities. Project Summary Case Management System The delivery of all employment and training services is supported through the implementation of a Case Management System, a web-based Commercial-off-the-shelf (COTS) application. COTS Software The Cúram Enterprise Business Application Suite has been selected as the COTS solution for implementing the Case Management System. 2

5 Why COTS? Ready to use solution Overall reduced development and maintenance cost Predictable license costs Time to market Widespread use leading to greater reliability Comprehensive functionality COTS vs. In-House Similarities Still a development project Infrastructure, Design and Development effort Testing effort Differences No Business and/or Functional Requirements No Architecture COTS is a Black Box 3

6 Challenges Requires a shift in testing MINDSET and APPROACH Unknown Design and Development Processes and Methods by vendor Unknown level of Quality Integration Issues Compatibility Issues Upgrades and Patches Vendor Support SDLC -COTS Fit Gap Analysis Customization & Configuration UAT Integration Test 4

7 SDLC & Best Practices Requirement Phase Business requirements are defined in the form of Business Processes Fit Gap Analysis The result of Fit Gap Analysis activity leads to the design phase Best Practices Get involved in the Fit Gap Analysis exercise Get training Get access to Runtime version of the software and learn the application SDLC & Best Practices Architecture & Design Phase Application architecture, design and data model are given The focus is on application customization, configuration and integration based on the output of Fit Gap Analysis The idea is to use OOTB functionality as much as possible Best Practices Base tests on functional or business processes used in the real world environment Clear distinction between OOTB and customized functionality Define high level business test scenarios by following the 80/20 rule 5

8 SDLC & Best Practices Development Phase Limited or no coding Configuration & Customization Development integration; i.e. Build Management Best Practices Adopt a modular or phased development approach Automated, centralized and continuous build integration process Centralized Change Management Process SDLC & Best Practices Testing Phase Limited or no unit test Functional testing is redundant by its very nature The focus is on E2E integration test The primary test of COTS product is Validation The level of quality is unknown Best Practices Avoid the idea of "testing everything Base your tests on real-world functional and business processes Manage expectations regarding the level of quality Automate as early as possible 6

9 Testing Solution Strategy Test Planning Clearly define the testing approach in regards to application functionality; i.e. Customized vs. OOTB functionality Identify the integration points within the application and outside of the application Define the change management, release management and defect management processes Define the testing approach in regards to application upgrades, service packs and patches Define the test data creation strategy Define the automated, regression and performance test strategy Testing Solution Strategy Test Cases There is no business requirements or functional specifications to create test cases from Test cases are created based on business process flows and scenarios QA staff need to learn the application and business in order to create proper test cases Business team need to get involved in creating and reviewing test cases Application s user guide, help and training material are helpful sources for creating test cases 7

10 Testing Solution Strategy Test Phases It all comes down to two test phases: Integration Test and User Acceptance Test Unit and System Functional Tests are either redundant or limited There are two levels of integration test: Application Integration Test: the integration between OOTB and customized functionality System Integration Test: the integration between the application and other systems Testing Solution Strategy Test Types Functional Test Business Process Test User Interface Test Regression Test Non-Functional Test Performance Test Security / Intrusion Test Compatibility Test Accessibility & Compliance Test Globalization Test 8

11 Testing Solution Strategy Test Execution Adopt a modular or phased testing approach Start testing the customized and complex functionality first Start the security testing as early as possible; i.e. Authentication and Authorization testing Product service packs, upgrades and patches require great deal of regression testing Have a clear strategy about logging and fixing defects related to customized vs. OOTB functionality Have a solid test data creation strategy Perform multi-platform testing Testing Solution Strategy Automated & Performance Test Start automation as early as possible Use automated scripts for regression test, test data creation and compatibility test Create scripts for each major functions and then run the critical business processes as your performance scenarios Use a real workload distribution model The main focus of performance test is on the infrastructure and integration points Consider vendor s recommendations for improving performance 9

12 Project Examples Critical Business Processes & Integration Points Project Examples Test Cases About 200 different test scenarios created With all data variations, about 600 individual test cases created Test Execution Functional Test: there was a requirement to capture SIN as one of the Client s information: Positive and negative tests performed for this new field The integration of this field with other modules was tested Application Integration Test: Look up for Postal Code using a third party application to populate client s address information System Integration Test: integration with login, payment system and reporting 10

13 Project Examples Test Automation All 200 different test scenarios were automated Automated test scenarios used to create test data Within 2 days the data for entire application was populated for business in their Training environment (600 clients) Performance Test All critical business processes and functions were scripted Performed the test for 250 concurrent virtual users Most of the performance issues discovered and tuning performed were on the infrastructure and integration points; e.g. servers configuration, database connection pool, etc. A few OOTB application functionality related issues found as well Conclusion Invest in training and learning the COTS at the beginning of the project Testing is based on Validation and customer s definition of Quality Fit for Use Avoid the idea of testing everything It s all about E2E Integration Test Beware of Architecture and Design limitations and design your tests accordingly Invest in Automated Testing early in the project lifecycle including automating the test data Be mindful of product s upgrades, patches and service packs Involve business in testing 11

14 Questions 12