MCGILL UNIVERSITY Montreal, Quebec September 20 21, A DMAIC Framework for Improving Software Quality in Organizations: Case Study at RK Company

Transcription

1 MCGILL UNIVERSITY Montreal, Quebec September 20 21, 2016 A DMAIC Framework for Improving Software Quality in Organizations: Case Study at RK Company

2 Team Composition Racha Karout Anjali Awasthi

3 Outline 1. Introduction 1.1 Background 1.2 Problem Definition 1.3 Research Objectives 2. Literature Review 3. Solution Approach 4. Conclusions and Future Works

4 1.1 Background To compete in today s world, every business needs to improve. Software has increasingly become a critical component in many industries (telecoms, banking, insurance, etc.). Software quality is crucial and poor quality is not acceptable. Software development has not been consistently successful.

5 What is Software Quality? Software quality is the degree to which a system, component or process meets specified requirements, in other words, the degree to which a system, component or process meets customer or user needs or expectation (IEEE, 1991). The software should not have bugs that reduce the quality attributes (functionality, reliability, usability and maintainability) (Chang et al., 2006). There should not be issues that affect its ability to maintain or re-establish its level of performance. Easy to use and maintain.

6 1.2 Problem Definition In today s market competition and the need for rapid delivery, software quality is often sacrificed, leading to the failure of the software project The use of traditional methodology (waterfall) with the current market pace, continuous change of customer requirements and rapid development of technology plays a major role in poor software quality. People jump to solutions without fully understanding the problem or finding the root cause of poor quality.

7 1.3 Research Objectives 1. Improve software quality in organizations. 2. Identify the root cause(s) of the problems that result in poor quality. 3. Identify improvement opportunities based on the root cause(s) of poor quality. 4. Show how companies can control their processes to ensure the improvements applied are directing the project to the expected level of quality.

8 2. Literature Review Six Sigma Common Software Quality Problems Software Development Processes Kanban Test Automation Metrics and Reports Research Gaps

9 2.1 Six Sigma Is a business process improvement approach that seeks to find and eliminate causes of defects and errors, reduce cycle times and cost of operations (Evans & Lindsay, 2005) It is based on a simple problem solving methodology DMAIC (Hung & Sung, 2011) Quality principles and six sigma: 1. Focus on customers 2. Participation and teamwork 3. A process focus supported by continuous improvement.

10 2.2 DMAIC Framework Define: clearly define the problem Measure: gather facts, collect data and observations and measure the internal process. Analyze: identify the root cause(s) of the problems and focus on why defects occur. Improve: generate and implement ideas for improving or resolving the problem. Control: focus on maintaining the improvements.

11 2.3 Common Software Quality Problems During project implementation both technology and business environment change. This leads to products that do not meet the needs of customers. (Williams & Cockburn, 2003) Customers are only involved during requirement collecting in traditional software development (Hossain et al., 2013), while in today s market demand, customer requirements continuously change. In the traditional plan-driven software development process, work is coordinated by managers (Moe et al., 2010), thus giving software development team less control (Stankovic et al., 2013) When developers and testers do not develop a shared mental model, they will not work together to address the project issues (Yu & Petter, 2014). Late discovery of bugs in the traditional software model leads to an increased cost to fix the faults (Monassar & Govardhan, 2010)

12 2.4 Software Development Processes - Waterfall Requirement Analysis System Design Implementation Testing Deployment Maintenance

13 2.5 Software Development Processes Agile Agile promotes quick response to changing environments, change in user requirements and accelerated deadlines (Eriksson et al., 2005). It focuses on four core values 1. Individuals and interactions over processes and tools. 2. Working software over comprehensive documentation. 3. Customer collaboration over contract negotiation. 4. Responding to change over following a plan (Dyba & Dingsoyr, 2008).

14 How can agile techniques enhance software quality System metaphor: It is helpful for communication between the team members and users. It enhances maintainability, efficiency, reliability and flexibility. Architectural spike: It helps to determine and test potential architecture. It enhances correctness. Onsite customer feedback: It helps developers refine and correct requirements. It enhances responsiveness to changing customer requirements. Refactoring: It improves software quality factors such as efficiency, reliability and testability. Pair programming: It improves design quality factors such as correctness, verifiability and testability and reduce defects. Stand-up-meeting: It improves software reliability and flexibility. Continuous integration: It improves integrity, usability and testability. Acceptance testing: It improves testability.

15 Agile Methods - Scrum

16 2.6 What is Kanban System? Number of cards equivalent to the capacity of a system are placed in circulation. It is a pull system as new work is pulled into the system when there is capacity to handle it, rather than being pushed. The core properties that Kanban uses are: 1. Visualize workflow 2. Limit work in progress 3. Measure and manage flow 4. Make process policies explicit (Anderson, 2010)

17 2.7 Test Automation It is the process of writing a computer program to do testing. Once test have been automated, they can be run quickly and repeatedly (Hooda, 2012) Benefits of test automation: Defects can be found at an early stage. Can discover defects that manual testing can t find. Helps increase test coverage. Helps increase management confidence in the software. It executes significantly faster than human users. Can be reused to test different versions with different configurations or platforms. With increased test coverage, more bugs are prevented from escaping to production which in turn reduces cost radically. Can be considered as a safety net for developers especially in case of necessary code refactoring.

18 2.8 Metrics and management Reporting The purpose is to show that the system is predictable, the organization exhibits business agility, and there is focus on flow, and there is clear development of continuous improvement Some of the metrics that can be used: Tracking work-in-progress Lead time Throughput Defect removal efficiency DRE=E/E+D

19 2.9 Research Gaps Subramanian et al. (2007) conducted a study that examines how CMM and the IS implementation strategies impact software quality and project performance. They mentioned a limitation in their study that is focused primarily on CMM and mentioned other methods such as Total Quality Management and Six Sigma could also be considered in empirical research. Card (2004) argued the need for more academic research in software process improvement methods. However, this thesis focuses on using six sigma DMAIC framework to improve software quality in organizations and as a consequence ensures the success of the software project.

20 3. Solution Approach RK Company Description Define Phase Measure Phase Analysis Phase Improvement Phase Control Phase

21 DMAIC - Tools Phase Define Measure Analyze Improve Control Technique CTQs SIPOC diagram Pareto charts 5-Why technique Cause and effect Interrelationship diagram QFD Measurement metrics

22 4. RK Company and DMAIC One of the fast growing software companies in Canada (Medium size) DMAIC is applied to one of the biggest and core projects The project has been in development for the past few years. Waterfall methodology. Team consists of: Six sub development teams System test team (test the product manually) Test automation team (Current focus is to create test tools for the project) Data collection: Interviews conducted with development manager and QA manager. Production bugs were collected from the reporting system and classified by type, severity and seasonality.

23 4.1 DMAIC Define Phase - CTQs Critical to quality (CTQs): are what the customer expect of a product or service. The most important quality attributes for the project under study are: Security Reliability Assurance Efficiency Maintainability

24 DMAIC Define Phase SIPOC Diagram Supplier Input Process Output Customer Organization Test Manager Tool Create manual test cases for new planned features New feature test suite Project manager QA team High level requirements Execute test cases for each fully implemented new feature New feature test results document (HLR) Development QA manager Report bugs for new features List of identified bugs for new feature manager System requirements document Development team Verify bug fixes for new features Modified build after bug fixes QA manager Project stable build Development manager Retest all new features when all new planned features are Regression test results Organization Testing Tools fully implemented. Project manager Updated test cases Testing environment Perform regression testing Updated regression test suite to include Test plan Report bugs found in regression priority one test cases of newly implemented features Regression test suite Verify bug fixes for regression testing Legacy system knowledge Perform manual test case maintenance (clean up)

25 4.2 DMAIC Measure Phase Pareto Chart Pareto chart based on the type of errors: Pareto chart based on seasonality: Pareto Chart Pareto Chart % % % 80% 60% % 80% % 20% % 0 0% % % % Number of bugs Cumulative Percentage 80% Marker Number of Bugs Cumulative Percentage 80% Marker

26 DMAIC Measure Phase Pareto Chart Pareto chart based on the severity: Pareto Chart Medium Major Minor Critical Number of Bugs Cumulative Percentage 80% Marker 120% 100% 80% 60% 40% 20% 0% RK company should focus on fixing functional and backend services. Years 2013 and 2014 have the highest number of bugs and testing and bug fixes should be concentrated on the features that got released in these 2 years as well as the affected features. Medium and major bugs should have the main focus. However, the focus should also be on critical bugs as they should not be even found in production.

27 4.3 DMAIC Analysis Phase 5 Why Technique 1. Why there is too many bugs in production? o Because it wasn t tested properly. 2. Why it wasn t tested properly? o Because many regression and performance tests were cut. 3. Why test cases were cut? o Because the test schedule is not adequate for the number of tests that should be performed 4. Why there wasn t enough time to execute all necessary tests? o Because developers never deliver on time. 5. Why developers never deliver on time? o Because of the may features that are added at each release and in some cases client requirements or new features are added or UI changes are done in the middle of implementation leading to a change in the scope and the current methodology does not properly adapt to the requirements change.

28 DMAIC Analysis Phase Cause and Effect Diagram Requirements Methodology Flow Test schedule Scope creeps Manual Not followed Many legacy regression No single owner Long sprints Many new features Continuous change Change adaptation Late delivery Poor quality Not detailed No explanation limited No frequent update Obsolete Friction Training Investigation No Fix No Updates Documentation Communication Tools

29 DMAIC Analysis Phase Interrelationship Diagram Large number of bugs found in production / Poor Quality Requirements Methodology Flow Testing Schedule Documentation Communication Tools

30 DMAIC Analysis Phase Conclusion Several tools were used to separate the symptoms from the causes to identify the root cause of the problem. Several issues were identified such as: Developers rarely deliver a stable build on time to system test team The sprint is too long. Continuous change of customer requirements. The project is large and complex. Leading to the discovery that the current waterfall methodology uses is the main cause of the large number of bugs in production

31 4.4 DMAIC Improvement Phase Agile 1) Move from waterfall to agile methodology to gain: a) Flexibility: to be able to adapt quickly and effectively to changing requirements b) Cycle time reduction: to speed up work so that customer response is improved and to reduce non-value added steps that leads to a reduction in cost. As a result, cost, quality and productivity are improved.

32 DMAIC Improvement Phase Scrum 2) Use scrum methodology that allow splitting a large team into smaller subteams which is already the case of the project under study. Scrum implementation: Start with daily stand up meetings First sprint objective is to demonstrate any piece of user functionality. Focus on sprint procedure (sprint planning, daily scrum, review meetings and retrospective meeting).

33 DMAIC Improvement Phase Kanban 3) Use Kanban together with scrum to improve service delivery, catalyze continuous improvement and achieve constant pace. Kanban implementation: Define start and end point of process visualization Identify the type of work that arrive and exit with the workflow Draw the cards wall Allocate capacity within the Kanban system. Design the cards to facilitate the pull system.

34 DMAIC Improvement Phase Test Automation Test automation is a necessity in agile How to start automating: Have a good understanding of the product and technologies used. Determine test cases to be automated. Be involved as early as possible in the development life cycle. Select suitable tool(s) for automation. Have a good understanding of the team member s level of experience and skills.

35 DMAIC Improvement Phase Good Quality Test Automation Have single objective per test case. Keep the test small Keep test cases independent Group and organize test cases in a specific logic. Avoid Redundancy of test cases. Centralize reusable test code. Avoid fragile automation test tools. Automate test in parallel with feature implementation Enhance test execution time by parallelizing the test run.

36 4.5 DMAIC Control Phase Metrics and Reports Measure the process and results and take corrective actions when necessary. RK company can measure and monitor performance by: Tracking the work in progress. Measuring lead time. Measuring throughput. Tracking number of bugs found in production. Measuring defect removal efficiency.

37 Conclusions and Future Works

38 5. Conclusions Strengths: 1. Focus on eliminating inefficiency 2. Assist in identifying the root causes of defects 3. Assist in executing quality improvement efforts Weaknesses: 1. Requires total cooperation of the company 2. Requires significant amount of data collection and analysis Opportunities: 1. Distinction in competitive market 2. Provides customer-driven excellence 3. Improves return on investment Threats: 1. Dysfunctional organizational culture 2. Lack of creativity and cooperation of the workforce.

39 6. Future Works Find ways to reduce cycle time especially for large features. Find proper tools and methods to facilitate the design phase in SDLC. Have improved and reliable test automation tools. Improve agile processes with varying team members expertise.

40 Thank You!