Fighting the WAR on BUGs A Success Story. Duvan Luong, Ph.D. Operational Excellence Networks

Transcription

1 Fighting the WAR on BUGs A Success Story Duvan Luong, Ph.D. Operational Excellence Networks

2 The WAR Support Verification Investigation Design Construction Requirements

3 High Casualties of the BUG WAR Total Bug-related-casualties Count Customer Found System Test R&D Total Count Low Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May 100% Bug-related-casualties Rate Casualty Rate 80% 60% 40% 20% 0% Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May 3

4 Strategy for Fighting the BUG WAR Profile the bugs Leverage the best bug fighting practices S.M. Prepare the organization for fighting bugs Make bug fighting priority # 1 and S.B. B.T. M.Y. H.W. 4

5 Profiling the Bugs Architect/ Design 45 % 20 % Requirements/ Investigation 35 % % of defect introduced by activity Construction/ Test & Bad Fixes Reference: Caper Jones, "Applied Software Measurements", McGraw Hill, 1991 Reference: "CASE Make Strides toward Automated Software Development", Computer Design, January

6 Profiling the Bugs Life Cycle Phase (Where bugs introduced) Requirements Gathering Architecting Designing Implementing/ Coding Testing/ Bug Fixing Product Supporting Requirements/ Specifications Functionality HW Interfaces SW Interfaces User Interfaces Functional Description Process (Inter-process) Communication Bug Type Data Definition Module Design Design Logic Description Error Checking Design Standards Code Logic Computation Data Handling Module Interface Implementation Code Standards Test HW Test SW Development Tools, Integration SW 6

7 Leverage the Best Bug Fighting Practices 35 Bug Removal Efficiency 30 Bugs/ Kloc Errors/kloc Remaining Inspections Unit Test Integration System Test Op Demo Operations Removal 50% 20% 50% 40% 20% Efficiency Typical Bugs Detection Profile Reference: M.Dyer, "The Cleanroom Approach to Quality Software development", John Wiley & Sons, Inc.,

8 Leverage the Best Bug Fighting Practices Industry Software Bug fixing Costs Defect caught in JAD $100 (1 hr = $100) (Joint Application Development) Prototyping $100 (1 hr) Requirement review $100 (1 hr) Design inspection $150 (1.5 hr) Code inspection $ hr) Unit test $250 (2.5 hr) Function test $500 (5.0 hr) System test $1000 (10 hr) Field test $1000 (10hr) Reference: Caper Jones, "Applied Software Measurements", McGraw Hill,

9 Leverage the Best Bug Fighting Practices Test/Detection Activity Review Inspection Unit Test Integration Testing System Testing Performance Testing Stress Testing Alpha Testing Beta Testing Requirements/ Specifications M M H H H M H Functionality M M H H H H H HW Interfaces M M H M SW Interfaces M M H H M User Interfaces M M H H H Functional Description H H M M M Bug Type Process (Interprocess) Communication M M H Data Definition H H H M M Module Design H M H M Design Logic Description H H M Error Checking H H H M M Design Standards H H H M Code Logic H H H Computation M M H Data Handling M M H M M M Module Interface Implementation M M M Code Standards H H M H Test HW M M M M M M Test SW M M M M M M M Development Tools, Integration SW M M H (high), M (Medium): Level of detection effectiveness - Blank : Low level of detection effectiveness

10 Prepare the Organization for Fighting Bugs Training Tools and Support Setting Policy and just do it! 10

11 Prepare the Organization for Fighting Bugs Training 100% of all Engineers were trained in inspection/review Common Language Common Expectations Heightened awareness Moderators assigned and trained for all Sections in the Organization 11

12 Prepare the Organization for Fighting Bugs Tools and Support Purchased dedicated laptop and projectors for use in inspections and reviews. Developed the Logging tool to facilitate the recording of inspection/review data Created online help to improve the tool use-ability. Enable the use of inspection/review data for later analysis of the process/product quality: Built Bob Grady s defects classifications into the tool. Inspection Champions for the Lab was assigned to work closely with Engineers. Holding regular moderator meeting to communicate and continue to fine tune the inspection process and tools. Regular analysis on inspection/review data to determine the process effectiveness. 12

13 Prepare the Organization for Fighting Bugs Policy and Process Included inspection plan in the checkpoint reviews. Inspection champion identified for each section to decide the inspection/review requirements for mandatory deliverables. All new and changed code required inspection. Inspection/review data is tracked and published at Section and Lab level. 13

14 How to Fight Bugs Just do it! Inspection Results Inspection Effort (hours) Defects Found Defect Discovery Rate Bugs 0 Bugs C/S Defects August September October November December January February March April May June Review Results 408 SNSL Review Results C/S Defects C/S Defects August September October November December January February March April May June

15 How to Fight Bugs Implementation periods in the past 20 months First Period 5 months All Peer Reviews Code Inspections C/S Code Reviews & Deskchecks C/S Medium / Low Medium / Low Document Inspections C/S Medium / Low Document Reviews & Deskchecks Medium / C/S Low Latest Period 10 months Code Inspections C/S All Peer Reviews Medium / Low Document Inspections C/S Medium / Low Second Period - 5 months All Peer Reviews Code Reviews & Deskchecks C/S Medium / Low Document Reviews & Deskchecks Medium / C/S Low Code Inspections C/S Medium / Low Document Inspections C/S Medium / Low Code Reviews & Deskchecks C/S Medium / Low Document Reviews & Deskchecks Medium / C/S Low 15

16 Statistics in Fighting Bugs Inspection/Review Statistics in the past 20 months Total hour spent: Total bugs found: Total C/S bugs found: All bugs found efficiency: C/S bugs found efficiency: 2933 hours 2725 bugs (all severities) 464 C/S bugs.93 bug/hour ~ 1 hour per bug.16 bug/hour ~ 6 hour per C/S bug 16

17 How to Fight Bugs Inspection/Review Bug Types in the past 20 months Previous Analysis (5 months) 65% Code 1. Functionality 21% 2. Data Handling 18% 3. Logic 13% 4. Standards 13% Previous Analysis ( 5 months) 74% Code & Docs 1. Req/Spec 36% 2. Standards 12% 3. Functionality 8% 4. SW I/F, Func. Desc & Logic 6% each Current Analysis (10 months) 66% Code & Docs 1. Req/Spec 32% 2. Functional Desc. 11% 3. Module Design & Logic 8% each 4. Functionality 7% 17

18 How to Fight Bugs Inspection/Review Bug Causes in the past 20 months Previous Analysis (5 months) Code 1. Wrong 35% 2. Missing 29% 3. Better Way 24% Previous Analysis (5 months) Code & Docs 1. Missing 32% 2. Unclear 24% 3. Wrong 23% Current Analysis (10 months) Code & Docs Missing 35% Wrong 24% Unclear 15% 18

19 How to Fight Bugs Inspection/Review Bug Origin in the past 20 months Previous Analysis (5 months) Code 1. Coding 59% 2. Design 21% 3. Req/Spec 12% Previous Analysis (5 months) Code & Docs 1. Req/Spec 47% 2. Coding 37% 3. Design 10% Current Analysis (10 months) Code & Docs 1. Req/Spec 45% 2. Coding 26% 3. Design 18% 19

20 How to Fight Bugs 1 st Period Number of Bugs Number of Bugs Bug Infestation by Phase Requirements Design Coding Other Development Phase Bugs Found during Code Inspection User I/F Test S/W Standards Software I/F Req/Spec Other Module Design Logic Descr. Logic Locks Hardware I/F Functionality Functional Desc. Error Checking Data Handling Computation 20

21 How to Fight Bugs 2nd Period Bug Infestation by Phase Investigation Design Construction Verification Maintenance (Inter)Proc. Comm. Computation Data Def. Data Handling Error Checking Func. Descr. Functionality Locks Logic Logic Descr. Module Design Other Race Conditions Req/Spec. Software I/F Standards Bugs Found during Document and Code Inspections 21

22 70 How to Fight Bugs 3rd Period Bug Infestation by Phase Investigation Design Construction Verification Maintenance Investigation Design Construction Verification Maintenance Computation Data_Def. Data_Handling Error_Checking Func._Descr. Functionality Locks Logic Logic_Descr. Module_Design Module_I/F Other Race_Conditions Req/Spec. Software_I/F Standards User_I/F Bugs Found during Document and Code Inspections 22

23 Total Bug Found Profile Monthly Bugs Found Insp. & Rev. R&D System Test Customer Bugs 100% Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Monthly Bugs Found Distribution Bugs Distribution % 80% 60% 40% 20% 0% Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul 23

24 Bug Backlogs Statistics Internal Defect Backlogs Customer Defect Backlogs Defect Backlogs Q1Y1 Q2Y1 Q3Y1 Q4Y1 Q1Y2 Q2Y2 Q3Y2 Q4Y2 Q1Y3 Q2Y3 Q3Y3 Q4Y3 Q1Y4 Q2Y4 Q3Y4 Q4Y4 Q1Y5 Q2Y5 Q3Y5 Q4Y5 24

25 Inspection/ Review Savings 1 st 20 months Inspection/Review cost to find Cost if found during maintenance (in Engineering Hour) Peer Review Period Engineering Hours Engineering Hours Cost to Find 1 st period 2 nd period 3 rd period Cost to Fix Cost to fix where found by Inspection/Review Cost if fixed during maintenance Engineering Hours Peer Review Period Defect find savings Defect fix savings total savings Inspection/Review Savings st Period 2 nd Period 3 rd Period In the first 20 months of Inspection/Review implementation, saved approximately 15 Engineering years or $ 2.7 million. 25

26 We Won! Support. and the kingdom was cleared of Bugs once again!. Verification Investigation Design Construction Requirements

27 Next Step Support Verification Construction Design Investigation Requirements 27

28 Next Step Front-end focus on Requirements/Specification to reduce the amount of introduced bugs More focus on more usage and advanced Inspection practices to improve efficiency and ROI Improve code inspection to prevent and detect defects earlier Shift the focus of Inspection from Destructive - Bug finding to Constructive develop right product. 28

29 References Mike Dyer, "The Cleanroom Approach to Quality Software development", John Wiley & Sons, Inc., 1992 Bob Grady, Practical Software Metrics for Project Management and Process Improvement, Prentice Hall, 1992 Capers Jones, "Applied Software Measurements", McGraw Hill,