SPIN AND RE-SPIN A WEB TO CATCH ALL POSSIBLE BUGS: A NEW WAY TO BUILD AND CONTINUOUSLY REFINE A PROCESS PERFORMANCE MODEL

Transcription

1 SPIN AND RE-SPIN A WEB TO CATCH ALL POSSIBLE BUGS: A NEW WAY TO BUILD AND CONTINUOUSLY REFINE A PROCESS PERFORMANCE MODEL NAME: Bin Cong with co-authors Eric Zhou,Kaijian Xu TITLE: Chief Process Architect and Professor ORGANIZATION: CRS Tech and Cal State Fullerton

2 Acknowledgement and the context of the work presented Mr. Eric Zhou and Kaijian Xu from Aviation Motor Control System Institute (AMCSI) in China made major contributions to the work presented. The context of work presented 1. AMCSI develops system solutions for engine control whose Software Development Dept. (the OU) focuses on embedded software development and maintenance; 2. Software Development Dept., consists of 7 professional teams (requirement, design, coding, interface, testing, QA, PA), and all software related work comes from the System Dept who will provide software development and maintenance requirements; 3. The lifecycle of a typical system (product) is over 20 years, thus majority of software work is maintenance not new system development. Majority of software developed and maintained by OU is level A; 4. Quote from the Top Manager: It would not matter that much for me if we deliver on time and if we save some money for our customers, but we will face severe consequences if a plane crushes due to bugs caused by our work.

3 ORGANIZATION OVERVIEW The basic idea Skynet: the framework and its usage in maintenance projects Skynet for new release development Quality debt driven mgt and improvement Conclusion and future work

4 THE ULTIMATE QUALITY GOAL Figure from DO-178C No leaked software defect that would cause a catastrophic failure condition for the aircraft is allowed!

5 QUALITY VS. PROCESS Development Figure from DO-178C Post-Development DAL(n)=F(O1,O2,,O71) n={a,b,c,d,e} Om = {0,1} Figure from DO-178C DAL A: 10^-9 Figure from ARP4761 Structural Quality Deployment According to DO- 178C, to ensure the software quality can reach level A, we have to satisfy 71 objectives which comes from 10 processes.

6 Process Partition LOCALIZATION Development 1. System Requirement allocated to Software 2. Constraints Post- Development Defect Leakge =0 REQ DES Coding INT SysT Criteria Definition {Xr} {Xd} {Xc} {Xi} {Xt} {Xn}: Process Quality Attributes Y(Defect Leakage) = F({Xr},{Xd},{Xc},{Xi},{Xt})

7 Change Request THE FLOW TO IMPLEMENT A CHANGE REQUEST Create Ticket Update Code Verify Change Obj Code User Verify CR Dev Sol. Sol. Rev. Chg Code Debug Stat. Ana. UT Test Req. Dev. Test Case UG Test Fnl Rev. The major activities performed C -> U -> V IN VS OUT {X1} {X2} {X3} Defect Leakage=0 Two Types of Quality Attributes: P (Preventive): Contribute to Defect Injection Prevention A (Assessment): Contribute to Defect Detection Yn = F({X1},{X2},{X3})

8 SKYNET ( 天网 )IS a web that will catch all bad things or bad people in Chinese. 天网恢恢疏而不漏!

9 THE FRAMEWORK OF SKYNET FOR CR DEVELOPMENT CR Gate 1 (QPPO) Gate 2 (QPPO) Gate 3(QPPO) Eva Risk Eva Risk C U V Eva Risk 缺陷零泄漏 缺陷零泄漏 缺陷零泄漏 缺陷零泄漏 缺陷泄漏 0-defect PPBs-> Sub-PPMs-> 过程基线 过程性能模型 变更方案评审缺陷密度 : (0,0.23) 个 / 条变更 - C 变更方案评审缺陷密度 : 缺陷密度 = 评审规模 评审速率 - C 升级验证缺陷密度 : (0,5.16] 个 /KLOC - V 升级验证缺陷密度 : 缺陷密度 = 测试用例覆盖率 测试执行工期紧张度 - V Quality Attributes Set: P and Decrease defect injection, Increase defect detection 迭代回归完善准则 天网 泄漏缺陷的类型 需求理解错误 派生需求错误 Leaked Defect Types QPPOs 数据检查缺失 初始化错误 <-PPBs 数据流错误 库函数使用错误 库函数错误 关联更改

10 RELATIONS BETWEEN DEFECT TYPE AND QUALITY ATTRIBUTE (EXAMPLE FROM C PHASE) Defect Category The establishment: 1. The creation of 8 defect types based on analysis over 1000 previous escaped defects. 2. Validate them with about 100 unused defects. 3. The core staffs brainstorm activities (quality attribute) performed during each phase and which defect types it will prevent or detect. 4. The 1 st version of Skynet was validated with 8 defects (one from each category).

11 RELATIONS BETWEEN DEFECT TYPE AND QUALITY ATTRIBUTE Defect Category The maintenance: 1. Conduct escaped defect prevention analysis to re-spin the net by adding new activities or refine the existing activities. 2. Review the effectiveness of the net to remove/refine the attributes that are not catching bugs periodically. 3. The net represent the best OU s knowledge on how to prevent and detect software defects.

12 LEAKED DEFECT ANALYSIS 1. Every leaked defect will be analyzed to identify where and how it is injected and where it should have been detected. 2. The flow goes from data analysis to process failure analysis (PM and QA) to proper improvement action analysis (Domain experts). 3. The related process elements (quality attributes) will be enhanced, that is to re-spin the net.

13 TARGETED QUALITY CONTROL MGT Each defect type is matched with a network of activities (attributes) that will could either prevent or detect this type of defects. If the constrain prevents adequately performing an activity, then a risk assessment will be conducted. Three things could happen: make it up later; ensure follow up activities will do the job; or add it to the Quality Debt Mgt System. Every defect category has a network of quality attributes distributed in the Skynet.

14 SKYNET IN ACTION OVER A SEQUENCE IMPLEMENTATION OF CR(S) P A Set up QPPOs/Skynet based on possible defects types involved. PPBs/PPMs used to assess initial risks Assess risks and mitigation plan after completing an activity C G After Action Review at the end of implementation of a CR. Interim goal analysis and risk review at each gate. SPC analysis will be conducted to make proper adjustments.

15 EXTEND SKYNET TO RELEASE DEVELOPMENT IN OUT Req Design OS AS CSCI CT/ST ND 新研 CR 维护 需求分析 变更输入确认 软件设计 变更方案制定 OS 开发实现 OS 代码修改 AS 开发实现 AS 代码修改 CSCI 集成调试 CSCI 集成调试 配置项 & 系统测试 变更验证 软件交付 内外部使用 In use Deployment

16 QUALITY DEBT DRIVEN MANAGEMENT/IMPROVEMENT Quality is the thing, but there are project constrains: time, environment, resources, lack of skills, and etc. Thus, shortcuts have to be taken from time to time. Another type of quality debt is caused by process deficiency. Quality debt: Deficiency in Implementing a quality attribute. Debt Mgt System: It is ok to cut the corner when necessary provided it is recorded with proper attributes and payback plan is developed. The goal is before the system is actually used, all quality debt must be paid back.

17 THE NEW RELEASE QUALITY PLAN Release Defined Process for every release Current Release Quality Attributes Planning Quality Attributes: New Quality Goal: Debt Payback Goal Old release s debt payback items The CI Items How to evaluate attribute conformance

18 Three approaches to evaluate attribute conformance: Auto analysis: checklist items can be verified by tools. QA verification: attributes can be judged by given rules and cannot be auto-checked; Expert Assessment: attributes cannot be judged by rules but can only be judged by experience. THREE VERIFICATION APPROACHES

19 REAL TIME RELEASE QUALITY STATUS REVIEW Product debt Debt type Verification checklist

20 QUALITY DEBT DRIVEN MGT Quality Objectives (Minimize Quality Debts) Quality Debts: 1. Process non-compliance; 2. {Xn} non-compliance (Xn} Baselines Models Review, Analysis, Testing and Audit Historical debts Sys Reqs Phase Review Project QPPOs Everyone focuses on the same thing: Achieve quality goals under the system schedule constrain. If quality goal needs revision, then debt must be managed. Phase1 Phase2 Delivery Schedule Constrain

21 As an example, Open Loop Testing is added to spin the net. DEFECT AND DEBT DRIVEN IMPROVEMENT Process Refinement Standard Process Debt Analysis Unstable Process Criteria Update Leaked Defect (Xn} Baselines Models QPPO PIO Identification Project & Release Ver & Audit 质量债务

22 SOME CONTRIBUTIONS MADE Allow the model factors closely linked to the key activities. Provide a complete picture for planning and monitoring quality control work. Provide a real time risk assessment mechanism. Make the development process a network not a linear line. By linking defect with a set of related quality attributes, we make 0-defect a possibility. The quality debt driven management and improvement provide best ROI in both project mgt and process improvement work. The use of quality attributes makes modular process design natural which enhance both process tailoring and improvement.

23 SOME EARLY RESULTS The 1 st quarter of 2016 vs The 1 st quarter of 2017 Jan March, 2016 Jan March, 2017 Num of leaked software defects Total number of software defects Ratio 20% 11%

24 FUTURE WORK Extend and refine the model with the integrated quality debt mgt system to cover the whole life cycle of the product development: Y-C- S-D (from concept all the way to final development) with all releases. Continue to refine the relationships between defect category and the mapping quality attributes. Establish QPPOs associated with each defect category. Develop a tool to support the use and improvemnt of Skynet.

25 Thank you.