Software Quality for Non-Software Professionals Presented by: Benjamin Schumeg ARDEC 13-14 MARCH 2017 UNPARALLELED COMMITMENT &SOLUTIONS Act like someone s life depends on what we do. Distribution Statement A: Approved for Public Release U.S. ARMY ARMAMENT RESEARCH, DEVELOPMENT & ENGINEERING CENTER
STATEMENT OF INTENT Better Understanding of Software Quality High-Level Review of Models, Methods, and Tools Discussion on Impacts, Benefits, and Challenges Demonstrate Involvement Throughout Project Example Software System for Analysis Review Quality Procedures on Sample Tasks Impart the Importance of Involving Software Quality in All Aspects of Project Lifecycle Plan Adjust Do Check 2
BIOGRAPHY Software Quality Engineer, US Army ARDEC, 2004 Present Weapons Fire Control and Software Quality, Reliability and Safety Engineering Division of Quality Engineering and System Assurance (QESA) M777A2 155mm Artillery M119A3 105mm Artillery Certified Software Quality Engineer, ASQ, 2013 Technical Rotation to NASA-Johnson Space Center, 2016 Software Quality Assurance Lead, Commercial Visiting Vehicles to the International Space Station M777A2 M119A3 ISS 3
COUPLE QUESTIONS Questions to Audience Anyone New to Quality? How Many Hardware Quality? How Many Software Quality? How Many Have Written Software Before? Familiar With Waterfall or Agile???????? 4
SOFTWARE WORKLOAD TRENDS DoD Systems Increasingly Rely on Software Warfighter Interacts With Software Frequently http://www.sae.org/events/dod/presentations/2012/dod_maintenance_symposium.pdf 5
SOFTWARE QUALITY IMPACT F-35 Joint Strike Fighter Most Costly and Ambitious Acquisition Program DoD Inspector General (DoDIG) Performed Quality Assurance Inspection of Pratt & Whitney Engines Found Inadequate Software Quality Management Practices Outdated Software Development Plans, Traceability Issues, and Incorrect Software Classification Insufficient Software Quality Assurance Organization DoDIG Recommended Multiple Improvements Improve Software Quality Management Systems, Software Development Plan, Traceability, and Software Classification Empower and Improve Software Quality Assurance Organization Internal and External Review Process Determine Impact of Poor Software Quality on Delivered Engines http://www.dodig.mil/pubs/documents/dodig-2015-111.pdf 6
SOFTWARE QUALITY IMPACT Mars Climate Orbiter Martian Weather Satellite Launched 1998 Crashed on Mars Due to Erroneous Location Information Incident Analysis Showed Discrepancy in Interface Newton vs. Pound Force Resulted In Accumulating Navigational Path Errors Ultimately Space Craft Contact Was Lost and Likely Destroyed in Atmosphere Reports Pointed to Multiple Possible Failures Lack of Confidence in Software Quality Assessments Software Engineers Expressed Concerns Poorly Defined Interface and Review Misunderstanding Impact on Differences NASA http://spectrum.ieee.org/aerospace/robotic-exploration/why-the-mars-probe-went-off-course 7
SOFTWARE QUALITY IMPORTANCE Space Shuttle Orbiter Flight Software Utilized 1970s-Era Computer for Handling Shuttle Flight Control First Fly-By-Wire System In Production Quickly Adjust to Failures, Such as Engine Issues Heavily Dependent On Software Software Quality Was Top Priority Crew Safety Resulted in High Quality and Reliability Near Zero Software Issues Found In Flight NASA https://www.nasa.gov/mission_pages/shuttle/flyout/flyfeature_shuttlecomputers.html 8
Software Quality 9
10
Cost to Fix SOFTWARE QUALITY GOALS Increase Customer Satisfaction Independent Assessment of Products and Processes Help Prevent, Identify, Reduce, and Eliminate Defects Ensure Functionality and Product Quality Reduce Costs of Development, Rework, and Maintenance Produce Reliable and Valid Products and Processes Reliable Valid Reliable Project Time Not Valid Not Reliable Valid 11
SOFTWARE QUALITY CHALLENGES Cost of Quality Hard to Quantify Invisible Products Quality Schedule Continuous Throughout Project, Not an Afterthought Quality s Significance in Processes and Products Quality is Beyond Just Testing Tradeoffs With Other Factors Cost, Schedule, Performance, and Product Improper Level of Rigor During Qualification Impacts Safety 12
DEFINITIONS Quality Management System Aggregate of the organization s quality-related organizational structure, policies, processes, work instructions, plans, supporting tools, and infrastructure [Westfall 2009] Software Quality Engineering Processes and activities needed to define, plan, and implement the quality management system for software-related processes, projects, and products [Westfall 2009] Quality Assurance Planned and systematic means for assuring management that defined standards, practices, procedures, and methods of the process are applied [CMMI 2011] 13
HARDWARE VS. SOFTWARE Hardware Requirements Design Manufacturing Testing Operation Software Requirements Design Coding Testing Fielding Measurable Characteristics Defined Specifications and Interfaces Implementable Design Controlled, Repeatable Production Quality Certification Continuous Improvement 14
Waterfall Software Model 15
WATERFALL Traditional Well-Defined Requirements Known Upfront Change Can Be Expensive Suitable for Lower Risk, Defined Projects Concept & Planning Software Requirements Software Design Code Development Integration Testing Qualification Testing Software Baseline 16
SOFTWARE V-MODEL Concept & Planning Software Baseline Software Requirements Qualification Testing Software Design Integration Testing Code Development 17
CONCEPT AND PLANNING - PROCESS Identify Project and Customer Needs Project Type and Criticality Acquisition Strategy Milestone Phases Schedule Imposed Constraints Development Methods Release Plans????? 18
CONCEPT AND PLANNING - QUALITY Scope Software Quality Effort Lessons Learned Tools and Methodology Hardware and Software Needs Confidence for Release Concept and Planning Activities Review Available System Requirements Software Development Plan Risks and Mitigations Implement Quality Management Systems 19
CONCEPT AND PLANNING - QUALITY Example System Improved Automated Teller Machine Software (IATMS) Builds Upon Existing ATM Technology Adds Loan and Currency Exchange Critical Finance System In-House Development Waterfall Software Development Lifecycle Large, Supported Quality Scope Budget Limitations Known Software Development Plan Provided 20
SOFTWARE REQUIREMENTS - PROCESS Elicit User/System Requirements Allocate to Software Requirements Identify Business Rules, Constraints, and Interfaces Develop User Stories, Data Flow Diagrams, and Event/Response Tables Assess Impact and Prioritize Define Requirement Change Process 21
SOFTWARE REQUIREMENTS - QUALITY Requirement Analysis Verify Complete, Clear, Concise, and Unambiguous Confirm Assumptions, Constraints, Limitations, and Boundaries Review Requirements for Quality Attributes (Sample) Safety Testability Reliability Usability Availability Performance Security Accuracy..... 22
SOFTWARE REQUIREMENTS - QUALITY Examples from Draft IATMS Software Requirements Specification for Review by Software Quality The software shall accept all currencies for exchange The software shall quickly compute the exact change from all currency exchange transactions The software shall accept all loan information as required in the most recent loan application form The software shall provide a final decision on the loan application within 10 seconds of all data submitted to the loan processor The software shall encrypt and decrypt all transmissions with the loan processor 23
SOFTWARE REQUIREMENTS - QUALITY Requirements Trace Verify Allocation of User Needs Identify Missing or Extraneous Requirements Reveal Relationships Managed Changes Along Links Requirement to Design to Test Bi-directional 24
SOFTWARE REQUIREMENTS - QUALITY ID User Need Requirement Design Test Case 1 USER-5.1.1 REQ1_134 DESIGN_A1 4.1.2.3 2 USER-5.1.2 REQ1_135 4.1.2.4 3 USER-5.1.3 REQ1_136 DESIGN_A6 4.1.3.1 4 REQ1_137 DESIGN_A9 4.2.8.8 5 USER-7.1 REQ1_734 DESIGN_B2 4.6.2 6 USER-7.2 7 USER-8.0 REQ1_1855 DESIGN_C4 4.7.1 8 USER-9.0 REQ1_2744 DESIGN_C5 Identify User Allocation Missed or Extraneous Software Design Incomplete Missing Test Cases 25
SOFTWARE REQUIREMENTS - QUALITY Requirements Tagging Safety, Mission Critical, High Risk Failure Mode Effect Analysis Level of Rigor Configuration Change Control Support Change Management Impact Analysis Manage Change, Churn, and Creep..... 26
SOFTWARE REQUIREMENTS - QUALITY Software Test Plan Define Test Management Activities Document Review Processes Identify Environments Document Test Criteria Scope of Test Events 27
SOFTWARE DESIGN - PROCESS Decompose Requirements to Design Architecture and Computer Software Configuration Items (CSCIs) Identify Design Quality Characteristics Cohesion, Coupling, Complexity, Reusability Allocation of Functions to CSCIs Assess Design Constraints Interface Analysis Identify Coding Standards 28
SOFTWARE DESIGN - QUALITY Architecture Verify CSCIs Identified Allocation and Trace to CSCIs Complete Interfaces Reviewed Interface Control Documents Available Design Constraints Identified Decomposition Completed Architecture, CSCIs and Other Units Quality Characteristics Mapped CSCI 1 CSCI 2 29
SOFTWARE DESIGN - QUALITY Design Reviews Held Preliminary Design Review Critical Design Review Quality Analyzes and Presents Initial Analysis Software Test Description Contains Test Case and Procedures Requirements and Design Analyzed Proves Verification of Software 30
SOFTWARE DESIGN - QUALITY get_loan_inputs() send_application() (delay) (delay) Software Design for Loan Processing Identify Possible Race Condition Missing External Connection display_result() (delay) 31
32
CODE DEVELOPMENT - PROCESS Translate Software Design to Code Implement Static or Dynamic Analysis Tools Define and Implement Build Process Develop and Execute Unit Tests 33
CODE DEVELOPMENT - QUALITY Source Code Analysis Unused/Dead Code Code Inspection for Traceability to Design Review for Defects Where Possible Coding Standard Adherence Complexity McCabe Cyclomatic Complexity Source Lines of Code (SLOC) Changes Note Volatility May Adjust Release Process 34
CODE DEVELOPMENT - QUALITY Configuration Management Identified Software Products Build Procedure Review Unit Test Analysis Review of Unit Test Procedures Analyze Results Confirm Code and Path Coverage 35
CODE DEVELOPMENT - QUALITY Unit Test Input Output 1 a = 0 increase(a) 2 a = 1 reduce(a) Path Testing Complete? Sufficient Unit Test Cases? a = 0.5, -1, test 36
INTEGRATION TESTING - PROCESS Assemble and Compile Code into CSCIs Confirm Interoperability Verify Internal and External Interfaces Prepare for Formal Qualification Testing 3 4 1 2 CSCI 1 1 2 3 4 CSCI 2 37
INTEGRATION TESTING - QUALITY Documentation Review Review Plans, Procedures and Results Verify Configuration Management Identify Emulators and Simulators Verify Software Products Identified and Integrated Implement and Manage Defect Process Verify and Validate Test Tools 38
INTEGRATION TESTING - QUALITY Software Integration Testing Verify Interfaces and Connectivity Execute Black Box and White Box Tests Boundary Inputs Valid and Invalid Environment Environment Input? Output Input Output Black Box White Box 39
INTEGRATION TESTING - QUALITY 3 5 1 2 1 2 3 5 CSCI 1 Integrated Products Testing Interface Between Components Fails Identify Failure Cause Build Analysis Determines Incorrect CSCI Integration CSCI 2 40
QUALIFICATION TESTING - PROCESS Determine Qualification Readiness Execute Qualification Testing Dry Run and Formal Qualification Testing Execute Test Cases and Procedures Record Defects and/or Problem Change Reports (PCR) Manage Defect and Change Requests Execute Regression Testing As Needed Provide Results Test Case Result Defect # 4.1.2.4 4.1.3.1 4.2.8.8 PCR_29 4.6.2 PCR_43 41
QUALIFICATION TESTING - QUALITY Conduct Test Readiness Reviews (TRR) Assess Quality of Software and Test Documentation Verify Environment and Configuration Identify and Manage Risks Certify Capability to Enter Test Event Execute Qualification Testing Dry Run and Formal Qualification Testing Execute Test Cases and Procedures Record Defects and/or Problem Change Reports Perform Regression Testing 42
QUALIFICATION TESTING - QUALITY Manage Defects and PCRs Define Priorities and Impacts Participate in Data Review Boards Monitor Arrival Rates and Defect Density Software Test Report Final Analysis of Software and Quality Contains Configuration, Environment Data, and Results Can Feed Release Process PCR_29 43
QUALIFICATION TESTING - QUALITY Defects By CSCI Affected Concern on Quality Anywhere? 44
QUALIFICATION TESTING - QUALITY Test Event is 12 Days Is There a Schedule Concern? 45
SOFTWARE BASELINE - PROCESS Software Products Packaged Final Management Approval Release to Customer and Stakeholders Feedback Gathering and Analysis IATMS 46
SOFTWARE BASELINE - QUALITY Software Products Packaged Verification and Validation of Build Configuration Audits Final Management Approval Release Review Board Support Certification Against Products and Process Release to Customer and Stakeholders Quality Check Release Methods Customer Manuals/Documentation Review Feedback Gathering and Analysis Support Management of Quality Escapes 47
Agile Software Model 48
49
AGILE Iterative Development Welcomes Change Quick Turnaround Constant Feedback Continuous Improvement Quality Built In All Steps High Team Cohesion Adjust Plan Check Do [SAFe] 50
Capability Maturity Model Integration (CMMI) 51
CMMI Capability Maturity Model Integration (CMMI) Essential Elements of Effective Processes Defined Processes and Process Areas Develop and Maintain Quality Products and Services Improved Predictability and Consistency of Products Continuous Process Improvement [CMMI 2011] 52
CMMI Model Contains Multiple Process Areas By Category Process Management Product Management Engineering Support Goals and Practices Generic Goals and Practices Defined Goals Across All Categories and Processes Specific Goals and Practices Unique Goals Specific to Process Area [CMMI 2011] 53
CMMI VERIFICATION Category: Engineering Purpose: Ensure Selected Work Products Meet Their Specifications Specific Goals and Practices Prepare for Verification Select Work Products for Verification Establish the Verification Environment Establish Verification Procedures and Criteria Perform Peer Reviews Prepare for Peer Reviews Conduct Peer Reviews Analyze Peer Review Data Verify Selected Work Products Perform Verification Analyze Verification Results [CMMI 2011] 54
CMMI MATURITY LEVELS 5 Optimizing Continuously Improved, Incrementally and Innovatively 4 Quantitatively Managed Controlled Through Statistics and Other Techniques 3 Defined Well Understood, Standardized, Defined, Controlled by Organization 2 Managed Planned, Documented, Performed, and Controlled by Project, Reactive 1 [CMMI 2011] Initial Unpredictable, Uncontrolled, Reactive ARDEC CMMI v1.3 Level 5 Independently Assessed 55
IEEE 12207 56
IEEE 12207 Systems and Software Engineering Software Life Cycle Processes Provides Framework for Developing and Managing Software Includes Terminology, Process, and References Covers Supply, Development, Operation and Maintenance Aspects Extensive Body of Standards and Documents to Support IEEE 12207 Can Supplement CMMI Processes https://standards.ieee.org/findstds/standard/12207-2008.html 57
Quality Tools 58
AUDITS Independent Evaluation of Products and Processes Against Known Criteria Functional Configuration Audit Certification Audit of Software Against Requirements Verification of Complete Trace Product Achieves Performance and Functional Goals Physical Configuration Audit Confirm Documentation Products Are Complete Versions and Revisions Specified, Clear, and Accurate All Process Specifications Met 59
PEER REVIEWS Work Products Reviewed Continuously Draft and Baseline Formal and Informal Methods Various Stakeholders and Experts Involved Resolve Defects Immediately Core Process in CMMI 60
DEFECT CONTAINMENT MATRIX Track Where and When Defects Found Resolve Defects As Soon As Possible Defects Found in Future Phases are Escapes Reduce Escapes to Improve Quality 61
Some Final Thoughts 62
63
SOFTWARE QUALITY INTEGRATION Development Requirements Verification and Validation Software Quality Customer/ Stakeholder Safety Reliability 64
CONCLUSION High-Level Introduction to Software Quality Topics Reviewed Popular Models, Methods, and Tools Just Scratched Surface of Software Quality Increased Understanding of Benefits and Challenges Demonstrated Involvement Throughout Project Impact Software Quality Has on All Phases Importance of Having Software Quality Plan Adjust Do Check 65
CONTACT INFORMATION Contact Information: Benjamin Schumeg ARDEC RDAR-EIQ-WE benjamin.j.schumeg.civ@mail.mil 66
REFERENCES [Westfall 2009]: Westfall, Linda. The Certified Software Quality Engineer Handbook, Milwaukee, WI: ASQ Quality Press, 2009 [CMMI 2011]: Chrissis, Mary Beth. CMMI For Development : Guidelines for Process Integration and Product Improvement, Boston, MA: Pearson Education, Inc., 2011 [SAFe]: Scaled Agile Framework. Scale Agile Inc., www.scaledagileframework.com. Accessed 23 January 2017 67
Backup Data 68
Additional Agile Data 69
AGILE Agile Manifesto/Principles (Excerpts Only) Our highest priority is to satisfy the customer through early and continuous delivery of valuable software Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale Working software is the primary measure of progress Continuous attention to technical excellence and good design enhances agility At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly Plan Adjust Do For Full Manifesto: http://agilemanifesto.org/principles.html Check 70
PLAN Prepare Backlog Review Backlog of User Stories Features and Enablers Verify Stories are Clear and Concise Define Acceptance Criteria Estimate Story Points Organize and Sequence Stories Plan Iteration Define Length and Velocity of Iteration Select Candidate Stories Refine Stories Utilize Story Points to Plan Quantity Story Story Story Story Plan Adjust Do [SAFe] Check 71
DO Execute Iteration Review Stories for Implementation Execute Development Track Story Flow and Process With Visualization Extreme Programming Constant Code Reviews Simplistic Designs When Possible Real-time Compilation of Software Continuous Integration and Feedback With Automated Unit Testing Scrum Daily Status Meeting Covering Past, Present, and Future Short Length (15 Minutes) To Do Doing Done Plan Story Story Story Adjust Do Story [SAFe] Check 72
CHECK AND DEMO Integration Test Continuous Integration Testing When Possible Internal or External System Integration Additional External Testing Supported Demonstration Working Product Shared with Teams Customer Feedback Gathered Stories Can Be Rejected Removed in Future Iteration Plan Adjust Do [SAFe] Check 73
ADJUST Customer Feedback Adjust Future Iterations Refine User Stories Retrospective What Did and Not Go Well During Iteration Improve the Process Adjust Iteration Velocity Adjust Estimations Plan Plan Adjust Do Adjust Do [SAFe] Check Check 74
SCALED AGILE FRAMEWORK [SAFe] 75
CMMI Process Area Examples 76
CONFIGURATION MANAGEMENT Category: Support Purpose: Establish and Maintain Integrity of Work Products Specific Goals and Practices Establish Baselines Identify Configuration Items Establish a Configuration Management System Create or Release Baselines Track and Control Changes Track Change Requests Control Configuration Items Establish Integrity Establish Configuration Management Records Perform Configuration Audits [CMMI 2011] 77
PROCESS AND PRODUCT QUALITY ASSURANCE Category: Support Purpose: Provide Staff and Management with Objective Insight into Processes and Work Products Specific Goals and Practices Objectively Evaluate Processes and Work Products Objectively Evaluate Processes Objectively Evaluate Work Products Provide Objective Insight Communicate and Resolve Noncompliance Issues Establish Records [CMMI 2011] 78