Measuring and Assessing Software Quality Issues, Challenges and Practical Approaches Kostas Kontogiannis Associate Professor, NTUA kkontog@softlab.ntua.gr
The Software Life Cycle Maintenance Requirements Specifications Analysis System Architecture System Design Implementation Testing Operation Use Case Models Expressed in terms of Structured by Realized by Implemented by Verified by Evolved by class... class... class... class... Application Domain Objects Subsystems / Components Solution Domain Objects Source Code Test Cases Analysis Models Project Management Processes, Project Implementation Processes, Organizational Support Processes 2
The Software Systems Landscape Different types of systems have different types of quality assessment challenges: Safety and Mission-Critical Systems (Level A E) Power plant monitoring & control, Navigation, guidance & control, Communications, Emergency handling systems Non mission-critical Systems Executive Support Systems, Management Information Systems, Office Automation Systems, Transaction Processing Systems Vertical vs. Horizontal software scope Custom made, specific industry vs. generic framework 3
Software Quality ISO 9126 External and Internal Quality Functionality Reliability Usability Efficiency Maintainability Portability Suitability, Accuracy, Interoperability, Security, Functionality, Compliance Maturity, Fault tolerance, Recoverability, Compliance Understandability, Lernability, Operability, Attractiveness, Compliance Time behavior, Resource Utilization, Compliance Analyzability, Changeability, Stability, Testability, Compliance Adaptability, Installability, Coexistence, Replaceability, Compliance Quality in Use Effectiveness Productivity Safety Satisfaction 4
The Four Variables The main four variables of a project Development cost, Time, Quality, Scope Only three of these variables can be (more or less) freely adjusted Development cost, time and quality are bad control variables The number of developers can only be incrementally increased Deadlines are often predetermined externally Low quality upsets customers and developers Scope is the only real control variable 5
Software System Stakeholders Usability, End-user programmability, Ease of adoption, Customizability, Extensibility, Security, Safety, Performance, Installability, Documentation Process fitness for use, Price, Competitive positioning and strategic advantage in industry, Process improvement, Cost reduction Users Maintainability, Reliability, Testability, Robustness, Modularity, Cost, Reqs. Compliance, Feature provision, Time to market, Vertical vs. Horizontal markets Vendors Acquirers Regulatory Agencies Standards and Guidleiines, Independent Verification and Validation, Safety, Security, Compliance 6
Software Quality: The Vendor s Perspective Software quality is assured with techniques applied at the: Process Level: Unified Processes Capability Maturity Model Integrated (CMMI) ISO 9126 / ISO 14598 / Square Independent Verification and Validation IV & V Product / System Level: External System Properties Analysis Internal System Properties Analysis Security Assurance Compliance Reliability 7
Software Quality: The Vendor s Perspective Software quality evaluation Product level guidelines External Systems Properties Analysis Functional adequacy Volatility Accuracy and Precision Interface standards compliance Internal System Properties Analysis (pure Metrics) Traceability Defects per KLOC or Defects per Function Point Average Cyclomatic Complexity per Module Information Flow Time and effort to repair and maintain Change-Impact analysis 8
Software Quality: The Regulatory Perspective Analysis Aspects and Levels of Severity (DO-178B) Safety (ISO 26262) Economy (SOX) Security (ISO 27001) Environment related (ISO 14001) 9
Software Quality: The Regulatory Perspective Evaluation Techniques Functional or black box testing Inspection of development documentation guided by checklists Unit testing with test coverage criteria Analysis of fault tolerance construct in the software design and code Reliability growth modelling User interface and documentation inspection Verification of the conformity to interface standards Performing usage experiments with real users Execution time measurement Benchmark testing Analysis of the design to determine the algorithmic complexity 10
Evaluation Methods Software Quality: The Acquirer s Perspective Software Engineering Issues Review of user and technical product documentation Evaluation based on supplier courses and training Assessment of software engineering process Review of operating history with supplier Operating history requirements Operating history review Review of supplier capability, support, and quality system Review of operating history with customers Prototyping 11
Software Quality: The Acquirer s Perspective Other Evaluation Methods Analysis of software architecture design (maintainability) Fault tree analysis of software (safety, reliability) Statistical random usage based testing of software product (reliability) Dynamic analysis of code to check syntax and semantics for correctness (reliability) Hazards analysis of software design (safety, reliability) Review of software requirements specification (functionality) Code inspection (functionality) Black-box testing of software (functionality) Benchmark testing (efficiency) Analysis of requirements traceability (maintainability) Simulated faults at the interfaces between components (robustness). 12
Software Quality: The User s Perspective Source IHS and ISO 14598 13
Summary Software quality is an engineering characteristic of the system and its assessment should follow a prescribed process The software system type determines the assessment methodology Vertical and custom made applications require careful analysis of suitability, while frameworks and general purpose solutions require careful analysis of customizability Internal product attributes are more difficult to be examined by acquirer or end-user, as opposed to external attributes or quality in use attributes 14
Measuring and Assessing Software Quality Issues, Challenges and Practical Approaches Kostas Kontogiannis Associate Professor, NTUA kkontog@softlab.ntua.gr