Software Quality and Risk Analysis

Size: px

Start display at page:

Download "Software Quality and Risk Analysis"

Lewis Phelps
5 years ago
Views:

1 Software Quality and Risk Analysis Master Software Technology / Software Engineering, Utrecht University 15 october 2007 Arent Janszoon Ernststraat 595-H NL-1082 LD Amsterdam info@sig.nl

2 Software Improvement Group Company Spin-off from CWI in 2000, self-owned, independent Management consultancy grounded in source code analysis Nominated for the Innovator Award 2007 (to be decided October 23) Services Software Risk Assessments (snapshot) and Software Monitoring (continuous) Toolset enables to analyze source code in an automated manner Experienced staff transforms analysis data into recommendations We analyze over 50 systems annually Focus on technical quality, primarily maintainability / evolvability 2 I 49

3 Who is using our services? 3 I 49 Financials / Insurance companies Financials / Insurance companies Government Government Logistical Logistical IT IT Other Other

4 Our services 4 I 49 Software Risk Assessment Remeasurement Monitor Software Analysis Toolkit Portfolio Monitor

5 Software Analysis Toolkit (SAT) Extraction ANTLR, sglr, regular expressions, off-the-shelf front-ends Analyses Metrics, lots Dependencies, data, control, inheritance, import, include, Clones Presentation Charts, graphs, matrices, etc Web pages (DocGen, Monitor) OLAP cube 5 I 49

6 Software Risk Assessment Report 6 I 49 Presentation Interpretation, reconciliation, evaluation Facts Facts Facts Automated analysis Documentation Interviews Benchmark Source code

7 Software Quality Monitor 7 I 49 Annual Report Board Interpretation by SIG experts Quarterly Report IT Management Monitor Web portal Software Engineers Source code

8 The issue 8 I 49 Companies innovate and change Software systems need to adapt in the same pace as the business changes Software systems that do not adapt lose their value The technical quality of software systems is a key element

9 Functional vs technical quality 9 I 49 low cost & risk Technical quality high cost & risk Functional quality Software with high technical quality can evolve with low cost and risk to keep meeting functional and non-functional requirements.

10 But 10 I 49 What is software quality? What are the technical aspects of quality? How can technical quality be measured?

11 The bermuda triangle of software quality CMMI (Scampi) ISO COBIT SAS70 Process (organizational) TickIT ISO9001:2000 Security ISO17799 ISO27001 BS I 49 Six Sigma TMap ISTQB J2EE (IBM) MCP (Microsoft) ITIL People (individual) Siebel (Oracle) Product ISO 9126 ISO Prince2 DSDM Project (individual) RUP (IBM) PMI

12 Software product quality standards ISO/IEC 9126 Software engineering -- Product quality 1. Quality model 2. External metrics 3. Internal metrics 4. Quality in use metrics 12 I 49 ISO/IEC Information technology -- Software product evaluation 1. General overview 2. Planning and management 3. Process for developers 4. Process for acquirers 5. Process for evaluators 6. Documentation of evaluation modules

13 ISO/IEC 9126, Part 1 Quality perspectives phase metrics 13 I 49 software product internal quality build 9126, Part 3 external quality test 9126, Part 2 effect of software product quality in use deploy 9126, Part 4

14 ISO/IEC 9126, Part 1 Product quality model: internal and external ISO/IEC 9126 Software Product Quality 14 I 49 functionality suitability accuracy interoperability security reliability maturity fault-tolerance recoverability usability understandability learnability operability attractiveness maintainability analysability changeability stability testability efficiency time behavior resource utilisation portability adaptability installability co-existence replaceability

15 ISO/IEC 9126, Part 1 Product quality model: technical quality ISO/IEC 9126 Software Product Quality 15 I 49 functionality suitability accuracy interoperability security reliability maturity fault-tolerance recoverability usability understandability learnability operability attractiveness maintainability analysability changeability stability testability efficiency time behavior resource utilisation portability adaptability installability co-existence replaceability

16 ISO 9126, Part 1 Maintainability (= evolvability) Maintainability = Analyzability: easy to understand where and how to modify? Changeability: easy to perform modification? Stability: easy to keep coherent when modifying? Testability: easy to test after modification? 16 I 49 Maintain Analyze Change Stabilize Test

17 ISO 9126 Part 2,3: metrics External metrics, e.g.: Changeability: change implementation elapse time, time between diagnosis and correction 17 I 49 Testability: re-test efficiency, time between correction and conclusion of test Internal metrics, e.g.: Analysability: activity recording, ratio between actual and required number of logged data items Changeability: change impact, number of modifications and problems introduced by them Critique Not pure product measures, rather product in its environment Measure after the fact

18 So 18 I 49 What is software quality? What are the technical aspects of quality? How can technical quality be measured?

19 A Challenge Use source code metrics to measure technical quality? 19 I 49 Plenty of metrics defined in literature LOC, cyclomatic complexity, fan in/out, coupling, cohesion, Halstead, Chidamber-Kemener, Shepperd, Plenty of tools available Variations on Lint, PMD, FindBugs, Coverity, FxCop, Fortify, QA-C, Understand, Integrated into IDEs But: Do they measure technical quality of a system?

20 Source code metrics Lines of code (LOC) 20 I 49 Easy! Or SLOC = Source Lines of Code Physical (! newlines) Logical (! statements) Blank lines, comment lines, lines with only } Generated versus manually written Measure effort / productivity: specific to programming language

21 Source code metrics Function Point Analysis (FPA) A.J. Albrecht - IBM Objective measure of functional size 21 I 49 Counted manually IFPUG, Nesma, Cocomo Large error margins Backfiring Per language correlated with LOC SPR, QSM Problematic, but popular for estimation

22 Source code metrics Cyclomatic complexity 22 I 49 T. McCabe, IEEE Trans. on Sw Engineering, 1976 Accepted in the software community Academic: number of independent paths per method Intuitive: number of decisions made in a method Really, the number of if statements (and while, for,...) Software Engineering Institute:

23 Source code metrics Coupling Efferent Coupling (Ce) How many classes do I depend on? Afferent Coupling (Ca) How many classes depend on me? Instability = Ce/(Ca+Ce)! [0,1] Ratio of efferent versus total coupling 0 = very stable = hard to change 1 = very instable = easy to change 23 I 49

24 A Challenge Do metrics measure technical quality? 24 I LOC Java code source code analyzer =

25 A Challenge 25 I LOC Java code source code analyzer =

26 Wealth of technical data at code level 26 I 49 must be translated into: Quality information Business risks Decisions at system level.

27 Layers ISO 9126 quality sub-characteristics e.g. changeability influences 27 I 49 can be caused by can be measured by source code properties e.g. complexity indicates source code measures e.g. cyclomatic complexity

28 Mapping source code properties onto quality sub-characteristics 28 I 49 Complexity Volume Duplication Unit size Unit testing Analysability X X X Changeability X X Stability X Testability X X X

29 Source code properties and metrics Volume LOC, within the context of a single language Man years via backfiring function points 29 I 49 Complexity per unit McCabe s cyclomatic complexity, SEI risk categories, %LOC for each category Duplication Duplicated blocks, threshold 6 lines, %LOC Unit size LOC, risk categories, %LOC for each category Unit testing Unit test coverage Number of assert statements (as validation)

30 Code duplication Definition Code duplication measurement 30 I 49 0: abc 1: def 2: ghi 3: jkl 4: mno 5: pqr 6: stu 7: vwx 8: yz 34: xxxxx 35: def 36: ghi 37: jkl 38: mno 39: pqr 40: stu 41: vwx 42: xxxxxx Number of duplicated lines: 14

31 Code duplication Code duplication 31 I % % Lines Percentage % 22% JSP Front-end Back-end handwritten Generator A B C D

32 Duplication Complexity Duplication Unit testing Volume Unit size 32 I 49 Analysability X X X Changeability X X Stability X Testability X X X Rank duplication Duplicate blocks Over 6 lines String comparison Remove leading spaces 0-3% 3-5% 5-10% 10-20% %

33 Complexity per unit Quality profiles Aggregation by averaging is fundamentally flawed 33 I 49

34 Complexity Volume Complexity Unit size Duplication Unit testing 34 I 49 Analysability X X X Changeability X X Stability X Testability X X X Software Engineering Institute complexity risk 1-10 low medium high >50 very high Rank Maximum relative LOC moderate high very high 25% 0% 0% 30% 5% 0% 40% 10% 0% 50% 15% 5% - - -

35 Rating example 35 I 49 Volume Complexity Unit size Duplication Unit testing Analysability X X X Changeability X X Stability X Testability X X X

36 That s all? Practical Fast, repeatable, technology independent Sufficiently accurate for our purposes Explainable 36 I 49 Beyond core model Only one instrument in Software Risk Assessments and Sofware Monitor Weighting schemes Dynamic analysis Quality of process, people, project

37 37 I 49 What is software quality? What are the technical aspects of quality? How can technical quality be measured?

38 Case 1 Curbing Erosion System About 15 years old Automates primary business process Maintenance has passed through various organizations New feature requests at regular intervals 38 I 49 Questions Improve management s control over quality and associated costs Metrics in this example Volume Duplication

39 Case 1 Curbing Erosion 39 I 49

40 Case 1 Curbing erosion 40 I 49 All growth is caused by duplication There is no real productivity

41 Case 2 Systems accounting from code churn System 1.5 MLOC divided over 7000 files Estimated 240 people divided over 25 subcontractors 41 I 49 Questions Is staffing justified? Metrics in this example Code churn = number of added, changed, deleted LOC

42 Case 2 Systems accounting from code churn 42 I 49

43 Case 2 Systems accounting from code churn Volume over time 43 I Lines of Code loc months

44 Case 2 System accounting from code churn Volume over time 44 I 49 Lines of Code months loc fte

45 Case 3 Learn from failure System Electronic commerce Replacement for functionally identical system which failed in rollout Outsourced development 45 I 49 Questions Monitor productivity and quality delivered by the developer Metrics in this example Volume Complexity

46 Case 3 Learn from failure 46 I 49

47 What should you remember from this lecture? 47 I 49 Lessons learned Technical quality matters in the long run Technical quality can be measured A few simple metrics are sufficient If aggregated in well-chosen, meaningful ways The simultaneous use of distinct metrics allows zooming in on root causes Further topics Dependency analysis Coding standard violations Requirements Architecture

48 48 I 49 Shameless Commercial Alert

49 Software Improvement Group 49 I 49 Internships Jobs Research cooperation Very interesting clients You get to see lots and lots of source code from all over the world Either Software Engineering or Consultancy skills (or both) See

50 More info? Feel free to contact 50 I 49 Dr. ir. Joost Visser E: j.visser@sig.nl W: T:

Measuring Software Product Quality

Measuring Software Product Quality Eric Bouwers June 20, 2013 T +31 20 314 0950 info@sig.eu www.sig.eu Software Improvement Group Who are we? Highly specialized advisory company for cost, quality and risks