the Advanced Research Project on Software Metrics by the Ministry of Economy, Trade and Industry, Japan (METI)

Size: px

Start display at page:

Download "the Advanced Research Project on Software Metrics by the Ministry of Economy, Trade and Industry, Japan (METI)"

John Parks
6 years ago
Views:

1 the Advanced Research Project on Software Metrics by the Ministry of Economy, Trade and Industry, Japan (METI) Copyright (C) Mitsubishi Research Institute, Inc.

2 Overview Introduction (Background, Issues, Objective) Phase 1 Survey of Public Projects relating to Metrics in Japan Phase 2 Selection of a Set of Practical Metrics 1. Survey of Metrics Usage 2. Consolidating a Set of Metrics 3. Guiding Usage of a Set of Practical Metrics and its Benefits Conclusion & Next Plan (Phase 3) Copyright (C) Mitsubishi Research Institute, Inc. 1

3 Introduction Copyright (C) Mitsubishi Research Institute, Inc. 2

4 Introduction Background and Issues Background System/software is rapidly becoming larger in scale and sophisticated. Social impact caused by system outage is strongly recognized. The importance of system/software quality is increasing. System/software quality requirements and evaluation by quality metrics become important. Issues In Japan, we have some projects for product quality metrics of system/software (including ISO/IEC 250nn SQuaRE series) Each definition of metric, formula and measurement method depends on the projects Accordingly, every stakeholder does not treat quality metrics in unified manner. Copyright (C) Mitsubishi Research Institute, Inc. 3

5 Introduction Objective and Approach Objective Realization of a set of quality metrics which every stakeholder can treat in unified manner Developer meets quality needs of users considering quality life cycle Acquirer (evaluator) evaluates quality using metrics that are shared with developers Approach 3 years project Phase 1 (2009) Survey of Public Projects relating to Metrics in Japan Phase 2 (2010) Phase 3 (2011) Selection of a Set of Practical Metrics 1. Survey of Metrics Usage 2. Consolidating a Set of Metrics 3. Guiding Usage of a Set of Practical Metrics and its Benefits Promotion to Industries and Contribution to International Standardization Copyright (C) Mitsubishi Research Institute, Inc. 4

6 Phase 1. Survey of Public Projects relating to Metrics in Japan Copyright (C) Mitsubishi Research Institute, Inc. 5

7 Phase 1. Survey of Public Projects relating to Metrics in Japan Outline of Activities Survey and list up organizations of industries and their output on quality metrics Analyze the results and make mapping onto quality life cycle of ISO/IEC SQuaRE series. Copyright (C) Mitsubishi Research Institute, Inc. 6

8 Phase1. Survey of Public Projects relating to Metrics in Japan Project and their Output Project (Organization) UVC II (JUAS) NFR-Grade (IPA/SEC) Common Reference (IPA/SEC) SLA in software development (JEITA) SLA guidelines (JEITA) Purpose Definition of Non Functional Requirements(NFR) specification. Definition/suggestion of NFR specification. Reliability control at the development of the system which has an influence on the society. Improving quality of IT service by applying SLA to software development. Realizing appropriate balance about cost, risk and quality of services between acquirer and supplier. Output Overview 230 metrics in 10 domains (includes definition measurement method, interpretation of measured value) Guide to use metrics in software life cycle. Model system (16 characteristics) NFR grades Chart (92 items) Item list (236 metrics) Tree diagram Guide to use NFR-Grade System profiling In-process metrics 11 product quality metrics Quality metrics in software development Connection evaluation measure between a development process and operative, enhancement processes Processes using SLA Service level objective (481 metrics) Check list for introducing SLA (800 items for assessment) Case studies Copyright (C) Mitsubishi Research Institute, Inc. 7

9 Phase1. Survey of Public Projects relating to Metrics in Japan Mapping onto Quality Life Cycle ISO/IEC Quality Model Requirements Quality in Use Needs Products SLA Guid deline UVCII NFR-G Grade Common Reference SLA in Software development System Quality in Use Model System and Software Product Quality Model Is Used For Is Used For Is Used For Is Used For System Quality In Use Requirement Computer System Quality Requirement Software Product Quality Requirement Process Requirement Validation Verification Validation Verification Validation Implementation System Quality In Use Computer System Quality Software Product Quality Process Quality Projects System/Software Quality Life Cycle Model Copyright (C) Mitsubishi Research Institute, Inc. 8

10 Phase 2 Selection of a Set of Practical Metrics 1. Survey of Metrics Usage 2. Consolidating a Set of Metrics 3. Usage of a Set of Metrics and benefits of Using Copyright (C) Mitsubishi Research Institute, Inc. 9

11 Phase2. Selection of a Set of Practical Metrics Outline of Activities (1) Survey of metrics usage Survey of usage frequency of metrics in industries, with questionnaire which describes rearranged metrics used in all projects from the viewpoint of quality characteristics in ISO/IEC (2) Consolidating a set of metrics Edit a set of practical metrics considering usage frequency of metrics and WG member s opinion (3) Guiding usage of a set of practical metrics and its benefits Guide the use of the recommended set of metrics for system/software products quality requirements specification Copyright (C) Mitsubishi Research Institute, Inc. 10

12 Phase 2 (1) Survey of Metrics Usage Objective and Procedure Objective Clarify the following points from the actual situation of the industry Relationship among needs in use, product needs, functional and non-functional requirements, and quality characteristics Usage frequency of metrics Procedure 1. Select target systems 2. Survey each target system Question 1. Product Outline and Usage Scenario Question 2. Main Quality in Use Model and Using Metrics Question 3. Main Product Quality Model and Using Metrics Copyright (C) Mitsubishi Research Institute, Inc. 11

13 Phase 2 (1) Survey of Metrics Usage Questionnaire for Survey of Using Metrics ISO/IEC Quality in Use Model Internal/External Product Quality Model ISO / IEC ,3,4 Quality in Use Metrics Questionnaire ISO/IEC Quality in Use Model Effectiveness Efficiency Satisfaction Freedom from risk Context coverage UVCII 230metrics NFR-Grade 236metrics Common Reference 11metrics SLA in Software development SLA Guideline 481metrics Rearrange for Questionnaire Product Quality Metrics Questionnaire ISO/IEC Product Quality Model Functional suitability Performance efficiency Compatibility Usability Reliability Security Maintainability Portability Copyright (C) Mitsubishi Research Institute, Inc. 12

14 Phase 2 (1) Survey of Metrics Usage 1. Target System Field No. System/Software Product Finance/ # 1 Stock trading system Insurance # 2 Insurance product capitalization system company # 3 Financial institution counter terminal system # 4 Auto fare collection system using contactless IC card (Transportation) Public service # 5 Power supply, customer information control system (Electricity) company/ organization World Wide Web based service and contents # 6 Information system providing customer information(electricity) # 7 Agricultural information system # 8 Earthquake early warning service # 9 EC SITE, Cyber mall #10 Multi player online game system used from mobile/pc #11 Various information provision, Registration system #12 Educational learning system, Contents control system(school/education) #13 Built-in design supporting tool (Development tool) Copyright (C) Mitsubishi Research Institute, Inc. 13

15 Phase 2 (1) Survey of Metrics Usage Q.1 Product Outline and Usage Scenario, Example#4 Product Outline Case # : System Field Usage stage of system (business type) Related built-in system Architecture System processing form Usage Scenario #4: Auto fare collection system using contactless IC card Transportation Transportation business (Transport (ITS)), Finance/insurance business (Electronic settlement (e-money)) Facility equipment system Autonomous distribution Batch processing, Online transaction processing Stakeholder Usage Scene Main function Station staff Passenger Customer Issuing/selling IC cards Registering for re-issuance of IC card in the event loss. Passing the ticket gate by presenting an IC card Charging money to the IC card Shopping an IC card Issuing a new IC card Registration of re-issuance of lost or broken IC card Reading/writing of IC card, controlling ticket gate door, processing usage information Writing the amount paid in the IC card on the ticketing device Device or network for reading and writing IC card is required at a shop. Copyright (C) Mitsubishi Research Institute, Inc. 14

16 Phase 2 (1) Survey of Metrics Usage Q.2 Quality in Use Model and Using Metrics, Example#4 Important Needs, Risks Quality in Use Characteristics (Sub Characteristics) Quality in Use Model of #4 Quality Requirement Using Metrics (selected by questionnaire) Data on media (card) Effectiveness Functional and system needs to completeness of be consistent. data Mistake frequency Possible to pass the ticket gate smoothly with simple action even at busy times. Operation can be continued even at malfunction or disaster. Satisfaction (Usefulness) Freedom from risk (Economic risk mitigation) Accurate and high speed processing of reading, writing and fare calculation Autonomous distribution system configuration Work effectiveness Work completion degree Handling time Operation time Service provision time(handling trouble/general inquiry) No. of claims from customer Opportunity loss (check with a systemization evaluation sheet) Copyright (C) Mitsubishi Research Institute, Inc. 15

17 Phase 2 (1) Survey of Metrics Usage Q.3 Product Quality Model and Using Metrics, Example#4 Important Needs, Risks Correct fare collection and high speed processing are requisite conditions for passing the ticket gate. Consistency of data of media (card) and system is necessary. Mutual usage with other vendors is possible. System operates stably. There are many everyday transactions, so impact of operation shutdown is extremely large. Preventing leakage of personal information. Preventing data falsification. Product Quality Characteristics (Sub Characteristics) Functional suitability (Functional completeness) Functional suitability (Functional correctness) Compatibility (Interoperability) Reliability (Maturity) Reliability (Fault tolerance) Security (Confidentiality) Security (Functional completeness) Product Quality Model of #4 Quality Requirement Accurate and high speed processing of reading, writing and fare calculation Functional completeness of data Compatibility Redundant type, Autonomous distribution Redundant type, Autonomous distribution Encryption technology, operation control Encryption technology, operation control Using Metrics Copyright (C) Mitsubishi Research Institute, Inc. 16 N/A N/A N/A N/A N/A N/A N/A

18 Phase 2 (2) Consolidating a Set of Metrics Consolidating Procedure 1. Sort metrics by score of usage frequency of metrics for each quality sub-characteristics 2. Collect opinion of expert from industries about metrics that should be selected 3. Determine metrics in consideration of result of 1 and 2 above Phase2 (1) Survey of Metrics Usage Input 1. Usage frequency of metrics 3. Determination Set of Metrics 2. Expert opinion Copyright (C) Mitsubishi Research Institute, Inc. 17

19 Phase 2 (2) Consolidating a Set of Metrics Selected Set of Metrics Quality in Use Characteristics The number of Metrics Product Quality Characteristics Effectiveness 4 Functional Suitability 12 Efficiency 7 Performance Efficiency 22 Satisfaction 3 Compatibility 4 Freedom from Risk 12 Usability 22 Context Coverage 3 Reliability 32 Total 29 Security 22 Maintainability 21 Portability 16 Total 151 Copyright (C) Mitsubishi Research Institute, Inc. 18

20 Phase 2 (2) Consolidating a Set of Metrics Template and Example Item Meaning Example ID Identification of quality characteristics Re-3 Quality Characteristics Name of quality characteristic Reliability Quality Sub Characteristics Name of sub quality sub characteristic Maturity Metrics Name of metrics Test coverage rate Explanation of metrics Reference Information Indicating for what sort of evaluation How great is the coverage rate in units of purpose a metric can be used. statement when a test is carried out for Evaluation methods are also indicated each program. Or, how many tests partially as an example. including all passes are performed. Indicating the abbreviated code of report in which pertinent metrics are described. Supplemental information is partially attached following the abbreviated codes. Indicating the recommended degree ++high, +low Common Reference, UVCII <Report> Investigative Report on Measure for System/Software Product Quality Requirement Definition and Evaluation Copyright (C) Mitsubishi Research Institute, Inc

21 Phase 2 (3) Guiding Usage of a Set of Practical Metrics and its Benefits Usage and its Benefits Usage Select metrics from a set of metrics for; System quality in use requirements specification and evaluation System/software products quality requirements specification and evaluation Benefits of using the recommended set of metrics Effective selection of metrics depend on quality characteristics for target system Every stakeholders can treat in unified manner Copyright (C) Mitsubishi Research Institute, Inc. 20

22 Phase 2 (3) Guiding Usage of a Set of Practical Metrics and its Benefits Usage of a Set of Metrics in Quality Life Cycle Set of Metrics ISO/IEC25010 Quality Model Requirements Quality in Use Needs Products Quality in use Metrics System Quality in Use Model Is Used For System Quality In Use Requirement Validation System Quality In Use Before ISO/IEC ,3,4 UVCII NFR-Grade Product Quality Metrics Common Reference SLA in Software development SLA Guideline System and Software Product Quality Model ISO/IEC Quality Model Is Used For Is Used For Computer System Quality Requirement Software Product Quality Requirement Verification Validation Verification Validation Implementation Computer System Quality Software Product Quality System/Software Quality Life Cycle Model Copyright (C) Mitsubishi Research Institute, Inc. 21

23 Phase 2 (3) Guiding Usage of a Set of Practical Metrics and its Benefits Usage for Specification of Quality in Use Requirements Usage Scenario Important Needs, risks, Issues Quality in Use Needs ISO/IEC25010 Quality in Use Model Target Quality in Use Model Set of Practical Quality in Use Metrics Quality in use Requirement Specification System Quality in Use Requirement User s needs and risk Occurrence of a large amount of loss due to wrong ordering (securities dealing system) Example of Finance system Quality in Use Characteristics (Sub) Effectiveness Freedom from risk (Economic risk mitigation) Recommended Metrics Work effectiveness Failure frequency Work completion degree No. of delays of delivery time No. of missing part cases Copyright (C) Mitsubishi Research Institute, Inc. 22

24 Phase 2 (3) Guiding Usage of a Set of Practical Metrics and its Benefits Usage for Specification of Product Quality Requirements System Quality in Use Requirement Functional, Non Functional Requirement Requirement trace result should be 100% New order reception processing: 2mili. sec. Connection spec. (protocol) determined after working with securities company Select Characteristics ISO/IEC Product Quality Model Set of Practical Product Quality Metrics Select Metrics Function that Target Sys./soft. Sys./soft. should be Product Quality Product Quality realized Model Requirement (Specification) Computer System, Software Product Quality Requirement Example of Finance/Insurance system Product Quality Characteristics (Sub) Functional Suitability (Functional correctness) Performance efficiency (Time behavior) Usability (Appropriateness recognizability ) Recommended Metrics Accuracy Functional correctness of calculation Response time Turnaround time Throughput Functional completeness of description Copyright (C) Mitsubishi Research Institute, Inc. 23

25 Conclusion & Next Plan (Phase 3) Copyright (C) Mitsubishi Research Institute, Inc. 24

26 Conclusion & Next Plan (Phase 3) Summary In Japan, some public project relating to metrics Definition of metric, usage etc. depends on each project Every stakeholder does not treat quality metrics in unified manner. Conform to ISO/IEC Consolidate a set of practical metrics Considering actual usage situation of metrics in industries Unique name, explanation of a set of metrics Every stakeholder can treat in unified manner Conform to ISO/IEC25010 Copyright (C) Mitsubishi Research Institute, Inc. 25

27 Conclusion & Next Plan (Phase 3) Next Plans Phase 3 (2011-) Promotion to Industries and Contribution to International Standardization Improve a set of metrics Evaluate survey results on utilizing a set of metrics for industries Revise a set of metrics in consideration of the survey results Publish a revised set of metrics and usage guide Propose to ISO/IEC 25022(Measurement of quality in Use) ISO/IEC 25023(Measurement of system and software product quality) Copyright (C) Mitsubishi Research Institute, Inc. 26

28 Thank you More Information Copyright (C) Mitsubishi Research Institute, Inc. 27

Investigative Report on Measure for System/Software Product Quality Requirement Definition and Evaluation

Investigative Report on Measure for System/Software Product Quality Requirement Definition and Evaluation March 2011 Ministry of Economy, Trade and Industry, Japan Software Metrics Advanced Project Product