Risk Based Testing -Why we need RBT? -Types of risks -Managing risks -Methods of evaluation & risk analysis -Costs and benefits Ladislau Szilagyi www.euroqst.ro
Definitions (ISTQB glossary) Risk = a factor that could result in future negative consequences; usually expressed as impact and likelihood. Risk based testing = a testing strategy intended to reduce the level of product risks and inform the stakeholders of their status, starting in the initial stages of a project. It involves the identification of product risks and the use of risk levels to guide the test process. Safety = the capability of the software product to achieve acceptable levels of risk of harm to people, business, software, property or environment in a specified context of use.
Risk outcome Risk = negative outcome? not always Positive risks are opportunities, desired by both the Project Manager and the stakeholders, and may positively affect the project ( such as increasing the ROI or finishing the project ahead of time ). But, positive risk may generate negative risks ( finishing a part of the project way before schedule will create a lot of slack, as other resources are not scheduled to work on the project until much later ).
Risk types Product risk Functional Non-functional Project risk External Organizational Technical
Project risk categories Risk types External: service provider related issues; client related issues;
Risk types Organizational: skill and staff shortages; personal and training issues; political issues, such as: problems with testers communicating their needs and test results; failure to follow up on information found in testing and reviews (e.g. not improving development and testing practices). improper attitude toward or expectations of testing (e.g. not appreciating the value of finding defects during testing).
Risk types Technical: problems in defining the right requirements; the extent that requirements can be met given existing constraints; the quality of the design, code and tests.
Risk dimensions Likelihood = the estimated probability that a risk will become an actual outcome or event Impact = the damage that will be caused if the risk become an actual outcome or event
Risk dimensions Risk dimensions are depending on the context: Automotive: Exposure (the relative expected frequency of the operational conditions in which the damage can possibly happen) Control (the relative likelihood that the user can act to prevent the damage) Severity of the damage Avionics: Threat Vulnerability Consequences
Risk factors Technical Complexity of technology and teams Personnel and training issues among the business analysts, designers, and programmers Conflict within the team Contractual problems with suppliers Geographical distribution of the development organization Legacy versus new approaches
Risk factors Tools and technology Bad managerial or technical leadership Time, resource and management pressure Lack of earlier quality assurance High change rates High earlier defect rates Interfacing and integration issues
Risk factors Business Frequency of use of the affected feature Damage to image Loss of business Potential financial, ecological or social losses or liability Civil or criminal legal sanctions Loss of license Lack of reasonable workarounds Visibility of failure leading to negative publicity
Risk options Ignore Assume Delegate Mitigate Contingency planning
Establish the context Risk identification Risk analysis Risk assessment Risks sorting Risk mitigation Risk contingency Risk reporting RBT activities
Establishing the context Study the business domain Identify the stakeholders Define a framework Planning of the remaining activities
Product Risk Identification Risk statement Something may fail in some way due to some circumstances Something The component or feature where the problem could occur Fail in some way What potential failure Some circumstances The reasons or vulnerabilities, why we are concerned
Product Risk Identification techniques Expert interviews Project matrix Independent assessment Use of risk templates Lessons learned Risk workshops Brainstorming Checklists
Probability Product Risk Analysis techniques Informal fast, cheap, but not accurate Pragmatic Risk Analysis and Management (PRAM) Systematic Software Testing (SST) Product Risk Management (PRisMa) High 1 2 4 3 Low Low Consequence High
Product Risk Analysis techniques Formal accurate, but takes time, expensive Cost-of-exposure Quality function deployment Fault tree analysis Hazard analysis Failure Mode Effect Analysis
FMEA - Example of formal Risk Analysis technique Failure Mode Effect Analysis Define the System Identify Potential Failure Modes & Their Causes Evaluate the Effects on the System of Each Failure Mode Identify Failure Detection Methods Identify Corrective Measures for Failure Modes 3 factors: Severity = The criticality of the effects of bugs in this failure mode, should any exist, from 1 (most damaging) to 5 (least damaging), Likelihood = The probability of and extent of impact associated with bugs included in this failure mode, from 1 (most probable) to 5 (least probable). Priority = The importance of fixing bugs in this failure mode, should any exist, based primarily on the ability of the delivered system to meet customer needs, though also on logistical project issues, regulatory or standards compliance, or other business considerations, from 1 (most important to fix) to 5 (least important to fix).
Product Risk Analysis techniques
Product Risk Mitigation techniques Non-testing related Testing related Choosing an appropriate test design technique Reviews & inspection Reviews of test design Level of independence Most experienced person The way re-testing is performed Regression testing
RBT activities
Project Risk Mitigation techniques Non-testing related Testing related Early preparation of test ware Pre-testing test equipment Pre-testing earlier versions of the product Tougher entry criteria Requirements for testability Participation in reviews of earlier project results Participation in problem and change management Monitoring of the testing progress and quality
Questions?