It s time to be more Optimistic about Negative Testing 12 th Annual International Software Testing Conference Saroj Patnaik 20-October-2012

Transcription

1 It s time to be more Optimistic about Negative Testing 12 th Annual International Software Testing Conference 2012 Saroj Patnaik 20-October-2012

2 Background Our existing QA approach: Understanding the user specification and exercises the Application Under Test (AUT) from a black box perspective Establish requirement traceability matrix to show a traceability of requirements to tested features and results To test and ensure all the features do what they were specified to do External feedback - we are seeing, most of the defects customers are seeing is related to unusual conditions that the application was not tested to handle Internal feedback - we are also hearing repeatedly statements like The failure occurred because customers are using the product in a way it was not designed for Therefore, QA needs to start thinking a lot more about how to anticipate and test for failure modes and other conditions that product may encounter in the real world Developers and testers needs to work closure to ensure testers understand potential failure modes and include appropriate hooks to drive the software into failure conditions during testing 2

3 Effectiveness of Testing Testing is said to be effective only when both positive and negative test cases are prepared and executed Positive test case Test to Pass Negative test case Test to Fail So, negative test cases should be a part of our testing effort, however Time and Cost estimation needed for negative testing has never been thought during estimation procedure Our estimation should reflect that negative testing is not a single phase, but spans and included in a range of activities Project with fixed budget or deadlines, negative tests may replace other testing type, but we need to evaluate which tests to leave out needs some kind of assessment We also need to consider our investment in terms of hardware, tools licenses and training 3

4 Intend of Negative Testing Negative testing aimed at showing software that does not work The goals is to see how easy it is to break the system, and having done that, how easy it was to get it up and running again Discovery of faults that result in significant failures; crashes, corruption and security breaches Observation and measurement of a system s response to external problems (networking, data bandwidth), error handling & exceptions Exposure of software weakness and potential for exploitation Strategic Directions There are often further aims of negative testing that do not set the scope of the activity, but nevertheless dictate the choices made in designing and running the tests. These can include: Prompt exposure of significant faults Learning about function through the study of dysfunction Verification (and possible enhancement) of risk model* used to prioritize testing Documentation of common failures, characteristic symptoms, and running fixes 4

5 The Life-Cycle Approach of Negative Testing - Planning Negative testing does not have a well-defined position in a waterfall or agile or iterative process, therefore not seen as a distinct phase in Software Testing Life Cycle (STLC) Typically, negative testing is most often used during system and integration testing and is performed by testers within those test teams Negative testing is an open-ended activity, therefore cannot be planned in a detailed way, but must be approached pro-actively Planning of negative testing should include following activities: Formal design and execution of negative tests Designing and performing exploratory negative tests Assessing the risks and possible exploitations of a weakness Diagnostic tests and related problem logging Communication within the team of exploits and weakness 5

6 The Life-Cycle Approach of Negative Testing - Schedule and Staffing Phase Approach Staffing Requirement Analysis, System Design, Unit Testing System and Integration Testing User Acceptance Testing, Beta Testing Formal techniques can be used to derive closed sets of negative tests. These should match the exception handling and validation functionality in the code. Analysis to derive the tests can itself expose important flaws Execution of scripted tests, and less-scripted primary and secondary tests React to weaknesses found Assess risk and potential for exploitation Diagnosis Allows fair assessment of user-facing error messages Valuable input to further negative testing, and can help to improve the test designers understanding of failure modes and weaknesses May allow a near-live system to be pushed beyond its performance limits, or tested for fail-over or recovery (tests may need to be scheduled away from general UAT) Business Analyst, Developer and experienced Test Analyst with good Observational skills and good knowledge of the underlying technology Scripted testing may be done by less experienced testers Exploratory testing must be done by experienced testers with good Observational skills and good knowledge of the underlying technology Business Analyst, Users and experienced Test Analyst with good Observational skills and good knowledge of the underlying technology 6

7 The Life-Cycle Approach of Negative Testing - Test Designing Some of the well-known techniques can be considered during negative test designing are: Boundary Value Analysis(BVA) and Equivalence Class Partitioning(ECP) State Transition testing Test against known constraints Failure Mode and Effects analysis Concurrency Sometime a fully scripted test may be hard to write and maintain, therefore a checklist can also help during design Lists of input that should be tried when checking validation or failure with bad input Known exploitation techniques Lists of error messages Environment / configuration outside constraints Steps along a happy path and variation 7

8 The Life-Cycle Approach of Negative Testing - Test Execution and Stop Criteria Prioritize and execute tests to determine the effectiveness and robustness of the exception handling system early the results of later tests will depend on its reliability and accuracy Prioritize and execute test to explore the system by looking for potential weaknesses Prioritize and execute test based on known exploitations which can find bugs, or allow to increase the confidence that the system is robust Prioritize and execute tests that tend to lead to a crash during early in execution life cycle Prioritize and execute test based on known failure modes, particularly those where impact analysis indicates serious consequences Since negative testing is open-ended, and has the potential to make coverage go down, therefore test coverage (number done vs. number planned) is not a useful way to judge when it is best to stop. The most obvious approach is to stop negative testing when we are no longer finding any significant new issues. This assessment should always be backed up with an idea of how much of the system has been observed under test. 8

9 Tools and Environment Negative testing, like performance and stress testing, need its own environment to avoid compromising other test work The most effective tools for negative testing is Exploratory Testing, which give a new perspective of the system Capture-n-replay tools have their place particularly for setting up tests and avoiding manual errors during repetitive tasks but negative testing depends on manual observation of the system s actions, however automated the test execution System inspectors tools such as network, CPU, and memory monitors, can give an early warning of failure, and also helps in further designing and enhancing negative tests A tool to refresh the system after a test is particularly important for negative testing, where failures can leave the system in a compromised state 9

10 Criticisms of Negative Testing Negative testing comes in for a lot of flack which is questioned, excluded and occasionally ridiculed. This attention and frequent selfjustification means that it is often well aimed, and an effective way of finding good bugs Some frequently heard criticisms are: That s an acceptable failure But that would never happen in normal use Don t the other tests do that? Why find a problem that we can t fix? 10

11 Takeaway Negative testing is a core skill of experienced testers, and requires an opportunist, exploratory approach to get the best value from the time spent Negative testing can not only find significant failures, but can produce invaluable strategic information about the risk model underlying testing, and allow overall confidence in the quality of the system. However, it can reveal fundamental flaws in the project, as well as the product Negative testing is open-ended and hard to plan granularly. It needs to be managed proactively rather than over planned Although negative testing is a powerful and effective approach, it is also a hard-to-manage task that has the potential to produce unwelcome information 11

12 References Testing Computer Software (Second Edition): Common Software Errors: Cem Kaner, Jack Falk, Hung Quoc Nguyen Whittaker, How to Break software, Addison-Wesley 2002 Beizer, Boris. Software Testing Techniques, van Nostrand Reinhold, 1990 Kaner, Bach, Pettichord, Lessons Learned in Software Testing, Wiley

13 13 Q & A

14 Thank You