Test Workflow. Michael Fourman Cs2 Software Engineering

Transcription

1 Test Workflow Michael Fourman

2 Introduction Verify the result from implementation by testing each build Plan the tests in each iteration Integration tests for every build within the iteration System tests for the end of the iteration Design and implement tests by creating test cases that specify what to test test procedures that specify how to perform the tests executable test components to automate the tests (if possible) Perform the tests and systematically handle the test results Defective builds are sent to other workflows (e.g., design and implementation) for defect fixing

3 Context Test System Procedures Plans Expected Results Results Evaluations Defects Tester Test Components System/Component Under Test

4 Test Focus Core Workflows Requirements Inception Phases Elaboration Construction Transition Analysis Design Implementation Test Preliminary 1 2 n n+1 n+2 m m+1 Iterations

5 Testing in in the Software Life Cycle In general, whenever there is an implementation result, there is a test Test at each build Inception phase: initial test planning, prototype testing Elaboration phase: test architectural baseline Construction phase: significant testing at each build Transition phase: re-test fixes and regression tests Regression tests: in a new build, re-apply tests from old builds to make sure nothing broke in new build Evolve test model create new test cases for next build refine prior test cases into regression tests remove obsolete tests and corresponding test procedures and components

6 Artifacts Test model artifacts Test cases Test procedures Test components Test subsystem packages for complex test models Other artifacts Test Plan Defects Test Evaluation

7 Test Model Describes how executable components (e.g., builds) in the implementation model are tested by integration tests and system tests Describes how specific system aspects are to be tested e.g., usability testing of user interface or user manual

8 Test Case Specifies one way to test the system What to test (requirements) With which inputs Expected results Test conditions and constraints

9 Test Case Examples Test a use case or a scenario in a use case (black box) actor interaction, pre-conditions, action sequences, post-conditions Test a use-case realization (white box) verify interaction between components Installation tests, configuration tests, negative (abuse) tests, stress tests Organize similar tests in input/result matrix Family of test for different paths through a use case

10 Test Cases Special tests of system as a whole Installation tests verify that the system can be installed on the customer platform and that it executes correctly when installed Configuration tests verify that the system works correctly in different configurations Negative tests (abuse tests) try to cause the system to fail Test outside the intended design, e.g., wrong configuration, wrong input, etc. Reveals weaknesses in design Stress tests show problems with insufficient or shared resources

11 Test Procedures Specify how to perform one or several test cases or parts of them Instructions for tester in manual test case Instructions for interacting with test automation tool to create executable test components, then instructions to integrate and execute those test components One test procedure may cover several test cases One test case may require several test procedures Test procedure instructions often reflect flow-ofevents descriptions, including input values, how to enter input values, and how to verify results

12 Test Components Automate one or several test procedures or parts Provide test inputs, control and monitor test execution, and possibly analyze and report results Developed with scripting languages, programming languages, or test automation tools test drivers, test harnesses, and test scripts Complex test cases and procedures may require formal analysis, design, and implementation modeling of the test components

13 Test Plan Describes testing strategies, resources and schedule Strategy What kind of tests to perform for each iteration Test objectives Required level of test and code coverage Test acceptance criteria Trade-off cost of test vs. improved quality and reduced risk Test Plan may be defined in conjunction with the model artifacts from other workflows analysis (black box) design and implementation (white box) Tests may include artifact reviews before implementation tests

14 Test Results Results are the output data of the test and/or the behavior of the system during test Analysis of results: pass or defect Defect: a system anomaly (software fault or a defect found in artifact review) capture symptom in sufficient detail to allow problem resolution capture defect in problem and resolution tracking system Test evaluation: summary of overall test results test-case coverage, code coverage defects and defect status

15 Test Workflow Activities and Workers Test Engineer Plan Test Design Test Evaluate Test Integration Tester Perform Integration Test System Tester Perform System Test Component Engineer Implement Test

16 Workers Test designer: responsible for integrity of the test model plan tests: test goals and schedule select and define test cases, test procedures evaluate test results Component engineer: responsible for development of test components that automate some of the test procedures consider independent test

17 Workers Integration tester: perform integration tests and report defects may be same person as system integrator in implementation workflow System tester: perform system tests and report defects focus on externally observable behavior consider users, beta customers, etc. as testers

18 Activity: Plan Tests Describe testing strategy Inputs: use case model, supplemental requirements, design model Develop test cases and procedures that have a minimum of overlap test the most important test cases test requirements associated with the highest risk Estimate resources needed for the testing effort, including human and system resources Schedule the testing effort

19 Activity: Design Integration Test Cases Derive test cases from use-case realizations design (component interaction) Develop tests with minimal overlap that test an interesting path or scenario through use case Primarily consider interaction diagrams Combinations of actor input, output and system start state that lead to interesting scenarios that employ the classes that participate in the diagrams

20 Activity: Design System Test Cases Test system as a whole Test combinations of use cases instantiated under different conditions Hardware configurations System loads Number of actors Database size etc. Prioritize combinations that are required or are likely to function in parallel are likely to influence each other if they function in parallel involve multiple processes frequently use system resources or use them in complex and unpredictable ways

21 Activity: Design Regression Test Cases Modify test cases from earlier builds to be used as regression tests in subsequent builds Make more flexible and resilient to changes to the system under test Specifically address previously found defects Automate where possible

22 Activity: Identify and Structure Test Procedures Work test case by test case and suggest testing procedures for each one Reuse existing test cases May need to modify them to be reusable Align test procedures to subsystems under test Minimize overlap Isolates change to test resulting from change to subsystem

23 Activity: Implement Test Create test components from test procedures With test automation tools Perform or specify actions using test automation tools Record actions as a test component, such as a Visual Basic script When programming explicitly Design and implement using test procedures as specifications Provide input and output data generation, visualization, interpretation and management utilities Use spreadsheets and database applications

24 Activity: Perform Integration and System Test Perform the integration tests relevant to a build Manually perform test procedures or execute test components Compare test results with the expected test results Investigate test results that deviate from the expected Faulty system and/or faulty test? Report defects to the component engineer responsible for the components likely to contain the fault Report defects to the test designers For use in evaluation of overall test results System tests are performed the same way as integration tests

25 Activity: Evaluate Test Test designers prepare metrics of test acceptance with respect to system quality level Test completeness and coverage Percentage of test cases executed Percentage of code and paths tested Reliability Categorize defect severity Analyze trends of discovered defects and tests passed with expected results o Compare with prior experience to predict amount of testing still required to reach an acceptable defect level

26 Activity:Evaluate Test Suggest actions Suggest further tests Relax test criteria Deliver parts of the system that passed; rework and retest parts that failed Document test completeness, reliability and actions

27 Summary Test Model Test Plan Test cases specify what to test Test procedures specify how to perform test cases Test components automate the test procedures Test Evaluation Defects to be fed into other core workflows