Test Evaluation. Test Facets. Customer understandable. Spell Check. Idempotent Tests

Transcription

1 22 Test Evaluation "Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence." Edsger Dijkstra This chapter examines test characteristics show how the they may be made more maintainable. (** Order okay? **) Test Facets Here is a list of things to look for in tests. Some of these came from Gerard Meszaros, Ward Cunningham, and Rick Mugridge. Overall, remember that the tests represent a shared understanding between the customer unit and the developer and testing units. Customer understandable The test should be written in the customer terms (ubiquitous language) (**Ref Chapter x**). The tables should represent the application domain. If the standard tables shown in the examples are not sufficient, make up tables that are understandable by the user. (**Ref Chapter Table Presentation**). Use whatever way of expressing the test that most closely matches the customer s way of looking at things. Try multiple ways to see which way is most suitable to the customer. Use what is easiest for the customer. If a basic action table is not as understandable, add graphics to make it look as a dialog. If what looks like a printed form is needed for understanding, rather than just the data, use that as expected output. You can also have a simple test for the data for easier debugging of what might be wrong with the printed form (**Ref Chapter x**) Unless the customer wants values, use names. For example, use Good and Excellent as customer types, not customer types 1 and 2. Spell Check Tests should be spell-checked. Tests are meant for communication. Mis-spelled words hinder communication. The spell check dictionary can contain triad agreed upon acronyms and abbreviations. These acronyms and abbreviations should be defined in a glossary. Idempotent Tests Tests should be idempotent. They should work the same way all the time. Either they consistently pass or consistently fail. An erratic test does not work the same way all the time. Types of erratic tests are Interacting Tests that share data and Unrepeatable Tests for which first and following executions are 142

2 different because something in the state has changed. Paying attention to setup (**Ref Chapter X**) can usually resolve erratic tests. Test Sequence Ideally tests should be independent so they can run in any sequence without dependencies on other tests. However, as previously noted, you often have required workflows. You then must run acceptance tests in the workflow sequence. To ensure that tests are as independent as possible, ensure that a common setup (**Ref Chapter **) occurs when necessary. As noted in that chapter, the setup part of a test ( Given ) should ensure that the state of the system is checked to see that it matches what is required by the test. Internally this can be done by either checking that the condition is as described or by making the condition to be that which is described. Workflow Tests Workflow tests are sequences of tests that need to be run in a particular order to demonstrate that the system processes the sequence correctly. Try to have a happy path and an exception path in the workflow. For example with the CD rental limit of 3, have a test for renting 3 CDs (the happy path) and 4 CDs (the exception path). Not all exceptions have to be tested if they result in the same workflow, just different output values. Variations of business rules or calculations, unless they affect the workflow, should be tested separately. Have at least one positive and one negative test. For example, have a test with a Customer ID that is valid and one that is not valid. Use case tests should have a single action tables in them. Workflows which have multiple use cases within them may have multiple action tables in their tests. Try not to have too many complicated workflow tests. Other Points Organize tests with the setup (Given) / action (When) / outcome (Then) sequence Only have the essential detail in a test to keep it simple. Avoid lots of input and output columns. Break it into smaller tables or show common values in the headers. (**Ref Chapter X**) Minimal maintainable unless the desired behavior has changed, the test should not have to change, regardless of any implementation changes underneath. As much as practical, 100% of the code should be covered by tests either acceptance tests or unit tests. There may be some exceptional conditions that are difficult to reproduce in a end-to-end preproduction platform. Test logic should not be in the production code. production code. Tests should be completely separable from the Automate the tests so that they can be part of a continuous build. (** See Appendix X for versions of automation**) Tests and automation should be developed separately. Understand the test first and then explore how to automate it 143

3 Test Conditions A set of tests should abide by these three conditions: (** Ref Amir Kolsky **) : 1. A test should fail for a well-defined reason (that is what the test is checking). The reason should be specific and easy to diagnose. Each test case has a scope and a purpose. The failure reason relates to the purpose. 2. No other test should fail for the same reason. Otherwise you may have a redundant test. You may have a test fail at each level an internal business rule test and a workflow test that uses one case of the business rule. If the business rule changes and the result for that business rule changes, then you will have two failing tests. Tests should not be redundant. You want to minimize overlapping tests. But if the customer wants to have more tests because they are familiar or meaningful to him, do it. Remember the customer is in charge of the tests. 3. A test should not fail for any other reason. This is the ideal, but often hard to achieve. Each test has a three part sequence Setup, Action/Trigger, Expected Result. The purpose of the test is to ensure that the actual result is equal to the expected results. A test may fail because the setup did not work or the action/trigger did not function or that the actual results were not returned correctly. Test Failure As noted before (**Chapter ref**), a failing test is a requirement. A passing test is a specification of how the system works. Initially, before any implementation is created, every acceptance test should fail. If not, then you need to examine any acceptance test that passes. You should determine why it passed. Is the desired behavior that the test checks already covered by another test? If so, then the new test is redundant. Does the implementation already cover the new requirement? Is the test itself not testing anything? For example, the expected result may be the default output of the implementation. Test Redundancy You want to avoid test redundancy. Often that may occur when you have data dependent calculations. For example, here are the rental fees for different category CDs that was shown in ( ** Ref chapter XX**) CD Rental Charges Category Standard Rental Days Standard Rental Charge Regular 2 $2 $1 Golden Oldie 3 $1 $.50 Hot Stuff 1 $4 $2 Extra Day Rental Charge Here were the tests that were created. 144

4 Rental Charges Type Days Cost? Regular 3 $3 Golden Oldie 3 $1 Hot Stuff 3 $8 Do you need all these tests? Are they equivalent? They all use the same underlying formula (Cost = Standard Rental Charge + (Number Rental Days Standard Rental Days) * Extra Day Rental Charge). (** Footnote Depending on how the code is written, a single test might provide 100% code coverage. If you have that coverage, do you need more tests **). When the first test passes, the other tests also pass. What if there are lots of categories? Say there are 100 different ones that all use this same formula. That would be a lot of tests. But if Cathy the customer wants to see them, that s her call. Fragile Tests Fragile tests are sensitive to changes in the state of the system and with the interfaces it interacts with. The state of the system includes all code in the system and any internal data repositories. Handle sensitivity to external interfaces by using test doubles as necessary. (** Ref Chapter x **). In particular, the clock should be controlled through a test double. Random events should be simulated so that they occur in the same sequence for testing. Changes to a common setup can cause a test to fail. If there is something particular that is required for the test, then the test could check for the assumptions it makes about the condition of the system. If the condition is not satisfied, the test fails in setup, rather than in the check of expected outcome. This makes it easier to diagnose why the failure occurred. Tests should only check for the minimum amount of expected results. This makes them less sensitive to side effects from other tests. Many programs require that the platform they are installed upon meet particular requirement, such as a particular version of an operating system. When the program is installed, the installation process checks that those requirements are met. If not, the installation terminates. This approach is more user friendly than letting a program be installed and then the program failing because the environment is wrong. However the program makes an assumption that the environment does not change after it is installed. Sometimes the installation of another program changes the environment and causes the first program to fail. To be less fragile, the program should check for the required environment every time it starts up. If it is not as expected, the program should notify the user with an error message. That makes it much easier for a user to determine what the problem is than a This program had a problem message. Acceptance Tests One Part of The Matrix Acceptance tests are not the whole picture. They go hand in hand with requirements. And there are the other boxes on the matrix ( ** Ref Chapter **), usability, performance, etc. (** move **) 145

5 Acceptance Tests And Unit Tests Outside-in development provides a context. The set of acceptance tests for a story gives a context for the unit tests for the implementation. You do not need to translate all acceptance tests into unit tests. That would be duplication. On the other hand, the acceptance tests can be implemented using unit test frameworks, such as XUnit. It depends on the collaboration between you, the customer, and the tester. The XUnit tests should be readable as shown in the example in Chapter (**Ref Chapter**), so that they can be matched with the customer s expectations. If you don t have a customer providing you examples and you have a technical tester, then you might as well code all acceptance tests in Xunit. But if neither of those conditions apply, then having readable tests provides a double check. Production Bugs One of the most important measures for a team process is the number of bugs that escaped to production. You should examine the root cause or causes of each escaped bug. Was the case in production not covered by a test case and if so, why not? Were the data values not expected? For example, the program assumed a value between 1 and 100, but the value in production was 101. Examining why bugs escaped may lead you to discover how to prevent more of them from escaping in the future. Separation of Concerns The more that concerns can be separated, the easier it can be to maintain the tests. With separation, changes in the behavior of one concept do not affect other concepts. Here are some concepts that can be separated, as shown earlier in this book Separate business rules from how the results of business rules are displayed. (**Ref Chapter Separate Concerns**) Separate the calculation of business rule, such as a rating, from the use of that business rule. (**Ref Chapter **) Separate each use case or step in a workflow. (** Ref Chapter **) Separate out validation of an entity from use of that entity. You can have separate tests for the simplest things. For example, the Customer ID formatting functionality needs to be tested. The test can show the kinds of issues that the formatting deals with. If the same module is used anywhere a Customer ID is used, then other tests do not have to perform checks for invalid Customer IDs. And if the same module is not used, then you have a design issue. ID Format ID Valid? Notes 007 Y 1 N Too short 0071 N Too long 146

6 Summary Acceptance tests should be customer readable Avoid fragile or erratic tests by paying attention to setup and test doubles Avoid test redundancy 147