Checking formal specifications by testing. How to enhance the value of your test results with requirement observers

Transcription

1 Checking formal specifications by testing How to enhance the value of your test results with requirement observers

2 BTC Embedded Systems AG Company established in 1999, today: 100 employees We provide: intelligent and automated test solutions to the automotive market with a special focus on model-based development ISO

3 BTC Embedded Systems AG Continuous effort in research and innovation Partnerships e.g. with OFFIS Oldenburg and Oxford Innovation Ltd Member of Safetrans Active participation in numerous research projects 3

4 BTC Embedded Systems Partial User List 4

5 Challenges in the automotive industry today Challenges Increasing Complexity of Software and Electronic Systems Increasing amount of Test Cases and Test Data Increasing number of Safety Critical Requirements New applications requiring fundamentally different test approaches Questions How good are my Requirements? Is any Requirement violated? Are all Requirements completely tested?

6 Example - ACC Let's look at an example of a software requirement for Adaptive Cruise Control(ACC). Requirement 1: When the brake- or acceleration-pedal is pressed, the ACC system shall be deactivated within 20ms in any situation.

7 Example - ACC How do you make sure that the requirement is fulfilled by the system? You create a test case Test Case 1 * 1 Step = 10 ms Interfaces / Time Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 [Input] Brake Pedal [Input] Accel Pedal [Input] Vehicle Speed [Input] Distance Simulation System [Output] ACC Active ACC deactivated GOOD Record outputs 7

8 Example - ACC What about side effects that could lead to a violation of the requirement? ACC was active for 10 hours. ACC start button is kept pressed Both pedals are pressed at the same time Gear is set to Neutral Acceleration Pedal is pressed very short time Possible solution: Create more test cases for the requirement Time consuming creation Time consuming execution Every relevant situation must be explicitely designed as a test case for this requirement 8

9 Example - ACC Requirement 2: Absolute acceleration value shall never exceed 2.5km/h per second. Can you imagine test case for simulation? NO REASONS:. This requirement just prohibits wrong behavior, a specific situation is not given. We cannot define any specific inputs to the system. 9

10 Idea What if your PC could understand your requirements?

11 Formal Test: How to make testing more complete? Problem: Testcases are typically created per requirement It might stay undetected, if e.g. Test No.1 violates Requirement No.4 Solution: Use a Requirement Observer to automatically observe the status of each requirement during the complete test process. Requirements-based Testing Formal Test Req. 1 Test1 Req. 1 Test1 Req. 2 Test2 Req. 2 Test2 Req. 3 Test3 Req. 3 Test3 Req. 4 Test4 Req. 4 Test4 Manually defined Verdict Automatic formal Verdict 11

12 How to get there? Verification methods: from manual Review to Formal Test Manual reviews tedious and costly, no fun Simulation more fun, partial automation/interactive, not very systematic approach Requirements-based testing (RBT) mandatory method for verification and validation, highly automated, very systematic approach Formal Testing using formal requirement observers combined with RBT, highly automated, much stronger verification result compared with classical RBT How to get to a requirement observer? You need to teach the meaning of the requirement to your computer/hil Teaching means: Clearly defined Syntax Clearly defined Semantic ->Formal language (also recommended by ISO 26262)

13 Languages for teaching requirements to a computer Problem1: Some languages that might be used to express requirements are not formal too close to the implementation language Simulink/Stateflow Python.m scripts Problem2: Formal languages are often considered to be too mathematical and too difficult to learn Example of a formal specification in LTL Solution: A tool and a method that allows engineers to take their textual requirements and intuitively derive semi-formal and formal notations

14 Universal Pattern: formal modeling of requirements Easy to Use Close to natural language Step-by-Step wizard from informal via semi-formal to formal 14

15 Requirements Coverage Events and conditions are translated into goals Goals represent the different possibilities of making the event true Requirements Coverage can be measured automatically 15

16 Formal Test with Requirement Observers - Benefits Requirement Observers Are automatically generated from Formal Requirements Have access to Variables of the Test System Provide an automatic verdict, showing if a Formal Requirement is covered or violated Significantly increase the test depth Observer 1 Test Tool SUT Observer n No modification to current test setup needed No additional test data needed Informal Req. 1 Formal Req. 1 Informal Req. n Formal Req. n No additional test runtime needed

17 Formal Test - Workflow Real-Time BTC EmbeddedSpecifier Recorded Test Data Test Environment Formal Specification Test Environment Import BTC EmbeddedTester Export Create & Execute Requirement Observer Requirement Status Fullfilled / Violated Test Data Formal Specification Observe Formal Test System Under Test Test Cases Test case XY violated Requirement 5 Requirement fullfilled 17

18 Example Generated Layout in dspace ControlDesk Example 18

19 Additional Use Cases Requirements-based Test Generation Formal Verification Generate test cases automatically from formalized requirements Ensure completeness of the generated test cases thanks to the clear definition of requirements coverage Use Model Checking to automatically perform a complete mathematical proof, showing that a requirement can never be violated by a system under test Modelling/Coding Safety Requirements Formalization System under test Formal Specification TargetLink ANSI C-Code Formal Requirement Code does not fulfill the requirement Counter example Formal Verification Formal Verification using Model Checking technology for complete mathematical proof Code fulfills the requirement 19

20 Summary What if your PC could understand your requirements? Better requirements! Better verification!

21 Thank you. Dr. Udo Brockmeyer