Complementary methods for designing safety necessities for a Safety-Bag component in experimental autonomous vehicles

Transcription

1 Complementary methods for designing safety necessities for a Safety-Bag component in experimental autonomous vehicles Manel Brini, Paul Crubillé, Benjamin Lussier and Walter Schön Sorbonne universités, Université de Technologie de Compiègne, CNRS, Heudiasyc UMR 7253, CS , Compiègne Cedex France {manel.brini,paul.crubille,benjamin.lussier,walter.schon}@utc.fr Summary This work presents a study to improve the safety of experimental autonomous vehicles in the Heudiasyc laboratory. This study confirms that the use of these vehicles involves significant risks during experiments, and that integration of an Independent Safety Component called Safety-Bag in the vehicle architecture can significantly reduce theses risks. The Safety- Bag carries out the on-line verification of safety necessities by checking the vehicle s current state with safety rules and taking or disabling actions to ensure a safe behavior. In our work, we present and we apply two methods for risk analysis (FMEA and HazOP-UML) to design these safety necessities in the case of experimental autonomous vehicles. We also began to evaluate some safety necessities by experimenting with our robotized Fluence vehicle named IRIS in our laboratory on the SEVILLE runway and on the VILAD testbed. Keywords Safety-Bag, dependability, Safety, Fault tolerance, FMEA, HazOp-UML, autonomous vehicles. I. INTRODUCTION Autonomous automotive vehicles are mobile robots which evolve in an open environment and need to respect strict traffic rules. They should also be able to react to dangerous situations. Depending of their autonomy s level, they may need to carry out their mission in different contexts and to perform complex tasks. In order to do that, they use complex artificial intelligence software based on declarative mechanisms, which cause their validation and verification to be difficult. In addition, autonomous vehicles are fast and powerful mobile robots, able to accumulate a large amount of energy. Thus, they are a source of danger to their operator and their environment. Their dependability thus becomes a technological challenge for their industrialization as their safety has to be guaranteed. We propose to integrate an Independent Safety Component called Safety-Bag, in order to reduce risks from the complex and incompletely validated control systems. The Safety-Bag allows software and hardware fault tolerance but needs safety necessities to detect problems and put the system in a safe state. The second section of this article presents a brief context about autonomous systems architecture, the Safety-Bag component, and two dependability analysis methods: FMEA and HazOp-UML. The third section briefly presents the experimental autonomous vehicle that we are working on, and the implementation of its Safety-Bag. The fourth section presents a severity rating necessary for the FMEA and HazOp- UML risk analyses, respectively presented in sections V and VI. Section VII describes how we have moved from safety requirements derived from safety analyses to safety necessities implementable on the Safety-Bag. Section VIII presents a comparison of the results obtained from the FMEA and HazOp-UML risk analysis to generate safety necessities necessary for the development of our Safety-Bag. The last section presents our approach to evaluate the relevance of our safety necessities by running driving scenarios. These scenarios are tested on the Fluence IRIS vehicle on the SEVILLE runway and the VILAD testbed in the Heudiasyc laboratory. Finally, the paper ends with conclusions and perspectives for further work. II. GENERAL CONTEXT/ STATE OF THE ART We present in this section the hierarchical architecture, the most common control architecture used in autonomous vehicles. We then define the Safety-Bag approach by citing some examples of Safety devices existing in the literature. We finish this part by introducing the two risk analysis techniques FMEA and HazOp-UML. A. AUTONOMOUS AUTOMOTIVE VEHICLES SAE (Society of Automotive Engineers) [16] and OICA (International Organization of Motor Vehicle Manufacturers) [17] have each defined a classification for autonomy regarding intelligent vehicles. We use the OICA classification in our work, which consists of 6 levels. Level 0: Corresponds to vehicles without autonomous functions. Level 1: Corresponds to vehicles equipped with ADAS functions such as speed regulation or limitation. Level 2: Corresponds to vehicles which have limited autonomy in some particular situations such as park assist. In this case, the driver remains fully responsible for driving the vehicle. Level 3: Corresponds to vehicles which are able to drive independently under the supervision of a human driver. Managed driving situations and performance may be limited. The vehicle should be able to recognize its limits

2 and to inform the driver by leaving him a reasonable time to regain control. Level 4: Differs from level 3 by the fact that the autonomous system should ensure complete driving in the intended use cases, even if the driver doesn t respond to a request for manual return. Level 5: Corresponds to fully autonomous vehicles in all circumstances. They don t require human supervision. The experimental autonomous vehicles in the Heudiasyc laboratory are located somewhere along the levels 3 and 4 of this classification. In our experimental platform case, we consider that putting the system in a safe state can always be achieved by alerting the driver and switching to manual mode. Figure 1. A hierarchical architecture style In most autonomous cars, the vehicle s autonomy is implemented through the hierarchical architecture presented in Figure 1 [14]. It usually consists of three layers, although it can have more or less than that: A reactive layer (or functional layer): It is responsible for the execution of basic tasks requested by the upper layers. It performs a set of low-level actions like sensor data acquisition and controls hardware actuators. This layer s frequency is around 100 Hz to allow real-time actions and reactions. An executive layer implementing situation categorization and reactive navigation: This layer supervises the functional layer and uses data derived from sensors to identify the vehicle s situation and to generate trajectories that the lower layer will reactively follow. This layer s frequency is around 10 Hz. A planning layer: this layer produces a high level plan (a succession of roads and intersections) that the vehicle will follow to go from its current position to its destination. As planning can take several seconds (or even minutes), this layer s frequency is quite low and totally unadapted to real time constraints. B. Safety-bag Independent from the functional system, a Safety-Bag or Independent Safety Component is responsible for supervising the system s commands and enforcing safety rules to avoid catastrophic failures of the system. To reduce common cause of failures, it must be specified and developed independently from the functional system, and have means of action and detection independent from the faults to be tolerated. It monitors the operational system, and in case of danger sets the system in a safe state [1]. This approach was used effectively for critical applications such as: the ELEKTRA rail system [8], the SPAAS project for an autonomous spacecraft [2], the SPIN nuclear plant monitoring [8] and a mobile robot excavator [7]. C. FMEA The failure Mode, Effects and Criticality Analysis (FMEA) is often used in the automotive, railway and space industries. According to IEC [13], the objective of the FMEA risk analysis is to establish a criticity s order of system degradation through single failures to determine which components may require special attention and monitoring during design or operation. The FMEA is a bottom-up method that examines the possible failure modes of system components in order to determine the effects of such failures on equipments and the system s performances [5]. Its main goal is thus to establish the list of failures of each component and determine their occurrence rate and the severity of their consequences on the system s behavior. This analysis is then used to define risk reduction measures, and particularly avoid catastrophic consequences in terms of human life or economical value. The FMEA analysis is presented in the form of a table detailing each component, its different types of failures and the associated effects, including their severity, using each failure to determine possible safety requirements. The FMEA table may also includes the failure rate of each component, and the means of detection, action or correction. D. HazOp-UML The Hazard and Operability analysis (HazOp) aims to identify the system s dangers for safety, their possible causes, their consequences and the recommended actions to minimize their probabilities of occurrence [12]. Originally, HazOp was developed in the 1970s by Imperial Chemical Industries (ICI) to process thermohydraulic systems. The HazOp method has the advantage of identifying the system s hazards systematically by applying to each parameter a guide word, producing a deviation [5]. HazOP-UML is an extension of the HazOP method using diagrams of the Unified Modeling Language (UML) to study the system s behavior and its interactions with other actors in the environment. This study uses in particular use cases and sequence diagrams, and typically takes place at the beginning of the design process before the development phase. Three types of attributes are associated to each use case to precise: Preconditions: they are an obligatory conditions before the beginning of the use case.

3 Invariants: they are an obligatory conditions that should be true during the use case. Post-conditions: they are a necessary conditions after the end of the use case. The next step is to apply a set of guide words for each attribute of each use case. Among these guide words, we find as regards to use cases: No/none: The condition is not evaluated and can have any value. Other than: The condition is evaluated wrongly (for example, it is evaluated as true even though it is false). As well as: The condition is correctly evaluated but other unexpected conditions unfavorably affect the system. Part of: The condition is partially evaluated /Some conditions are missing. Early: The condition is evaluated earlier than required. Late: The condition is evaluated laterer than required. The product of the HazOP-UML analysis is thus in the form of a table detailing for each use case the combination of all guide words with all attributes, each combination possibly causing a deviation of the system s behavior. The deviations can be seen as somewhat similar to the failures of the FMEA analysis, and be used to define safety requirements for the system. III. SAFETY-BAG FOR AUTONOMOUS VEHICLES We now present the architecture of an experimental autonomous Fluence vehicle called IRIS and developed in the Heudiasyc laboratory. Figures 2 and 3 show a generic architecture of the system, respectively with and without Safety-Bag. The control application of the vehicle is implemented using the common hierarchical architecture previously presented. Figure 2. Deployment without Safety-Bag The vehicle is equipped with both proprioceptive sensors (ie. sensors giving information on the system s state; such as odometers, inertial measurement unit, etc.) and with exteroceptive sensors (ie. sensors that give information on the system s environment; such as GPS, lidar, camera, radar, etc.). Acceleration, braking and steering commands are sent to the actuators through analog signals produced by the command application and D/A converters. A Stop Process Button allows the driver to disable the automatic control and immediately regain manual control. Visual and auditory alarms can alert the pilot of the application s state and possible problems if the command application can produce a correct diagnosis. A Safety-Bag component was designed and implemented for this robotized car. Figure 3. Deployment with Safety-Bag Adding any component, even a fault tolerant one, can introduce new faults and failures in the system. As a single fault should not cause failure of the whole Safety-Bag, it is composed of two functionally diverse computers monitoring each others. One is called the Safety-Bag rules checker, and the other the Safety-Bag supervisor as shown in the figure 3. The control application sends now commands to the Safety- Bag rules checker. This checker verifies the commands against a set of safety rules, enabling them if considered safe, while blocking them or even taking safety actions otherwise. We call safety necessity a safety rule combined with its possible safety responses. A safety necessity is typically expressed as a if-then-else condition. The checker and supervisor monitor each other by the exchange of heartbeats. If the Safety-Bag supervisor no longer receives the signal from the Safety-Bag rules checker, it raises alarms and disables the vehicle actuators via a MOSFET as the autonomous behavior of the system can no longer be guaranteed. This has the same effect as the pilot pressing the Stop Process Button. If the Safety-Bag rules checker no longer receives the signal from the Safety-Bag supervisor, it advises the driver to stop the experiment even if there is no immediate risk, as the Safety-Bag is now vulnerable to a single fault.

4 The Safety-Bag can trigger visual and auditory alarms. It records all events related to safety, which allows postexperimental analysis. IV. SEVERITY RATING We present here the first step of risk analysis which consists in defining a table of severity rating (as shown in figure 4). This is used to rate each components failure or deviation in the analyses, but also to guide risk reduction approaches. Level 0: It corresponds to the system s nominal operation, where the software components are able to generate correct commands while the hardware components correctly apply the produced commands. The other levels require the pilot s intervention and an halt of the experiment. Level 1: Experimentations should be stopped, but the application is still somethat capable of properly driving the vehicle. For example, the cameras image quality is poor, or the process detecting navigable space has too much uncertainty. Level 2: The time needed for the driver to take over is sufficient to not pose any difficulties. For instance, the application has lost its GPS and can not know nor plan its position on maps, but remains capable of detecting immediate obstacles and avoiding them. The correct corresponding alarm is raised. Level 3: The control application has stopped and the vehicle is no longer controlled. An alarm is raised to warn the test driver to regain control immediately. The driver must be trained to properly handle the situation. Level 4: The application has stopped and the vehicle is no longer controlled. No alarms are raised. It is a complicated situation for a driver to handle because of the lack of warning and the possible need of rapid reaction. Level 5: An invalid command is sent or maintained on the vehicle actuators. It is extremely difficult for a driver to regain control in this situation, and the accident only depends whether or not obstacles are present on the vehicle s course. For example, a strong acceleration is maintained while the vehicle is turning. V. FMEA RISK ANALYSIS FOR AUTONOMOUS VEHICLES The figure 5 is an extract from the FMEA analysis table of the experimental autonomous vehicle in the Heudiasyc laboratory without Safety-Bag. This extract shows that our experimental autonomous vehicles have numerous failure modes with significant failure rates and severe consequences, indicating very high risks. We introduced additional columns to distinguish the failure s effects on the control component and the final effects on the vehicle s behavior. We also added a column describing the safety requirements that can protect from this failure, which is a common practice in the industry. In addition, we introduced in the table subcolumns to give indicative values for both response time (the Figure 4. Severity rating time needed for the recovery action to bring the vehicle to a safe trajectory) and detection time (the time needed to detect the failure; if there is a diagnostic device, this is the time to raise an alarm. Otherwise, it is an estimated time needed by the driver to recognize the situation as dangerous). The failure rate λ (failure probability per hour) is difficult to identify accurately without performing a huge number of experiments. In most cases, the given values are best guesses from an engineering point of view. For hardware failures, we took them as pessimistic values, from our knowledge of the system and its components. For software failures, experimental components are developed in research context and can be tested only briefly before being integrated into the system. The practice leads us to believe the failure rates of such failures are significantly high. As an example, in the first line of the Figure 5, a locked down failure in the control application leads to an uncontrollable vehicle as the actuators commands are not updated. The accident may become unavoidable before the driver is able to resume manual control. The severity of such a failure is 5. The corresponding safety requirement states that the system needs to be able to detect failures of the control application and subsequently block automated commands while warning the driver to regain manual control. Sensors failures can also cause dangerous situations if the software generates inappropriate orders due to its inputs. In our experimental vehicle, actuators failures are more frequent than in a standard car as the components added to interface with the actuators are not as reliable as industrialized automotive equipment. Moreover the software components are developed in an experimental context and can not be tested extensively before being integrated into the complete system. We believe that their failure rates are probably orders of magnitude higher than what we wrote into the FMEA table. As part of a dependability analysis, a FMEA analysis should

5 Figure 5. An extract of FMEA for autonomous vehicles without Safety-Bag be as complete as possible, and may include a large number of lines depending on the number of considered components. In our case (not taking into account the control application s subcomponents nor the exteroceptive sensors), the table for the vehicle without Safety-Bag has 14 components and 11 additional elements for the Safety-Bag. The determination of safety requirements is presented in Section VII. VI. HAZOP-UML RISK ANALYSIS FOR AUTONOMOUS VEHICLES To identify a relevant set of use case to perform a HazOp- UML analysis for autonomous vehicles, we relied on the list of conditions cited on pages 28 and 29 of [15], pending the production of similar recommendations by the European authorities. From 28 situations described in this document, we have identified a total of 25 use cases: 8 generic use cases related to our autonomous vehicle, 14 use cases extending the use case 5, 2 use cases included in the use case 8 and 1 use case extending the use case 6. The 8 generic use case of autonomous vehicles are: UC1: switch to autonomous mode, UC2: switch to manual mode, UC3: generate an itinerary 1, UC4: follow an itinerary, 1 Itinerary: An itinerary is a list of successive steps from the vehicle s origin that allows to reach a destination chosen by the operator. UC5: generate a kinetic trajectory 2, UC6: follow a kinetic trajectory, UC7: generate alarms, UC8: correctly know the state of the autonomous vehicle and its environment. In this article, we detail in particular the use case Follow a kinetic trajectory. The list of associated attributes is as follows: Pre-conditions: the kinetic state 3 estimated by the vehicle is equivalent to the real kinetic state. the kinetic state used by the vehicle s software is not erroneous. the kinetic trajectory does not cause the loss of the vehicle s control. The navigable space 4 contains the kinetic trajectory. Invariants: The vehicle s projection on the road stays in the navigable zone while following the kinetic trajectory. The vehicle kinetic state estimated by the vehicle remains at a limited distance from the kinetic trajectory. The vehicle components don t fail. 2 Kinetic trajectory: The kinetic trajectory is successive sets of position, attitude and speed of the vehicle that describes its intended immediate movement on the road. 3 Vehicle Kinetic state:the vehicle kinetic state consists of the position, the attitude and the speed of the vehicle. 4 Navigable space: It is a set of admissible vehicle position where the vehicle does not collide with other objects or people. (safety margins not taken into account).

6 Post-conditions: The vehicle knows the next kinetic trajectory. The vehicle s kinetic state is at the end of the previous kinetic trajectory. The vehicle estimated kinetic state is equivalent to the vehicle s real kinetic state. In the HazOp-UML table figure 6, we have associated with each deviation its consequences and possible causes, their severity and finally the safety requirements needed to protect from those consequences. Unlike the method proposed in [5], we do not determine directly the safety necessities for the Safety-Bag, but we begin by identifying safety requirements related to the vehicle behavior s deviation, in a similar way to safety analysis carried out by engineers in the FMEA method. Indeed, all safety requirements may not be implemented by the Safety-Bag component, and another another step is required in our opinion. VII. DETERMINATION OF SAFETY NECESSITIES FROM FMEA AND HAZOP-UML SAFETY REQUIREMENTS Once analyses of the system are performed, we need to determine the safety necessities from the safety requirements. These safety necessities will subsequently be implemented as safety rules monitored by the Safety-Bag, and safety actions to be taken if those rules are violated. However, not all safety requirements are implementable by the Safety-Bag, as for example the line 2 of the figure 7 or the line 3 of the figure 8. Other methods may be required such as redundancy of critical hardware components. Particularly, the checks to be carried out may be too complex to be performed by the Safety-Bag. Indeed, the Safety-Bag must remain sufficiently simple to be validated easily, and thus best have the form of conditional blocks in an imperative language (if-then-else clauses). All the safety requirements should be individually studied, and some examples of our analysis results are presented in the figures 7 and 8. For each safety requirement, the means of its implementation are detailed, and if the Safety-Bag is considered one of these means, a safety necessity is identified. This safety necessity specifies what the Safety-Bag should observe, how it identifies failures from these observations, and what it should do once a fault is detected. The safety necessities are formulated in a natural language, before being expressed in a formal language, as shown in the HazOp-UML method [5]. In figures 7 and 8, we have only formulated the safety necessities in a natural language. We obtained 21 security requirements from 42 attributes through the HazOp-UML analysis of the use case Follow a kinetic trajectory. Based on these 21 safety requirements, we extracted 10 safety necessities. The FMEA approach allowed us to identify 11 safety necessities through 16 safety requirements. VIII. FMEA/ HAZOP COMPARISON VIA THE SAFETY REQUIREMENTS In this section, we compare the results obtained between the two methods presented. Indeed, the FMEA and the HazOp- UML analysis have the same main goal to identify unacceptable risks to the system and its environment, as part of a process to reduce them to an acceptable level. The FMEA method focuses on internal components of the system (in our case, experimental software components, hardware components and the Safety-Bag at some point). However, the HazOp-UML focuses more on the vehicle s functions processes and its interactions with the environment. This explains the fact that there are common elements in both analyses, but also different ones. For example, the safety requirements from HazOp-UML, lines 1 and 2 do not correspond to any safety requirements derived from the FMEA analysis. Conversely, some components that are mentioned in the FMEA (eg. line 2) are not included in the HazOp-UML. In regards to common elements, we identified five similar necessities, such as line 1 FMEA and line 5 HazOp-UML, and line 5 FMEA and line 5 HazOp-UML. The safety requirement from line 21 of Hazop-UML is very general and need to be derived further before being translated into safety necessities, a common process in the industry. In fact, a specific FMEA analysis may be the easiest way to derive that safety requirement. Finally, we have found that the results of the HazOp-UML analysis may significantly depend on the choice of the representation and the expression of system s design, and thus the person who realizes it. Indeed from our point of view, the final result can vary according to how are chosen and expressed the use cases, the sequence diagrams and their attributes. Although still dependent of its analyst s skills and knowledge, the FMEA appears to us as more systematic, even if poorer in terms of studying the system s process. As previously stated, we found 5 similar safety requirements in both analyses, which gives us more confidence in them. We also found 5 and 6 unique safety requirement in respectively the FMEA and the HazOp- UML analysis, which makes us think of the two methods as complementary. Indeed, we can consider them as two diverse ways to obtain safety requirements for a critical system. IX. FIRST EXPERIMENTAL RESULTS The safety-bag component has been implemented in the Fluence-IRIS in the Heudiasyc laboratory. We were able to perform the first experimental results on both the Seville runway and the VILAD testbed. A. EXPERIMENTATIONS ON THE SEVILLE RUNWAY We performed the first experimental results on the SEVILLE runway successively with 4 drivers, which are laboratory members. The SEVILLE runway is a 60 meters road consisting of a straight line with a roundabout at each end. The objective of these tests is both to validate the implementation of the Safety-Bag and have a measurement of the pilots response time which is required to complete the FMEA study. The nominal

7 SIL: Safety Integrity Level Figure 6. An extract of HazOp-UML Figure 7. Safety requirements and necessities from a FMEA

8 Figure 8. Safety requirements and necessities from a HazOp-UML scenario is as follows: the car accelerates for 5 seconds until 20 km/h, then the speed is maintained for 5 seconds and finally the car brakes and stops autonomously. Pilots should react if the actual situation deviates from this nominal scenario by activating the Stop Process Button and stopping the vehicle. In the figure 9, we have described the faults injected in the system with erroneous commands. For example, when the vehicle restarts after shutdown, the undesired command is acceleration. In this case, the Safety-Bag correctly detects the erroneous command, raises an alarm and stops the system. Note that the associated safety necessity was specifically developed for this scenario to test the implementation of our Safety-Bag and should not be implemented on all autonomous vehicles. Moreover, the alarm signals raised by the Safety-Bag allow the drivers to detect and react to incorrect behavior more quickly. We found that for 1/4 of these cases, without the Safety-Bag intervention, the drivers do not react and when they react, they react slowly taking about 4s. On the other hand, if the Safety-Bag raises an alarm, the drivers react at average in 2s, as shown in table I. Note that with the Safety-Bag, the maximal reaction time is 5s. It is an unfavorable case when the drivers did not quickly rely on the SPB. B. EXPERIMENTATIONS IN THE VILAD TESTBED We plan to validate identified safety necessities using the Fluence IRIS (robotized Renault Fluence) vehicle and the VILAD testbed. The VILAD testbed integrates the Scaner Studio vehicle simulator and force feedback motors which opposes the power of the engine and those of the brakes to simulate a real vehicle s efforts. It allows either a human driver or the command application to control the real car in a virtual world. Of course, part of the sensors such as inertial sensors, lidars and radars, and the GPS must be emulated by the simulator. We are in the process of determining activities and fault injections needed to validate several of our safety necessities. X. CONCLUSION AND PERSPECTIVES In our study, an Independent Safety Component called Safety-Bag is proposed to reduce the risks in an experimental autonomous vehicles. This approach detects control failures such as locked outputs or dangerous commands as well as some actuators and sensors failures. It also recovers from the detected errors, either by rejecting dangerous commands, enforcing actions to put the system in a safe state, or ultimately giving the control back to the operator. However, to avoid introducing new logical faults in the system, potentially causing catastrophic failures, the behavior of the Safety-Bag must be simple and easy to validate, which limits the expressiveness of its safety necessities. To prevent failure in the Safety- Bag, it also needs enough redundancy to detect and tolerate internal faults. The elimination of the most serious and most immediate risk situations is done by alerting the pilot and switching to manual mode. Thus, trained and vigilant pilots remain essential in the vehicles, but the Safety-Bag ensures the detection of some of the most critical errors and allows more response time to the driver.

9 Figure 9. Scenarios with deviation Driver reaction time Average reaction time Standard deviation Min Max Without Safety-Bag and more With Safety-Bag Table I REACTION TIME WITHOUT AND WITH SAFETY-BAG. In this paper, we presented two analysis methods: FMEA and HazOp-UML. These methods are used to determine the safety necessities required by the Safety-Bag to detect and react from problems. They analyze the system from two different points of view and appear complementary for identifying the safety necessities. However, it should be noted that the results of these analyzes (in particular the HazOp-UML analysis) depend greatly on the analyst s skills. In addition, not all safety requirements are necessarily implementable by the Safety-Bag, which must remain simple enough to be easily validated. Moreover, the Safety- Bag should not involve decisions based on risky components (such as data fusion or artificial intelligence mechanisms). Nonetheless, the first tests performed on the SEVILLE runway confirm the interest of using the Safety-Bag in autonomous vehicles, and the interest of FMEA and HazOP-UML to design its safety necessities. [8] P. Klein, The Safety-Bag Expert System in the Electronic Railway Interlocking System Elektra, Expert System with Applications, [9] G. Guesnier, J. F. Hamelin, & J. M. Peyrouton, Centrale nucléaires N4: l informatique au service d une conduite plus sure, Epure, [10] MIL-STD-1629A 24 November 1980: Procedures for performing a Failure Mode, Effects, and Criticality Analysis [11] Lawley, H. G.,(1974) Chemical Engineering Progress, vol 70, no 4 page 45 "Operability studies and hazard analysis" [12] IEC Hazard and operability studies (HAZOP studies) : Application Guide. International Electrotechnical Commission, [IEC10] [13] IEC Functional Safety of Electrical/Electronic/Programmable Electronic Safety-Related Systems, Part 7 : Overview of techniques and measures. International Organization for Standardization and International Electrotechnical Commission, [14] C. Urmson & all, Environments: Boss and the Urban Challenge, Journal of Field Robotics 25(8), (2008) [15] Accelerating the next revolution in roadway safety, NHTSA, Septembre 2016 [16] SAE J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems [17] OICA Automated Driving Definition for Levels of Automation Informal document No. WP March 2014 ACKNOWLEDGMENTS This work was carried out and funded under the EQUIPEX ROBOTEX. It was supported by the French Government, through the program Investissements d avenir managed by the Agence Nationale de la Recherche. (Reference: ANR-10- EQPX). REFERENCES [1] Lussier B. (2007). Tolérance aux fautes dans les systèmes autonomes. [2] Blanquart J., Fleury S., &Hernek, M. (n.d.). Software Safety Supervision On-board Autonomous Spacecraft. [3] David P. & Guiochet J. (2005). Etude et analyse de différents dispositifs externes de sécurité-innocuité de type safety bag. [4] Baudin E., Blanquart J. P., Guiochet J. & Powell D. (2007). Independent Safety Systems for Autonomy. [5] Mekki Mokhtar A. (2012). Processus d identification de propriétés de sécurité-innocuité vérifiables en ligne pour des systèmes autonomes critiques. Toulouse: Université de Toulouse. [6] Mekki Mokhtar A., Blanquart J.-P. & Guiochet J. (n.d.). Safety Trigger Conditions for Critical Autonomous Systems, 18th Pacific Rim Int. Symp. on Dependable Computing (PRDC 2012), Niigata, Japan, [7] Pace C., Seward D., & Sommerville I. (n.d.). A Safety Integrated Architecture for an Autonomous Excavator. IEEE,Proc. 17th Int. Symp. on Automation and Robotics in Construction, Taiwan, 2000.