IAH Temporary Revision Ed 9 / Section 6

Transcription

1 General Overview This Temporary Revision (TR) to the IOSA Audit Handbook (IAH) Edition 9 introduces a new Section 6 to the IOSA Audit Handbook containing the procedures and guidance for AOs and operators for auditing the effectiveness of implementation. Please note that this new Section 6 is applicable to both Auditors and Operators. Effective Date of the Temporary Revision This TR becomes effective 29 Apr Page 1 of 19

2 Section 6 Auditing the Effectiveness of Implementation Guidelines for Operators, AOs & Auditors Page 2 of 19

3 Section 6 - Table of contents 0. Background Auditing Effectiveness Definition of Effective Elements of Effectiveness Documentation / Implementation Suitability Effectiveness Criteria Ground Rules for Effectiveness Assessment Designation ISARP Structure Desired Outcome Suitability Criteria Effectiveness Criteria Umbrella ISARP Procedure for Airlines Ensuring Effectiveness Documentation Internal Auditing for Effectiveness Addressing Lack of Effectiveness (Non-conformity) Procedure for Auditors Nature of Effectiveness ISARP Procedure Evaluating Desired Outcome Assessing Suitability Assessing Effectiveness Overall ISARP Assessment Recording of Suitability Recording of Effectiveness Status Recording on ISARP Level Recording for Umbrella ISARP Scenarios for Recording Repeated ORG Provisions Corrective Actions for Effectiveness...18 Page 3 of 19

4 0. Background Since the introduction of the IOSA program, Standards and Recommended Practices (ISARPs) have been evaluated for two aspects: documentation and implementation. The requirement to document an ISARP in a controlled manner is straight forward. However, the requirement to implement an ISARP includes that when a system, process or procedure has been implemented, it must be monitored to ensure desired outcomes are achieved. The operator needs to check whether desired outcomes are achieved and whether they are in line with what the operator is intending (effective implementation). The requirement to ensure desired outcomes are achieved has always been there, however it is uncertain whether it has been systematically assessed and captured in the audits until now. IOSA is introducing a new method under which the effectiveness of ISARP implementation will be audited: In addition to pure implementation, operators will also be assessed whether an IOSA requirement as it is documented and implemented, is also effective. This new method will introduce several improvements to the program. Following are the main benefits: The assessment is giving a better representation of the effectiveness of implementation of the ISARPs, rather than that is purely based on conformity. Implementing SMS standards more effectively can also improve an operator s overall safety performance. Many accidents in the most recent IATA Safety Report list ineffective Safety Management Systems as contributing factors to accidents Regulators around the world are also looking more into effective implementation of Safety Management Systems for their operators. By assessing this through the IOSA audit, regulatory recognition may also be enhanced in the long run, which benefits all involved in the program Page 4 of 19

5 1. Auditing Effectiveness 1.1 Definition of Effective The term effective has been defined as follows: Effective means, the Desired Outcome, as specified in the Assessment Tool is achieved when an ISARP is assessed as documented and implemented and additionally, the defined suitability and effectiveness criteria are fulfilled. 1.2 Elements of Effectiveness As per the above definition, to achieve effectiveness, four elements need to be in place for all relevant ISARPs: documentation implementation suitability effectiveness Documentation and implementation have to be in place as a basis for effectiveness. These two elements also continue to be the only requirements to achieve conformity with an ISARP for the purpose of IOSA registration. Additionally, the implementation should be suitable and needs to fulfill all effectiveness criteria. Suitability will be discussed and challenged by the auditors, while effectiveness criteria will be verified in detail Documentation / Implementation A pre-condition for a standard to be assessed as effective is conformity based on the current IOSA audit method. This means that both, documentation and implementation must be in place. There is no change in the current methodology when assessing these two aspects of an ISARP. Conformity with the ISARP is still only based on these two aspects. Page 5 of 19

6 1.2.2 Suitability One new element of the effectiveness assessment is the one of suitability. Suitability has not been assessed under the current methodology, or only in a very limited manner. The suitability component requires a system, process, procedure or equipment, etc. to correspond with the size, complexity and nature of the operations. Quantitative and qualitative aspects need to be taken into consideration for this assessment. The factors to consider when assessing suitability may change from one standard to the next. In general, they are assessed against size, nature, and complexity of the operator s environment. The important thing is that all factors are taken into consideration collectively, like different dimensions to a single assessment. They cannot be individually assessed and then compared from one organization to the next. To illustrate this point, one could take two operators, each with 100 aircraft and 8000 employees. One is operating from one main base using a single aircraft type with a workforce that is largely made up of the local population. The second operator has the same number of aircraft, however the 100 aircraft are divided into five different types, the operator has 3 different bases in two countries and the workforce has culturally diverse backgrounds. If size was to be regarded separately in this context, both operators would be considered identical or very similar. It is however evident that this one aspect alone cannot be regarded separately. Another factor such as organizational complexities, processes, types of operations etc. needs to be considered. The third factor which is usually to be considered is the nature of operations such as destinations, customer segments, scheduled vs non-scheduled, passenger vs. cargo, etc. This will also ultimately loop back into complexity. When assessing suitability, all these factors need to be combined together and held against the operator s systems, processes and procedures. Suitability means that whatever has been implemented is adequate and appropriate to the operator s environment. Besides the ideal situation of being suitable, two extremes to the suitability are possible (too much or too little). Real issues would usually be associated with the lower extreme of the scale. In other words, one can expect that in the majority of the cases the implementation would be insufficient, rather than too much. An example to illustrate an extreme could be the reporting tool. If an operator has employees spread over different offices, a paper-based reporting system, where employees need to fill in a form, which can only be picked up and dropped off in one location, would be considered not suitable. Rather than introducing prescriptive elements for suitability, the method includes factors against which suitability needs to be discussed with the operator. Auditor judgement and common sense will have to be emphasized here. The outcome of the discussion with the operator is then captured in the auditor comment if the auditor has come to the conclusion that the implementation is not suitable. Page 6 of 19

7 1.2.3 Effectiveness Criteria Besides the suitability criteria, there are effectiveness criteria. Unlike for the suitability criteria, the effectiveness criteria are specifically defined. They go beyond what the ISARP spells out and contain all elements that are expected from a system, process, procedure, etc. to be effective. While the suitability criteria are the basis for a discussion, the effectiveness criteria must be individually verified and implemented by the operator. The example in Ch. 1.5 below illustrates examples of these criteria. 1.3 Ground Rules for Effectiveness Assessment As the methodology for auditing effectiveness is new, it is important to lay down certain ground rules for its application. The main ground rules are contained in this subchapter and are if deemed necessary further detailed in the following subchapters. The assessment of conformity is completely independent of the assessment of effectiveness which means, an operator can be assessed as being in conformity with an ISARP, even though that particular ISARP may have been assessed as not being effectively implemented. The assessment of effectiveness has no effect on the registration status of an operator. The overall result of the effectiveness is captured in one recommended practice. ISARPs to be assessed for effectiveness are pre-defined and designated as such; only designated ISARPs are evaluated. The suitability criteria must be discussed. Suitability will not affect the final result of the umbrella ISARP at this time Effectiveness criteria must all be in place in order to have a positive effectiveness assessment. 1.4 Designation The components of suitability and effectiveness are incorporated into the audit methodology on an ISARP level. ISARPs which are to be assessed for effectiveness are designated and only those ISARPs will be evaluated for effectiveness. As mentioned before, the existing component in the implementation assessment (verification that desired outcomes are achieved) continues to remain in place and the auditors should not deviate from previous practices of challenging the operators on this topic. ISARPs which are to be specifically assessed for effectiveness under this new methodology have a tag similar to the SMS provisions. The newly introduced effectiveness tag is [Eff]. Whenever an ISARP has this tag, it must be assessed according to this new methodology. The ISARP would then also have the Desired Outcome, Suitability, and Effectiveness criteria published. Page 7 of 19

8 1.5 ISARP Structure Since the new methodology has specific criteria which are to be verified by the auditors, they have to be incorporated into the ISM. Since these criteria are neither guidance material nor auditor actions, a new set of ISARP components has been introduced. They are called Assessment Tool. The Assessment Tool for the purpose of the effectiveness verification is added to the ISARP as follows: ORG 1.x.x ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text ISARP Text [Eff] Note: (if applicable) Assessment Tool Desired Outcome Suitability Criteria Effectiveness Criteria Auditor Actions Auditor Action 1 Guidance Material Guidance Material Text Guidance Material Text Guidance Material Text Guidance The following page illustrates a draft example based on an actual ISARP. Page 8 of 19

9 Sample: ORG Risk Assessment and Mitigation ORG The Operator shall have a safety risk assessment and mitigation program that includes processes implemented and integrated throughout the organization to ensure: (i) Hazards are analyzed to determine corresponding safety risks to aircraft operations; (ii) Safety risks are assessed to determine the requirement for risk mitigation action(s); (iii) When required, risk mitigation actions are developed and implemented in operations. [SMS] [Eff] (GM) Note: Conformity with this ORG standard is possible only when the Operator is in conformity with all repeats of this ORG standard in other ISM sections. Refer to the IAH for information that identifies such repeats. Assessment Tool Desired Outcome The Operator maintains an overview of its operational risks and through implementation of mitigation actions, as applicable, ensures risks are at an acceptable level. Suitability Suitable to the size, complexity and nature of operations Number and type of analyzed hazards and corresponding risks Means used for recording risks and mitigation (control) actions. Safety data used for the identification of hazards is appropriate Effectiveness (i) All identified hazards are analyzed for corresponding safety risks. (ii) Safety risks are expressed in at least the following components: Likelihood of an occurrence; Severity of the consequence of an occurrence. Likelihood and severity have clear criteria assigned. (iii) A matrix quantifies safety risk tolerability to ensure standardization and consistency in the risk assessment process which is based on clear criteria. (iv) Risk register(s) across the organization capture risk assessment information, risk mitigation (control) and monitoring actions. (v) The risk mitigation (control) actions include time lines, allocation of responsibilities and risk control strategies such as hazard elimination, risk avoidance, risk acceptance, risk mitigation. Page 9 of 19

10 1.5.1 Desired Outcome The first part of the Assessment Tool is the desired outcome. It contains what should be the result of the process / mechanism as required by the ISARP. The desired outcome may already contain some key expectations which an auditor can verify Suitability Criteria The overall suitability has to be assessed to ensure that a system, process, procedure, etc. addresses the operational and organizational environment. Since it is difficult to specify absolute suitability values, this section concentrates on factors to take into consideration when discussing the suitability. These factors may include specific metrics to observe, but do not specify the value this metric should have. In general, suitability will be evaluated keeping the operator s size, nature and complexity of the operational environment in mind as they usually will influence the suitability the most Effectiveness Criteria The effectiveness criteria specify what needs to be in place in order to achieve the desired outcome. The effectiveness criteria must all be in place to assess the corresponding ISARP as effective. The criteria, to a large part, contain elements which an auditor would normally be looking for, but if absent, could not lead to a finding since they are not specifically required by the standard. 1.6 Umbrella ISARP Same as for the SMS, there is an umbrella ISARP which captures the overall state of effectiveness for the operator. This ISARP is a recommended practice and is only in conformity if all ISARPs with an [Eff] designator have been assessed as being effective. Even one single missing item in the effectiveness criteria of one ISARP would result in a non-conformity in the umbrella ISARP. ORG The Operator should demonstrate that systems, processes and procedures specified by the ISARPs identified with the [Eff] symbol are achieving the designated Desired Outcome. Note: Conformity with this ORG recommended practice is possible only when the Operator demonstrates effectiveness of implementation for all ISARPs designated with the [Eff] symbol. Page 10 of 19

11 2. Procedure for Airlines 2.1 Ensuring Effectiveness The whole concept is introduced as a recommendation. Nothing new is required of an operator to ensure the continued validity of the IOSA registry. Even if an operator has been assessed as not implementing certain ISARPs effectively, this operator can still retain its IOSA registration status. The operator needs to ensure sufficient efforts are invested in implementing the corresponding ISARPs effectively and needs to be able to demonstrate it. 1. To properly prepare, an operator needs to consult the Assessment Tool in the IOSA Standards Manual for the corresponding ISARPs and identify the required elements. 2. The first element of the Assessment tool is the desired outcome. It is important that it is well understood what the desired outcome of the ISARP is as the rest of the elements, specifically suitability and effectiveness, are there to support this desired outcome. For [Eff]-designated ISARPs, the desired outcome is defined in the standard, whereas for the rest of the ISARPs the operator has some flexibility in defining it to ensure effective implementation. 3. The next step is ensuring the suitability of the implemented process. This will depend on the size, complexity and nature of the operational environment. Clear criteria should be developed based on which the suitability is evaluated internally. Achieving the desired outcome should always be in the foreground of this evaluation. Ideally, there would also be some metrics associated which makes the achievement of the desired outcome measurable. This will ultimately give an indication of whether implementation is suitable. Another method to utilize could be benchmarking with other operators of similar setup. This can also help in making the assessment and ensuring that the suitability is given. The operator should be ready to communicate the rationales which went into this internal assessment during the IOSA audit. Any type of benchmarking and analysis should carry some form of proof (records). 4. Lastly, the specific effectiveness criteria need to be in place. The operator needs to study them and make sure they are all in place and fully implemented. Every effectiveness criterion must be in place to receive a positive assessment for effectiveness. If only one item is missing, the assessment would automatically become negative for that particular ISARP. It is the operator s burden to provide sufficient proof of implementation. 2.2 Documentation The main focus of the effectiveness methodology is the implementation ensuring that all processes are carried out in an effective manner. Therefore, there is no documentation required for both, the umbrella ISARP as well as for the criteria underneath the [Eff]-designated ISARPs. Therefore, the umbrella ISARP does not require a document reference in the audit checklist. Page 11 of 19

12 2.3 Internal Auditing for Effectiveness The procedure for internal auditing for effectiveness as per ORG (and / for Edition 12 of the ISM) is the same as the one to be used by the auditors for the purpose of the registration audit. Procedures should therefore be followed as described in chapter 4 below for the purpose of completing the internal audits. The concept of effectiveness is a recommendation at this time. If the internal assessment program concludes that effectiveness is not assured, it is not mandatory to rectify the observation raised against the umbrella ISARP. 2.4 Addressing Lack of Effectiveness (Non-conformity) The result of lacking effectiveness would create an observation against the umbrella ISARP in ORG, recommending demonstration of effective implementation. Effectiveness criteria are contained in several ISARPs designated with the [Eff] tag. Any missing criteria will be recorded in the auditor comment of the umbrella ISARP, and only the umbrella ISARP will be assessed as an observation. Detailed guidance on how to record the non-conformity in the umbrella ISARP can be found in the auditor guidance in Chapter 3. Corrective action would follow conventional procedures: a root cause needs to be identified and a corrective action plan needs to be established and agreed on with the AO. Once verified, corresponding final corrective action needs to be taken. The main difference to all other ISARPs is the potential level of complexity of the wording in the Corrective Action Record (CAR) as each affected ISARP needs to be individually addressed in a single CAR. The action to be taken would be the same logic as with any other non-conformity. The main difference is that corrective action would probably not be as simple as with some other issues. Page 12 of 19

13 3. Procedure for Auditors Before any assessment for effectiveness is performed, the auditor first needs to assess an ISARP for documentation and implementation in the conventional way. These two aspects are a pre-requisite for a standard to be assessed as being effective. If either documentation or implementation are missing or partly missing, the standard cannot be assessed as being effectively implemented. In such a scenario, the umbrella ISARP would automatically be assessed as non-conformity. 3.1 Nature of Effectiveness With the introduction of this method, a first step towards a performance-based audit of individual ISARPs is taken. This being said, the IOSA audit is not shifting into a full performance-based audit. When discussing the suitability criteria, the auditor does not audit against prescriptive specifications. Instead, the auditor will have to take into account the individual environment of the operator and determine whether the implementation is suitable. This is a first form of performance-based approach. To illustrate the difference between a compliance and performance-based audit, the following example can be taken. Under IOSA, as well as most regulatory environments, an operator needs to have a training program in place. The training program contents are very prescriptive and detailed. The desired outcome of the training program is often missing. Compliance is achieved as long as the training contains the topics which are contained in the regulations. Performance-based requirements focus more on the outcome rather than on the way something has to be accomplished. Going back to the concept of training, a performance-based audit of a training program would focus on what an operator is trying to achieve with the training. It should be captured and verified somewhere. An operator may have a very comprehensive training program in place, with very elaborate training materials and facilities. This alone however, does not guarantee that the people being trained are actually competent to do what they have been trained in, or in other words, if the delivery of the training is effective. This would show most directly in exam results (provided the exams are effective in testing knowledge and understanding) and more indirectly in the day to day operation. If a training program as described above is in compliance with prescriptive regulatory requirement, it does not necessarily mean that the training program is effective. If trained staff members are unable to apply knowledge in the day to day operation due to an ineffective implementation of the training program, the desired outcomes would not be met (competent staff members who can apply the knowledge). The IOSA audit is not changing into a performance-based audit by any means. When discussing suitability, it should however be clear that applying a purely compliance-based audit method during the interview would create some problems. The approach should therefore be shifted slightly during the discussion on suitability. Page 13 of 19

14 3.2 ISARP Procedure The following describes the procedure to evaluate / assess the three components of effectiveness which are defined for each ISARP. The methodology changes slightly between those components Evaluating Desired Outcome The first level of assessing effectiveness is to study the desired outcome. This part of the Assessment Tool describes what the ISARP is trying to accomplish and what the operator should be addressing with the system / process / procedure. The desired outcome usually already contains certain concepts which can be verified usually on a qualitative level. The desired outcome however does not provide very detailed components. To verify the desired outcome, the auditor needs to change the course of questioning and interviewing from verifying specific pieces of evidence, to asking more globally how the operator accomplishes this desired outcome and which factors contribute to reaching this outcome Assessing Suitability The second part is the assessment of suitability. Suitability translates to whatever is in place will effectively support the desired outcomes and the overall objectives. For the time being, Suitability criteria are not prescriptive and they need to be assessed with regards to the size, nature, and complexity of the operational environment. These three aspects will dictate largely what an operator needs to have in place. For the suitability, it is very difficult to define specific, absolute criteria which need to be in place. As a result, the assessment of suitability has a certain risk of being subjective. To counter that, the auditors need to ensure that the suitability is compared with industry standards and benchmarking, rather than using individual views. It is also equally important that the auditors challenge the operator for its own method to evaluate suitability. In other words, the auditor should be asking the operator how it ensures suitability and in what ways this is measured. The desired outcome will play a major role in the assessment for suitability. If the desired outcome is achieved, one can already argue that the suitability is achieved to a great extent. If the desired outcome is not achieved, it may be a consequence of unsuitable systems / processes / procedures. In order for the auditor to mark the suitability as insufficient, it must be very clear and distinctive. If it is just a fine line between the auditor s expectation and what the operator has in place, the suitability should not be assessed as insufficient. Page 14 of 19

15 3.2.3 Assessing Effectiveness The effectiveness criteria, compared to suitability criteria, are more straight-forward and prescriptive. Each of these criteria should be identifiable within the operator s system, operational environment, or organization. It is important that during the audit, each item is individually assessed. The methodology does not look for a generally effective state, but one where all the required criteria are in place. Only if all the effectiveness criteria are present, can the assessment be effective. Some of the items listed in the effectiveness criteria may not be as clearly identifiable by speaking to just one person. In those cases it is important that an accurate sample is taken from the employee base. Unlike in the traditional method of documented and implemented, the effectiveness criteria do not need to be documented in a controlled document. The focus is on the implementation. It is possible that the effectiveness criteria for a particular ISARP list(s) one or more other ISARPs. In these cases, the listed ISARP needs to be in conformity in order for the credit to be given for that particular criterion. It is however not necessary that the identified ISARP is assessed as being effective, unless it is specifically mentioned Overall ISARP Assessment Once all three components of the Assessment Tool have been assessed, the auditor can then make an assessment on the full ISARP regarding the effectiveness. If the result of that assessment is not positive (meaning the ISARP is not effectively implemented), it is important to note what in particular was not in place. This is important since the operator needs to know the specific missing items to define what type of corrective action is to be taken, in case corrective action is planned. As mentioned beforehand, the effective assessment can only be made if ALL the required criteria are in place. If only one is missing, the assessment must be non-effective. Suitability, while fully recognized as being relevant, will not affect the assessment of the umbrella ISARP for the time being. 3.3 Recording of Suitability The suitability criteria are generally not very specific and not prescriptive in nature. Therefore it is difficult to make an objective assessment on suitability. For this reason, the suitability needs to be addressed and discussed with the operator in order to come to a conclusion. For the purpose of the audit, the suitability will not affect the overall effectiveness assessment as recorded in the umbrella ISARP. In other words, if an auditor concludes that suitability is not given for a specific ISARP, but all the effectiveness criteria are fulfilled, the final assessment in the umbrella ISARP could still be conformity - meaning Effective. The suitability assessment however will need to be captured in the auditor comment of the affected ISARP. The auditor would make a statement in the comment field that he or she has made the assessment that the suitability is not provided, including a justification / reason. Page 15 of 19

16 Example of an auditor comment on suitability Suitability Due to the size and complexity of the Operator s organization, the reporting tool is deemed unsuitable by the auditor because it does not cater for the large number of staff members who may not be very well educated in the use of complex electronic forms and tools. For a frontline handling agent, the use of the reporting tool would seem too complex. 3.4 Recording of Effectiveness Status Depending on the scenario, the result of the assessment of the effectiveness criteria needs to be recorded in the proper location. Unlike the discussion of the suitability, which has no effect on the umbrella ISARP, missing effectiveness criteria will have an effect on the umbrella ISARP. The following illustration gives an overview of what needs to be recorded where: Page 16 of 19

17 3.4.1 Recording on ISARP Level The effectiveness criteria are individually listed in the audit checklist for each ISARP that is designated with an [Eff] designator. The criteria can be checked separately. As the auditor verifies each item, it must be ticked/checked in the checklist, respectively in the audit software. If all criteria of all ISARPs are implemented and the whole ISARP is assessed as being effective, no further recording is necessary. If one or more effectiveness criteria are not implemented, the respective criteria are not ticked/checked in the checklist or audit software, which indicates to the reader of the report what is missing. The individual missing effectiveness criteria are then brought forward into the umbrella ISARP into the auditor comment. For simplification, the effectiveness criteria are numbered and they should be referred to by the number (i), (ii), Recording for Umbrella ISARP If parts of one ISARP, a full ISARP or multiple ISARPs have been assessed as non-effective due to the missing implementation of effectiveness criteria, the umbrella ISARP needs to be assessed as non-conformity for implementation. The individual ISARP would also show which effectiveness criteria have not been fulfilled as the corresponding items would not be checked / marked in the checklist. Since the umbrella ISARP covers multiple ISARPs, any missing effectiveness criterion will need to be attached to this one ISARP and the auditor comment needs to reflect that accordingly. To ensure the operator and any potential readers of the report can clearly identify the deficiencies, the comment needs to reference the ISARP by number, together with a description, respectively a reference to the criterion by number. Example of an auditor comment for the umbrella ISARP The Operator does not effectively implement all ISARPs designated by the [Eff] symbol. ORG The effectiveness criterion (i) is not implemented. The effectiveness criteria (i) (iv) are not implemented. The effectiveness criterion hazard register is not implemented. The introductory auditor comment is standardized and should always be used in case of a nonconformity against the umbrella ISARP: The Operator does not effectively implement all ISARPs designated by the [Eff] symbol. Page 17 of 19

18 3.4.3 Scenarios for Recording The following table intends to summarize the different scenarios which are possible. The table should be read from left to right, whereas the first column describes how many [Eff]-designated ISARPs are affected. Affected ISARPs Suitability criteria Effectiveness criteria Auditor comments in [Eff]-ISARP Umbrella ISARP All [Eff]-ISARPs Suitable for all Effective for all No comment Conformity / no comment One or more Not suitable for Effective for all Comment in auditor Conformity / no comment [Eff]-ISARPs one or more [Eff]- comment field of ISARP One or more [Eff]-ISARPs ISARPs Suitable for all Effectiveness criteria missing for one or multiple [Eff]-ISARPs which is not suitable No comment Not implemented Standard Auditor comment plus One, multiple or all [Eff]-ISARPs Not suitable for one or more [Eff]- ISARPs Effectiveness criteria missing for one or multiple [Eff]-ISARPs Comment in auditor comment field of ISARP which is not suitable ISARP Nr and list of missing criteria Not implemented Standard Auditor comment plus ISARP Nr and list of missing criteria 3.5 Repeated ORG Provisions If an ORG ISARP is repeated in other operational disciplines, and it has an [Eff] tag, the same logic applies to the effectiveness assessment of the ORG ISARP as for the assessment of conformity for [SMS] provisions. More specifically, the following rule applies: If an ORG ISARP has an [Eff] tag, and it is repeated in other operational disciplines (whether the ORG ISARP has an [SMS] tag or not is not relevant), and the suitability and / or effectiveness are assessed as not suitable / not effective in one or more of the operational disciplines, the corresponding suitability / effectiveness assessment is automatically carried over to the corresponding ORG ISARP. 3.6 Corrective Actions for Effectiveness For a non-conformity raised against the umbrella ISARP, a conventional CAR would be generated and the operator would have the option to close it if there is a wish to do so. The procedure to close this CAR would follow the conventional method to close a CAR. The main difference is that the CAR will need to address individual ISARPs by number to ensure corrective actions can be clearly linked to the correct standard and criteria. Page 18 of 19

19 As with any other CAR, the operator would need to come up with root cause, corrective action plan and final corrective action. The auditor would then verify the evidence and record which evidence has been verified. It can be expected that missing effectiveness criteria would be difficult to address within short periods of time particularly during renewal audits. Page 19 of 19