IS-BAO Audit Procedures Manual 2015

Transcription

1 IS-BAO Audit Procedures Manual 2015 Suite 16.33, 999 University Street Montreal, Quebec, H3C 5J9, Canada Tel Fax:

2 IS-BAO Audit Procedures Manual Copyright All rights reserved No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without the prior permission of IBAC. January 1, page - ii

3 IS-BAO Audit Procedures Manual RECORD OF AMENDMENTS Number Date Date Entered Entered By Second Edition January 1, 2004 included included 1 January 1, 2005 included included Third Edition January 1, 2006 included included 1 January 1, 2007 included included 2 January 1, 2008 Included included 3 January 1, 2009 included included 4 January 1, 2010 included included 5 January 1, 2011 included included Fourth Edition January 1, 2012 Included included 1 January 1, 2013 included included 2 January 1, included included 3 January 1, 2015 included included January 1, page - iii

4 IS-BAO Audit Procedures Manual Page intentionally left in blank January 1, page - iv

5 IS-BAO Audit Procedures Manual LIST OF EFFECTIVE PAGES Page No. Effective Date Page No. Effective Date ii thru x Jan 1, thru 52 Jan 1, ii Jan 1, Jan 1, 2014 iii Jan 1, Jan 1, 2014 iv Jan 1, Jan 1, 2014 v Jan 1, Jan 1, 2014 vi Jan 1, Jan 1, 2014 vii Jan 1, Jan 1, 2014 viii thru x Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, 2014 January 1, page - v

6 IS-BAO Audit Procedures Manual Page No. Effective Date Page No. Effective Date 77 Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, Jan 1, 2014 Page intentionally left in blank January 1, page - vi

7 IS-BAO Audit Procedures Manual FOREWORD This Manual is prepared by the and is issued under the authority of the IS-BAO Standards Board. The Audit Procedures Manual has been developed pursuant to the International Standards for Business Aircraft Operations (IS-BAO). The Manual is designed to provide for a systematic and independent examination to determine if an organization is in conformity with the provisions of IS-BAO. Application of IS-BAO by an aviation operator is voluntary. Concurrently, measurement of the organization s adherence to the recognized industry code of practice is the operator s choice. A variety of options are available for conducting audits to determine adherence to IS-BAO. Operators can apply the procedures for: a. internal audits; b. as a basis for development of a programme for audits of sub-contracted flight operations; c. third party independent external audit for company reference only; or d. use of a third party accredited auditor for registration purposes. IBAC has developed a system of registering operators as being in conformity with the IS-BAO standards. This allows the managers of the operation to demonstrate to the organization s executives, Board of Directors and customers that the organization adheres to an internationally recognized code of practice, not unlike the International Standards Organization (ISO) system. Operators that qualify after undergoing an audit by an accredited third party auditor will receive a Certificate of Registration from IBAC. The audit procedures were developed in recognition of proven industrial management auditing concepts and follow the principles set out in ISO Guidelines for Quality and/or Environmental Management System Auditing. The procedures will ensure that audits are completed consistently and in accordance with a systematic, objective and proven process. The Manual also describes the IS-BAO Standards Board responsibilities in relation to the standards. January 1, page - vii

8 IS-BAO Audit Procedures Manual Page intentionally left in blank January 1, page - viii

9 IS-BAO Audit Procedures Manual TABLE OF CONTENTS Record of Amendments... iii List of Effective Pages... iv Foreword... vii Table of Contents... ix 1.0 Purpose Purpose Statement Structure of the Manual Definitions and Audit Terminology IS-BAO an International Standard for Business Aircraft Operations Background IS-BAO Standards Board Keeping the IS-BAO Current IS-BAO Revisions IS-BAO Registration Audits Certificate of Registration Authority for Registration Authority for the Audit Program Accredited Auditors Auditor Qualifications HAI Helicopter Mission Specific Standards (HMSS) IS-BAO Workshops Audit Program Quality Assurance Audit Objectives Audit Principles Conflict of Interest Confidentiality Audit Monitoring Observers Audit Feedback Audit Procedures Audit Purpose The Audit Standard Audit Scheduling Audit Planning Audit Opening Meeting Audit Protocols Using the Audit Protocols Conducting the Audit Determining if an Audit is Successful Judgement and Decision Making Closing Meeting Agreement Between Operator and Auditor Evaluating the Operator s SMS What is a Safety Management System (SMS)? Measuring Safety Performance Stages in the Development of a Safety Management System Determining the Objectives of the Evaluation Preparations to Evaluate the SMS Scoping the Evaluation Measuring Soundness, Appropriateness and Effectiveness Conducting the Evaluation Preparing Audit Findings Introduction Making Findings January 1, page - ix

10 IS-BAO Audit Procedures Manual 6.3 Determining Criticality Audit Reports Report Standards Determining Audit Results Audit Report Contents Audit Forms Audit Report Form Detailed Audit Protocols IS-BAO Audit Finding Form Appendix A Example of Auditor/Operator Agreement Appendix B Example of Audit Planning Letter Record of Amendments... iii List of Effective Pages... iv Foreword... vii Table of Contents... ix 1.0 Purpose Purpose Statement Structure of the Manual Definitions and Audit Terminology IS-BAO an International Standard for Business Aircraft Operations Background IS-BAO Standards Board Keeping the IS-BAO Current IS-BAO Revisions IS-BAO Registration Audits Certificate of Registration Authority for Registration Authority for the Audit Program Accredited Auditors Auditor Qualifications HAI Helicopter Mission Specific Standards (HMSS) IS-BAO Workshops Audit Program Quality Assurance Audit Objectives Audit Principles Conflict of Interest Confidentiality Audit Monitoring Observers Audit Feedback Audit Procedures Audit Purpose The Audit Standard Audit Scheduling Audit Planning Audit Opening Meeting Audit Protocols Using the Audit Protocols Conducting the Audit Determining if an Audit is Successful Judgement and Decision Making Closing Meeting Agreement Between Operator and Auditor Evaluating the Operator s SMS January 1, page - x

11 IS-BAO Audit Procedures Manual 5.1 What is a Safety Management System (SMS)? Measuring Safety Performance Stages in the Development of a Safety Management System Determining the Objectives of the Evaluation Preparations to Evaluate the SMS Scoping the Evaluation Measuring Soundness, Appropriateness and Effectiveness Conducting the Evaluation Preparing Audit Findings Introduction Making Findings Determining Criticality Audit Reports Report Standards Determining Audit Results Audit Report Contents Audit Report Form Detailed Audit Protocols IS-BAO Audit Finding Form Appendix A Example of Auditor/Operator Agreement Appendix B Example of Audit Planning Letter Record of Amendments... iii List of Effective Pages... iv Foreword... vii Table of Contents... ix 1.0 Purpose Purpose Statement Structure of the Manual Definitions and Audit Terminology IS-BAO an International Standard for Business Aircraft Operations Background IS-BAO Standards Board Keeping the IS-BAO Current IS-BAO Revisions IS-BAO Registration Audits Certificate of Registration Authority for Registration Authority for the Audit Program Accredited Auditors Auditor Qualifications HAI Helicopter Mission Specific Standards (HMSS) IS-BAO Workshops Audit Program Quality Assurance Audit Objectives Audit Principles Conflict of Interest Confidentiality Audit Monitoring Observers Audit Feedback Audit Procedures Audit Purpose The Audit Standard Audit Scheduling Audit Planning January 1, page - xi

12 IS-BAO Audit Procedures Manual 4.5 Audit Opening Meeting Audit Protocols Using the Audit Protocols Conducting the Audit Determining if an Audit is Successful Judgement and Decision Making Closing Meeting Agreement Between Operator and Auditor Evaluating the Operator s SMS What is a Safety Management System (SMS)? Measuring Safety Performance Stages in the Development of a Safety Management System Determining the Objectives of the Evaluation Preparations to Evaluate the SMS Scoping the Evaluation Measuring Soundness, Appropriateness and Effectiveness Conducting the Evaluation Preparing Audit Findings Introduction Making Findings Determining Criticality Audit Reports Report Standards Determining Audit Results Audit Report Contents Audit Forms Audit Report Form Detailed Audit Protocols Safety Management System Organization and Personnel Requirements Training and Proficiency Flight Operations Operations in International Airspace Aircraft Equipment Requirements Aircraft Maintenance Requirements Company Operations Manual Emergency Response Plan Environmental Management Occupational Health and Safety Transportation of Dangerous Goods Security IS-BAO Audit Finding Form Appendix A Example of Auditor/Operator Agreement Appendix B Example of Audit Planning Letter Record of Amendments... iii List of Effective Pages... iv Foreword... vii Table of Contents... ix 1.0 Purpose Purpose Statement Structure of the Manual Definitions and Audit Terminology IS-BAO an International Standard for Business Aircraft Operations Background IS-BAO Standards Board... 3 January 1, page - xii

13 IS-BAO Audit Procedures Manual 2.3 Keeping the IS-BAO Current IS-BAO Revisions IS-BAO Registration Audits Certificate of Registration Authority for Registration Authority for the Audit Program Accredited Auditors Auditor Qualifications HAI Helicopter Mission Specific Standards (HMSS) IS-BAO Workshops Audit Program Quality Assurance Audit Objectives Audit Principles Conflict of Interest Confidentiality Audit Monitoring Observers Audit Feedback Audit Procedures Audit Purpose The Audit Standard Audit Scheduling Audit Planning Audit Opening Meeting Audit Protocols Using the Audit Protocols Conducting the Audit Determining if an Audit is Successful Judgement and Decision Making Closing Meeting Agreement Between Operator and Auditor Evaluating the Operator s SMS What is a Safety Management System (SMS)? Measuring Safety Performance Stages in the Development of a Safety Management System Determining the Objectives of the Evaluation Preparations to Evaluate the SMS Scoping the Evaluation Measuring Soundness, Appropriateness and Effectiveness Conducting the Evaluation Reserved Preparing Audit Findings Introduction Making Findings Determining Criticality Audit Reports Report Standards Determining Audit Results Audit Report Contents Audit Forms Audit Report Form Detailed Audit Protocols Safety Management System Organization and Personnel Requirements Training and Proficiency Flight Operations January 1, page - xiii

14 IS-BAO Audit Procedures Manual 7 Operations in International Airspace Aircraft Equipment Requirements Aircraft Maintenance Requirements Company Operations Manual Emergency Response Plan Environmental Management Occupational Health and Safety Transportation of Dangerous Goods Security In-Flight Inspection IS-BAO Audit Finding Form Appendix A Example of Auditor/Operator Agreement Appendix B Example of Audit Planning Letter Appendix C - Auditing Aircraft Management Companies January 1, page - xiv

15 IS-BAO Audit Procedures Manual January 1, page - 1

16 IS-BAO Audit Procedures Manual 1.0 PURPOSE 1.1 Purpose Statement The Audit Procedures Manual provides IS-BAO auditors and operators with procedures for the conduct of audits pursuant to the IS-BAO standards established by the IS-BAO Standards Board The Audit Procedures Manual was initially reviewed by the IBAC Governing Board at its meeting of December 15, 2001 at which time approval in principle was given and responsibility for the ongoing maintenance of the Manual delegated to the IS-BAO Standards Board. 1.2 Structure of the Manual The IS-BAO Audit Procedures Manual consists primarily of standards and protocols for the conduct of IS-BAO registration audits. Explanatory and guidance material is included where considered necessary for clarity and guidance. 1.3 Definitions and Audit Terminology The following definitions and explanatory material apply to the IS-BAO auditing processes. Adequate: Assessment: Audit: Audit activities: Audit conclusion: Audit finding: Audit leader: Audit report: Closing meeting: Compliance: Conformity: Evaluation (SMS): Hazard: fulfilling minimal requirements; satisfactory; acceptable; sufficient. an appraisal of procedures or operations based largely on experience and professional judgement. a systematic and objective review of an operator s operations and maintenance framework to verify conformity with IS-BAO standards, and good aviation safety practices. those activities and procedures by which information is obtained to verify the audited operator is in conformity with applicable IS-BAO provisions. Such activities may include, but are not limited to, interviews, observations, inspections and the review of files and documents. the outcome of the audit provided by the auditor after consideration of the audit objectives and all audit findings. a problem or cause-and-effect statement, and is accompanied by specific examples of the observed condition. These are significant issues requiring the operator's immediate attention. Findings may relate to one or more nonconformities but should be sufficiently significant to require urgent action. an individual accredited by the accredited auditing organization responsible for the planning and conduct of an audit, including the production of an audit report. a standardised means of reporting the audit findings to operators. a meeting at the end of the audit, the purpose of which is to brief the client on the audit findings. a product or activity that meets required characteristics and was performed in accordance with the required processes and system requirements. The output of one or more processes and the controls under which the processes occurred. meets required characteristics. The fulfilment of a requirement in the IS-BAO. a systematic assessment of the appropriateness and effectiveness of an operator s efforts to manage the safety of the operation. the condition or circumstance that can lead to physical injury or damage. January 1, page - 2

17 IS-BAO Audit Procedures Manual Inspection: Non-conformity: Major nonconformity: Minor nonconformity: Objective evidence: Objectivity: Operator: Procedure: Process: Product: Protocol: Quality assurance: Random sample: Remedial Action Plan: Risk: Safety: Sampling: Satellite/Additional Operating Base Standard: Verification: Working papers: the basic activity of an audit, involving examination of the specific characteristics of the operator s systems, procedures, processes and documents. the non-fulfilment of a requirement specified in the IS-BAO. a non-conformity that in itself constitutes a significant risk to the safety of the operation. Note: A significant risk to safety is any risk identified as moderate or high via the methodologies described in ICAO Doc 9859, Safety Management Manual. a non-conformity that does not constitute a significant risk to safety. It may be an isolated instance of non-conformity, a lack, or error in documentation of a process or procedure that was correctly executed, or a similar instance. information, which can be proved to be true, based on facts obtained through observation, measurement, test or other means. freedom of bias engendered by the illogic of emotion or prejudice. A person, organization or enterprise engaged in or offering to engage in an aircraft operation. a series of steps followed in a methodical manner to complete an activity what shall be done and by whom, when, where, and how it shall be completed; what materials, equipment, and documentation shall be used, and how it shall be controlled. a system of activities that uses resources to transform inputs into outputs. the end result of a procedure or process. a document, which organizes audit procedures into a general sequence of audit steps and describes the steps in terms of specified standards or practices to be verified. that part of the overall management of the IS-BAO audit program that focuses on providing confidence that quality requirements of the program are being fulfilled. sample drawn in such a manner that every possible item has an equal chance to be selected. a plan to resolve a non-conformity that describes the actions that will be taken and the time frame for completing them. the consequence of a hazard, measured in terms of severity and likelihood. the state in which the risk of harm or damage is limited to an acceptable level as low as is reasonably practical. the inspection of a representative portion of a particular characteristic to produce a statistically meaningful assessment of the whole. an airport or heliport other than the operator/ management company s main base at which one or more aircraft and personnel are based and managed by the operator/management company. established criteria used as a basis for measuring an operator s level of conformity. the independent review, inspection, examination, measurement, testing, checking, observation, and monitoring to establish and document that products, processes, practices, services and documents conform to specified standards. This includes evaluating the effectiveness of management systems. all documents required by the auditor or audit team to plan and implement the audit. January 1, page - 3

18 IS-BAO Audit Procedures Manual 2.0 IS-BAO AN INTERNATIONAL STANDARD FOR BUSINESS AIRCRAFT OPERATIONS 2.1 Background In February 2000, the IBAC Governing Board agreed to conduct an analysis of the feasibility of developing standards for business aircraft operations. As a result, a project was initiated under the direction of the IBAC Safety and Flight Operations Work Group The project was completed in three phases through 2000 and The first phase involved a proof of concept. An initial standards document was developed for discussion purposes and a number of flight department representatives met in a series of focus group meetings to consider the proposal and its potential benefits. Following the focus group sessions, the Governing Board reviewed the results and approved the development of the standards and two Generic Company Operations Manuals (GCOMs), one in a format familiar to operators in North America and one in a format familiar to many European operators The second phase involved the finalization of the Standards, and the GCOM documents. This was done through a number of focus groups, a thorough review of comments from a number of sources, and on-site testing at a number of volunteer flight departments The third phase consisted of the development of processes for management of the IS-BAO through authority issued by the IBAC Governing Board to the Standards Board. The product of Phase 3 was a Business Plan for the ongoing management of the IS-BAO and the Audit Procedures Manual, and all related requirements, including the programme funding The IS-BAO an International Standards for Business Aircraft Operations was released in early 2002, following completion of the project. At the time, the IBAC Governing Board approved in principle a Business Plan for the ongoing maintenance of the standards and the related programs such as the auditing procedures In 2010 and 2011 IBAC worked with the Helicopter Association International (HAI), the British Helicopter Association (BHA) and the European Helicopter Association (EHA) to adapt the IS- BAO to include helicopter operations. Pursuant to the agreement under which this project was undertaken, the HAI developed a set of helicopter mission specific standards (HMSS), to augment the IS-BAO. Operators who have implemented the IS-BAO can undergo an IS-BAO audit and become registered as being in conformance with the IS-BAO. They then can undergo an audit relative to their specific mission and also become registered as being in conformance with those standards. 2.2 IS-BAO Standards Board The IS-BAO Standards Board has been delegated responsibility by the IBAC Governing Board for the maintenance of the IS-BAO standards and for overseeing the auditing process The Standards Board consists of members nominated by the IBAC Members associations and the IS-BAO partner helicopter industry associations. At least three members must be aircraft operators, one must be involved in the helicopter industry and one must have aircraft maintenance expertise The Standards Board meets at least once per year to consider proposed changes to the IS-BAO and the Audit Procedures Manual. Professional standards services are provided by the IBAC Secretariat through the IS-BAO Program Director who will assemble and provide an analysis of recommendations for change. January 1, page - 4

19 IS-BAO Audit Procedures Manual 2.3 Keeping the IS-BAO Current Proposals for change to the IS-BAO can come from many sources. In essence, all recommendations will be assessed by the IBAC Director, IS-BAO Program and any proposals that will enhance the efficiency or effectiveness of the IS-BAO, will be submitted to the Standards Board for consideration. 2.4 IS-BAO Revisions Decisions of the Standards Board to amend the IS-BAO are contained in the Standards Board meeting decision record that is posted at IS-BAO Standards Board. The current IS-BAO documents are loaded on the IS-BAO web site where they may be downloaded by IS-BAO holders and accredited auditors During the period Jan 1 to Jun 30 registration and renewal audits may be done, at the discretion of the operator and the auditor, in accordance either with the standards of the previous year or the current year. However, on and after July 1 audits must be conducted in accordance with the current year standards. January 1, page - 5

20 IS-BAO Audit Procedures Manual 3.0 IS-BAO REGISTRATION AUDITS 3.1 Certificate of Registration Operators who have purchased the IS-BAO and completed implementation may choose to register their adherence to IS-BAO standards with IBAC. There are many benefits of doing this, as described in IBAC s IS-BAO brochure and repeated on the IBAC website Further to the advantages described in the brochure, operators may find that insurance companies look favourably on clients that meet an internationally recognized code of practice IBAC will issue a Certificate of Registration valid for two years from the end of the month in which the audit was completed, unless otherwise specified, to operators that successfully complete an audit by an accredited third party auditor. The Certificate is in an attractive format that can be wall-mounted so that staff, passengers and customers are able to see that the organization has met the well-recognized international industry standard The IS-BAO registration system is progressive in nature, requiring registrants to achieve and maintain SMS program standards. A registrant is required to progress to and maintain at least Stage Two to remain registered. Stage Three is an optional but desirable level of achievement for operators. Other features of the registration program are: a. The Certificate of Registration is normally issued for a period of two years from the end of the month in which the audit was completed, unless otherwise specified. b. Operators who have demonstrated a high degree of SMS maturity and sustainability during a Stage Three audit may, upon recommendation by the auditor, be granted registration for a three-year period. c. If a registration renewal audit is conducted within the 90-day period prior to expiry of the operator s registration, the Certificate of Registration will be renewed for the full period from the original expiry date. d. Registrations that have lapsed for more than six months will be treated as initial applications, unless an extension has been previously approved. e. During the period Jan 1 to Jun 30 initial and renewal audits may be performed, at the discretion of the operator and the auditor, in accordance either with the standards of the previous year or the current year. However, on/after July 1 audits must be conducted in accordance with the current year standards. f. In the event that an operator fails to progress to at least or maintain sstage twotwo, follow procedures shown at the following link: IS-BAO Policy Letter IS-BAO Policy on SMS Progress Requirements. g. In cases where a registered operator is split into two or more separate entities, it may be possible to register both at the same stage level. See IS-BAO Policy Letter IS-BAO Policy on Registration With Split Entity for details At the completion of the registration audit, the auditor will forward a completed audit report form and protocols to the IS-BAO Audit Manager at auditmanager@ibac.org for review. The operator will then complete the IS-BAO registration or renewal application and forward it to the IBAC Montreal office with the appropriate fee. An electronic copy of the organization s logo is requested for inclusion on the certificate. A second certificate or additional certificates to be carried on board aircraft as proof of registration is provided are available at no additional cost. Additional certificates are available for a nominal fee. 3.2 Authority for Registration The IBAC Governing Board has delegated to the Audit Manager authority to issue of the Certificate of approve IS-BAO Rregistrations to the IS-BAO Audit Manager. Registration recommendations will be made to the Audit Manager by an accredited third party auditor. The Certificate of Registration will be issued when the recommendation and the related audit report January 1, page - 6

21 IS-BAO Audit Procedures Manual have been reviewed and accepted by via the audit review process described in IBAC policy at the following lthe IS-BAO Audit Manager or the person appointed to act on his/her behalf. January 1, page - 7

22 IS-BAO Audit Procedures Manual ink: Audit Program Oversight Policy. 3.3 Authority for the Audit Program The IS-BAO Standards Board has been delegated accountability for the audit programme by the IBAC Governing Board. As such, the Audit Procedures Manual is issued under the authority of the IBAC Governing Board IBAC retains no auditing responsibility or authority for companies wishing to use the Audit Procedures Manual or the Internal Audit Procedures Manual for internal auditing purposes. Nevertheless, IBAC is interested in experiences of such operators and is willing to accept recommendations from all sources. 3.4 Accredited Auditors IBAC will issue audit accreditation and appropriate credentials, to individuals that apply for such accreditation and who meet the requirements established in this Manual and applicable IBAC policies. These requirements reflect those specified in ISO Guidelines for Quality and/or Environmental Management Systems Auditing. A current list of accredited auditors, the scope of their accreditation and their organizational affiliation is maintained on the IBAC website. The scope of an auditor accreditation is based on their aviation experience and areas of expertise. can be: fixed wing: rotary wing, or both fixed and rotary wing Registration audits must be conducted under the direction of an accredited IS-BAO auditor who is experienced in the type of aviation operations primarily being conducted and whose scope of accreditation includes that type of operations. Where required, a subject matter expert who is not an accredited auditor and does not fully meet the qualifications specified in section 3.5 or an auditor in training may conduct a segment of an audit under the supervision of an accredited auditor. a. Auditors with little or no experience in either aviation operations or maintenance may not conduct a full audit in which they sign the audit report form as the principal or audit team leader unless they meet additional requirements shown at Accreditation Requirements. b. Auditors not conducting an IS-BAO audit within the past 12-months must receive additional training and provide proof of audit knowledge through testing. These issues are described in detail at web address shown in item a., above.auditor Currency Requirements c. The auditor shall only audit operators whose language the governing and technical documents are in is one in which they can speak and read, unless they utilize the services of a language specialist, who must be part of the audit team. 3.5 Auditor Qualifications The IBAC standards for the qualifications and experience required for accreditation of all auditors are described in IS-BAO Policy at the following link: Auditor Accreditation Requirementsas follows (For complete information regarding auditor qualifications and accreditation procedures, see : January 1, page - 8

23 IS-BAO Audit Procedures Manual working knowledge of the operational and maintenance provisions of civil air regulations and International Civil Aviation Organization standards and recommended practices; working knowledge of aviation operations or maintenance management; a minimum of five years experience in aviation operations or maintenance; submit a course completion certificate for an ISO, ASQ, or equivalent audit course or provide proof of substantial experience in aviation safety auditing; for initial accreditation have completed a Fundamentals of IS-BAO Workshop and IS-BAO Auditing Workshop within the previous 24 months: for accreditation renewal have completed an IS-BAO Auditing Workshop within the previous 24 months. If the auditor has not conducted an IS-BAO audit within the current validity period, a Fundamentals of IS-BAO Workshop is also required for accreditation renewal; provide evidence of experience in aviation safety management; and demonstrate excellent written and oral communication skills The scope of the auditor s accreditation will be issued in accordance with the above policy. The knowledge and experience for the scope of the auditor s accreditation are as follows: Fixed Wing. a working knowledge of fixed wing aviation operations or maintenance management, and. a minimum of three years experience in fixed wing aviation operations or maintenance. Rotary Wing. a working knowledge of rotary wing operations or maintenance management, and. a minimum of three years experience in rotary wing operations or maintenance. Both Fixed and Rotary Wing Meet both of the above knowledge an experience requirements Procedures for Auditor Accreditation Those wishing to obtain audit accreditation should complete the online application on the IBAC website at Initial Auditor Application Form. Applicants should ensure that their application contains sufficient information to demonstrate that they meet the requirements set out in section 3.5 of this document and attach all auditor and SMS course certificates and resume Auditor Currency An auditor not conducting an IS-BAO audit within their accreditation validity period must participate in both the Fundamentals of IS-BAO and IS-BAO Auditing Workshops for accreditation renewal. Auditors not conducting an audit or completing an IS-BAO workshop within the previous 12 months must present evidence of completing one of the following: a Fundamentals of IS-BAO Workshop; an approved aviation SMS or auditor course, either in a classroom or on-line; or a substantive level of implementation support for a client preparing for an IS-BAO audit. Additionally, each auditor completing one of the above requirements must also take an IS-BAO auditor requalification examination consisting of not less than 40 questions. 3.6 HAI Helicopter Mission Specific Standards (HMSS) January 1, page - 9

24 IS-BAO Audit Procedures Manual HAI will accredit IS-BAO auditors, who meet the standards specified in their accreditation process, to conduct audits for operator HMSS accreditation. The HAI auditor accreditation and HMSS audit requirements are managed by HAI. 3.7 IS-BAO Workshops Fundamentals of IS-BAO Workshops are held throughout the year to assist operators who are implementing the IS-BAO and to meet IS-BAO auditor accreditation requirements. IS-BAO Auditing Workshops specifically for auditor accreditation and renewal will also be conducted throughout the year. Information on the workshops is posted at IS-BAO Workshop Information and the workshop schedule is posted at IS-BAO Workshop Schedule. 3.8 Audit Program Quality Assurance In order to provide for continuing improvement of the IS-BAO system and uniform application of the audit programme, the IBAC IS-BAO Audit Manager administers a quality assurance (QA) programme. The QA programme includes: the initial evaluation of persons who apply to become accredited IS-BAO auditors; assessment of the applicant s performance during the IS-BAO Workshops; reviews of audit reports; quality assurance consultations with auditors and operators; and monitoring of operator audits. The activities of the QA program are guided with the principles contained in ISO A full description of the IS-BAO audit program oversight provisions may be found at the following link: Program Oversight 3.9 Audit Objectives The audit objectives are: a. to determine that the operator is in conformity with the IS-BAO standards; b. to observe and assess the operator s adherence to the recommended practices contained in the IS-BAO; c. to assist the operator to identify any safety deficiencies; d. where a Certificate of Registration is desired by the operator, to make a recommendation accordingly to the IBAC Secretariat; and e. to observe upon and make recommendations to the Standards Board regarding potential changes to the IS-BAO standards Audit Principles Audit principles are: a. Transparency and disclosure; b. Timeliness; c. Systematic, consistent and objective; d. Fairness; and e. Quality Transparency and disclosure January 1, page - 10

25 IS-BAO Audit Procedures Manual The philosophy of no secrets is very important during the audit and in the reporting process. By openly discussing concerns or apparent shortcomings, as soon as they are identified, the auditor will provide the operator with the opportunity to provide additional information or in some cases take immediate action to resolve the issue. Audit results including all identified non-conformities and findings, will be provided to the operator at the end of the audit through a verbal debriefing and subsequent provision of a written report. The written report must be consistent with the verbal debriefing. The report or reports, provided to the operator must be consistent with the report that is provided to the IBAC Audit Manager Timeliness The audit report will be produced and provided by the auditor to the client operator on a timely basis. When IS-BAO registration is desired: a. Client operator accepts the report and develops remedial action plan if required. b. Auditor accepts the remedial action plan for minor non-conformities. c. Auditor will forward the completed audit report, including protocols and findings to IBAC S Ithe S-BAO Audit Manager for review and acceptance. Auditors must submit the completed report to IBAC within 5 weeks following the closing meeting or inform the Audit Manager of the circumstances causing the delay. d. Client operator will submit the registration application or the renewal application form to the IBAC Montreal office. e. IBAC will issue the Certificate of Registration Systematic, consistent and objective Audits will be conducted in a systematic, consistent and objective manner. There should be no variation in the scope, depth and quality of the audits from that which was agreed between the operator and the auditor. However, it must be recognized that the depth of investigation may have to be modified during the course of the audit in order that the auditor can accurately identify the cause(s) of non-conformities Fairness Audits are to be conducted in a manner such that operators are given every opportunity to monitor, comment and respond. As previously noted such opportunities should be provided as soon as a specific non-conformity is identified or suspected Quality Audits will be conducted in recognition of quality audit concepts including the use of qualified auditors. The four fundamental rules for auditing should be applied: a. Audits provide information for decisions. b. Auditors are qualified to perform their tasks. c. Measurements are taken against defined requirements. d. Conclusions are based on facts Conflict of Interest It is a well acceptedwell-accepted principle that auditors should not audit their own work. Therefore, auditors who do provide consulting services to operators should not normally conduct January 1, page - 11

26 IS-BAO Audit Procedures Manual IS-BAO registration audits if the audit involves auditing of any work that they or their firm, have done At the same time it is understood that an auditor may often be in the best position to provide strategic guidance to an operator on addressing a shortcoming or solving a problem and that, as such, should not disqualify them from conducting future audits. When offering strategic guidance, the auditor may present two or more options without indicating a bias toward one. For example, the auditor may point out that fatigue management guidance is located in the IS-BAO Implementation Guidance (IG), the IS-BAO Generic Company Operations Manual (GCOM), and on various websites. However, if the auditor offers opinion or outright assistance, they have crossed the line and provided consulting services. Therefore, auditors must govern themselves accordingly, and should they wish to conduct an IS-BAO registration audit of an operator to whom they or members of their firm have provided consulting, they must obtain prior approval from the IBAC Audit Manager and disclose the full details of these facts in the Audit Report Confidentiality Audits conducted for the purpose of registration to IS-BAO are operator confidential. IBAC will retain audit reports in a secure location and they will be made available only to IBAC staff and the Standards Board. Results of the audit will be used only for the purposes of determining if an organization is to receive a Certificate of Registration and for audit programme quality assurance Any audit conducted pursuant to the IS-BAO Audit Procedures Manual for any purpose other than registration will be provided to the operator alone and, as such, IBAC will not retain a copy or record of results Audit Monitoring IBAC maintains an audit quality assurance program that includes monitoring a percentage of the registration audits. The Monitors are either IBAC staff or persons designated by the Director, IS- BAO Program. The Monitors will only observe the audit and will not become involved in the audit process. Auditors and the operator will be advised in advance if an audit will be monitored. Details of the monitoring program, procedures and protocols may be found at the following link: Monitoring Program 3.14 Observers Observers to an audit are personnel other than IBAC staff who participate in the audits, with permission of the operator, for audit training and audit educational reasons. Observers can provide some assistance in the administration of the audit, but may not actively participate in interviews and the determination of audit findings, unless the team leader, for training purposes, duly authorises and monitors them Audit Feedback A request for audit feedback iswill be sent to all client operators after their audit has been reviewed by the IBAC IS-BAO Audit Manager. This feedback is used to improve the IS-BAO is program and audit quality. Input from client operators is very important in maintaining the quality and appropriateness of the IS-BAO program. recipients of the feedback form Operators are encouraged to submit a describeption of their audit experience and offering suggestions for improvement. January 1, page - 12

27 IS-BAO Audit Procedures Manual Page intentionally left in blank January 1, page - 13

28 IS-BAO Audit Procedures Manual 4.0 AUDIT PROCEDURES 4.1 Audit Purpose Aircraft operators choose to be audited to assist them in effectively managing the safety of their operation and to ensure that they are in conformity with the requirements of the IS-BAO and with their manuals, procedures and directives. For an audit to be effective it must both assess the operator s conformity with the IS-BAO, and be of value to the operator. By reviewing manuals and procedures, examining records, interviewing personnel and observing operations the auditor can arrive at a conclusion of conformity and the effectiveness of their safety management efforts. 4.2 The Audit Standard The basic standard to which the audit is conducted is the IS-BAO. In conducting the audit the auditor must determine if the operator is in conformity with the requirements that are applicable to the category of aircraft operated (aeroplanes or helicopters) and the type and nature of the operations conducted. The IS-BAO provides considerable latitude for the operator to develop programs and procedures that are suited to the size and complexity of the individual operation and to publish them in manuals and directives. While a small operation, such as one aircraft and less than five people, still must meet the standards specified in the IS-BAO they would be expected to use basic programs, systems and procedures than a large operation. In conducting the audit the auditor must not only determine if these programs and procedures are in conformity with the requirements of the IS-BAO, but that they are appropriate to the size and complexity of the operation and that it is being conducted in conformity with the company manuals and directives as they relate to IS-BAO requirements The operator Safety Management System (SMS) is the cornerstone of the IS-BAO. Hence, it is a key item of the audit. An evaluation of the operator s SMS will include the related standards that are an integral part of their SMS. These related standards include: a. organization structure, b. duties and responsibility of personnel, c. training and proficiency programs, d. flight operations procedures and processes, e. standard operating procedures, f. maintenance control system, etc. All of these components of the operator s SMS will be audited to the level required to determine that it is appropriate and effective. For initial audits and other Stage One audits, as well as the SMS evaluation the auditor must complete all of the remaining audit protocols contained in Chapter 8. When using the audit protocols in Chapter 8 it must be understood that they only paraphrase the standards, and as such, the standards in the IS-BAO must be considered when conducting the audit When conducting Stage Two or Stage Three audits the evaluation of the effectiveness of the SMS will involve in-depth examination of the systems, procedures and processes related to the standards that are an integral part of their SMS, such as those noted in Section 4.2.2, a thru f, to ensure they are appropriate and effective in managing safety risks and that they meet the related IS-BAO requirements. In that case the detailed protocol for chapter 3 (SMS) will be the primary audit document and the results of the SMS evaluation should be fully documented in that protocol. It will be supplemented by the detailed protocols for chapters 4 thru 15. The auditor will determine whether the detailed protocols for chapters 4 thru 15 will be audited in-depth or checked for basic conformance. Although sampling is allowable at the discretion of the auditor, at January 1, page - 14

29 IS-BAO Audit Procedures Manual least 50% of the protocols for each chapter (4 thru 15) must be assessed to substantiate that the operator is in conformity with the requirements of the IS-BAO. 4.3 Audit Scheduling Operators wishing to be scheduled for a registration audit should consult the list of accredited auditors on the IBAC Website at Accredited Auditors and liaise directly with the auditors Under normal circumstances, IBAC does not have to be consulted, but IBAC or any of its Member Associations, are prepared to provide assistance if needed.once a registration audit is scheduled, the auditor shall send the following information to Audit Manager via the following link as soon as practical: Planned Audit Notification a. The planned date of the audit, b. The name of the operator, c. The planned scope, full system or partial and SMS Stage, d. Names of other audit team members, and e. How many days onsite planned. 4.4 Audit Planning In order to enhance the efficiency of the audit and to minimize the risk of there being a difference in expectations between the operator and the auditor, it is recommended that the auditor provide the operator with information on the audit process and the audit protocols that will be used. It is also recommended that a formal audit agreement be established between the auditor and the client operator. Appendix BA contains an example of such an agreement. Once the agreement has been finalised the auditor should develop an audit plan and provide it to the client operator. Appendix CB contains a sample covering letter that auditors may adapt to their own use if they do not already have such a document Auditors should also obtain copies of relevant operator documents including confirmation that the operator holds an authorized current copy of the IS-BAO. This can be achieved by confirming that the operator has an IS-BAO website password that permits them to download updates to the IS-BAO. It may be useful for auditors to develop a questionnaire to obtain relevant information on operators with whom they are not familiar. It is strongly recommended that the auditor have the operator complete a copy of the IS-BAO audit protocol that is contained in section 8.2 indicating where and how each element has been addressed by the operator. This will expedite the work of the auditor enabling a focus on an evaluation of the operator s safety management system and facilitating spot-checking of the other elements as is necessary Most audits are performed for a single operator controlling all aspects of the operation. However, aircraft management companies providing comprehensive scheduling, flight operations and maintenance services also may receive IS-BAO registration. These companies require special audit considerations and procedures that are described in shown in brief in Appendix C and in detail IBAC policy at Aircraft Management Companies For renewal audits the auditor should: a) obtain the previous audit documentation from the operator to ensure that all non-conformities have been rectified effectively. If the operator does not have a copy of the audit, the auditor January 1, page - 15

30 IS-BAO Audit Procedures Manual may ask the operator to obtain a copy from IBAC. (IBAC will not send a copy of the audit directly to the auditor); and b) review of all the IS-BAO standards and recommended practices that have been incorporated since the last assessment to ensure the operator has effectively updated their policies, processes, procedures, and programs to reflect these changes. 4.5 Audit Opening Meeting The Auditor or Audit Team Leader is responsible for arranging for an Audit Opening Meeting with the manager of the operation and appropriate personnel. The agenda for the meeting should include: a. Introductions; b. Purpose of the audit (registration, organization internal reference, etc.); c. Schedule for the audit, including those to be interviewed; d. Overview of the audit protocols (checklists) that will be used; e. Channels of communication, and f. Dispute resolution. 4.6 Audit Protocols The audit protocols (checklists) have been prepared to guide the auditor through the process of assessing whether or not the operator is in conformity with the standards specified in the IS-BAO. Auditors must be aware that the audit protocols paraphrase the standards specified in the IS-BAO and as such, if there is any question regarding a particular protocol they must make reference to the IS-BAO. Auditors must complete the parts of any protocol used and review the protocol with the operator as part of the audit Closing Meeting. 4.7 Using the Audit Protocols The Detailed IS-BAO audit protocol in section 8.2 follows the structure of the IS-BAO. This protocol is used to do an in-depth assessment of one or more IS-BAO element (chapter). When this protocol is used the auditor should determine and indicate whether the operator is in conformity or not, with each of the referenced requirements. If that requirement is not applicable to the operator write check N/A in the appropriate column. The IS-BAO contains some standards and recommended practices that apply to all operators and others that apply only to either fixed wing operators or rotary wing operators. Some of the requirements refer specifically to aeroplanes or helicopters and as well the aeroplane specific requirements have an A suffix to the element of item number and the helicopter specific items have an H suffix. Also, some of the general provisions would not be applicable where the operator does not engage in that activity. Therefore, auditors must ensure that the audit report accurately reflects the type and nature of the operations conducted and the operator s conformance with the appropriate IS-BAO requirements If an item was not audited, the auditor should leave the three columns blank and note in the Remarks column the reason why it was not audited. The right hand column is used to record remarks related to the effectiveness of the operator s programs, systems and procedures and objective evidence of any non-conformities. Comments relating to recommended practices, good aviation safety practices or activities that are especially well done may also be recorded. These are items that can add value for the operator. At this point it is not necessary to determine whether there are minor or major non-conformities. That can be done when all instances of nonconformity and the related objective evidence have been analysed and, if appropriate, Findings are made. In such cases and Audit Finding Forms would then be prepared. See Chapter 6 - Preparing Audit Findings. January 1, page - 16

31 IS-BAO Audit Procedures Manual When a detailed protocol (8.2) item includes sub-elements, the auditor shall indicate their assessment of each of these sub-elements with a Y, N, or NA adjacent to each item, with appropriate references and remarks associated with the same. 4.8 Conducting the Audit An IS-BAO audit should be conducted as a systems audit. A system is a group of processes all working together to achieve a common goal. As such, the audit will examine the operator s safety management system, programs, procedures, and processes and assess how they are integrated to ensure conformity with the IS-BAO and effectively manage the safety of the operation. Therefore, the audit will be conducted at a higher level than a process or product audit. The audit will be conducted by collecting objective evidence through observations, interviews and examination of documents. This evidence will then be compared with the requirements of the IS- BAO and analysed. If there are shortcomings, that information will be used to develop observations or statements of non-conformity and if necessary, findings which identify the cause(s) of the problems. It must be remembered that non-conformities and findings must describe the problem, not the symptoms of the problem. Figure 4.1 General Model for Auditing While gathering objective evidence the auditor should ensure that they interview a variety of operator personnel as well as management personnel. This will help to determine the level of knowledge and understanding for key IS-BAO program elements throughout the organization and to form a complete picture of the organization and its activities While this process will facilitate an effective audit for a variety of types of operations, Appendix CIBAC policy regarding Auditing Management Companies presents some considerations for the audit of aircraft management companies which may be involved in commercial and noncommercial operations or non-commercial managed aircraft operations. 4.9 Determining if an Audit is Successful As the IS-BAO is based on industry best practices and many of the standards are performance based, that is they identify what must be achieved not how it is to be done, a considerable amount of judgement must be used in determining if an audit is successful. In general terms, the 1 Dennis R. Arter, Quality Audits for Improved Performance (3 rd ed. Milwaukee: Quality Press, 2003) p. 5. January 1, page - 17

32 IS-BAO Audit Procedures Manual audit is considered successful if the operator is in conformity with the standards contained in the IS-BAO in that there are no major non-conformities A minor non-conformity would be one that does not constitute a significant risk to safety. It may be an isolated instance of non-conformity, a lack, or error in documentation of a process or procedure that was correctly executed, or a similar instance. A major non-conformity is when there is a pattern of instances of minor non-conformities or a non-conformity with a standard specified in the IS-BAO that in itself constitutes a significant risk to the safety of the operation As previously mentioned, it is important that instances of apparent non-conformity be discussed with the operator as soon as they are noted. This will provide the client with an opportunity to provide additional information or in some cases, to undertake action to resolve the non-conformity prior to completion of the audit The auditor must provide evidence that the operator has implemented the processes and procedures to support required SMS performance for Stage 2 or Stage 3 registration. If the evidence is insufficient for SMS Effectiveness, the auditor may determine that a major nonconformity must be assessed. If minor non-conformities also exist, the auditor will document all non-conformities and remedial action plans and proceed with the closing meeting If the audit reveals a major non-conformity, the auditor will send the audit documents to the Audit Manager for review and they will be held until the auditor has obtained confirmation that all nonconformities have been rectified. The auditor will validate the successful implementation of all remedial action plans through a follow-up/partial audit, possibly no more than one day, consisting of a thorough review of the protocols that are related to the findings to ensure conformance If the follow-up/partial audit submitted to IBAC verifies that all applicable remedial actions have been taken and accepted by IBAC, the operator s registration will be renewed and remain valid for 24 months from the end of the original base audit month. For example, if the operator s base audit month is April and the audit is conducted in February and it reveals significant Findings such that IBAC would not be able to issue a normal 24 month registration, a follow-up/partial audit will be required to validate Remedial Actions. If that follow-up/partial audit takes place in August and is accepted by IBAC, the registration will be renewed for 24 months from the original base audit month of April Judgement and Decision Making As the IS-BAO provides considerable latitude for the operator to develop programmes and procedures that are suited to the individual operation and to publish them in manuals and directives, a high degree of judgement by the auditor is required when determining nonconformity. As the cornerstone of the IS-BAO is the operator s safety management system, it should be key in the auditor s decision making process. There would be a major non-conformity if there is clear evidence that: a. the operator has not correctly identified the safety-risks associated with the operation; or b. effective mitigation has not been developed and applied for a safety-risk assessed as Medium or higher After that point, the key consideration is the effectiveness of the process or procedure that the operator has developed and applied to mitigate the safety-risks inherent in the operation. Consideration of the company culture and the hazard identification and tracking system may be of assistance in making determinations. The company culture should be to effectively involve all operator personnel in the organization s safety management system. The organization s hazard identification and tracking system should actively involve all operator personnel, and management should analyse observations, develop and implement appropriate solutions and track their effectiveness. If there is clear evidence that these two key components of the operator safety management system are in place, considerable latitude in application of IS-BAO standards January 1, page - 18

33 IS-BAO Audit Procedures Manual is appropriate. Chapter 5 provides more information on evaluation of the operator s SMS and taking into consideration the stage of development that it is at Of course, it is a given that the process and procedures used by the operator must be in compliance with the civil aviation rules of the State of Registry and for commercial operations the State of the Operator Closing Meeting A Closing Meeting will be conducted for all audits prior to the departure of the auditor or audit team The purpose of the Closing Meeting is to communicate to the manager of the operation and other appropriate personnel, the results of the audit, and to do so immediately after completion of onsite activities The Closing Meeting will consist of a discussion of the audit findings, observations and results. It will be organized in accordance with the audit protocol, covering each module in turn The format and major headings for the Closing Meeting should be as follows: a. objective and scope of the audit; b. summary of the audit procedures, (including a list of those interviewed); c. an assessment of the overall appropriateness and effectiveness of the operators safety management system and other management system controls; d. non-conformities and audit findings; and e. where appropriate, audit programme quality and process recommendations to the Standards Board Agreement Bbetween Operator and Auditor Operators are responsible for arrangements with an auditor. If an IBAC Certificate of Registration is the desired outcome, an accredited third party auditor must conduct the audit Operators should consult the IBAC website or contact the IBAC Secretariat to obtain a current list of accredited auditors Although the agreement between the accredited auditor and the operator is solely the responsibility of the two parties involved, IBAC has provided an example contract (Appendix BA) that parties may wish to use as the basis for developing an agreement. January 1, page - 19

34 IS-BAO Audit Procedures Manual 5.0 EVALUATING THE OPERATOR S SMS 5.1 What is a Safety Management System (SMS)? A Safety Management System (SMS) is a process to explicitly identify, manage and measure the safety risks that inevitably occur in all aviation operations. It is the cornerstone of the performance based IS-BAO and it includes the key elements of the IS-BAO standards such as: a. organization structure, b. duties and responsibility of personnel, c. training and proficiency programs, d. flight operations procedures and processes, e. standard operating procedures, f. maintenance control system, etc. Therefore, it must be kept in mind when discussing the operator s SMS that these key elements are an integral part of it The IS-BAO has been designed so operators can customize their operator s safety management activities to suit the size and nature of their operation and their specific operating conditions. The operator s safety-risk profile, or the methods each organization uses to identify and manage safety-risks are particularly important when the auditor is planning and conducting the evaluation, and when determining the relative significance of findings. This chapter of the Audit Procedures Manual has been developed to assist auditors in dealing with the range of differences that may result from the application of these performance-based standards in a broad range of specific situations. 5.2 Measuring Safety Performance The goal of the SMS evaluation is to determine whether the operator is managing the safety-risks of the operation to a level as low as reasonably practical. This is done by assessing the soundness, appropriateness and effectiveness of the operator s safety management activities. Figure 1.1 refers. Figure 5.1 Measures that Determine Optimal Performance January 1, page - 1

35 IS-BAO Audit Procedures Manual Although auditors will explicitly evaluate the SMS, they will, at the same time, gauge and comment on aspects of the operator s safety culture. An operator s safety performance is the sum of the achievements of its formal SMS and the attitudes and values that influence the behaviour of everyone in the organization. The guidelines for evaluating the effectiveness of an SMS, described in section 5.8, assess indicators of safety culture Stage 2 and Stage 3 Audits must contain substantive auditor comments to validate SMS performance. These comments should provide examples of qualitative and/or quantitative methods to verity effectiveness. 5.3 Stages in the Development of a Safety Management System The implementation and operation of an SMS takes time, even for mature aviation organizations. Therefore, the auditor must determine a reasonable level of performance that can be expected when evaluating the SMS. Figure 5.2 illustrates the different stages of an SMS and guidelines are provided in section 5.4 for determining the stage of development of the operator s SMS. Figure Stages of Maturing of an SMS 5.4 Determining the Objectives of the Evaluation As noted earlier, the maturity of a Safety Management System will vary from operator to operator, and within the same operation over time. The auditor must take this into account before conducting an evaluation. It is counter-productive to measure an SMS that is just being implemented in the same way as one that has been in operation for years. What is as low as reasonably practical will differ from one operator to another and evolve with the maturity of the operator s SMS. Yet evaluation processes must be fair, and findings consistently derived. Therefore, an auditor s first step before each evaluation is to determine, with the operator, the objective of that evaluation The stages of SMS development described below are illustrated in Figure Stage One The SMS is Documented, Approved, Resourced and Being Implemented A Stage One Evaluation is conducted: a. For initial applicants for IS-BAO registration; b. For a renewal audit if considerable weaknesses were identified in the previous evaluation; or c. At the request of an operator (e.g. after a change in ownership or management) or at the discretion of the auditor (e.g. after significant organizational change). 2 Adapted from B. Frohlich, Performance Measurement of Safety Management Systems in Safety Performance Measurement, edited by J. van Steen, European Process Centre, Institute of Chemical Engineers, Rugby, UK, January 1, page - 2

36 IS-BAO Audit Procedures Manual The objectives of a Stage One Evaluation are to confirm that: a. The necessary SMS infrastructure is in-place; and b. Past and planned safety management activities are appropriately targeted. Consequently, the evaluation focuses mainly on assessing the soundness and appropriateness of the SMS. Auditors should plan approximately 75% of their activities to be spent measuring the soundness of the SMS, and approximately 25% for assessing appropriateness. Guidance is provided in section Stage Two SMS Functioning, Results Being Measured A Stage Two Evaluation is conducted of an operator that has successfully completed either a Stage One or Stage Two Evaluation. The objectives of a Stage Two Evaluation are to confirm that: a. Safety management activities are appropriately targeted; and b. Safety-risks are being effectively managed. Consequently, a Stage Two renewal audit focuses on assessing the appropriateness and effectiveness of the SMS, and only samples indicators of SMS soundness. Auditors should plan approximately 30% of their activities to measure soundness, and concentrate their attentions on areas that were previously weak or on areas that are particularly important (e.g. safety profile, reporting program, hazard tracking process). They should plan to focus the remaining 70% of their activities on assessing appropriateness and effectiveness. The proportion may favour appropriateness for companies still implementing their SMS (e.g. 50% appropriateness and 20% effectiveness) and effectiveness for more mature SMS (e.g. 20% appropriateness and 50% effectiveness). Guidance is contained in section Stage Three SMS Sustained and Supported by an Ongoing Improvement Process A Stage Three Evaluation is conducted during an evaluation of an operator that has successfully completed at least one renewal audit. The objectives of a Stage Three Evaluation are to confirm that: a. Safety management activities are fully integrated into the operator s business; and b. A positive safety culture is being sustained. Consequently, a Stage Three Evaluation primarily assesses the effectiveness of the SMS, and only samples indicators of SMS soundness and appropriateness. Auditors should plan to spend approximately 75% of their activities measuring effectiveness, and only assess soundness and appropriateness where necessary to understand the causes of potential deficiencies identified in effectiveness. Guidance is included in section Preparations to Evaluate the SMS The auditor must involve the operator when determining the objectives of the evaluation. In the case of an initial audit, the auditor should advise the operator of the objective; provide an outline of the planned activities; and coordinate the on-site visit. In the case of a renewal audit, the auditor should: a. Contact the operator, and arrange to review the previous audit report, and as a minimum, a current copy of the Safety-Risk Profile or equivalent hazard identification and risk analysis system; b. If there is uncertainty, consider contacting the auditor of the previous evaluation; c. Determine an appropriate objective for the evaluation (Stage One, Two or Three); January 1, page - 3

37 IS-BAO Audit Procedures Manual d. Discuss and obtain agreement with the operator; and e. Coordinate the necessary on-site activities. Note: The objective of the SMS evaluation must be determined prior to commencement of the audit. SUMMARY Determine the objectives of the Safety Management System evaluation Stage One Initial applicants Renewal audit (previously weak performance) As requested or required Stage Two Most renewal audits Stage Three Renewal (exceptional performance) Sample indicators of soundness, appropriateness and effectiveness in proportion to the stage of development of the SMS Note: Structure the evaluation to reflect the agreement made with the operator. 5.6 Scoping the Evaluation As noted earlier, safety performance is best if the SMS is sound, safety activities are appropriate, and the actions of executives, managers and staff consistently and effectively reduce safety-risks to a level as low as reasonably practical. For this reason, the SMS evaluation framework is designed to measure indicators of soundness, appropriateness and effectiveness Because operations vary in size and complexity, the aircraft flown, the airspace and airports used, the geographic and meteorological conditions encountered, and the experience and skills of the managers and staff employed; optimal safety management varies from operator to operator. The auditor s evaluation must consistently and fairly measure the safety performance of each operator, despite the differences. 5.7 Measuring Soundness, Appropriateness and Effectiveness Soundness is measured to determine whether the operator has the necessary foundation for proactive safety management. It is also the first general diagnosis of the operator s safety health, and therefore its safety culture. Most of the indicators of soundness can generally be found by reviewing formal documents, files and procedures. The information can be subsequently verified by interviews or observations. When assessing soundness, the auditor seeks to answer the question: Is the infrastructure in place to reduce safety-risks to a level as low as reasonably practical? Appropriateness is measured to determine if safety activity is targeted to consistently manage the operator s safety risks. An SMS may be soundly managed, but it may not be appropriately targeted. This may happen when the operator s SMS has not been well conceived, developed, or implemented, or in situations where there has been significant organizational or operational change. The indicators of appropriateness are generally found by reviewing documents, files, and procedures; by conducting interviews with managers and operational staff; and by observing January 1, page - 4

38 IS-BAO Audit Procedures Manual the operation first-hand. When assessing appropriateness, the auditor seeks to answer the question: Are safety activities purposefully and appropriately targeted to reduce safety risks to a level as low as reasonably practical? Effectiveness is measured to determine if the desired results are being achieved. Circumstances embedded in human, operational, and organizational factors will conspire to impede safety performance, even in companies with a well-managed, well-targeted SMS. The indicators of effectiveness are generally found when conducting interviews, or when observing the operation (after which the observations are validated by follow-up interviews, or by examining files or documents). When assessing effectiveness, the auditor seeks to answer the question: Is everybody effectively engaged in, and capable of, reducing safety risks to a level as low as reasonably practical? 5.8 Conducting the Evaluation The evaluation is conducted by assessing each of the SMS requirements specified in the IS-BAO for soundness, appropriateness, and effectiveness utilizing the Chapter 3 of the Detailed ISBAO Audit Protocols in section 8.2 of this APM. Auditors should customize their evaluation to suit the operation being examined. For example a small operation with one or two people would be expected to have a basic SMS, while the SMS of a large operation would be expected to be much more intricate SMS Stage One Evaluation The aim of the Stage One evaluation is to confirm that the SMS is documented, approved, resourced and being implemented. The auditor examines indicators of the soundness of the SMS foundation, and of the appropriateness of the operator s safety management activities. The SMS is considered sound if the operator has the required components of an SMS in place based on the ICAO framework of 4 components and 12 elements. The auditor determines the level of detail required to evaluate soundness. Section 6.3 contains guidelines for determining the relative importance of the deficiencies that might be identified. Appropriateness is assessed for all 12 elements of the SMS. Although effectiveness is not required to be assessed for a stage 1 evaluation, the auditor may do so if the SMS maturity allows and if it provides value to the operator SMS Stage Two Evaluation The aim of a Stage Two evaluation is to confirm that: the SMS is functioning, results are being measured, safety management activities are appropriately targeted, and safety-risks are being effectively managed. The auditor should concentrate on assessing the appropriateness and effectiveness of the operator s safety management activities utilizing the detailed protocols. Although the auditor is expected to validate soundness for renewal audits, an in-depth evaluation of soundness is only required for those areas that have changed since the previous evaluation, or which warrant reexamination. For example, if there has been a significant change in any of the following areas, a more in-depth examination of the related SMS elements, to include soundness, would be appropriate: January 1, page - 5

39 IS-BAO Audit Procedures Manual Structure of the organization Roles of key managers Staffing levels Aircraft type Routes, or Missions. In cases where there has not been significant change since the previous evaluation, the auditor may wish to examine, in more depth, the components that were either weak in the previous assessment, or which are key to effective safety performance SMS Stage Three Evaluation The aim of a Stage Three evaluation is to confirm that: SMS is sustained and supported by an on-going improvement process, A positive safety culture is sustained, and Safety management activities are fully integrated into the operator s business. A Stage Three operator is expected to have the following characteristics: a. Widespread employee commitment to and involvement with improving safety performance, including visible leadership by top management; b. A sustained desire for good safety performance as a goal in itself, not just as an issue of regulatory compliance; c. Consistently, with interest, seeks the causes of deficiencies, rather than the allocation of blame; d. Effective communication of safety information within and outside the organization, including safety performance trends; e. A deeply held commitment to the benefits of continuous evaluation and improvement; f. Managerial awareness of the issues that adversely impact the operator s safety culture; g. Primary goals that include safety, that do not focus only on cost or financial targets, and that, with other goals, are actively and proactively measured; h. Adequate allocation of financial and other resources for safety management; i. Initiatives to learn from the safety performance of external organizations; and j. Safety performance measures that include those of effectiveness of activities of processes, rather than just the results of these activities. The on-going improvement process should include regular evaluation of safety management activities in relation to stated SMS objectives plus the evaluation of the effectiveness of risk controls. It also should include a formal process to identify the causes of sub-standard performance of the SMS, determine the implications of sub-standard performance of the SMS in operations, and eliminate or mitigate such causes. For Stage Three evaluations, auditors are shallencouraged to conduct an in-depth evaluation of effectiveness utilizing the detailed protocols along with the following performance indicator examples. The possible methods by which the information can be obtained are provided in brackets with the following abbreviations employed: doc. manuals, formal documents, procedures, files, etc; int. interviews with managers, staff or stakeholders; obs. on-site or inflight observations; exp. the experience of the auditor. January 1, page - 6

40 IS-BAO Audit Procedures Manual 1. Safety Policy and Objectives Stage 3 SMS Performance Indicators a. Are comments on the efficacy of the safety policy and related policies actively solicited from all stakeholders, managers and staff? (doc., int., obs., exp.) b. Are the policy/policies periodically and thoroughly reviewed and when appropriate, revised? (doc., int., obs., exp.) c. Is top management fully aware of the human and organizational factors that endanger the operation and do they continuously make decisions mindful of those factors? (doc., int., obs., exp.) d. Does management drive the SMS process and vigorously seek excellence? (int.) e. Are responsibilities for safety and contributing to the SMS effectiveness fully imbedded throughout all levels of the organization? (doc., int.) f. Do the appraisals of personnel at all levels include related SMS targets (activities and results) and is exemplary behaviour rewarded? (doc., int.) g. Are safety management activities being continually monitored, evaluated and improved when opportunities are identified? (doc., int.) h. Have managers and staff clearly demonstrated that they are capable of performing their roles to proactively manage safety? (obs., exp.) i. Is there evidence of the documentation evolving as the other elements of the SMS matured? (doc., int.) 2. Safety Risk Management a. Are the policies and procedures to mitigate safety-risks periodically reviewed for applicability and effectiveness and modified when found wanting. (int., obs., doc.) b. Does the mitigation account for predictable human, operational and organizational limitations? (int., obs., exp.) c. Are risk assessment and management fully integrated into all operations and maintenance activities? (int., obs., doc.) d. Is the information indicating the need to change respected and valued? (doc., int., obs.) e. Are long and short-term plans to improve safety performance driven by multi-level analyses of the safety-risk management database and integrated with business planning. (int., obs., doc.) f. Is there solid evidence that managers and staff are fully engage in and committed to, enhancing safety performance? (int., obs., doc., exp.) g. Is their consistent feedback that has been effective in ensuring the participation of managers and employees? (doc. int., obs.) h. Is the information used not only to enhance day-to-day safety, but also to improve longterm safety management? (doc., int.) i. Are the results of safety management activities evident in the operations manual and its amendments? (obs., doc., int.) j. Are there effective processes to ensure that any amendments to the Operations Manual that result from safety management activities are made in a timely manner? (obs., doc., int.) k. Are employees highly motivated to report occurrences and does the environment exist where they are fully prepared to make such reports when it involves admitting that they made mistakes? (int.) January 1, page - 7

41 IS-BAO Audit Procedures Manual l. Are occurrences analyzed and the lessons learned integrated into the appropriate programs and procedures? (doc., int.) 3. Safety Assurance a. Is the effectiveness of safety management activities periodically evaluated and modified to improve the organization s safety performance. (doc., int.) b. Is there effective use of tools such as internal safety surveys? (doc., int.) c. Is there an effective ongoing program to evaluate safety performance? (doc., int., obs., exp.) d. Is the information obtained used to enhance appropriate programs and procedures? (doc., int., obs., exp.) 4. Safety Promotion 5.9 Reserved a. Is there an established effective process to ensure that the results of SMS activities are fully integrated into the appropriate training programs? (int., doc.) b. Are employees highly motivated to report apparent safety deficiencies and fully confident that there will not be retribution? (int.) c. Are there well established processes to expeditiously evaluate safety information and integrate it into safety management activities? (doc., int. obs., exp) January 1, page - 8

42 IS-BAO Audit Procedures Manual 6.0 PREPARING AUDIT FINDINGS 6.1 Introduction Findings are significant issues which require urgent action by the operator's to correct. Findings may relate to one or more non-conformities but should be sufficiently serious to require urgent action. As previously noted, a minor non-conformity would be one that does not constitute a significant risk to safety. It may be an isolated instance of non-conformity, a lack, or error in documentation of a process or procedure that was correctly executed, or a similar instance. A major non-conformity is when there is a pattern of instances of minor non-conformity or a non-conformity that in itself constitutes a significant risk to the safety of the operation. Non-Conformity (NC) Condition Consequence Minor Single NC Low risk to safety Finding or ACTION REQUIRED statement in protocols Multiple related NCs Increased risk to Finding safety Major Significant risk to safety Finding 6.2 Making Findings Figure 6.1. Non-Conformity/Finding Matrix Findings must be significant issues that need the operator s attention. They must relate to standards specified in the IS-BAO, must be substantiated by factual information and in most cases should be validated by objective evidence of different forms or sources of information (for instance, corroborating information from: different documents; documents and observations; or documents, interviews and observations). Findings of an IS-BAO audit must address the degree to which safety risks are being managed to a level as low as reasonably practical (i.e. the objective of all evaluations of SMS). As such, the findings should always relate to management of the safety-risks, not to the operational indicators of deficient safety management. Consequently, the findings are developed after the components of the SMS and related elements have been assessed (as described in chapter 5). Auditors must take the information from their assessments, and use logic and experience to determine the cause(s) of the problem. It is these cause(s) that make up the findings. It is important that the auditor address the disease not the symptoms of the disease Several examples are provided below: a. A Safety-Risk Profile that failed to identify a number of key hazards might lead to observations by the auditor of inappropriate operational procedures and training. By seeking the cause(s), the auditor s finding might target the procedures used to conduct the Hazard Analysis, rather than the profile, the operational procedures or the training. b. An auditor might identify that plans to revise self-dispatch procedures had been slipping for some time. Further inquiry or other observations might lead to a finding regarding an absence of resources to modify the procedures, and possibly an absence of managerial oversight of this slippage. c. An auditor might observe a number of maintenance related issues. By reviewing the Maintenance Control System and interviewing personnel, deficiencies in the Maintenance Control System, or related procedures, training or managerial oversight might be identified as the causes of these issues. January 1, page - 9

43 IS-BAO Audit Procedures Manual It is therefore important that indications of non-conformity and the related objective evidence collected in the course of the audit be carefully analysed to determine the cause(s) of the problem. When the cause has been identified it should be presented as a Finding with the non-conformities and other objective evidence, used as supporting material. If it is determined that one or more minor non-conformities are simply slips, lapses or omissions that can easily be corrected and there is no significant underlying problem, rather than making a Finding the non-conformity should be noted in the Analysis of Non-conformities section of the Audit Protocol along with the comment, ACTION REQUIRED. Full Conformity should not be selected on the Audit Report Form if there are any instances of non-conformities, regardless of how they are documented, i.e. on the Finding forms or in Analysis of Non-conformities block A major non-conformity, by definition, will be sufficiently critical that a Finding must be made. However, the same degree of analysis must be undertaken to identify the cause(s) of the problem Findings are to be recorded on the Audit Finding Form that is contained in section It should be noted that a finding cannot be based on non-conformity with a Recommended Practice contained in the IS-BAO, but that fact can be used in conjunction with other nonconformity information to substantiate a conclusion. 6.3 Determining Criticality Figure 6.2. Process for Documenting Non-Conformities Auditors must be able to consistently determine the criticality of their findings so operators can plan appropriate remedial action. Criticality is determined by judging the scope of the finding (i.e. by determining whether the consequence is far reaching), and by assessing its impact on the safety of the operation. The measure differs from an Initial and a Renewal audit IS-BAO Audit with a Stage One SMS Evaluation An IS-BAO audit that is conducted of an operator where their SMS is at a first stage of development and implementation will focus on the degree to which the SMS and other IS-BAO January 1, page - 10