Implementation of OAG recommendations. Status Report

Transcription

1 Implementation of OAG recommendations Status Report O r g a n i s a t i o n Civil Aviation Authority of New Zealand V e r s i o n Final (DW ) A u t h o r COO D a t e o f t h i s d o c u m e n t 1 October 2010

2 CONTENTS Contents 2 Executive Summary 3 Performance improvement at the CAA 5 BACKGROUND 5 ACTION SUBSEQUENT TO THE 2005 REPORT 5 CURRENT CONTEXT 6 Organisational Development 6 Considerations in the implementation of the OAG recommendations 7 What is different this time? 8 OAG Recomendations: Summary status of corrective actions 10 OAG Recomendations: Detailed status of corrective actions 14 OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No OAG Recommendation No Output quality measures 37 Introduction 37 Methodology 37 Analysis 37 Results 37 2 P a g e

3 EXECUTIVE SUMMARY This report is the first in a series detailing the CAA s progress implementing the recommendations made by the Office of the Auditor General (OAG) in the report of its follow-up audit of the CAA s certification and surveillance functions released on 29 June In that report the Controller and Auditor General concluded that the CAA s response to the recommendations made in a 2005 OAG report regarding the performance of those functions had been inadequate. The report went on to identify factors the OAG considered had contributed to the CAA s poor response and made a number of recommendations that should enable the CAA to make necessary improvements to strengthen its certification and surveillance work. The Board and senior management of the CAA have acknowledged the CAA s inadequate response to the OAG s 2005 report and have reviewed that performance in order to learn from the mistakes made. While the Board and management of the time were committed to change, their efforts failed because they: underestimated the magnitude of the change process required and as a consequence did not treat the required work as significant organisational change; focussed on process change without adequately addressing people issues; and did not introduce any measurement of the effectiveness of the change it did undertake. The organisational arrangements introduced during a restructure in 2000 were also a contributor. Responding to these lessons, the CAA is approaching the implementation of the OAG s 2010 recommendations in the context of a wider and longer-term organisational change effort. While this first status report focuses primarily on the short term high priority fixes associated with the OAG s recommendations, the wider context is important and will assume more prominence as time goes on. Work on the wider organisational development and culture issues is in the initial planning stage and, where possible, projects implementing OAG recommendations will be used to model the values and behaviours envisaged in the to be state of the CAA 2-3 years hence. Additional action has been taken to ensure that there will be no repetition of the inadequate response in This is despite the very tight financial constraints within which the CAA is working and the conflicting high priority demands on its current resource. Significant examples of this additional action include: the internal audit function has been strengthened and outsourced; a Chief Operating Officer position has been created to drive change and consistency in the organisation; and improved monitoring and measurement of certification and surveillance output quality has been introduced. The three largest tranches of work flowing from the OAG s recommendations constitute work related to the CAA s: certification function; surveillance function; and safety data analysis capability. A project to improve the certification function is currently in progress and slightly ahead of schedule. Current indications are that it will prove to be very effective and that it will be complete in October. Work on the surveillance function has been brought forward from a scheduled start in November and is currently in the initial planning phase. While elements of the work to improve the CAA s safety data analysis capability have been deferred pending the outcome of the funding review, some parts of the recommendation are under action. The current Board and management of the CAA accept the recommendations made in the OAG s 2010 report and are committed to implementing them as a matter of urgency. The arrangements already in place and under development will provide a high level of assurance 3 P a g e

4 that the recommendations will be implemented effectively and sustainably. The scheduling of implementation reflects the outcome of careful prioritisation and consideration of the risks associated with conflicting demands for finite funding. In this regard, the CAA is closely monitoring its financial performance and if actual results exceed budget, additional funding will be applied to the implementation of OAG recommendations to the extent practicable. The CAA considers that the current implementation programme is the fastest it can achieve in the current circumstances and that the scheduling of the programme does not create any unacceptable risks. 4 P a g e

5 PERFORMANCE IMPROVEMENT AT THE CAA Background 1. Since 1997, the Office of the Controller and Auditor General (OAG) has carried out four audits of the CAA s certification and surveillance functions. Audits conducted in 1997 and 2000 identified deficiencies in the CAA s exercise of its surveillance function, particularly in relation to the identification and targeting of high risk aircraft operators. 2. A further audit, conducted in 2005, noted that little action had been taken to address the recommendations in the 1997 and 2000 reports. The 2005 report expressed particular concerns with the application of certification processes within the General Aviation (GA) Group and with the CAA s exercise of its surveillance function. It noted, however, that the CAA had commenced a significant review of the surveillance function, which envisaged the integration of the certification, surveillance and risk assessment projects (then underway) into a single computer-based system. 3. During the period June to August 2009 the OAG conducted a follow-up audit of the CAA s performance of its surveillance and certification functions to establish whether the CAA had addressed the 10 recommendations made in the 2005 report. In the report of the 2009 audit released on 29 June 2010, the Controller and Auditor General concluded that the CAA s response to the OAG s 2005 recommendations had been inadequate 1. The report went on to identify the factors the OAG considered had contributed to the CAA s poor response and made 14 recommendations 2 that the OAG considered should enable the CAA to make necessary improvements to strengthen its certification and surveillance work. 4. In response to the OAG s audit, the Chairman of the Authority wrote to the Minister of Transport proposing an arrangement in which the Authority would monitor implementation of the OAG s recommendations on a quarterly basis and would subsequently report the results of that monitoring to the Minister. The Minister accepted the Chairman s proposal and this document constitutes the first of the resulting progress reports. The status outlined is with effect 6 September Action subsequent to the 2005 report 5. In its recent report the OAG identified the following factors as contributing to the CAA s inadequate response to the OAG s 2005 report: governance of, and accountability for, the CAA s certification and surveillance functions are ineffective, the strength of the CAA s regulatory focus is unclear, and there is insufficient guidance to ensure that regulatory responses are appropriately and consistently applied, the CAA s management practices are not focused on improving staff performance, and it has not been receptive to change, the CAA s management oversight of implementing and using the new certification and surveillance processes is inadequate, and the CAA has not given enough attention to improving its organisational proficiency in auditing. 6. Analysis of the improvements sought by the OAG in its 2005 report shows that implementation of the improvements would have required significant organisational change on 1 The OAG considered that of the 10 recommendations made only one had been fully addressed, eight had been partially addressed, and one had not been addressed directed to the CAA and 1 to the Ministry of Transport. 5 P a g e

6 the part of the CAA. Given this fact, it is apparent that much of the improvement effort failed because the CAA: underestimated the magnitude of the change process required and as a consequence did not treat the required work as significant organisational change; focussed on process change without adequately addressing people issues; and did not introduce any measurement of the effectiveness of the change it did undertake. 7. The organisational structure was also a contributor. In 1999 at the request of the then Authority, Human Synergistics NZ Ltd. made recommendations to restructure the CAA as part of efforts to improve client relationships and rectify a general lack of client orientation. The Authority accepted the recommendations of the report and in 2000 the CAA moved from a structure reflecting function, e.g. certification and surveillance, to one organised around client groupings, e.g. General Aviation and Airlines. While Human Synergistics Ltd. pointed to certain advantages of the new structure, it also identified a loss of functional expertise and consistency (for outputs and activities) as risks of the new structure. This was, in large part, due to the lack of clear ownership of any function performed by two or more of the CAA s new client oriented groups. It proposed the creation of a Professional Standards Unit to mitigate the risk by monitoring standards and enforcing consistency. With the benefit of hindsight it can be seen that the risk mitigation was ineffective. Knowledge of these factors enables action to be taken to compensate for the disadvantages of the current organisational arrangements while still gaining the benefit of its advantages. 8. Given the benefit of the OAG s most recent recommendations and the lessons of history the CAA is now better equipped than ever to implement effective corrective action. Current Context Organisational Development 9. The primary focus of this report is on the CAA s implementation of the recommendations made by the OAG relating to deficiencies in the performance of the CAA s core certification and surveillance functions. Nevertheless, if those efforts are to have any chance of success, they must be treated as part of a wider organisational change rather than a series of individual process changes. Thus, while urgent work is being done to improve and standardise deficient processes highlighted by the OAG, that work must be seen in the context of the longer term organisational change and development that is necessary to sustain and enhance work done in the short term. 10. In May of this year, the position of Chief Operating Officer (COO) was created within the CAA and the Authority has delegated responsibility, accountability and authority to that position to: lead change in the organisation by improvement to processes and procedures; promote and implement a performance culture in the CAA; improve the organisation s performance in the delivery of its functions and the timely introduction of changes; and improve the efficiency and consistency of CAA processes. 11. While efforts to date have primarily focussed on the means of addressing the OAG recommendations, initial planning is now underway on the means of addressing organisational culture and development issues. The planning of this organisational development work will be integrated with the current business planning cycle. Thus, essential work such as defining the to be nature of the organisation 2-3 years hence will be integrated with routine strategic planning work currently scheduled to start in mid October. 12. It is worth noting that while organisational development planning is in its early stages, it has been decided to utilise the Surveillance Improvement Project, which entered its planning phase on 6 September 2010, as a medium to model for staff the values, behaviours and 6 P a g e

7 methods that will be promoted in future. Planning for and creating short-term wins is an essential element of maintaining momentum in an organisational change effort and the Surveillance Improvement Project offers a means of doing this. 13. The next report in this series will provide more information on the CAA s medium and longer term organisational development objectives and the means of achieving them. Considerations in the implementation of the OAG recommendations 14. The three largest tranches of work flowing from the OAG s recommendations constitute work related to the CAA s: certification function; surveillance function; and safety data analysis capability. 15. As will be seen from an examination of the corrective actions identified for each OAG recommendation listed later in this report the CAA has prioritised these bodies of work in the order shown above. Some of the more important factors taken into account in developing this prioritisation include the: safety significance of the particular deficiencies observed by the OAG; magnitude and complexity of the work required in each area; related resource requirements (human and financial); current financial position of the CAA (budgeted $2.6M deficit for the 2010/11 FY); relationships and dependencies between different bodies of work; links to the current review of CAA funding; and competing resource demands and the priority of that work (in particular the ability to re-prioritise the allocation of existing resource). 16. In some cases, the outcome of this prioritisation exercise indicated that it was not practicable to commence full implementation of a particular OAG recommendation immediately. In those cases consideration has been given to: some form of alternative interim action; whether it is possible to break the recommendation down to constituent parts and action parts of the recommendation; and whether it is possible to begin establishing groundwork or infrastructure that will support the subsequent implementation of the recommendation at a later time. 17. The handling of Recommendation 8 3 is an example of this approach. If the recommendation is broken down, it is seen to have two elements: improve the integrity and reliability of safety data in its Management Information System ; and improve the analysis of this data. Consideration of the effort involved in improving the CAA s safety analysis capability showed that it would involve considerable expense which could not be funded in current circumstances. Therefore, this requirement was deferred and provision of appropriate capability will be sought in the CAA s funding review. It was, however, considered possible to improve the integrity and reliability of the safety data held and considerable work has already been undertaken, or is 3 We recommend that the Civil Aviation Authority give priority to completing the project to improve the integrity and reliability of safety data in its Management Information System, and improve the analysis of this data so that it can be used to better inform regulatory decision-making. 7 P a g e

8 programmed, to first establish and then rectify the shortcomings in that data. Put simplistically, the approach taken has been do as much improvement work possible within applicable constraints to ensure that if, and when, the funding review provides the resource to improve the analysis capability, it will have good data to work with. 18. A similar approach is taken with a number of the other recommendations which cannot be fully implemented immediately. This has been done in order to obtain early benefit from the incremental improvement provided and also to build momentum for change. 19. The scheduling of the work on the certification and surveillance functions deserves brief mention. Both functions have considerable safety significance and it would be difficult to assign priority to one or the other on the basis of safety. The rectification of the shortcomings of the certification function is, however, a considerably less complex and resource intensive undertaking than the improvement of the surveillance function. Thus, the Certification Improvement Project started first and is on schedule for completion in October. Work on the surveillance function has been brought forward from a planned start date in November 2010 and is in the initial planning phase. Existing internal resource was seconded to the project with effect from 6 September 2010 and at this stage it is anticipated that the project will run for months. What is different this time? 20. As noted previously, following the release of the 2005 OAG report, the CAA created an action plan to implement the recommendations of the OAG report. The Certification, Surveillance, and Risk Assessment and Intervention projects mentioned earlier were created and the projects ran their course with the CAA regularly reporting progress and ultimately completion of the required work. Unfortunately, in 2010 the OAG found that two of the three projects had not delivered the improvements sought in its 2005 report. 21. It would be reasonable for the CAA s monitoring agencies and others to seek assurance that the current situation is different to that in 2005 and ask why history will not simply repeat itself. Fortunately, there are significant differences between now and then. Some factors that distinguish the current situation from that in 2005 are: greater senior management and Authority acceptance of the urgent need for change; the focus of the current Authority on the need to implement the OAG recommendations and, most significantly, the need for assurance of the effectiveness of any corrective actions; the internal audit function has been strengthened and outsourced; the creation of the COO role which provides a single owner and accountability for the operational processes utilised by the CAA s operational groups and the organisation s change efforts; acknowledgement that organisational, rather than simply process change is required; recognition of the need to measure the effectiveness of change efforts; the recent establishment of a programme and project management capability; acceptance of the need to adequately resource required change and commitment to provide that resource; and a significantly different Authority and management team from those in place in In combination, these factors make the current context for implementation of the recent OAG recommendations significantly different to that in They provide a high degree of assurance that effective change will take place and will be sustained. It must be noted, however, that the pace of that change will be influenced by the level of funding that can be applied to support it. In this regard, the CAA is closely monitoring its financial performance and 8 P a g e

9 if actual results exceed budget, additional funding will be applied to the implementation of OAG recommendations to the extent practicable. 9 P a g e

10 OAG RECOMENDATIONS: SUMMARY STATUS OF CORRECTIVE ACTIONS Table 1 provides a summary report on the status of work underway and planned to implement the recommendations made by the OAG. In many cases, a number of corrective action projects will be involved in full implementation of a recommendation. The table provides a high level summary of the status of work against each recommendation, relates each recommendation to the CAA s strategic priorities expressed in the 2010/13 Statement of Intent, provides an indication of the likely status in 12 months time, and where applicable, provides an indication of the scale of funding needed to complete the work. More detailed information about the status of work to implement the recommendations and the results of the internal audit of work done to date are provided in Table 2. Status legend C P Complete In progress but too early in the project to fully address the underlying problem and/or some project risk remains. No significant progress. Constrained by some factor e.g. lack of human resource or funding. P IA Satisfactory progress being made and nearing completion with low project risk. Interim action being taken. The underlying problem is not being fully addressed for a specific reason although interim action is being taken which has a mitigating effect and/or addresses elements of the problem. OAG recommendation No.1. We recommend that the Civil Aviation Authority put in place measures to better assess the effectiveness of its certification and surveillance functions and use these measures to report and account to the Board for its performance in achieving its outcomes. Table 1: Summary status of corrective actions Planned completion Dec 2011 subject to funding no later than July 2011 Current Status IA Strategic intent Funding requirements Projected status in 12 mths Effective entry control and monitoring. Effective analysis and intervention. Approximately $50-$100k not currently budgeted or available from internal reprioritisation. P No.2. We recommend that the Board extend its internal audit of the Civil Aviation Authority to include assurance over the executive management team s assessment of how well the Airlines Group s and General Aviation Group s certification and surveillance are contributing to its strategic priorities and achieving its overall goals and objectives. Dec 2011 subject to funding no later than July 2011 IA Effective entry control and monitoring. Effective analysis and intervention. Approximately $50-$100k not currently budgeted or available from internal reprioritisation (same funding as 1 above). P 10 P a g e

11 OAG recommendation No.3. We recommend that the Board s Audit and Risk Management Committee take a more active role to ensure that the Airlines Group and the General Aviation Group actually address the internal audit findings. Planned completion Process in place ongoing monitoring. Status Strategic intent Funding requirements Projected status in 12 mths C Effective entry control and monitoring. Effective analysis and intervention. N/A C No.4. Not applicable. Recommendation made to the Ministry of Transport. No.5. We recommend that the Civil Aviation Authority prepare and implement better measures of the strength and effectiveness of its regulation of the civil aviation sector, including measures to assess the relative effectiveness of advisory and enforcement actions. Dec 2011 subject to funding no later than July 2011 Effective entry control and monitoring. Effective analysis and intervention. Approximately $50-$100k not currently budgeted or available from internal reprioritisation (same funding as 1 above). P No.6. We recommend that the Civil Aviation Authority clarify how its regulatory focus is to be applied in practice through certification, surveillance, and other regulatory action by providing more detailed guidance to staff about what circumstances constitute a significant risk to public safety, and what action they should take when these safety risks are identified. Dec 2010 P Effective entry control and monitoring. Effective analysis and intervention. Approximately $40k budgeted in the 2010/11 FY. C No.7. We recommend that the Civil Aviation Authority give priority to completing the project to improve the integrity and reliability of safety data in its Management Information System, and improve the analysis of this data so that it can be used to better inform regulatory decisionmaking. Feb 2012 subject to availability of funding with effect Jul IA Effective analysis and intervention. Approximately $450k PA. Capability sought in the current review of CAA funding. P 11 P a g e

12 OAG recommendation No.8. We recommend that the Civil Aviation Authority assess, and, where necessary, provide training to improve its managers capability to effectively manage and lead staff. This includes improving the staff performance assessment process in the General Aviation Group. Planned completion Dec 2012 Status Strategic intent Funding requirements Projected status in 12 mths IA Capable and efficient business processes, information systems and people. N//A P No.9. We recommend that the Civil Aviation Authority give priority to completing the project to review and improve the surveillance process and tools, and ensure that all managers and auditors are using the new certification and surveillance processes. Sep 2011 P Effective entry control and monitoring. Approximately $180k budgeted in the 2010/11 FY. P No.10. We recommend that the Civil Aviation Authority introduce more robust quality assurance of certification and surveillance work, including input into planning for certification and surveillance, reviewing the results, and moderating auditors findings. Sep 2011 P Effective entry control and monitoring. Approximately $180k budgeted in the 2010/11 FY (same funding as 9 above). P No.11. We recommend that the Civil Aviation Authority provide better guidance to its auditors on the level of documentation that needs to be retained as evidence of the certification and surveillance work that has been carried out, and reinforce the importance of clearly documenting the basis for decisions that involve serious consideration of evidence for a judgement to be made. Sep 2011 P Effective entry control and monitoring. Approximately $180k budgeted in the 2010/11 FY (same funding as 9 above). P No.12. We recommend that the Civil Aviation Authority provide better guidance to its auditors on how to apply the fit and proper person criteria when carrying out assessments of senior persons. Oct 2010 P Effective entry control and monitoring. N/A C 12 P a g e

13 OAG recommendation Planned completion Status Strategic intent Funding requirements Projected status in 12 mths No.13. We recommend that the Civil Aviation Authority give priority to providing training in risk-based audit methodologies for its auditors, to ensure that they have the appropriate skills to carry out effective certification and risk-based surveillance. Sep 2011 P Effective entry control and monitoring. Approximately $180k budgeted in the 2010/11 FY (same funding as 9 above). P Plus approximately $150k pa from 1 Jul No.14. We recommend that the Civil Aviation Authority provide detailed guidance to its auditors on risk-based auditing, including how information about risk can be used to tailor audits at the planning stage, how this information should be documented, how systemsbased auditing should be applied, and how risk influences the size of samples checked during audits. Sep 2011 P Effective entry control and monitoring. Approximately $180k budgeted in the 2010/11 FY (same funding as 9 above). Plus approximately $150k pa from 1 Jul 2011 (same as 13 above). P 13 P a g e

14 OAG RECOMENDATIONS: DETAILED STATUS OF CORRECTIVE ACTIONS Table 2 provides a detailed description of the work identified by the CAA as necessary to implement the recommendations of the OAG and the timing of the work. In many cases, a number of corrective actions will contribute to implementing a single recommendation. The table details, with respect to each OAG recommendation, what the problem is, why it is a problem, and how it will be addressed. The CAA is committed to the scheduled start and end dates for each body of work (except where a dependency on the funding review is identified) with detailed planning for each action to be undertaken at or before its scheduled start date. Table 2: Detailed status of corrective actions OAG Recommendation No.1. We recommend that the Civil Aviation Authority put in place measures to better assess the effectiveness of its certification and surveillance functions and use these measures to report and account to the Board for its performance in achieving its outcomes. Underlying problem leading to the recommendation: The CAA does not have any robust measures for the effectiveness of its certification and surveillance functions. This means: CAA is unable to report deliverables in terms of its contribution toward strategic goals and desired safety and security outcomes; the Authority is limited in its ability to monitor the performance of the organisation; and the organisation is limited in its ability to prioritise the allocation of resource to different types of intervention. Intervention/Corrective actions Internal audit assessment and comment CAA Remarks 1A. Introduce improved quality measures for certification and surveillance (Output Class 2 performance measures). Assess the quality of surveillance and certification outputs and correct any deficiencies found. Complete. (Ongoing reporting on a quarterly basis has been established.) output measures have been refined and focussed on better indicators of quality, quantity and timeliness; an internal quality review system has been put in place to monitor and report on output; a feedback loop is established as an input to staff performance appraisal and process monitoring; and the first quarterly report has Assessment: The project to implement quality monitoring of certification and surveillance activities has been completed and internal audit have reviewed the moderation process in action. The corrective action is designed to improve the consistency of application of certification and surveillance outputs which is a subset of the problem. It does not, however, address the primary recommendation to measure the effectiveness of the certification and surveillance functions. No assessment of the work to develop measures of the 1A. The action taken makes the reasonable assumption that improved quality and consistency of outputs will more effectively contribute to the CAA s desired safety outcomes. While the action does not completely address the problem identified it has been taken as an interim and supplementary measure, pending the CAA s development of robust measures for the effectiveness of its certification and surveillance functions in contributing toward its strategic goals and desired safety and security outcomes. 14 P a g e

15 1B Develop measures of the effectiveness of interventions. Provide a measure of the effectiveness of the CAA s regulatory interventions. been provided to the A&RM Committee at its 24 September 2010 meeting. Output Class 2 (surveillance and certification) quality measures specified in the SOI will be met. Currently scheduled to start on 1/07/11 and finish on 23/12/11. It is estimated that the deployment and validation of method would cost approximately $50-$100k. This work is not funded in the current budget and at the time of writing there does not appear to be any reasonable way of reallocating existing funding to do the work. The scheduling shown above assumes it will not proceed in this FY. Capability for the initial deployment and further development will also be sought as part of the current review of CAA funding. The CAA will also take up a recent offer by the MoT to assist in the development of desired measures. Benefits not yet planned or assessed. effectiveness of interventions possible. Comment: The development of effectiveness measures is key to resolving this recommendation. Development of such measures that will be practical and add value to CAA will be a significant challenge based on the experience overseas. This work is yet to be commenced due to funding issues. The Output Class 2 work is positive but a subset only. 1B. The CAA acknowledges the desirability of developing and utilising measures of the effectiveness of interventions in achieving outcomes but notes that this is a highly complex issue and, in the absence of clear measurements, aviation safety regulators world-wide are forced to make value judgements about the way in which they balance the application of the various regulatory interventions required by their legislation. The complexity reflects the fact that aviation safety regulators cannot control safety outcomes they are merely one of a number of influences on them. The CAA currently has an untested method that may provide insight into how effectively different types of regulatory intervention contribute to achievement of the desired safety outcomes for the sector. The validation of this method is not funded in the current budget and no reasonable reprioritisation of expenditure is currently available. The scheduling shown assumes the work will not proceed in this FY. While being at the leading edge of such development work carries some risk, its desirability is acknowledged. Actual financial position versus budget will be monitored over time and if finance becomes available the work will proceed. The MoT has also offered to assist in the development of measures and the CAA will take it up on this offer. 15 P a g e

16 OAG Recommendation No.2. We recommend that the Board extend its internal audit of the Civil Aviation Authority to include assurance over the executive management team s assessment of how well the Airlines Group s and General Aviation Group s certification and surveillance are contributing to its strategic priorities and achieving its overall goals and objectives. Underlying problem leading to the recommendation: In the past, internal audit has focussed more on procedural compliance than effectiveness. Compliance with policy and procedures does not necessarily equate to the achievement of goals and objectives or even the production of quality outputs. Intervention/Corrective actions Internal audit assessment and comment CAA Remarks 2A. Assessment: 2A. 2B Amend Audit and Risk Management (A&RM) Committee charter. Adjust the focus of internal audit from solely compliance to include effectiveness. Design and implement Internal Audit Monitoring. Provide the A&RM Committee with a quarterly report on the quality of outputs. Draft Charter presented to the 3 August A&RM Committee meeting and adopted by the committee to be approved by the Authority. Clarity of expectation on oversight role. Arrangements in place through internal audit plan for 2010/11. Clear line of independent quarterly reporting to A&RM Committee on management s assessment. The A&RM Committee Charter has been amended. It addresses the governance oversight element of the recommendation. The structure is in place to enable the recommendation to be addressed. The assurance work still needs to be undertaken. See next corrective action. The internal audit programme will report on Output Class 2 quality monitoring process which addresses consistency of output from certification and surveillance. Work is still required to look at effectiveness. First report will go to September 2010 A&RM Committee. Internal audit report will not cover contribution to strategic priorities and goals. It will report on consistency of application of process across certification and surveillance. Work is still required by management to look at effectiveness. Comment: The Charter of the A&RM Committee was amended at the Authority s 3 Aug 2010 meeting to specifically include a requirement that it monitor the executive management team's assessment of how well the CAA s certification and surveillance functions are contributing to its strategic priorities and achieving its overall goals and objectives. 2B. As an interim measure, the assumption is made that quality outputs will contribute to achievement of desired outcomes. The quarterly monitoring and reporting on progress against the CAA performance improvement programme milestones will therefore include the results of Output Class 2 quality monitoring. In addition, the monitoring is being done to ensure certification and surveillance are conducted as intended, which will lay the foundation for subsequent assessment of their relative contribution to CAA s goals. See also the comments against recommendation 1. The A&RM Committee is set to oversee the assurance provided by internal audit regarding effectiveness. Assurance on 16 P a g e

17 Airlines and GA s certification and surveillance contributing to strategic priorities and overall goals will be considered once effectiveness measures are determined. In the interim, assurance will be provided over improving the quality of certification and surveillance outputs. 17 P a g e

18 OAG Recommendation No.3. We recommend that the Board s Audit and Risk Management Committee take a more active role to ensure that the Airlines Group and the General Aviation Group actually address the internal audit findings. Underlying problem leading to the recommendation: In the past the Audit and Risk Management Committee tended to accept at face value corrective actions proposed to close internal audit findings. Proposed corrective actions may not effectively rectify the cause of the finding and prevent recurrence. It is important for internal audit to assess the effectiveness of the action before closing the finding or to recheck its effectiveness at some later time and for the A&RM Committee to require this level of assurance. Intervention/Corrective actions Internal audit assessment and comment CAA Remarks Internal audit follow up process in place to give independent comment on progress to A&RM Committee. Internal audit monitoring of OAG and other audit findings reported quarterly to A&RM Committee. Ongoing action by internal audit provider and A&RM Committee. Agreed as part of the internal audit plan for 2010/11. Independent review of progress to ensure A&RM Committee is able to ensure effective actions being undertaken. Assessment: The monitoring process is in place to report on progress. This will ensure any lack of progress/appropriate focus is identified. A follow up process has been agreed in the internal audit plan. Reporting will commence for the December A&RMC meeting. The A&RM Committee is better placed to have active monitoring role. Internal audit findings will be addressed if resources are available to management; if not then A&RM Committee will know. The internal audit provider and the A&RM Committee will in future more closely monitor the effectiveness of the corrective actions taken in response to internal audit findings raised against the CAA s operational groups. Comment: Monitoring in place to enable an active A&RM Committee role. OAG Recommendation No.4. Not applicable. Recommendation made to the Ministry of Transport. 18 P a g e

19 OAG Recommendation No.5. We recommend that the Civil Aviation Authority prepare and implement better measures of the strength and effectiveness of its regulation of the civil aviation sector, including measures to assess the relative effectiveness of advisory and enforcement actions. Underlying problem leading to the recommendation: The CAA does not have any robust measures for the effectiveness of its different regulatory functions in contributing toward its strategic goals and desired safety and security outcomes. Without robust measures of the effectiveness of different regulatory interventions, it is difficult for the CAA to determine whether it is using the optimal mix of interventions in order to achieve desired goals. Intervention/Corrective actions Internal audit assessment and comment CAA Remarks Develop measures of the effectiveness of interventions. Provide a measure of the effectiveness of the CAA s regulatory interventions. Scheduled to start on 1/07/11 and finish on 23/12/11. N/A. Assessment: Not assessed. Comment: No auditable progress to date. The CAA acknowledges the desirability of developing and utilising measures of the effectiveness of interventions in achieving outcomes but notes that this is a highly complex issue and, in the absence of clear measurements, aviation safety regulators world-wide are forced to make value judgements about the way in which they balance the application of the various regulatory interventions required by their legislation. The complexity reflects the fact that aviation safety regulators cannot control safety outcomes they are merely one of a number of influences on them. The CAA currently has an untested method that may provide insight into how effectively different types of regulatory intervention contribute to achievement of the desired safety outcomes for the sector. Unfortunately, the validation of this method is not funded in the current budget and no reasonable reprioritisation of expenditure is currently available. The scheduling shown assumes the work will not proceed in this FY. While being at the leading edge of such development work carries some risk, its desirability is acknowledge. Actual financial position versus budget will be monitored over time and if finance 19 P a g e

20 becomes available the work will proceed. The MoT has also offered to assist in the development of measures and the CAA will take it up on this offer. 20 P a g e

21 OAG Recommendation No.6. We recommend that the Civil Aviation Authority clarify how its regulatory focus is to be applied in practice through certification, surveillance, and other regulatory action by providing more detailed guidance to staff about what circumstances constitute a significant risk to public safety, and what action they should take when these safety risks are identified. Underlying problem leading to the recommendation: The OAG considers that guidance and training provided to staff is not sufficient to ensure consistency of regulatory decision-making in like circumstances. It is a basic regulatory principle that rules and standards must be applied consistently and fairly across the sector and regulation should be predictable in order to give stability and certainty to those being regulated. Intervention/Corrective actions Internal audit assessment and comment CAA Remarks Development of Regulatory Tools Policy. Provide greater guidance to staff on regulatory decision-making. In progress. N/A. Assessment: The framework envisaged by the project currently underway is designed to meet this specific recommendation. Contract let and progressing. 1-2 week delay against original deadline of 31 Aug Level of confidence in outputs not assessed. Project underway but with slight slippage. This work will be completed by the next quarterly report. It will satisfy the OAG recommendation but will go beyond that to provide staff and the aviation community with guidance on the appropriate/likely choice of regulatory tools in various circumstances. Comment: No auditable evidence beyond signed contract to date. 21 P a g e

22 OAG Recommendation No.7. We recommend that the Civil Aviation Authority give priority to completing the project to improve the integrity and reliability of safety data in its Management Information System, and improve the analysis of this data so that it can be used to better inform regulatory decision-making. Underlying problem leading to the recommendation: Much safety data held by the CAA lacks integrity and is of limited use for analysis. In addition, the CAA s safety analysis capability could be improved. The analysis of safety data should inform regulatory decision-making, the identification of safety risk and the direction of CAA resource. The CAA s limited capability in this area limits its ability to make use of data for these purposes. Intervention/Corrective actions Internal audit assessment and comment CAA Remarks 7A. Assessment: 7A. 7B. 7C. Appointment of Data Standards Manager. Improve the integrity of data input to the management information system. Aviation related concern (ARC) process review Establish the integrity of information input to the management information system from ARCs. Safety Findings Review. Establish the integrity of information input to the Planned (appointment by 1 Nov 10) N/A. Complete N/A (review to establish status). In progress on schedule for completion 17 Dec Stage 1 complete. The decision to appoint a Data Standards Manager addresses one element of the overall capability requirement. The action is subject to recruiting an individual and was not assessed. The ARC Review has established the current status. The review is complete and it found systemic failure so the next stage is critical. Stage 1 of the Safety Findings project is complete and has established the current status. Detailed plan for stage 2 to be completed based on findings. Stage 1 review found culture, knowledge and quality control issues. Next step is critical. Comment: Broader project to enhance data capability is not funded. The measures undertaken to date are subsets of the overall project. Recommendation is being partially addressed but the full project scope needs While resource constraints currently preclude significant enhancement of analysis capability (see below) precursor work can be done to ensure that data input to the system is reliable. Thus, when analysis capability is improved, it will have quality data to work with. The position description is established, role sized, advertising and recruitment process has started. 7B. This review was conducted to assess and report on the quality of the outputs from the ARC process (which are inputs to the Management Information System). The report highlighted concerns with the quality and consistency of the outputs. Required improvement will be implemented in a separate project see Aviation related concern process improvement work below. 7C. This review is being conducted to assess and report on the quality of the inputs to the Management Information System from the raising of safety 22 P a g e

23 management information system from Safety Findings and identify any rectification action required. N/A (review to establish status). to be delivered to fully meet the recommendation. findings. It will also identify any required corrective actions. 7D. 7D. Aviation related concern process improvement. Establish the integrity of information input to the management information system from ARCs. Scheduled to start 1 July Not assessed not planned yet. Rectification work required to correct deficiencies identified in a recent review of this process see above. This work is not funded in the current budget but actual financial performance/position versus budget will be monitored as the year progresses and if finance becomes available the work will proceed. Capability for this work may also become available in subsequent financial years or following the current review of CAA funding. 7E. 7E. Enhance safety data analysis capability. Enhance analysis capability. Scheduled to start on 1/07/11 and finish on 10/2/12. N/A. This work is estimated to cost approximately $450k pa for 5 years and at present it is considered that substantial progress will not be possible until such funding is available. Capability for this work will be sought as part of the current review of CAA funding. 23 P a g e

24 OAG Recommendation No.8. We recommend that the Civil Aviation Authority assess, and, where necessary, provide training to improve its managers capability to effectively manage and lead staff. This includes improving the staff performance assessment process in the General Aviation Group. Underlying problem leading to the recommendation: Management capability to lead and manage staff is inconsistent and the setting of expectations for staff is inconsistent across different parts of the organisation. It can lead to inconsistency in the way in which a particular function is performed by different parts of the organisation and the effectiveness of that function. Intervention/Corrective actions Internal audit assessment and comment CAA Remarks 8A. 8B Improve/refocus performance management. Improve the staff performance assessment process in the GA Group, improve the Performance Review and Development (PRD) process to provide a more obvious link to the SOI objectives and train managers on the application of the new system. Leadership team development. Improve senior managers capability to effectively manage and lead staff. Completed (ongoing monitoring of effectiveness via Output Class 2 quality measurement). Improved performance of functions assessed by Output Class 2 Quality monitoring processes. In progress. In progress not assessed. Assessment: The work on improving and refocusing the performance management system is a subset of the overall recommendation. There needs to be a needs assessment process and then focused management training. See below. For this specific corrective action the project has now delivered a monitoring process that is ongoing. Feedback into performance management is incorporated into policy but yet to be undertaken. The process should improve outputs across certification and surveillance. Individual s performance evaluation is required by the policy; still to be performed but is planned. Leadership development not yet fully developed not assessed. Comment: Overall plan currently being developed to address recommendation. Two individual actions above are positive but mainly precursors. 8A. Note that on-going testing of the effectiveness of this action will be provided by monitoring of Output Class 2 quality measures. Separate work (outside the scope of this project) is planned on strengthening the current link between merit and remuneration. 8B. Concordia NZ Ltd. was contracted to assess current status (survey involving senior and 3 rd tier managers) and provide advice on opportunities for improvement of management performance. First report was provided 11 Aug 2010 and next steps currently being developed. Next planning meeting 21 Sep This is one element in the development of a larger change programme that is being coordinated with the current business planning process. It will also be informed by the outputs from a Value for Money study to be initiated in October. The next quarterly report will provide more detail on this broader development plan. 24 P a g e