Value for Money Study

Transcription

1 Smart decisions. Lasting value. Value for Money Study Delivery of benefits from the National Audit Office s IT-enabled change programme National Audit Office December 2017

2

3 National Audit Office Value for Money Study Delivery of benefits from the NAO s IT-enabled change programme Ordered by the House of Commons to be printed on 12 th December 2017 This report is presented to the House of Commons pursuant to Paragraph 26(3) of Schedule 2 to the Budget Responsibility and National Audit Act Value for Money Study: Delivery of benefits from the NAO s IT-enabled change programme HC 584 Parliamentary Copyright House of Commons 2017

4 This report is prepared solely for the use of the Board and senior management of The National Audit Office. We accept no responsibility for any reliance that third parties may place upon the report. Where this report refers to matters of law, it should not be taken as expressing any formal opinion whatsoever. Crowe Clark Whitehill LLP is a member of Crowe Horwath International, a Swiss verein (Crowe Horwath). Each member firm of Crowe Horwath is a separate and independent legal entity. Crowe Clark Whitehill LLP and its affiliates are not responsible or liable for any acts or omissions of Crowe Horwath, or any other member of Crowe Horwath and specifically disclaim any and all responsibility or liability for acts or omissions of Crowe Horwath or any other Crowe Horwath member Crowe Clark Whitehill LLP Crowe Clark Whitehill LLP is registered to carry on audit work in the UK by the Institute of Chartered Accountants in England and Wales and is authorised and regulated by the Financial Conduct Authority Crowe Clark Whitehill 4

5 Contents Introduction... 6 Executive Summary... 9 BIP benefits realisation Use of IT in delivery of FA products Use of IT in delivery of VFM products Use of IT in knowledge management Appendix 1 Terms of Reference Appendix 2 24 actions for successful transformation Appendix 3 Challenges to wider use of data analytics Appendix 4 Management response to recommendations Crowe Clark Whitehill 5

6 Introduction 1.1. The National Audit Office (NAO) supports Parliament in holding government to account for spending public money. In so doing the NAO helps to improve the way public services are delivered. The NAO audits the financial statements of all central government departments, agencies and other public bodies and reports the results to Parliament. Its value-for-money reports conclude on the effectiveness, efficiency and economy of government spending The NAO saves hundreds of millions of pounds for the taxpayer each year. Its remit is to examine the effectiveness and efficiency of public spending as well as the economy. The NAO is well regarded by its stakeholders and attracts top quality people to join the organisation. Its continued success is, however, contingent on its ability to adapt and develop in a public sector and audit environment which is constantly changing. Much of the NAO s work is focused on transformation in government and so it must practice what it preaches and ensure that its people understand and experience good change management The NAO has developed a new, user-focused approach to change management, with better prioritisation and planning. This has been overseen by the Change Management and Assurance Committee (CMAC), which was established in The CMAC makes sure the NAO change portfolio is aligned with the NAO s strategy, and is well managed and working towards a set of common principles. The NAO s approach to change management is set out in the NAO Change Framework and is based on best practice standards including the Government Digital Service Design Principles 1 and the DSDM Agile Project Framework The NAO has a new Digital Strategy, spanning a three-year strategic horizon. The strategy outlines a digital vision which governs strategic decisions about the NAO s use of technology. The vision for digital is based on four key guiding principles, all digital investment must be; secure, simple, useroriented and of long term value. All digital change initiatives must adhere to these key principles and comply with the NAO s change governance model. The areas covered in this report align closely with the core pillars of the digital strategy The NAO is already delivering on its wider Business Improvement Programme (BIP) which focuses on ensuring the NAO s operating model is optimised to support the effective delivery of its business strategy. BIP is a major programme of change that the NAO embarked on in 2015, which aims to redesign, simplify and standardise the NAO s key business processes, covering project management and resource management, people management and financial management. BIP aligns to the NAO s strategic objective to deliver high performance The new change process has been followed for the BIP implementation, which included extensive user engagement and an agile approach. This approach focuses on value and quality, implementing the highest priority requirements first. In a survey of BIP users, just 1 in 10 respondents felt they had not been given the opportunity to contribute their thoughts and ideas to BIP. The implementation has kept to schedule, and a lot has been achieved in a relatively short amount of time The NAO aims to keep up-to-date in its use of digital techniques to maximise its efficiency and audit effectiveness, and is seeking to deploy digital techniques to enhance its risk assessment processes. The NAO has developed in-house expertise to be able to scrutinise and analyse big and complex data sets Knowledge management is an area that is central to the NAO s strategy and operating model. The NAO gathers much knowledge and has developed pockets of deep expertise. As well as a way of Crowe Clark Whitehill 6

7 reducing costs through eliminating duplication, effective knowledge management increases the speed and quality of response as a direct result of better knowledge access and application. Scope of work conducted for the report 1.9. As part of its own external scrutiny, the Board of The National Audit Office commissions its external auditors, Crowe Clark Whitehill ( Crowe ), to undertake a value for money (VFM) study annually to assess the extent to which the organisation is achieving VFM in its use of resources. This year the review focused on the delivery of benefits from the NAO s IT-enabled change programme, including how IT is being used for better knowledge management and for the delivery of NAO outputs. The review addressed the following key questions: Does the NAO have clear ambitions regarding the use of IT, including for knowledge management and the delivery of NAO outputs? How has the NAO identified, defined, tracked and optimised the realisation of benefits? How is the NAO ensuring it has systems and technology that are fit for purpose and cost effective? Is the NAO getting the most out of its systems and maximising its return on investment in technology? In addressing the key questions of this VFM study we undertook an assessment of the following areas: Review of strategy and plans for investment in technology, including for knowledge management and the delivery of NAO outputs. Review benefits realisation management for the NAO s IT-enabled change programme, and identify any risks to planned benefits realisation. Understand how IT is currently being used, what current projects are ongoing and how these are being coordinated and governed, including for knowledge management and the delivery of NAO outputs. Review of how key investment decisions are made, and how the results of pilot projects and cost-benefit analyses are being taken into account. Understand how the NAO keep up with best practice, and how it ensures it gets the most out of its relationship with the Big 4 through the framework agreements. Interviews conducted and documents reviewed We held interviews with more than 30 NAO staff who have understanding of the BIP programme and the use of IT in the delivery of NAO products and management of knowledge. We obtained and reviewed supporting documents in order to corroborate the information gained during interviews Crowe Clark Whitehill 7

8 Survey of BIP users We provided input to the design of the survey of BIP users that was conducted by the NAO Engagement and Change team during April and May 2017 and have included references to the survey results in our report. Meetings with framework partners We held meetings with two Big 4 framework partners with a view to benchmarking the NAO s current practice and exploring how the NAO benefits from the sharing of knowledge and skills as part of the framework contracts. The results of these meetings have been referred to in this report. Review of literature Additionally, as part of benchmarking the Financial Audit practice of the NAO, we conducted a literature review of recent publications regarding the use of technology in audit. These included: Data Analytics for External Auditors (ICAEW, 2016); The use of data analytics in the audit of financial statements (FRC, 2017); and Request for Input, Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics (IAASB, 2016), along with responses. Review of internal audit work We reviewed the work conducted by the NAO Internal Audit function on the internal audit of BIP Benefits Realisation conducted during April We discussed the draft report for this internal audit with the Head of Internal Audit, and reviewed and discussed the audit work papers the internal auditor who conducted the fieldwork. The work papers were confirmed to support the findings and conclusions contained in the draft audit report. Finding and recommendations The findings of this VFM study are detailed in the main body of this report, with a summary included in the Executive Summary section. We have made a total of 13 recommendations which are included in the Executive Summary. Appendix 4 provides the details of the NAO management s response to each recommendation along with an implementation date and owner Crowe Clark Whitehill 8

9 Executive Summary High-level scope 2.1. This review focused on the management of the delivery of benefits from the NAO s IT-enabled change programme, including how IT is being used for better knowledge management and for the delivery of NAO outputs. The review addressed the four key questions noted at paragraph 1.9 above and our conclusions are summarised below. Does the NAO have clear ambitions regarding the use of IT, including for knowledge management and the delivery of NAO outputs? 2.2. The NAO s new Digital Strategy clearly captures the NAO s ambitions for the use of IT, and has the vision to offer a secure and well-integrated digital environment that enables the NAO to deliver high quality audits, add value to Parliament and clients, and run its operations efficiently. The Digital Strategy includes key guiding principles and enterprise architecture design principles that must be taken into account in decisions about what systems and processes to buy, design or build In particular, the NAO s ambitions for the use of IT as part of its Business Improvement Programme (BIP) are clearly articulated in the BIP business case. The NAO s financial audit development programme (FADP) includes a project for digital audit which covers the use of IT in delivery of financial audit products. The ambition for this project is clearly articulated in the NAO s updated Digital Strategy. The NAO s ambition regarding the use of IT in the delivery of VFM products is captured in the NAO s VFM development programme The NAO places importance on knowledge management, and we saw evidence of all the elements of a knowledge management system in terms of culture, people, processes and technology. However, the NAO still needs to complete its work on an overall knowledge management strategy to ensure these elements are fully integrated and managed as a system. How has the NAO identified, defined, tracked and optimised the realisation of benefits? 2.5. A benefits realisation plan is in place for BIP and has been closely followed and scrutinised in the organisation. Since the FADP started as a continuous improvement programme rather than a change programme, it does not have a benefits realisation plan. However, it is now classified as a change programme which means that the benefits for this programme will now be defined and tracked. The usage of IT tools developed for the VFM practice, as well as user feedback, is being monitored to ensure that these tools are providing the desired benefits We saw evidence of the tracking of benefits for the individual elements of the NAO s knowledge management system, for example measuring and optimising the use of tools, and measuring the impact of training on cultural change. However, there is an opportunity, with the completion of a knowledge management strategy for the organisation, to track and optimise benefits in a more integrated way Crowe Clark Whitehill 9

10 How is the NAO ensuring it has systems and technology that are fit for purpose and cost effective? 2.7. The selection process for the new system implemented as part of BIP evidences the selection of the most fit-for-purpose and cost-effective option. For financial audit, the electronic work paper and workflow system was designed specifically for the NAO s Financial Audit practice and is based on the needs of users, recognising the different types of organisations the NAO audits. We agree with the NAO s assessment that this system is a cost-effective solution that will continue to provide the necessary functionality in the near term The data analytics tools being developed using existing in-house expertise are cost effective and are designed specifically to meet the NAO s needs. The off-the-shelf data analytics tool is, however, only being used for low complexity tasks for large data sets. However, the NAO has recently used its in-house expertise with this, and other tools, to centrally develop and provide advanced analytics to the practice. The NAO continues to investigate how it can be used more widely as well as alternative tools and methods for different types of analytics and user needs, and there may be scope to use it in a more cost-effective way. The NAO conducted some market analysis to determine if an alternative solution exists to the current core analytics tool and approach to its use for financial audit. For example, alternative solutions for those who do not need to use the analytic tool s full capability. However, while alternatives do exist, the NAO assessed the short-term disruption and cost of transferring to an alternative tool would outweigh any benefit. The NAO plans to continue the use of this tool for the next three years and during this time determine if there are better delivery models and tools to meet the practice s, and Parliament s, changing needs in line with the increasingly digital environment in which it operates The NAO s new change framework provides a rigorous and transparent way of ensuring that investments in new systems and technology that are fit for purpose and cost effective. The FADP is now classified as a change programme, and will follow the change framework methodology, for future investments in systems and technology Many of the tools used by the VFM practice have been developed using existing in-house expertise, and these tools are being continuously improved in response to user feedback, to try and ensure that they are providing a fit-for-purpose and cost-effective solution. The VFM audit work paper and workflow solution currently in place is a two-system solution based on existing investments in technology. This system is cost effective and provides an acceptable level of functionality and audit trail, however, the NAO recognises the need to review to what extent the current solution could be replaced with more integrated, open systems Many of the tools used for knowledge management have been developed in house and are fit for purpose and cost effective. The new system implemented as part of BIP is an important tool for promoting good knowledge management. Is the NAO getting the most out of its systems and maximising its return on investment in technology? The NAO s new change framework methodology is designed to ensure the NAO maximises its return on investment in technology and this framework has been applied to BIP. A recent BIP lessons learnt exercise did, however, identify opportunities for the programme to better follow the change methodology In terms of the use of IT in the delivery of financial audit products, the NAO is generally getting the most out of its systems and maximising its return on investment in technology, though in the case of the off-the-shelf data analytics tool, the NAO recognises it is not making the most of its investment 2017 Crowe Clark Whitehill 10

11 in this tool. The NAO has introduced two new tools during 2017 and therefore needs to ensure it monitors take up by audit teams to ensure it is maximising its return on investment Many of the VFM tools being developed in house are making use of the NAO s existing SharePoint platform and are thus maximising the NAO s return on investment in this technology. We saw evidence that the NAO is maximising its return on investment in the specific tools developed for knowledge management. However, there is scope to get more out of the NAO s SharePoint platform as part of an integrated knowledge management strategy. Key findings BIP benefits realisation: In 2015 the NAO embarked upon its Business Improvement Programme (BIP), the key elements of which are: Replacement of legacy finance and HR systems, which were at the end of their lives and no longer supported. Redesign of internal processes to improve both the efficiency and quality of support services on which frontline audit delivery depends. Behavioural change needed to achieve ongoing business benefits As part of BIP, the NAO has replaced legacy systems with a cloud-based, integrated solution that aligns the key business activities of project management and resource management, people management and financial management. This system results in better automated and simplified processes, which means it is easier to keep data up-to-date, everything is done in real time, and there is just one source of the truth with no need for reworking In contrast, historically, the data generated from the legacy systems was less reliable due to conflicts between different systems and an incomplete understanding on the part of users regarding how to operate the systems. It was therefore difficult to get a consistent view of audit or corporate projects without manual work and meetings to verify data The NAO has over time sought to improve its approach to managing change. In particular, BIP was set up to do change in a different way. The approach adopted for user engagement was one of the most significant aspects of this programme. It aimed to be more user focused, gathering the views of people to both inform requirements and to help prepare for change. At Appendix 2 we have highlighted 24 specific actions that the consulting firm McKinsey has identified can triple the odds of success in implementing major change. In our view the approach to change adopted by the NAO is closely aligned to these actions The implementation of core functionality of the new system in September 2016 was a large and complex implementation to create an operational system. The remaining system capability is being rolled out in smaller, separate streams. We have seen evidence that the NAO has made good progress in delivering these additional elements of the system and the elements that remained to be implemented at the time of this VFM study were largely related to the people management module of the system. In our view, BIP has delivered an ambitious set of deliverables, largely to the intended timetable, with any changes to the timetable being discussed and approved by the BIP Programme Board The supplier of the new system has provided the NAO with its off-the-shelf module for people management, however, this module is less well developed than the modules for project and resource management, and financial management. This means that more effort is needed to 2017 Crowe Clark Whitehill 11

12 develop and custom build some of the people management elements of the system. In June this year the supplier announced new strategic partnerships and has proposed changes to its offerings with regards to people management. The NAO is currently working with the supplier to gain an understanding of the new product on offer and the costs of transitioning. At present it is too early to indicate how the NAO will choose to proceed and what the implications will be for the programme In terms of benefits realisation, the Leadership Team, Directors and Private Office are seeing the realisation of qualitative benefits as the system has helped to improve the planning and profiling of work and is now generating corporate dashboards which are facilitating more data-driven oversight and providing better support for decision making According to the original plans, once the new system is fully embedded, the NAO aims to achieve annual benefits by 2018 of 1.1 million, which could be increased to 1.6 million based on the achievement of stretch targets resulting from additional headcount reductions. The latest forecast, as reported to the NAO Board in July 2017, anticipates annual benefits of 1.3 million, which is within the planned range and reflects an achievement of stretch targets of 100, We have seen evidence that BIP benefits realisation has been closely followed and scrutinised in the NAO. There has been regular monitoring by the BIP Programme Board, which has been chaired by an Executive Leader as the Senior Responsible Owner of the Programme and includes membership of key people from across the organisation. The NAO Director of Internal Audit has been attending the BIP Project Board meetings in an observer capacity and has closely monitored BIP benefits realisation since the initiation of the programme There has also been an ongoing programme of work conducted by the NAO internal audit team looking at multiple aspects of BIP. In particular, an internal audit of BIP benefits realisation was reported to the NAO Audit Committee at its meeting in May This internal audit focused on providing assurance on the framework for reporting BIP benefits and costs to the NAO Board. This audit resulted in an internal audit opinion of adequate assurance. It was agreed by the Audit Committee that this work by the internal audit team would be shared with us in order to avoid duplication of work. As part of this VFM study, we reviewed the draft report for this internal audit and confirmed the internal audit work papers support the findings and conclusions contained in the report The BIP team carried out a lessons learnt diagnostic exercise at the end of the first six month of embedding the new system. This helped to identify the key technical and behavioural challenges encountered in implementing the new system, and enabled actions to be agreed to address these issues We provided input to the design of a survey of BIP users that was conducted as part of this exercise. The majority of survey respondents believed that BIP and the new system will help improve ways of working at the NAO. However, the results show that while a core of users have taken to the new system, many others have not found it particularly intuitive. With benefit of hindsight, the NAO assesses that the initial training and support model for the roll out of the new system could have been better targeted The BIP team have explained that they foresaw that it would take time to bed the new system in and for people to adapt to a wide scale change in process. The survey results were largely as they expected at this early stage, and have enabled the BIP team to identify clear actions that need to be taken as part of its commitment to continue developing the system and providing further training and support. The survey results provide a baseline for the NAO to monitor improvement in take up of the system and user engagement over time In our view, the NAO has taken a rigorous, action-oriented approach to the BIP implementation, which is closely aligned to good practice, and the challenges experienced are to be expected in this type of process of significant change. The BIP team anticipated several stages of acceptance that users would likely need to pass through, and ongoing training and support is being provided to 2017 Crowe Clark Whitehill 12

13 enable the learning process necessary to take users through the change. Despite the challenges in embedding the new ways of working in the culture of the organisation, the NAO remains on track to make the savings planned BIP was carried out in line with the NAO s newly introduced Change Framework, which is supported by a newly created Engagement and Change function. This brings additional capacity and skills for managing change, and ensures the NAO has an efficient, joined-up approach to undertaking change which is articulated in a change roadmap for the organisation. The Change Framework sets out the NAO s approach to undertaking change and provides principles that are expected to be applied to individual change projects As part of the NAO s change methodology, change projects pass through a number of phases, with clear objectives and deliverables for each. Movement through the phases requires passing through a gate review process, which provides an independent and objective mechanism to ensure that change is well delivered, and helps the NAO make sure its investments in change are optimised This framework is, however, still in the process of being fully embedded and the Engagement and Change Function has identified the following areas of focus for 2017/18: Improve and embed the change framework. Review the skills and capacity to deliver the change portfolio (need to improve requirements gathering and testing capability already identified). Support project teams in shaping projects and applying the change framework. Enhance guidance and make it easier to consume through a SharePoint Guidance site. Recommendations In order to address the key technical and behavioural challenges identified in the lessons learnt diagnostic, the NAO should continue to improve the usability of the new system, reinvigorate user engagement and continue to provide the training and support necessary to secure user buy in The NAO should continue with its commitment to a professional approach to change management by ensuring the Change Framework is fully embedded, with support at all levels of the organisation for compliance with the framework When embedding the Change Framework, particular areas for attention should be to ensure change projects have clear problem statements and requirements, and have baseline data and meaningful measures of project success. It is also important to ensure all change projects go through the gate review process and that project teams follow the gate review recommendations or provide explanation where recommendations are not followed. Use of IT in delivery of FA products The NAO s approach to investing in technology for financial audit The use of technology in the financial audit process enables reliance to be placed on controls built into IT systems and enables large volumes of data to be analysed and unusual transactions identified. However, the experience of large audit firms is that effective use of data analytics in the financial statement audit requires substantial investment in hardware, software, skills and quality control. Significant use of IT-dependent audit techniques has typically been restricted to these firms, where the size and complexity of their clients justifies the use of technology-enabled audit and provides an appropriate return on investment Crowe Clark Whitehill 13

14 2.36. Though the NAO is much smaller than the large audit firms, its client base includes some large and highly complex organisations. For this client base, the NAO proactively seeks to ensure that its service is on par with the large firms. The NAO estimates, however, that 80% of its clients are relatively noncomplex for whom there are more limited opportunities for a greater use of technology in the financial audit process. The NAO s aim is to invest judiciously and continue to use existing tools whilst these continue to provide an appropriate and cost effective way of delivering a high quality audit Nonetheless, the NAO is conscious that central Government has the ambition to expand the use of technology in its operations, which will present greater opportunities and need to embed technology in the audit process. The NAO s overall assessment, as at March 2017, is that it is behind the Big 4 audit firms in the use of technology, however, it is making targeted investments that are enabling it to close the gaps where necessary, and are consistent with its strategy to be a fast adopter of changes where they have been proven to add value to the audit process In our opinion, the NAO s approach to adopting technology-enabled audit is appropriate given the characteristics and pace of change of its client base. We believe this approach has the advantage of enabling quality audit work to be delivered in a cost effective way whilst also ensuring the NAO s approach remains relevant and comparable with the large commercial firms. How the NAO is building its technology-enabled financial audit capability The NAO initiated a broad ranging programme called the financial audit development programme (FADP) in 2015, which is designed to ensure that the NAO s approach to financial audit delivers a high quality, competitive product. Historically the individual projects under the FADP have been classified as continuous improvement projects. However, the FADP is now classified as a change programme by the NAO s Change Management and Assurance Committee, which has the remit of ensuring the NAO s change portfolio is aligned with its strategy, well managed and working towards a set of common principles Digital Audit is a key project within the FADP, which aims to ensure that the NAO has sufficient IT capability to allow it to deliver audits that can deploy the full range of IT-enabled audit techniques for those clients where such an approach is appropriate and needed. The ambition for this project is clearly articulated in the NAO s updated Digital Strategy which was being finalised at the time of our VFM study. This project has three streams to it: I. Increased use of automated controls testing for complex, transactional based clients The NAO has plans to increase the extent to which it applies an automated controls-based approach from 21 clients in 2016/17 to 35 clients in 2018/19, by focusing initially on the largest NAO clients that rely most on automated processes and then smaller clients where appropriate. II. Embedding data analytics as a key audit tool to help deliver consistency, quality, insight and efficiency Beginning in 2015/16, the NAO has been using in-house expertise to develop its own data analytics tools, which is enabling the roll out of quality-assured analytics routines. This is a model that follows good practice in the financial audit industry. The key ongoing initiatives at the time of this VFM study were the development of data analytics tools for analysing journals and for the testing of purchase-to-pay systems. These initiatives are ensuring the NAO s use of data analytics is comparable to that of comparable private-sector audit firms. In addition to its centrally-developed data analytics tools, the NAO continues to use its core, off-the-shelf data analytics tool, which has been in use for many years and is embedded in the financial audit working practices. This tool is provided directly to financial auditors and its use has generally been limited to a number of key financial audit tasks, such as sampling 2017 Crowe Clark Whitehill 14

15 and confirming the validity of trial balances. The use of this tool is not mandatory at the NAO and other tools such as Excel are being used as an alternative. In adopting this decentralised approach, the NAO has not sought to apply this tool to its financial audit work as extensively as it possibly could. The NAO has assessed that the short-term disruption and cost of transferring to an alternative tool would outweigh any benefit, and therefore plans to continue the use of this tool for the next three years and during this time determine if there are better options. III. Use of technology to enhance operational efficiency The NAO s use of tools to increase the efficiency of the audit process is comparable to that of the private-sector audit firms, and includes tools for: audit workflow and working paper documentation; online bank confirmations; and online collaboration. Tools currently used in NAO financial audit The table below shows a breakdown of the tools used in the NAO financial audit process. Purpose Tool Use Enhanced operational efficiency Data Analytics Project and Resource Management module Electronic audit work paper solution Confirmation.com Huddle Off-the-shelf data analytics tool Data analytics apps (developed in house) Tableau Catflap Audit project management Audit workflow and working paper documentation system Online service to automate the bank confirmation process Online collaboration tool providing a secure means for transfer and storage of audit documentation Key financial audit tasks, such as sampling and confirming the validity of trial balances Apps for journals testing and purchase-to-pay testing Data visualisation In-house developed system to process, store and provide controlled access to bulk client data Comparison to other Supreme Audit Institutions Supreme audit institutions (SAIs) are national agencies responsible for auditing government revenue and spending. In different countries their legal mandates, reporting relationships, and effectiveness vary, reflecting different governance systems and government policies. Our impression based on material we have seen is that the NAO is leading rather than lagging in the use of IT when compared to other SAIs Crowe Clark Whitehill 15

16 Knowledge and skills transfer from framework partners The NAO aims to benefit from its relationships with its framework partners through knowledge and skills transfer, and though it has been using these relationship to keep up to date with the latest industry developments, there was a perception from the conversations we had as part of this VFM study that there is an opportunity for greater skills transfer. Joint working was identified as a possible way for NAO staff to benefit from skills and knowledge transfer, in particular through NAO IT audit staff working within framework partner IT audit teams for the duration of audits. Recommendations The NAO should follow through with its plans to ensure the development programme for the Financial Audit practice, whose overriding purpose is to deliver enhanced quality across the NAO s audit estate, is brought into line with the new NAO Change Framework. This will have the benefit of ensuring suitable project management and governance is in place, as well as ensuring fit with other projects in the corporate portfolio and alignment with the NAO s architecture and digital strategy Where digital audit techniques, such as data analytics, are assessed as being a key means of delivering audit quality, applying aspects of the change framework, such as engagement, requirements and testing, will be important to ensure a successful ramping up of the use of such inhouse developed data analytics tools. Other aspects of the change framework, such as putting in place a benefits realisation plan and following the gate review process, will be an effective way to demonstrate that systems and technology are fit for purpose, cost effective and maximise the NAO s return on investment In the case of the off-the-shelf data analytics tool, given the use of this tool is not compulsory and audit teams can opt to use other tools, it may be the case that only a limited number of licences are required in practice. The NAO recognises this and as part of a recent licence renewal, the site licence was replaced with a substantially reduced number of licences. Licences are only provided to users directly requesting them and usage will be monitored to ensure the appropriate number are held. The NAO should therefore continue to monitor the usage of this tool and adjust the number of licences held accordingly More generally, the NAO should assess the usage of the tools used in the financial audit process to ensure it is making most of its investment in technology. Particularly, for the tools that have been newly introduced, such as Confirmation.com and Huddle, the NAO should monitor whether take up is as expected and whether the licensing model being followed is the most cost effective As part of ensuring greater skills transfer to NAO staff from framework partners, the NAO should continue to pursue opportunities for greater joint working, for example, through NAO IT audit staff working alongside framework partner IT auditors on specific audits. Use of IT in delivery of VFM products The NAO s Value for Money (VFM) practice produces studies that evaluate the value for money of public spending and include recommendations to help government improve public services. Parliament uses the NAO s VFM work to hold government to account for how it spends public money. Each study examines a major area of government expenditure, with the objective of forming a judgement on whether VFM has been achieved. In common with the Financial Audit practice, the project and resource management module of the new system implemented as part of BIP is used for operational planning, resource allocation and audit project management Crowe Clark Whitehill 16

17 2.50. The NAO s electronic audit work paper solution was designed for the financial audit process and has been perceived by the VFM practice to be not best suited for VFM workflow. A SharePoint application, NAO Tank, is instead being used to manage VFM workflow and team collaboration. NAO Tank was, however, not designed to be an audit documentation storage system and VFM teams are required to store key audit documents in the NAO s electronic audit work paper solution on completion of each assignment, which serves as the document repository for the VFM practice. This results in two-system solution for VFM workflow management (NAO Tank) and documentation storage (NAO s electronic audit work paper solution) The NAO s electronic audit work paper solution is a closed system which means there is limited potential for automated integration with other NAO systems. This results in a small amount of additional effort needed to rekey some basic audit information and manually transfer documentation into this electronic work paper system. This two-system solution is based on existing investments in technology and is, in our view, cost effective and provides an acceptable level of functionality and audit trail. The NAO, however, recognises the need to review to what extent the current solution could be replaced with more integrated, open systems The compliance rates for VFM teams completing the storage of key documentation in the electronic work paper system have historically been low as there was no formal requirement for teams to do this. Compliance rates have now, however, increased considerably as a result of training and the inclusion of a requirement in the audit close-down checklist for teams to complete the storage of key documentation The VFM practice uses a large number of additional tools to support its work, the majority of which enable VFM teams to access the knowledge they need to perform their VFM work. Many of these tools have been developed in-house, in particular using the strong IT and technology skills of the Methods, Economics and Statistics Hub (MESH), which is one of the NAO s communities of practice. Notably, MESH leads the NAO s increasingly strong analytical capability to scrutinise and analyse big and complex data sets to improve efficiency and insight in its analysis to support its assurance work for Parliament MESH represents the NAO at the International Organisation of Supreme Audit Institutions (INTOSAI) and the European Organisation of Supreme Audit Institutions (EUROSAI) on the subjects of data analytics and big data. It is evident from the NAO s extensive engagement with Supreme Audit Institutions (SAIs) on data analytics that the NAO is taking a lead role in sharing knowledge in this area, and is recognised as a leading expert on the use of data analytics in both VFM and financial audit. Recommendations The NAO should consider the costs and benefits of the possible options for moving to a better integrated, open system for audit workflow and documentation. The introduction of any new solution should take into account the NAO s digital strategy and enterprise architecture, and be considered in the context of the NAO s change roadmap to ensure appropriate prioritisation given other organisational priorities and available resources In the meantime, the NAO should continue with efforts to increase the compliance rate for VFM teams completing the storage of key documentation in the NAO s electronic work paper solution in order to ensure the complete audit trail for each audit is available in this system Crowe Clark Whitehill 17

18 Use of IT in knowledge management Developing and applying knowledge is a key part of the NAO s strategy, and we saw many examples of how knowledge management is incorporated into the NAO s practices and supported by the use of IT The NAO s data governance and data management has improved under BIP. The new system increases the use of common data, which promotes a wider culture of knowledge sharing and collaboration. However, the NAO still has a wide variety of knowledge management solutions in place locally which are not joined up Notably, an ongoing project in the VFM practice has used the NAO s strategic platform, SharePoint, to create a site named One Stop Shop, which brings together information that people previously had to look for in separate places. The NAO intends to explore ways of optimising the use of SharePoint to develop additional integrated, user friendly and flexible solutions to better support the NAO s information and knowledge management capability Strategic work on knowledge management was conducted during 2015 and 2016, which consisted initially of the preparation of a Knowledge Capability Strategy, and subsequently of work to define an information and knowledge strategy. However, resource demands from BIP meant that the NAO took a business decision to put the strategic work on knowledge management on hold whilst BIP was being prioritised In the interim period, important improvements in knowledge management have been achieved through specific SharePoint projects and also through BIP itself, which improves the NAO s information and knowledge management capability through notably easier collaboration, better data and enhanced integration. Now with BIP in the process of being bedded in, there is an opportunity to continue with work to put in place an overall strategy for knowledge management and define overall responsibility for knowledge management in the NAO More generally, given that developing and applying knowledge is a key part of the NAO s strategy it is important that good governance, clear strategic priorities and clear roles and responsibilities for information and knowledge management are developed to ensure user experience, system management and system development are predictable, effective and focussed. The NAO Digital Design Authority, which has been formalised as part of the new digital strategy, is to be responsible for ensuring that any new requirements or initiatives fit with the NAO s digital principles, architecture and strategy. Recommendations The NAO should continue with its strategic work on knowledge management with a view to implementing an overall vision and organising framework for the many knowledge management activities taking place in the organisation. This will have the benefit of making sure all activities are aligned and that culture, people, processes, and technology work together to ensure that knowledge is managed effectively in support of the NAO s business strategy A senior leader with overall responsibility for knowledge management should be identified to drive knowledge management in the organisation. Some key activities to undertake would include: Establishing clear priorities for knowledge management. Developing and championing best practice. Ensuring the clarity and integration of the NAO s tool offering, with a clear remit for each tool. Empowering owners to update and share information in order to avoid placing a burden on central resources and to ensure information is kept up to date Crowe Clark Whitehill 18

19 Measuring and reporting on knowledge management performance In order to improve knowledge and information sharing and findability, it will be important to work with digital services and the NAO Digital Design Authority to ensure good data governance and management, aligned with a well-defined information architecture, and a more integrated technology offering that is aligned to digital strategy. Consideration should be given to building a roadmap of solution architecture so that the delivery of information and knowledge management functionality can be aligned with this roadmap Crowe Clark Whitehill 19

20 BIP benefits realisation 3.1. The Business Improvement Programme (BIP) business case expresses the NAO s ambition to have a single and reliable data and information environment that increases the NAO s standards and efficiency and improves the pace of decision making across the organisation. In order to do this, the NAO has replaced legacy systems, which were redundant and unsupported, with an integrated solution that aligns the key business activities of project management and resource management, people management, and financial management, whilst also taking the opportunity to streamline processes and empower people to make decisions. This new system is cloud-based and uses the Salesforce platform Historically, the data generated from the legacy systems was inconsistent due to conflicts between different systems and variation in the commitment by users to keep data up to date. This meant time needed to be spent manually checking, re-working and integrating data into separate offsystem spreadsheets to ensure the quality that the Leadership Team needed for effective decision making. Figure 1 shows the case for change as articulated in the BIP Business Change and Engagement Strategy, which shows how BIP addresses these issues. Figure 1 BIP case for change Source: NAO BIP Business Case - September BIP promises clear benefits. An integrated system, with better automated and simplified processes, means it is easier to keep data up-to-date. Everything will be done in real time with just one source of the truth and no need for reworking. Creating a data-driven office in this way will improve planning and resource allocation, improve decision making, improve collaboration and team working, save time and money, and shift efforts from internal administration to making greater external impact Crowe Clark Whitehill 20

21 3.4. The BIP business case identified a number of key conditions on which the delivery of benefits depends, namely: Achieving cultural and behavioural change. Producing a solution that is recognised by NAO people as easy and intuitive to use. Successful integration with other NAO systems. Alignment of BIP with process developments in the individual service areas The NAO has over time sought to improve its approach to managing change. In particular, BIP was set up to do change in a different way. The approach adopted for engagement was one of the most significant aspects of this programme. It aimed to be more user focused, gathering the views of people to both inform requirements and to help prepare for change. As an example, over half the Office attended user labs which were held to help understand user requirements and priorities. What has been achieved to date 3.6. Phase 1 of BIP went live on 19 September This initial deployment covered the core system functionality, which included project planning, resource allocation and core finance with limited people management capability. This was a large and complex implementation to create an operational system, with the implementation of the remaining capability of the system being rolled out in smaller, separate components The first significant test of the new software and ways of working was the operational planning for the 2017/18 financial year which took place successfully in December 2016 and January Roll out of additional functionality as part of Phase 2 began in the first half of 2017 covering the move to the new outsourced payroll service, and capability in people management for employee benefits and performance management. Phase 3 began in summer 2017 covering skills management, workforce planning and recruitment The supplier of the new system has provided the NAO with its off-the-shelf module for people management, however, this module is less well developed than the modules for project and resource management, and financial management. This means that more effort is needed to develop and custom build some of the people management elements of the system In June this year the supplier announced new strategic partnerships and has proposed changes to its offerings with regards to people management. This announcement means that the NAO will need to explore future options for the best way to achieve all the required system functionality for people management. The NAO is currently working with the supplier to gain an understanding of the new product on offer and the costs of transitioning. At present it is too early to indicate how the NAO will choose to proceed and what the implications will be for the programme. Benefits realisation The NAO Leadership Team, directors and Private Office are seeing the realisation of qualitative benefits as a result of better planning and reporting. The 2017/18 operational planning has resulted in improved transparency and information sharing. The system has helped to improve the planning and profiling of work and is now generating corporate dashboards which are facilitating more datadriven oversight and providing better support for decision making Crowe Clark Whitehill 21

22 3.11. This original BIP business case assumed annual benefits of 1.1 million once the system is fully embedded, which are to be achieved by 2018 through: Headcount reductions in the back office as processes are streamlined; Increased productivity of frontline teams through easier and quicker to use systems and better resource allocation tools; Reduced dependency on temporary staff and contractors to support frontline work, achieved by improved project planning and better utilisation of staff; Reduction in net technology spend as legacy systems are retired It was estimated that these annual benefits could be increased to as much as 1.6 million based on the achievement of stretch targets resulting from additional headcount savings. Forecast sustained annual benefits by 2018 of 1.3 million were reported to the NAO Board in January and July 2017, which is within the planned range and reflects an achievement of stretch targets of 100,000. Table 1 below summarises the details of the BIP benefits and costs as reported to the NAO Board in July Table 1 BIP costs and benefits summary reported in July 2017 Source: BIP Benefits Realisation Update to the NAO Board July The July report to the Board explained that annual costs are forecast to be 48,000 higher than the baseline due to increased numbers of annual licences being required for the new system over those originally planned. A key reason for this is that licences are needed for additional staff to undertake work on new audit clients such as the BBC, Bank of England and financial services companies. However, taking into account the forecast achievement of stretched targets, the overall net sustained annual benefits are forecast to be 130,000 higher than the baseline The two-year contract with the supplier of the new system ends in January 2018 and the NAO will have to undertake a new procurement exercise and have recognised that existing discount terms need to be maintained to avoid an increase in costs. The NAO is also exploring options to mitigate the future risk of vendor lock in We have seen evidence that BIP benefits realisation has been closely followed and scrutinised in the NAO. There has been regular monitoring by the BIP Programme Board, which has been chaired by the NAO Chief Operating Officer and includes membership of key people from across the 2017 Crowe Clark Whitehill 22

23 organisation. The NAO Director of Internal Audit has been attending the BIP Project Board meetings in an observer capacity and has closely monitored BIP benefits realisation since the initiation of the programme There has also been an ongoing programme of work conducted by the NAO internal audit team looking at multiple aspects of BIP. In particular, the NAO internal audit team conducted an audit of BIP Benefits Realisation during April 2017 which focused on the framework to capture and report on benefits and costs accurately. The final report for this audit was presented to the Audit Committee at its meeting on 26 May 2017, which was attended by Crowe. This audit resulted in an internal audit opinion of adequate assurance and concluded that there is an adequate and effective framework of control to mitigate the risks to the achievement of the following objectives: Benefits are consistent with the original business case and are recorded accurately and completely. Benefits are directly attributable to the programme and the impact has been realised in cash terms. The methodology has been approved by the Board and leadership team and reports are accurate and complete The BIP Benefits Realisation internal audit focused on the costs and benefits as reported in the BIP Benefits Realisation Paper for the Board dated January As part of this VFM study, we discussed the draft report for this internal audit with the Head of Internal Audit, and the audit work papers were reviewed and discussed with the auditor who conducted the fieldwork. We confirmed the work papers support the findings and conclusions contained in the draft audit report. It was agreed at the May 2017 Audit Committee meeting that the work of the internal audit team would be shared with us in order to avoid the duplication of work. In addition to reviewing this work by the NAO internal audit team, we did not perform any additional work to independently verify the BIP benefits and costs that have been reported to the NAO Board. Results of BIP diagnostic The BIP team anticipated that, in the first six months of adoption, users would still be learning how to use the system and the system would still be undergoing configuration and enhancement, and, at the end of this period, they carried out a BIP lessons learnt diagnostic exercise in order to: Identify what worked well and examples of good practice. Identify outstanding (technical and behavioural) barriers to change and current issues. Agree actions to address those issues As part of this exercise, a survey of BIP users was conducted during April and May 2017 to help pinpoint the key areas where further action is needed to support people with the learning process, and identify the key issues with the new system and business processes that need to be addressed. Crowe provided input to the design of this survey as part of this VFM study. Users opinions were sought on: the usability of the system, how helpful it is in supporting them in their roles, and their overall opinions of the system so far The BIP team foresaw that the new system would take time to bed in and the survey scores are largely as expected and do not impact on the planned benefits. In our opinion, the survey results are typical of any situation where there is a major system change which is at a relatively early stage and users are still in the process of adapting to the change Crowe Clark Whitehill 23

24 3.21. The results showed that whilst a core of users have taken to the system, others have expressed they do not find it particularly intuitive. For example, according to survey comments, some users are finding the allocations processes overly complex and they still have difficulty in getting the information from the planners that they need. With the benefit of hindsight, the BIP team considers the initial training and support model could have been more targeted and better timed Figure 2 shows the survey respondents overall opinions of the system. Encouragingly, many more respondents than not were of the opinion that BIP and the new system will help improve ways of working at the NAO. For the vast majority of respondents the new system was providing clarity regarding resource allocation with people only being booked where needed. Similarly, a large majority of users believed their data in the system is up-to-date At the time of the survey, however, approximately one third of respondents indicated they did not trust the data in the new system yet, and around a half perceived the organisation did not have one version of the truth yet. Although nearly 60% of respondents said they still used spreadsheets or workarounds in addition to the new system, it was not clear whether this was due to people using data from the system to create their own visualisations or whether they were creating their own data outside the system, thereby perpetuating different sources of the truth. Figure 2 Overall opinions of the new system How much do you agree with the following statements? BIP and the new system will help improve ways of working at the NAO: I book only the resources I need I am clear what projects I am working on I use workarounds/ spreadsheets in addition to the system The office has one version of the truth My data is up to date I trust the data in the new system 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Source: NAO BIP survey results, April Crowe Clark Whitehill 24

25 3.24. It is important to recognise that the survey was responded to by all levels of staff and without a demographic analysis it is likely that the more numerous junior members of teams views impact on the scoring. Importantly, at the time of our VFM study, the key decision makers in the organisation were using data from the new system for their decision making. This group in particular has benefited greatly from the availability of integrated, real-time information which is providing them with key information for decision making These results show there is still some way to go to bed the new system in, and also reflect the fact that people from all levels of the organisation responded to the survey. We found evidence from conversations as part of this VFM study that, at more operational levels, spreadsheets continue to be used out of habit, even though the same information is now available in the new system As intended, the results of this survey have enabled the BIP team to identify clear actions that now need to be taken as part of its commitment to continue developing the system and providing training and support necessary to take users through the change. The Business Change and Engagement Strategy the NAO prepared for BIP anticipated several stages of acceptance that users would likely need to pass through Priority actions agreed as part of the BIP diagnostic include: training events offering a variety of training to match different user needs; user experience testing to identify areas where the system can be simplified and usability improved; and root and branch reviews and workshops looking at issues related to system functionality The survey results also provide a baseline for monitoring improvements in user take up over time. The expectation is that users will also remain committed to their part in making BIP work and will make the required effort to learn how to use the system It is important that the results of this survey are taken in context given that negative reaction to change is a common occurrence and is to be expected in how individuals respond to a major change in the workplace. This phenomenon is dealt with extensively in change management literature and can be explained as a resistance to losing the tried and known in the face of having to move to the untried and unknown In our view, the NAO has taken a rigorous, action-oriented approach to the BIP implementation, which is closely aligned to good practice. In particular, the approach followed by the NAO is closely aligned to the 24 specific actions 4 that the consulting firm, McKinsey, has identified can triple the odds of success in implementing major change (see Appendix 2). These practices include communicating effectively, leading actively, engaging and empowering employees, and creating an environment of continuous improvement. Need to embed change framework The NAO s new Change Framework sets out its approach to undertaking change, and is supported by a newly created, director-led Engagement and Change function. This has strengthened the NAO s capacity and skills for managing change, and ensures the organisation has an efficient, joined-up approach to undertaking change. 3 Burke, W., 2011, Organization Change: Theory and Practice, London: Sage, pp Crowe Clark Whitehill 25

26 3.32. Whereas in the past the NAO had more resources and which enabled the different parts of the organisation to develop more independently, now, with less resources, change is being integrated into one story, which is articulated in a change roadmap for the organisation The Change Framework provides principles that are expected to be applied to individual change projects and also provides guidance on project methodology and assurance, which helps to ensure projects have a clear problem statement and requirements, and have baseline data and meaningful measures of project success. The good practices followed for the BIP change process are based on the principles described in the NAO s Change Framework As part of the NAO s change methodology, change projects pass through a number of phases, with clear objectives and deliverables for each. Movement through the phases requires passing through a gate review process, which results in a decision on the project, namely: continue to next phase, further work needed, put on hold or abandon Gate reviews help the NAO to make sure that its investments in change are optimised by allowing teams to reflect on progress to date and providing an independent and objective mechanism that draws on internal expertise in order to identify issues and risks and assess the quality of solution design processes. Reviews are also important for effective portfolio management notably in ensuring alignment between projects and considering the project in a wider context The Change Framework is, however, still in the process of being embedded. For example, though it was used for the implementation of the new system as part of BIP, the BIP diagnostic identified that there were a number of lessons learnt that would have been addressed automatically if the Change Framework had been more fully embedded at that time. These included matters related to more closely following project methodology and ensuring that the significant investment in people engagement needed to inform requirements and support change management is maintained post go-live The Engagement and Change Function has identified the following areas of focus for 2017/18: Improve and embed the change framework. Review the skills and capacity to deliver the change portfolio. The need to improve requirements gathering and testing capability has already been identified. Support project teams in shaping projects and applying the change framework. Enhance guidance and make it easier to consume through a SharePoint Guidance site Crowe Clark Whitehill 26

27 Use of IT in delivery of FA products The role of technology in financial audit 4.1. A financial audit involves an independent examination of an entity s systems, books and records to ensure that the financial statements, fairly reflect that organisation s financial performance and position. The NAO s financial audits give assurance over two aspects of government expenditure: the truth and fairness of financial statements; and that transactions are in accordance with parliamentary expectations and relevant authorities (regularity) A key element is the checking of the validity of financial transactions. Historically this has been done by manually checking a sample of transactions back to supporting documentation, potentially in conjunction with evaluating and, where appropriate, testing manual controls. The use of technology enables reliance to be placed on controls built into IT systems. For example, so long as the control of approving purchases has operated as intended, reliance can be placed on the control to permit reliance on the validity of the purchase made Computer systems that financial auditors now use have the storage capacity and processing speed to check a large volumes of data which means that whole populations of data, rather than a sample, can be analysed and unusual transactions identified Testing IT controls and undertaking data analysis, however, has a cost to it. The audit of IT controls involves a good deal of information gathering on controls and supporting systems, and requires access to IT auditors with the necessary technical skills to perform the testing. For data analytics, downloading data and writing the analysis routines to interrogate data is time consuming and relies on skills not typically found in financial audit teams. Auditors need to weigh these infrastructure costs against the value these techniques add to the audit approach According to a Financial Reporting Council (FRC) thematic review published in , the use of audit data analytics (ADA) in financial audit is not as prevalent as the market might expect, and important barriers and challenges exist to the wider use of ADA by audit firms. In particular, auditors have indicated that they find challenges in fitting the audit evidence derived from data analytics into the current audit evidence model within the FRC s International Standards on Auditing (UK). We have provided an overview of the challenges to the wider use of ADA in Appendix 3, which we have based on the International Auditing and Assurance Standards Board (IAASB) Request for Input 6 and responses received, and the ICAEW's 2016 publication, 'Data analytics for external auditors' The FRC concluded that to date, the tools developed by the global audit firms have focussed largely on risk assessment procedures, through analysing populations of data to identify which items to test, and in particular that the use of data analytic techniques is pervasive in the UK for journal entry testing in response to the requirements of auditing standards Technology has also been used to enhance the operational efficiency of the audit process. All significant financial audit organisations, in both the private and public sectors, have to some extent invested in automation of their audit work papers. Also audits involve the sharing of large volumes of information between client and auditor, increasingly technology is being used to allow this information to be shared via secure, online methods. 5 FRC Audit Quality Thematic Review, The Use of Data Analytics in the Audit of Financial Statements, January IAASB Request for Input: 'Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics', September ISA The Auditor's Responsibilities Relating to Fraud in an Audit of Financial Statements 2017 Crowe Clark Whitehill 27

28 The NAO s approach to investing in technology for financial audit 4.8. Typically, significant development and leading-edge use of IT-dependent audit techniques has been restricted to the larger audit firms, particularly the Big 4 firms 8 where the size and complexity of their clients justifies the use of technology-enabled audit and provides an appropriate return on investment Though the NAO is much smaller than a Big 4 firm, its client base includes large and highly complex organisations such as HM Revenue & Customs, Department for Work and Pensions, Home Office and Ministry of Defence. In addition the NAO is now the auditor of companies and corporations that have historically been audited by the Big 4, for example, Network Rail, UK Asset Resolution and the BBC. For this client base, the NAO is proactively seeking to ensure that its service is competitive with the large audit firms However, the NAO estimates that nearly 80% of its clients are relatively noncomplex for whom there are more limited opportunities for a greater use of technology in the financial audit process. The NAO s aim is to invest judiciously and continue to use existing tools whilst these continue to provide an appropriate way of delivering a high quality audit. In the NAO s own assessment, it does not need to be a market leader or innovator in technology-enabled audit, but it does need to be a fast adopter of changes where they have been proven and are being widely adopted within the industry Nonetheless, the NAO is conscious that central Government has the ambition to expand the use of technology in its operations, which will present greater opportunities to embed technology in the audit process. To ensure that the NAO was well placed to respond to these digital changes the NAO has recruited a senior IT audit professional from a Big 4 firm to lead the NAO s technologyenabled financial audit initiatives. This senior level appointee has assessed how the NAO s financial audit approach compares private sector audit firms regarding the use of technology This assessment included commissioning a Big 4 firm to review a small sample of NAO audit files and compare the approach taken with that the Big 4 firm would have adopted had they been doing the review. This led to an assessment that the Big 4 do use technology in their audit approach more extensively than the NAO. However, the NAO s lower reliance on technology, for the sample selected, could in most cases be justified on the basis of efficiency, in that the cost of investing in a more technology-enabled audit would not be justified on the basis of reduced audit effort given the current maturity of clients IT systems and processes The NAO s overall assessment, as at March 2017, is that it is behind the Big 4 in the use of technology (see tables 3 and 4). The NAO is using this assessment to inform the targeted investments that it is making to close the gaps where appropriate, and its approach is consistent with its strategy to be a fast adopter of changes where they have been proven to add value to the audit process In our opinion, the NAO s approach to adopting technology-enabled audit is appropriate given the characteristics and pace of change of its client base. The key advantage of this approach is that it enables quality audit work to be delivered in a cost effective way, whilst also ensuring the NAO remains competitive with the large commercial firms. 8 These large multinational firms dominate the market for the financial audit of the largest and most complex organisations 2017 Crowe Clark Whitehill 28

29 How the NAO is building its technology-enabled financial audit capability As part of its proactive approach, the NAO, in 2015, initiated a broad ranging programme called the financial audit development programme (FADP). The FADP has a remit to look at all aspects of the way the NAO conducts financial audits as a means of internal challenge to ensure that the NAO s approach to financial audit delivers a high quality and market competitive product. Historically the individual projects under the FADP have been classified as continuous improvement projects. However, the FADP is now classified as a change programme by the NAO s Change Management and Assurance Committee, which makes sure the NAO s change portfolio is aligned with its strategy, well managed and working towards a set of common principles An important element of the FADP is a key project entitled Digital Audit, which aims to ensure that the NAO has sufficient IT capability to allow it to deliver audits that can deploy the full range of ITenabled audit techniques for those clients where such an approach is appropriate and needed. The ambition for this project is clearly articulated in the NAO s updated Digital Strategy which was being finalised at the time of our VFM study. Digital Audit has three streams to it: Increased use of controls-based testing for complex, transactional based clients. Embedding data analytics as a key audit tool to help deliver consistency, quality, insight and efficiency. Use of technology to enhance operational efficiency Each of these streams is expanded upon below. Increased use of controls based testing for complex, transactional-based clients The NAO audits a diverse range of institutions, including all government departments, and certifies around 370 accounts each year. On an evolutionary basis, the NAO is seeking to increase the extent it applies a controls-based approach, expanding the use of business controls-based auditing and IT controls where it is possible to do. Table 2 shows the growth of the NAO s use of controlsbased auditing to 2016/17, along with projected growth to 2018/19. The focus has been on largest NAO clients that have high reliance on automated processes. Over time, a controls-based approach will also be applied to the smaller clients where appropriate. Table 2 The NAO s increasing use of controls-based auditing Rollout 2015/ / / /19 No. of audits where reliance is placed on automated IT controls Source: NAO Technology enabled audit progress update presentation to Leadership Team on 25 April Crowe Clark Whitehill 29

30 Embedding data analytics as a key audit tool Beginning in 2015/16, the NAO has been using in-house expertise to develop its own data analytics tools, which is enabling the roll out of quality-assured analytics routines. This model is closely aligned to the FRC s good practices for effective use of ADA techniques as it involves centrallysupported roll out of a small number of standard data analytics tools which are clearly defined within the audit methodology and have specialist support for data capture Similarly, the IAASB, in its Request for Input regarding the use of technology in audit, noted that some firms are using audit delivery models that are different to the traditional engagement team structures, with certain specialised expertise being centralised and supporting numerous audit teams. Both of the NAO framework partners (two Big 4 firms) that we spoke to as part of this VFM study had central data analytics hubs that process client data using their own proprietary standardised tools and provide the results to the financial audit teams At the time of this VFM study, there were two key ongoing initiatives: Use of an NAO developed data analytics tools for analysing journals Journals are manual adjustments made to accounting records and are a key area of audit focus given the associated risk of error or fraud. The journal analytics tool provides teams with: (i) a comprehensive set of potential risks to assess each journal against; and (ii) a userdirected automated method of scoring them to provide clear clarification of the basis on which high risk journals are identified for further investigation. This has been rolled out across 11% of the audits the NAO undertook in 2016/17 and the intention is to eventually roll out to all clients. More focussed feasibility studies of business process specific data analytics for a smaller portion of the NAO client base Pilots in 2016/17 have focussed on the use of data analytics for risk assessment purposes on three clients, principally to support the testing of purchase-to-pay systems Table 3 shows the NAOs progress and projections for the roll out of data analytics tools. Table 3 The NAO s roll out of data analytics tools Data Analytics Rollout (no. of audits) 2015/ / / /19 Journal data analytics to Business process analytics Source: NAO Technology enabled audit progress update presentation to Leadership Team on 25 April Crowe Clark Whitehill 30

31 4.23. The NAO conducted an assessment of how it compares to the market in the use of data analytics, and concluded that, as a result of its rollout of data analytics for journals, it is making progress in closing gaps when compared to private-sector firms, where journal data analytics are widely used. Regarding the use of business process data analytics, the NAO is in its first year of use compared to mid-tier firms, where use is still limited, and Big 4 firms where data analytics are being used for selected business processes In addition to its centrally-developed data analytics tools, the NAO continues to use its core, off-theshelf data analytics tool, which has been in use for many years and is embedded in the financial audit working practices. This tool is provided directly to financial auditors and its use has generally, with the exception of the automated journal analytics tool, been limited to a number of key financial audit tasks, such as sampling and confirming the validity of trial balances. In adopting this decentralised approach, the NAO has not sought to apply this tool to its financial audit work as extensively as it possibly could. In particular, the use of this tool is not mandatory and other tools can be used instead The NAO conducted some market analysis to determine if an alternative solution exists to the current core analytics tool and approach to its use for financial audit. For example, alternative solutions for those who do not need to use the analytic tool s full capability. However, while alternatives do exist, the NAO assessed the short-term disruption and cost of transferring to an alternative tool would outweigh any benefit. The NAO plans to continue the use of this tool for the next three years and during this time determine if there are better delivery models and tools to meet the practice s, and Parliament s, changing needs in line with the increasingly digital environment in which it operates. Any different delivery models and tools will likely be introduced on a phased basis. Use of technology to enhance operational efficiency Audit workflow and working paper documentation The NAO s electronic audit work paper and workflow system has been in place for several years and was adapted specifically for the Financial Audit practice. This system is fit for purpose and provides a cost-effective solution. The NAO s monitoring of the market indicate that the other solutions available do not provide sufficient improvement in functionality that would warrant the significant cost of moving to a new solution in the near term Online bank confirmation service For 2016/17 an automated system for bank confirmations was implemented through the provider Confirmation.com, which automates the process of sending requests and receiving responses from banks and reduces the time audit staff need to spend on this activity Online collaboration tool The NAO has now rolled out a collaboration tool, Huddle, which was made available to all audit teams for 2016/17 work. This allows for a more secure means transferring and sharing information between NAO and clients than the historic reliance on attachments and memory sticks, and helps to manage information requests by tracking what has been requested; what has been received; and what is pending The NAO also conducted an assessment of its position compared to the market with regard to tools to increase the efficiency of the audit process. The NAO concluded that it has closed the gap with regard to tools for online confirmations and collaboration, where these are embedded for privatesector audit firms and the NAO is in its first year of office-wide use. Additionally, it concluded that it is on par with mid-tier audit firms in having an embedded electronic work paper and workflow 2017 Crowe Clark Whitehill 31

32 system. In the case of the Big 4, these multinational audit firms have developed ERP 9 audit tools for audit work papers and workflow which provide additional functionality such as: smart documentation which helps guide the auditor through the key steps of testing in each area of the audit; and management information which helps assess the extent to which teams are complying with the firm s methodology Other important ways in which technology is being used by the NAO to enhance the operational efficiency of the audit process include: The project and resource management module of the new system implemented as part of BIP is used for the planning and management of each audit. This module provides the master data for each audit and has enabled teams to better plan audits and track progress. The testing of payroll often involves time consuming calculation to estimate payroll costs using base data. A tool has been developed in-house by the NAO to automate this calculation. This tool was piloted for one client during 2016/17. The NAO s investment in audit technology The experience of large audit firms is that effective use of data analytics in the financial statement audit requires substantial investment in hardware, software, skills and quality control. The two framework partners interviewed as part of this VFM study, noted that, in their experience, first-year audits using data analytics resulted in significantly more investment, as well as disruption for both auditor and management, than subsequent audits. These firms also mentioned that the benefits they have experienced so far from a greater use of technology in the audit process have mainly been quality and insight rather than cost savings Table 4 summarises the NAO s plans to invest in audit technology, with these investment plans covering the following areas: Audit Software This continues existing spend in this area and includes the ongoing licensing of the NAO s electronic audit work paper software. IT Audit Skills This investment includes technical training to build the skills of IT audit specialists, and enable them to provide on-the-job support and transfer of knowledge to IT auditors working on audits. Data Analytics This includes investment in data analytics and visualisation tools and people skills to exploit these tools. The NAO is achieving the roll out of centrally-developed data analytics tool by refocussing existing resources, and envisages that no new people will need to be recruited to accomplish this. 9 Enterprise resource planning (ERP) software integrates different functions into one complete system to streamline processes and information across the entire organization Crowe Clark Whitehill 32

33 Enabling Automation This includes the cost of collaboration tool and bank confirmation software as well as the cost of assessing the increasing range of tools available to improve audit productivity. Opportunities exist, for example, in the areas of automated methods of performing standard repetitive tests and automated document creation. Table 4 NAO investment profile in audit technology between and Audit Technology ( 000s) Audit software IT audit skills Data analytics Enabling automation As % of 2016/2017 NAO audit fees ( 29,969k) As % of 2016/2017 spend on goods and services ( 22,765k) 0.67% 2.27% 2.00% 1.88% 0.88% 2.99% 2.64% 2.46% Source: NAO Technology enabled audit progress update presentation to Leadership Team on 25 April 2017 Comparison to other Supreme Audit Institutions Supreme audit institutions (SAIs) are national agencies responsible for auditing government revenue and spending. In different countries their legal mandates, reporting relationships, and effectiveness vary, reflecting different governance systems and government policies With regard to SAI data analytics benchmarking, comparator data in this field is difficult to obtain and would need to be interpreted in the context of the different SAI environments and remits. It was thus not possible to obtain comparator data which provides meaningful insight within the timeframe of this VFM study. However, it is evident from the NAO s extensive engagement with SAIs on data analytics that the NAO is taking a lead role in sharing knowledge in this area internationally. Our impression based on material we have seen is that the NAO is leading rather than lagging in the use of IT when compared to other SAIs Crowe Clark Whitehill 33

34 Knowledge and skills transfer from framework partners The invitation to tender documents the NAO sent to its framework partners specify that one of the key objectives of the framework contract is to help build the skills of the NAO, and mentions that this is to be achieved by enhancing the capability of the NAO by skills and knowledge transfer During interviews as part of this VFM study it was confirmed that there is regular interaction between the NAO and the framework partners regarding the latest developments in data analytics in the financial audit industry. There was a perception, however, that there is an opportunity to get greater skills transfer from framework partners It was mentioned by the framework partners that in the past there have been examples of joint working in the form co-sourcing and secondments that have enabled the transfer of skills to NAO staff. One of the framework partners, who is conducting IT audit work on an NAO client, suggested that 2 or 3 NAO IT auditors could join the firm s IT audit team for the duration of the audit in order to gain on-the-job training and skills transfer Crowe Clark Whitehill 34

35 Use of IT in delivery of VFM products 5.1. The NAO s Value for Money (VFM) practice produces studies that evaluate the value for money of public spending and include recommendations to help government improve public services. The VFM practice aims to bring the best from across the NAO to every VFM product, and has the ambition to stay ahead of those it audits by having expertise in government s areas of priority and delivering cross-cutting insights and expertise, leading to lasting improvements. See Figure 3 for VFM practice vision. Figure 3 VFM Practice Vision - The Best of the NAO Source: NAO Senior Management Conference 2016 VFM practice presentation 5.2. In common with the Financial Audit practice, the project and resource management module of the new system implemented as part of BIP is used for audit project management and provides a clear view of the schedule and progress of each audit. This system is also used for planning and resource allocation by both practices The NAO s electronic audit work paper solution was designed for the financial audit process and has been perceived by the VFM practice to be not best suited for VFM workflow. A SharePoint application, NAO Tank, is instead being used to manage VFM workflow and team collaboration. NAO Tank was, however, not designed to be an audit documentation storage system and VFM teams are required to store key audit documents in the NAO s electronic audit work paper solution on completion of each assignment, which serves as the document repository for the VFM practice. This results in two-system solution for VFM workflow management (NAO Tank) and documentation storage (NAO s electronic audit work paper solution) The NAO s electronic audit work paper solution can be described as a closed system which means there is limited potential for automated integration with other NAO systems. This results in a small amount of additional effort needed to manually transfer documentation into this electronic audit work paper and also rekey some basic audit project information that is already in project and resource management module of the new system implemented as part of BIP This two-system solution is based on existing investments in technology and is, in our view, cost effective and provides an acceptable level of functionality and audit trail. The NAO, however, 2017 Crowe Clark Whitehill 35

36 recognises the need to review to what extent the current solution could be replaced with more integrated, open systems The compliance rates for VFM teams completing the storage of key documentation in the electronic work paper system have historically been low as there was no formal requirement for teams to do this. Compliance rates have now, however, increased considerably due to additional training and training and the inclusion of a requirement in the audit close-down checklist for teams to complete the storage of key documentation. For studies published in the second half of , the overall compliance rate was 69% with regard to ensuring all mandatory items had been stored in electronic work paper system The VFM practice uses a large number of tools to support its work. We listed of these, along with descriptions, in Table 5. In particular, the development programme for the VFM practice includes an ongoing project called Product Knowledge which aims to improve how the NAO collects and reuses its product assets and resources. Product Knowledge consists of a site on the NAO s SharePoint platform named One Stop Shop, which brings together information that people previously had to look for in separate places. Table 5 Tools used by VFM Practice VFM tool One Stop Shop VFM guidance portal Data service Back-catalogue analyser Analysis knowledge tool Framework site Analytics site Description of tool A searchable collection of documents and insights about NAO projects. For VFM it provides information across the project cycle including: Concept papers from the study proposal stage covering the rationale, strategic fit, hypotheses, main study questions and the risks and opportunities the study presents; Proof of Concept slide packs prepared during fieldwork and analysis setting out initial findings; Outputs produced at publication including NAO reports and supplementary material such as methodologies, podcasts, and infographics; and links to corresponding PAC reports and Treasury Minutes; and post-project lessons learned about strategic issues, clients and project delivery. Also covers investigations and some financial audits. Provides easily accessible and up-to-date guidance for teams to use. Provides easy access to a wide variety of data already published by government departments. Links datasets together to increase insight. Bespoke tool for searching and retrieving content from published VFM reports. Details of the analysis used and lessons learnt in previous studies. Provides a back catalogue of re-usable NAO-developed frameworks with which to approach assurance work. Provides a small number of re-usable NAO-developed frameworks, likely to be relevant to about 80% of NAO assurance work and enhanced by ongoing contribution of relevant NAO findings to support comparative judgements Crowe Clark Whitehill 36

37 Recommendations tracking StyleWriter Tracking of all recommendations made in NAO reports. Software to ensure that VFM reports are drafted in a clear, concise and readable style, Source: NAO Senior Management Conference 2016 VFM practice presentation 5.8. A number of the tools used by the VFM practice have been developed with the help of the Methods, Economics and Statistics Hub (MESH), which is one of the NAO s communities of practice. In particular, MESH has used its strong IT and technology skills to help develop the Data Service, Back-catalogue analyser, and Analysis knowledge tool shown in Table 5. MESH has a total of nine disciplines many of which rely heavily on technology, such as data analytics, modelling and mapping. According to data provided by MESH, a total of 24 of the 42 major VFM products published between June 2015 and June 2016 had substantial MESH input. Notably, MESH leads the NAO s increasingly strong analytical capability to scrutinise and analyse big and complex data sets, and people from MESH represent the NAO at INTOSAI and EUROSAI on the subjects of data analytics and big data. In particular, other SAIs have been keen to replicate the NAO s Data Service Crowe Clark Whitehill 37

38 Use of IT in knowledge management Why knowledge management is important to the NAO 6.1. The NAO is a collaborative organisation and developing and applying knowledge is a key part of the NAO s strategy (see Figure 4). Knowledge management enables collaboration and provides the means to find expertise from across the organisation which is increasingly important in helping the NAO make sense of those it audits, and how it has influence and impact on them. The NAO gathers much knowledge and has developed pockets of deep expertise. We saw many good examples from across the NAO of how IT is being used to manage knowledge and how knowledge management is incorporated into the organisation s practices. Figure 4 The NAO s strategic objective and enablers The NAO s overall strategic objective is accountability and public service improvement which is enabled through developing and applying knowledge, increasing influence and delivering high performance Source: NAO Strategy 2016/17 to 2018/19 The NAO manages knowledge in many ways Knowledge and influence (K&I) directors champion knowledge management 6.2. A director is designated responsibility to champion K&I for each cluster. Their activities to champion knowledge management include promoting and encouraging people to use tools and techniques to support knowledge, encouraging small-scale sharing events for show and tell, and representing 2017 Crowe Clark Whitehill 38

39 user needs in the further development of business processes and technology platforms. The K&I directors meet periodically to share experiences and learn about organisational K&I initiatives, for example at their meeting in May 2017 they received a demonstration of the new One Stop Shop tool for sharing VFM product knowledge. The way NAO is organised promotes knowledge management 6.3. The NAO is focused on building knowledge and expertise on priority issues shared between departments and achieves this through its clusters. Each cluster covers departments facing similar strategic issues. For example, financial and service sustainability; oversight and intervention; growth; or accessible and integrated services for people. This structure maximises the NAO s ability to share knowledge across audit teams and build subject matter expertise. It also enables comparative analysis to be carried out and encourages collaborative working While the clusters are organised around clients, there are also communities of practice (CoP), which are decentralised networks that work across the NAO that are organised around specialist knowledge, skills and capabilities relevant to all clusters. As mentioned previously, the Methods, Economics and Statistics Hub (MESH) is one of the NAO s communities of practice, which has a total of nine disciplines including modelling, mapping and data analytics. MESH provides training and practical opportunities for NAO people to develop specialist skills within one or more disciplines Outside MESH the NAO has five CoP that are based on subject expertise relevant to operational delivery. These are: Commercial and Contracting; Corporate Finance; Digital; Operations and Process Management; and Project and Programme Management Delivery. These CoP provide targeted expertise to audit teams engaged in client-facing work or VFM studies. By encouraging broad membership and participation in the communities, the CoP promote the development and sharing of specialist knowledge and expertise amongst NAO people. Knowledge management behaviours are encouraged 6.6. The Way We Work is a development programme consisting of three modules designed to promote a culture of collaboration and empowerment. Module 1 has been completed, and according to data from the NAO Learning and Development team, a total of 34 courses were run and 712 people trained. The aim of this module was to help everyone at the NAO optimise the way they develop, share and apply their knowledge This programme helps support the achievement of specific knowledge and collaboration behaviours necessary to underpin BIP. These behaviours include involving and consulting colleagues across the NAO to access the right skills and knowledge, and to reusing the best work from across the NAO Regarding knowledge management, the desired outcome of the Way We Work programme is for NAO people to be more skilled in leveraging the NAO s cross-government insights and expertise through effective knowledge sharing, increased collaboration and comparative work. Furthermore, actively sharing knowledge more readily will build effective client facing teams, promoting best practice through enhanced collaboration. The NAO will benefit from this wider knowledge sharing and be able to apply this to its work Crowe Clark Whitehill 39

40 6.9. The NAO performance management system has also been significantly reengineered to reinforce the weighting placed on collaboration and knowledge sharing. VFM practice promotes knowledge management Knowledge is central to the VFM practice s vision (see Figure 3) and, in order to achieve this vision, the VFM development programme aims to embed four behaviours that are considered key to improving VFM work, which include to access best NAO skills and knowledge (see Figure 5). The behaviours are supported by six operating principles to translate them into concrete actions, which include routine reuse of best cross-office analytics and insights and cross-office collaboration as normal practice. Figure 5 Key behaviours and operating principles for VFM practice Source: NAO Senior Management Conference 2016 VFM practice presentation Financial audit practice promotes knowledge management The NAO s Financial Audit practice has a very structured process for knowledge management which follows a clear annual cycle. The NAO s financial audit is regulated through the Audit Quality Review team of the Financial Reporting Council and is required to have formal firm-wide procedures which set out how audits should be performed. The overall approach to financial audit work is set out in the NAO Financial Audit Manual which has been updated to reflect guidance on the use of IT audit and data analytics Crowe Clark Whitehill 40