Summary QRP Report INSIDE THIS REPORT

Transcription

1 Quality Review Process A summary of the results of ACiW s quality review of all suppliers 2002/2003 audit appointments and inspection work INSIDE THIS REPORT PAGES 2-11 Foreword Introduction Summary of performance Future developments Quality assurance Quality control Output and impact Reference: Author: 953A2004 Final Simon Edge Head of APA Date: September 2004

2 Foreword 1. The 2002/2003 year was the first in which the Audit Commission in Wales (ACiW) took full responsibility for managing the quality review of the audit and inspection work preformed in Wales. In managing this task, ACiW retained the framework of the Audit Commission s Quality Review Process (QRP). The QRP provided a sound basis of approach for the work and, through close co-operation with QRP colleagues in England, resulted in minimal additional work for suppliers who operate framework contracts in both England and Wales ACiW took an early decision to include inspection work within the QRP. In the lead up to the creation of the Wales Audit Office (expected on 1 April 2005) and the development of quality assurance and review approaches for the new body, we also needed to accommodate the specific circumstances of: ACiW s being an in-house audit and inspection supplier. The QRP work performed and reported here is the responsibility of the ACiW s client or regime regulatory function which is managerially and operationally independent of ACiW s in-house audit and inspection supplier; PwC a longstanding audit supplier in Wales; and KPMG and Deloitte these two firms did not have appointments in Wales during 2002/2003. However, they were undertaking 2003/2004 audit work during the review period which presented an opportunity for early discussions on quality issues. These factors caused us to focus the QRP into the following areas: quality assurance an assessment of the network of controls and activities operated by suppliers to ensure that the ACiW s requirements are met; Page 2 of 11

3 4. quality control the post event review of work to ensure that the ACiW s requirements were met in practice and, as a consequence, the effectiveness of the previously assessed quality assurance measures; and output and impact a more subjective assessment of whether the audit and inspection work reviewed will have had a positive impact which combines satisfaction surveys, performance monitoring and product reading. This includes identifying and sharing good practices amongst suppliers. Some of the work in these three areas is complete, some is ongoing. This report presents a public summary of where the ACiW s public audit and inspection regime is in terms of quality and where we intend to improve as we move forward to the creation of the Wales Audit Office. Introduction In meeting their responsibilities and in exercising their powers auditors and inspectors must carry out their work in line with the Code of Audit Practice (the Code), and ACiW s approved inspection methodologies. In performing their work ACiW expects auditors and inspectors to make a clear contribution towards improving public services maximising the benefit to the public and other stakeholders and minimising the costs of regulation and compliance. In Wales during 2002/2003 we had appointed external auditors from the ACiW s own staff and one private firm of auditors (collectively called audit suppliers) and inspectors, who are exclusively ACiW staff, working at 56 principal audited or audited and inspected bodies. The current picture post NHS reconfiguration and the introduction of Local Health Boards, and the expansion in the ACiW supplier base is: Supplier Local Government* (audited and inspected bodies) NHS (audited bodies) ACiW Deloitte KPMG PwC Total * including police, fire, probation and national parks. Total (audited, and audited and inspected bodies) As with the ACiW s own appointed auditors, firms act under the ACiW s regime but independently of the ACiW when discharging their responsibilities as appointed auditors. ACiW worked to the performance targets set by the Audit Commission for suppliers for 2002/2003, amended to include inspection: achieving an overall satisfaction level of 75% from audited and inspected bodies; Page 3 of 11

4 achieving 95% for meeting other ACiW requirements that can be numerically quantified (eg, timeliness of reporting, compliance with Annual Audit Letter requirements); achieving high quality performance for the QRP elements that do not lend themselves to a numerical performance scoring scheme (eg, quality assurance, and quality control) for which narrative assessment is given; and achieving 100% timeliness of issuing audit opinions and Annual Audit Letters. Summary of performance 10. The summary of performance arising from the QRP work undertaken this year is: ACiW QRP element Quality assurance (QA) Quality control Output and impact All suppliers performance in Wales All suppliers had QA systems and procedures in place that ACiW could place reliance on as part of the QRP. We identified 28 instances where ACiW requirements had not been fully met or complied with at the seven (a 10% sample) audits reviewed. The level of compliance in Wales compares favourably with the previous year s results and is indicative of the thoroughness of suppliers review approaches. New suppliers have given positive assurances of the procedures they are applying to ensure high quality work is delivered. Satisfaction: The levels of satisfaction reported in health and local government, and in relation to inspection / relationship management which meet or exceed the 75% target are very encouraging. There is scope for auditors, inspectors and relationship managers to improve bodies satisfaction with assessment and assurance work in both health and local government. Timeliness of audit opinions: All local government opinions that could be, were issued by the due deadlines. Four opinions were outstanding for legitimate reasons involving technical issues and or questions and objections to the accounts. All opinions in health that could be, were issued by the due deadlines. Five opinions were outstanding for legitimate reasons involving technical issues and the unavailability of officers. Annual Audit Letters: The quality of letters was generally good but there was scope to increase the clarity and impact with which issues are raised for members attention. We found 10 instances where requirements were not, or not fully, met giving an overall level of 98% compliance. 11. Individual and collective issues of good practice and improvement opportunities arising from this year s QRP have been shared with auditors and relationship managers for their consideration and action. Page 4 of 11

5 Future developments New quality assurance and review arrangements will need to be in place by 1 April 2005 ready for the start of the Wales Audit Office. The ACiW and National Audit Office will work together to develop appropriate arrangements in conjunction with audit suppliers. If you would like to give us your views on any issues raised in this report, please Simon Edge, Head of Audit Policy and Appointments (Wales) at s-edge@auditcommission.gov.uk or write to him at Deri House, 2-4 Park Grove, Cardiff, CF10 3PA. Page 5 of 11

6 Quality assurance Each year we carry out an assessment of the extent to which we can rely on suppliers internal systems of quality assurance (QA) as part of the quality control review (QCR) programme. This year we included the application of supplier s QA arrangements to the specific requirements of the ACiW s regime. Most particularly this included the Wales Programme for Improvement, and the impact of Wales Assembly Government requirements. Our QA work covered all suppliers audit and inspection work under the following headings: Risk management: QA arrangements: overall risk; ethical considerations; resources; relationships with other parties; training and development; assignment of personnel; consultation; independent review; technical matters; staff performance and appraisal; QA arrangements: engagement level Administration: direction, supervision and review; client records; other commission requirements; non code work; complaints; welsh language scheme; Internal QCR arrangements: leadership & responsibilities; supplier internal QCRs; Specialist Areas: all other QA activities 16. Our reviews showed that all suppliers had QA systems and procedures in place that ACiW could place reliance on as part of the QRP. The results of the QA review were used to help us focus and tailor the remainder of our QRP work. Page 6 of 11

7 Quality control 17. The circumstances of ACiW s suppliers were significantly different during 2002/2003 and the QCR work was tailored to reflect this: Supplier Circumstances QCR work ACiW Deloitte KPMG PwC 2002/2003 audit and inspection work undertaken. No 2002/2003 audit work undertaken (but the firm was starting work on 2003/2004 audit work). No 2002/2003 audit work undertaken (but the firm was starting work on 2003/2004 audit work). 2002/2003 audit work undertaken. Full review of 2002/2003 audit and inspection work. Early opportunity taken to hold preliminary discussions and review of arrangements for planning and performing 2003/2004 audit work. Early opportunity taken to hold preliminary discussions and review of arrangements for planning and performing 2003/2004 audit work. Full review of 2002/2003 audit work QRP meetings and discussions were held with Deloitte and KPMG. Both firms provided positive assurances of the procedures they are applying to ensure high quality work is produced. The efficacy of the arrangements will be reviewed in the coming year. ACiW s risk-based framework for quality review allows that where a supplier: has quality assurance arrangements that can be relied upon; has complete and comprehensive internal quality control processes; and is willing to give ACiW s quality review team full access to its internal quality control papers, then ACiW can review and seek to place reliance on the supplier s internal quality control review (iqcr) work in place of undertaking its own external quality control reviews. To be considered complete and comprehensive, an iqcr process needs to: cover all Code of Audit Practice areas each year; review the work of all appointed auditors (including inspection work performed at local government bodies where appropriate) over a reasonable timescale; and be undertaken by appropriately qualified individuals who are independent of the appointed auditor (and where appropriate inspection work) being reviewed. The ACiW s in-house audit supplier and PwC both met the above requirements. We reviewed seven audits in whole or part, representing 10 percent of principal audited bodies in Wales. We found that the ACiW can rely on both suppliers iqcr process, which were well managed, comprehensive and thorough. ACiW s in-house audit supplier had made particular progress in implementing a process capable of reaching this standard, and by including inspection work within its quality review activities. Page 7 of 11

8 23. We use the term compliant for assessing compliance against Statements of Auditing Standards (SASs) for the financial statements aspects of the audit, and met for other ACiW audit and inspection requirements (Code, Letter of Guidance and other requirements). In addition to reporting good practices and improvement opportunities we use three assessment categories: Category Fully compliant/met Not fully compliant/met Not compliant/met Assessment All ACiW and SAS requirements fully met/complied with. The issues arising are not sufficiently serious to undermine the opinion or a Code objective/aciw requirement. The issues arising are sufficiently serious to undermine the opinion or a Code objective/aciw requirement Our review of suppliers internal quality control reviews identified 28 instances where ACiW requirements/sass had not been fully met/compliant with at the seven audits reviewed. This overall level of compliance in Wales compares favourably with the previous year s results where at two audits the QRP found 13 instances of work not fully meeting the Commission s requirements. This picture of performance is indicative of the thoroughness of suppliers review approaches and the continual raising of the bar particularly in relation to the documentation and evidencing standards auditors and inspectors should apply. It should also be seen in the context of the openness afforded to the QRP reviewers which is appreciated by the ACiW. Instances of good practice and improvement opportunities this year were: Good practices Common areas of good practice were: maintaining a robust review approach to identifying where requirements had not been fully met; comprehensive review methodology covering the suppliers procedures, professional standards and ACiW requirements; and integration of the internal quality review results into the development of the suppliers audit and or inspection approach and staff training; good review documentation; comprehensive training for internal quality control reviewers; sound management arrangements for managing the internal quality control review process. Improvement opportunities The main areas with scope for improvement this year were: including clearer qualitative assessment comments in review papers to express reviewers compliance conclusions; generally ensuring that reviewers documentation of issues is sufficient for later external review purposes; document the root cause of matters arising from the review work; auditors and inspectors need to remain constantly alert to the need for them to adequately document and evidence their work (Standing Guidance and SAS 400); and similarly, auditors need to remain constantly alert to being able to demonstrate SAS compliance. Page 8 of 11

9 Output and impact 27. Output and impact covers a more subjective assessment of whether the audit and inspection work reviewed will have had a positive effect on improving public services which combines satisfaction surveys, performance monitoring and product reading. Audited body satisfaction Health Audited body satisfaction questionnaires were sent to all 37 NHS bodies in Wales. A total of 11 replies were received representing a 30% response rate. Many representatives of the 22 recently created Local Health Boards thanked us for seeking their views about the quality of the work performed by their auditor, but declined the opportunity to provide feedback as relationships and performance were still in their formative stages. The following results must therefore be seen in this context: Local Government Statement Result Percentage Overall, the work of the ACiW focuses on the Neutral to strongly agreed 80% right areas. Overall, the ACiW appointed auditor provides an accurate assessment of the strengths and weaknesses of my organisation's financial position. The ACiW appointed auditor's work provides robust challenge to the way in which my organisation delivers services. Overall, how satisfied are you with the assessment and assurance work that ACiW's appointed auditor provides. Overall, I am satisfied with the service that ACiW's appointed auditors provide my organisation. Neutral to strongly agreed 90% Neutral to strongly agreed 63% Satisfied or very satisfied 63% Neutral to strongly agreed 90% Audited body satisfaction questionnaires were sent to the unitary authorities in Wales. We received a total of 62 replies from 13 bodies representing a very high, 77%, response rate. For the first time, this year we surveyed bodies views about audit, inspection, and the role of relationship management which became operational during 2002/2003. Satisfaction was generally high and the relationship manager role was seen as a positive development even if it was felt to be too early to fully judge the impact of the role. Page 9 of 11

10 Statement The amount of work carried out by the ACiW appointed auditor at the body is justified, given their jointly agreed Wales Programme for Improvement/ risk assessment. Allowing for the fact that the Relationship Manager (RM) role has been in existence for less than a year, it has already helped to effect change and improvement. The following provide robust challenge: inspection work; audit work; and role of the RM. The role of the RM helped to join up external regulation within local government Overall, how satisfied are you with the assessment and assurance work that ACiW's appointed auditor provides. Overall, satisfaction with the service provided by: audit; and inspection/rm. Result Neutral to strongly agreed 78% Neutral to strongly agreed 85% Neutral to strongly agreed 65% 63% 52% Neutral to strongly agreed 69% Satisfied or very satisfied 66% Neutral to strongly agreed 82% 85% 32. Clearly ACiW, its auditors, inspectors and relationship managers have further work to do to improve the level of challenge provided and, to improve bodies general satisfaction with assessment and assurance work in both health and local government. Timeliness 33. Four local government opinions were issued after their due deadlines. In all cases it was appropriate for the auditor to delay issuing the opinion due to the need to resolve technical issues and or questions and objections to the accounts. In the NHS, there were five opinions issued after their respective deadlines. Again, it was appropriate for the auditor to delay issuing the opinions due to issues concerning NHS reconfiguration, and one instance where the auditor was obliged to await the return of officers who had not signed the accounts before taking annual leave. Page 10 of 11

11 Annual Audit Letter reading We read all Annual Audit Letters issued to principal audited and inspected bodies in 2002/2003. We assessed each letter s compliance with ACiW requirements and, by trying to put ourselves in the position of the recipient, whether it was a good read that contained clear messages that would have an impact. Looking across nine areas of the ACiW s requirements for each of the letters, we found three instances where we felt letters did not communicate effectively or were not clearly addressed as being issued to members, and seven instances where letters were not clearly dated as being issued by the deadline on 31 December. This equates to a level of 98% compliance with ACiW requirements. The quality of letters was generally good but there was scope for some letters to be considerably shortened to increase the clarity and impact with which issues are raised for members attention. Some of the short-format letters read could conversely have been improved by ensuring that issues are not lost or diluted by the summarisation process of producing a short-format letter. Reviewers made the following general comments: What was done well Compliance with ACiW requirements: high levels of compliance with Code of Audit Practice requirements; Impact: key issues were identified and stated clearly; positive statements were used; letters were written clearly, easy to read, and generally well-presented; and good use was made of plain language, tables, diagrams and charts. What could be improved Content: all letters should refer to the availability of Welsh language translation versions; refer to qualifications in the accounts up front as a key message; state clearly any consequences of inaction to the key issues raised; clearly show the fundamental issues affecting the body e.g. provide more information on financial standing where this is linked to a key issue - sharpen key issues and where appropriate pull together messages for successor bodies; provide a follow up on past recommendations; explicitly state that the letter covers all work done since the previous one; explicitly refer to the letter s issue to members, addressing it to them; date the letter s issue accurately; and consider giving negative assurance nothing has come to our attention where appropriate. Format: standard letter templates should be tailored to small or single purpose audited and inspected bodies circumstances; and avoid blank numbered pages at the start of letters. Page 11 of 11