Performing a Successful Audit. Fundamentals of Auditing ERO Compliance Audit Process Jim Hughes Manager, Audit Assurance and Oversight

Transcription

1 Performing a Successful Audit Fundamentals of Auditing ERO Compliance Audit Process Jim Hughes Manager, Audit Assurance and Oversight

2 Objectives At the end of this session, participants will be able to: Define the ERO Compliance Audit and its Goals (Quality, Consistency & Credibility) Describe authority and responsibilities in conducting a compliance audit Describe your role as a Lead Auditor Demonstrate basic information gathering methods; list the tools for auditors Explain how the ERO compliance audit process is conducted Describe terms: GAGAS, professional judgment, and evidence: relevant, valid, reliable, stacking, sufficient o Generally accepted government auditing standards

3 Definition Compliance Audit: A systematic, objective review and examination of records and activities to determine whether a Registered Entity meets the requirements of applicable Reliability Standards (NERC ROP). On-Site Off-Site

4 Goals for Compliance Audits Quality Government auditing standards (GAGAS) (Chapters 1-3, 7&8) Consistency Auditor tools Credibility Reasonable assurance (Relevant, sufficient, appropriate evidence, and professional judgment)

5 Areas Targeted for Consistency Regional Entity program implementation Compliance audit process Auditor tools Compliance measurement of the Reliability Standard requirements Evidence required to substantiate findings Guidance provided in RSAWs and CANs We want to avoid outliers

6 Auditors Creed Auditors will perform with the highest ethical standards of independence, integrity, and accountability, and will maintain objectivity from external and internal influences when using their professional judgment in all aspects as they perform their responsibilities.

7 Document Authority Hierarchy 1 Energy Policy Act of Section 215 FERC Implementing Rule 18CFR39 (Order 672) 2 NERC Rules of Procedure (ROP) NERC Compliance Monitoring & Enforcement Program (CMEP) GAO Standards/ GAGAS 3 Auditor Practices, Procedures, and Tools FERC Guidance orders CANs NERC Directives 4 Reliability Standard Audit Worksheet (RSAW) Compliance Auditor Manual Feedback forms Checklist Questionnaires Intro and Exit Presentations Compliance Audit Survey Audit Report Template Work History/Conflict of Interest

8 Compliance Monitoring and Enforcement Program (CMEP)

9 Compliance Monitoring and Enforcement Program (CMEP) CMEP-Appendix 4C: Policy document that implements the Rules of Procedure (ROP) Based on GAGAS Monitor, assess, and enforce compliance with Reliability Standards Establishes framework for regional compliance program implementation o ROP has it basis in the Generally accepted government auditing standards

10 Other Resources: FERC guidance order on compliance audits No. AD o FERC staff provided guidance on training, audit requests for information, pre-audit preparation, documentation to show compliance, Assessing a compliance culture & other Compliance Process Directives & Bulletins 22 Other Resources

11 Team Leader Role Facilitator Leverage the technical expertise of your team Setting expectations Entity, Team, Observers Team Leader Ensure consistency among the team Maintain team focus When to call caucus Stakeholder management Manage Conflict Advocacy

12 Use of Auditor Tools Audit Preparation Audit Execution Audit Follow Up Compliance Audit & Internal Compliance Culture Surveys Audit Checklist Compliance Audit Process Feedback Forms Questionnaire RSAW RSAW Audit Checklist Reliability Standard Audit Worksheet (RSAW) Work History, Conflict of Interest and Non Disclosure Agreement Intro Presentation Template (Regional) Exit Briefing Presentation Template also known as findings (Regional) Audit Checklist Audit Report Template

13 Audit Checklist 13

14 NERC Rules of Procedure Require: Initial determination (possible violation) NERC and Regional Entity compliance staff have the authority and responsibility to make initial determination of potential compliance or noncompliance. Confidentiality (Section 1500 of ROP) audit team members shall maintain the confidentiality of information.

15 NERC Non-disclosure & Confidentiality Agreements Refer to NERC ROP Section

16 Audit Preparation Plan the Audit Field Work - Conduct the Audit Report - Wrap Up and Follow Up

17 Expectations set before the onsite audit For your team Independence and Objectivity Team Member Roles Dress code Behavior Confidentiality/Evidence/Notes Observers

18 For the entity Data collection authority Evidence presentation expectations Logistic expectations (meeting rooms, lunch etc.) Behavior Methodology to be used SME expectations and data presentations Reliability Standard Organization

19 Plan the Audit Write an Audit Plan Milestones ICP o Compliance History o Events o 13 Questions Understand the Entity and may need to expand scope Functions Applicable Standards o AML and Annual Plan

20 Manage Audit Scope Risk! FERC approved NERC Reliability Standards 102 approved by FERC for 2011 The FERC approved NERC Reliability Standards are considered by FERC to be good industry practice (111) o The standards are the audit teams measure for compliance Rick Based Scope

21 Actively monitored standards Audit Scope - AML Annually select a subset of the NERC reliability standards and requirements 39 actively monitored in 2011, 37 in 2012 NERC and Regions have input Represents the minimum auditable Based on Risk to the reliability of the BPS Entities must comply with all standards (see FERC Guidance order AD section C, item 13)

22 Practical Exercise Actively Monitored List

23 AML Practical Exercise You are leading a future audit & need to identify minimum scope by Standard and Requirement Break up into 4 teams and do the following: Each team has at least one internet access Each team select a function to identify minimal audit scope RC BA TOP GOP/GO o Each team take ~15 minutes to identify the minimum audit scope and each report out the scope and lessons learned

24 AML Practical Exercise

25 Internal Compliance Program Assessing Risk for Audit Scope

26 FERC Orders Policy Statement on Enforcement Docket No. PL , 113 FERC 61,068 (October 20, 2005) Revised Policy Statement on Enforcement Docket No. PL , 123 FERC 61,156 (May 18, 2008) Policy Statement on Compliance Docket No. PL ,125 FERC 61,058 (October 16, 2008) Policy Statement on Penalty Guidelines Docket No. PL , 130 FERC 61,220 (March 18, 2010), suspended on April 15, 2010 Revised Policy Statement on Penalty Guidelines Docket No. PL ,132 FERC 61,216 (October 17, 2010)

27 Some inconsistencies between Regions Timing Voluntary On-site vs. Off-site audits Evaluating an ICP All Regions moving toward evaluations of ICPs based on the FERC 13 questions provided in the 2005 orders Is there alignment between the ICP, the registered entity s organization chart and responses to the 13 questions?

28 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) Does the company have an established, formal program for internal compliance? Signature, Title, Date Is it signed by a senior officer? Does it grant authority and responsibilities for the ICP? Does the document have version control? Is it well documented and widely disseminated within the company? Dissemination should be throughout the organization, not just compliance o Common Area For Improvement (AFI) lack of dissemination throughout entire organization, such as HR, Legal, Executive, Accounting, etc. Does the ICP identify when, where and to whom it was disseminated?

29 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) Is the program supervised by an officer or other high-ranking official? The oversight position should be clearly identified, along with the responsibilities and requirements of the Compliance Manager An Organizational Chart should identify clear organizational responsibilities Direct access to the President, CEO, or Board should be outlined in the ICP, and the Organizational Chart A common AFI is inconsistency of the named oversight position or titles in the Organizational Chart

30 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) Does the compliance official report to or have independent access to the chief executive officer and/or the board of directors? The ICP needs to specifically identify how and when the oversight position has access to the CEO/Board A common AFI is that there is no independent access to the CEO/Board, or it is not identified on the Organizational Chart or stated in the ICP

31 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) 5. Is the program operated and managed so as to be independent? Independence from day to day decisions over NERC Reliability is needed The ICP manager should not have responsibility over any direct implementation of a Standard How to achieve independence: Utilize other departments such as Human Resources, Legal, or internal Auditing to operate or manage the Program Use other outside sources such as other related facilities or plants

32 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) Are there sufficient resources dedicated to the compliance program? An independent compliance officer is necessary to be considered fully staffed Is there enough resource for the entity based on its size? Budget should identify how much of the resource is dedicated to the ICP and funding should be managed independently A common AFI is that funding/budget is not mentioned in the ICP

33 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) Is compliance fully supported by senior management? For example, is senior management actively involved in compliance efforts and do company policies regarding compensation, promotion, and disciplinary action take into account the relevant employees compliance with Commission regulations and the reporting of any violations? The Compliance Program needs a clear description of how Senior Management is involved and must discuss how corrective actions are ensured. o Example of participation with Senior Management: Meetings on a monthly or other specified basis Agenda topics such as new standards, Self Certifications, Audits, Internal Assessment status and results

34 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) How frequently does the company review and modify the compliance program? ICP should clearly identify how frequently the review takes place o An annual review basis may not be sufficient based upon changing of Standards o Semi-Annual or shorter basis is best How frequently is training provided to all relevant employees? The ICP should document how and when training takes place o Initial, refresher, and ongoing training, use of real world examples

35 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) Is the training sufficiently detailed and thorough to instill an understanding of relevant rules and the importance of compliance? Detailed training should be provided to employees with direct responsibility for Reliability Standards A common AFI is that high level awareness training is missing

36 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) In addition to training, does the company have an ongoing process for auditing compliance with Commission regulations? The ICP should identify the periodicity, by whom, and how self assessments are performed Self assessments or audits should take place on an off cycle from regular audits or self Certifications Keep evidence to show compliance during off cycle internal reviews

37 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) How has the company responded to prior wrongdoing? Did it take disciplinary action against employees involved in violations? When misconduct occurs, is it a repeat of the same offense or misconduct of a different nature? The ICP should include a direct tie from failure to follow Reliability Standards to disciplinary action, up to and including termination A common AFI is the ICP lacks direct mention of discipline

38 Policy Statement on Enforcement Docket No. PL ,125 FERC 61,058 (October 20, 2005) Does the company adopt and ensure enforcement of new and more effective internal controls and procedures to prevent a recurrence of misconduct? The ICP documentation should include a description of self assessment and should have enough steps or controls to keep the violation from occurring again Once weakness is found it should be reported to top management (feedback loop) to work on how to strengthen the weakness

39 Final preparations for the audit Documents sent by Lead Auditor prior to on-site audit: Check your checklist On-site audit agenda with team assignments Completed surveys /questionnaires Relevant procedures/documents from audited entity RSAWs for applicable standards Pre-Audit Meetings Authority document All team members must sign appropriate Confidentiality Agreements and Code of Conduct statements prior to on-site audit

40 Pre-Audit Preparation Final preparations for the audit Assignments to the Audit Team Members Review Evidence / RSAWs 3 Year ERO Assessment ( 118) Develop Questions Identification of Issues Hold team meetings to ensure alignment Know the Entity s ICP o PVs, Events, review survey 13 Qs o Open Mitigation Plans

41 Conduct Audit Plan the Audit Field Work - Conduct the Audit Report - Wrap Up and Follow Up

42 Team introduction and roles Audit scope (including the standards) Discuss audit methodology Agenda Authority to request and retain data Introductory Presentation Express that scope may increase based upon findings Confidential information Expectations Discuss feedback forms

43 Practical Exercise FAC-011

44 Practical Exercise Your team is auditing an RC where you are reviewing FAC-011 System Operating Limits Methodology for the Operations Horizon Audit window is Jan 1, 2009 to present Prepare to discuss with the group Perform an Impact Assessment Determine Applicable Effective dates Determination of impact to audit Present what revision(s) is applicable Effective Date? 44

45 Mandatory Effective Date 45

46 Reliability Standards Audit Worksheet (RSAW) Used to organize and document evidence gathered during an audit Evidence put into report findings table Provides clarity in regards to the intention of the Standards and provides an agreed upon (consensus) audit approach for each requirement Revised to include FERC Order 693 language & Questionnaire 22

47 Gather Evidence- Just the Facts Substantiate your findings to support: Compliance/possible violation Credibility Scrutiny Checks and balances How to substantiate your findings Appropriateness of evidence (quality) Sufficiency (quantity) Testimonial evidence (interviews) Observation Reasonable assurance May need to increase Scope or Suggestion

48 Appropriateness of Evidence Relevance to requirements Validity (sound reasoning) Document Quality: Document title, definition Revision level, date Effective date Authorizing signatures Registered Entity must demonstrate compliance No Evidence = Possible Violation Clearly document the team s conclusion in the RSAW and the evidence that a reasonable individual could come to a similar conclusion

49 Practical Exercise RSAW

50 Directions Scope is PER Requirement R3 Complete as if on-site Complete the note section o Write the notes as if you were going to copy into the final report o Assume if listed, it is verified as fact Task: if the hypothetical entity in compliance?

51 Professional Judgment Professional judgment represents the application of the collective knowledge, skills, and experiences of all the personnel involved with an assignment Knowledge Skills (SME) Experiences Good faith application Integrity

52 Evidence - Reasonable Assurance Sufficient, appropriate evidence Relevant, Valid and Reliable Is there enough to persuade someone that the findings are reasonable? Professional judgment Will the team s findings withstand scrutiny by others?

53 What you need to know GAGAS for lead auditors Comply with GAGAS when performing compliance audits Examples: Work History and Conflict of Interest o Independence Adequately plan o Prepare audit plan Reasonable assurance o Sufficient, appropriate evidence Professional judgment o Document significant decisions

54 The four Bullets Positives Suggestions or Areas For Improvement Open Issues Findings (Work In Progress) Transparency! Golden Rule - Rules End of Day De-Brief

55 Exit Briefing Presentation Summation of audit findings Suggestions/concerns regarding audit process Authority to retain evidence Report process Timeline Draft Critical infrastructure (confidentiality) Right to due process Provide entity with opportunity to ask questions Reminder feedback forms

56 Audit Feedback Forms Used for the audited entity and the audit team members Based on performance indicators for onsite audits Scorecard for onsite audits and lead auditors Designed to improve the audit process based on feedback *Included as a reference in the course manual

57 Wrap-up and Report the Audit Plan the Audit Conduct the Audit Wrap up and Report

58 Your Role Post-Audit Produce draft audit report to NERC Properly gather, collate and archive the teams working papers Both paper and electronic Only one evidence package is kept CEII Evidence SGI Evidence Remind audit team to destroy / delete any individual notes or evidence after they have been recorded in the RSAWs and turned over to the team lead for appropriate safeguard and storage

59 Key Take-Away Stakeholder Management Understand the Document Authority Hierarchy Confidentiality Understand the Tools for auditors-transparency Evidence - Focus on the facts Validate the evidence (reasonable assurance) Substantiate your findings (Corroborating of evidence) The Regional Entity determines compliance The Registered Entity must demonstrate compliance No evidence = Possible violation Use SMEs! 4 Bullets Quality, Consistency, Credibility A fair and consistent audit benefits all!

60 Questions and Closing Remarks