Agenda Standards Committee Process Subcommittee July 30, :00 p.m. 4:00 p.m. Eastern

Transcription

1 Agenda Standards Committee Process Subcommittee July 30, :00 p.m. 4:00 p.m. Eastern Conference Dial-in Information: Phone: Access Code: Security Code: Register for ReadyTalk access here: Webinar Access Administrative 1. Introductions and Chair s Remarks 2. NERC Antitrust Compliance Guidelines and Public Announcement* 3. Approval of Meeting Notes from June 09, 2015* 4. Review of Agenda and Objectives Agenda Items Standards Drafting Team Training Modules B. Nutter Technical Guidance Document Process* L. Lee SCPS and PMOS Meeting Frequency Discussion* L. Lee 4. Quality Review* L. Lee 5. Communication Facilitation L. Lee a. Work with NERC staff to develop ways for: i. Ballot pool members to communicate with each other; and ii. SC members to communicate with members in their segment 6. SCPS Activities* (Those Listed Below Will be Discussed) a. SCPS Work Plan* L. Lee i. Project Assignments ii. Conceptual Projects 7. Communication with Industry L. Lee 8. Items Slated for Presentation at Next Standards Committee Meeting L. Lee

2 9. Review of Actions/Assignments K. Street 10. Future Meetings a. Meetings in coordination with Standards Committee: September 22, 2015 (1:00 5:00 p.m.) Atlanta; and December 8, 2015 (1:00 5:00 p.m.) Marina del Rey, CA. b. Interim Conference Call: October 15 (1:00 4:00 p.m.) 11. Adjourn *Background materials are included. Meeting Agenda SCPS July 30,

3 Antitrust Compliance Guidelines I. General It is NERC s policy and practice to obey the antitrust laws and to avoid all conduct that unreasonably restrains competition. This policy requires the avoidance of any conduct that violates, or that might appear to violate, the antitrust laws. Among other things, the antitrust laws forbid any agreement between or among competitors regarding prices, availability of service, product design, terms of sale, division of markets, allocation of customers or any other activity that unreasonably restrains competition. It is the responsibility of every NERC participant and employee who may in any way affect NERC s compliance with the antitrust laws to carry out this commitment. Antitrust laws are complex and subject to court interpretation that can vary over time and from one court to another. The purpose of these guidelines is to alert NERC participants and employees to potential antitrust problems and to set forth policies to be followed with respect to activities that may involve antitrust considerations. In some instances, the NERC policy contained in these guidelines is stricter than the applicable antitrust laws. Any NERC participant or employee who is uncertain about the legal ramifications of a particular course of conduct or who has doubts or concerns about whether NERC s antitrust compliance policy is implicated in any situation should consult NERC s General Counsel immediately. II. Prohibited Activities Participants in NERC activities (including those of its committees and subgroups) should refrain from the following when acting in their capacity as participants in NERC activities (e.g., at NERC meetings, conference calls and in informal discussions): Discussions involving pricing information, especially margin (profit) and internal cost information and participants expectations as to their future prices or internal costs. Discussions of a participant s marketing strategies. Discussions regarding how customers and geographical areas are to be divided among competitors. Discussions concerning the exclusion of competitors from markets. Discussions concerning boycotting or group refusals to deal with competitors, vendors or suppliers.

4 Any other matters that do not clearly fall within these guidelines should be reviewed with NERC s General Counsel before being discussed. III. Activities That Are Permitted From time to time decisions or actions of NERC (including those of its committees and subgroups) may have a negative impact on particular entities and thus in that sense adversely impact competition. Decisions and actions by NERC (including its committees and subgroups) should only be undertaken for the purpose of promoting and maintaining the reliability and adequacy of the bulk power system. If you do not have a legitimate purpose consistent with this objective for discussing a matter, please refrain from discussing the matter during NERC meetings and in other NERC-related communications. You should also ensure that NERC procedures, including those set forth in NERC s Certificate of Incorporation, Bylaws, and Rules of Procedure are followed in conducting NERC business. In addition, all discussions in NERC meetings and other NERC-related communications should be within the scope of the mandate for or assignment to the particular NERC committee or subgroup, as well as within the scope of the published agenda for the meeting. No decisions should be made nor any actions taken in NERC activities for the purpose of giving an industry participant or group of participants a competitive advantage over other participants. In particular, decisions with respect to setting, revising, or assessing compliance with NERC reliability standards should not be influenced by anti-competitive motivations. Subject to the foregoing restrictions, participants in NERC activities may discuss: Reliability matters relating to the bulk power system, including operation and planning matters such as establishing or revising reliability standards, special operating procedures, operating transfer capabilities, and plans for new facilities. Matters relating to the impact of reliability standards for the bulk power system on electricity markets, and the impact of electricity market operations on the reliability of the bulk power system. Proposed filings or other communications with state or federal regulatory authorities or other governmental entities. Matters relating to the internal governance, management and operation of NERC, such as nominations for vacant committee positions, budgeting and assessments, and employment matters; and procedural matters such as planning and scheduling meetings. NERC Antitrust Compliance Guidelines 2

5 Public Announcements REMINDER FOR USE AT BEGINNING OF MEETINGS AND CONFERENCE CALLS THAT HAVE BEEN PUBLICLY NOTICED AND ARE OPEN TO THE PUBLIC Conference call version: Participants are reminded that this conference call is public. The access number was posted on the NERC website and widely distributed. Speakers on the call should keep in mind that the listening audience may include members of the press and representatives of various governmental authorities, in addition to the expected participation by industry stakeholders.

6 NERC List Policy NERC provides lists, or listservs, to NERC committees, groups, and teams to facilitate sharing information about NERC activities; including balloting, committee, working group, and drafting team work, with interested parties. All s sent to NERC listserv addresses must be limited to topics that are directly relevant to the listserv group s assigned scope of work. NERC reserves the right to apply administrative restrictions to any listserv or its participants, without advance notice, to ensure that the resource is used in accordance with this and other NERC policies. Prohibited activities include using NERC provided listservs for any price fixing, division of markets, and/or other anti competitive behavior. 1 Recipients and participants on NERC listservs may not utilize NERC listservs for their own private purposes. This may include announcements of a personal nature, sharing of files or attachments not directly relevant to the listserv group s scope of responsibilities, and/or communication of personal views or opinions, unless those views are provided to advance the work of the listserv s group. Use of NERC s listservs is further subject to NERC s Participant Conduct Policy for the Standards Development Process. Updated April Please see NERC s Antitrust Compliance Guidelines for more information about prohibited antitrust and anti competitive behavior or practices. This policy is available at

7 Meeting Notes Standards Committee Process Subcommittee June 9, :00 p.m. 4:00 p.m. Central Administrative 1. Introductions and Chair s Remarks L. Lee brought the meeting to order on June 2, 2015 at 12:00 p.m. CT. L. Lee welcomed the attendees. 2. NERC Antitrust Compliance Guidelines and Public Announcement K. Street reviewed the NERC Antitrust Compliance Guidelines and announced that the meeting was public. 3. Approval of Meeting Notes from April 30, 2015 The meeting notes were approved. 4. Review of Agenda and Objectives L. Lee reviewed the Agenda and Objectives. Agenda Items 1. Standards Balloting System B. Nutter B. Nutter provided an update on the Standards Balloting System (SBS), noting that there are still issues with ballot pools, and that NERC Staff will continue to monitor the SBS to determine the appropriate next steps. Barb also noted that there are still individuals that are joining ballot pools and not voting or abstaining. 2. Endorse SPM Process Documents developed in response to Audit recommendation P. Heidrich SCPS reviewed but did not endorse SPM Documents developed in response to Audit recommendation, as these documents will be used as NERC internal documents and not distributed or publicly posted. 3. SCPS Activities (Those Listed Below Will be Discussed) a. SCPS Work Plan L. Lee i. Project Assignments 1. Modification to Standards Committee (SC) Quality Review (QR) Process Document A small group of Standards Committee Process Subcommittee (SCPS) members have been working with NERC Staff to develop a QR guideline

8 ii. document and a QR form for SC. The plan going forward is for the small team to meet when more of the small team members are available and able to review and submit their comments on the current state of the documents. Then, the documents will be submitted to the SCPS at large for review. 2. Consensus Building and SAR Development Task is complete and will be scheduled to be presented to the SC at the next in-person meeting (September 2015). 3. Consolidate SC Resource Documents--SCPS Chair to follow up with SC Chair as to status of the Field Test Guideline document. 4. Improve standards development process--scps will collaborate with Standards Leadership to evaluate this item and determine next steps. 5. CEAP Pilot Project--SCPS will collaborate with Standards Leadership to evaluate this item and determine next steps. 6. Process to improve the effectiveness of balloting B. Nutter provided an update on the Standards Balloting System (SBS), noting that there are still issues with ballot pools, and that NERC Staff will continue to monitor the SBS to determine the appropriate next steps. Barb also noted that there are still individuals that are joining ballot pools and not voting or abstaining. 7. SPM use of prompt and substantive-- A motion was adopted during the March 2015 Standards Committee meeting to retain the words prompt and substantive in the SPM, but consider defining or clarifying the terms at some time in the future. The SCPS will work jointly with NERC staff to prepare a recommendation to the SC on whether to define or clarify the terms, and if so develop the accompanying definition or clarification. This project has been deferred until end of SDT Training Materials a small group of SCPS members received a set of training materials (Module 1) from Standards to review. Comments were provided to Monica Benson May 22. NERC staff is currently reviewing comments received and working on content for Module Resource Document Project (Phase 2)--Scope document approved by the SC at the May SC meeting. A draft list of documents has been compiled with document owners, most recent revision date and proposed review frequency identified. Next step is to finalize the spreadsheet of all Standards Resource documents and develop review criteria. Conceptual Projects Meeting Notes SCPS June 2,

9 1. Interpretation Process A. Wills is now the assigned NERC Legal representative. A. Wills provided Interpretation research and presented to the SCPS during the March in-person meeting. Further research is needed to determine next steps to determine appropriate means of alignment with FERC expectations, current processes, and best means by which to address interpretations. SCPS will collaborate with Standards Leadership to evaluate this item and determine next steps. 4. Communication with Industry L. Lee The last item communicated to industry was the SCPS Charter, which was communicated through the March 30 April 5, 2015 Weekly Standards and Compliance Bulletin. 1 The Charter is also posted on the related files page for the SCPS Items Slated for Presentation at Next Standards Committee Meeting L. Lee 6. Review of Actions/Assignments K. Street Consolidate SC Resource Documents--SCPS Chair to follow up with SC Chair as to status of the Field Test Guideline document. Improve standards development process--scps will collaborate with Standards Leadership to evaluate this item and determine next steps. CEAP Pilot Project--SCPS will collaborate with Standards Leadership to evaluate this item and determine next steps. Process to improve the effectiveness of balloting there are still issues with ballot pools, and that NERC Staff will continue to monitor the SBS to determine the appropriate next steps. SDT Training Materials--NERC staff is currently reviewing comments received and working on content for Module 2. Resource Document Project (Phase 2)--Scope document approved by the SC at the May SC meeting. Next step is to finalize the spreadsheet of all Standards Resource documents and develop review criteria. Interpretation Process Conceptual Project-- SCPS will collaborate with Standards Leadership to evaluate this item and determine next steps. 7. Future Meetings 1 March 30 April 5, 2015 Weekly Standards and Compliance Bulletin, available at: 2 SCPS related files page, available at: Related-Files.aspx. Meeting Notes SCPS June 2,

10 a. Meetings in coordination with Standards Committee: September 22, 2015 (1:00 5:00 p.m.) Atlanta; and December 8, 2015 (1:00 5:00 p.m.) Marina del Rey, CA. b. Interim Conference Calls: July 30 (1:00 4:00 p.m.); and October 15 (1:00 4:00 p.m.). 8. Adjourn L. Lee thanked members for their participation and adjourned the meeting. Meeting Notes SCPS June 2,

11 Agenda Item 14 Standards Committee July 15, 2015 Improving the Procedure for Technical Review of Reference Documents Action Endorse the SCPS, working with NERC staff, to review the Standard Process Manual (SPM) section 11 procedure, and provide the Standards Committee (SC) with its recommendations, if any, for changes at or before the September 2015 SC meeting. Background The SPM section 11 permits the SC to approve the posting of reference documents: The Standards Committee shall authorize the posting of all supporting references 32 that are linked to an approved Reliability Standard. Prior to granting approval to post a supporting reference with a link to the associated Reliability Standard, the Standards Committee shall verify that the document has had stakeholder review to verify the accuracy of the technical content. While the Standards Committee has the authority to approve the posting of each such reference, stakeholders, not the Standards Committee, verify the accuracy of the document s contents. 32 The Standards Committee s Procedure for Approving the Posting of Reference Documents is posted on the Reliability Standards Resources web page. Attachment 1 is the procedure referenced in footnote 32. This procedure leaves it to unspecified NERC staff to post a potential reference document for stakeholder review, with the author responding to comments and deciding when all of the comments have been adequately addressed. The role of NERC staff with respect confirming technical correctness is vague, and the authors appear to be in control: Authors Agenda Item 2 Standards Committee Process Subcommittee July 30, 2015 Develop a report that includes a response to each comment (based on criteria in the Standard Processes Manual) submitted and provide the report to NERC staff. Based on the comments received, either revise the reference and request another posting for comment or submit the reference to NERC staff for submission to the Standards Committee. Reliability Guidelines in Technical Committee Charters At its May 7, 2008 meeting, the Board approved separate Planning Committee (PC) and Operating Committee (OC) charter revisions that defined an identical process for the committees issuing Reliability Guidelines. An identical provision was later added to the Critical Infrastructure Protection (CIPC) charter, although in practice CIPC refers to them as Security Guidelines. Since current provisions from each charter identical, the OC charter provisions are in Attachments 2. The subject matter of any Reliability Standard is under one or more technical committees, and the OC, PC, and CIPC charter-imposed processes for developing Reliability Guidelines could be adopted for all ensuring accurate technical content for SC-authorized reference documents. Attachment 3 contains a proposed revision which is intended for SCPS review and input at a future SC meeting.

12 Agenda Item 14a Standards Committee July 15, 2015 Standards Committee Procedure Approving the Posting of Reliability Standard Supporting References Purpose: Conditions: Responsibility: Authors Authors Standards Administrator NERC staff NERC staff Standards Administrator To ensure that each supporting reference posted with a link to an approved Reliability Standard has received an open, inclusive, technical review. When a drafting team or technical committee has prepared a reference document that supports the implementation or understanding of an approved Reliability Standard and wants the Standards Committee to approve posting the document with a link to the approved Reliability Standard. Activity Draft a reference to aid stakeholders in the implementation or understanding of a Reliability Standard Submit the reference to NERC staff, with a request to post the reference document for at least a 30-day public comment period Format the reference according to the NERC Style Guide Post the reference with a comment form that includes, as a minimum, the following questions: Does this reference document aid in either the implementation or understanding of the associated standard? Is the terminology in the document consistent with the related standard? If not please explain. Has this been approved through some other open process? Please Identify. Please provide any other comments you have on this reference document that you haven t already provided. Announce the posting of the reference document for a comment period Assemble the comments submitted during the comment period and send a copy to both the Authors and NERC staff

13 Authors NERC staff Standards Committee Standards Administrator Develop a report that includes a response to each comment (based on criteria in the Standard Processes Manual) submitted and provide the report to NERC staff. Based on the comments received, either revise the reference and request another posting for comment or submit the reference to NERC staff for submission to the Standards Committee Include approval of the reference in the next regularly scheduled Standards Committee meeting. Provide a summary that includes: Dates the reference was posted Number and diversity of commenters (number of industry segments and NERC Regions represented) Issues resolved with changes to the reference Issues unresolved Recommendation on whether to approve posting the reference Approve posting the reference document if the following conditions have been met: Stakeholders indicated the reference would aid in the implementation or understanding of the associated reliability standard Stakeholders indicated that the reference is technically correct and provides a complete treatment of the subject Stakeholders have not identified any issues with the reference document that should preclude posting If approved, on the Approved Reliability Standards web page, add a link from the standard to the approved reference Version History Version Date Owner Change Tracking 1 March 10, 2008 NERC (Standards Endorsed Committee Endorser) 2 May 16, 2014 NERC (Standards Committee Endorser) 3 October 6, 2014 NERC (Standards Committee Endorser) 3 December 9, 2014 NERC (Standards Committee Endorser) Updated template Updated job titles Updates endorsed by the Standards Committee Standards Committee Procedure Approving the Posting of Reliability Standard Supporting References 2

14 Appendix 3 Reliability Guidelines Approval Process Agenda Item 14b Standards Committee July 15, Reliability Guidelines. Reliability guidelines are documents that suggest approaches or behavior in a given technical area for the purpose of improving reliability. Reliability guidelines are not binding norms or mandatory requirements. Reliability guidelines may be adopted by a responsible entity in accordance with its own facts and circumstances Approval of Reliability Guidelines. Because reliability guidelines contain suggestions that may result in actions by responsible entities, those suggestions must be thoroughly vetted before a new or updated guideline receives approval by the OC. The process described below will be followed by the OC: a. New/updated draft guideline approved for industry posting. The OC approves for posting for industry comment the release of a new or updated draft guideline developed by one of its subgroups or the committee as a whole. b. Post draft guideline for industry comment. The draft guideline is posted as for industry-wide comment for forty-five (45) days. If the draft guideline is an update, a redline version against the previous version must also be posted. c. Post industry comments and responses. After the public comment period, the OC will post the comments received as well as its responses to the comments. The committee may delegate the preparation of responses to a committee subgroup. d. New/updated guideline approval and posting. A new or updated guideline which considers the comments received, is approved by the OC and posted as Approved on the NERC website. Updates must include a revision history and a redline version against the previous version. e. Guideline updates. After posting a new or updated guideline, the OC will continue to accept comments from the industry via a web-based forum where commenters may post their comments. i. Each quarter, the OC will review the comments received. At any time, the OC may decide to update the guideline based on the comments received or on changes in the industry that necessitate an update. ii. Updating an existing guideline will require that a draft updated guideline be approved by the OC in step a and proceed to steps b and c until it is approved by the OC in step d. 3. Review of Approved Reliability Guidelines. Approved reliability guidelines shall be reviewed for continued applicability by the OC at a minimum of every third year since the last revision. 2 Standards Committee authorization is required for a reliability guideline to become a supporting document that is posted with or referenced from a NERC Reliability Standard. See Appendix 3A in the NERC s Rules of Procedure under Supporting Documents. NERC Operating Committee Charter September of 18

15 Agenda Item 14c Standards Committee July 15, 2015 Standards Committee Procedure Approving the Posting of Reliability Standard Supporting Documents Purpose: To ensure that each supporting document posted with a link to an approved Reliability Standard pursuant to Section 11 of the Standards Processes Manual (SPM) has received an open, inclusive, technical review. The process below permits anyone to submit an intended supporting document for review, including individuals or a drafting team. The process assumes that the technical committees charters have an identical reliability guideline approval process which will be used to approve intended supporting documents. 1. The document s author shall submit the document to the Director of Standards, who shall transmit it to the chair of one or more of the NERC technical committees 1 that has expertise related to the document s subject matter The committee chair(s) receiving the document shall establish an ad hoc review group comprised of committee members or other persons that have expertise on the document s subject matter. The review group shall initiate discussions with the document s author so that the document may be considered by the full technical committee(s) at the earliest opportunity. 3. The full committee s(committees ) shall be provided the following for consideration: a. The author s original document; and b. A modified document that reflects the input of the ad hoc review group. 4. Once the committee(s) approves (approve) a version of the document, which may include further modifications by the committee(s), the document shall be posted for industry comment until the document is approved by the committee(s). After consideration of comments, once the committee(s) approves(approve) the final document, the committee s(committees ) chair(s) shall transmit the approved document to the Director of Standards who shall present it to the Standards Committee for its action under Section 11 of the SPM. 5. The committee(s)-approved document shall be updated per the reliability guideline approval process, and any approval of subsequent updates, after consideration of industry comments, shall be transmitted by the committee s (committees ) chair(s) to the Director of Standards who shall present it to the Standards Committee for its action under Section 11 of the SPM. 1 The NERC technical committees are the Critical Infrastructure Protection Committee (CIPC), Planning Committee (PC), or Operating Committee (OC). 2 In most cases, one technical committee s expertise will be sufficient to address the document s subject matter; however, the procedure is written to allow more than one technical committee to receive the document. In this case, the committees are expected to coordinate all aspects of this procedure.

16 Version History Version Date Owner Change Tracking 1 March 10, 2008 NERC (Standards Committee Endorser) Endorsed 2 May 16, 2014 NERC (Standards Committee Endorser) Updated template 3 October 6, 2014 NERC (Standards Committee Endorser) 3 December 9, 2014 NERC (Standards Committee Endorser) Updated job titles Updates endorsed by the Standards Committee

17 Agenda Item 3 Standards Committee Process Subcommittee July 30, 2015 Agenda Item 13 Standards Committee July 15, 2015 Request for PMOS and SCPS to Recommend to SCEC a SC Meeting Frequency Action Assign the Project Management and Oversight Committee (PMOS) and Standards Committee Process Subcommittee (SCPS) to review the draft Reliability Standards Development Plan and provide the Standards Committee Executive Committee (SCEC) options on the meeting frequency for The SCEC, working with NERC staff, will then present options to the Standards Committee in September 2015 in setting a schedule for Background Most NERC Standing Committees meet on a quarterly basis. The NERC Standards Committee met monthly for several years to address FERC directives and develop a set of results-based standards. In 2016, the focus on existing standards will shift to periodic reviews, resulting in a more deliberate pace of standards development. If the Standards Committee meets less frequently, it is important to retain flexibility to address regulatory deadline-driven matters and some of the more routine document postings without increasing project schedules. Therefore, as noted in section 7 of the Standards Committee Charter, unscheduled Standards Committee calls of limited scope, and the SCEC acting on the Standard Committee's behalf, are additional tools available to authorize postings of SARs, Reliability Standards, and standards-related documents. Recent meetings have lasted four hours or less. To justify continued in-person meetings, it may be required to consolidate matters into a fewer set of more impactful meetings.

18 Agenda Item 4 Standards Committee Process Subcommittee July 30, 2015 Reliability Standard Quality Review Form Project Name: Standard: Date of Review: The standard drafting team (SDT) conducted a quality review in accordance with the NERC Guideline for Quality Review of NERC Reliability Standards Project Documents and recommends that the Standards Committee authorize the proposed documents for formal posting and balloting. Background The NERC Standard Processes Manual (SPM) Section 4.6 requires NERC staff to coordinate a Quality Review 1 of the Reliability Standard, Implementation Plan, Violation Risk Factors (VRFs) and Violation Severity Levels (VSLs), in parallel with the development of the Reliability Standard and Implementation Plan to assess whether: (1) The documents proposed for posting are within the scope of the associated Standard Authorization Request (SAR); (2) The Reliability Standard is clear and enforceable as written; and (3) The Reliability Standard meets the criteria specified in: NERC s Benchmarks for Excellent Standards 2 and Criteria for governmental approval of Reliability Standards. 3 The drafting team considered the results of the quality review and decided upon appropriate changes. Quality Review Summary Add 1-2 paragraphs summarizing the Quality Review that was performed. The summary should contain: When the Quality Review for this project was conducted Who participated? Whether the SDT deviated from any of the criteria (i.e. P81) and the SDT s rationale Any other nuances that are relevant The SDT hereby recommends that the SC authorize this project for [insert either SAR comment period or initial comment and ballot period]. 1 The SPM s Quality Review requirements also apply to new or revised definitions and Reliability Standard interpretations See FERC Order No. 672.

19 NERC Standard Developer Date Drafting Team Leadership Date Guideline for Quality Review of NERC Reliability Standards Project Documents 2

20 Guideline for Quality Review of NERC Reliability Standards Project Documents Updated July 20, 2015 Purpose To review standards through the development process for content, quality and administrative criteria. Introduction The NERC Standard Processes Manual (SPM) Section 4.6 requires NERC staff to coordinate a Quality Review 1 of the Reliability Standard, Implementation Plan, Violation Risk Factors (VRFs) and Violation Severity Levels (VSLs), in parallel with the development of the Reliability Standard and Implementation Plan to assess whether: 1. The documents proposed for posting are within the scope of the associated Standard Authorization Request (SAR); 2. The Reliability Standard is clear and enforceable as written; and 3. The Reliability Standard meets the criteria specified in NERC s Benchmarks for Excellent Standards 2 and criteria for governmental approval of Reliability Standards. 3 Process and Participants An effective Quality Review strategy requires incorporating the provided in this document throughout the development process. Additionally, an overall Quality Review specifically focused on the qualitative and administrative criteria should be conducted prior to documents being posted for industry comments. Quality Review participants should include people that have not been heavily involved in the development of the project documents, so they would be bring independence (i.e. cold eyes ) to the review. Further, participants may include: Standards Developer NERC legal NERC compliance PMOS representative Subject matter expert assigned to the project Independent subject matter expert Independent legal or editing experts SDT leadership 1 The SPM s Quality Review requirements also apply to new or revised definitions and Reliability Standard interpretations See FERC Order No. 672.

21 NERC administrative staff It is at the Standard Developer s and SDT Chair s discretion to determine the appropriate Quality Review participants, the PMOS representative on the project must be involved, and the SDT ultimately must approve final project documents. Industry, NERC, and the Standards Committee have developed an Enhanced Periodic Review Template that will guide the Quality Review for quality and content. Note that much of the criteria in the template are also provided further below in this document: The following sections with and reference documents links provide additional suggestions and guidelines for the Quality Review. The sections below outline: Content questions, which should be considered throughout the development of the standard. The content is the responsibility of the SDT; Quality criteria, which should be considered throughout the development but also specifically considered during a pre-posting review. This is to be addressed by the participants conducting the Quality Review (listed above); and Administrative criteria, which should be specifically reviewed prior to posting for industry comment. This is the responsibility of NERC staff, although the Quality Review may identify some suggestions. Certain criteria may be considered in multiple sections. Content Although this guideline is intended to be a best practices document to guide and inform Quality Review, documents should be reviewed for content, using the below criteria: Is a standard required? Does each of the requirements support reliability (i.e. a Reliability Principle 4 )? Do the requirements meet the Paragraph 81 criteria 5 or would a guideline 6 be more appropriate? If so, is the content steady-state? 4 The Adequate Level of Reliability Task Force Reliability Principles, July 8, The Adequate Level of Reliability Task Force and supporting documents can be located on the NERC website at: 5 Paragraph 81 refers to this paragraph in the March 15, 2012 FERC Order Accepting with Conditions the Electric Reliability Organization s Petition Requesting Approval of New Enforcement Mechanisms and Requiring Compliance Filing; North American Electric Reliability Corporation, 138 FERC 61,193 at P 81 (2012). 6 The NERC technical committees develop guidelines. The processes for each are contained in each committee s charter. The Planning Committee s Report/Reliability Guideline Approval Process for approving guidelines is contained in Appendix 4 of its charter; the Operating Committee s Reliability Guidelines Approval Process is contained in Appendix 3 of its charter. Guideline for Quality Review of NERC Reliability Standards Project Documents 2

22 Are the appropriate actions, for which there should be accountability, included or is there a gap? 7 Does the standard identify the appropriate Functional Entities, as identified in the Functional Model, and applicability? Is the content of the requirement technically correct, including identifying who does what and when? Does the standard address Federal Energy Regulatory Commission directives? Does the Reliability Standards conform to Ten Benchmarks for an Excellent Reliability Standard (included below for reference)? Is it technologically neutral? Are the expectations for each function clear? Is it a higher solution than the lowest common denominator? Is it measureable? Does it have a technical basis in engineering and operations? Can it be practically implemented? Quality Project documents must conform to fundamental quality principles and should be reviewed for quality, using the below criteria: Should the requirement stand alone as is or should it be consolidated with other requirements or standards? Do the requirements meet the Paragraph 81 criteria 8 or would a guideline 9 be more appropriate? Does the standard identify the appropriate Functional Entities, as identified in the Functional Model, and applicability? Is the content of the requirement technically correct, including identifying who does what and when? 7 The areas where risks to the BPS are not adequately mitigated in the standards may be referred to as gaps throughout this document. 8 Paragraph 81 refers to this paragraph in the March 15, 2012 FERC Order Accepting with Conditions the Electric Reliability Organization s Petition Requesting Approval of New Enforcement Mechanisms and Requiring Compliance Filing; North American Electric Reliability Corporation, 138 FERC 61,193 at P 81 (2012). 9 The NERC technical committees develop guidelines. The processes for each are contained in each committee s charter. The Planning Committee s Report/Reliability Guideline Approval Process for approving guidelines is contained in Appendix 4 of its charter; the Operating Committee s Reliability Guidelines Approval Process is contained in Appendix 3 of its charter. Guideline for Quality Review of NERC Reliability Standards Project Documents 3

23 Does the Reliability Standards conform to Ten Benchmarks for an Excellent Reliability Standard? Is it drafted as a results-based standard (RBS) requirement (performance, risk (prevention) or capability) and does it follow the RBS format (e.g., sub-requirement structure)? See Results Based Standards materials development guidance Are the expectations for each function clear? Does the requirement align with the purpose? Is it measureable? Is it complete and self-contained? Is the language clear and does not contain ambiguous or outdated terms? Does it use consistent terminology? Additional Reference documents include: Acceptance Criteria of a Reliability Standard The NERC Functional Model NERC Rules of Procedure FERC Order 672 (Order containing the 16 factors for a standard) Additional FERC Orders (including Order 748, Order 890, and Order 729) Administrative The Standard Developer should ensure that appropriate administrative staff conducts a final review to mitigate typographical errors and to ensure document compliance with the NERC Style Guide. Additionally, the Standard Developer must ensure that the final documents include the following: Correct document templates Spelling and grammar reviews Consistency among documents Updated version history Redlines to last approved Uncorrupted document conversion to.pdf format Appropriate Data Retention requirements VRFs and VSLs 10 meet the necessary criteria Responses to Comments address all industry comments 10 See 123 FERC 61,284 Order on Violation Severity Levels Proposed by the Electric Reliability Organization, issued June 19, Guideline for Quality Review of NERC Reliability Standards Project Documents 4

24 Clear and comprehensive Implementation Plan Enforcement date language for Canada and/or Mexico Implementation of the Guideline Standards Authorization Form (SAR): A quality review is conducted prior to the SAR being posted Informal Comment Period: A quality review is conducted prior to the Standard(s) and supporting material being posted Initial Comment Period and Ballot: o A quality review is conducted prior to the Standard(s) and supporting material being presented to the Standards Committee (SC) for authorization to post o A Reliability Standard Quality Review Form is presented to the SC with the standard(s) and supporting materials Additional Comment Period and Ballot: o Final Ballot: o A quality review is conducted prior to the Standard(s) and supporting material being posted A quality review is conducted prior to the Standard(s) and supporting material being posted Submittal of the standard(s) and supporting material for Board adoption: o o A quality review is conducted prior to the Standard(s) and supporting material being submitted for the Board package A review form is completed Guideline for Quality Review of NERC Reliability Standards Project Documents 5

25 Guideline for Quality Review of NERC Reliability Standards Project Documents Updated May June 8827, 2015 Purpose To review standards through the development process for content, quality and administrative criteria. Introduction The NERC Standard Processes Manual (SPM) Section 4.6 requires NERC staff to coordinate a Quality Review 1 of the Reliability Standard, Implementation Plan, Violation Risk Factors (VRFs) and Violation Severity Levels (VSLs), in parallel with the development of the Reliability Standard and Implementation Plan to assess whether: 1. the documents proposed for posting are within the scope of the associated Standard Authorization Request (SAR); 2. the Reliability Standard is clear and enforceable as written; and 3. the Reliability Standard meets the criteria specified in NERC s Benchmarks for Excellent Standards 2 and criteria for governmental approval of Reliability Standards. 3 Process and Participants An effective Quality Review strategy requires incorporating the provided in this document throughout the development process. Additionally, an overall Quality Review specifically focused on the qualitative and administrative criteria should be conducted prior to documents being posted for industry comments. Quality Review participants should include people that have not been heavily involved in the development of the project documents, so they would be bring independence (i.e. cold eyes ) to the review. Further, participants may include: Standards Developer NERC legal NERC compliance PMOS representative Subject matter expert assigned to the project Independent subject matter expert Independent legal or editing experts SDT leadership 1 The SPM s Quality Review requirements also apply to new or revised definitions and Reliability Standard interpretations See FERC Order No. 672.

26 NERC administrative staff It is at the Standard Developer s and SDT Chair s discretion to determine the appropriate Quality Review participants, the PMOS representative on the project must be involved, and the SDT ultimately must approve final project documents. Industry, NERC, and the Standards Committee have developed an Enhanced Periodic Review Ttemplate that will guide the Quality Review for quality and content. Note that much of the criteria in the template are also provided further below in this document: Enhanced Periodic Review Template The following sections with and reference documents links provide additional suggestions and guidelines for the Quality Review. The sections below outline: Content questions, which should be considered throughout the development of the standard. The content is the responsibility of the SDT; Quality criteria, which should be considered throughout the development but also specifically considered during a pre-posting review. This is to be addressed by the participants conducting the Quality Review (listed above); and Administrative criteria, which should be specifically reviewed prior to posting for industry comment. This is the responsibility of NERC staff, although the Quality Review may identify some suggestions. Certain criteria may be considered in multiple sections. Content Although this guideline is intended to be a best practices document to guide and inform Quality Review, documents should be reviewed for content, using the below criteria: Is a standard required? Does each of the requirements support reliability (i.e. a Reliability Principle 4 )? Do the requirements meet the Paragraph 81 criteria 5 or would a guideline 6 be more appropriate? 4 The Adequate Level of Reliability Task Force Reliability Principles, July 8, The Adequate Level of Reliability Task Force and supporting documents can be located on the NERC website at: 5 Paragraph 81 refers to this paragraph in the March 15, 2012 FERC Order Accepting with Conditions the Electric Reliability Organization s Petition Requesting Approval of New Enforcement Mechanisms and Requiring Compliance Filing; North American Electric Reliability Corporation, 138 FERC 61,193 at P 81 (2012). 6 The NERC technical committees develop guidelines. The processes for each are contained in each committee s charter. The Planning Committee s Report/Reliability Guideline Approval Process for approving guidelines is contained in Appendix 4 of its charter; the Operating Committee s Reliability Guidelines Approval Process is contained in Appendix 3 of its charter. Guideline for Quality Review of NERC Reliability Standards Project Documents 2

27 If so, is the content steady-state? Are the appropriate actions, for which there should be accountability, included or is there a gap? 7 Does the standard identify the appropriate Functional Entities, as identified in the Functional Model, and applicability? Is the content of the requirement technically correct, including identifying who does what and when? Does the standard address Federal Energy Regulatory Commission directives? Does the Reliability Standards conform to Ten Benchmarks for an Excellent Reliability Standard (included below for reference)? Quality Project documents must conform to fundamental quality principles and should be reviewed for quality, using the below criteria: Should the requirement stand alone as is or should it be consolidated with other requirements or standards? Do the requirements meet the Paragraph 81 criteria 8 or would a guideline 9 be more appropriate? Does the standard identify the appropriate Functional Entities, as identified in the Functional Model, and applicability? Is the content of the requirement technically correct, including identifying who does what and when? Does the Reliability Standards conform to Ten Benchmarks for an Excellent Reliability Standard? Is it drafted as a results-based standard (RBS) requirement (performance, risk (prevention) or capability) and does it follow the RBS format (e.g., sub-requirement structure)? See Results Based Standards materials development guidance Is it technologically neutral? Are the expectations for each function clear? Formatted: Superscript Formatted: Superscript Commented [KS1]: Should this be added or moved to the Content section? Commented [KS2]: Should this be added to the Content section? 7 The areas where risks to the BPS are not adequately mitigated in the standards may be referred to as gaps throughout this document. 8 Paragraph 81 refers to this paragraph in the March 15, 2012 FERC Order Accepting with Conditions the Electric Reliability Organization s Petition Requesting Approval of New Enforcement Mechanisms and Requiring Compliance Filing; North American Electric Reliability Corporation, 138 FERC 61,193 at P 81 (2012). 9 The NERC technical committees develop guidelines. The processes for each are contained in each committee s charter. The Planning Committee s Report/Reliability Guideline Approval Process for approving guidelines is contained in Appendix 4 of its charter; the Operating Committee s Reliability Guidelines Approval Process is contained in Appendix 3 of its charter. Guideline for Quality Review of NERC Reliability Standards Project Documents 3

28 Does the requirement align with the purpose? Is it a higher solution than the lowest common denominator? Is it measureable? Does it have a technical basis in engineering and operations? Is it complete and self-contained? Is the language clear and does not contain ambiguous or outdated terms? Can it be practically implemented? Does it use consistent terminology? Commented [KS3]: Should this be added or moved to the Content section? Commented [KS4]: Should this be added to the Content section? Commented [KS5]: Should this be added or moved to the Content section? Commented [KS6]: Should this be added or moved to the Content section? Additional Reference documents include: Acceptance Criteria of a Reliability Standard The NERC Functional Model NERC Rules of Procedure FERC Order 672 (Order containing the 16 factors for a standard) Additional FERC Orders (including Order 748, Order 890, and Order 729) Administrative The Standard Developer should ensure that appropriate administrative staff conducts a final review to mitigate typographical errors and to ensure document compliance with the NERC Style Guide. Additionally, the Standard Developer must ensure that the final documents include the following: Correct document templates Spelling and grammar reviews Consistency among documents Updated version history Redlines to last approved Uncorrupted document conversion to.pdf format Appropriate Data Retention requirements VRFs and VSLs 10 meet the necessary criteria Responses to Comments address all industry comments Clear and comprehensive Implementation Plan Enforcement date language for Canada and/or Mexico 10 See 123 FERC 61,284 Order on Violation Severity Levels Proposed by the Electric Reliability Organization, issued June 19, Guideline for Quality Review of NERC Reliability Standards Project Documents 4

29 Implementation of the GuidelineThe procedure Standards Authorization Form (SAR): A quality review is conducted prior to the SAR being posted Informal Comment Period: A quality review is conducted prior to the Standard(s) and supporting material being posted Initial Comment Period and Ballot: o A quality review is conducted prior to the Standard(s) and supporting material being presented to the Standards Committee (SC) for authorization to post o A Reliability Standard Quality Review Form is presented to the SC with the standard(s) and supporting materials Additional Comment Period and Ballot: o A quality review is conducted prior to the Standard(s) and supporting material being posted Final Ballot: o A quality review is conducted prior to the Standard(s) and supporting material being posted Submittal of the standard(s) and supporting material for Board adoption: o A quality review is conducted prior to the Standard(s) and supporting material being submitted for the Board package o A review form is completed Commented [KS7]: Keith & Andy this section describes when the procedure will occur, rather than what the procedure entails. This section may need to be retitled. Guideline for Quality Review of NERC Reliability Standards Project Documents 5