From Audit Requirements to Checklists to Evidence Gathering Plans Linda Westfall 12 October 2017
Turning Requirements into Audit Results Audit Requirements Audit Inputs Audit Initiation Audit Checklists Objective Evidence Gathering Plan Audit Preparation Objective Evidence Nonconformances & Observations Audit Executions Audit Results Audit Reporting
Audit Requirements Evaluation Criteria Audit requirements provide the objective evaluation criteria against which conformance/compliance is evaluated. Examples of audit requirements include: Written organizational quality policies Documented objectives (e.g., budgets, programs, contracts) Customer or organizational quality specifications, standards, or procedures Product requirements and/or specifications Governmental or regulatory requirements Industry standards
Process Documentation Hierarchy Industry Level Process Documentation Industry Standards Quality Manual Organization Level Process Documentation Standard Processes Standard Work Instructions, (Guidelines, Templates, Checklists) Program/Project Level Process Documentation Quality Plans Project/Program Specific or Tailored Processes Project/Program Specific or Tailored Work Instructions
Checklists Checklists are: Lists of yes/no questions Tools to help add organization & structure Tools to ensure complete coverage of audit criteria Precise, measurable, factual Checklists correspond to the audit requirements & help ensure complete coverage within the scope of the audit.
Turning Requirements into Checklist Items Requirement: The organization shall ensure that the person(s) doing the work are competent on the basis of appropriate education, training, or experience. Checklist Item: Do the person(s) doing the work have the appropriate education, training, and/or experience? Verification activities are conducted to ensure that the design and development outputs meet the input requirements Are the design and development outputs appropriately verified against the requirements?
Generic Checklist Items Examples
Generic Checklist Items Examples (cont.)
Generic Checklist Items Examples (cont.)
Gathering Objective Evidence An auditor measures compliance/conformance by gathering & analyzing objective evidence. Objective evidence is: Information which can be proved true based on facts Observed or documented evidence uninfluenced by prejudice, emotion or bias For each checklist item plan the technique(s) that will be used to gather objective evidence for that item.
Objective Evidence Gathering Techniques
Observing an Event or Process An auditor can observe work in progress to see if it meets requirements. The auditor observes if: The product was made or activity performed according to documented procedures or work instructions It was done by the designated responsible person The proper equipment and/or tools were used Employees were familiar with policies & procedures & that they know their roles & responsibilities The auditor should watch or be present without participating actively.
Interviews Interview questions should: Be open-ended & context-free Have know, expected answers Focus on the product/process/ problem -- not the individual Be organized in a logic order The auditor should seek corroboration: More than one interviewee says the same thing Another audit team member hears the same thing A record, document or other objective evidence verifies response
Open-Ended Questions Examples of open-ended vs. close-ended questions: Open-Ended Question What procedures do you follow when performing your work? What reviews & approvals are involved in releasing your work products? How do you communicate the problems you encounter? Close-Ended Question Do you follow the XYZ procedure when you do your work? Do you perform a peer review on your work products? Does the software lead approve your work products? Do you record the problems you find in the defect tracking tool?
Context Free Questions Examples of context free questions: Ask This What documentation do you use when performing this task? How do you verify the quality, completeness & consistency of your work products? How do you track your project s progress? Not This How do you use the SOW Template when performing this task? What steps do you use when conducting unit testing of your work product? How do you use Microsoft Project to track your project s progress?
Examining Records Quality records are examined to ensure that evidence exists to: Demonstrate product conformance to requirements Appropriate implementation of the processes If it s not written down, it never happened Examples of quality records: Meeting minutes Reports Completed forms or checklists Metrics or data Documented approvals
Examining Documents If documents were supplied as audit inputs, examination was accomplished during document review. Based upon information obtained during execution: Additional examination of documents supplied as input Examination of additional documents requested during the audit
Measurement Measurement can be accomplished through: Collecting & analyzing new data Analyzing existing data Examining existing metrics Looking for: Patterns & trends Escapes Out of control values Conformance to entry/exit criteria
Sampling Rarely is there enough time or resources to: Interview every auditee about every checklist item Examine every work product Examine every quality record Auditing is based on the concept of sampling.
Sampling Techniques Non-statistical sampling methods: Haphazard sampling Block selection Judgmental selection Statistical sampling methods: Random sampling Systematic sampling Stratified sampling
Sampling Plan Objective evidence may be gathered by sampling. Determine sample size & selection based on: Complexity Size/volume Risk Past problems Time available Use sampling methods to select a sample Don t let auditee select the sample If no nonconformances are found move on Consider minimal sample size guidelines (4/10, 10/100, 20/1000)
Evidence Gathering Plans Yes No Checklist Item Evidence Gathering Plan Does the person performing the task have access to applicable processes and/or work instructions? Are the processes or work instructions up-to-date (latest revision)? Observation Interviews Examine records & documents Measurement Test
Evidence Gathering Plans (cont.) Yes No Checklist Item Evidence Gathering Plan Are the entry criteria to the process met before the process is started? Are the exit criteria to the process met before the process is completed? Observation Interviews Examine records & documents Measurement Test
Evidence Gathering Plans (cont.) Yes No Checklist Item Evidence Gathering Plan Are the correct person(s) performing each task correctly per the required processes and/or work instructions? Observation Interviews Examine records & documents Measurement Test
Evidence Gathering Plans (cont.) Yes No Checklist Item Evidence Gathering Plan Do the person(s) doing the work have the appropriate education, training, and/or experience? Observation Interviews Examine records & documents Measurement Test
Evidence Gathering Plans (cont.) Yes No Checklist Item Evidence Gathering Plan Are the required activities being performed in an effective & efficient manner? Observation Interviews Examine records & documents Measurement Test
Turning Requirements into Audit Results Audit Requirements Audit Inputs Audit Initiation Audit Checklists Objective Evidence Gathering Plan Audit Preparation Objective Evidence Nonconformances & Observations Audit Executions Audit Results Audit Reporting
Questions?
Contact Information Linda Westfall 3000 Custer Road Suite 270, PMB 101 Plano, TX 75075-4499 phone: (972) 867-1172 email: lwestfall@westfallteam.com www.westfallteam.com