THE COMMANDER NAVY REGION, SOUTHWEST (CNRSW) HAZARDOUS MATERIALS (HAZMAT) OPERATIONS QUALITY ASSURANCE SURVEILLANCE PLAN (QASP) 20 June 2000
TABLE OF CONTENTS 1.0 INTRODUCTION 1 1.1 PURPOSE... 1 2.0 OVERVIEW.2 2.1 DEVELOPMENT... 2 2.2 METHODOLOGY... 2 2.3 SERVICE PROVIDER RESPONSIBILITY... 2 2.4 GOVERNMENT RESPONSIBILITY... 2 2.5 NON-CONFORMANCE... 2 2.6 PENALTIES... 2 3.0 PART I. QUALITY ASSURANCE REQUIREMENTS.4 3.1 OVERVIEW OF CONTRACT REQUIREMENTS (CR)... 4 4.0 PART II. QUALITY ASSURANCE METHODS 5 4.1 OVERVIEW... 5 4.2 ONE HUNDRED (100) PERCENT SURVEILLANCE... 5 4.3 MANAGEMENT INFORMATION SYSTEMS... 5 4.4 PLANNED SAMPLING... 5 4.5 RANDOM SAMPLING... 5 4.6 CUSTOMER COMPLAINTS... 6 4.7 CUSTOMER SATISFACTION... 6 4.8 COMBINING METHODS OF SURVEILLANCE... 6 4.9 OVERVIEW OF SURVEILLANCE METHODS BY FUNCTIONAL AREA... 6 4.10 CONTRACT ADMINISTRATION RESOURCES... 8 4.10.1 Contract Administration Staff... 8 4.10.2 Quality Assurance Evaluators... 8 5.0 PART III. QUALITY ASSURANCE SURVEILLANCE SCHEDULE 9 5.1 QUALITY ASSURANCE EVALUATOR SCHEDULE... 9 5.2 ACTUAL SURVEILLANCE... 9 5.3 SURVEILLANCE DOCUMENTATION... 10 5.4 CONTRACT COMMUNICATION... 12 5.5 REVISIONS TO THE QASP... 12 6.0 PART IV. QUALITY CONTROL PLAN REVIEW.13 6.1 QUALITY CONTROL PLAN INSPECTION PROCEDURE... 13 7.0 PART V. SAMPLING PLANS 14 Tables Table No. Title Page No. Quality Assurance Surveillance Plan: CNRSW HAZMAT i
1 Performance Requirements Summary Q-7 2 Contract Administration Organization Size Q-8 3 Suggested Sample Size Q-10 4 Quality Assurance Evaluator Checklist Q-11 Quality Assurance Surveillance Plan: CNRSW HAZMAT ii
1.0 INTRODUCTION 1.1 Purpose The Quality Assurance Surveillance Plan (QASP) is designed to assist the Government in monitoring the quality and quantity of services that may be provided by either a Government or a commercial organization. The plan must comply with the requirements of the Performance Work Statement (PWS), regardless of whether the service provider is the Government or a contractor. (For purposes of this plan, the party performing the requirements of the PWS is referred to as the Service Provider (SP).) The QASP consists of the following parts: Part I, Quality Assurance Requirements, contains an overview and general description of the elements relating to the quantity and quality of the services required by the PWS. It references functional and performance requirements in all areas of the contract. Part II, Quality Assurance Methods, provides an assessment of the risks and benefits associated with choosing a particular surveillance method for a given performance requirement. It also identifies any critical logistics, protocol, mission sensitive, or similar considerations that, if not dealt with appropriately, may adversely impact the successful performance of PWS requirements. The Contracting Officer (KO) will use information from this section to decide on the level of quality assurance (QA) resources which can be supported and the amount of risk which can be assumed. Part III, Quality Assurance Surveillance Schedule, guides the Quality Assurance Evaluator (QAE) in scheduling and documenting the performance of the SP that is performing the work described in the PWS. Part IV, Quality Control Plan Review, provides the QAE(s) a method to review the SP s Quality Control (QC) Plan, past performance history, any certification, such as ISO 9000, and the actual performance of services. Depending on all of these factors, the level of surveillance can be adjusted upwards if there are performance problems or downwards if performance is at or above the desired levels. Part V, Sampling Plans, provides the QAE(s) with information on the method of surveillance, acceptable quality level, lot size, sample size, performance standard and procedure for inspecting SP performance. Quality Assurance Surveillance Plan: CNRSW HAZMAT Q-1
2.0 OVERVIEW 2.1 Development This QASP has been developed to provide the QAE(s) an effective and systematic surveillance method for selected functions described in the PWS. 2.2 Methodology This QASP provides a systematic method to evaluate services required (e.g., quality, quantity, timeliness, etc.) and not the details of how the SP accomplishes the work. This QASP identifies the various surveillance methods that, when properly utilized, assure the Government of satisfactory SP performance. 2.3 SP responsibility This QASP is based on the premise that the SP, and not the Government, is responsible for management and quality control actions to meet the terms of the contract. The performance requirements and standards outlined in the CNRSW PWS recognize SP performance will not be perfect and unforeseen and uncontrollable problems do occur. However, good management and the use of an adequate quality control plan will allow the SP to operate within specified performance requirements. The SP is the owner of the QC process and will institute procedures that, if followed, will produce the desired outcomes. The SP develops, implements, and modifies the QC process. QAE(s) conduct an initial review of the quality control process to ensure its adequacy. The QAE(s) will vary the level of surveillance depending on the SP's conformance to quality levels. The QAE(s) will use statistically valid samples to ensure that the SP's process is accomplishing the desired performance standards of the contract. 2.4 Government responsibility The Government intends to minimize the level of Government involvement and inspection to allow the SP to responsibly perform to, or exceed, the contract standards. The Government's recent experience supports the notion that responsible SP's can produce at acceptable levels of quality without extensive surveillance. If the SP's performance is not satisfactory, and it appears that the SP's QC process has not produced the desired result, the QAE(s) has the option to increase quality assurance surveillance in order to protect the Government's rights. 2.5 Non-conformance Any non-conformance with contract requirements is a defect. The terms defective and discrepancy are used in reference to a service output that does not meet the output s associated standard as defined in the contract. If any Government action or lack of action causes unacceptable performance which results in a defect, then these defects shall not be counted against the SP. All costs associated with rework due to SP defects are the responsibility of the SP. The KO can use the inspection of the service clauses in the contract to address nonperformance, defective performance and performance at less than the standards required by the contract. 2.6 Penalties In addition, failure to meet performance standards or comply with regulations may result in contract penalties. Contract penalties may include suspension or termination for cause, delay of Quality Assurance Surveillance Plan: CNRSW HAZMAT Q-2
progress payment, reduced payment, or a decision not to exercise contract options. The KO will determine the penalty for non-compliance based on his/her judgment and the severity of noncompliance. SP negligence will almost certainly result in contract cancellation. If an award fee is used with the contract, the loss of potential incentive payments to the SP can also influence positive contract performance. Quality Assurance Surveillance Plan: CNRSW HAZMAT Q-3
3.0 PART I. QUALITY ASSURANCE REQUIREMENTS 3.1 Overview of Contract Requirements (CR) The following Contract Requirements (CR) are referenced in the PWS and the Quality Assurance Methods section of this document. These requirements will be in effect at the start of the contract performance. CR-1. Material Management. Refers to the total effort involved in operational and material management (PWS Section 2.2). CR-2. Issue Material. Refers to the activities involved in supplying material to customers in response to a customer request (PWS Section 2.2.6). CR-3. Hazardous Waste (HAZWASTE) Services/Transfer Hazardous Material to Waste Management Office. HAZWASTE Services refers to the process of disposing of HAZMAT that is no longer in a usable condition. Transfer Hazardous Material to Waste Management Office refers to the process of preparing SP-generated waste for the disposal process. It does not include the actual disposal of the waste (PWS Section 2.2.5.6 and 2.3.11).. CR-4. Store Material. Store material refers to the activities involved in maintaining the integrity of any HAZMAT inventory (PWS Section 2.2.4). Quality Assurance Surveillance Plan: CNRSW HAZMAT Q-4
4.0 PART II. QUALITY ASSURANCE METHODS 4.1 Overview This section discusses the risks and benefits of acceptable quality assurance surveillance methods. The QAE(s) will recommend one or more of the surveillance methods listed below for use on each of the tasks within the functional areas defined in the Technical Exhibits of the PWS. The allocation of the QAE(s) will be based on the QAE(s) technical expertise and experience, as well as the requirements of the contract and the performance of the SP. 4.2 One Hundred (100) Percent Surveillance This is usually the most appropriate method for infrequent tasks or tasks with stringent performance requirements, e.g. where safety, health, or national security is a concern. With this method, performance is inspected/evaluated at each service occurrence. The results of the SP s overall performance are then evaluated to determine acceptability of the lot. This is the most costly method of surveillance and cannot be performed in all cases due to resource constraints. 4.3 Management Information Systems Existing Management Information Systems (MIS) can be used for surveillance. The results are similar to 100 percent inspection in that the QAE(s) can check every reported occurrence of the service output (i.e., through the use of tracking reports and audits of databases). The use of a MIS differs, however, in that there is no physical inspection of the output. Instead, the collected information is compared to a contract standard, and on the basis of the comparison performance can be accepted or rejected. To be usable in a QASP, the information in the MIS must describe the SP s output or performance rather than the methods of performance. 4.4 Planned Sampling This method, sometimes called periodic sampling, consists of the evaluation of tasks selected on other than a 100 percent or random basis. This method may be appropriate for tasks that occur infrequently, and where 100 percent inspection is neither required nor practicable. 4.5 Random Sampling This is usually the most appropriate method for recurring tasks. While 100 percent inspection leaves little margin for error, it is too expensive to be used in most cases. With random sampling, services are sampled to determine if the level of performance is acceptable. Computer programs are available to assist in establishing sampling procedures. For tasks surveyed by random sampling procedures, random selection will be generated by either a statistically valid computer program or by using a random number table. When selecting statistically valid random samples, the number of selections should exceed the required sample size by at least 10 percent to allow for situations where a selected sample cannot be evaluated. Replacement samples must be used in the order drawn. See Table 3, Suggested Sample Size. When random sampling is the method of surveillance, any customer complaints or observed unacceptable performance by the SP, other than during a sample observation, shall not be counted as defects. These complaints and observations should be noted and reinforce the validity of the sample results. Only defects discovered during sample observations will be counted. Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-5
4.6 Customer Complaints Although usually not a primary method of surveillance, customer complaints are valuable supplements to more systematic methods. For example, in a case where random sampling indicates unsatisfactory service, customer complaints can be used as substantiating evidence. In certain situations where customers can be relied upon to complain consistently when the quality of performance is poor, (e.g. dining facilities), customer complaints may be a primary surveillance method, and customer satisfaction an appropriate performance standard. In all cases, complaints should be documented on a standard form. The KO will determine what action to take if the SP continuously receives customer complaints for the same services. The QAE(s) should provide relevant documentation to the KO to aid in remedial action. 4.7 Customer Satisfaction Measuring customer satisfaction is different than keeping track of and responding to specific customer complaints. To measure customer satisfaction, and to use it from a quality assurance surveillance perspective, the survey must be administered in a statistically valid manner. Nonscientific surveys that are developed and administered locally should not be used for quality assurance purposes. While they may be extremely useful to local management, they are not valid for use in award fee determination or deduct analyses. Since HAZMAT operations are customer focused, using a statistically valid customer survey methodology is a useful method of quality assurance. The idea is to develop a survey instrument that captures customer views about various aspects and the overall level of their satisfaction with the service that they receive. By conducting the survey of a statistically valid sample of customers, the quality assurance organization can establish a baseline of customer satisfaction levels. Then, by periodically repeating the survey, the level of customer satisfaction can be compared to the baseline thus providing a comparison of customer satisfaction over time. Maintaining or improving the level of customer satisfaction would be the goal of meeting the performance requirement. 4.8 Combining Methods of Surveillance The various methods of surveillance are not mutually exclusive. Indeed, if more than one factor is pertinent to a specific function or individual task, they should be considered in the selection of the surveillance method(s). 4.9 Overview of Surveillance Methods by Functional Area Table 1, Performance Requirements Summary, provides the QAE(s) with an effective method to monitor the activities of the SP. The purpose of the inspection process is to measure SP performance against specified performance standards to assure acceptable work accomplishment. Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-6
Performance Requirement CR-1 Manage HAZMAT CR-2 Issue HAZMAT CR-3 HAZWASTE/Transfer material to Waste Office Table 1 Performance Requirements Summary Performance Measure 1. All HSMS scheduled inventory adjustments shall be authorized before entering into system 2. All unscheduled HSMS adjustments shall be reported to the COR within five working days 3. Greater than or equal to two stock turns for Government-owned inventory per contract year starting from date of contract award 4. Total cost value of A-condition materials provided to customers IAW stated SP pricing methodology 1. Customer satisfaction in order fulfillment as indicated by customer survey results 2. Orders are fulfilled according to customer established priorities 3. Book value of C/A material issued is > 30% of total cost value issued 1. Greater than 35% reduction in total SP generated hazardous waste per contract year starting from date of contract award using Government provided baseline 2. No Notification of Violations (NOV) resulting from external agency operational inspections (OSHA, EPA) or Major violations resulting from internal operational inspections 3. No more than three Notification of Findings (NOF) resulting from external agency operational inspections or Minor violations resulting from internal operational inspections 4. NOF resulting from external agency operational inspections or Minor violations resulting from internal operational inspections shall not be repeated over the contract period of performance Method of Frequency Acceptable Error Rate Surveillance 100% Surveillance As Required 0% Random Sampling Monthly 0% Stratified Sampling Monthly 25% Stratified Sampling Monthly 5% 100% Surveillance supplemented by customer complaints Semi-Annual 20% Random Sampling Quarterly 10% 100% Surveillance Monthly 33.3% 100% Surveillance Monthly 15% 100% Surveillance Each Inspection 0% 100% Surveillance Each Inspection 0% 100% Surveillance Each Inspection 0% CR-4 Store Material 1. HAZMAT inventory accuracy level Random Sampling Monthly 5% Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-7
4.10 Contract Administration Resources 4.10.1 Contract Administration Staff Contract Administration staff will be required to administer the contract, which involves compliance reviews, payment processing, and monitoring of contract operation closeout. The appropriate number of contract administration staff is recommended by OMB Circular A-76, Revised Supplemental Handbook, March 1996, and is shown in Table 2, Recommended Size of Contract Administration Organization. Table 2 Recommended Size of Contract Administration Organization MEO Staffing Contract Administration Staff (FTEs) 10 or less.5 11-20 1 21-50 2 51-75 3 76-100 4 101-120 5 121-150 6 151-200 7 201-250 8 251-300 9 301-350 10 351-450 11 401-450 12 451 & Above 2.5 percent of in-house MEO staffing *According to the OMB Circular A-76, the CNRSW HAZMAT MEO is allowed 3 FTE for contract administration 4.10.2 Quality Assurance Evaluators The quality assurance organization exists under the authority of the KO. The Contracting Officer s Representative (COR) works for the KO and is the day-to-day person responsible for the operation of the contract. Under the COR are the QAEs. The QAE(s) must be fully qualified to meet the major responsibilities of the position: maintaining complete and accurate documentation, a good working relationship with the SP, thorough knowledge of the contract requirements, expert knowledge of the area under surveillance, and an understanding of contracting procedures. The COR will appoint QAE(s) by letter with a copy provided to the KO. The QAE(s) must be objective, fair, and consistent in evaluating SP performance against the standards. The QAE(s) will not have the authority to make legal interpretations, impose tasks not in the contract, supervise SP employees, or waive contract requirements. Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-8
5.0 PART III. QUALITY ASSURANCE SURVEILLANCE SCHEDULE 5.1 Quality Assurance Evaluator Schedule QAE(s) will develop a monthly surveillance schedule based on the surveillance plan s contract requirements. The monthly schedule (i.e., planned surveillance schedule) will be completed no later than the 25th of the preceding month. Copies of the schedule shall be sent to the KO and COR. The schedule shall be marked FOR OFFICIAL USE ONLY and shall not be shown to the SP. Changes to the monthly surveillance schedule will be posted weekly and copies sent to the KO and COR. QAE(s) shall document the reason for the changes. It may not be possible to make changes to the extent that samples might become statistically valid. It may be necessary to implement the original plan to work on the specific problem area with the KO and COR. 5.2 Actual Surveillance Actual surveillance should be comparable to the planned surveillance schedule with any deviations noted in writing. If less than the required number of inspections are made or inspections are made outside the randomly selected sample, the ability to project from random sample results is lost. If this should occur, the QAE(s) shall document the reasons the inspections were missed and consider the performance results for those missed inspections acceptable. This alternative increases the Government s risk of accepting a defective lot. Therefore, it is essential that the exact number of scheduled inspections be accomplished to maintain an effective quality assurance system. For surveillance utilizing the random sampling technique, QAE(s) shall reference Table 3, Suggested Sample Sizes, when determining the monthly lot and sample size for those specific tasks. The QAE(s) shall determine and adjust the sample size based on population size, criticality of the task, and the past performance of the SP. Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-9
Table 3 Suggested Sample Size Population Size Normal Sample Size Medium Sample Size 2-8 2 2 2 9-15 3 2 2 16-25 5 3 3 26-50 8 5 5 51-90 13 5 5 91-150 20 8 8 151-280 32 13 13 281-500 50 20 13 501-1,200 80 32 20 1,201-3,200 125 50 32 3,201-10,000 200 80 32 10,001-35,000 315 125 50 35,001-150,000 500 200 80 150,001-500,000 800 315 80 > 500,001 1250 500 125 Source: MILSTD105D Small Sample Size 5.3 Surveillance Documentation The QAE(s) shall document the results of the surveillance on the appropriate SP performance checklist, or customer complaint form. All discrepancies/defects will be annotated in a Contract Discrepancy Report (CDR). The CDR and the accompanying checklists and forms shall be submitted monthly to the KO. Table 4, Quality Assurance Evaluator Checklist, shows an example of a checklist. Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-10
Table 4 Quality Assurance Evaluator Checklist SP S PERFORMANCE CHECKLIST Checklist Ref. No.: Contract No. SP: Date (mm/dd/yy): Contract Name: Contract Requirement (CR): Work Requirements (Performance Indicators) AQL (Acceptable Quality Level) SP s Defect Rate Performance Rating Recommended Action Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-11
5.4 Contract Communication QAE(s) shall notify the SP s representative of any discrepancies and/or concerns. Although notification should be immediate, it should be no later than five (5) days after a defect is discovered. This notification will be annotated in writing and a copy maintained for the KO s records. Additionally, QAE(s) will continuously keep the KO informed of the SP s performance. 5.5 Revisions to the QASP Revisions to this surveillance plan are the joint responsibility of the COR and the KO. Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-12
6.0 PART IV. QUALITY CONTROL PLAN REVIEW 6.1 Quality Control Plan Inspection Procedure The Contract Administration Organization will obtain the SP s QC schedule not later than the 25th of the preceding month. This provides the QAE(s) a method to review the SP s QC Plan, past performance history, and the actual performance of services. At a minimum, the QC plan shall monitor the requirements presented in the PWS. Depending on all of these factors, the level of surveillance can be adjusted upwards if there are performance problems or downwards if performance is at or above the desired levels. Additionally, the QAE(s) will record defects in the QC Program for failure of the SP to recognize valid customer complaints and record unacceptable performance for contract requirements. In addition, the QAE(s) may also schedule a formal set of progress/performance review meetings between the SP, KO and COR. These meetings could be monthly, or more frequently if performance problems occur. The purpose of the meetings would be to discuss any performance problems from the prior period and review the SP s QC plan for the upcoming period. The idea is to establish a good working relationship between the Government and the SP with a focus on quality performance. If there are important planning issues or special requirements, these issues can also be discussed at these meetings. Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-13
7.0 PART V. SAMPLING PLANS Sampling Guide #1 CR-1: Manage HAZMAT 1. Method of Surveillance: 1. 100% Surveillance 2. Random Sampling 3. Stratified Sampling 4. Stratified Sampling 2. Acceptable Error Level: 1. 0% 2. 0% 3. 25% 4. 5% 3. Lot Size: TBD by Contract Administration 4. Sample Size: TBD by Contract Administration 5. Performance Measure: 1. All HSMS scheduled inventory adjustments shall be authorized before entering into system. 2. All unscheduled HSMS adjustments shall be reported to the COR within five working days. 3. Greater than or equal to two stock turns for Government-owned inventory per contract year starting from date of contract award. 4. Total cost value of A-condition materials provided to customers IAW stated SP pricing methodology. 6. Inspection Procedure: TBD by Contract Administration Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-14
Sampling Guide #2 CR-2: Issue HAZMAT 1. Method of Surveillance: 2. Acceptable Error Level: 1. 20% 2. 10% 3. 33.3% 1. 100% Surveillance 2. Random Sampling 3. 100% Surveillance 3. Lot Size: TBD by Contract Administration 4. Sample Size: TBD by Contract Administration 5. Performance Measure: 1. Customer satisfaction in order fulfillment as indicated by customer survey results. 2. Orders are fulfilled according to customer established priorities. 3. Book value of C/A material issued is >.3 of total cost value issued. 6. Inspection Procedure: TBD by Contract Administration Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-15
Sampling Guide #3 CR-3: HAZWASTE/Transfer Material to Waste Office 1. Method of Surveillance: 2. Acceptable Quality Level: 1. 15% 2. 0% 3. 0% 4. 0% 1. 100% Surveillance 2. 100% Surveillance 3. 100% Surveillance 4. 100% Surveillance 3. Lot Size: TBD by Contract Administration 4. Sample Size: TBD by Contract Administration 5. Performance Measure: 1. Greater than or equal to 35% reduction in total SP generated hazardous waste per contract year starting from date of contract award using Government provided baseline. 2. No Notification of Violations (NOV) resulting from external agency operational inspections (OSHA, EPA) or Major violations resulting from internal operational inspections. 3. No more than three Notification of Findings (NOF) resulting from external agency operational inspections or Minor violations resulting from internal operational inspections. 4. NOF resulting from external agency operational inspections or Minor violations resulting from internal operational inspections shall not be repeated over the contract period of performance. 6. Inspection Procedure: TBD by Contract Administration Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-16
Sampling Guide #4 CR-4: Store Material 1. Method of Surveillance: 1. Random Sampling 2. Acceptable Quality Level: 1. 5% 3. Lot Size: TBD by Contract Administration 4. Sample Size: TBD by Contract Administration 5. Performance Measure: 1. HAZMAT inventory accuracy level 6. Inspection Procedure: TBD by Contract Administration Quality Assurance Surveillance Plan: CNRSW-HAZMAT Q-17