QUALITY ASSURANCE SURVEILLANCE PLAN TABLE OF CONTENTS 1.0 INTRODUCTION PART I QUALITY ASSURANCE REQUIREMENTS PLAN... 3

Transcription

1 QUALITY ASSURANCE SURVEILLANCE PLAN TABLE OF CONTENTS 1.0 INTRODUCTION Quality Assurance Program Goal Concept of Quality Assurance Scope of (QASP) Three Part QASP Quality Assurance Requirements Plan (QARP) Quality Management Assessment Plan (QMAP) Quality Assurance Surveillance Implementation Plan (QASIP) Relationship of QASP to the Contract PART I QUALITY ASSURANCE REQUIREMENTS PLAN Basic Principles Contract Administration Responsibilities Contracting Officer (KO) Contracting Officer s Representative (COR) Quality Assurance Evaluators Assessment of Quality Needs Critical Requirements Requirements Routine Requirements Contract Performance Requirements Allowable Deviation Concept Documentation of Deviation Substantially Complete Work PART II QUALITY MANAGEMENT ASSESSMENT PLAN Assessment Methodology Inspections One Hundred Percent Inspection Planned Sampling Special Inspections Random Sampling Customer Complaints Customer Complaint Program Automated Customer Complaints/ Inspections of Requirements Surveillance of PWS Requirements Q- i

2 3.4.1 Surveillance of Critical Requirements Surveillance of Requirements Surveillance of Routine Requirements Analysis of Surveillance Results Analysis Using 100 Percent Inspections Analysis Using Planned Sampling Analysis Using Customer Complaints Interpretation of Results of Analysis Excellent Performance Satisfactory Performance Unacceptable Performance PART III QUALITY ASSURANCE SURVEILLANCE IMPLEMENTATION PLAN (QASIP) Surveillance for Critical Requirements Surveillance for Requirements Surveillance of Routine Requirements Supporting Documentation Inventory of Services Worksheet Monitoring Guide and Inspection Checklists Master Inspection Schedule Attachments Q- ii

3 LIST OF FIGURES Figure 1. Contract Discrepancy Report (CDR) Figure 2. Quality Assurance Inventory of Monitoring Requirements Figure 3. Quality Assurance Master Inspection Schedule Figure 4. Analysis of Performance Worksheet Figure 5. Performance Monitoring Guide (PMG) 1 Generating Standard Reports Figure 6. Performance Monitoring Guide 2 Individual Processing Report Figure 7. PMG 3 ESR Construction, Management, and Accountability Figure 8. PMG 4 Uniformed Services Identification Cards Figure 9. PMG 5 Divisional In-Processing Figure 10. PMG 6 Electronic Personnel Security Clearance Requests Figure 11. PMG 7 Leave Management Figure 12. PMG 8 Unauthorized Absence (UA) and Deserter Personnel Actions Figure 13. PMG 9 Transient Tracking Figure 14. PMG 10 Disciplinary Reporting and Documentation Figure 15. PMG 11 Recruit Out-Processing Figure 16. PMG 12 Permanent Change of Station (PCS) Orders Management Figure 17. PMG13 Recruit Transfer Figure 18. PMG 14 Recruit Separation Figure 19. PMG 15 Recruit Pay Processing Q- iii

4 1.0 Introduction PSD RTC Great Lakes QUALITY ASSURANCE SURVEILLANCE PLAN 1.1 Quality Assurance Program Goal The goal of the Personnel Support Activity Detachment, Recruit Training Command, Great Lakes (hereinafter referred to as PSD) quality assurance program is to ensure that the Recruit Training Command, Naval Training Center, Great Lakes, Illinois (hereinafter referred to as RTC) receives Personnel and Pay Administrative Support Services (PASS) as directed by the Commander-in-Chief, U.S. Atlantic Fleet, and is satisfied with products and services delivered by the Service Provider. 1.2 Concept of Quality Assurance The Government will conduct systematic surveillance of all aspects of this contract to ensure that either the in-house Government or commercial contractor activity is fulfilling all requirements of the performance work statement (PWS). However, the Government s quality assurance program is not a substitute for the Service Provider s quality control program. The Government expects the Service Provider to detect and correct deficiencies before products and services are delivered to customers. The Government will rely on the Service Provider s quality control program, which shall be part of each commercial offeror s technical proposal and part of the Government s technical performance plan. The quality control plan shall contain the Service Provider s methodology for ensuring the delivery of high quality products and services in accordance with the requirements of Section C.5 of the performance work statement. The Service Provider shall perform quality control inspections based upon the items listed in the PWS, the Performance Requirements Summary (PRS) and upon receipt of customer feedback. The Service Provider s quality control plan shall contain checklists for all items to be inspected. Customer feedback is an essential element of quality control. The Service Provider shall develop a customer complaint form that may be transmitted via electronic mail to the Service Provider s quality control manager and to the Government s quality assurance evaluator (QAE). The Service Provider is responsible to produce, maintain, and provide for audit of all quality control records and reports and all records associated with the investigation and resolution of customer complaints. 1.3 Scope of (QASP) The PSD quality assurance program is designed to provide comprehensive and systematic surveillance of all requirements of the PSD contract. This QASP describes the procedures the Government will use to monitor the Service Provider s compliance with the requirements of the PWS. In some cases, specific requirements will be the basis for this measurement, in other cases subjective judgment and evaluation by Government personnel will be the Q- 1

5 determining measurement. This plan describes the methodology necessary to make quantitative and qualitative evaluations of Service Provider performance under the contract. Its requirements apply to the Service Provider, regardless of whether the service Provider is a contractor or a Government entity. 1.4 Three Part QASP This is divided into three parts: Part I Part II Part III Quality Assurance Requirements Plan Quality Management Assessment Plan Quality Assurance Surveillance Implementation Plan Quality Assurance Requirements Plan (QARP) The QARP is based upon the PWS. It contains an overview and general assessment of the elements relating to the quantity and quality needs of the PWS services and products. It refers to information in the PWS used to measure success in accomplishment of mission requirements and customer expectations Quality Management Assessment Plan (QMAP) The QMAP is tailored to the actual needs of the project. It provides an assessment of the risks and benefits, including a comparative assessment of cost associated with the use of potential surveillance methods. The Contracting Officer will use information in the QMAP to decide upon the level of quality assurance resources to fund and the amount of risk the Government is willing to assume Quality Assurance Surveillance Implementation Plan (QASIP) The QASIP is tailored to the levels of surveillance described in the QMAP. It takes into consideration the Service Provider s quality control plan and performance history. The QASIP is the actual working document that guides the quality assurance evaluator (QAE) in evaluating and documenting Service Provider performance. 1.5 Relationship of QASP to the Contract This QASP is not part of the solicitation nor will it be made part of any resulting contract. The Government will provide the successful Service Provider an information copy of the QASP to enable the Service Provider to enhance its quality control program that interrelates with the Government s QASP. The Government will retain the right to change the surveillance methods and quality assurance procedures as well as to increase or decrease the degree of surveillance efforts at any time necessary to assure contract compliance. Q- 2

6 QUALITY ASSURANCE REQUIREMENTS PLAN 2.0 Part I Quality Assurance Requirements Plan 2.1 Basic Principles While the Service Provider is inherently and ultimately responsible for the performance and quality of all products and services specified in the contract, the Government must be able to confirm that the quantity and quality of goods and services received conform to contract requirements. Government quality assurance procedures evaluate the products and services delivered by the Service Provider. These products and services may result either from Service Provider-developed procedures or from Government-specified procedures. When the output is based on Service Provider procedures, the procedure is examined on an exception basis; that is, satisfactory service output as specified in the contract normally indicates that the Service Provider is using satisfactory procedures. When the Government specifies the procedure, compliance with the procedure is the desired output. 2.2 Contract Administration Responsibilities This section of the QARP briefly defines the responsibilities of key personnel involved in contract administration and quality assurance: the Contracting Officer; the Contracting Officer s Representative (COR), and the Quality Assurance Evaluator (QAE) Contracting Officer (KO) The Contracting Officer has the authority to administer the PSD contract. The KO may delegate many of the day-to-day contract administration duties to the COR and QAE(s). However, certain contractual actions such as certification of invoices, negotiation and issuance of contract modifications, resolution of Service Provider claims and disputes, issuance of Contractor Deficiency Reports, cure notices, or show-cause letters, termination of the contract, and contract close-out are retained by the KO Contracting Officer s Representative (COR) The COR represents the Contracting Officer and functions as the Service Provider s point of contact. The COR serves as the day-to-day manager of all or a specified portion of the contract. As such the COR may be the manager or the Officer-in-Charge of the residual organization. The COR supervises the QAE(s). If modifications to the contract are necessary, the COR will assist the Contracting Officer in preparing and negotiating the modifications. If there are problems with Service Provider performance, the COR must inform the Service Provider of the problems and recommend to the Contracting Officer what Q- 3

7 adverse contractual actions should be taken (such as Contractor Deficiency Report or issuance of a cure notice). The COR must coordinate matters of contract interpretation with the Service Provider and the Contracting Officer Quality Assurance Evaluators The QAE plays a key role in contract administration. They serve as the eyes and ears of the Contracting Officer and COR. The QAE performs the actual contract surveillance and reports to the COR. The QAE performs surveillance as required by this QASP and makes recommendations to the COR for issuance of contract discrepancy reports or letters of commendation. The QAE makes recommendations to the COR for the validation of satisfactorily completed work and for administrative actions based on unsatisfactory work or non-performed work. The QAE assists the COR in identifying necessary changes to the contract, preparing Government estimates, conducting quality assurance meetings, approving submittals, and maintaining work files. The QAE furnishes the COR with any requests for changes, deviations, or waivers to the contract Limitations on the Authority of the QAE The QAE only has the authority delegated by the Contracting Officer. A QAE does not have the authority to allow the Service Provider to deviate from contract requirements, to direct the Service Provider s methods of performance, or to issue modifications directly to any of the Service Provider s personnel unless methods being used are unsafe Standards of Behavior for QAE QAEs must conduct themselves at all times in a manner commensurate with the responsible positions they occupy. The QAE should maintain cordial but professional working relationships with the Service Provider s management and personnel. However, the QAE must guard against undue friendliness or social association. When possible, the QAE should work only with the Service Provider s designated representative at the site. Sometimes, the QAE will also have to work with various subordinates and subcontractor representatives. The QAE should ensure that all such contacts are made with the full knowledge and consent of the Service Provider. At the start of the contract, the QAE should make clear to the Service Provider that the QAE intends to be strict about compliance with the contract but reasonable in interpretation of subjectively judged requirements. Although the QAE intends to maintain a good working relationship between Service Provider personnel and themselves, the QAE will not let anything divert them from faithful protection of the Government s interests. Q- 4

8 Although the QAE places full confidence in the Service Provider, that confidence must be continually renewed by demonstrated performance Prohibition Against Gratuities or Favors The QAE is prohibited by law from accepting any gratuities or favors of any nature whatsoever either directly or indirectly from any Service Provider. The Service Provider should be made aware of the pertinent statutes and legal action that can be taken for violations. 2.3 Assessment of Quality Needs All requirements are assigned one of three priority rankings: critical, significant, and routine Critical Requirements Critical requirements are mission essential. The Government has an interest in knowing with a high degree of confidence that critical requirements are performed properly. Therefore, the Government will inspect for contract compliance from the beginning to the end of the contract for all critical requirements using either 100 percent inspections, or planned sampling. In some instances, when outcomes are readily apparent to the customer, customer complaints may be used. An example of a critical requirement would be to begin paying Recruits on time Requirements requirements are also important and may require intervention by the Government to demand a higher standard of quality control on the part of the Service Provider. The Government must be assured the Service Provider is knowledgeable and capable of performing the significant requirements but can give some latitude if there is a high degree of confidence that the Service Provider is performing well. The Government will therefore begin with either 100 percent or planned inspections. If the Service Provider meets all expectations, the Government may scale back its efforts and rely on customer complaints to identify any future quality problems. However, if the number of validated customer complaints rises to an established threshold, the Government will resume its previous inspections. An example of a significant requirement would be construction of the Enlisted Service Record not later than close of business on the 1-1 day of training Routine Requirements Routine requirements should account for the majority of the contract effort. While the Government still requires quality work from the Service Provider, the Government also wants to conserve scarce resources by not performing inspections when the Service Provider proves to be doing a good job. PSD has a unique situation in that virtually all of their customers are Recruits. Because of their status and inexperience, the Recruits will be less vocal about complaints than would be a similar population of Sailors with more time in the Navy. Eventually, there Q- 5

9 will be complaints from the Recruit Division Commanders when the situation has deteriorated significantly. To avoid reaching this threshold, the Government will begin the contract performance period by relying on planned inspections of routine requirements as well as customer complaints. Only when the Government gains confidence that the Service Provider is providing the desired quality level will the Government revert to customer complaints as its primary means of monitoring routine contract performance requirements. However, the Government will still conduct infrequent planned inspections on a schedule determined by the Government. Such monitoring will continue so long as the number of validated customer complaints remains low. If a pre-determined threshold of validated customer complaints is exceeded, the Government will step up its quality assurance efforts by conducting special inspections of product outputs and services. The Government will also examine the Service Provider s quality control program to determine why the quality level has dropped. Only when the Government regains confidence that the Service Provider has corrected its quality problems will the Government revert to customer complaints as its primary means of monitoring this contract performance requirement. An example of a routine requirement would be the suppression of the pay for Recruits that have been on unauthorized absence for ten days. 2.4 Contract Performance Requirements Contract performance requirements are shown in the Performance Work Statement (PWS). The Performance Requirements Summary (PRS) summarizes the requirements of the PWS. The PRS shows the PWS paragraph number and requirement, the standard, the method of surveillance, quality ranking category, and the allowable deviation from the standard Allowable Deviation Concept Even though the Government is paying to have all work performed as specified, on time, all the time, there will be occasional errors and omissions, referred to as defects, on the Service Provider s part. The Service Provider shall be held responsible for all defects. However, there is obviously a difference between an occasional defect and a gross number of defects. Allowable deviation is a tool used to distinguish between satisfactory performance (few defects) and unsatisfactory performance (excessive defects). Depending on the requirement evaluated and the evaluation method selected, allowable deviation may be stated as the number of occurrences or as a percentage Allowable Quality Level (AQL) For the purposes of this QASP, the allowable deviation is called the allowable quality level for 100 percent inspections and planned sampling. The AQL is expressed as a percentage of defects. The AQL only applies when inspections are conducted for critical requirements, significant requirements at the beginning of the contract performance period, and special inspections. Q- 6

10 Complaint Threshold PSD RTC Great Lakes The allowable deviation for significant requirements is expressed as both an allowable quality level and a threshold level of validated customer complaints. The allowable deviation for routine requirements is expressed as a threshold level of validated customer complaints. Once the threshold is reached, the Government will increase surveillance of the requirement. The complaint threshold only applies to routine and significant requirements since critical requirements will always be subject to inspections Documentation of Deviation Thorough documentation of unperformed or poorly performed work is essential. The QAE must document unperformed work or poorly performed work by compiling facts describing their inspection methods and results. The COR and the QAE will then document nonconformance with the contract. The documentation, together with any recommendations, will then be forwarded to the Contracting Officer. Only the Contracting Officer can make decisions on administrative actions Substantially Complete Work Work is considered substantially complete where there has been no willful departure from the terms of the contract and no omission of essential work. The Service Provider has honestly and faithfully performed the work required and the only variance consists of minor omissions or deficiencies. In general, work is substantially completed when percent or more is satisfactorily completed. The percentage selected depends upon the type of work performed. This concept relies somewhat on subjective judgment, however, and there are no clear guidelines established. Q- 7

11 QUALITY MANAGEMENT ASSESSMENT PLAN 3.0 Part II Quality Management Assessment Plan 3.1 Assessment Methodology The Government will evaluate the quality of services delivered under the provisions of this contract via inspections conducted by the Government and customer complaints. The specific evaluation technique employed will be contingent upon the assigned priority ranking of the requirement and the Service Provider s performance track record. 3.2 Inspections The most comprehensive means of determining whether the Service Provider has met all contract requirements is to inspect the results. Such inspections confirm that value is received for the money or that problems exist. The principal disadvantage of inspections is that they are resource intensive and, therefore, are more costly than other methods of surveillance. Inspection should be used for those items that are critical, where defective service may not be readily apparent to customers, and when the monitoring of customer complaints indicates a quality problem with the service One Hundred Percent Inspection One hundred percent inspections require complete inspection of a contract requirement. One hundred percent inspections should be used for requirements that are especially critical or where there is some reason for suspecting that the performance standard is not being met. One hundred percent inspection may also be used for monitoring scheduled contract requirements (such as one-time deliverables, scheduled submissions, and infrequently identified requirements). One hundred percent inspections are the most costly; however, they provide positive proof of delivery of critical requirements Planned Sampling Planned sampling provides a systematic way of looking at service output and forming conclusions about the Service Provider s level of performance. Evaluation by planned sampling is designed to inspect some part but not all of the products and services being monitored. Specific occurrences of contract requirements that are to be monitored are selected for evaluation prior to their scheduled accomplishment. Sample selection is based on a subjective rationale and sample sizes are usually arbitrarily determined. With this type of evaluation, the Service Provider knows that work performed at selected locations is more likely to be monitored than work at other locations, and the QAE is able to direct efforts to those areas where inspection is most needed. Planned sampling, unlike random sampling, does not provide a sound statistical means of making comparisons between observed and overall performance, and the Service Provider s overall level of performance cannot be determined. Q- 8

12 The cost of planned sampling varies with the level of inspections. Such latitude is important to manage limited resources and focus inspections on known or suspected problems areas Special Inspections Special inspections consist of ad hoc evaluations of contract requirements whenever customer complaints indicate that the Service Provider is not delivering a quality service. This method is very similar to planned sampling except for the omission of a pre-announced schedule. The costs of special inspections are usually higher than for regular planned sampling because they are used in reaction to identified quality problems Random Sampling Historically, random sampling was the predominant type of inspection. It was used not only to determine contract compliance, but it was also used to assess deductions. Current Navy thinking is to forgo deductions and to focus on improving quality control in the work processes used to produce products and services. This QASP does not intend to use random sampling as an inspection technique. However, random sampling may be employed if the KO determines it to be necessary. There are two ways of applying random sampling for quality assurance surveillance. The first is used only for performance evaluation and allows deductions to be taken only for observed defects. The second is random sampling for performance evaluation and extrapolated deductions. The second method allows deductions against the whole population based on the inspection of the sample. The two procedures follow the same steps up to the point of determining if performance has been satisfactory or unsatisfactory. Problems may arise if Government inspectors fail to follow random sampling procedures precisely. 3.3 Customer Complaints Quality assurance resources to inspect the performance of service are extremely scarce and costly. Consequently, they must be used efficiently. To achieve a cost-effective balance between the cost of inspections and achieving reasonable confidence that the Service Provider is meeting the contract requirements, this QASP intends that service quality will be primarily monitored through customer complaints. Customer complaints both traditional and automated, will be the chief means of monitoring routine requirements and significant requirements after the Government is satisfied that the Service Provider is delivering quality work. The customer complaints will be used to monitor quality and provide an indication that quality problems may exist. When customer complaints reach the established threshold, the QAE will conduct a special inspection to document the problem Customer Complaint Program The Service Provider shall be required to establish a customer complaint program. At the beginning of the contract, the COR will send letters to the RTC, Naval Education and Training Command (NETC) Operations (N3), and receiving Personnel Support Activities. These letters will inform them of the quality assurance program and solicit Q- 9

13 their active participation. They will also include a customer complaint form for them to use. It is the responsibility of the Service Provider to record, research, and resolve each complaint. After the complaint has been resolved, the Service Provider shall complete the complaint form and forward it to the QAE Automated Customer Complaints/ Inspections of Requirements The PSD is unusual in that input to the major Navy personnel and pay systems, Navy Standard Integrated Personnel System (NSIPS) and Master Military Pay Account (MMPA), must be accepted by the respective automated system. If the entry is inaccurate, it will be rejected. Rejections must be corrected (in accordance with the timeframes specified in the PWS) in order to process/pay Recruits. Therefore, rejections, per se, become a de facto customer complaint. These automated customer complaints may pertain to all levels of requirements: critical, significant, and routine. NSIPS and EARS calculate rejection rates on a daily and cumulative basis. The Government will use the NSIPS and EARS statistics in deciding whether or not to conduct unplanned inspections of functional areas subject to automated customer complaints. In making this decision, the Government will coordinate with the respective NSIPS/MMPA functional manager to determine if the rejections are attributable to systemic error or the result of Service Provider error. 3.4 Surveillance of PWS Requirements Surveillance of Critical Requirements Surveillance of critical requirements will be either via 100 percent inspections or planned inspections. Initially, the Government will conduct 100% inspections of the critical requirements identified in the PRS. If the Service Provider demonstrates the capability to furnish high quality products and services, the QAE may revert to planned inspections. Based on the Service Provider s demonstrated performance, the QAE may lengthen the time between planned inspections. Conversely, if the Service Provider s performance is below expectations, the QAE will document findings to the Contracting Officer for possible administrative action. The QAE may employ increased surveillance until the Service Provider demonstrates that the quality control program is again within allowable deviation limits. When inspections reveal that the AQL has not been exceeded for at least two months in a row, the QAE may return to normal surveillance Surveillance of Requirements requirements are important, though not as vital as critical requirements. Surveillance will initially use planned inspections to determine the Service Provider s ability to comply with the contract requirement. The QAE will follow inspection guides and checklists. If the AQL is exceeded, the QAE will forward the findings to the Contracting Officer for possible administrative action. The QAE will then employ increased surveillance to Q- 10

14 assess Service Provider performance. As with critical requirements, the Service Provider must then show improved quality for at least two consecutive months in order for the QAE to revert to normal surveillance. Unlike the procedures for critical requirements, the QAE may determine that if the Service Provider performs well over time (at least two consecutive months), inspections may be suspended. The Government will then rely upon customer complaints to assess compliance with the contract. The Government will continue to rely upon customer complaints until the number of validated customer complaints reaches the specified threshold, whereupon the QAE will return to inspections Surveillance of Routine Requirements Routine requirements have lower priorities than critical or significant requirements and are monitored from the beginning through customer complaints where possible. Provided that the number of validated customer complaints remains below the established threshold, the Government will continue to rely upon customer complaints as defined in paragraphs 3.3 through above. If, however, the number of customer complaints exceeds the established threshold for a product or service, the QAE will begin conducting special inspections. If the special inspections reveal no problems or if subsequent planned inspections are satisfactory over a two-month period, the QAE may revert back to customer complaints. If, however, the special inspections reveal problems, the QAE will forward findings to the Contracting Officer. Planned inspections will continue as long as the QAE determines that quality problems exist. For those routine requirements where customer complaints would not provide a reliable indicator of performance, special inspections will be conducted periodically. Whenever a customer is not satisfied with the quantity, quality, or delivery time for a product or service, the customer will complete a customer complaint form or otherwise report the problem to the Service Provider and to the QAE. The Service Provider will then investigate the validity of the complaint and take appropriate action. If the complaint is valid, the Service Provider will correct the error or re-perform the needed actions and notify the customer. If the complaint is found not to be valid, the Service Provider will inform the customer as to why the complaint is not valid (e.g., a request out of scope of the contract) and provide a copy of the response to the QAE. The Service Provider shall be required to forward a copy of all completed customer complaint forms to the QAE assigned to the site. The QAE will track complaints each month and will report any negative trends to the COR. The QAE will also report on a monthly basis the findings of special inspections. 3.5 Analysis of Surveillance Results Analysis Using 100 Percent Inspections One hundred percent inspection requires that every occurrence of a performed service be monitored. Contract requirements subject to evaluation by this method are those that occur infrequently, are essential, or are costly to perform. Evaluation schedules for 100 percent inspections will be prepared each month just as they are for planned sampling. Q- 11

15 Allowable Quality Level PSD RTC Great Lakes AQL may be stated as either a percentage or a number Analysis Observed defects for a service monitored by inspection will be totaled at the end of each month. The total number of defects is divided by the population size. This calculation gives the actual defect rate Performance Evaluation The computed defect rate is compared to the AQL. If the defect rate is greater than the AQL, the Service Provider s overall level of performance for that contract requirement is classified as unsatisfactory. If the observed defect rate is equal to or less than the AQL, the overall level of performance for the item evaluated is classified as satisfactory Analysis Using Planned Sampling Surveillance based on planned sampling is a more subjective process than random sampling. Planned sampling is generally used in one of two ways. First, it can provide a one-time subjective evaluation of Service Provider performance. Second, it can be used to detect a change in the level of performance (i.e., trend analysis). This method requires that the sample selection criteria be well documented and consistently applied from period to period and that there are no other intervening factors Allowable Quality Level The AQL is usually stated in terms of the number of defects detected per unit time period (for example, three times per month). There is no specific relationship between sample size and AQL. However, when the AQL is expressed as a percentage, the sample size should be chosen so that one defect does not exceed the AQL Level of Evaluation The levels of evaluation appropriate for planned sampling are judgmental. In order to perform trend analysis from planned sampling, criteria for sample selection should be applied consistently from period to period. The number of evaluations conducted may be reduced in those instances where the Service Provider has established a good performance record. In the case of poor performance, the Government may increase the level of evaluation, focusing on known problem areas. In either case, the reasons for the change should be documented Evaluation Procedure To ensure valid results, the QAE should use planned sampling evaluation sheets and follow a detailed inspection schedule. Schedules may be developed monthly to coincide with the Service Provider s monthly schedule of work Population Size The first step in this procedure is to define a unit of output for each service that is to be monitored by planned sampling. The unit of output should be the same as that defined in the PRS. The number of occurrences per month of the service is the population subject to inspection. Q- 12

16 Sample Selection PSD RTC Great Lakes The criteria to be used for sample selection should be documented. The rationale for criteria and sample size requirements should be stated. The sample size requirements are based on Service Provider performance. A larger sample size should be used for poor performance and a smaller sample size for good performance. Any impact on services that receive little or no evaluation must also be documented Worksheets Using an established selection criteria and the Service Provider s work schedule to determine when the performance of the service is scheduled, a list of locations equivalent to the sample size required is developed Analysis Observed defects for services monitored by planned sampling will be totaled at the end of each month. For each service, the total number of defects will be compared to the AQL. When the observed total number of defects is equal to or less than the reject level, the Service Provider s overall performance for the given service evaluated is satisfactory. When the observed total number of defects is greater than the reject level, the Service Provider s overall performance is judged to be unsatisfactory indicating further inspection or corrective action is required. Defect rates are monitored for a period of time to detect changes in Service Provider performance level Evaluation Evaluation of Service Provider performance using defect rates based on planned sampling is a subjective assessment. The AQL is used as a benchmark for planned sample results. That is, when the number of defects exceeds some specified number (for example, three) overall performance is considered unsatisfactory Analysis Using Customer Complaints Complaint threshold refers to the number of validated complaints received over a given time period (usually one month) Documenting Customer Complaints Each traditional customer complaint brought to the Service Provider (either in person, via , or by telephone) must be documented. Information about the complaint should be recorded on a customer complaint form. The Service Provider shall retain the forms for the entire contract period. All customer complaints and associated resolution shall be reported to the QAE Analysis The Service Provider shall summarize validated complaints each month by product or service. The NSIPS and EARS reports will also serve to record automated customer complaints. The results must never be combined with other evaluation results (100 percent inspections or planned sampling). If the assumption is made that customer awareness is similar from month-to-month, trend analysis can be used to test for variation in the number of complaints received each month. Q- 13

17 3.6 Interpretation of Results of Analysis Analysis of all types of contract monitoring will result in one of the following outcomes: excellent performance, satisfactory performance, or unacceptable performance Excellent Performance When there are very few or no deficiencies and the Service Provider has performed in the best possible manner, the Government may reduce its level of surveillance Satisfactory Performance The Service Provider s performance is satisfactory if the number of defects does not exceed the complaints threshold or AQL for planned sampling or 100 percent inspection. The QAE may suggest to the COR that an increased level of surveillance be used for important services which show defect rates approaching the AQL and that the Service Provider be notified that the performance is marginal Unacceptable Performance When the AQL for any service has been exceeded, the Service Provider s performance is unsatisfactory and is, therefore, unacceptable. The QAE should suggest one or more of the following actions: That the level of surveillance be increased until the Service Provider demonstrates acceptable performance over a period of time. That the Contracting Officer or COR meet with the Service Provider to discuss discrepancies, trends, and intended corrective measures. That the Contracting Officer issue a Contract Discrepancy Report (Figure 1) for each service that has exceeded its AQL. QUALITY ASSURANCE SURVEILLANCE IMPLEMENTATION PLAN 4.0 Part III Quality Assurance Surveillance Implementation Plan (QASIP) The QASIP is tailored in consideration of the level of quality assurance surveillance decided upon in the Quality Management Assessment Plan after receipt and review of the Service Provider s quality control plan and history of past performance. Q- 14

18 The QASIP is the actual working document that guides the QAE in documenting consistent achievement or betterment of the service quality requirements. 4.1 Surveillance for Critical Requirements The PRS identifies all critical requirements and the allowable quality level. The QAE will use either 100 percent inspections or planned inspections to monitor critical requirements. Customer complaints may be used to monitor some critical requirements where unsatisfactory performance would be immediately evident to the customer. The NSIPS and EARS reports may be used to monitor critical requirements which would be reflected as automated customer complaints. The QAE will notify the COR whenever monthly inspections reveal the Service Provider has exceeded an AQL. The QAE will take no reporting action if the AQL is not exceeded, other than to record results of all inspections. 4.2 Surveillance for Requirements The PRS identifies all significant requirements. At the beginning of the contract performance period, the QAE will monitor significant requirements using planned inspections. The QAE will notify the COR whenever monthly inspections reveal the Service Provider has exceeded an AQL. The QAE will take no reporting action if the AQL is not exceeded, other than to record results of all inspections. If the Service Provider does not exceed the AQL for two consecutive months, the Government may discontinue inspections and will instead rely upon customer complaints. Reliance on validated customer complaints will continue provided the number of validated customer complaints remains below the threshold level shown in the PRS. If the threshold level is exceeded for a month, the QAE will begin special inspections. 4.3 Surveillance of Routine Requirements Customer complaints will be the normal surveillance method of routine requirements. All PWS requirements not listed on the PRS will be considered routine requirements. Because of the nature of PASS services, some routine requirements may also be listed on the PRS. Provided the number of valid complaints is at or below the complaint threshold shown in the Inventory of Services Worksheet for Routine Requirements developed by the QAE, customer complaints will continue to be the means of monitoring the Service Provider s contract performance. If the number of validated customer complaints exceeds the complaints threshold, then the QAE will begin conducting special inspections (typically planned sampling). In addition, the QAE will examine the Service Provider s quality control procedures to determine whether there are problems identifying and correcting defects in the delivery of products and services. Special inspections will continue until the Service Provider has demonstrated for at least a two-month period that defects remain below the AQL percentage shown in the PRS. If there is a positive trend and confidence that quality Q- 15

19 problems have been corrected, the Government may revert to customer complaints as the means to monitor the Service Provider s performance for routine requirements. 4.4 Supporting Documentation In order for the QAE to effectively use any of the given evaluation methods, detailed procedures for documenting performance must be developed. While each QAE will develop their own QASIP, there are several supporting documents that may be useful in preparing a specific surveillance plan Inventory of Services Worksheet Each QAE should develop a Inventory of Services for the service requirements to be inspected and/or monitored at their site. This inventory is a simple list of services to be monitored in numerical sequence. The worksheet serves two purposes: (1) it provides a comprehensive listing of services required, and (2) serves as a tool to schedule the service requirements for inspection. The worksheet lists the assessment method, frequency, and AQL of each service requirement Monitoring Guide and Inspection Checklists A monitoring guide and inspection checklist should be developed for each requirement to be inspected. This document is completed by the QAE during an inspection. It shows the specific tasks to be checked and whether or not the inspection is satisfactory. The inspection checklist is the formal documentation for all Government quality assurance evaluations performed. It is used by the QAE to bring defects to the Service Provider s attention. All instances of non-compliance detected will require Service Provider initials on the original inspection checklist, indicating notification of the problem. Inspection checklist formats vary depending on the type of contract and the service performed Master Inspection Schedule Each contract requirement to be monitored will be scheduled on a regular basis, normally monthly, on a Master Inspection Schedule. This schedule is completed by the QAE and contains the service requirements and inspection frequencies indicated in the Inventory of Services. Once it is completed, the QAE accomplishes the evaluations using the inspection checklists. Evaluation schedules that indicate all contract requirement occurrences to be inspected on any given day should consider the Service Provider s work schedule. Use of the consolidated Master Inspection Schedule allows the QAE to prioritize inspections and identify when the inspection workload exceeds the availability of inspectors Attachments Examples of forms, performance monitoring guides, and other items that may be of use to the QAE are shown as Figures 1 through 19. Q- 16

20 Figure 1. Contract Discrepancy Report (CDR) 1. CONTRACT NUMBER DISCREPANCY REPORT NUMBER: 2. TO: (SERVICE PROVIDER & SUPERVISOR S NAME) 3. FROM: (NAME OF COR) DATES 4. PREPARED ORAL NOTIFICATION RETURNED BY SERVICE PROVIDER ACTION COMPLETE 5. DISCREPANCY OR PROBLEM: (DESCRIBE IN DETAIL-INCLUDE PWs REFERENCES. ATTACH CONTINUATION SHEET IF NECESSARY) 6. SIGNATURE OF QAE: 7. TO: (CONTRACTING OFFICER) FROM (SERVICE PROVIDER) Q- 17

21 8. SERVICE PROVIDER RESPONSE AS TO CAUSE, CORRECTIVE ACTION AND ACTIONS TO PREVENT RECURRENCE: ATTACH CONTINUATION SHEET IF NECESSARY. (CITE APPLICABLE QC PROGRAM PROCEDURES OR NEW QC PROCEDURES) 9. SIGNATURE OF SERVICE PROVIDER REPRESENTATIVE DATE: 10. GOVERNMENT EVALUATION: (ACCEPTANCE, PARTIAL ACCEPTANCE, REJECTION: ATTACH CONTINUATION SHEET IF NECESSARY) 11. GOVERNMENT ACTIONS (PAYMENT DEDUCTION, CURE NOTICE, SHOW CAUSE, OTHER) CLOSE OUT NAME TITLE SIGNATURE DATE SERVICE PROVIDER NOTIFIED QAE COR Q- 18

22 Figure 2. Quality Assurance Inventory of Monitoring Requirements GUIDE NUMBER PERFORMANCE REQUIREMENT ASSESSMENT METHOD FREQUENCY SAMPLE SIZE ALLOWABLE QUALITY LEVEL (AQL) 1 Generating Standard Reports Customer Complaints Continuous 2 per Month QC Monitor 2 Individual Processing Report Customer Complaints Continuous Zero 3 ESR Construction, Management, and Accountability 4 Uniformed Services Identification Cards Planned Sampling Weekly 1 Division 2 errors Planned Sampling Weekly 1 Division 1 error 5 Divisional In-Processing Planned Sampling Weekly 1 Division 2 errors 6 Electronic Personnel Security Questionnaire Requests Customer Complaints Continuous 1 per month 7 Leave Management Planned Sampling Monthly 10 Records 4% error 8 Unauthorized Absence (UA) and Deserter Personnel Actions Planned Sampling Monthly 10 Records 4% error 9 Transient Tracking Planned Sampling Weekly 10% of Transient Records 1% error 10 Disciplinary Reporting and Documentation Planned Sampling Monthly 10 Records 4% error 11 Recruit Out-Processing Planned Sampling Weekly 1 Division 0.5% error 12 Permanent Change of Station (PCS) Orders Management Planned Sampling Weekly 1 Division 0.5% error 13 Recruit Transfers Customer Complaints Continuous Zero 14 Recruit Separation Planned Sampling Monthly 50 Records 4% error 15 Recruit Pay Processing 100% Inspection Weekly 0.5% error Q- 19

23 Figure 3. Quality Assurance Master Inspection Schedule (Completed entry shown only for illustrative purposes) PREPARED BY: Performance Requirement Date and Guide Number 1 Generating Standard Reports PMG MONTH AND YEAR: Responsible Monitor or Inspector Site Operations Manager Signature Outcome: Sat or Unsat Q- 20

24 Figure 4. Analysis of Performance Worksheet TO: Contracting Officer s Representative CC: Service Provider Quality Control Manager Date: Please find following the results of an inspection/monitoring (type circled) conducted on: Date of Inspection 100% Inspection Scheduled Inspection Planned Sampling Customer Complaints Special Inspection 1. PWS Requirement: 2. Period of Performance: 3. Population: 4. Sample Size: 5. Report of Inspections: a. Number of Items Inspected: b. Number of Items Found Defective: c. Defect Rate (4b / 4a): % d. Allowable Quality Level (AQL): 6. Report of Customer Complaints Monitored: a. Number of Customer Complaints Validated: b. Allowable Quality Level (AQL): 7. Assessment of Performance: a. Satisfactory (Defect Rate less than or = to Line 5d or 6b): b. Unsatisfactory (Defect Rate = to or greater than Line 5d or 6b): Q- 21

25 Figure 4. Analysis of Performance Worksheet (continued) 8. Past Inspection Performance (Trend): Last Inspection Date and Number Defects = Two Inspections Past Date and Number Defects = Three Inspections Past Date and Number Defects = 9. Recommendations: (circle and add remarks) Increase Level of Surveillance to: Decrease Level of Surveillance to: Add Special Inspections of: Continue present level of surveillance. Send Letter of Appreciation to: 10. REMARKS Inspector/Monitor (printed): Inspector/Monitor Signature: Q- 22

26 Figure 5. Performance Monitoring Guide (PMG) 1 Generating Standard Reports Requirement: PWS Reference: Priority: Surveillance Method: Frequency: AQL: Generating Standard Reports C Routine Customer Complaints Continuous 2 per month Sampling Procedure: This task is a non-critical surveillance item and shall be inspected if a problem is discovered with the generation of standard reports to the DGR within specified timelines. The Quality Control Monitor (QCM) will inspect this requirement whenever customer complaints are received from the DGR, but may inspect this requirement at any time to confirm satisfactory performance. Inspection Procedure: Determine whether customer complaints regarding timely delivery of standard reports are valid or invalid. Determine whether or not standard report delivery delay was due to a condition within the Service Provider s control. Date and Location of Inspection: TASK Priority Method Frequency 1. Are recurring reports generated and Routine Customer Continuous delivered to the DGR within the Complaints specified timelines? OVERALL RATING Satisfactory / Unsatisfactory Inspector/Monitor Comments & Recommendations: Signature: Q- 23