QUALITY ASSURANCE PLAN OKLAHOMA DEPARTMENT OF HUMAN SERVICES ENTERPRISE SYSTEM (MOSAIC PROJECT)

Transcription

1 QUALITY ASSURANCE PLAN OKLAHOMA DEPARTMENT OF HUMAN SERVICES ENTERPRISE SYSTEM (MOSAIC PROJECT) MOSAIC Quality Assurance Plan v04.02

2

3 Prepared by: Approved by: QUALITY ASSURANCE PLAN APPROVALS QA/QC Program Quality Manager Quality Team Program Manager MOSAIC Project Sponsor MOSAIC Quality Assurance Plan v

4 TABLE OF CONTENTS 1.0 OVERVIEW General Information Purpose Scope QA Team Roles and Responsibilities Resources for QA Team QUALITY ASSURANCE TASKS QA Team Tasks and Matrix Conducting QA Team Tasks QA Schedule Evaluate Planning Oversight Evaluate Project Management Evaluate Quality Management Evaluate Training Evaluate Requirements Management Evaluate Operating Environment Evaluate Development Environment Evaluate Software Development Evaluate System and Acceptance Testing Evaluate Data Management Evaluate Operations and Business Oversight Evaluate Software Products Review Process Evaluate Component Deliverable (Release) Process Evaluate Media Certification, Storage and Handling Process Evaluate Non-Deliverable Software Certification Evaluate Performance Standards PROJECT DELIVERABLES QA Activities for Deliverables Walkthrough for Deliverables REVIEWS AND AUDITS Verify Document and Artifact Deliverable Review Verify Project Management Compliance Reviews Conduct Process Audits and Reviews Evaluation TESTING PROCESS AND ENVIRONMENTS System and User Acceptance Testing Processes System Environments Quality Control Testing Process VALIDATION AND ACCEPTANCE PROCESS Deliverable Acceptance Contractor's System Certification Acceptance Plans and Releases Federal Acceptance and Approval Preparation Food and Nutrition Service (FNS) Acceptance Requirements Health and Human Services (HHS) Acceptance Requirements 43 2 MOSAIC Quality Assurance Plan v04.02

5 7.0 ISSUE REPORTING AND CORRECTIVE ACTION Escalation Procedure for Resolution Disputes Corrective Action Process Recording Defects (Incidents) in Software Code or Documentation QUALITY METRICS Measurements Monitor and Control Trend Analysis Process Improvement Analysis 48 APPENDIX A. QA TEAM TRAINING 50 APPENDIX B. QA TASKS MATRIXES 51 APPENDIX C. PROCESS AUDIT AND REVIEW SCHEDULE 57 APPENDIX D. COMPLIANCY REPORT 60 APPENDIX E. SOFTWARE TOOL EVALUATION CHECKLIST 61 APPENDIX F. PERFORMANCE STANDARDS EVALUATION CHECKLIST 62 APPENDIX G. PILOT EVALUATION CHECKLIST 63 APPENDIX H. IMPLEMENTATION EVALUATION CHECKLIST 64 MOSAIC Quality Assurance Plan v

6 1.0 OVERVIEW 1.1 General Information. This Quality Assurance Plan (QA Plan) presents a framework for managing quality assurance activities, which when followed, will verify delivery of quality products and services for the Oklahoma Department of Human Services (OKDHS) Enterprise System (MOSAIC Project). The QA Plan will: Provide Project standards and procedures to be used as the basis for Quality Assurance (QA) Team reviews and audits Document how to plan, implement, and assess the effectiveness of QA and Quality Control (QC) operations Outline and reference QA and QC procedures used to evaluate overall Project and product performance, business and technical processes, Deliverables software, and traceability within all MOSAIC Project documentation Be implemented by the Project QA Team. Contractor will assist OKDHS staff throughout the lifecycle of the Project. 1.2 Purpose. Quality Assurance activities performed will verify that Project management and Project Deliverables are of high quality and meet quality standards as determined by the MOSAIC Project stakeholders. The purpose of the QA Plan is to: Define Project QA organization Identify QA tasks and responsibilities Implement quality standards and objectives to provide defect-free systems and business and technical processes that are delivered on time and within budget, and are maintainable Provide reference documents and guidelines to perform QA and QC activities Define standards, practices, and conventions used in carrying out QA and QC activities Provide tools, techniques, and methodologies to support QA and QC activities and reporting. 1.3 Scope. The QA Plan will establish QA and QC functions and processes for the lifecycle of the Project that conform with OKDHS approved QA standards and procedures and comply with state and federal laws and regulations Quality Management Scope will implement a Quality Management methodology and process that will verify all activities necessary to design, develop, implement, and utilize a product or service are effective and efficient with respect to the Enterprise System and its performance. 4 MOSAIC Quality Assurance Plan v04.02

7 1.3.2 Quality Assurance Scope will implement the Quality Assurance methodology and processes to verify that all the processes, methodologies, and technologies used for the Project and used to build and utilize the product perform as specified Quality Control Scope will verify and validate that Deliverables comply with quality standards and Project requirements, and are complete and correct QA and QC tools will be utilized to verify and track the Project lifecycle activities, create and track associated documents for those activities, and verify completeness of product and that it met the requirements through QC testing The QA Plan will ensure: Quality Control system development, evaluation and acceptance standards are developed, documented, and followed throughout the Project lifecycle QA Team records QC measurement results for use in reevaluating and analyzing quality standards and processes Results from software quality reviews and audits performed by the QA Team are submitted to QA/QC to track conformance to predetermined software requirements and standards Test results adhere to predetermined acceptance standards Confirm Deliverable and milestone acceptance criteria are met. 1.4 QA Team Roles and Responsibilities. The QA Team has the responsibility to report directly to the OKDHS Program Manager if the quality of the Project or product is being jeopardized or compromised. The QA Team will follow the escalation process as defined in this plan. While in practice this rarely occurs, as almost all defects are correctly addressed at the project level, the fact that the QA Team can escalate above the project level gives it the ability to keep the majority of these defects at the project level. Figure 1 reflects the reporting structure for resolving QA issues Program Quality Subject Matter Experts (SMEs) report directly to the QA/QC, but they may interact directly with Project Teams, including Contractors. They have the authority to delegate responsibilities of interacting functions Program Quality Manager and Quality Team will review the QA Team work and have the final approval. QA Team will identify compliance areas as either conforming or non-conforming with the QA standards, procedures, and guidelines set forth in the QA Plan with the goal of ensuring compliance with QA requirements. MOSAIC Quality Assurance Plan v

8 Figure 1. Roles of QA Team Members Relative to the Project The roles of the QA Team members that influence and control Project quality include OKDHS Program Manager, responsible for: Establishing a quality program by committing the Project to implement quality standards and methodologies. Reviewing and approving the QA Plan. Gaining approval from the Decision Team for implementation of the QA Plan. Resolv ing and following-up on any quality issues escalated to this level by the QA Team. Facilitating identification of a person or group independent from the Project to audit and report on the Project s QA Team function. Facilitating identification of the quality factors to be implemented in the system and software. 6 MOSAIC Quality Assurance Plan v04.02

9 Facilitating identification of, developing, and maintaining planning documents, such as the Project Management Plan Program Quality Manager and Quality Team, responsible for: Implementing the quality program in accordance with the approved QA Plan which defines QA standards, procedures, and guidelines. Reviewing and approving the QA activities to be performed by QA/QC. Reviewing and approving the QA Team s work and recommendations during all phases of the program. Resolv ing and following-up on any quality issues escalated to this level by the QA Team. Identifying, developing and maintaining Quality Management planning documents such as the Risk Management Plan, Change Management Plan, and QA Plan Quality Assurance/Quality Control (QA/QC), responsible for: Implementation and adherence to the quality program in accordance with the approved QA Plan which defines QA standards, procedures, and guidelines. Identifying the QA activities to be performed by QA Team. Reviewing and approving the QA Team s work and recommendations during all phases of the program. Resolving and following-up on any quality issues raised by QA Team. Identifying, developing, and maintaining planning documents, such as Test Plans, Standards and Process Manuals (Quality Control Testing and Migration Approval), and this QA Plan. Implementing practices, processes, and procedures as defined in OKDHS Enterprise System RFP, Project Plan, Business or Technical Requirements and Design Documents, SOW, and QA Plan QA Team, responsible for: Reviewing and commenting on the QA Plan. MOSAIC Quality Assurance Plan v

10 Implementing the quality program in accordance with this QA Plan. Resolving and following-up on any quality issues raised by QA Team related to software development and hardware implementation activities. Identifying and evaluating the quality factors to be implemented in the system (software and hardware). Implementing practices, processes, and procedures as defined in OKDHS Enterprise System RFP, Project Plan, Business or Technical Requirements and Design Documents, SOW, and QA Plan Contractor Project Manager and Contractors, responsible for: Reviewing and commenting on the QA Plan. Implementing the quality program in accordance with this QA Plan. Resolving and following-up on any quality issues raised by QA Team related to software design, application development, and hardware implementation. Identifying and evaluating the quality factors to be implemented in the software and hardware. Implementing the software design/development practices, processes, and procedures as defined in the Project Software Development Methodology and Development Requirements Document, Functional and Technical Design Documents, and QA Plan Analysis, Design, Development, & Test (ADDT) Teams, responsible for: Reviewing and commenting on the QA Plan. Implementing the quality program in accordance with QA Plan. Identifying and evaluating the quality factors to be implemented in the software and hardware. Implementing the software design/development practices, processes, and procedures as defined in the Project Software Development Methodology and 8 MOSAIC Quality Assurance Plan v04.02

11 Program Quality SME(s) and Quality Control Testing (QCT) Supervisor, responsible for: Reviewing and commenting on the QA Plan. Implementing the quality program in accordance with QA Plan. Resolving and following-up on any quality issues raised by QA Team related to Change Management and QCT Standards and Process Manual. Ensuring the quality factors are implemented in QCT of software as related to Change Management. Implementing the practices, processes, and procedures as defined in Project Change Management Plan, Standards and Process Manuals (Quality Control Testing and Migration Approval), and QA Plan Quality Control Testers, responsible for: 1.5 Resources for QA Team Reviewing and commenting on the QA Plan. Implementation and adherence to the quality program in accordance with this QA Plan. Resolving and following-up on any quality issues raised by QA Team related to software testing. Verifying the quality factors are implemented in the system, specifically software. Implementing the software test practices, processes, and procedures as defined in Quality Control Standards and Process Manual, and Software Development Methodology, and QA Plan Facilities and Equipment. The QA Team will have the same access to facilities and equipment as the MOSAIC Project Team. QA Team will have access to computer resources to perform QA functions such as process and product evaluations, audits, reviews, and quality control testing Testing Sites. Testing sites that will be established will have all the necessary tools and documentation available to perform QA and QC activities QA Team Members. The QA Team selected and assigned to the Project has demonstrated experience in quality assurance as it relates MOSAIC Quality Assurance Plan v

12 to software engineering and software development and how it directly impacts the business units and their functions. The QA Team has a marked and explicit understanding of quality assurance and QA implications to the Project, Project Team, and how it directly relates and results in quality results for overall Project success Independent Verification & Validation (IV&V) Contractor. The IV&V Contractor will assist OKDHS by determining if development products conform to requirements and whether software satisfies the intended use and user needs. The determination includes assessment, analysis, evaluation, review, inspection, and testing of software products and processes. The IV&V Contractor will work with the QA Team and will follow procedures established or referenced in the QA Plan that will be used as the basis for managing the quality assurance of Project Deliverables Quality Control Testing Team. OKDHS will determine how and when QA/QC staff will be brought into the Project to perform quality control testing. This will also be defined for the Staffing Plan QA Team Training. To establish a QA Team that is experienced in quality assurance activities and to supplement their formal training and experience, they will be trained on OKDHS specific activities to achieve an effective Quality Program. As required, other Project Team members will be provided with training to support the QA activities. Any QA training requirements will be coordinated with the Project Team. Training may be conducted in several formats as specified in the Training Plan. The training schedule will be compatible with the Project schedule. Appendix A provides a matrix that identifies the required skills to perform QA tasks to implement the QA Plan. 2.0 QUALITY ASSURANCE TASKS The scheduling of QA tasks is driven by the Project Plan. Each QA task is performed in relationship to the Project activities that are taking place. One or more QA tasks may be performed concurrently. A task is considered completed when the required report, such as QA Report or Compliancy Report, is satisfactorily completed or action items have been closed. The tasks identified in this section requiring coordination and cooperation with the Project Team will be performed by the QA Team as defined in the Enterprise System RFP, QA Plan, and Project Standards and Guidelines reference materials. 2.1 QA Team Tasks and Matrix. The QA/QC will ensure tasks are performed to verify the quality of the Deliverables on the Project. The QA Team will assist the QA/QC in ensuring the QA procedures are executed in accordance with the QA Plan The QA/QC will monitor Project staff activities and review products for compliance to applicable standards and procedures by utilizing this QA Plan methodology. The results of QA Team 10 MOSAIC Quality Assurance Plan v04.02

13 monitoring and analysis along with QA Team's recommendations for corrective action will be reported to Project Program Quality Manager and Quality Team, and as required, to OKDHS Program Manager All documents and software approved by the OKDHS Program Manager for release to the business units must have been reviewed and approved by QA Team. The QA task matrix is presented in Appendix B. 2.2 Conducting QA Team Tasks. To conduct QA/QC activities identified in Section 2, checklists will be designed and used as a tool for conducting task audit/review/evaluation to verify required steps have been performed. The results of these evaluated tasks will be documented using the Compliancy Report reflected in Appendix D, or sample evaluation checklists in Appendices E, F, G, and H. All checklists will be reviewed with appropriate team members, and submitted to the QA/QC for review Adding QA Team Tasks. Any QA/QC tasks not identified in the planning stages of the Project can be added to the Quality Team's scheduled activities by contacting the Project Program Quality Manager or Quality Team for approval In the tables describing QA Team Tasks, Q indicators in the Task # column are activities that will be preformed in addition to the IV&V RFP tasks required. 2.3 QA Schedule. QA schedules are closely coordinated with the Project Deliverables and product development. Process compliance audits will be performed at the beginning of each new phase of development to verify that the appropriate processes are correctly implemented as defined in the planning documents. Unscheduled audits will be made during each phase of development to verify that the processes and procedures are being followed. At the completion of a software development phase, QA Team will review and report whether all steps required to transition to the next phase have been accomplished. 2.4 Evaluate Planning Oversight. As stated in the OKDHS Enterprise System RFP, key staff within each business area serves on the Project Decision Team. OKDHS Program Manager will report to the Executive Sponsors, who report directly to the OKDHS Chief Executive Officer (CEO). The Project Decision Team will perform overall oversight of the Project and will receive input from an independent Project Manager and the Independent Verification and Validation (IV&V) Team. MOSAIC Quality Assurance Plan v

14 TASK ITEM Contract Verification Feasibility Study TABLE 1 - PLANNING OVERSIGHT QA TEAM TASKS TASK # TASK DESCRIPTION PO-1 Evaluate and verify that the obligations of Contractor, Subcontractors, and external staff, including terms, conditions, statement of work, requirements, technical standards, performance standards, development milestones, acceptance criteria, and delivery dates, are clearly defined. QA Team will verify that performance metrics are included that allow tracking of Project performance and progress against the criteria set by OKDHS. PO-2 Perform ongoing evaluations and reviews of OKDHS methodologies used for the Feasibility Study, verifying the Feasibility Study is objective, reasonable, measurable, repeatable, consistent, accurate, and verifiable. PO-3 Review and evaluate PAPD(U)s and IAPD(U)s. PO-4 Review and evaluate the Cost Benefit Analysis. 2.5 Evaluate Project Management. The overall guidance and direction to the Project is provided by the OKDHS Program Manager. The facilitation of Project tasks and Deliverables will be organized in a Project work plan with recommended sequence and schedule. The QA Team will review the joint Project management efforts as needed and initiate corrective actions where appropriate to verify Project delays and defects are minimal. TABLE 2 PROJECT MANAGEMENT - QA TEAM TASKS TASK ITEM Project Sponsorship Management Assessment TASK # TASK DESCRIPTION PM-1 Evaluate and recommend improvements to verify continuous executive agreement, participation, support, and commitment, and that open pathways of communication exist among stakeholders. PM-2 Verify that OKDHS management has signed off on all changes that impact Project objectives, cost, or schedule. PM-3-Q Review and propose Project schedule, scope, and expenditure controls. PM-4 Evaluate Project management and organization, to verify that lines of reporting and responsibility provide adequate technical and managerial oversight of Project. PM-5 Evaluate and report findings on Project progress, resources, budget, schedules, workflow, and reporting. PM-6 Evaluate and review coordination, communication, and management to verify stakeholders work collaboratively and follow Project Communication Plan. 12 MOSAIC Quality Assurance Plan v04.02

15 TASK ITEM Project Management Business Process Reengineering Risk Management TABLE 2 PROJECT MANAGEMENT - QA TEAM TASKS TASK # TASK DESCRIPTION PM-7 PM-8 Evaluate Project Management Plans and procedures to verify they are developed, communicated, implemented, monitored, and complete. Evaluate Project Management Plans and Project reports to verify Project status is accurately tracked using defined Project metrics. PM-9-Q Evaluate and verify that Project planning complies with Project Management Plan requirements. PM-10-Q Evaluate and verify that milestones and completion dates determined by OKDHS are planned, monitored, and met. PM-11-Q Evaluate and verify Deliverables are entered and tracked by the approved Deliverables Management Tool and comply with Deliverables requirements. PM-12 Evaluate and verify that the Tracking Tool for Project issues and defects documents issues and defects as they arise; enabling communication of issues and defects to appropriate stakeholders; documents a mitigation strategy as appropriate; and tracks issues and defects to resolution. Tracking issues and defects will include, but not be limited to, technical and development efforts. PM-13-Q Evaluate and verify that Defect (Incident) Tracking Reports are provided for any defects that could place in jeopardy any Project component presented in the Project Management Plan, and complies with Defect Tracking Report requirements. PM-14-Q Review, analyze, and propose the Deliverable content, review, and submission process. PM-15-Q Review, critique, and propose issue resolution and escalation procedures. PM-16 PM-17 PM-18 PM-19 Evaluate the lifecycle development methodology(s), such as waterfall, evolutionary spiral, rapid prototyping, and incremental, to verify the methodology(s) is appropriate for the system being developed. Evaluate and verify Project capabilities and plans to redesign business systems to achieve improvements in critical measures of performance, such as cost, quality, service, and speed. Evaluate and verify the Enterprise Architecture Methodology has the strategy, management backing, resources, skills, and incentives required for effective change. Evaluate and verify that the Project Risk Management Plan is created and followed. MOSAIC Quality Assurance Plan v

16 TASK ITEM Change Management Organizational Readiness TABLE 2 PROJECT MANAGEMENT - QA TEAM TASKS TASK # TASK DESCRIPTION PM-20 Evaluate Risk Management Plan and procedures to verify that risks are identified and quantified and that Mitigation Plans are developed, communicated, implemented, monitored, and complete. PM-21-Q Review and be familiar with Risk Management Plan PM-22 compliance standards and procedures. Evaluate and verify that Project Change Management Plan is created and followed. PM-23-Q Evaluate and verify that requirements identified as having potential defects are reviewed with the Change Control Committee to determine the impact or necessity of the change. PM-24-Q Evaluate and verify that Change Control Management activities comply with Change Control Management requirements. PM-25 Communicatio PM-26 n Management Configuration Management PM-27 PM-28 PM-29 PM-30 PM-31 PM-32 Evaluate Change Management Plan and Organizational Readiness Plan and procedures to verify they are developed, communicated, implemented, monitored, and complete and that resistance to change is anticipated and prepared for. Verify that Project Communication Plan is created and followed. QA Team will verify Organizational Management is addressed in the Project Communication Plan. Evaluate Communication Plan and strategies to verify they support communications and work product sharing between all Project stakeholders, and verify Communication Plan and strategies are effective, implemented, monitored, and complete. Evaluate and verify Change Management Plan and procedures associated with the development process. Evaluate and verify that all critical development documents, including, but not limited to, requirements, design, and code, are maintained under the required level of control. Evaluate and verify that processes and tools are in place to identify code versions and rebuild system configurations from source code. Evaluate and verify that all source and object libraries are maintained for training, testing, and production, and that formal sign-off procedures are in place for approving Deliverables. Evaluate and verify that processes and tools are in place to manage system changes, including formal logging of change requests and review, prioritization, and timely scheduling of maintenance actions. 14 MOSAIC Quality Assurance Plan v04.02

17 TASK ITEM Project PM-35 Estimating and Scheduling TABLE 2 PROJECT MANAGEMENT - QA TEAM TASKS TASK # TASK DESCRIPTION PM-33 Evaluate and verify that mechanisms are in place to prevent unauthorized changes to the system and prevent authorized changes from being made to the wrong version. PM-34-Q Evaluate and verify that the configuration management process is in place to control and manage the baseline configuration for all hardware, communications equipment, network equipment, application development security principles, security controls, application deployment, and PM-36 operational configurations. Evaluate and make recommendations on Project estimation and scheduling process to verify that Project budget and resources are adequate for the Work Breakdown Structure and schedule. Review and evaluate schedules to verify that adequate time and resources are assigned for planning, development, review, testing, and rework. Project Staff PM-37 Evaluate the job assignments, skills, training, and experience of the staff involved in program development to verify that they are adequate for the development task. Project Organization Sub- Contractors and External Staff, if any PM-38 PM-39 PM-40 PM-41 PM-42 PM-43 PM-44 Verify that lines of reporting and responsibility provide adequate technical and managerial oversight of the Project. Verify that Project organizational structure supports training, process definition, independent quality assurance, Configuration Management, product evaluation, and any other functions critical to Project success. Evaluate the use, in Project development, of Subcontractors or other external sources of Project staff, such as IT staff from another State of Oklahoma organization. Evaluate and verify that the obligations of Subcontractors and external staff, including terms, conditions, statements of work, requirements, standards, development milestones, acceptance criteria, and delivery dates, are clearly defined. Evaluate and verify that Subcontractor's software development methodology and product standards are compatible with OKDHS standards and environment. Verify that Subcontractor has and maintains the required skills, staff, plans, resources, procedures, and standards to meet Subcontractor's commitment. QA Team will evaluate and verify the feasibility of any off-site support of the Project. Evaluate and verify that any proprietary tools used by Subcontractors do not restrict the future maintainability, MOSAIC Quality Assurance Plan v

18 TABLE 2 PROJECT MANAGEMENT - QA TEAM TASKS TASK ITEM OKDHS Oversight TASK # TASK DESCRIPTION PM-45 PM-46 PM-47 portability, and reusability of the Enterprise System. Verify that OKDHS oversight is provided in the form of periodic status reviews and technical interchanges. Verify that OKDHS has defined the technical and managerial inputs the Subcontractor requires, including reviews, approvals, requirements, and interface clarifications, and has the resources to supply them on schedule. Evaluate Project oversight to verify that OKDHS, not Contractor, exercises ultimate responsibility for monitoring Project cost and schedule. 2.6 Evaluate Quality Management. QA Team will verify Contractor assists in updating the QA Plan, which will document how to plan, implement, and assess the effectiveness of Quality Assurance and Quality Control operations. The QA Plan will fully define the entire Quality system, including, but not limited to, organizational structure, roles and responsibilities, processes, and resources required to implement and manage the QA Plan. TABLE 3 QUALITY MANAGEMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION Quality Assurance QA-1 QA-2 QA-3 QA-4 QA-5 QA-6 QA-7-Q Evaluate and make recommendations about Project Quality Assurance Plan, procedures, and organization. Verify the fidelity of all defined processes in all phases of the Project is monitored. Verify that the quality of all products produced by the Project is monitored by formal reviews and sign-offs. Verify that Project self-evaluations are performed and that measures are continually taken to improve the process. Monitor the performance of QA processes and designated staff by reviewing QA processes and reports, and by performing spot checks of system documentation. Contractor will evaluate findings and performance of the processes and reports. Verify that Project Management supports the appropriate levels of independence for QA activities. Review and be familiar with QA Plan compliance standards and procedures. 16 MOSAIC Quality Assurance Plan v04.02

19 TABLE 3 QUALITY MANAGEMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION Process QA-8 Definition and Product Standards QA-9 QA-10 QA-11 Evaluate and make recommendations about all defined processes and product standards associated with system development. Evaluate and verify that all major development processes are defined and that the defined and approved processes and standards are followed in development. Evaluate and verify that processes and standards are compatible with each other and with system development methodology. Evaluate and verify that all process definitions and standards are complete, clear, up-to-date, consistent in format, and easily available to Project staff. QA-12-Q Evaluate and verify that Progress/Status Reports comply with Communications Plan requirements. QA-13-Q Evaluate and verify that all QA Reports comply with the Reporting requirements as stated in the QA Plan. QA-14-Q Evaluate and verify that Requirements Discovery and Documentation processes are followed as defined in the Project Requirements Management Plan. QA-15-Q Evaluate and verify that Functional Requirements Discovery and Documentation processes are followed as defined in the Project Requirements Management Plan. 2.7 Evaluate Training. The QA Team will review Training Plan and the training systems along with any physical training space needs to verify they all meet the requirements. MOSAIC Quality Assurance Plan v

20 TASK ITEM User Training TR-1 and Documentation TR-2 TABLE 4 TRAINING QA TEAM TASKS TASK # TASK DESCRIPTION TR-3 TR-4 Developer TR-5 Training and Documentation TR-6 TR-7 TR-8 TR-9 Verify that Training Plan and processes provide adequate training to system users. Verify that user-friendly training materials and Help Desk services are easily available to all users. Verify that all required standards and processes, and related documentation, are easily available to users. Verify that all training is provided on time, and is evaluated and monitored for effectiveness, with additional training provided as needed. Verify that the training plan and processes provide adequate training to system developers. Evaluate and verify that developer training is technically adequate, appropriate for the Development Phase, and available at appropriate times. Verify that all required policy, process, and standards documentation is easily available to developers. Verify the required knowledge transfer occurs for maintenance and operation of the new system. Evaluate and verify that all training is provided on time and is evaluated and monitored for effectiveness, with additional training provided as needed. TR-10-Q Evaluate and validate training results. TR-11-Q Evaluate and verify that Project staff involved in any processes is trained in accordance with Training Plan in the required procedures and standards applicable to area of responsibility to do the job correctly. 2.8 Evaluate Requirements Management. Requirements analysis establishes a common understanding of the business unit's requirements between that business unit customer and the software Project Team. QA Team will verify that the requirements process reviews are conducted in accordance with the standards and procedures established by the Project Team. The Contractor will assist OKDHS in ensuring that all Enterprise System RFP requirements are not only met, but requirements are also incorporated and traceable within all documents, models, Deliverables, etc.; and accomplished efficiently and effectively. 18 MOSAIC Quality Assurance Plan v04.02

21 TABLE 5 REQUIREMENTS MANAGEMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION Requirements Management Security Requirements RM-1 RM-2 RM-3 RM-4 RM-5 RM-6 RM-7-Q RM-8-Q RM-9-Q Evaluate and make recommendations about the Requirements Management Plan processes and procedures for managing system requirements. Evaluate and verify that requirements are well-defined, understood, documented, and traceable throughout all phases of the Project. Evaluate and verify the allocation of system resources per hardware and software requirements. Evaluate and verify that software requirements can be traced through design, code, and test phases to verify that the Enterprise System performs as intended and contains no unnecessary software elements. Evaluate and verify that requirements are under formal configuration control. Evaluate and verify Project security for managing requirements and performing the risk analysis on each requirement. Evaluate and verify that requirements are reviewed to determine if they are clearly stated and consistent. Evaluate and verify that changes to requirements, work products, and activities are identified, reviewed, and tracked to closure. Evaluate and verify that the prescribed processes for defining, documenting, and allocating requirements are followed and documented. RM-10-Q Evaluate and verify that requirements are managed, controlled, and traced by the approved Requirements Change Management Tool. RM-11-Q Evaluate and verify that Requirements Reviews comply with Requirements standards and guidelines. RM-12 RM-13 RM-14 Evaluate and verify that processes and equipment are in place to back-up Project information and files and archive them safely at appropriate intervals. Evaluate and make recommendations on Project standards and procedures for ensuring the system is secure and privacy of client information is protected. Evaluate and verify Project restrictions to verify system and information access control. MOSAIC Quality Assurance Plan v

22 TABLE 5 REQUIREMENTS MANAGEMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION Requirements Analysis Interface Requirements Reverse Engineering RM-15-Q Evaluate and verify security testing and functional tests are performed. RM-16 Verify that the Requirements Analysis of OKDHS and federal and state requirements and objectives has been performed to verify that Enterprise System requirements are understood, well-defined, and meet federal and state regulations. RM-17 Verify that all stakeholders have been consulted about the desired functionality of the Enterprise System, and users have been involved in prototyping the user interface. RM-18 Verify that performance requirements, such as timing, response time, and throughput, meet user requirements. RM-19 Evaluate and verify that user maintenance requirements for the Enterprise System are completely specified. RM-20-Q Evaluate and verify that the requirements analysis process and associated requirements reviews are conducted in accordance with the standards and procedures established by the Project and as described in Project RFP, SOW, and Requirements Management Document. RM-21-Q Evaluate and verify that action items resulting from reviews of the requirements analysis are resolved in accordance with RM-22 RM-23 RM-24 RM-25 these standards and procedures. Evaluate and verify that all system interfaces are exactly described, by medium and by function, including input/output control codes, data format, polarity, range, units, and frequency. Evaluate and verify approved interface documents are available and that appropriate relationships, such as interface working groups, are in place with all agencies and organizations supporting the interfaces. Evaluate that requirements specifications have been developed for all hardware and software subsystems in a level of detail to verify successful implementation and business continuity. Evaluate and verify that a well-defined Transition Plan and process for reengineering the system is in place and is followed, if a legacy system or a transfer system is or will be used in development. The process, depending on the goals of the reuse/transfer, may include reverse engineering, code translation, re-documentation, restructuring, normalization, and re-targeting. 20 MOSAIC Quality Assurance Plan v04.02

23 2.9 Evaluate Operating Environment. The QA Team will evaluate software and hardware compatibility in all operating environments to verify that system requirements have been met, are maintainable and upgradeable. TABLE 6 OPERATING ENVIRONMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION System Hardware System Software OE-1 OE-2 OE-3 OE-4 OE-5 OE-6-Q OE-7-Q OE-8-Q OE-9-Q Evaluate current and projected system hardware configurations to verify that performance meets system requirements. Evaluate and verify that hardware is compatible with existing OKDHS processing environment, maintainable, and easily upgradeable. Evaluation and verification will include, but is not limited to, CPUs and other processors, memory, network connections, bandwidth, communication controllers, terminals, printers, telecommunications systems (LAN/WAN), and storage devices. Evaluate and verify current and projected outside Contractor support of the hardware and OKDHS hardware Configuration Management Plan and procedures. Evaluate current and projected system software to verify that capabilities meet system requirements. Evaluate and verify that software is compatible with existing OKDHS hardware and software environment, maintainable, and easily upgradeable. Evaluation and verification will include, but is not limited to, operating systems, middleware, and network software, including communications and file-sharing protocols. Evaluate and verify that system requirements are traceable throughout all phases of the Project. Evaluate and verify that relevant documents are updated and based on approved requirements changes. Evaluate and verify that the agreed upon requirements follow the Systems Standards and Guidelines. Verify that design walkthroughs evaluate compliance of the design of requirements, identify defects in the design, and evaluation and report design alternatives. OE-10-Q Participate in walkthroughs and verify all walkthroughs are conducted. OE-11-Q Identify defects, verify resolution for previous identified defects, and verify change control integrity. OE-12-Q Selectively review and audit the content of system documents. MOSAIC Quality Assurance Plan v

24 TABLE 6 OPERATING ENVIRONMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION Database Software System Capacity OE-13-Q Evaluate and review demonstration prototypes for compliance with requirements and standards, verify that the demonstration conforms to standards and procedures, identify lack of compliance with standards, and determine corrective actions. OE-14 Evaluate current and projected database products to verify that capabilities meet system requirements. OE-15 Evaluate and verify the data format of the database is easily convertible to other formats, supports the addition of new data items, is scalable, easily refreshable, and compatible with existing OKDHS hardware and software, including any online transaction processing (OLTP) environment. OE-16-Q Review and evaluate the software tools are available to provide database back-up, recovery, performance analysis, and data creation control. OE-17 Evaluate the existing processing capacity of the system and verify that it is adequate for current statewide requirements for both batch and online processing. OE-18 Evaluate the historic availability and reliability of the system, including the frequency and criticality of system failure. OE-19 OE-20 OE-21 Evaluate the results of any volume testing or stress testing. Evaluate any existing measurement and capacity planning program and will evaluate the system's capacity to support future growth. Make recommendations about changes in processing hardware, storage, network systems, operating systems, Enterprise System software, and software design to meet future growth and improve system performance. OE-22-Q Evaluate and verify the system's response time meets requirements stated in the RFP. OE-23-Q Evaluate and verify the system accessibility complies with RFP requirements. OE-24-Q Evaluate and verify that Operational Support complies with the Operational Support requirements. OE-25-Q Evaluate and validate system tests and procedures Evaluate Development Environment. The QA Team will evaluate software and hardware for compatibility in all development environments to verify that development requirements have been met and documentation complies with Project Software Development Methodologies and documentation standards. 22 MOSAIC Quality Assurance Plan v04.02

25 TABLE 7 - DEVELOPMENT ENVIRONMENT - QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION Development Hardware Development Software DE-1 DE-2 DE-3 DE-4 DE-5 DE-6 DE-7 DE-8-Q DE-9-Q Evaluate current and projected development hardware configurations to verify performance meets the requirements of system development. Evaluate and verify that hardware is compatible with existing OKDHS development and processing environment, maintainable, and easily upgradeable. Evaluation and verification will include, but is not limited to, CPUs and other processors, memory, network connections, bandwidth, communication controllers, terminals, printers, telecommunications systems (LAN/WAN), and storage devices. Evaluate and verify current and projected outside Contractor support of the hardware. Evaluate current and projected development software to verify capabilities meet system development requirements. Evaluate and verify that software is compatible with the existing OKDHS hardware and software environment, maintainable, and easily upgradeable. Evaluate the environment as a whole to verify it shows integration. Evaluation will include, but is not limited to, operating systems, network software, Computer-Aided Software Engineering (CASE) Tools, Project management software, configuration management software, compilers, cross-compilers, linkers, loaders, debuggers, editors, and reporting software. Evaluate and verify current and projected outside Contractor support of the software. Evaluate and verify that documentation complies with Project Software Development Methodology and Documentation standards. Evaluate and verify that development requirements comply with Development standards and guidelines Evaluate Software Development. The system design process is to develop decisions about the system's behavioral design and other decisions affecting the selection and design of system components. Project Design Documents Standards & Guidelines template will define the organization and content of the Functional Design Document and Technical Design Document. MOSAIC Quality Assurance Plan v

26 TABLE 8 SOFTWARE DEVELOPMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION High-Level Design Detailed Design SD-1 SD-2 SD-3 SD-4 SD-5 SD-6-Q SD-7-Q SD-8-Q SD-9 SD-10 SD-11 SD-12 SD-13 Evaluate and make recommendations on existing high-level design products to verify that the design is workable, efficient, and meets all system and system interface requirements. Evaluate and verify that high-level design products conform to design methodology and standards. Evaluate and make recommendations on high-level process, standards, methodologies, and CASE tools used. Evaluate and verify that design requirements can be traced back to system requirements. Evaluate and verify that all design products are under configuration control and formally approved before detailed design begins. Evaluate and verify that the software requirements analysis process and associated requirements reviews are conducted in accordance with the standards and procedures established by the Project and as described in the Project RFP, SOW, and Requirements Management Process. Evaluate and verify that action items resulting from reviews of the software requirements analysis are conducted in accordance with the standards and procedures established by the Project and as described in the Project RFP, SOW, and Requirements Management Process. Evaluate and verify that Design Reviews are conducted in accordance with the standards and procedures established by the Project and as described in the Project RFP, SOW, and Requirements Management Process. Evaluate and make recommendations on existing detailed design products to verify that the design is workable, efficient, and meets all high-level design requirements. Evaluate and verify that detailed design products conform to design methodology and standards. Evaluate and make recommendations on the detailed design and analysis process, standards, methodologies, and CASE Tools used. Evaluate and verify that design requirements can be traced back to system requirements and high-level design. Evaluate and verify that all design products are under configuration control and formally approved before coding begins. 24 MOSAIC Quality Assurance Plan v04.02

27 TABLE 8 SOFTWARE DEVELOPMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION SD-14-Q Evaluate and verify that the method, such as the appropriate Software Development Repository, used for tracking and documenting the development of a software unit is implemented and is kept current. Job Control SD-15 Evaluate and make recommendations on the existing job control and the process for designing job control. SD-16 Evaluate the system's division between batch and online processing to verify system performance and data integrity. SD-17 Evaluate batch jobs to verify appropriate scheduling, timing, and internal and external dependencies. SD-18 Evaluate and verify that job control language scripts are under configuration control. Code SD-19 Evaluate and make recommendations on the standards and process for code development currently in place. SD-20 Evaluate the existing code base to verify portability and maintainability, taking into account software metrics, including, but not limited to, modularity, complexity, and source and object size. SD-21 Evaluate code documentation to verify quality, completeness including maintenance history, and accessibility. SD-22 Evaluate the coding standards and guidelines to verify compliance with these standards and guidelines. Evaluation will include, but is not limited to, structure, documentation, modularity, naming conventions, and format. SD-23 Evaluate and verify that developed code is under configuration control and is easily accessible by developers. SD-24 Evaluate and verify use of software metrics in management and quality assurance. Unit Test SD-25 Evaluate and make recommendations on the plans, requirements, environment, tools, and procedures used for unit testing system modules. SD-26 Evaluate the level of test automation, interactive testing, and SD-27 interactive debugging available in the test environment. Evaluate and verify the test process, that test results are verified, that the correct code configuration has been tested, and that the test is appropriately documented. SD-28-Q Evaluate and verify that the software development methodology, associated code reviews, and software unit testing are conducted in conformance with the standards and procedures established by the Project. MOSAIC Quality Assurance Plan v

28 TABLE 8 SOFTWARE DEVELOPMENT QA TEAM TASKS TASK ITEM TASK # TASK DESCRIPTION Software Development Lifecycle SD-29-Q Evaluate and verify that action items resulting from reviews of the code are resolved in accordance with these standards and procedures established by the Project. SD-30-Q Evaluate and verify that the system or module presented for acceptance must account for all required functionality, training, conversion, documentation, and any other related requirements of the Contract and federal and State of Oklahoma rules, regulations, and policies for the components comprising the release. SD-31-Q Evaluate and verify that Contractor in conjunction with OKDHS will provide the Iterative Application Software Development Methodology. SD-32-Q Evaluate and verify that documentation and computer program materials are approved and placed under library control. SD-33-Q Evaluate and verify the establishment of formal release procedures for approved documentation and software versions. SD-34-Q Evaluate and verify that library controls prevent unauthorized changes to the controlled software and verify the incorporation of all approved changes Evaluate System and Acceptance Testing. QA Team activities will verify software integration and test activities combine individually developed components together in the testing environment to verify that they work together to complete the software and system functionality. A Pilot will verify the functional and technical usability of the Enterprise System in a limited production environment. 26 MOSAIC Quality Assurance Plan v04.02