3 PART THREE: WORK PLAN AND IV&V METHODOLOGY (SECTION 5.3.3)

Transcription

1 3 PART THREE: WORK PLAN AND IV&V METHODOLOGY (SECTION 5.3.3) Emagine IT s approach to Independent Verification and Validation (IV&V) has been shaped over the years by hands-on experience and contributions as thought leaders in the cybersecurity arena. Simply put, we address the objectives, challenges, transformation, and ultimately the critical success factors necessary for mission success. We incorporate Project Management Body of Knowledge (PMBoK) best practices, Capability Maturity Model Integration (CMMI) Level 3 appraised methodologies, and our International Organization of Standardization (ISO) , 27001:2013, and 17020:2012 certified frameworks to design and develop our most comprehensive IV&V solution. The key IV&V objectives of the Emagine IT methodology include: Provide continuous assessment of software, system products, and processes throughout the life cycle Facilitate early detection and correction of errors Reduce the effort required to remove faults, via early detection Demonstrate hardware, software, system requirements are complete, accurate, consistent, and testable Enhance management insight into process and product risk Support the life cycle processes to ensure compliance with program performance, schedule, and cost requirements Enhance operational correctness and product maintainability As shown in Figure 1 below, Emagine IT developed a 12-step IV&V methodology by taking the best of breed from industry standards (National Institute of Standards and Technology (NIST), CMMI, ISO, PMBoK) and combining them with years of Subject Matter Expert (SME) experience. Figure 1. Emagine IT s IV&V Methodology Air Force Medical Services (AFMS) Contract IV&V Efficiencies On the AFMS contract we successfully processed more Authority to Operate/Interim Authority to Operate (ATO/IATO) packages in the first six months after contract award than the combined 18 months under the previous contract; a more than 300% increase in complete package efficiency. Our approach enabled Emagine IT to reduce the number of staff required from that of the previous vendor s contract. Under that contract, 14.5 Full Time Equivalents (FTE) were required and Emagine IT is doing the same work with only 6 FTEs, while increasing staff productivity and the quality of deliverables by nearly 600%. 3.1 IV&V ENGAGEMENT OVERVIEW The first and most important step in the IV&V process is the Planning Phase. As part of the Planning Phase, Emagine IT begins by conducting an analysis of the scope of the project, application, or system in relation to: cost, size, schedule, complexity, criticality, security, Page 1

2 safety, risk, team complexity, organizational complexity, and requirements maturity. IV&V factors need to be analyzed, resulting in a costeffective IV&V effort that is tailored to the scope of the application. During the planning phase, we conduct independent verification against program planning to measure readiness and prepare a baseline to work from. These independent variables include: Development Schedules Software Development Plans Security Assessment Plan System Engineering Plans Project Management Plans Test Plan Quality Assurance Plans Hardware Development/Acquisition Plans Configuration Management (CM) Plans Commercial of the Shelf (COTS) Plans Risk Management Plans Emagine IT then prepares a tailored Work Plan to address the IV&V activities for each project under review. This follows the Emagine IT IV&V Methodology and/or any other applicable customer-specific guidance documents or policies. This plan is fairly brief and contains an introduction and a list of activities to be performed for each development stage. The Work Plan is structured by phase and includes key activities for each phase with target due dates. At a minimum, the Emagine IT Work Plan describes the following: System profile Points of contact IV&V schedule Key activities by phase IV&V Team organization Key deliverable due dates Scope of the IV&V effort (Approach, Activities, Tailoring) Tailoring of IV&V tasks and/or checklists based on project and/or scope The IV&V schedule is coordinated with the customers project master schedule so that IV&V schedules can be consolidated across all contractors and staff. The Work Plan is generally the first major deliverable produced by Emagine IT and it is updated regularly throughout the project as we respond to the specific needs of the project. Upon completion of the IV&V assessment (documentation review, interviews, vulnerability and penetration tests, analysis, and reporting), Emagine IT conducts a lessons learned session. The lessons learned session incorporates feedback from the project team members in addition to the IV&V team internal evaluation. We evaluate feedback to determine opportunities for improvements to processes, people, policies, and deliverables. We provide the results from the lessons learned to the customer as a final deliverable with recommendations and an action plan. 3.2 TOOLS, RECOMMENDATIONS, DELIVERABLES Table 1 outlines a baseline of standard checklists and tools maintained by Emagine IT for use during evaluations. They may be used as is, or tailored as necessary. Table 1. Emagine IT IV&V Baseline s Name Project WBS and This checklist is used to determine if critical activities have been planned for in a project s Work Breakdown Schedule Structure (WBS) and schedule. This checklist is applied to the Integrated Master schedule for the project. Readiness Review This checklist specifies review of all system documentation, raw scan results, configuration checklists, and interviews with system and process owners. Document Review Requirements Review Preliminary Design Detailed Design This checklist forms the basis for documentation reviews and is tailored to match the type of document under review. It serves as an aid to determining the overall quality of a document as to readability, utility, correctness, and completeness. This checklist is used to determine whether a given concept, set of requirements, design, test, etc., demonstrates that the requirements documentation or system satisfies its specified acceptance requirements. This checklist is used to aid in assessing the top-level design, as well as the allocation of requirements to software components, and to determine whether the Preliminary Design Review resolved open issues concerning the handling of high-level design requirements. This checklist is used to aid in determining if all the software requirements have been translated into a viable software design, and whether the Critical Design Review resolved open issues concerning the handling of critical requirements. Page 2

3 Name Process Review CM Code Review Unit Testing Review Software Development Files Review Test Readiness Review Section 508 Review This is a sample process review checklist with an emphasis on CM practices. This checklist is used to determine whether CM Procedures document and plans for performing configuration control; providing access to documentation and code under configuration control; and controlling the preparation and dissemination of changes to master copies of software and documentation so they reflect only approved changes. This checklist identifies evaluation criteria that can be used to determine whether the software design has been correctly implemented using code that adheres to programming standards and conventions. This checklist is used to determine if adequate test procedures to test each Computer Software Unit were developed and documented. Each unit was coded and tested ensuring that the algorithm(s) and logic employed are correct and satisfy the specified requirements; all necessary revisions to the design documentation and code were made; all necessary retesting was performed; and test results were recorded. This checklist is used to determine whether software development files contain material pertinent to the development or support of the software including: requirements, design considerations, constraints, documentation, program design language and source code, test data; status information; and test requirements, cases, procedures, and their results. This checklist is for evaluating the Test Readiness Review to ensure that adequate preparations were made prior to performing System Integration Test, and System Acceptance Testing. This checklist is for performing a Section 508 assessment, as required. These requirements are described in section 508 of the Rehabilitation Act, which requires access for both members of the public and federal employees to such technologies when developed, procured, maintained, or used by Federal Agencies. It is included here to provide guidance to the IV&V Team as to the Section 508 requirements. The goal of conducting IV&V assessments is to get an unbiased review of a project or system against the agreed upon requirements. On many engagements, we have identified situations where significant or major defects/vulnerabilities have been discovered, that cannot and should not, wait to be addressed until later in the project lifecycle. In the Planning Phase, Emagine IT defines major risks and the escalation procedures to address the risks. In addition, we rely heavily on constant communication throughout the entire IV&V assessment process. Table 2 below outlines the typical documentation, deliverables, and reports that Emagine IT s IV&V Team provides on any IV&V assessment. We include additional deliverables to address customer-specific requirements, as specified in a Statement of Work (SOW). Table 2. Representative Documentation and Deliverables Documents, Deliverables, IV&V Work Plan The IV&V Team prepares a tailored plan to address the IV&V activities for each development project under review. Review Plan Typically, the IV&V Team provides the Review Plan within 10 days of the scheduled review. Reviews are performed when scheduled by the IV&V Team or when directed by DMS Customer. Completed s The IV&V Team completes the appropriate checklist for each IV&V task. Completed checklists may be included as part of a Technical submitted for the specific IV&V task. The IV&V Team reports on the individual IV&V phase tasks in a letter of findings (technical report), Technical s which is issued as necessary. The technical reports may document interim results and status. The reports may be in a format appropriate for technical disclosure (for example, technical reports or memos). We generate reports for each technical task performed during the course of the IV&V program. Document Review Comments Memorandum of Record (MOR) Feasibility Assessment Pen Testing Letter of Authorization The IV&V Team prepares and submit document review comments in a letter of findings. For each document that requires review, the IV&V Team submits a letter of findings within 30 days of document receipt. A formal memo format that can be used for meeting minutes, comments, and status reports, or to highlight a significant issue or milestone. This report contains a detailed analysis of the IV&V Team s assessment of the alternatives including: Assessment methodology Alternatives with accompanying analysis Ranking of alternatives Recommendations with rationale Any risks that accompany the recommendations and alternatives Also known as the Rules of Engagement for penetration testing that may be performed for a DMS customer. Page 3

4 Documents, Deliverables, Requirements Verification Matrix (RVM) Anomaly Risk Assessment and IV&V Risk Register Security Assessment Plan Plan of Actions and Milestones Test Plan IV&V Test Procedures and Use Cases IV&V Test Special Studies s IV&V End of Phase Summary s Stage Gate/Production Readiness Review Recommendation IV&V Final (Security Assessment ) Progress s Meeting Minutes Trip s IV&V Metrics s Contractor Team/Security Roster Executive Level Project A tool to verify that system requirements are in accordance with the IV&V standards. An Anomaly is prepared and for submission to the DMS Customer Program Manager when anomalies detected by the IV&V team. Emagine IT will provide Anomaly s to the DMS customer no later than 3 days after anomaly detection. A formal report documenting the results of the risk assessment described in the IV&V Standards and Procedures. The detailed technical test plan that outlines the tools, dates, scope, and level of effort to conduct the security assessment for any engagement. Record of all identified vulnerabilities that tracks the status of the vulnerability and mitigation plan. All of the activities of the testing cycle to include environmental variables, users, timeframes of testing, results, formal build procedures, what types of tests, etc. These procedures and use cases will be provided sequentially throughout the Build and Test and Integration Test Phases in preparation for Acceptance Testing. These reports will be used to document the IV&V Team s witnessing of test activities, including software installation, test setup, test execution, problem reporting, data recording, and data analysis. As required, the IV&V Team will conduct and report on technical, cost, and schedule trade-off analyses related to system development (for example, changes in standards or technology) For large system development efforts, the IV&V Team may prepare and submit an IV&V End of Phase Summary for each lifecycle phase to include the following information: of IV&V tasks performed Assessment of overall system/software quality Recommendations to proceed to next phase Lessons learned An End of Phase Summary when provided comes at the conclusion of each of the following phases: Vision Phase, Definition Phase, Construction Phase, Deployment Phase, Support Phase When applicable, IV&V will encourage a pre-stage Gate meeting where all outstanding issues can be addressed prior to the Stage Gate or Program Readiness Review (PRR). The report includes: List of outstanding issues Risks relevant to Stage Gate/PRR Recommendation for Stage Gate/PRR All contingencies that impact the recommendation The IV&V Project Manager or team lead may be asked to sign the Stage Gate/PRR recommendation. The IV&V Final is issued at the end of the System Deployment Phase or at the conclusion of the IV&V effort. The report includes the following items: Summary of all lifecycle IV&V tasks Assessment of overall system quality Recommendations Lessons learned Provide a summary of all IV&V activities performed for the program. Provides a detailed description of the attendees, action items, and discussions of all meetings. The IV&V Team may prepare (as needed) and submit formal trip reports to the DMS customer for each trip in support of the project. This report is used in reporting metrics performance on a monthly basis. This is delivered at the project inception and includes a list of the complete project staff including name, labor category, task, title, whether they have a badge, and whether they are on- or off-site. This report provides DMS customer leadership insight into the key issues facing a particular development project. This report provides the IV&V Team s formal assessment of the issues and provides a forum for IV&V to convey information to senior management. Page 4

5 Documents, Deliverables, IV&V Lessons Learned Vulnerability Scans Penetration Testing At the conclusion of each In Process Review (IPR), the IV&V Team provides lessons learned for the quarter under review. Raw results from the scans, and reports resulting from any vulnerability scans performed during the IV&V assessment. of all vulnerabilities, risks, and findings discovered during the penetration testing phase of an assessment. Page 5