NATO PROJECT ASSESSMENT MODEL

Transcription

1 AQAP 2050 NATO PROJECT ASSESSMENT MODEL AQAP 2050 (September 2003) I ORIGINAL

2 AQAP 2050 Page blank II ORIGINAL

3 AQAP 2050 NORTH ATLANTIC TREATY ORGANIZATION NATO STANDARDISATION AGENCY (NSA) NATO LETTER OF PROMULGATION September NATO PROJECT ASSESSMENT MODEL is a NATO/PFP UNCLASSIFIED publication. The agreement of interested nations to use this publication is recorded in STANAG is effective on receipt. 3. It is permissible to distribute copies of this publication to Contractors and Suppliers and such distribution is encouraged. Jan H ERIKSEN Rear Admiral, NONA Chairman NSA III ORIGINAL

4 AQAP 2050 Page blank IV ORIGINAL

5 Record of Changes AQAP 2050 Change Date Date Entered Effective Date By Whom Entered V ORIGINAL

6 Table of Contents AQAP 2050 Page number List of Acronyms 1 Chapter 1 - Overview References Objectives Application 2 Chapter 2 - Assessment Model 3 Chapter 3 - Roles and Responsibilities Assessment Sponsor (AS) Project Manager (PM) Assessment Team (AT) Assessment Team Leader (ATL) Project Liaison (PL) Independent Peer Reviewer-Optional (PR) Enterprise Customer (EC) Assessment Management Board (AMB) Model Management Team (MMT) 5 Chapter 4 - Assessment Process Process Activities Tasks Establish Assessment Capability Purpose Tasks Responsibility Products Adopt Assessment Capability Purpose Tasks Responsibility Products Initiate Assessment Purpose Tasks Responsibility Products Plan Assessment Purpose Tasks Responsibility Products Conduct Assessment Purpose 11 VI ORIGINAL

7 AQAP Tasks Responsibility Products Analyze and Report Assessment Results Purpose Tasks Responsibility Products Follow-up Purpose Tasks Responsibility Products Analyze and Improve Purpose Tasks Responsibility Products 14 Annex A - Assessment Issue Description A-1 VII ORIGINAL

8 AQAP 2050 LIST OF ACRONYMS AMB AP AS AT ATL EC MMT PL PM PMT PR Assessment Management Board Assessment Process Assessment Sponsor Assessment Team Assessment Team Leader Enterprise Customer Model Management Team Project Liaison Project Manager Project Manager Team Peer Reviewer VIII ORIGINAL

9 AQAP 2050 Chapter 1 - Overview The purpose of this publication is to describe and provide guidance on NATO Project Assessment Model (NPAM). The NPAM is a model for assessing projects to, identify specific project issues, provide recommendations, and extract cross-project systemic issues. This document describes the model, and provides guidance at high-level on its use, and feedback of results into common information repository(s). 1.1 References The material included in this standard is based on the Program Assessment Architecture developed by the U.S. Department of Defense under the Tri-Service Assessment Initiative. a. Allied Quality Assurance Publication (AQAP) 100 Ed 3, The NATO Policy on an Integrated Systems Approach to Quality Through the Life Cycle. b. ISO10006:1997, Quality Management -- Guidelines to quality in project management. c. Phased Armaments Programming Systems (PAPS). d. AQAP 2000 NATO Policy on an Integrated Systems Approach to Quality Through the Life Cycle. e. ISO 19011:2002, Guidelines for quality and/or environmental management systems auditing. 1.2 Objectives The objectives of the NPAM are as follows: a. Provide assistance to Project Management Teams (PMTs) in order to: (1) Identify issues within and outside of their control; (2) Recommend areas for improvement; (3) Provide follow up assistance as needed; (4) Improve organization-wide processes. b. Focus on project management issues throughout the system lifecycle. Noted that NPAM is a tool to assess the project itself (not to assess any stakeholders -PM, PMT or contractor/subcontractor-). c. Enable identification and use of common/systemic issues across projects through: (1) Monitoring statistical outcomes; (2) Establishing benchmarks; 1

10 (3) Maintaining and disseminating lessons learned. AQAP 2050 d. Be consistent with NATO quality policy and guidance; based upon approved International standards and best practices. 1.3 Application The NPAM is intended for: a. NATO and national implementation for acquisition, development and/or maintenance projects. b. Collaborative multi-national projects. c. Bi-lateral agreements with Partnership for Peace Nations. 2

11 AQAP 2050 Chapter 2 - Assessment Model The NPAM is comprised of an Assessment Process (AP) and a documented set of issue categories that the assessment should cover. The issue categories provide a consistent baseline by which assessment results are reported. The AP is comprised of eight activities, which are described in this document. The activities include: a. Establish Assessment and Maintain Capability. b. Adopt Assessment Capability. c. Initiate Assessment. d. Plan Assessment. e. Conduct Assessment. f. Analyze and Report Assessment Results. g. Follow up. h. Analyze and Improve. The Issue Categories provide a top-level structure for identifying project issues. All of the issue categories should, at least, be considered by the Assessment Team (AT). Those issues which are deemed to be high priority will be subject to more detailed analysis. Additional issue categories may be added to the assessment. The Issue Categories are: a. External Constraints. b. Mission Requirements. c. Financial. d. Resources. e. Management. f. Processes. g. Product. h. Schedule. i. User / Customer. j. Project Specific. It can be chosen only a few issue categories for conducting the assessment. Also, in many cases it would be convenient to conduct more than one assessments during the project life cycle. 3

12 AQAP Assessment Sponsor (AS) The requestor of the assessment. Chapter 3 - Roles and Responsibilities For a specific project, the AS could be the Project Manager (PM) or sometimes the Enterprise Customer (EC) when is related to mentioned specific project or some other authority involved in this specific project and interested in project outcomes. This person determines the scope and use of the assessment. This person also ensures that assessment results and recommendations are reviewed for quality and consistency at both the project and enterprise levels and provides feedback to the Model Management Team (MMT). This feedback includes the assessment of process performance and non-attributable (sanitized) assessment data. 3.2 Project Manager (PM) The assigned manager for a specific project. Responsible for all aspects of the project and related decisions. The primary user of the project assessment results. 3.3 Assessment Team (AT) This team is comprised of a Team Leader and selected participants from National and/or NATO resources. The Assessment Team is responsible for executing the core activities of the assessment process after assessment initiation. Team makeup is based on the projected assessment requirements. Team members are independent from the assessed project. 3.4 Assessment Team Leader (ATL) Responsible for the planning and conduct of a specific assessment in accordance with the Assessment Method and related guidance. The ATL is appointed by and interfaces directly with the AS. The ATL organizes and manages the AT. 3.5 Project Liaison (PL) Representative from the project/organization who will interface with the AT. The PL is responsible for providing background information to the team, and assistance with site visit logistics and planning. 3.6 Independent Peer Reviewer-Optional (PR) The person or persons designated to independently review the assessment findings, results, conclusions, and recommendations for a given assessment. The PR evaluates the assessment results for compliance with the assessment process and to 4

13 AQAP 2050 verify that assessment conclusions and recommendations are consistent with the findings. The PR is not a member of the AT. 3.7 Enterprise Customer (EC) A customer for enterprise level analysis of results and products related to a specific project. Examples of EC are national/international acquisition organizations. These organizations could be NATO agencies or commands, NATO/PfP Nations or MoD/Defence organizations. 3.8 Assessment Management Board (AMB) Responsible for overall model policy, maintaining the assessment capability, and monitoring its use, effectiveness, and efficiency. The AMB is supported by the MMT. Note: It is envisioned that this function will be performed by a NATO organization. 3.9 Model Management Team (MMT) This team is responsible for the maintenance and improvement of the NPAM as well as for developing training concept and material. Responsible for non-attributable (sanitized) data collection, systemic analysis, and reporting, and assessment process performance analysis. Note: It is envisioned that this function will be performed by a NATO organization. 5

14 Chapter 4 - Assessment Process 4.1 Process Activities There are eight activities which comprise the assessment process: a. Establish and Maintain Assessment Capability. b. Adopt Assessment Capability. c. Initiate Assessment. d. Plan Assessment. e. Conduct Assessment. f. Analyze and Report Assessment Results. g. Follow up. h. Analyze and Improve. Figure 1 depicts the overall NATO Assessment Process. The characteristics of the process are described below. NATO PROJECT ASSESSMENT MODEL - PROCESS MODEL ESTABLISH and and MAINTAIN ASSESSMENT CAPABILITY by by Assessment Assessment Management Management Board Board NATO NATO PROJECT ASSESSMENT METHODOLOGY MODEL STAKEHOLDERS STAKEHOLDERS STAKEHOLDERS 2. ADOPT ASSESSMENT METHODOLOGY CAPABILITY by by Enterprise Enterprise Customer Customer SPONSOR, PM and AT INITIATE INITIATE ASSESSMENT ASSESSMENT by by Sponsor Sponsor PLAN PLAN ASSESSMENT by Team by ATL Leader CONDUCT CONDUCT ASSESSMENT ASSESSMENT by by AT Team Presentation to PM - AT SPONSOR 7. FOLLOW UP ACTION by Project by PM Manager Update Through Assessment Management Board SYSTEMIC ISSUES ANALYSE & IMPROVE by Methodology by MTT Team FEEDBACK ANALYSE ANALYSE & REPORT REPORT by by Team AT/ ATL Leader Core Activities REPOSITORY: - SANITIZED DATA DATA (controlled by PM) - PROCESS FEEDBACK by Methodology Maintain and Improve Team by MMT INITIAL ASSESSMENT REPORT FINAL ASSESSMENT REPORT REPORT 6

15 From a wide range of analysed assessments, systemic issues contents a description of common issues arose during the assessment performance, typical problems found, key issues to deal with in a specific project, etc. Four of the assessment process activities are designated as core assessment activities. The core activities are: a. Initiate Assessment. b. Plan Assessment. c. Conduct Assessment. d. Analyze and Report Assessment Results. These core activities encompass the assessment tasks and interactions necessary for specific project assessments. These activities are the primary responsibility of the designated project AT Tasks Each activity in the AP is comprised of tasks, responsibility and products. The tasks delineate the specific actions required to implement the activity. Individual tasks may be tailored to best address the characteristics of the project being assessed, the assessment time frame, and the assessment scope. The AT should document the manner in which the tasks are tailored in the assessment plan. Assessments are conducted on individual projects in accordance with the AS requirements. There are different types of assessments, generally defined in terms of the purpose and scope of the information needs of the AS and the PM. They include: a. Assessments of project general situation. b. Assessments focused on specific project issues. c. Assessments focused on specific project organizations. d. Software and/or system assessments. Assessments are focused on the success of the project. No individual assessment results are provided by the AT to AS and/or EC without the concurrence of the PM. The PM controls the release and dissemination to MMT of all project specific assessment results. The MMT is responsible for analyzing and reporting systemic issues to stakeholders (whoever involved in the AP) in order to support enterprise level management and decision-making. Each activity that comprises the AP is individually described in the following sections (Figure 1 refers). The activities are described in terms of: a. The purpose of the activity. b. The tasks which comprise the activity. c. Responsibilities for the implementing the activity. 7

16 d. Products of the activity. 4.2 Establish Assessment Capability Purpose The Establish Assessment Capability activity is to implement and maintain the NPAM. This includes providing the necessary infrastructure to maintain and improve the capability and to perform systemic analyses Tasks Implement repository and method of use; Implement training concept and materials; Establish a MMT; Develop and implement model performance measurement process and an associated database Maintain the document Responsibility AMB Products List of capable assessment offices; MMT; Training materials; Systemic analysis results and associated information technology infrastructure. 4.3 Adopt Assessment Capability Purpose The Adopt Assessment Capability activity is the process by which the NPAM is adapted/translated for application. Note: This involves ensuring the technical and management infrastructure, and qualified technical resources, are available to effectively conduct assessments. Implementation of the tasks associated with this activity results in an organization capable of performing assessments within that Enterprise and improving the manner in which the assessments are conducted. 8

17 4.3.2 Tasks Implement the standard assessment model Establish assessment team member qualifications and associated team selection criteria Develop and update the participant s qualifications, training, and experience database Responsibility EC Products Adopted Model documentation; Assessment tool, technique, and reference library and associated components; Model performance measurement database; Assessment education and training courses and materials; AT members capability database. 4.4 Initiate Assessment Purpose The Initiate Assessment activity is the first of the four core assessment activities and must be initiated only by AS. This activity includes those management and technical tasks required to initiate the AP. Follow-up assessments may be initiated based upon recommendations from a previous assessment report, or may be a result of feedback by a PM when previous assessment recommendations did not provide necessary corrective action Tasks Establish the need for, and the agreement to conduct an assessment. This task includes determination of the scope of the assessment to be conducted Appoint the ATL Appoint the PL (in accordance with PM's proposal) Responsibility AS. 9

18 4.4.4 Products Agreement to conduct an assessment; Assessment scope; Identification of the PL. 4.5 Plan Assessment Purpose The Plan Assessment activity includes those management and technical tasks required to conduct the assessment. Implementation of these tasks results in the coordination of a project AT, a consensus plan/approach for conducting a project assessment, and an initial profile of project characteristics and issues in accordance with the assessment scope Tasks Obtain, disseminate, and review background information. This includes characteristics and associated project documentation Communicate with the PM to identify concerns. From this communication, determine the initial project issues Identify team participants based upon the initial project issues, project domain/functional area, and assessment scope. Team members should be a mix of experienced assessors and subject matter experts related to project initial list of issues Ensure team members are trained in the NPAM Obtain appropriate confidentiality agreements for participants, in order to avoid that data collected during project assessment activities can be spread out In cooperation with the team members, plan the project assessment and document the approach. Planning information may include: Assessment objectives and strategy; Assigned assessment resources; Selected assessment issue categories and prioritization; Assessment schedule; Information needs/sources; Assessment products Responsibility ATL Products Initial issues; AT; Confidentiality agreements; Assessment plan. 10

19 4.6 Conduct Assessment Purpose The Conduct Assessment activity includes identification of strengths, concerns, and risks of the assessed project based upon consideration of the NPAM Issue Categories (Annex A), and in-depth investigation into critical project issues. This investigation may include customer and supplier site visits (only to prime and first tier sub-suppliers), interviews, and review of objective evidence. The Conduct Assessment activity includes validating and updating the initial assessment issues Tasks Identify and document issues in accordance with the assessment scope and the Issue Categories. Confirm and/or corroborate findings that support the identified issues. The ATL will maintain an interface with PL to discuss any changes to the assessment plan Responsibility AT Products Documented findings; Initial observation briefings as necessary. 4.7 Analyze and Report Assessment Results Purpose The Analyze and Report Assessment Results activity includes tasks required to relate the issues that have been individually evaluated, analyze and integrate the results in terms of cause and effect, and report the results with appropriate recommendations for the project. As a result of this activity, the assessment customers (AS and PM) understands key project issues and receives actionable recommendations. In addition, sanitized project assessment results (findings arose during the assessment of a specific project and related exclusively to the NPAM application itself and the AP) should be made available for systemic analysis Tasks Conduct in-depth analysis of key project issues Document findings and results to create the final assessment issue profile. 11

20 Evaluate and describe issues in terms of associated project objectives, assumptions, and constraints. Address strengths, problems, risks, and lack of information related to issue findings Prioritize the issues with respect to project impact Analyze issues to determine cause and effect Generate recommendations related to the issues and causes, and determine whether a future assessment should be recommended Develop a draft assessment report. It must reflect the consensus of the AT. The report may be reviewed by a PR. The draft report is reviewed with the PM. The PM is given the opportunity to comment and/or add additional information Finalize the report and disseminate to the PM and the AS Provide feedback to the MMT about issues concerning the application of model including required performance measures, assessment model related lessons learned, and sanitized issues from the assessment Solicit feedback on the assessment from the AT, the PM and AS to the MMT Responsibility ATL Products Final Assessment Report to the PM and AS; Assessment feedback and sanitized assessment results to the MMT. 4.8 Follow-up Purpose The Follow-up activity includes those tasks required to produce and implement the action plan Tasks The PM should use the Final Assessment Report to determine and prioritize the actions that will be taken to address the issues Determine and prioritize actions that will be taken by the project and others Designate resources, schedules and responsibility for the actions. 12

21 PM must report to AS about the Project Manager's action plan. Also he will provide to AS reports about the Action Plan fulfillment (when it would be required by AS or systematically) Determine need for a future assessment Track actions to closure Responsibility PM Products Project Manager s action plan. 4.9 Analyze and Improve Purpose The Analyze and Improve activity includes tasks required to determine systemic issues that are affecting project performance. Further, this activity includes tasks required to maintain the NPAM and evaluate its performance. This includes analysis of sanitized assessment results and process feedback across a wide range of assessments Tasks Perform statistical analysis from sanitized assessment data to determine systemic issues, including state of practice in compliance with current allied publications Determine recommended changes to this allied publication and other appropriate actions based upon the above Determine the benefits of applying the NPAM based upon feedback received from assessed PMs and Sponsor Make systemic issues available for use by all stakeholders Suggest maintenance and improvement actions for the NPAM and associated products based upon feedback received to the Assessment Management Board. 13

22 Maintain and improve the supporting information technology infrastructure, including the repository. The following is a list of sanitized assessment data that could be included in the repository: Assessments Results (project profile, assessment plan, assessment schedule,...) and Lessons Learned (both customer and AT feedback) Responsibility MMT Products Systemic issue analysis information; Proposed updates to the NPAM and supporting materials; Proposed updates to allied publications and other actions; Updates to the information technology infrastructure, including the repository. 14

23 ANNEX A ASSESSMENT ISSUE DESCRIPTION A-1

24 Issue Category Issue Sub - Issue 1. External Constraints 1.1 Regulatory Environment Legal Policy Audits & Assessments 1.2 Workplace Environment Cooperation Morale Culture 1.3 Political Environment Legislative Agendas Customer s Agenda Supplier s Agenda 1.4 Other Constraints Corporate Industrial Scheme 2. Mission Requirements 2.1 Operational Requirements Reasonableness Stability Dependencies Change Tolerance 3. Financial 3.1 Funding Sufficiency Timeliness Continuity/Stability Flexibility 3.2 Budget Allocation Variance Control 4. Resources 4.1 Personnel Qualifications Staffing Availability 4.2 Facilities Capital Equipment Infrastructure 4.3 Tools Support Tools Information Systems 4.4 Government Furnished Equipment Information 4.5 Supplier Furnished Commercial Off-The-Shelf (COTS) Non-Developed Items (NDI) Developed Items (DI) 4.6 Selection of Prime Contractors/Supplier Integrity Longevity Sub-suppliers 5. Management 5.1 Acquisition Strategy Acceptability Feasibility Suitability 5.2 Project Planning Acceptability Feasibility Suitability 5.3 Project Management Organisation Suitability Monitoring and Control Change Management A-2

25 Issue Category Issue Sub - Issue 5.4 Contracting and Subcontracting A Conditions/Constraints Cost Accounting Progress Tracking Arrangements Timeliness Change Management 5.5 Communication Interfaces Openness Teamwork 5.6 Quality Management Quality Management System 5.7 Documentation Management Availability Adequacy 6. Processes 6.1 Conformance Compliance Process Employment Process Compatibility 6.2 Capability Fitness for Purpose Efficiency Improvement 7. Product 7.1 Requirements Completeness Correctness Feasibility Stability Traceability 7.2 Design Architecture Scale Complexity Technology Effectiveness Interoperability Reuse 7.3 Attributes Usability Performance Dependability Reliability/Availability Supportability/ Maintainability Reusability Portability Efficiency Integrity Survivability Safety 7.4 Technical Constraints Human Factors Health & Safety Security Natural Environment Protection 8. Schedule 8.1 Constraints Feasibility Dependencies Contingency Plans 8.2 Progress Estimation Accuracy Visibility Progress Performance

26 Issue Category Issue Sub - Issue Rework 9. User/Customer 9.1 Satisfaction Involvement User Acceptance 9.2 Transition Transition Support Training 9.3 Support Maintenance Assistance Warranty 10. Project Specific Table 1. Issue Structure A-4

27 0.1 Issue Categories Chapter 0 - Introduction The issue categories are used to capture the high-level concerns (risks, problems and uncertainties/lack of information) generated during the assessment. They are applicable to the whole effort being assessed, i.e., the organizational, systems and software aspects of a project or enterprise. The issues are dynamic and multi-dimensioned in nature. This means that an issue will change dependent on time (e.g., life cycle phase), stakeholder perspective (e.g., customer or supplier), priority of importance, triggering event (i.e., the event(s) that initiated the issue), and the objectives, assumptions and constraints that exist when the issue is being examined. 0.2 Issue Structure Application The list of issues is not meant to be exhaustive, but rather a minimum set of concerns that the ATL and team members should survey to help initiate and sustain the assessment. Further, the issues are only described in general terms and are not intended to cover the specifics of every possible situation the team may find itself assessing. It is required that each issue category will be reviewed by the AT as part of the issue identification process. However, it is not expected that every issue or sub-issue will necessarily be addressed during an assessment. It is not expected that every team member will be an expert in every issue category. However, the team as a whole should be able to relate the general issue descriptions to the specific circumstances of the effort under assessment. Additionally, some issues necessarily overlap or are repeated because of certain items that require emphasis based upon a perspective or triggering event. Again, the issue categories should be used as a starting point only, and be supplemented with other domain specific questionnaires, surveys, taxonomies, personal experience, etc., as the situation requires. A Project Specific issue category exists to allow an AT to add new issues to the structure where appropriate. The sub-issues have a customer and supplier perspective from which they may be assessed. Consequently, the AT shall be aware that there is both a customer and supplier aspect. Many times the assessment guidance provided in the perspectives are nearly identical, but sometimes they are not. For example, the political agendas of the customer and supplier may be very different from one another, while their ability to hire personnel may be very similar. Be aware that even when a sub-issue s A-5

28 perspective appears the same, the actions or recommendations to address the issue may be very different because of the unique perspective involved. The guidance provided follows a convention where the word evaluate is used to examine things that are factual in nature, i.e.; they can be objectively measured. The word contrast is used to try to show differences, while compare is meant to convey similarities. The word assess is used to render some sort of judgment. Finally, the ATL should endeavor to look for issues beyond this list to reduce the possibility of issue blindness, i.e., seeing only what is on the list of issues. A-6

29 Chapter 1 - External Constraints Scope: This issue category concerns the context in which the project or enterprise operates, the boundaries that constrain it, and any externalities that may influence it. 1.1 Regulatory Environment Scope: This issue concerns the regulatory constraints under which the project or enterprise operates Legal Scope: The effort s compliance with international, national and local laws. Customer Perspective: Examine general legal constraints, impacts of acquisition regulations, and potential changes in legislation. Evaluate the effort s compliance with appropriate legal requirements. Assess whether the level of compliance is acceptable. Supplier Perspective: Examine legal requirements imposed on the effort. Evaluate copyrights, patents, trade secrets or intellectual property rights. Assess the impact of non-compliance with or changes to the legal requirements Policy Scope: The current governmental, NATO policies, and industry standards that influence the effort s acquisition, development, operational or sustaining engineering. Customer Perspective: Examine policy constraints, policy that may be changing, and/or policies that mandate the use of specific technology solutions. Evaluate these policies for consistency. Assess whether these policies have any implications on the effort. Supplier Perspective: Examine internal corporate policies (e.g., purchasing, finance, personnel administration, labor, at contract at risk policies) as well as governmental policies. Evaluate the constraints these policies place on the effort. Assess the impact of changes in policy Audits & Assessments Scope: The compliance with previous external and internal reviews/audits/ assessments performed on the effort. Customer Perspective: Examine previous reviews/audits/assessments performed on the assessed project and those performed on projects of a similar nature. Evaluate the recommendations and their implementations, which recommendations were not A-7

30 followed, and the rationale for not following the recommendations. Assess the adequacy of the previous reviews/audit/assessments and their recommendations. Supplier Perspective: Examine previous reviews/audits/assessments. Evaluate the recommendations and their implementations, which recommendations were not followed, and the rational for not following recommendations. Assess the validity and/or impacts of not following previous recommendations (e.g., the likelihood of passing the next major decision milestone). 1.2 Workplace Environment Scope: This issue concerns the general situation existing within the stakeholders workplace Cooperation Scope: The level of teamwork existing among the effort s stakeholders. Customer Perspective: Examine the work agreements in place, the teamwork that exists, including cooperation within internal groups. Evaluate the willingness to compromise, the number and types of confrontations that occur (e.g., between customer and sponsor, customer and supplier), the conflict resolution process, and cooperation received from supplier as well as projects/programs with which the effort must interface. Assess whether the level of cooperation is acceptable. Supplier Perspective: Examine the work agreements in place and the level of teamwork that exists. Evaluate the acceptance of reasonable direction and instructions from customer or other parties, the degree of commitment to effort by the stakeholders, the general openness towards resolving problems and risks. Evaluate the cooperation that exists within the organization (e.g., with corporate headquarters) as well as among subcontractors/vendors. Assess the impact of changes in the level of cooperation on the effort Morale Scope: The degree of enthusiasm and commitment that exists among the effort s stakeholders and their personnel. Customer Perspective: Examine the morale of the effort s stakeholders. Evaluate the personnel turnover rate, personnel satisfaction, leadership visibility, the number of crisis management meetings, the number of changes in schedule. Assess whether the morale is acceptable. A-8

31 Supplier Perspective: Examine the morale of the effort s stakeholders. Evaluate the personnel turnover rate personnel satisfaction, the current reputations of the stakeholders, the productivity levels, the promptness of remedial actions. Assess the impacts of a change in morale on the effort Culture Scope: The unwritten rules of organizational behavior that exist within the effort s stakeholders that may influence the success of the effort. Customer Perspective: Examine the organizational culture (customer and supplier). Evaluate their organizational assumptions, how decisions are made, unwritten codes of behavior, and where power resides. Assess whether the varied organizational cultures support the effort or whether culture incompatibilities will ensue. Supplier Perspective: Examine the organizational cultures (customer, supplier, vendors, and subcontractors). Evaluate their codes of conduct, vision statements, what is socially valued, internal paths leading to success. Assess the impacts of changes or incompatibilities in culture. 1.3 Political Environment Scope: This issue concerns the general situation existing within the overall political environment Legislative Agendas Scope: Current and pending legislation, a sense of Governmental or other legislative bodies or institutions that may influence the effort s success. Customer Perspective: Examine current and pending legislation. Evaluate the potential impacts of pending/proposed legislation, related support (e.g., lobbying groups, newspapers, think tanks) on the effort. Assess whether the legislative agendas support the effort. Supplier Perspective: Examine the potential impacts of pending/proposed legislation; and supporting organizations. Assess the impacts of changes in legislative agendas on the effort Customer s Agenda Scope: The political objectives of effort s customer and/or sponsor. Customer Perspective: Examine political objectives of the customer, the customer s sponsor, support contractors and consultants. Compare and contrast their respective political objectives. Evaluate the actions of the sponsors to safeguard the effort and A-9

32 their willingness to champion the effort. Assess whether the agendas support the customer s agenda for the effort. Supplier Perspective: Examine the customer s agenda. Evaluate impact of the customer s (or supporting contractors) political goals conflicting with the technical goals of the effort. Assess the impact of changes in the customer s agendas Supplier s Agenda Scope: The political (market and economic) objectives of the effort s supplier and other team members. Customer Perspective: Examine the supplier s (and its subcontractors) agenda. Evaluate the influence of the supplier on international or political environment, the economic strength of supplier, the current competitive environment, the supplier s long and short-term market/financial goals. Assess whether the supplier s agenda support the customer s agenda. Supplier Perspective: Examine all the stakeholders agendas. Compare and contrast the political and economic goals of supplier and team members against technical/customer s goals for project. Evaluate the actions of the supplier to safeguard the effort and champion the effort (e.g., risk sharing). Assess the impacts of agenda conflict or changes in anyone s agenda. 1.4 Other Constraints Scope: This issue concerns other external constraints that impact the project Corporate Scope: The current company-wide policy procedures and processes regulated by the own company for its function that influence the effort s acquisition. Customer Perspective: Examine company policy constraints, procedures and best practices to be applied to the effort, policy that may be changing, and/or policies that establishes or propose the use of specific technology solutions. Evaluate these policies, procedures and best practices for consistency. Assess the implications of such policies, procedures and best practices on the effort. Supplier Perspective: Examine company policy constraints, procedures and best practices to be applied to the effort, policy that may be changing, and/or policies that establishes or propose the use of specific technology solutions. Evaluate these policies, procedures and best practices for consistency. Assess the implications of such policies, procedures and best practices on the effort. A-10

33 1.4.2 Industrial Scheme Scope: This issue concerns the industrial scheme (e.g. joint venture, international Competitive Biding, Partnership, ) under which the project will be executed and its impact on the assessed project. Customer Perspective: Examine the scheme to see how management is shared among the participant companies, and evaluate associated rules and procedures. Evaluate the rules established to regulate their participation, responsibilities and distribution of tasks and charges to meet contract requirements. Assess the responsibilities determined for each participant and their contribution to the effort. Supplier Perspective: Examine the scheme to see how management is shared among the participant companies, and evaluate associated rules and procedures. Consider the marketplace, legal and data sharing environments. Assess the responsibilities, budget and financial distribution of each participant with respect to their contribution to the effort. A-11

34 Chapter 2 - Mission Requirements Scope: This issue category concerns how well a mission requirement can be supported or the validity of a mission-related capability against a threat. 2.1 Operational Requirements Scope: This issue concerns the forces that can influence the mission requirement Reasonableness Scope: The operational requirements and the effort s technical approach. Customer Perspective: Examine the operational requirements. Assess the reasonableness of the requirements. Assess whether the implementation is reasonable given the proposed technology and acquisition, development and/or support approach. Supplier Perspective: Examine the operational requirements. Assess the reasonableness of the requirements. Assess whether the implementation is reasonable given the proposed technology and acquisition, development and/or support approach. Assess the impact of operational requirement changes to the reasonableness of the effort Stability Scope: The stability of the operational requirements. Customer Perspective: Examine the threat environment, operational doctrine, legislation, and/or operational & test feedback. Evaluate each for changes (e.g., threat projections, pending doctrinal changes, interoperability requirements, legislative proposals) that might modify the mission requirements. Assess the impact of operational requirements change/volatility on the effort. Supplier Perspective: Examine the threat environment, operational doctrine, legislation, and/or operational & test feedback. Evaluate each for change (e.g., pending doctrinal changes, interoperability requirements, legislative proposals) that might modify the mission requirements. Assess the impact of operational requirements change/volatility on the effort s functionality, cost and schedule Dependencies Scope: The impact of external dependencies on the operational requirements. Customer Perspective: Examine operational requirements. Evaluate their dependency on the existence of other mission requirements (e.g., those provided by A-12

35 other platforms). Assess the impact of changes in mission requirements on the effort and/or by the effort on other missions. Supplier Perspective: Examine operational requirements. Evaluate their dependency on the existence of other mission requirements (e.g., those provided by other platforms). Assess the criticality of this dependency on the capability to support the effort s mission Change Tolerance Scope: The capability of the effort to manage the impacts of changes to the mission requirements. Customer Perspective: Examine the risk management, change management, requirements management, and rework procedures. Assess their robustness. Assess whether they can smoothly handle a change in mission requirements. Supplier Perspective: Examine the risk management, change management, requirements management, and rework procedures. Assess their robustness. Assess whether they can smoothly handle a change in mission requirements. A-13

36 Chapter 3 Financial Scope: This issue category concerns the process of monetary control being exercised over the effort. 3.1 Funding Scope: This issue concerns the process involved in funding the effort Sufficiency Scope: The amount of funding available for the effort and the support for continued funding of the effort. Customer Perspective: Examine the funding profile and expenditure rates. Evaluate the adequacy of the funding amounts against the effort s objectives. Assess the continued funding support of effort by sponsors. Assess whether the funding and support are acceptable. Supplier Perspective: Examine the funding received. Evaluate its adequacy against contractual requirements and obligations. Assess any impacts of funding slow-down and/or reductions. Assess the impacts of changes in the support of effort by customer and internal sponsors Timeliness Scope: The funding profile and timeline. Customer Perspective: Examine the funding profile and timeline. Compare and contrast the funding timeline against contractual requirements and expenditures. Evaluate whether the funding profile maps to when the resources are required, factoring in lead times. Assess the adequacy of the timeline s fit against the requirements and expenditures. Supplier Perspective: Examine the funding profile and timeline. Compare and contrast the funding timeline against contractual requirements and expenditures. Assess the impacts of changes in the funding timeline on current contractual requirements and liabilities (e.g., cash flow requirements) Continuity/Stability Scope: The funding profile and timeline. Customer Perspective: Examine the funding profile and timeline. Evaluate the continuity and stability of the funding profile. Assess whether the continuity/stability of the funding is acceptable. A-14

37 Supplier Perspective: Examine the funding profile and timeline. Evaluate the continuity/steadiness of the funding profile. Assess the impacts of changes in continuity/stability (e.g., a sudden reprogramming of the funding) against effort s cost, schedule, and technical objectives Flexibility Scope: The funding profile, timeline, funding process and the type (i.e., color ) of funding available. Customer Perspective: Examine the funding profile, timeline, funding process and funding type. Evaluate the capability to reallocate or transfer funding. Assess whether there is sufficient funding flexibility. Supplier Perspective: Examine the funding profile, timeline and process. Evaluate the capability to absorb changes to funding. Assess the impacts of changes in funding profile on the effort. 3.2 Budget Scope: This issue concerns the process of budgetary control exercised over the effort Allocation Scope: The budgetary process and disbursements. Customer Perspective: Examine the budgetary process and disbursements. Evaluate the appropriateness of the funding allocation (e.g., the allocation of funding to subcontractors) as well as the value for money spent. Contrast the allocation against current and proposed work. Assess whether the allocation is acceptable. Supplier Perspective: Examine the budgetary process and disbursements. Compare and contrast the current and proposed work against budget allocations and expenditures. Assess mix and balance of the allocations against the work performed. Assess the impact of changes to the allocation mix (e.g., a funding shortfall) Variance Scope: The budgeting process and expenditures. Customer Perspective: Examine the budgeting process and expenditures. Evaluate the accuracy and precision of the budgetary expenditures. Contrast the funding against expenditures. Assess the impact of any variance in estimated versus actual budget. Assess whether the accuracy of the budget is acceptable. A-15

38 Supplier Perspective: Examine the budgeting process and expenditures. Contrast the funding against expenditures. Assess the impact of any variance in estimated versus actual budget. Assess the impact of changes in the budgetary information Control Scope: The budgetary process and expenditures. Customer Perspective: Examine the budgetary control process. Evaluate how funds are spent, monitored and reported (e.g., comparing budgeting allocated funds and expenditures profile). Assess whether the financial controls are acceptable. Supplier Perspective: Examine the budgetary control process. Evaluate how funds are spent, monitored and reported. Assess the impacts of changes to the financial control process. A-16

39 Chapter 4 Resources Scope: This issue category concerns the assets needed and available to meet the effort s objectives, including but not limited to personnel, facilities, training, etc. 4.1 Personnel Scope: This issue concerns the quality of the expertise available to the effort Qualifications Scope: The relevant knowledge, experience and training of the personnel assigned to the effort. Customer Perspective: Examine the expertise of the personnel (customer and supplier) assigned to the effort. Contrast the relevance of the expertise, domain knowledge, and training against effort s needs. Assess the adequacy of the expertise. Supplier Perspective: Examine the expertise of the supplier personnel assigned to the effort. Contrast the relevance of the expertise, domain knowledge, and training against effort s needs. Assess the impacts of any change in the level of expertise Staffing Scope: The staffing profile. Customer Perspective: Examine staffing profile. Evaluate the customer and supplier staffing. Compare and contrast the staffing adequacy, mix, turnover, apportionment balance, and timeliness of staffing against current and proposed work. Assess the adequacy of the staffing profile. Supplier Perspective: Examine staffing profile. Compare and contrast the staffing adequacy, mix, turnover, apportionment balance, and timeliness of staffing against current and proposed work. Assess whether the personnel bid for are assigned to the effort. Assess any changes in the staffing profile Availability Scope: The current economic/organizational situation in regard to personnel supply and demand. Customer Perspective: Examine the competitive marketplace as well as constraints on personnel availability (e.g., personnel ceilings, promotion ceilings, internal competition for qualified personnel, etc.). Evaluate the availability of qualified A-17

40 personnel to support the effort (customer and supplier). Assess whether there exists an ability to hire and retain qualified personnel. Supplier Perspective: Examine the competitive marketplace. Evaluate the availability of qualified personnel to support the effort (e.g., experienced Ada programmers). Assess the ability to hire and retain qualified personnel. Assess the circumstances that support or negatively impact the ability to put together a qualified team of people. Assess the impacts of changes in the marketplace. 4.2 Facilities Scope: This issue concerns the organizational technology, e.g., plant and machinery, required for supporting the effort Capital Equipment Scope: The profile of the capital equipment required in supporting the effort. Customer Perspective: Examine the capital equipment provided. Contrast the equipment against the effort s requirements. Assess whether the capital equipment is sufficient. Supplier Perspective: Examine the capital equipment provided. Contrast the equipment against the effort s requirements. Assess the impacts of any changes in capital requirements Infrastructure Scope: The infrastructure required for supporting the effort. Customer Perspective: Examine the infrastructure provided (e.g., office space, facilities, ergonomic, and safety aspects, etc.). Contrast the infrastructure against the effort s requirements. Assess whether the infrastructure is adequate. Supplier Perspective: Examine the infrastructure provided. Contrast the infrastructure against the effort s requirements. Assess the impacts of changes in infrastructure. 4.3 Tools Scope: This issue concerns the range of supporting tools, techniques, etc., required for supporting the effort. A-18

41 4.3.1 Support Tools Scope: The range of tools required for acquiring, developing or maintaining the effort. Customer Perspective: Examine the support tools provided. Contrast the tools functional match against the effort s requirements. Assess whether the tools provided are sufficient in number and quality. Supplier Perspective: Examine the support tools provided. Contrast the tools against the effort s requirements. Assess the level of trained personnel on the tools. Evaluate if the tools provided are sufficient in number and quality. Assess the impacts of any changes in development tools Information Systems Scope: The information systems required in supporting the effort. Customer Perspective: Examine the information systems provided that support the effort (e.g., MIS, financial accounting, communications/ information capture including web-based capabilities). Assess their adequacy against both the customer and supplier s requirements to manage the effort. Supplier Perspective: Examine the information systems provided that support the effort (e.g., MIS, financial accounting, communications including web-based capabilities). Evaluate their adequacy against the effort s needs. Assess the impacts of information system changes. 4.4 Government Furnished Scope: This issue concerns the items available from the government that are being used (or planned for use) in the effort Equipment Scope: The Government Furnished Equipment (GFE) required to support the effort. Customer Perspective: Examine the GFE provided. Compare and contrast GFE against the effort s requirements. Evaluate its availability, reliability, maintainability, supportability, etc. Assess whether the GFE used is adequate. Supplier Perspective: Examine the GFE provided. Compare and contrast the GFE against the effort s requirements. Evaluate its availability, reliability, maintainability, supportability, etc. Assess the impacts of any changes in GFE. A-19