Assessment RFP Development Toolkit Introduction to the Toolkit Joseph A Martineau Senior Associate Who is the Audience for this Toolkit? This toolkit is targeted toward state education agency (SEA) staff responsible for developing requests for proposals (RFPs) for contracted assessment services. This is likely to be primarily assessment directors and their staff, so they are the intended audience. However, developing an RFP for contracted assessment services will also likely require input from procurement, contracts, finance, and information technology directors and their staff (with the understanding that not all agencies will have all roles). In sections needing input from external staff, I attempt to assist assessment staff with including the types of information that external staff will need to provide. Although this toolkit was developed with SEA assessment staff in mind, it can be adapted for use by analogous staff in assessment consortia. The same is true of regional service agencies (or intermediate school districts) and local education agencies (LEAs, or local school districts) with sufficient human resources devoted to managing assessment services contracts. How Can this Toolkit be Helpful? This document builds on lessons I have learned over thirteen years writing RFPs and advising others responsible for doing so. Over those thirteen years I have served in various roles, including SEA Psychometrician, SEA Program Manager, SEA Assessment Director, SEA Deputy Superintendent, and consultant or advisor to various other people in similar roles. My experience in writing RFPs and implementing or managing the resulting contracts covers roles on the front line of operations, middle management, senior management, and external consultant. More to the point, I also have thirteen years of experiencing the consequences of how RFPs were developed. Some of those consequences were good, some were bad, and some were downright awful. A good RFP can protect and balance a state s interests in the areas of quality of deliverables, meeting timelines, reducing strain on staff, and minimizing cost. A good RFP can also maximize fairness for bidders, reduce the likelihood of protests from non-winning bidders, and help to foster good relationships with vendors that are essential to effective collaboration. A bad RFP can create all kinds of headaches for SEAs, such as exploding costs, heavy strain on staff, poor relationships with the vendor, critical missed deadlines, poor-quality deliverables, disastrous security breaches, and anger and distrust from constituents. From my experience, I can definitively say that it is worth the effort of creating a strong RFP to avoid such problems. Toolkit Component: Model RFP Outline Components of the Toolkit This is the main component of the toolkit. It provides a comprehensive outline of topics to include in an RFP for contracted assessment services. It is not intended to be a one-stop shop, but a
Assessment RFP Development Toolkit Introduction to the Toolkit 2 resource from which SEA staff can pick and choose what is applicable to their programs, ignoring the rest. The comprehensiveness of the model RFP is to assist SEA staff avoid forgetting important components. The comprehensiveness is not intended to signal that every section is applicable for every RFP. Likewise, SEA staff should not feel bound by the structure developed in the model RFP. They should be encouraged to adapt components (and their ordering) and document formats to meet SEA needs and procurement requirements. The model RFP incorporates several key lessons learned in my experience managing over 15 contracts with different vendors. Those lessons learned are addressed in the following areas: Balancing Adequate Specification with Flexibility in the Scope of Work. Some vendors are comfortable working in a high-stakes environment that is frequently buffeted by political, technical, and logistical issues. Such vendors allow for program evolution over time by modifying deliverables, services, and timelines without significant cost if the scope of effort and/or deliverables remain reasonably similar. Others can be much more risk averse, preferring a legalistic approach in interpreting the scope of work. They may require official changes to the contract even for minor changes in scope or effort, and may do so at considerable increased cost. The level of specificity detailed in the model RFP outline protects the SEA s interest by reducing exposure to risk of increased cost, poor contractor performance, and poor-quality deliverables. It also helps bidders utilize the scope of work to develop stronger cost proposals. The approach to specifying bid costs described in the model cost proposal worksheets (described below) balances precision with the ability to respond to contingencies, in many cases without needing to change the contract. Ability to Provide Customized Solutions. Some vendors have a core set of products, services, and timelines that are unreasonably difficult to customize, even in minor ways. The reasons may include corporate policy, corporate culture, program management infrastructure or information technology infrastructure. The level of specificity detailed in this model outline is intended to minimize the risk of vague bids that only appear to satisfy the requirements of the RFP (when the bids in fact describe standard products, deliverables, services, and/or timelines that conflict with RFP requirements). Issuing a highly detailed RFP can help bid evaluators identify bidders that are unwilling or unable to customize as needed, which facilitates evaluating the importance of such unwillingness or inability with more complete information. Clarity on Identities and Roles of Client, Vendor, and Third Party Stakeholders. Confusion sometimes arises regarding which organization is the primary client. While various entities have a stake in the contractor s work (e.g. legislatures, school boards, advocacy groups), it is essential that the both bidders and selected vendors clearly know who assigns, oversees, and evaluates work; and clearly know and (are committed to using) proper channels of communication. While these characteristics describe below are not universal, the general structure of appropriate relationships in a state testing program is as follows: The funding body is the legislature. The procurement/budgeting/auditing body is the executive branch, led by the governor. The policy oversight body is the board of education. The vendor s client is the SEA.
Assessment RFP Development Toolkit Introduction to the Toolkit 3 The SEA s stakeholders/clients are students and educators (generally represented by advocacy organizations such as leaders of unions and professional associations.) It is important that an RFP clearly state the roles of various groups and that the statement of work and terms and conditions are clearly aligned with those roles. When appropriate roles are muddied, it can create considerable confusion and conflicts of interest. For example, When vendor, legislators or state board members develop independent relationships without including the SEA, it can create a conflict of interest for both the legislature (a body that typically establishes policy and appropriates funds) and for the vendor (an entity that needs clarity about who directs and evaluates work performance). When vendors develop independent relationships with leaders of advocacy groups (such as unions and professional associations) outside the SEA, it can create a conflict of interest for the association heads (in their appropriate job of lobbying for what is best for their constituents) and the vendor (in terms of being able to intentionally influence lobbying by advocacy organizations that may result in statutory favoritism). A strong and detailed RFP is unlikely to remedy tendencies of various parties to muddy the waters. It can, however, help in developing a contract that is difficult to reinterpret to the liking of any interest-conflicted parties. Need for Senior Level Management Pulse Taking. This is important for two reasons. First, senior management of SEAs and vendors may have a firm grasp of the strength and performance of their own teams, the health of the relationship between SEA staff and vendor staff, the financial health of the contract, or the operational health of the contract. Others may not for a variety of reasons. Second, operational management and frontline staff of SEAs and vendors may have a firm grasp of the current policy environment that senior management must navigate in their roles. Others may not for a variety of reasons. The model RFP outline contains several items intended to help SEAs and vendors avoid problems that can arise if staff at different levels have varying degree of understanding. These items include issue and risk tracking, periodic issue and risk reporting to senior management, appropriate SEA and vendor communications, appropriate SEA and vendor contact personnel, escalation protocols and contacts, regular operational management meetings, and periodic executive management meetings. Ability to Demonstrate Capacity. SEAs should guard against the risk that a vendor will bid products and services that are beyond its capacity to deliver (e.g. software/hardware solutions that exist in theory only.) With a well-written RFP and a highly detailed response, it is more difficult to disguise unsupported claims. However, a detailed RFP cannot eliminate this risk. SEA staff should remember the following potential remedies when developing an RFP and reviewing bids. In developing an RFP, it may be appropriate to include the following requirements: o Referrals to other clients for whom comparable work has been performed (in terms of substance, complexity, and reasonable volume) o Sample deliverables provided to another client (redacted appropriately)
Assessment RFP Development Toolkit Introduction to the Toolkit 4 o Temporary access to an online development/test site demonstrating the full capabilities of software and/or hardware products (with an offer to sign appropriate non-disclosure agreements). In evaluating a bid, it may be appropriate for the procurement office to require one or more bidders to improve the clarify of their bids (including incorporation of clarifications in any resulting contract). Such requirements might include the following: o Holding an in-person presentation or demonstration by the bidder with time for questions and answers. o Obtaining additional referrals if the bid evaluation team does not find bidders references comments sufficiently informative. o Obtaining additional sample deliverables if the bid evaluation team does not find those initial provided sufficiently informative. Toolkit Component: Instructions for Using the Model RFP Outline This component provides more detailed instructions on how to use the model RFP outline. It describes the types of items in the model outline that are included to help the RFP developers, and should be deleted from the final RFP. It describes the types of items in the model outline that are included to clarify the intent for potential bidders and should remain in the final RFP. It also describes special formatting that signals the intent of specific parts of the model RFP outline. Toolkit Component: Model Cost Proposal Worksheets The intent of the model cost proposal worksheets are to protect the SEA against uncertainty in its cost, to minimize protective costs vendors may include in bids as a hedge against uncertainty in their costs, and to provide a structure to minimize volatility that arises because of the least predictable part of any contract: contract changes. As with the model RFP outline, the model cost proposal worksheets are not intended to be used as is by SEA staff, but as a model that may be adapted to the SEA s program needs and procurement requirements. The model pricing sheet includes the following features developed to support the intent described above: Detailed line items to ensure that there is a shared understanding of the work to be completed and paid for. Clarification that a contract change request is required for new line items. Clarification that negotiation is needed between the SEA and the selected vendor to determine whether a contract change request is required for a line item whose nature has changed (regardless of volume). Specification of costs for goods, services, and deliverables at a highly detailed level. This serves to minimize SEA uncertainty in anticipating vendor invoices because only the described goods, services, and deliverables are valid items to include on vendor invoices. Detailed invoices minimize ambiguity that payments are made for appropriate goods, services, and deliverables. They also help vendors in providing invoices less likely to require multiple rounds of revision. Finally, detailed invoices also support SEA staff if an audit is conducted on expenditures of appropriated funds.
Assessment RFP Development Toolkit Introduction to the Toolkit 5 Separate fixed and variable (volume-based) costs. Separating these types of costs provides clarity to the SEA regarding where the vendors costs lie, and if a cost reduction is required, how the SEA can pinpoint potential cost savings to meet the cost reduction target. Likewise, if additional funds become available, separate fixed and variable costs can help SEAs identify the best value among several potential program improvements. An example of different types of costs is provided below: o A fixed cost (say, $150,000) for setting up an online testing platform may reflect that the platform must be set up each year no matter how manystudents take a test. o A variable (volume-based) cost (say $10,000) for setting up each unique test (e.g., grade and subject area) may reflect that the number of unique tests required each year may vary depending on legislative requirements. o A sliding variable (volume-based) cost may reflect that the number of unique tests administered becomes logarithmically less expensive with increases in volume, as in, the following: $50.000 per test for fewer than 100 tests administered $18.051 per test for fewer than 1,000 tests administered $ 1.354 per test for fewer than 10,000 tests administered $ 0.108 per test for fewer than 100,000 tests administered $ 0.009 per test for fewer than 1,000,000 tests administered $ 0.002 per test for 1,000,000 or more tests administered A mechanism for bidders to propose the described deliverable, alternative deliverables, and value-added options. Separate pricing for each year of the contract, including definitive pricing for awarded contract years and maximum pricing for optional extension years. The maximum pricing for extension years serves three purposes: (1) allowing bidders to account for uncertainty in the market and economy, (2) reducing SEA uncertainty regarding maximum cost exposure for extension years, and (3) assisting SEA staff in determining whether to extend a contract or issue a new RFP by allowing flexibility to negotiate extension year pricing with an understanding of the current market and economy. Elimination of volume-based contract change requests by including per-unit pricing applicable to any volume. A separate pricing schedule for contract changes. Because only materially modified or new line items require contract change requests, such changes cannot be anticipated in detail. To provide bidders and the SEA with a reasonable level of predictability in anticipating and pricing contract change requests, a different approach is taken: o Requiring bidders to specify a percentage markup to be billed in addition to the actual cost of goods, services, or deliverables the vendor must outsource. o Requiring bidders to specify an hourly or daily rate for services provided by internal vendor personnel that would fulfill the requirements of any contract change request (based on the types of internal vendor personnel described in the regular pricing schedule). Toolkit Component: Instructions for Using the Model Pricing Sheet This component provides more detailed instructions on how to use the model pricing sheet. It describes for the SEA how to use the model pricing sheet to develop the final pricing sheet for an
Assessment RFP Development Toolkit Introduction to the Toolkit 6 RFP, and how to use the pricing sheet in evaluating proposed costs by various bidders. It also describes for bidders how to use the pricing sheet in preparing a bid.