RFP Questions based on metrics Mini Guide [5]

Transcription

1 RFP Questions based on metrics Mini Guide [5] Background information for the Guideline for metrics in contracts Version Publication of Nesma

2 ISBN: Copyright Nesma 2015 All rights reserved. No part of this publication may be reproduced or made public in any way or in any form, without prior written consent by Nesma. After having been granted permission, the title page of the document that includes (parts of) this document must contain the following text: This work contains material from Metrics in contracts. Permission for publication has been granted by Nesma.

3 Table of Contents Table of Contents... 3 Preamble Introduction Purpose Size measures How to use the guideline Target audience Why to use this document Disclaimer Request for proposals (RFPs) What is meant with RFP Customer Issues Supplier Issues The effort/duration trade-off Duration versus Cost trade-off The impact of the effort/duration trade-off for RFP questions The effects of low, realistic and high estimates How to assess the reality value of proposals Example case QSM SLIM International Software Benchmarking Standards Group (ISBSG) Galorath SEER-SEM Differences between the tools Creating good RFP questions Another possibility: client oriented approach Other recommendations Conclusions References... 30

4 Preamble Contracting of software development projects and maintenance continues to be a difficult task for many organizations. They struggle to determine which questions they need to ask in the Request for Proposal (RFP) and contracting phases. These organizations wish to find the questions that would enable them to compare the bidding suppliers in an objective, yet meaningful way and they wish to select the right supplier based on this comparison. In practice, the industry sees many RFPs that seem to achieve this goal at first glance, but which offer a comparison that is not objective and meaningful at all. As a consequence, in many cases the wrong supplier is selected, which can (and often does) result in a failing project. Repeatedly, suppliers argue with client organizations about the objectivity of the tender and the reasons for missed offers and sometimes they even start legal action. Recently the urgency to improve the management of contracts and the execution of software projects again became evident in the conclusions of the Dutch Parliamentary Investigation ICT (the Elias committee) 1 : the Dutch government has insufficient control over the majority of their own IT projects. Many projects fail, while keeping a green light dashboard status until the end, even when the project gets cancelled. Nesma believes this Guideline for Metrics in Contracts, together with the new "Base Of Estimate" (BOE) 2 for software services (published as a recommended practice by the American Association of Cost Engineers (AACE)) will improve this situation significantly. In order to facilitate the IT industry to improve its software project success rate, Nesma started an initiative to develop a guideline for the use of function points and quality metrics in contracts/contracting. This publication zooms in on assessing suppliers performance in order to provide a knowledge base for readers who wish to start using the guideline. This guideline is the result of the work of the following persons: Harold van Heeringen, Sogeti Nederland B.V. (chair) Hans Kuijpers, Software Improvement Group (SIG) Rini Scholten, Kadaster Frans Schoot Uiterkamp, Rendeck automatisering Jolijn Onvlee, Onvlee Opleidingen & Advies Hans Bernink, Jim Bernink B.V. Marcel Pereboom, Mediaan The BOE for software service can be downloaded for free from the Nesma website: Mini Guide RFP questions based on metrics version 1.0 Page 4

5 1 Introduction In today s world, most medium and large companies in the world are involved in some kind of outsourcing. Many organizations believe that outsourcing is the perfect solution for managing a part of the (usually non-primary ) functions in the organization in an efficient and effective way. This may be true for certain tasks that are relatively easy to understand, like for instance catering or security of an organization. In these cases, knowledge transfer from the outsourcing company to the service supplier is relatively easy and the characteristics of the service to be delivered and its price are easy to agree on. IT development projects however are usually very complex. Knowledge transfer from the outsourcing company to the supplier is often a difficult task. Also, software functionality is more and more the driving factor for the success of organizations. Carrying out software development in the most efficient and effective way may result in a direct competitive advantage and therefore it is extremely important for organizations to make sure they select the right supplier when they outsource (part of) software development activities. Outsourcing companies often try to select the right outsourcing partners by trying to compare the different suppliers based on quantitative data. Although this idea is very good in theory, unfortunately the way it is done in practice is often quite the opposite. Actually, by focusing too much on getting the lowest price the opposite happens. As a consequence the least suitable supplier is chosen. Project overruns and other disasters are more the rule than the exception. The purpose of this guideline is to show that modern RFP management should take into account the laws and best practices from software metrics literature. Selecting the best supplier based on the comparison of objective comparable data can be the start of a successful relationship between the two organizations. However, it is very easy to select the wrong supplier if one does not know the right questions to ask. After reading this guideline, it should be clear which questions the outsourcing company should ask, which questions they should avoid, and other criteria that are important for selecting the right supplier. Mini Guide RFP questions based on metrics version 1.0 Page 5

6 1.1 Purpose This document describes the way Nesma believes that function point based metrics should be applied in RFPs regarding software realization projects. This applies to both new development projects and functional releases. In scope are both RFPs to select a single supplier for a one-time engagement and RFPs to select a supplier for a long term engagement. The scope therefore includes the following contract types: Software development projects, including new development, enhancements and (major) adaptive maintenance; Framework agreements. The scope is not limited to certain technologies or methods of implementation. Java,.Net, Oracle, and agile, waterfall, package implementation, websites, mobile apps, data warehouses, etcetera are all in scope. 1.2 Size measures The scope of this document is limited to ISO-certified function point based metrics (like Nesma [1], IFPUG [2], COSMIC [3] and FiSMA [4]). These methods comply to the ISO standard for Functional Size Measurement Methods [5] (i.e. ISO/IEC 14143) and therefore these methods are objective, repeatable, verifiable and defensible. These are essential criteria in contracting, because it reduces/prevents discussions regarding the size of the project or the application delivered. This means that size measures like source lines of code (slocs), IBRA points, SNAP points, Configuration points, Story points, Usecase points, nr. of pages of documentation, nr. of modules, number of RICEF objects, etcetera, are outside the scope of this document. Although these size measures may help organizations in some ways to estimate projects, or for other purposes, Nesma does not recommend to use these measures in contracts (or contracting) because these metrics are subjective (depending on the person doing the measurement), not repeatable, not verifiable and ultimately not defensible. It is important to understand that the ISO-certified functional size measurement methods only measure the size of the functional user requirements. Parts of the contract that are not related to the functional size can be measured with different methods. Mini Guide RFP questions based on metrics version 1.0 Page 6

7 1.3 How to use the guideline This guide is part of a set of mini guides composing a guideline for metrics in contracts. There is one document describing all of the mini guides named Guideline for metrics in contracts. It is recommended to read this overall guideline first. The division into different Mini Guides is intended to support different types of audiences. The reader can select the Mini Guide that is applicable for his or her interests and goals. It makes the total guideline easy to read. All of the mini guides are described shortly in the [1] Guideline for metrics in contracts. The documents that together make up the total guideline are: [1] Guideline for Metrics in contracts; [2] Mini Guide for Development Methodologies; [3] Mini Guide for Maintenance; [4] Mini Guide for Management; [5] Mini Guide for RFP questions; [6] Mini Guide for Functional Quality; [7] Mini Guide for Pricing Mechanisms; [8] Mini Guide for Technical Quality; [9] Mini Guide for Assessing Suppliers Performance (this document); [10] Mini Guide for Software Metrics based on FPA; [11] Mini Guide: Requirements for Supplier organizations; [12] Mini Guide: Requirements for Customer organizations. All of the mini guides are described shortly in the Guideline for metrics in contracts. Based on your role and/or interest, you may select all of the mini guides to read or just a selection of the mini guides that you feel are most interesting for you. Please check the website for the mini guides that have been published and the possibilities to obtain them. Mini Guide RFP questions based on metrics version 1.0 Page 7

8 1.4 Target audience This document is targeted to all people involved in contracting software development and maintenance in both client organizations and supplier organizations (not limited): Procurement; Bid management; Vendor management; Supplier management; Demand management; Delivery management; Account management; ICT management; Software Cost Engineers; Business management; Project management. 1.5 Why to use this document There are a number of reasons why you may be interested to read this document. A number of common reasons, such as (not limited): You are going to issue a Request for Proposal (RFP) in order to select a supplier to deliver a software product for you and you wish to be able to compare the bids from the suppliers in an objective way in order to select the right supplier; You wish to select a realistic offer instead of a cheap offer, because you realize that cheap offers often result in large overruns and in the end might turn out to be more expensive; You have to respond to an RFP and you wish to answer the metrics based questions in the way the client requires; You wish to prepare your metrics department in order to be ready to answer future RFPs in which the client organization uses this guideline; You want to know which subjects and details should be included in an RFP or contract to avoid disagreements afterwards. 1.6 Disclaimer A lot of care and effort has been put into this document and although Nesma believes that this document is of great value to the industry, Nesma does not accept any responsibility for results that occur after the use or misuse of this document. Nesma welcomes and appreciates all comments and additions for further improvement. These can be sent directly to Nesma (office@nesma.org). Mini Guide RFP questions based on metrics version 1.0 Page 8

9 2 Request for proposals (RFPs) 2.1 What is meant with RFP A Request for Proposal (RFP) is a solicitation made, often through a bidding process, by an agency or company interested in procurement of a commodity, service or valuable asset, to potential suppliers to submit business proposals.[6] In many cases, RFPs are also submitted to select outsourcing partners (or preferred suppliers ) for a period of time. During this period of time, the supplier may or may not do any work agreed upon in the contract. In this guideline the focus is on RFP management for a single software development project, but the same principles apply for framework agreements as well. 2.2 Customer Issues In general, the company that issues the RFP has to provide all the necessary information to the potential suppliers for them to be able to write a sound proposal. In general, the following information has to be submitted for an RFP on a specific project: Client corporate information; The bidding process like deadline for the definitive proposal, but also possible scheduled sessions for asking and answering questions; The functional requirements that have to be delivered in the system to be delivered; The non-functional requirements that have to be satisfied, like for instance security requirements and development language to be used; Decision criteria that the client organization is going to apply to select the most appropriate proposal. It is very important that the provided information is up-to-date and detailed. Especially in the case of fixed-price bids the proposal offered by the different suppliers usually also have a legal status. This means that when the client selects a specific proposal, the supplier is obligated to deliver the proposal against the price stated in the proposal. Needless to say that more detail in the description of the functional and the nonfunctional requirements lead to better proposals (as the supplier can reduce its risk percentage due to unforeseen requirement creep) and therefore to realistic prices. For the party that submits the RFP it is crucial to select the right outsourcing party, and to do so in a legally correct way. Of course the customer organization should provide general information like: Mini Guide RFP questions based on metrics version 1.0 Page 9

10 Schedule of the bid process. Are there going to be any information sessions for suppliers to ask questions? What date is the submission deadline for the quotation? When will the decision be communicated? What date should the project start? Organizational information. Who are the responsible persons in the client organization and how is the organization organized? General requirements that a supplier has to meet in order to be allowed to be in the bid process. An example could be a requirement that the organization must hold a CMMi level 3 certification. Solution details. Are there any limitations to the solutions that the suppliers have to take into account, like architecture or programming language? When preparing an RFP, the client organization also has to think of the criteria on which it will judge and select the most appropriate outsourcing party. This usually means that they will have to think of the most important characteristics of the project itself and of the party that will realize it. Characteristics that are usually considered are (not limited): Price; Quality; Productivity; Duration; Supplier creditability; Supplier references; Solution details. To be able to compare the different supplier quotations in an objective way, the questions are usually as quantitative as possible. Typical metrics based questions that are often seen in RFPs are: Price What is the price per function point that you offer for the realization (technical design, functional design, coding, unit testing, system testing) of this system in Java? What is the price per function point that you offer for the realization of change requests during the projects? What is the price per function point that you offer for the maintenance of the system after implementation? Mini Guide RFP questions based on metrics version 1.0 Page 10

11 Quality What is the number of defects per function point that is expected to be detected during systems testing? What is the number of defects per function point that is expected to be detected during user acceptance testing? What is the number of defects per function point that is expected to be detected during the first three months in production? What is the number of defects per function point that is expected to be detected per year after the first three months in production? Productivity What is your productivity expressed in hours per function point in the realization (technical design, functional design, coding, unit testing, system testing) of Java projects? These questions seem good questions that offer the possibility to select the best supplier offer. However in fact they are impossible to answer if one is familiar with current models from software metrics literature. 2.3 Supplier Issues In order to win the contract, supplier organizations compete against each other to score the highest on the client s decision criteria. Many of the typical questions asked in RFPs are related to metrics expressed in function points. For this reason supplier organizations should have an experience database with their project data sized in Nesma, IFPUG and/or COSMIC function points. Without this database, it is quite difficult to answer the questions above, and it is even impossible to objectively defend the answers given. Depending on the decision criteria included in the RFP, the commercial people of the supplier will try to bend the bid in a way that they think suits the decision criteria best. However, it is important to understand that according to McConnell [7] there is a distinction between target, estimation and commitment. The supplier should be very careful first to estimate the project very thoroughly. This should be done before handing over the results to the commercial organization that is going to translate the estimate into a quotation. Estimating and pricing are really two different disciplines, but of course they are related. In the next chapter, one of the most important software metrics laws is explained: the effort/duration trade-off. Mini Guide RFP questions based on metrics version 1.0 Page 11

12 3 The effort/duration trade-off In this chapter, one of the most important metrics laws regarding using functional software metrics in RFPs is discussed: the Effort/duration trade-off. 3.1 Duration versus Cost trade-off This is one of the most important and relevant software metrics laws and it is published in numerous books over the years. It indicates that the chosen duration of a project is a very important variable that drives the effort needed and therefore metrics like productivity (hours/fp) and cost/fp (figure 1) [8]. Figure 1: Duration vs. Effort/costs trade-off In this figure, it shows that for every software development project of a given size, it is possible to make a different estimate with regards to cost and effort, depending on the duration chosen. There is a whole range of valid scenarios from which one can choose. There is also an impossible area in which the project simply cannot be done. Also there is a duration area in which the estimation is not very practical. The reason: when a project takes too long, the benefits of the project will be less. Also when the team size is getting too low, not all necessary skills will be present in the team to carry out the work effectively and efficiently. Effort and cost will therefore be higher after the optimal duration. The black line that indicates the duration vs. effort/cost trade-off represents the productivity of a specific organization. Mini Guide RFP questions based on metrics version 1.0 Page 12

13 When we look at the realistic zone we see that "faster is more expensive". This law is based on the fact that to be able to deliver a project in a shorter duration than with an optimal team size, one has to increase the team size to be able to develop faster. However, as for instance the International Software Benchmarking Standards Group (ISBSG) [9] indicates, the optimal team size of a given project is about 4-6 persons. Any extra person on the team will reduce the productivity, as more communication paths arise, project management and planning will become more difficult, dependencies will increase and the number of defects will increase which in turn will increase the effort for testing and debugging. Even worse: this will increase the time for testing, so even more staff is needed on the team to make the duration indeed lower. Eventually this escalates into the impossible zone. Looking at figure 1, the following observations can be made: There is an impossible zone in which the project cannot be completed, as the staff needed to realize the shorter duration rises to infinity; The first possible duration indicates the minimal duration/maximum cost scenario; There are numerous estimations possible on the duration vs. effort/cost trade-off, each resulting in a different effort/cost estimation and therefore also in different productivity (hours per function point) and cost per function point metrics; There is an optimal effort/cost estimation, although it is hard to calculate where; There is a duration zone in which it is impractical to realize the software, but it is still possible. The cost may even go up in this zone as the team size gets so low that for instance one person is working part-time on the project and constantly has to switch between tasks. However this zone will give desirable scenarios in case the number of available staff is a limiting constraint on the project. 3.2 The impact of the effort/duration trade-off for RFP questions This paragraph uses an example to show the impact of this trade-off. Consider one of the before mentioned typical RFP questions that customer organizations often ask: What is the price per function point that you offer for the realization (technical design, functional design, coding, unit testing, system testing) of this system in Java? A professional supplier (let s call it Supplier A) would first size the functional requirements submitted by the customer organization. Let s assume the size is exactly FP. After sizing, a mature supplier would use its historical data of Java projects in combination with professional estimation tools to come up with figure 2. (Please note that there is no real data here, it is only for instructional purposes). Mini Guide RFP questions based on metrics version 1.0 Page 13

14 Figure 2: Duration vs. Effort/costs trade-off in a specific bid So, what would supplier A answer to this question? In fact, the question is unanswerable if the duration that the client has in mind is not known. If the time-to-market is only 6 months, the answer is /FP. If a duration of 12 months is OK for the customer, the optimal duration/costs trade-off can be offered, which is 500 /FP in this example. The people that are involved in estimating the project, will report this to the commercial people. The estimate involves a range of 1000 /FP in 6 months to 500 /FP in 12 months. The commercial people of supplier A will probably decide to quote the 500 /FP in their answer, in order to score points against the decision criteria scheme of the client RFP question. However, one should be aware of the fact that after the project is won, the project has to be carried out against this price. Only then, the negotiations over the duration for the project may start. It then turned out that the client required a 6 month duration, and the price was agreed on 1000 /FP. Of course, this is not a very good way to start the project and probably there will already be problems in the relationship during the negotiations. The reason for this: the RFP question was just not specific enough and lacked the duration prerequisite. The only solution for supplier A is to make sure the duration will be known before submitting the quote. Mini Guide RFP questions based on metrics version 1.0 Page 14

15 Another supplier (supplier B) might have guessed that the customer time-to-market is 6 months and quoted 800 EUR/FP. Although this supplier did not win the project, this is a better quote for the customer than the quote of the supplier that won the contract and after that renegotiated the effort (price)/duration trade-off. If the customer would have made the question more specific, adding the fact that his duration is 6 months, he would have received a quote of supplier A of 1000 EUR/FP and a quote from supplier B of 800 EUR/FP and he would probable choose supplier B. Yet another supplier ( unprofessional Supplier C) may not have formal estimation techniques in place and may not use historical data and professional estimation tools. This supplier will probably rely on expert estimates, probably resulting in a low quote, for example 300 EUR/FP. If the customer does not have a mechanism to exclude overoptimistic bids from the competition, this supplier may actually win the bid, but probably the problems start already then! The next chapter explains why. Mini Guide RFP questions based on metrics version 1.0 Page 15

16 4 The effects of low, realistic and high estimates Another software metrics law, described in McConnell [7], indicates that the project results are very much dependent on the estimation and planning of the project. The relationship is displayed in figure 3. Figure 3: Project Estimates and Realization The figure points out that in case of an overly optimistic estimation of the project, i.e. at a given duration of the project too few hours have been estimated, the realized extra costs of the project will be higher in a non-linear way. A number of known causes for this relationship are: Planning errors (too small team size, critical path errors, etc.); Too little time spent on requirements and design and so injecting more defects; Once the project is late, ineffective techniques are applied to get it back on track, like adding new people, making it only more expensive, not faster; More status meetings, extra management attention; Project stress results in more defects, each of which has to be logged, reported, analyze, corrected and tested again. Often corrected defects inject new defects. Project stress is real bad for the project result. When the project is estimated in a pessimistic way however, the extra costs of the realized project will rise in a linear way. This is due to Parkinson s law [10] that states that work expands to fill the available time. A second reason is student s syndrome [11] which states that when a project team gets too much time to do a task, they will wait Mini Guide RFP questions based on metrics version 1.0 Page 16

17 until it is the last possible moment to start with the task and then work really hard to complete it in time. The implications of these laws are evident. When a client organizations sends out an RFP and receives the different quotations of the different suppliers, it is crucial to be able to judge whether the proposals are realistic! Let s look at an example of how things go in day-to-day practice. Example An organization submits an RFP for a specific project and receives three quotations, A, B and C. The Estimations are shown in the next table figure. Figure 4: Project Estimates and Realization Example An organization that is not able to recognize that the quotation submitted by Supplier A is unrealistic may (and probably will) go for this one because it promises to be cheaper and faster. Who would not want that, right? The result will be disaster! The question now is of course: How can organizations assess whether a quotation is realistic or not? Mini Guide RFP questions based on metrics version 1.0 Page 17

18 5 How to assess the reality value of proposals There are multiple ways to assess the reality value of a proposal. Typically, you need a professional tool and/or a database with relevant historical data. In this chapter three possible ways of assessing the reality value of proposals are given using different tools: 1. QSM SLIM tool suite [12] 2. ISBSG repository New developments & enhancements [13] 3. Galorath SEER-SEM tooling [14] Other professional estimation tools or databases with relevant industry data can of course be used as well. These three are chosen because they are currently used the most in the industry. In this chapter, examples are given of how to carry out a reality check using the tools mentioned above. 5.1 Example case Consider an RFP that is send out by Organization X, containing the following RFP question: What is your productivity rate (hours/fp) for a moderately complex Java project of 500 function points and a duration of 30 weeks? Phases to include are technical design, coding, unit testing, systems testing and support of the user organization during the user acceptance test. Organization X receives three different proposals, displayed in the next table. Table 1: answers to RFP Proposal Size (FP) Duration (months) Effort (hours) Productivity (h/fp) , , , , , ,2 The size and duration were given, and also the activities in scope and expected complexity. The three answers should be very well comparable. Now it is important to exclude the answers that are unrealistically optimistic. Mini Guide RFP questions based on metrics version 1.0 Page 18

19 5.2 QSM SLIM QSM SLIM is one of the most widely used tool suites in the world when it comes to software cost estimation. When an organization has licensed the QSM SLIM tool suite, it is possible to simulate the estimations in SLIM Estimate or in SLIM Datamanager. This results in a Productivity Index (PI) that is implied in the quotation. The productivity index in QSM shows the productivity which is corrected with the duration. A higher PI means a higher productivity (lower hours/fp) at a given size and duration. The black line in figure 2 could for instance indicate the PI=15,0 line. Although the number of hours per function point is different on every point on this line, the PI could easily be the same. When the PI that is implied in the Estimate is known, it is possible to assess the reality value of the proposal. In QSM SLIM it is possible to compare the PI of an Estimate with the average PI for similar projects that are present in the QSM database (over projects) or with the PI of the projects that are stored in the organization project database. When an implied PI is much higher than the average PI for similar projects in the QSM database, the estimate is probably not very realistic. This implies that the productivity assumed in the quotation is much higher than average, so they think they need less hours/fp. Therefore the estimate may be considered too optimistic. In that case, the organization should ask the supplier to present proof that they are able to produce software with a productivity that is that high. In the next figure, the three proposals are simulated in QSM SLIM Datamanager. The PI of the three proposals is calculated by the tool, based on size, effort and duration. Figure 5: Simulation in QSM SLIM PI calculation Now it becomes possible to compare the PI with for instance the PI that is reported by QSM based on their Data set of Business Projects measured in function points. This analysis can be made in QSM SLIM Metrics. This shows the following: Mini Guide RFP questions based on metrics version 1.0 Page 19

20 Figure 6: QSM proposal assessment It seems that according to this tool, proposal 3 is the most realistic as it is closer to the PI that may be considered market average (the black line). Even so, the proposal still has a much better PI than the market average PI of about 15,2 at that size. When the suppliers of proposal 1 and 2 do not have a good explanation and no proof of the fact that they can deliver software with a productivity that much higher than market average, it would be advisable to choose proposal International Software Benchmarking Standards Group (ISBSG) A second way to assess the proposal is to use the ISBSG industry data, either through the online portal [14] or by licensing the data (in Microsoft Excel) [13]. This assessment works basically the same, but the main metric that can be compared here is the productivity in hours per function point. When this metric is used standalone, we have already seen in paragraph 5 that the usefulness is very low. However, when also the actual duration and size (and quality) of the project are taken into account when selecting the projects against which the supplier quotations are compared, the usefulness will be much higher. ISBSG also offers a specific tool called the ISBSG Reality Checker [16]. Although input options are at the moment quite restricted (size, platform, language type), ISBSG is in the process of updating the tool to accept more input criteria. Mini Guide RFP questions based on metrics version 1.0 Page 20

21 The reality check will find the ranges of effort spent and project duration elapsed in which the project estimations must fall to be assessed realistic. Using the ISBSG data portal or the ISBSG repository New Developments and Enhancements, first an appropriate data set has to be selected. Usually it is advisable only to select projects with data quality A or B. These projects submitted data that the ISBSG repository manager assessed as very high (A) or high (B). Furthermore, in this case it is a good idea to select only the projects measured in Nesma or IFPUG function points. The primary programming language should be Java and the size of the project should be not too far from 500 function points. The result of this search gives the following information: Table 2: ISBSG assessment Metric Percentile 25 Median Percentile 75 PDR (hours/fp) 6,3 9,1 16,3 Duration (months) 5,2 7,3 11,3 The assessment of the PDR (hours/fp) is made visual in the next figure. Figure 7: ISBSG proposal assessment In this case, it shows that the PDR of proposal 3 is actually quite worse than market average (median PDR) and even worse than the Percentile 75 of the selected data set. This means that over 75% of the selected projects were carried out against a better PDR than proposal 3. A quite conservative bid one might say. Mini Guide RFP questions based on metrics version 1.0 Page 21

22 Proposal 1 is clearly very optimistic, even better than the P25. In general, ISBSG data is considered to be skewed towards best-in-class data, as only mature companies are probably able to submit data to ISBSG and also willing to do so. This proposal should therefore be excluded if the supplier has no additional proof and historical data of comparable projects that show it is indeed much more productive than industry average. In this assessment example, based on ISBSG data the best offer could be proposal 2. However, also this proposal is better than market average, therefore also this supplier should offer proof that he is as productive as claimed. Based on this assessment, a choice should be made between proposal 2 and 3. Proposal 3 is relatively expensive (when hour rates are comparable), but the advantage is that the chance of the effect described before of projects starting with low estimates are avoided. With proposal 2, there is still a considerable risk of this happening, and the result may be a considerable more expensive project than proposed by proposal Galorath SEER-SEM A third way is to use the Galorath SEER-SEM [14] tool to estimate the project. The tool allows to use predefined knowledge bases from historical data collected by Galorath. The Galorath SEER-SEM tool provides the possibility to calculate an optimal effort scenario and a minimum time scenario. Especially the minimum time scenario is useful for assessing the reality value of a proposal. There are many parameters that can be set in the SEER-SEM tooling, but many of these parameters are preset when the appropriate knowledge bases are selected. The knowledge bases contain the parameter settings that have been found to be consistent in particular type of projects. After selecting the knowledge bases, only the size has to be entered to come up with a rough estimate. After that, the parameter settings can be altered by even more accurate settings and scenario analysis can start. The more is known about a project, the better the parameters can be set. Simulation using the SEER- SEM tooling results in the following realistic range: Mini Guide RFP questions based on metrics version 1.0 Page 22

23 Table 3: SEER-SEM assessment Scenario Duration (months) Effort (hours) PDR (hours/fp) Minimum time 5, ,6 Duration = 6,8 months 6, ,4 Optimal Effort 10, ,8 This simulation shows us that for this particular project, the answer to the question stated in the beginning of this section should be around hours / 500 FP = 9,4 hours/fp. Both proposals 1 and 2 seem too optimistic and therefore not realistic. If the suppliers can support their proposal with valid history data that they are able to produce projects faster and cheaper than the market, they still might be a good option to select. If they do not have such proof, the best thing to do is to select proposal Differences between the tools The different tools are developed based on different data and different formulas. This means that it is possible that one tool shows one proposal too be too optimistic, while the other tool comes up with a different conclusion. To get a better understanding of the reality value of a proposal, it is recommended to use at least two tools. Mini Guide RFP questions based on metrics version 1.0 Page 23

24 6 Creating good RFP questions Now that the problems are clear that supplier organizations face when trying to answer an RFP, and the implications are clear this has on the ability of client organizations to choose the right supplier, it is important to give some recommendations of RFP questions that should be used. Returning to the often-asked questions mentioned before: What is your productivity rate for Java projects The main recommendation is very evident. Make the question as specific as possible. A better question would already be: What is your productivity rate (hours/fp) for a moderately complex Java project of 500 function points and a duration of 20 weeks? However, this question still lacks a lot of context. it is not clear which activities should be included for instance. Will the supplier be in charge of the full lifecycle, or perhaps only technical design, coding and testing? it is crucial to supply this information! A much better question would therefore be: What is your productivity rate (hours/fp) for a moderately complex Java project of 500 function points and a duration of 20 weeks? Phases to include are technical design, coding, unit testing, systems testing and support of the user organization during the user acceptance test. This last question is easily answerable for suppliers that have a history base with experience project data. For the client it is then quite easy to compare the different supplier quotations and also to assess their reality to market averages or history data. Mini Guide RFP questions based on metrics version 1.0 Page 24

25 To conclude, good RFP questions contain the following information: Metric to compare between competitors, for instance o Productivity (hours/fp, Function points/hour, Productivity Index) o Costs (Cost/FP) o Quality (defects per function point, Mean-time-to-defect (MTTD)) Technology (for instance Java, Oracle or MS.NET); Size (in ISO standard, like Nesma, IFPUG or COSMIC Function Points); Technical/ Functional Complexity (for instance high/mediate/low); Phases/Activities to include (for instance Technical Design, Coding, Unit testing, systems testing); Development type (Agile, spiral, waterfall); Development location(s) (in house, at supplier site); Business area type (Government, Banking, Logistics, etcetera); Application type (Mobile app, Financial transactions, Data warehouse). Duration requested (days, weeks, months, years). It may therefore be recommendable to ask the questions in the format of (small) case studies. Even when the questions are good, and a meaningful comparison is very well possible, it is still very important to select a realistic proposal. Here it becomes difficult, because many people, and organizations, are used to select the cheapest one, expecting the goods or service to be delivered of equal quality. Well, in the software business it is not like that! Mini Guide RFP questions based on metrics version 1.0 Page 25

26 7 Another possibility: client oriented approach In some countries, client organizations are very reluctant to accept any metrics or data that is submitted by suppliers and they wish to have full control of the process. For instance in Spain, it is customary for client organizations to carry out a should costing activity. The client organization measures the functional size of a project and estimates the effort, duration, productivity and cost of the project using own historical data, external benchmark data and/or professional software estimation tools or models. The proposals of the suppliers should meet this estimate in order to be taken into consideration. In these type of bids, the supplier has to show the customer the value his proposal will deliver to the client and the client chooses the proposal that he feels gives him the best value for money. This is a completely different approach than the one described in the previous chapters, but apparently this can work in some cases. However, it may be hard to objectively decide which offer brings the best value for money. Should Costing can go as far as published by Airbus [19]. The Costing department of Airbus applies Should-costing for all parts of their airplanes. In collaboration with the suppliers it is decided that the costs estimated by Airbus is realistic and the supplier should commit to these costs plus a certain profit margin. Unless the supplier can actually prove that there is an error in the estimation model or the assumptions made, they have to comply to this. Mini Guide RFP questions based on metrics version 1.0 Page 26

27 8 Other recommendations This chapter gives a number of recommendations that you have to keep in mind when working with these types of questions. They are structured in Do and Don t recommendations. DO DON T Make sure a well defined and controlled change procedure is in place. This should result in a fair price that is paid for a change during the project, not the premium price because the supplier is already at a loss. Don t think that changes won t happen and don t think that the supplier will do them free of charge or against a low fee if no clear change procedure is agreed upon. Reward reality value more than lowest price. Use one of the assessment methods discussed to discard overoptimistic ones. Don t think that a very low offer will automatically be the best offer. Probably you will pay more in the end and nobody is going to be happy: another project failure. In case of an RFP for a specific project, carry out the functional size measurement yourself by a certified analyst and allow suppliers to review the functional size measurement report. This way, everybody is using the same starting point. Also possible is to have the suppliers do their own functional measurement instead of reviewing. However, they would need to submit the measurement report for the customer to judge the quality. Suppliers that measure the wrong size or are not able to produce a report that can be reviewed, should be rejected. Don t send out the requirements and have every supplier do their own sizing without reviewing their size reports. Some suppliers may not have certified analysts and will use a completely wrong size in their estimation, resulting in non-comparable offers. Use ISO standard for functional sizing, preferably Nesma, IFPUG or COSMIC function points. This makes sure the size is measured in an objective, repeatable, verifiable and defensible way. Don t think that size measures like source lines of code, FFPA, story points, IBRA points or any other measures than ISO measures can be used in contracting. These measures lack objectivity and can easily lead to huge problems in contract situations. Only use certified analysts to measure functional size. Have a review procedure in place to review the functional sizing. Don t expect people who are not certified, or people that are certified but do functional sizing only a few times a year, to come up with a good functional size measurement. Agile projects need to be treated in a completely different way than traditional (waterfall) projects. Don t use the same pricing framework for both agile and non-agile development projects. The Mini Guide RFP questions based on metrics version 1.0 Page 27

28 Remember that in agile projects the scope is variable, the duration and cost in principle not. dynamics are quite different, making it fairly impossible to use 1 framework for both types of development methods. Please read the Nesma publication on the differences between traditional development projects and agile development projects (To be released) Keep in mind that there are different variants of the same functional size measurement, for instance the derived methods of the Nesma method are Indicative FPA and Estimated FPA. For more information about this, check for instance this white paper. [17]. Always specifically state which method and variant is used with the version. Don t think that Nesma function points and Nesma onderhoudsfunctiepunten are the same. They are not! Ask for proof of the productivity a supplier claims that he can realize. Ask for the (anonymous) data of a number of relevant projects of the supplier. If he can t give this data, you know he is probably not mature enough to collect, measure and analyze completed projects. Check for instance paper 18] on how this proof may be obtained. Don t believe suppliers on their blue eyes when it comes to their claims. Always ask for proof. Specify the activities that are considered to be part of the Hours/FP or Price/FP metrics. Also specify the activities that are not part of these metrics. Additionally, specify what should happen when activities have to be done that are not described to be either in or out scope. Don t assume that you have specified all the possible project activities. There are always more! Allow the supplier to realize a fair profit. They do have an organization to sustain and may have shareholders who may expect a fair return on their investments. Do not try to get a too cheap deal at the expenses of the supplier. Beware of procurement staff with this mentality. If a supplier finds any ambiguity or unclearness in the RFP questions, always ask the client to explain. The client will send the answers to all suppliers involved in the RFP. Suppliers should not make assumptions on the rfp questions without verification by the client. Clients should not make assumptions on the proposals without verification by the supplier. Mini Guide RFP questions based on metrics version 1.0 Page 28

29 9 Conclusions There is a fundamental problem going on nowadays in the IT outsourcing industry. In many cases, unrealistically optimistic proposals are granted by the outsourcing organizations, resulting in failing projects. If the IT industry is not able to evolve into a higher degree of maturity, this practice will not be likely to improve in the near future. Only when both buyers and suppliers are going to speak the same language, proposals will at least be comparable and therefore assessable. The first thing outsourcing organizations should do is therefore to give as much information about the project as possible (including a detailed functional design) and to ask the metrics based question in such a way that it will be possible to compare the proposals. And also in such a way that it is possible to assess the reality value of the proposal with the use of industry tools or industry data. Unrealistically optimistic proposals that are not backed up by proof should not be chosen! Nesma believes that the methods described in this mini guide are crucial for organizations to select realistic offers instead of cheap offers. Focus on reality value will result in a higher percentage of successful projects, better software (fewer defects, higher maintainability), fewer problems between the client and the supplier(s) and happier people. This benefits everybody and also saves a lot of money! Mini Guide RFP questions based on metrics version 1.0 Page 29

30 10 References [1] Nesma, Netherlands Software Measurement user Association. Definitions and counting guidelines for the application of function point analysis, a practical manual. Version 2.1, 2004, (ISO/IEC 24570) [2] IFPUG, International Function Point User Group. Function Point Counting Practises Manual (ISO/IEC 20926) [3] ISO/IEC19761:2003, Software Engineering COSMIC, A Functional Size Measurement Method version 2.1, International Organization for Standardization ISO, Geneva, 2003, ) [4] FiSMA, (ISO/IEC 29881) [5] ISO standard for Functional Size Measurement Methods, ISO/IEC 14143, [6] Wikipedia definition RFP: [7] McConnell, Software Estimation: Demystifying the Black Art (Microsoft, 2006) [8] van Heeringen, H.S. - Successful software project outsourcing - How to limit the risk of failures, Proceedings of the Software Measurement European Forum (SMEF), Rome (Italy), June [9] Practical Project Estimation 3, Peter Hill, ISBSG, [10] Parkinson s law: [11] Student s syndrome: [12] Quantitative Software Management, Software Lifecycle Management tool suite, [13] ISBSG repository New Developments and Enhancements, [14] Galorath SEER-SEM, [15] ISBSG data portal: [16] ISBSG reality checker: Mini Guide RFP questions based on metrics version 1.0 Page 30