Using Economic Considerations to Choose Among Architecture Design Alternatives

Transcription

1 Usng Economc Consderatons to Choose Among Archtecture Desgn Alternatves Jayatrtha Asund Rck Kazman Mark Klen December 2001 TECHNICAL REPORT CMU/SEI-2001-TR-035 ESC-TR

2

3 Pttsburgh, PA Usng Economc Consderatons to Choose Among Archtecture Desgn Alternatves CMU/SEI-2001-TR-035 ESC-TR Jayatrtha Asund Rck Kazman Mark Klen December 2001 Product Lne Systems Unlmted dstrbuton subect to the copyrght.

4 Ths report was prepared for the SEI Jont Program Offce HQ ESC/DIB 5 Egln Street Hanscom AFB, MA The deas and fndngs n ths report should not be construed as an offcal DoD poston. It s publshed n the nterest of scentfc and techncal nformaton exchange. FOR THE COMMANDER Norton L. Compton, Lt Col, USAF SEI Jont Program Offce Ths work s sponsored by the U.S. Department of Defense. The Software Engneerng Insttute s a federally funded research and development center sponsored by the U.S. Department of Defense. Copyrght 2001 by Carnege Mellon Unversty. NO WARRANTY THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. Use of any trademarks n ths report s not ntended n any way to nfrnge on the rghts of the trademark holder. Internal use. Permsson to reproduce ths document and to prepare dervatve works from ths document for nternal use s granted, provded the copyrght and "No Warranty" statements are ncluded wth all reproductons and dervatve works. External use. Requests for permsson to reproduce ths document or prepare dervatve works of ths document for external and commercal use should be addressed to the SEI Lcensng Agent. Ths work was created n the performance of Federal Government Contract Number F C-0003 wth Carnege Mellon Unversty for the operaton of the Software Engneerng Insttute, a federally funded research and development center. The Government of the Unted States has a royalty-free government-purpose lcense to use, duplcate, or dsclose the work, n whole or n part and n any manner, and to have or permt others to do so, for government purposes pursuant to the copyrght lcense under the clause at For nformaton about purchasng paper copes of SEI reports, please vst the publcatons porton of our Web ste ( prnted 1/28/2002 3:16 PM verson number / rl

5 Table of Contents Abstract v 1 Introducton and Motvaton 1 2 Decson-Makng Context 3 3 The Steps of the CBAM The Trage Phase The Detaled Examnaton Phase 6 4 Theoretcal Bass of the CBAM and Detaled Descrpton Step 1: Choose Scenaros and Archtectural Strateges (ASs) Step 2: Assess the Relatve Importance of QAs (Elct QAScore ) Step 3: Quantfy the Benefts of ASs (Elct ContrbScore, ) Addressng Varablty n the Contrbuton Score Calculatng the Beneft Scores of the ASs Step 4: Quantfy the Costs of the AS and Incorporate Schedule Implcatons Step 5: Calculate Return for Each AS Step 6: Rank Order the ASs and Apply an Approprate Decson-Rule A Decson-Rule Based on Probablty Dealng wth Combnatons of Strateges Usng the Portfolo Theory Framework 14 CMU/SEI-2001-TR-035

6 5 Case Study Descrpton of the Proect Applyng the CBAM to the ECS Step 1: Choosng the Scenaros and Archtectural Strateges (ASs) Step 2: Assessng the Relatve Importance of QAs Step 3: Quantfyng the Benefts of the ASs Step 4: Quantfyng the Costs of ASs and Incorporatng Schedule Implcatons Step 5: Calculatng the Return for Each AS Step 6: Rank Orderng and Applyng an Approprate Decson-Rule Summary of the ECS Case Study 24 6 Related Work 25 7 Lessons Learned and Further Developments of the CBAM 27 8 Concluson 29 References 31 Appendx A: Mult-Attrbute Decson Theory for a Software Desgn Problem 35 Appendx B: Determnng the Probablty of Domnance of AS Return Values Case 1: Partal Overlap Case 2: Complete Overlap 42 CMU/SEI-2001-TR-035

7 Lst of Fgures Fgure 1: Fgure 2: Context of the Cost Beneft Analyss Method (CBAM) 3 ASs Mappng on a Beneft-Cost Plot Durng Trage Phase 6 Fgure 3: Components and the Influence of ASs 16 Fgure 4: Fgure 5: Effcent Portfolos of Archtectural Strateges 24 Maxmzed Boundary for a Two-Attrbute Problem 36 Fgure 6: Maxmzed Boundary wth Thresholds 36 Fgure 7: Sources of Uncertanty n Archtectural Beneft Assessment 38 Fgure 8: Expected Utlty Functons of QAs 39 Fgure 9: Case 1: Partal Overlap of ASs 41 Fgure 10: Case 2: Complete Overlap of ASs 42 CMU/SEI-2001-TR-035

8 v CMU/SEI-2001-TR-035

9 Lst of Tables Table 1: Elcted Trage Informaton 5 Table 2: Example of Scalng Parameters Elcted from Stakeholders 8 Table 3: Probablty of AS row >AS column 13 Table 4: Correlaton Matrx for ASs 16 Table 5: Descrpton of Qualty Attrbutes (QAs) 20 Table 6: Qualty Attrbute Ratngs of Stakeholders 20 Table 7: Template of AS Ratng Sheet 21 Table 8: Table 9: Aggregated Beneft Scores, Cost Values, Return Score of Top 10 ASs 22 Rank Orderng of ASs by Crteron (Top 10) 23 CMU/SEI-2001-TR-035 v

10 v CMU/SEI-2001-TR-035

11 Abstract The software archtecture forms an essental part of a complex software-ntensve system. Archtecture desgn decson-makng nvolves addressng tradeoffs due to the presence of economc constrants. The problem s to develop a process that helps a desgner choose amongst archtectural optons, durng both ntal desgn and ts subsequent perods of upgrade, whle beng constraned to fnte resources. To address ths need for better decsonmakng, we have developed a method for performng economc modelng of software systems, centered on an analyss of ther archtecture. We call ths method the Cost Beneft Analyss Method (CBAM). The CBAM ncorporates the costs and benefts of archtectural desgn decsons and provdes an effectve means of makng such decsons. The CBAM provdes a structured ntegrated assessment of the techncal and economc ssues and archtectural decsons. The CBAM utlzes technques n decson analyss, optmzaton, and statstcs to help software archtects characterze ther uncertanty and choose a subset of changes that should be mplemented from a larger set of alternatves. We also report on the applcaton of ths method to a real world case study. CMU/SEI-2001-TR-035 v

12 v CMU/SEI-2001-TR-035

13 1 Introducton and Motvaton The software archtecture s an essental part of a complex software-ntensve system. Shaw and Garlan [Shaw 96] state that, wth ncreasng complexty of a system, the specfcaton of the overall system,.e., ts software archtecture, becomes a more sgnfcant ssue than the choce of algorthms or data structures. The Archtecture Tradeoff Analyss Method (ATAM) [Kazman 00] provdes software archtects a framework to reason about the techncal tradeoffs faced whle desgnng or mantanng a software system. In the ATAM, we are prmarly nvestgatng how well the archtecture has been desgned wth respect to ts qualty attrbutes (QAs) that nclude modfablty, performance, avalablty, and usablty. It s these qualtes that shape the archtecture and consequently dctate the cost of buldng and mantanng a software system. The ATAM also analyzes archtectural tradeoffs, the places where a decson mght have consequences for several QA concerns smultaneously. The bggest tradeoffs n large, complex systems usually have to do wth economcs. How should an organzaton nvest ts resources n a manner that wll maxmze ts gans and mnmze ts rsk? Where ths queston has been addressed, t has prmarly focused on costs [Boehm 81], and even then these costs are prmarly the costs of buldng the system n the frst place, and not ts long-term costs through cycles of mantenance and upgrade. The benefts that an archtectural decson may brng to an organzaton are ust as mportant as the costs. Gven that the resources for buldng and mantanng a system are fnte, there must be a ratonal process that helps us choose amongst archtectural optons, durng both ntal desgn and ts subsequent perods of upgrade. These optons wll have dfferent costs, wll mplement dfferent features, each of whch brngs some beneft to the organzaton, and wll have some nherent rsk or uncertanty. Thus, we need economc models of software that take nto account costs, benefts, rsks, and schedule mplcatons. To address ths need for better economc decson-makng, we have developed a method for performng economc modelng of software systems, centered on an analyss of ther archtecture. We call ths method the CBAM (Cost Beneft Analyss Method). The CBAM bulds upon the ATAM to model the costs and the benefts of archtectural desgn decsons and provdes a means of optmzng such decsons. The CBAM provdes a structured ntegrated assessment of the techncal and economc ssues and archtectural decsons. In our framework, we ncorporate economc crtera lke beneft and cost that are derved from the techncal crtera lke qualty attrbute responses. CMU/SEI-2001-TR-035 1

14 The followng secton descrbes the decson-makng context behnd the CBAM, whle secton 3 provdes a broad outlne of the steps nvolved. In secton 4 we descrbe the steps n detal as well as the theoretcal bass of the CBAM. We apply the method to a real world proect, NASA s Earth Observatory System Dstrbuted Informaton System (EOSDIS) Core System (ECS), whch s outlned as our case study n secton 5. We descrbe related work n secton 6, dscuss our plans for mprovng the method n secton 7, and fnally conclude n secton 8. 2 CMU/SEI-2001-TR-035

15 2 Decson-Makng Context In the CBAM the software archtect or decson-maker wshes to maxmze the dfference between the beneft he or she derves from the system, and the cost of mplementng the desgn. The CBAM begns where the ATAM concludes, and n fact, depends upon the artfacts that the ATAM produces as output. Fgure 1 depcts the context for the CBAM. The busness goals of a software system are expected to nfluence the archtecture decsons made by software archtects or desgners. These archtecture decsons have economc as well as techncal mplcatons. The drect economc mplcaton s that of the cost of buldng/mplementng the system. The techncal mplcatons are the characterstcs of the software system, namely the qualty attrbutes. These qualty attrbutes n turn have economc mplcatons for the beneft that can be derved from the system. Busness Goals Archtecture Decsons Beneft P - Performance A - Avalablty S - Securty M - Modfablty Influences Cost Fgure 1: Context of the Cost Beneft Analyss Method (CBAM) When the ATAM s appled to a software system, we expect to have a set of artfacts documented upon completon. They are as follows: a descrpton of the busness goals that are crucal to the success of the system a set of archtectural vews that document the exstng or proposed archtecture a utlty tree that represents a decomposton of the stakeholders goals for the archtecture. The utlty tree starts wth hgh-level statements of QAs, decomposes these nto spe- CMU/SEI-2001-TR-035 3

16 cfc nstances of QA requrements (performance, avalablty, etc.) and realzes these as scenaros a set of rsks that have been dentfed a set of senstvty ponts (archtectural decsons that affect some QA measure of concern) a set of tradeoff ponts (archtectural decsons that affect more than one QA measure, some postvely and some negatvely) The ATAM results n a set of key potentally problematc archtectural decsons, based on QA scenaros elcted from the stakeholders. These archtectural decsons result n some specfc QA responses, namely, a partcular level of performance, securty, usablty, modfablty, and so forth. But those archtectural decsons also have an assocated cost. For example, f an archtectural decson s made to use redundant hardware to ncrease relablty, then ths has one cost consequence. If check pontng to a dsk fle s used nstead, then ths archtectural decson has a dfferent cost. Furthermore, both of these archtectural decsons wll result n a measurable level of relablty (perhaps measured as mean tme to falure or steadystate avalablty). These QA responses wll have some value to the organzaton developng software. Perhaps the organzaton beleves that ts stakeholders wll pay extra for a hghly relable system (a telephone swtch, for example) or that the organzaton wll get sued f the system s not hghly avalable (for example, a medcal montorng devce). The ATAM uncovers the archtectural decsons made and lnks them to busness goals and QA response measures. The CBAM bulds on ths foundaton by fllng n the pentagons n Fgure 1, by elctng the costs and benefts assocated wth these decsons. 1 Gven ths nformaton, the stakeholders can then decde whether to use redundant hardware, check pontng, or some other archtectural decson addressed at ncreasng the system s relablty. Or the stakeholders can choose to nvest ther fnte resources n some other QA perhaps belevng that hgher performance wll have a better beneft to cost rato. A system always has a lmted budget for creaton or upgrade, so every archtectural choce s, n some sense, competng wth every other one for ncluson. The problem addressed n ths techncal report s that, gven the nevtable constrants on cost, how should an archtect choose a small set of archtectural strateges to be mplemented from a large set of alternatves? The CBAM does not make decsons for the stakeholders; t smply ads them n the elctaton and documentaton of costs, benefts, and uncertanty and gves them a framework wthn whch they can apply a ratonal decson-makng process. 1 The CBAM does not nclude a new way of determnng costs (although we thnk that an archtecture-aware cost estmaton method s a desrable goal). It assumes that some method of cost estmaton already exsts wthn the organzaton. 4 CMU/SEI-2001-TR-035

17 3 The Steps of the CBAM The CBAM conssts of sx steps. 1. Choose the scenaros and archtectural strateges (ASs). 2. Assess the relatve mportance of QAs. 3. Quantfy the beneft scores of the ASs. 4. Quantfy the costs of the ASs and ncorporate schedule mplcatons. 5. Calculate the return for each AS. 6. Rank order the ASs and apply an approprate decson rule. When we start wth a CBAM exercse, there s a large number of desred archtectural changes or ASs. Performng a detaled CBAM exercse nvolvng exhaustve elctaton and analyss of each of these changes s an mpossble task. To wade through such a large space of possble changes makes t necessary to use a two-phase approach. Hence, the CBAM conssts of two phases: the trage phase, where rough order of magntude estmates are used to prune the decson space, and the detaled examnaton phase, where more detaled estmates are gathered about the more promsng archtecture desgn alternatves. The sx steps, descrbed above, are carred out n both phases. 3.1 The Trage Phase In the trage phase of the CBAM we elct the costs and benefts of the changes n a rough qualtatve manner. Another step s to fnd a consensus amongst the stakeholders about the defnton of the varous QAs of mportance as elcted durng the ATAM. Ths exercse s carred out so that the stakeholders have a common understandng about the attrbutes that they are gong to use as a bass for ratng the varous ASs. Table 1: Elcted Trage Informaton Archtectural Performance Avalablty Modfablty Cost Strategy (50) (30) (20) AS L AS M AS H AS H AS M AS L CMU/SEI-2001-TR-035 5

18 Durng trage, we request the experts to rate every AS by QA on a fve-pont scale (--, -, 0, +, ++). For the cost estmates we use a smple three-pont scale: low, medum, hgh. The varous QAs are assgned votes, on the bass of ther mportance to the system, such that the sum of the votes s equal to 100. An example of the elcted nformaton s shown n Table 1. The qualtatve scales are converted to rough quanttatve estmates for beneft as well as cost. The ASs are then plotted on a graph of beneft aganst cost as shown n Fgure 2. Ths exercse helps n vsualzng the alternatves. AS 4 AS 1 AS 6 AS 2 AS 15 AS 7 AS 3 AS AS 17 AS 5 AS 8 14 AS 9 AS 12 AS 13 AS 1 Low Medum Hgh Cost Fgure 2: ASs Mappng on a Beneft-Cost Plot Durng Trage Phase 2 The ovals surroundng the ASs depct the uncertanty regon. The chosen ASs are shown n bold. The number of ASs consdered for detaled analyss are pruned on the bass of havng a hgh beneft - low cost characterstc, other factors lke regulatons/standards and that they have no resource conflcts or dependences. In ths manner, trage s used to prune the set of ASs to a manageable number of alternatves that can be analyzed n detal. We expect about ASs for detaled examnaton. It s qute possble that some organzatons may not feel the need for a detaled analyss due to resource constrants. For some, order of magntude estmates or qualtatve measures are suffcent for ther decson-makng process. In these cases the trage could prove to be suffcent for ther needs. 3.2 The Detaled Examnaton Phase In the detaled examnaton phase we perform a detaled analyss on the ASs that were found to be promsng n the trage phase. The strateges are re-assessed for ther mpact on the software system s QAs, and a more quanttatve scale, whch vares from 1 to +1, replaces the -/+ scale of the trage phase. We descrbe the theoretcal bass and the step detals n the followng secton. 2 The ovals shown wthn each cost category are representatve and not actual postons. 6 CMU/SEI-2001-TR-035

19 4 Theoretcal Bass of the CBAM and Detaled Descrpton The archtecture desgn problem of analyzng tradeoffs between the varous qualty attrbutes of a system can be characterzed n a tradtonal mult-attrbute decson theory framework. We use ths framework as the bass for the CBAM. We explore the problem of uncertanty [Morgan 90] and also examne technques lke portfolo analyss [Markowtz 52] to tackle our problem. Mult-attrbute decson theory as descrbed by Keeney and Raffa [Keeney 76] s adapted to software systems, and the formalsms are presented n Appendx A. Complex software decson problems nvolve conflctng obectves wth multple attrbute measures. Usually, there s no alternatve that s superor, n all respects, to other alternatves,.e., no domnatng soluton. There are value tradeoffs between the varous qualty attrbutes of the system. For example, one alternatve may have hgh performance but low mantanablty, whle another alternatve has hgh mantanablty but low performance. Choosng between the two alternatves wll nvolve understandng the value udgments of the key stakeholders n the software proect. (A domnatng alternatve would be one wth hgh performance and hgh mantanablty but rarely do we fnd such easy and obvously optmal solutons to problems!) The decson-makng framework thus nvolves 1. determnng the scalng parameters or the relatve mportance of the attrbutes (whch we call QAScore ) 2. determnng the attrbute-specfc mpact or utlty values (whch we call ContrbScore,.) The product of these values gves us a measure of the beneft due to an AS and can thus be used as an evaluaton metrc. The detals are descrbed n the followng sectons. 4.1 Step 1: Choose Scenaros and Archtectural Strateges (ASs) In the frst step, we choose a smaller number of ASs for detaled analyss. Frequently t s the case that certan changes are dctated by external forces keepng up wth a compettor, beng forced to port to newer hardware, meetng government regulatons, complyng wth standards, etc. Thus some of the ASs consdered n the trage phase may be automatcally consd- CMU/SEI-2001-TR-035 7

20 ered n the next phase n spte of ther beneft/cost value. These are ndcated by AS 8 and AS 15 n Fgure 2. From the trage phase, we also dentfy the ASs that have a hgh beneft but low to medum cost; ths s shown by the dotted rectangle n Fgure 2. Some ASs may be excluded due to resource conflcts or lack of support from the management. Addtonally, some ASs that are outsde ths regon may be ncluded, because of dependences (.e., AS 13 may depend on AS 7 ). A few other ASs may be chosen for analyss due ther proxmty to already chosen ASs (lke AS 5 and AS 14 ). Thus a smaller set of ASs are chosen for detaled analyss. 4.2 Step 2: Assess the Relatve Importance of QAs (Elct QAScore ) Once the varous qualty attrbutes of mportance have been dentfed, the scalng parameter QAscore for each qualty attrbute,, s elcted. By desgn, we ensure that QAScore =100 and : QAScore 0 An example of elcted QAscore parameters s shown n Table 2. Table 2: Example of Scalng Parameters Elcted from Stakeholders Qualty Attrbute Stakeholder 1 Stakeholder 2 Stakeholder 3 Attrbute1 50 (1) 60 (1) 40 (1) Attrbute2 15 (3) 15 (2) 15 (3) Attrbute3 25 (2) 10 (4) 30 (2) Attrbute4 10 (4) 15 (2) 15 (3) QAscore The stakeholders are usually from the same organzaton or have smlar understandng of the goals of the system. They are frequently n accordance wth each other regardng the scalng parameters. However, there are tmes when they may show great dfferences of opnon. Ths dfference n opnon wll manfest as a hgh varablty of elcted values. The varablty could be ether due to the nherent uncertanty present n the system or that the stakeholders are not completely n agreement wth each other regardng the defnton of the scalng parameters. To test how much the stakeholders agree wth each other regardng the scalng parameters, we use the Kendall s 3 concordance coeffcent as a measure of agreement for the group as a whole [Kendall 55]. The more the group agrees over the rank of each attrbute, the hgher the concordance coeffcent. In the example shown n Table 2, we have 3 stakeholders and 4 attrbutes. The values n the columns show the elcted scalng parameters and the numbers n the bracket show the rank 3 It s smlar to Spearman s correlaton coeffcent but t s a non-parametrc measure. 8 CMU/SEI-2001-TR-035

21 of that partcular attrbute for that partcular stakeholder. Kendall s concordance coeffcent reflects how much the stakeholders agree on the rank of the partcular attrbute. In ths example, the Kendall s coeffcent of concordance, W, s (F-stat = 4.461, p = 0.08), whch shows a far level of concordance, though t s not sgnfcant. Though the values elcted from the stakeholders may dffer, a consstent ratng on the relatve mportance of each of the attrbutes shows that the stakeholders more or less agree as to whch attrbutes are mportant to the busness goals of the system. If the concordance value s low or not sgnfcantly greater than zero, then the stakeholders wll need to revst the busness goals of the system and defntons of the QAs. Ths s to ensure that the stakeholders have a common understandng of the QAs and ther respectve mplcatons on the busness goals of the system. Ths teraton s carred out to ensure that varablty due to lack of understandng s reduced. 4.3 Step 3: Quantfy the Benefts of ASs (Elct ContrbScore, ) A sngle archtectural strategy affects more than one QA. Some effects are postve, some negatve, and all to varyng degrees. When consderng large software systems, there s uncertanty regardng the exact effect of a partcular archtectural strategy on the system. Whle the effect of some changes could be measured approxmately through smulaton models, the effect of some changes can only be obtaned through expert elctaton of the experts of the system and ther understandng of system behavor. Another source of uncertanty and subectvty s the utlty assessment of varous levels of the qualty attrbutes. (See Fgure 7 n Appendx A.) In ths step we ask the experts to rate each archtectural strategy (AS ) n terms of ts contrbuton (ContrbScore, ) to each qualty attrbute (QA ) on a scale of 1 to +1. A plus 1 mples that the AS has a perceved best possble effect on the QA for the system and a mnus 1 means the opposte. These values represent the utlty ganed or lost by makng a partcular archtectural change (AS ) on each of the qualty attrbutes (QA ) Addressng Varablty n the Contrbuton Score So far we have explaned a determnstc case of elctng contrbuton scores. However, wth multple stakeholders and sources of uncertanty the elcted values wll be ranges and not sngle pont estmates. As explaned n the prevous secton, we assume that the varablty n contrbuton scores may be due to two followng reasons: 1. the exstence of uncertanty n the system response characterstcs or 2. a dfference n nterpretaton or understandng of the system or partcular subsystems by experts We address the varablty due to uncertanty n our analyss of elcted values explaned later n ths report. However, the varablty arsng due to the nterpretaton or understandng of CMU/SEI-2001-TR-035 9

22 the system s somethng we wsh to mnmze (f not elmnate completely). Here we use the Delph method [Lnstone 75] for obtanng consensus on the specfc mpacts of partcular archtectural strateges on the system. The specfc process we use s the followng: 1. Obtan the frst round of contrbuton scores from all experts ncludng a short one-lne statement as to why the expert has rated a partcular AS n that way. For example, the expert could annotate the comment: Improved loggng would allow systematc securty audts aganst the ratng for securty. 2. Also obtan self-evaluatons of the experts, wth regard to ther understandng of the mpacts of the AS, on a four pont scale ( Don t have any dea, have some dea, have a reasonable dea, and have a good dea ). 3. Comple scores to produce ranges and the comments for each AS. 4. Provde all the experts wth the range and comment nformaton and ask them to revew ther ratngs based on the collected nformaton. We do not reveal the expert raters dentty to reduce the possblty of nfluencng other raters. 5. Obtan a fnal range on contrbuton scores and comple the one-lne reasons gven by experts as documentaton. In ths manner, we expect to mnmze the varablty of elcted values due to lack of nformaton or lack of understandng of all the mplcatons of a change. We also gve the experts the choce to recuse themselves from ratng certan archtectural changes due to ther lack of understandng about the mpacts of an AS. The measure of varablty n the scores we use s the Kendall s concordance coeffcent. We compute ths coeffcent from the elcted contrbuton scores and use t as an ndcator of how consstent the raters are wth each other. A low coeffcent ndcates that the varablty n ratngs s hgh and t s plausble that some experts may be mssng certan nformaton that other experts possess. Thus, we could use the coeffcent as a measure of the stoppng pont for the teratve Delph process of crculatng nformaton and elctng ratngs. Though ths process seems tedous and tme consumng, we feel that ths s necessary to ncrease the obectvty of the ratngs and make sure that every expert has the same nformaton to make the ratng. Another beneft of ths exercse s to capture the nformaton contaned n the one-lne statements made by the experts. Ths may be used to buld models of the system and to generate utlty-attrbute curves. (See Fgure 8 n Appendx A.) Calculatng the Beneft Scores of the ASs At ths pont n the method we have elcted values for the scalng parameter, QAscore,to reflect the mportance of the QAs wth respect the busness goals and the utlty of each archtectural change, ContrbScore,. We can now calculate the beneft score for each archtectural strategy by the followng formula: 10 CMU/SEI-2001-TR-035

23 Beneft ( AS) = ( ContrbScore, ) ( Qattrb ) Snce the ContrbScore s a utlty estmate, the beneft score s expressed n utls. For example, consder a software system wth the followng scalng parameters: (performance, securty, avalablty, modfablty) = (25, 30, 15, 30) and the ratng of the archtectural strateges for the respectve QAs as follows: AS 1 (1, -0.5, 0.6, -0.4) AS 2 (-0.4, 1.0, 0.8, 1.0) The respectve beneft scores wll be: Beneft(AS 1 ) = (1)*25 + (-0.5)*30 + (0.6)*15 +(-0.4)*30 = 7 utls Beneft(AS2) = (-0.4)*25 + (1)*30 + (0.8)*15 +(1.0)*30 = 71.2 utls Ths beneft score utlty estmate s bounded between -100 and +100 utls. 4.4 Step 4: Quantfy the Costs of the AS and Incorporate Schedule Implcatons We have descrbed n detal a method to elct benefts of varous archtectural strateges. There are few studes that dscuss the beneft of archtectural strateges. Cost estmaton on the other hand, as far as the mplementaton cost s concerned, has receved consderable attenton n the software engneerng lterature [Boehm 88, Jones 99]. Typcally, most mature organzatons adopt ther own methods for cost estmaton across all proects. The CBAM does not nclude any new way of determnng costs. However, consderng that most cost estmaton technques are dependent on much fner detaled parameters lke lnes of code or number of varables and other mplementaton detals, we thnk that an archtecture-aware cost estmaton method s a desrable goal. Ths archtecture-aware cost model would enable us to obtan cost estmates based on the type of components used or the archtectural styles used to desgn the system. As of now, the CBAM assumes that some method of cost estmaton already exsts wthn the organzaton, even f t s ad-hoc, and we can drectly obtan cost estmates, C,foreachofthe archtectural strateges. In addton to elctng the costs and benefts of the ASs under consderaton, prudent plannng dctates that we estmate the schedule mplcatons of each AS n terms of elapsed tme, shared use of crtcal resources, and dependences among mplementaton efforts. Perhaps an AS s otherwse desrable, but does not ft n wth the organzaton s tme-to-market goals. Durng ths step we wll note any contenton for shared resources among these estmates (hardware, software, or personnel), for these wll also affect the feasblty of an AS. CMU/SEI-2001-TR

24 4.5 Step 5: Calculate Return for Each AS The next step n the process of rankng our archtectural strateges s to calculate the return score (r ). Ths score s gven by Return (AS ) = Beneft (AS ) Cost (AS ) The unts of the return score wll dffer accordng to the unts of beneft and cost elcted. Ideally the return score wll be a non-dmensonal number snce both the beneft and cost are utlty values, n utls or n dollars. Due to the range of values n beneft as well as cost, the calculated return scores have a range of values. The return score s smlar to a return-onnvestment (ROI) metrc used commonly n the fnancal lterature Step 6: Rank Order the ASs and Apply an Approprate Decson-Rule A decson rule that s based on the mean value of the return could be appled. The ASs could be rank ordered accordng to the mean value of ther return. Ths assumes that the underlyng dstrbuton of the return s symmetrc around the mean value. The top r ASs could then be chosen such that r =1 Cost ( AS ) K where K s the total budget amount avalable for the proect. The number of stakeholders s usually small and the mean value could be skewed consderably because one stakeholder vares consderably from other stakeholders. To account for ths, another crteron for rank orderng s the medan value of the return score. Smlarly, the top r ASs could be chosen, subect to the cost constrant as shown above A Decson-Rule Based on Probablty In the above two rankng methods, we have assumed that the central moments, namely the mean and medan estmates for return, accurately represent the AS. Usng the mean or medan values of return assumes that the underlyng dstrbuton of the return score s normal/log-normal. Consderng that uncertanty s farly hgh, and the underlyng dstrbuton at tmes skewed, an expected value decson rule may not be suffcent to capture the uncertanty. In ths secton we brefly descrbe a technque to ncorporate the uncertanty nto our decson. 4 The relaton between our defned return and ROI s: ROI = (return 1) x 100 f the unts of cost and beneft are the same. 12 CMU/SEI-2001-TR-035

25 Consderng that the number of stakeholders s usually small and each stakeholder conveys mportant nformaton about the elcted values, we assume that the underlyng dstrbuton of the return score s unform. Ths assumpton s reasonable consderng that the outler may be examnng the system from a perspectve that other stakeholders may not be able to apprecate. If each archtectural strategy s plotted along a sngle scale, accordng to ts return score, there could be overlap of scores between any two ASs. Gven that the values overlap, we cannot say for certan whch AS s preferable. For ths purpose we develop a probablstc technque, where we calculate the probablty that the return score of any AS s greater than the return of another AS. The detaled dervatons for two cases of possble overlap are provded n Appendx B. Case: 1: When there s partal overlap between AS and AS. prob 1 ( AS AS ) = { AS AS } 2 2R R,max Case:2:WhenAS completely overlaps AS. 1 prob AS AS = 2AS,max AS 2R,mn ( ) { AS },max where R s the total range of the return value for AS and AS,mn and AS,max are the mnmum and maxmum return values of AS respectvely.,mn For example, consder a sample set of 6 ASs. The above technque yelds the probablty matrx shown n Table 3. Each value n the matrx denotes the Probablty (Row-AS >= Column- AS). Table 3: Probablty of AS row >AS column AS1 AS2 AS3 AS4 AS5 AS6 Avg. AS AS AS AS AS AS The probabltes n Table 3 are used to determne the senstvty of our earler rankng decson. If we choose AS5 over AS6, then the table tells us that there s a 40% probablty that we may be ncorrect n our rank orderng. If we fx p >= 0.6, as a lmt for partal domnance, then we get the followng result: AS1> AS3, AS4, AS6 CMU/SEI-2001-TR

26 AS2> AS1, AS3, AS4, AS5, AS6 (Clearly a hghly ranked AS) AS3> AS4, AS6 AS4> AS5> AS1, AS3, AS4, AS6 AS6>AS4 We can see that n these condtons, AS2 s the most preferred AS, followed by AS5, AS1, AS3AS6,AS4nthatorder. We use ths probablstc framework to rank the dfferent archtectural strateges. We have shown how a decson rule based on a partcular probablty value, p, can be used to ncorporate the varance nformaton. We notce that the rank orderng based on the p-value s also reflected n the average value of the probabltes n the rows. Thus, smlar to earler rankng methods, we could rank order based on the average probablty value for any AS, and then choose the top r ASs, subect to the constrant of cost. An extra constrant n choosng the set of ASs could be that every AS chosen should have a return greater than the return of any other AS wth a probablty greater than p unless the other AS has already been chosen. Formally, ths s represented as follows: Let Chosen Set be represented by CS, then AS P( AS AS CS > AS CS ) p Substtutng the unform dstrbuton wth any other dstrbuton would requre the development of a dfferent analytcal soluton for calculatng the probabltes, though the underlyng prncple wll reman the same Dealng wth Combnatons of Strateges Usng the Portfolo Theory Framework The dea of dversfcaton of stocks wthn a portfolo by nvestors to reduce the standard devaton on the returns was shown by Markowtz [Markowtz 52]. The central dea behnd portfolo theory s to combne two assets n a portfolo that are not perfectly correlated n order to reduce the overall uncertanty on the returns. The dea behnd the portfolo approach to nvestment s to reduce the overall uncertanty by mplementng ether uncorrelated or negatvely correlated strateges [Butler 99]. For example, 14 CMU/SEI-2001-TR-035

27 we could have a partcular archtectural strategy, AS 1, whch has hgh beneft wth relatvely hgh cost and uncertanty. Another archtectural change, AS 7, could have low beneft and low cost and uncertanty. AS 1 s consdered as a hgh-rsk opton whle AS 7 s a low rsk opton. Further analyss of the dependences shows that AS 1 and AS 7 are competng technologes that are mutually exclusve and negatvely correlated as far as ther uncertantes are concerned. In smpler terms, we fnd that f AS 1 s successful, AS 7 wll not be and vce versa. Ths concept leads us to the generalzaton that there are a number of other changes AS and AS, whch, combned together, could reduce the overall rsk of the system even though some of them are doomed to falure. The archtectural changes thus chosen could be consdered to be a portfolo that attempts to ensure a certan level of success, whle smultaneously reducng the overall rsk. When combnng the varous archtectural strateges to form a portfolo, one also needs to look at the techncal feasblty of mplementng all of them at the same tme, as well as the mpled schedule and budget consderatons. Smlar to the problem of rank orderng varous ASs, we now consder varous ways of combnng ASs such that the resultant portfolo maxmzes return on nvestment and mnmzes the varance of the portfolo subect to a cost constrant. The return and varance of a portfolo contanng ASs s gven by: R = x r Var = x x σ σ ρ where = and ρ = 1 x C C The return (r ) s obtaned as descrbed n the prevous secton. The value σ s the uncertanty of the return score. Ths formulaton tells us that when we combne strateges that are correlated wth each other, then the varance on return for the combnaton s lower. The correlaton value, ρ, whch les between +1 and 1, s more challengng to obtan. For ths purpose, we ntroduce the concept of a dependency structure between the varous ASs under consderaton as a proxy for the correlaton value. Fgure 3 shows the nfluence of varous ASs on the components of a system. The detals of the component names are not mportant for our example. CMU/SEI-2001-TR

28 IOS DMS MSS DSS AS 7 AS 5 PLS DPS INS AS 1 AS 4 Fgure 3: Components and the Influence of ASs We use ths example to show how the dependency structure could be elcted. We see n ths example that AS 7 and AS 1 overlap they affect the same component (MSS). Snce they mplement competng technologes, they are consdered to be uncorrelated (.e., success of one guarantees falure of the other). Smlarly, we see that AS 1 and AS 4 do not have any overlap n terms of components affected. Based on ths nformaton, we elct the correlaton nformaton from the experts as shown n Table 4. The nformaton obtaned could be qualtatve (+/- H, M, L, or 0) or quanttatve (a number between 1 and +1). Table 4: Correlaton Matrx for ASs Arch. Strategy AS 1 AS 4 AS 5 AS 7 AS (+L) -0.6(-H) AS (+H) 0 AS (-L) AS 7 1 In tradtonal portfolo management, the nvestor chooses a partcular value of x, whch s the fracton of total nvestment n a portfolo n a partcular asset. In the case of software, ths flexblty s not avalable snce the desgners cannot nvest n fractons of an AS. However, n a portfolo of ASs, the value of x for AS vares dependng on the chosen set of ASs. If we have 4 strateges n consderaton, we can construct 15 portfolos ( 4 C C C C 4 ) wth varyng return, cost, and varance. These 15 portfolos are analyzed and a portfolo wth a reasonable return, cost, and varance s chosen for mplementaton. 5 We also observe that, as the number of consdered strateges ncreases, the number of possble combnatons of portfolos ncreases n the order n 2 and the calculatons become computatonally ntensve. 5 The proect manager determnes what s consdered reasonable. 16 CMU/SEI-2001-TR-035

29 The CBAM, through ts dea of quantfcaton of beneft, cost, and uncertanty helps us structure the problem of choosng ASs n the framework of well developed methods from nvestment theory. The superor technques now avalable to us could be appled to the problem for an economcally sound soluton. CMU/SEI-2001-TR

30 18 CMU/SEI-2001-TR-035

31 5 Case Study In ths secton we descrbe the applcaton of the CBAM to a real-world software proect to demonstrate the feasblty of ths method for large-scale proects. The CBAM helped structure an unstructured archtecture desgn problem and offered the proect manager a set of solutons to choose from. We frst descrbe the proect n bref to gve a sense about the magntude of the proect and then outlne the varous CBAM steps and the results from applyng them. 5.1 Descrpton of the Proect The Earth Observng System s a constellaton of NASA satelltes that gathers data about the Earth for the U. S. Global Change Research Program and other scentfc communtes worldwde. The Earth Observng System Data Informaton System (EOSDIS) Core System, also called the ECS, collects data from varous satellte downlnk statons for further processng. The msson of the ECS s to process the data nto hgher-form nformaton and make t avalable n searchable form to scentsts around the world. The goal s to provde a common way to store (and hence, process) data, and a publc mechansm to ntroduce new data formats and processng algorthms, thus makng the nformaton wdely avalable to the scentfc communty at large. The ECS processes an nput stream of hundreds of ggabytes of raw envronment-related data per day. The processng nvolvng the computaton of 250 standard products results n thousands of ggabytes of nformaton that gets archved at 8 datacenters n the Unted States. The ECS has mportant performance and relablty requrements. The long-term nature of the proect also makes modfablty an mportant requrement. 5.2 Applyng the CBAM to the ECS The ECS, wth about 1.2 mllon lnes of code n 12,000 modules and about 50 customer offthe-shelf (COTS) products, s a very large software system. We had seven stakeholders, chosen by the proect manager, to take part n the CBAM exercse Step 1: Choosng the Scenaros and Archtectural Strateges (ASs) Pror to the CBAM, an ATAM exercse was carred out for the ECS proect. As a result of ths exercse, 72 archtectural strateges were dentfed to mprove the system. Except for one strategy that was deemed crucal and non-negotable, the rest of them needed to be analyzed CMU/SEI-2001-TR

32 wth respect to ther benefts, costs, and uncertanty. When analyzng an archtectural strategy, the decson was whether or not the proect should mplement that strategy Step 2: Assessng the Relatve Importance of QAs To quantfy the relatve mportance of the varous QAs, the stakeholders frst outlned a lst of QAs that they felt were mportant to the software system s goals. These QAs, as shown n Table 5, were arrved at through consensus and dscusson of the defnton of each, as well as relatng them to the utlty tree constructed durng the ATAM exercse. The seven stakeholders then ndependently rated each of the QAs as descrbed n secton 4.2. The ratngs are shown n Table 6. The stakeholders then dscussed the values to arrve at a consensus value to be used through the rest of the method. Table 5: Descrpton of Qualty Attrbutes (QAs) Attrbute Mantanablty Operablty Relavalablty Scalablty Performance User Satsfacton Securty Flextensblty Descrpton Lowers mantenance cost of the system by makng t easer to fnd and fx problems and deploy mnor changes to exstng problems To make t cheaper to operate the system or do more load wth less people (ncrease effcency) Decreases the degradaton n throughput due to downtme; mnmzes the operator requrement n recovery of the requests; no key nputs/outputs to the system are lost Stable to ncrease the problem sze, capacty (hardware, operators) scales lnearly wth workload: system components scale lnearly wth problem sze Reduce the end user request latency whle ncrease system throughput Increase user empowerment/capablty; decreased response tme, accuracy, understandablty The ablty to mantan the ntegrty of ther data holdng and mantan prvacy of the user nformaton and reduce the loss of avalablty due to denalof-servce attacks Ablty to nsert or add maor new functonalty or products Table 6: Qualty Attrbute Ratngs of Stakeholders Qualty Attrbute Stk1 Stk2 Stk3 Stk4 Stk5 Stk6 Stk7 Mean Var. Consensus Mantanablty Operablty Relavalablty Scalablty Performance User Satsfacton Securty Flextensblty CMU/SEI-2001-TR-035

33 Usng the values elcted n Table 6 we calculate the Kendall s coeffcent, W, to be , ndcatng a hgh level of concordance (F-stat=31.1, p<0.001). Ths gves us confdence that the stakeholders agreed wth each other about the mportance of varous QAs wth respect to the busness goals of the system Prunng the Decson Space In the trage phase of the method, the stakeholders were provded wth the lst of 72 archtectural strateges that were consdered desrable changes to the system. The stakeholders ndependently rated each AS by QA on a 5-pont (-- to ++) scale. The cost estmate for each was also noted on a 3-pont scale (H, M, L). The ratngs were converted to a quanttatve scale (-1 to +1), and a total beneft was computed. The proect manager guded the dscusson to ensure consensus about the chosen strateges and that no mportant aspect was beng overlooked. Applyng a mxed strategy for evaluaton, the prunng process resulted n 25 ASs for detaled analyss Step 3: Quantfyng the Benefts of the ASs The detaled analyss phase began wth the experts ndependently ratng each of 25 ASs by the QAs on a scale of -1 to +1. A template of the ratng sheet s shown n Table 7. Table 7: Template of AS Ratng Sheet QAscore Total Comment AS Number QA1 QA2 QA3 QA4 AS1 AS2 AS3 The stakeholders vared consderably n ther ratngs for varous ASs and the ranges were recorded as a measure of the uncertanty that the stakeholders had about the effects of the ASs to the system attrbutes. The concordance score was computed to be (F-stat = 2.9, p<0.001), whch nforms us that t was farly low, yet stll sgnfcantly greater than zero. A follow up Delph exercse was ntated wth the experts Step 4: Quantfyng the Costs of ASs and Incorporatng Schedule Implcatons The costs of varous ASs were assessed n terms of person-months. Ths estmate was obtaned from two of the stakeholders who ndependently performed a cost estmaton exercse. CMU/SEI-2001-TR

34 No gudance regardng cost estmaton was gven and the experts appled exstng methods. The two stakeholders were found to be consstent n ther estmates Step 5: Calculatng the Return for Each AS The return for the varous ASs was calculated usng the formula specfed n secton 4.5. The results are shown n Table 8. The uncertanty n the form of range as well as standard devaton was also calculated Step 6: Rank Orderng and Applyng an Approprate Decson-Rule The ASs were then rank ordered based on ther mean and medan return score. To determne the senstvty of ths rank orderng, we then calculated the probablty matrx as explaned n secton The resultng rank orderng of our 25 ASs by the mean, medan, and probablty s shown n Table 9. Table 8: Aggregated Beneft Scores, Cost Values, Return Score of Top 10 ASs AS Number AGGREGATE BENEFIT SCORE MIN MAX MEAN MEDIAN COST Return (p. m.) MEAN MEDIAN AS AS AS AS AS AS AS AS AS AS The orgnal estmates dffered consderably untl a follow-up dscusson establshed that one stakeholder had ncluded cost of testng whle the other hadn t. The ensung adustment resulted n an almost perfect correlaton n cost estmates. 22 CMU/SEI-2001-TR-035

35 Table 9: Rank Orderng of ASs by Crteron (Top 10) Arch. Strategy Return based Rank Mean Medan Probab. Cost based Rank AS AS AS AS AS AS AS AS AS AS Portfolo Theory Framework to Pck a Set of ASs In secton we descrbed a technque to ncorporate the uncertanty as well as the dependency structure of the varous ASs to make a choce. Ths nvolved the elctaton of a dependency structure among the ASs, whch was used to populate the correlaton matrx. Ths matrx was then used to buld portfolos contanng varous ASs. The return, varance, total beneft, and total cost were computed for each of these portfolos. Consderng that we had 25 ASs, the total number of possble combnatons s gven by 25 C C C 25 = 33,554,431 Consderng that we have a budget and schedule constrants, we assume that the total number of ASs that can eventually be mplemented cannot exceed 8; we restrct the maxmum number of ASs n our chosen portfolo to be 8. We also assume that to get a reasonable beneft, the portfolo must contan at least 4 ASs. Even wth ths restrcton, the total number of possble combnatons s 1,805,155 ( 25 C C C 8 ). Ths s stll a farly large number and the analyss and comparson of portfolos took about 2.5 hours on a Pentum III, 450MHz processor. A portfolo that belongs to a set of portfolos that s non-domnated on the crtera of return and varance s called an effcent portfolo. The effcent portfolos are shown n Fgure 4. CMU/SEI-2001-TR

36 Return Varance Fgure 4: Effcent Portfolos of Archtectural Strateges 5.3 Summary of the ECS Case Study In ths secton we llustrated how we appled the CBAM to NASA s ECS proect. The feedback we receved from the ECS proect manager ndcates that the method holds sgnfcant promse for facltatng the decson makng process. A beneft perceved by the stakeholders was the structured approach to prune ther decson space for changes. The quantfcaton asssted them n obectvely assessng the varous ASs and mplementng those that would gve them the greatest return. The method aded the stakeholders n determnng how to allocate lmted resources to ther software evoluton effort and they partcpated enthusastcally. 24 CMU/SEI-2001-TR-035

37 6 Related Work Tradtonal software desgn methods address the techncal aspects of a software system by dentfyng the functonalty to be delvered durng the requrement specfcaton phase of product development. Analyss of the product wth respect to ts economc mpact, rsks, and tradeoffs s seldom performed. If done, t manly focuses on the cost of the system and the methods of estmatng cost accurately. Identfcaton of the source of beneft for a software system and ts quantfcaton s rarely carred out. Some attempts have been made to te the busness goals of the system to the techncal aspects of a software system. The Spral model of software development [Boehm 88] and the Next Generaton Process Model (NGPM) [Boehm 94] help system stakeholders converge on a system s next-level obectves, constrants, and alternatves. The NGPM uses the Theory W, whch nvolves dentfcaton of the system s stakeholders and ther respectve wn condtons. Usng a negotaton process among the stakeholders, a mutually satsfactory set of obectves, constrants, and alternatves s determned. Our method dffers from the NGPM n the aspects of the stage n the lfe cycle of the development process, as well as the level of abstracton wthn the software system. The NGPM addresses the requrements specfcaton stage and tres to understand the broad functonal requrements, whle our method s at the archtectural level of abstracton and expects that the functonal requrements are understood and already carred out. In some sense the CBAM s complementary to the NGPM. There are a few studes on understandng the qualty attrbutes of a software system. Johansson et al. [Johansson 01] use a survey method to understand how dfferent stakeholders vew dfferent qualty attrbutes and how they rank the mportance of each. They surveyed the software archtect, system desgner, and marketng manager of three dfferent organzatons (one educatonal, two commercal) as stakeholders. The results from the study ndcated that dfferent stakeholders prortze the varous qualty attrbutes dfferently n spte of a common organzatonal goal. They also conclude that there are nadequate metrcs to measure the mpact of the qualty attrbutes of software platforms. Ths leads to stakeholders not knowng the qualty attrbutes they need to focus on for mprovements. An area of further research they have dentfed s the feedback loop to the stakeholders about how the varous qualty attrbutes affect cost, qualty, and lead-tmes for proects bult on a software platform. Snce our method drectly follows an ATAM, the system stakeholders and experts already have a common level of understandng about the system goals and the attrbutes of mportance. In the CBAM we also revst the defnton of the varous QAs and make sure that all the stakeholders understand as well as agree about the defnton of each and how they affect the system goals. CMU/SEI-2001-TR