An Effective Path Selection Strategy for Mutation Testing

Transcription

1 An Effectve Path Selecton Strategy for Mutaton Testng Mke Papadaks and Ncos Malevrs Department of Informatcs, Athens Unversty of E.B. Athens, Greece {mpapad, Abstract Mutaton testng has been dentfed as one of the most effectve technques, n detectng faults. However, because of the large number of test elements that t ntroduces, t s regarded as rather expensve for practcal use. Therefore, there s a need for testng strateges that wll allevate ths drawback by selectng effectve test data that wll make the technque more practcal. Such a strategy based on path selecton s reported n ths paper. A sgnfcant nfluence on the effcency assocated wth path selecton strateges s the number of test paths that must be generated n order to acheve a specfed level of coverage, and t s determned by the number of paths that are found to be feasble. Specfcally, a path selecton strategy s proposed that ams at reducng the effects of nfeasble paths and conversely developng effectve and effcent mutaton based tests. The results obtaned from applyng the method to a set of program unts are reported and analysed presentng the flexblty, feasblty and practcalty of the proposed approach. Keywords: software testng, unt testng, mutaton testng, path testng, adequacy crteron, nfeasble paths I. INTRODUCTION It s a well known fact that the cost of software testng can reach 50% or even 60% of the total software development cost. In order to reduce the cost overhead, a lot of effort has been put by the software engneerng communty, n an attempt to automate the testng actvtes and thus reduce the overall software development cost. Usually, the test data generaton actvty s performed by a tool or tools whch partally or fully automate(s) the producton of the sought test data. The generated data must then be drven to the software whch s then checked n terms of ts behavour and valdty wth respect to the produced results. The usual way to evaluate the test thoroughness of a pece of software s to establsh a collecton of requrements that must be fulflled when the software s executed aganst test cases. Such a set of requrements can be certan test coverage crtera for example, that gude and evaluate the effectveness of the test data generated. Normally, the crtera selected to be fulflled are structural, derved by a herarchcal structure whch ncludes all the whte box crtera such as: statements, branches, data flow etc. Mutaton testng s a powerful fault-based testng technque ntroduced by Hamlet [1] and the DeMllo, Lpton and Sayward [2] and forms the focus of ths paper. Mutaton analyss nduces syntactcal alteratons of the code under test wth the am of producng mutant versons of the consdered code. Each program verson s called mutated verson and contans one smple syntactc change from the orgnal code. Test cases are used to execute these mutated versons wth the goal of dstngushng them from the orgnal one. A mutant s sad to be klled f there s a test that dstngushes ts output from the output of the orgnal program whereas, t s sad to be equvalent f there are not such dstngushng nputs. Assessng tests wth the kllng mutants rato s usually consdered as a measure of the qualty of the testng thoroughness. Fndng approprate test data accordng to a selected crteron can be very laborous [3] and fully or partally automatng ths actvty forms one of the major problems that nterest the software engneerng research communty. Unfortunately, ths argument holds for mutaton too. Most of the progress n the area has been reported by DeMllo and Offutt [4] n a technque called Constrant Based Testng (CBT) and also by Offutt, Jn and Pan [5] n the Dynamc Doman Reducton approach (DDR). Both attempts rely on the use of paths and symbolc evaluaton n order to construct sets of condtons under whch nputs should execute and nfect the program state of the consdered mutant programs. These approaches dffer n the way they solve the constrant sets and how they handle specfc program elements such as loops, arrays and complex expressons. Although experments have shown that both CBT and DDR are effectve,.e. more than 90% of mutants were klled n experments based on ether CBT or DDR approaches [4], [5], ssues concernng ther effcency and effectveness on consderable sze programs, stll reman. Generatng tests accordng to mutaton can be a bg burden because of the hgh number of the consdered mutants. Despte the progress recorded n reducng that number [6] the problem remans mostly unresolved and t s estmated to be proportonal to the product of the number of data references and the number of data objects [6] of the program under test. Addtonally, proportonal to the number of canddate mutants, are also the tme-consumng stages of: mutaton analyss, mutant generaton and executon phase. Thus, n order to reduce the overall effort spent durng

2 mutaton testng there s a need to generate a small number of tests that klls most of the consdered mutants at frst and then concentrate on a small number of the remanng lve mutants. The proposed strategy begns by producng a generc set of tests that kll a large proporton of mutants and then contnues on targetng ncrementally to each of the remanng lve ones. By dong so, the effort spent on mutaton analyss and on tryng to generate tests for equvalent and hard to kll mutants s reduced. The proposed approach reduces the kllng mutant s problem to a coverng branches problem. Treatng each mutant as a branch helps on focusng on specfc mutants and selectng canddate paths n order to generate data amng at kllng the specfed mutants. By usng path analyss the strategy tres to reduce the ncdence of nfeasble paths on the one hand, and to utlse most of the already selected feasble paths and thus reduce and balance the overall effort to kll each mutant on the other. The essence of our work can be summarsed as follows: Effcently select a proper subset of all paths as a canddate soluton to the problem of generatng mutaton adequate test data. Ths task can be qute dffcult and mpractcal f done n an ad hoc way. Heurstc approaches therefore that gude the selecton of canddate paths and result n feasble sets are needed n order to practcally employ mutaton testng. Therefore, the proposed strategy wll antcpate savngs n tme and effort requred by both the test data generaton and mutaton analyss actvtes. The rest of ths paper s organsed as follows: Secton II ntroduces some background materal. Secton III outlnes the path selecton strategy and secton IV detals the proposed approach. Secton V reports on the results obtaned through the applcaton of the proposed method. Sectons VI and VII dscuss some related to the present work and dscuss ts practcal use together wth some future drectons. Fnally n secton VIII conclusons are gven. II. BACKGROUND Testng crtera have the goal of selectng a subset of all possble test cases that have a hgh probablty of detectng errors. There are many types of testng requrements whch can be formed accordng to varous technques such as functonal, structural and fault-based. In general, structural testng crtera requre the examnaton of the nternal composton of the program s source code. Tests are derved from examnng certan program elements such as basc blocks, branches, paths etc., of the program under test. The crtera requrements goal s both to gude and evaluate the qualty of the test sets. Typcally tests are produced untl a predefned level of coverage s reached. The level of coverage accordng to a selected crteron s defned as the rato of ts test elements beng covered to the sum of the elements ntroduced. In branch testng for example, a path set consstng of feasble paths that cover all the program branches s chosen and tests that execute the selected paths are produced. As ponted n [7], [8], [9] the ncdence of nfeasble paths s responsble for the major part of the testng effort. Ideally, all testng strateges should ncrease ther effcency by selectng canddate sets wth a mnmum number of nfeasble paths. Testng based on fault based crtera, requres the exposton of some ntroduced faults. These crtera seed a number of faults nto the program s code and utlse the requrements for exposng them. The test coverage s defned accordng to the number of faults found by the test set to the number of faults ntroduced. Mutaton testng s a fault based testng technque that ntroduces faults by makng smple syntactc changes to the source code under test. The syntactc changes ntroduced are based on a set of rules called mutant operators. Accordng to the lterature, a seeded fault such as a mutant, n order to be exposed should satsfy three condtons known as Reachablty, Necessty and Suffcency [4], [10]. Based on these three condtons DeMllo and Offutt developed a test data generaton technque called Constrant- Based Test data generaton (CBT) [4]. The Reachablty condton states that the mutant statement must be exercsed wth test data. It must be noted that mutaton ntroduces one fault at a tme and all the program s executable statements apart from the mutated one are the same to the orgnal thus, the executon paths form the same executon computatons for both the orgnal and the mutated program versons. If tests can not execute the mutated statement, t s guaranteed that the tests have no chance of kllng the seeded mutant [4]. The Necessty condton states that the executon of the mutated statement must cause a dscrepancy of the orgnal program state [4]. In other words the outcome of the executon of both the orgnal and mutated statements must dffer. Otherwse because of the syntactcal equalty of the rest of the two program versons they wll never form dfferent computatons and never result n observable output dfferences. The suffcency condton states that the nfected program state must propagate up to the last program statement. Executon path and ts computatons must use the nternal dfferent value resulted at the mutated statement (necessty condton) and form a dfferent observable computaton from there onwards to the programs output. Current test data generaton approaches [4], [5] try to utlse drectly the reachablty and necessty condtons. The Suffcency condton because of ts hgh complexty s satsfed ndrectly through the satsfacton of the reachablty and necessty condtons. As found n [4], [11] tests that meet the reachablty and necessty condtons have a 90% chance of meetng the suffcency condton too. The proposed approach focuses on a drect handlng of the frst two condtons. The thrd condton (suffcency) s handled ndrectly through the fulflment of the frst two condtons and when needed through the selecton of alternatve paths. The proposed approach ntroduces an mproved strategy over the CBT method by posng practcal heurstcs and lmts on the test generaton methods. The necessty of employng a heurstc approach stems from the need to reduce the overall testng effort especally when usng a hgh demandng technque, such as mutaton. Tme and effort lmts are thus necessary to practcally perform such a testng actvty.

3 III. PATH SELECTION METHOD A. The extended shortest path method The problem of obvatng nfeasble paths durng the testng process was stated by Gabow, Maheshwar and Osterwel [12]. Predctng the nfeasblty of a program path a pror therefore by any means, must necessarly be heurstc. In an attempt to derve such a heurstc, Yates and Hennell [13] advanced, and argued the proposton that: A program path that nvolves q >= 0 predcates s more lkely to be feasble than one nvolvng p > q. Formal statstcal nvestgaton of ths proposton was undertaken n Yates and Malevrs [7], wheren t was concluded, wth great statstcal sgnfcance, that the feasblty of a path decays exponentally wth the ncreasng number of predcates t nvolves. As a result, Yates and Malevrs [7] proposed a path selecton method, extendng that of Yates and Hennell, to reduce, a pror, the ncdence of nfeasble paths amongst those that are generated for the purpose of branch testng. Although ntroduced to support branch testng, the method was founded only upon a consderaton of the number of predcates on a program path. Thus, the method does not seek to optmse, n any way, on any one testng crteron. Hence, t may be used wth equal valdty, and wthout bas, n an attempt to fulfl any other testng crteron. It s ths path generaton method that was adopted for the research reported n ths paper. The strategy for purposes heren can be detaled as: 1. Generate a set Π of program paths - each contanng a mnmum number of predcates - that cover the test elements of code unt. 2. Derve the percentage value of Coverage resultng from the feasble paths n Π. Whle Coverage<1 AND <k, repeat steps 3 and 4 once for each uncovered element e. 3. Generate the next canddate path Π e 4. If path found to be feasble, then recalculate the value of Coverage. Here, Π denotes the th shortest path through element e e and k = maxmum number of practcally vable paths (beyond whch the generaton of extra paths s prohbtve). The order of selectng the elements e at step 3 can be performed n varous ways. Here, they were selected at random. The optmal use of these two parameters (the order of selectng test elements, and the use of approprate k value) form a matter of future research. B. The method used to derve Π for branch testng e Each program path nπ s a shortest path, n terms of the number of predcates t nvolves, through some branch of the program, say C. The problem of generatngπ s thus that of generatng a set of shortest paths through C whch together cover all of C s branches. Moreover, f each arc j of G C, the Control Flow Graph (CFG) of C, s assgned a length W j such that: W j = the number of arcs emanatng from vertex - 1 * The requred path set can be dentfed as a set, Π, of shortest Start to Fnsh paths that covers the arcs of G C. (1) As n [13], Π, the shortest path from S to F n G xy C that passes through arc xy can be decomposed usng the prncple of optmalty as: (1) (1) (1) Π = Π ( S, x) xy Π ( y, F) (1) xy (1) Where Π ( p, q) denotes the shortest path from p to q for any par of vertces p and q, and denotes sequence n G C. In a smlar fashon, Π, the th shortest path n C through e a branch, can be expressed as: ( ) ( r) ( t) Π = Π ( S, x) xy Π ( y, F) (2) xy ( j) Where Π ( p, q) denotes the j th shortest path from vertex p to vertex q of G C, as s approprate, and r and t are ntegers satsfyng 1 r and 1 t ( r + 1), respectvely, and agan denotes sequence. IV. APPROACH DESCRIPTION A. Overvew Our approach manly reles on two observatons about the testng effort needed to acheve adequate or close to adequate mutaton coverage. The frst observaton ponts that there s a sgnfcant nfluence of the effort assocated wth mutaton based test data generaton and the number of test paths that must be generated n order to fulfl the reachablty, necessty and suffcency condtons of mutants as stated n secton II. Thus, the major part of the test generaton actvty s assocated wth the number of the selected paths that are found to be nfeasble. Effort, therefore, s depended upon the effectveness of the path generaton method/strategy n avodng nfeasble paths. Thus, any strategy whch does not seek to reduce the ncdence of nfeasble paths s non-practcal and leads to waste of effort proportonal to the number of consderable mutants. The second observaton comes from the applcaton of branch testng. Specfcally, the branch coverage crteron fulfls the reachablty condton of all mutants and the necessty and suffcency condtons for the majorty of them as presented n secton V. Moreover, for the majorty of the cases, there s no need to generate addtonal paths n order to acheve the combnaton fulflment of reachablty, necessty and suffcency condtons. Therefore, the addtonal effort

4 overhead to perform mutaton when branch testng, can be consderably reduced. Based on the above observatons, we propose the use of an effcent path selecton strategy n order to fulfl the branch coverage crteron frst and then ncrementally fulfl the necessty and suffcency condtons of the remanng mutants. The use of such an ncremental approach wll reuse effcently much of the nformaton ganed from the branch coverage necessty condton fulflment to the combnatonal fulflment of the necessty and suffcency condtons. The man dea of the proposed approach s to reduce the problem of fulflment of the necessty condtons of mutants to that of coverng branches. Ths s done by transformng the requred necessty constrants to branch predcates and representng them n the program control flow graph. The nnovaton of ths approach s the use of an enhanced control flow graph and ts respectve branch predcates n order to proceed wth the symbolc evaluaton and test data generaton phase. The beneft of dong so s the ncorporaton of mutaton based crtera to exstng path based test data generaton methods nhertng all ther merts. The proposed approach ams at program branches frst and consequently klls a larger number of the targetng mutants. Then the method tres to ncrementally cover the remanng lve mutants n turn. The reason for employng such an ncremental strategy s twofold. Frst, the branch testng exercse s a well studed problem and forms almost a unversal mnmum requrement for testng. Its nature gudes the test to a well formed subset of test requrements whch normally ncludes lmted nfeasble elements. Second, the test generaton process balances the effort between the varous test elements. Thus, the test strategy can stop at any tme resultng n a farly balanced level of testng thoroughness, havng exercsed proportonally all program elements, at the respectve tme. The proposed approach tres to mnmse the test generaton overhead caused by the nfeasble paths contaned n the program and by avodng trappng or spendng too much tme on equvalent mutants or too hard to kll mutants. B. The method used The representaton of the structure of a code unt by ts Control Flow Graph (CFG) s well known, understood and forms the bass of path testng for most of the whte box testng crtera. Unfortunately, ths s not so for mutaton, snce the graph structure does not represent or dentfy mutants. For mutaton a more elegant model s needed. Addtonally, mutaton testng requres a complete executon path n both the orgnal and the mutated program versons n order to cover-kll the concerned mutant. Followng the lterature suggestons about the weak mutaton hypothess [11] and by usng mutants necessty condtons [4], t becomes possble to buld a smple test model. By enhancng the programs CFG wth mutant related necessty constrants, a sngle graph representng-ncludng the varous mutants can be constructed. The enhanced graph s bult smply by addng a specal type of vertex for each mutant. Fgure 1. The Enhanced Control Flow Graph. Every mutant related vertex s connected wth ts orgnal correspondng node and represents the mutant related necessty constrant. Fgure 1 demonstrates the constructon of the enhanced control flow graph (the orgnal CFG on the left and the enhanced ncludng three mutants on the rght). The proposed approach test model s completed by addng weghts on the CFG nodes. The orgnal CFG nodes are assgned a weght value calculated as defned n secton III.B. The mutant related nodes are assgned an nfnte weght value n order to be gnored through the path selecton method. Recall that the selected canddate paths for a specfc branch are the k-shortest ones passng through that branch that gnore the nfnte weghted nodes. Snce a path not contanng any mutant vertex represents an exact path on the test code unt, that path can form the bass for the fulflment of mutant s reachablty condtons. Addtonally, every feasble path contanng one or more mutant vertces fulfls both the reachablty and necessty mutant condtons of the respectve mutants. Thus, coverng all enhanced control flow graph branches results n a straghtforward coverage of the weak mutaton crteron and the fulflment of the reachablty and necessty condtons of all program mutants. Based on the above observatons the proposed strategy tres to effcently cover all branches of the enhanced control flow graph. As ponted n [9] when testng accordng to one method targetng at specfc test elements there s a realsed coverage on the elements of another testng method. The realsed coverage s called collateral coverage [8], [9] and forms the bass of the proposed strategy. The strategy for present purposes can be detaled as: 1. Perform Branch testng for some k-value as descrbed n secton III. 2. Generate the Enhanced Control Flow Graph. 3. Derve the percentage value of mutaton coverage resultng from the collateral coverage of the branch tests. Whle Mutaton Coverage < 1 AND < k, repeat: steps 4 and 5 once for each uncovered mutant element m. 4. Generate the next canddate path m Π. 5. If path s found to be feasble then recalculate the value of Mutaton Coverage

5 The above method embodes the phlosophy of the Yates and Malevrs method [7] as t reduces the ncdence of nfeasble paths n generatng path sets to kll the mutants. In comparson wth the CBT [4] and DDR [5] methods, the proposed strategy uses a pathwse approach n order to overcome some of the techncal problems mposed by symbolc executon, such as loops and arrays. Addtonally as ponted n [14], t lmts wasted effort by avodng to symbolcally execute unnecessary program paths. As ponted by Coward [15] the use of path set selecton for gudng the symbolc executon process has many advantages over the tradtonal way of applyng symbolc executon to the whole program, and concludes that t s better to ntegrate a path selecton strategy wthn the symbolc executor to generate as few condtons as necessary to acheve a gven level of coverage. C. Incremental selecton of paths The path generaton method used for mutaton testng s n analogy to that used for testng branches. The sought path set conssts of the k-shortest program paths, agan n the sense of contanng a mnmum number of predcates, whch together cover all of the program s mutants. Agan wth the recourse to the prncple of optmalty, usng equaton (2), the k th shortest paths through a mutant vertex m and fulfllng the reachablty and necessty condtons can be defned as: Π ( k) m = Π ( r) ( S, x) xm mx Π ( t) ( x, F) ( j) Where agan Π ( p, q) denotes the j th shortest path from p to q for any par of vertces n the enhanced flow graph. Note that the path condton derved s based on a selected (k) path from the path set Π n conjuncton wth the necessty m condton related to mutant m. The strategy targets to weak mutaton and thus there s a possblty that generated tests may fal to also fulfl strong mutaton. To handle such stuatons the proposed strategy generates new paths that fulfl the necessty and suffcency condtons. All mutants are handled based on a k-value, the lmt on the number of generated paths, the strategy uses. The practcalty of the method reles on the selecton of an approprate k-value beyond whch all left alve mutants are treated as equvalent ones. However, the optmal choce and handlng of equvalent mutants based on the most approprate k-value s a matter of ongong research, see secton VII. D. The crteron for selectng uncovered mutants When there are more than one remanng uncovered mutant elements to be klled, the queston that s rased s: Whch s the order they should be selected n an attempt to kll them? In other words, whch mutant should we try to kll frst? The answer to the above queston s gven by usng the well establshed phlosophy that: a program path that nvolves q 0 predcates s more lkely to be feasble than one nvolvng p > q. Specfcally, the remanng lve mutants are classfed accordng to the length of the shortest path each one les on. Thus, the mutant that s selected to be klled s the one whose path s the shortest n length among the others. Tes are broken arbtrarly. V. PRELIMINARY EXPERIMENTS A. Expermental plannng Ths secton descrbes the results obtaned by applyng the proposed method to a set of 8 java program unts n an attempt to emprcally assess the effectveness of the proposed strategy. All the selected subjects have been used n prevous studes such as [4], [11] but were rewrtten n java for the purposes of ths paper. Table I presents the selected test subject detals. Test Object TABLE I. Descrpton SUBJECT PROGRAMS PROFILE Lnes of Code No. of Branches Mutants Equvalent Mutants Trangle Trangle classfcaton Newton Newton algorthm BSearch Bnary search algorthm Bubble Bubble Sort algorthm Eucld Eucldan algorthm Cal Calculate no. between two gven days Insert Inserton Sort algorthm Quad Quadratc formula Mutaton analyss was performed based on the fve mutaton operators proposed by the Offutt, Lee, Rothermel, Untch and Zapf [6] (.e. ABS, AOR, UOI, LCR and ROR) whch are found to have near equal strength as all mutaton operators prevously proposed [16]. The experments were performed usng the specfcaton of the above mutaton operators as set n [16]. All equvalent mutants were detected by manual analyss and elmnated from the test subjects n order to relatvely compute mutaton scores. B. Branch coverage and necessty constrant enhancement Ths secton reports results for the frst teraton of the proposed approach. The frst teraton stage conssts of employng the Yates and Malevrs method [7] to the selected set of programs for branch testng and usng the selected paths n conjuncton wth the mutant related constrants (mutant vertces). The experment examnes the mpact of performng branch testng and steps 4 and 5, of the algorthm n secton IV.B, once to every lve mutant. Table II reports on the number of feasble paths generated durng branch testng (Test Paths column), the branch tests generated (Branch Tests column), the tests resulted through the conjuncton of path condtons found durng branch testng wth the mutant related constrants, (Mutant Tests column) and the acheved mutaton scores for both the branch and

6 mutant test sets (M.S. Branch and M.S. Mutants column respectvely). The average mutaton score acheved for the experment sample s 81.94% for Branch and 90.2% for the Branch-Mutant test sets usng the same path sets. TABLE II. BRANCH AND MUTANT COV. WHEN BRANCH TESTING Test Object Test Paths Branch Tests Mutant Tests M.S. Branch M.S. Mutants Trangle % 94.4% Newton % 100% BSearch % 75% Bubble % 90.2% Eucld % 74.3% Cal % 96.0% Insert % 91.3% Quad % 100% Average % 90.2% M.S. used for Mutaton Score C. Leadng to hgher mutaton coverage levels Ths secton reports results from employng the proposed approach to the selected set of programs. Table III reports expermental detals of the proposed strategy (secton IV.B). The strategy for example appled on the trangle program at steps 1, 2 and 3 (performng branch testng), results n 8 paths and ther tests whch cover the 72.7% (table III M.S. Branch) of all non-equvalent mutants. Steps 4 and 5 when appled once to all lve mutants result n a mutaton score of 94.4% (table III M.S. Mutants) and form a 21.7% ncremental coverage on the mutaton score (table III M.S. Increment-Same Paths). In order to acheve adequate coverage the strategy generates 2 new paths (table III Path Increment) whch result n a 5.6% ncrement on mutaton coverage (table III M.S. Increment-New Paths), resultng n fully kllng all the non-equvalent mutants. The same apples to all of the programs used except Newton and Quad whch requred no extra paths as 100% kllng of the mutants was acheved wth the ntal test paths. TABLE III. Test Object Startng Paths DETAILS ON EMPLOYING THE PROPOSED STRATEGY Path Increment M.S. Increment (Same Paths) M.S. Increment (New Paths) Trangle % 5.6% Newton BSearch % 25% Bubble 4 2 2% 8.8% Eucld % 25.7% Cal % 4% Insert % 8.7% Quad % - M.S. used for Mutaton Score Fgure 2, presents the average mutaton score acheved per test case obtaned from the applcaton of the proposed strategy. The graph s a plot of the average mutaton score acheved for the 8 programs used aganst the ordered generated test cases. For example all the 5 generated test cases acheved on average 86.5% mutaton coverage. As t s apparent by lookng at the graph, a sgnfcant number of test cases are needed n order to fully kll all the lve mutants Fgure 2. M.S. acheved per test case VI. RELATED WORK Despte the usage of mutaton testng for a number of years, automated generaton of tests based on mutaton analyss has receved lttle attenton n the lterature. Most of the progress n the area has been carred out by DeMllo and Offutt n a method known as the Constrant Based Testng (CBT) Technque [4]. Ths approach uses control flow analyss, symbolc evaluaton [17], mutant related constrants and program executon n order to automatcally generate mutaton adequate test data. CBT s based on the observaton that a test case that klls a mutant must satsfy three condtons (reachablty, necessty and suffcency) as stated n secton II. The CBT technque has been mplemented n a tool named Godzlla for the test of Fortran programs and ntegrated wth the Mothra [16] mutaton testng envronment. Godzlla targets on the reachablty and necessty condtons and descrbes them as mathematcal systems of constrants. The reachablty condtons are descrbed by path expressons of all program paths that pass through a mutated statement. The necessty condtons are descrbed by specfc, to each mutant expresson, addtonal expresson(s) n order to nfect the program s state mmedately after the mutated statement. Godzlla conjons and tres to solve for each mutant ts reachablty and necessty constrants n order to produce some tests. In ths approach there s no straghtforward attempt to automatcally satsfy the suffcency condtons. Although CBT has been emprcally shown as an effectve technque yet t suffers from many shortcomngs assocated wth symbolc evaluaton. Specfcally, the handlng of arrays, loops, non lnear expressons and the path exploson are the man problems recognzed n [5]. In an attempt to address some of the drawbacks of the CBT method, the Dynamc Doman Reducton (DDR) [5] method was proposed. In DDR the man part of the CBT approach, the generaton of reachablty, necessty and suffcency constrants, remans, but enhancements have been made on the way these constrants are handled. The DDR method utlses the program control flow graph and based on a bnary search method over the nput varables doman, generates test data. In each branch predcate along a gven path the varables nvolved n the predcates are reduced so that the approprate

7 predcates are true for any assgnment of values from the doman. When all selected predcates have been examned, tests are produced based on the reduced doman set. The DDR approach treats the mutaton test generaton problem as a pathwse test data generaton problem. Its contrbuton reles on the generaton of data for a specfc path usng a search heurstc on the nput doman. Based on selected paths and usng a backtrackng mechansm Meudec [14] ntroduces an approach that ntegrates constrant logc programmng and symbolc executon. In ths attempt the benefts of the smplcty and back-trackng mechansms of constrant logc programmng are ntroduced and ths resulted n the development of a tool for the structural coverage of Spark ADA programs. Dynamc approaches based on searchng nput doman sets have also been proposed. In Sofokleous and Andreou [18], a framework that utlses two optmzaton algorthms s proposed, the Batch-Optmstc and the Close-Up for generatng structural tests. Ths approach uses a doman control mechansm whch starts wth small doman spaces and modfes ts boundares to larger ones at subsequent phases. In [19] a hybrd approach that ntegrates Genetc Algorthms as a search engne of the nput doman, a specal form of the DDR method on selected paths and symbolc executon based on the Yates and Malevrs method [7] was also suggested n the context of performng branch testng. The benefts of mutaton testng hghly depend on the number of mutants nvolved. Strateges nvolvng mutaton should therefore attempt to lmt the number of the mutants ntroduced on the one hand, and avodng to ntroduce equvalent ones on the other. Such an approach s proposed n [20] where the constructon of hgher order mutants s dscussed. In ths work t s suggested that the number of mutants and equvalent ones can be dramatcally lmted by ntroducng two or more mutants at a tme. A dfferent approach to heurstcally deal wth equvalent mutants s proposed n [21]. Accordng to the authors, dynamc nvarants are ntroduced nto the program under test. The mutants are assessed based on ther mpact wth the nvarants. By targetng to those wth a hgher level of mpact, a good measure of the adequacy of the test sute s establshed, whle lmtng the number of consdered mutants and the equvalent ones. However, both of these approaches rely on mutaton analyss rather than on generatng test data. VII. DISCUSSION AND FUTURE WORK The proposed method tres to effectvely generate test sets by targetng on one of the most promsng testng crtera. As suggested n [22] mutants can provde a good ndcaton of the fault detecton ablty of a test sute and that mutants tend to be smlar to real faults. Amng at a fault based crteron such as mutaton, results to be a qute dffcult and effort demandng task [3]. Thus, there s a need for practcal, effectve and systematc technques whch lmt the effort spent on the data generaton and mutant executon phases. The proposed approach, heurstcally deals wth the frst part of generatng test data by smplfyng and gudng the process n order to select tests wth a lmted waste of effort. The approach smplfes the problem of selectng test data by reducng t to generatng test data accordng to a selected path as ponted by Coward [15] and Meudec [14]. Addtonally, the proposed method n the present paper, lmts the waste of effort by reducng the effects of nfeasble paths as stated n secton IV. Targetng on practcal ssues of mutaton testng there s a need for heurstc approaches to automatcally reduce and bypass the equvalent mutants problem [3]. As stated by Yates and Malevrs [7] t s unlkely to be cost-effectve to generate more than (about) the frst 40 shortest paths through a branch n order to cover t. Although ther clam needs to be valdated for the case of mutants, t s unlkely to be generally nvald. The ncremental nature of the proposed approach results n cost-effectve solutons not only n terms of tme but also n terms of effort assocated wth each mutant. That way the effort spent on tryng to kll equvalent or hard to kll mutants s balanced and reduced accordng to the strategy taken. In vew of ths, a nave heurstc approach s to treat each lve mutant as an equvalent one after the k tres of the proposed method. As presented n secton VI there s a lmted number of automated tools for generatng tests accordng to mutaton analyss. The method presented n ths paper can be easly ncorporated nto an automated tool towards ths effect. Effort s made to ths drecton. Future work s drected towards conductng more experments n order to statstcally asses the effort, n terms of the paths requred to generate mutaton test data. A seres of experments s also planned n order to determne the optmal k-value n a cost effectve strategy. Experments consderng some large programs are also planned. VIII. CONCLUSION Ths paper presents a practcal approach to generate mutaton adequate test data. The proposed approach transforms the problem of kllng mutants to a path based test data generaton problem and apples an effcent path selecton strategy n order to acheve adequate coverage. The presented strategy targets on generatng mutaton adequate test sets n an effectve way. Ths s acheved by a strategy that reduces the effects of nfeasble paths and equvalent mutants. The employed strategy not only seeks to mnmse the test data generaton actvty but also tres to balance the effort nvested on kllng each mutant by stoppng the data generaton engne based on cost overhead lmts. Transformng the problem of mutaton nto a path based problem has the advantage of usng exstng path analyss strateges and tools. Thus, automated tools and testng strateges could be easly appled to fault based testng and mprove ts effectveness. Such a strategy wth an approprate adaptaton s the one presented here. A prelmnary expermentaton and assocaton of the effort needed wth the number of test paths used for branch testng was performed. Results from some benchmark programs were reported along wth an assocaton on the effort needed to acheve mutaton coverage wth branch testng. The results were very encouragng, as they showed that the ntal path set obtaned from the branch testng, on the eght programs used, provded a very good ntal path set for

8 kllng a good proporton of the mutants straght away, whereas only a small number of extra paths was necessary to be consdered n order to fully kll all the mutants for the sx of the eght programs. Addtonally, for the remanng two programs, the ntal path set was capable to generate tests that adequately cover all the programs mutants. Ths can be seen as the ablty of the proposed method to kll the majorty of the selected mutants, as one can contnue generatng extra paths untl such a task s fulflled. However, and as ths was dsplayed by the approprate graph, the effort nvolved n ncreasng such a coverage s not always advsable. Although the approach reles mostly on the weak-frm mutaton hypothess [11] t s clearly an effectve and practcal soluton n order to generate hgh qualty test sets. REFERENCES [1] R. G. Hamlet, Testng program wth the ad of a compler, IEEE Trans. Softw. Eng., vol. 3, 1977, pp [2] R. A. Demlo, R. J. Lpton, and F. D. Sayward, Hnts on test data selecton: Help for the practcng programmer, IEEE Computer, vol. 11, 1978, pp [3] A. J. Offutt, R. H. Untch, Mutaton 2000: Untng the Orthogonal, In Mutaton 2000: Mutaton Testng n the Twenteth and the Twenty Frst Centures, October 2000, pp [4] R. A. Demlo and A. J. Offutt, Constrant-Based Automatc Test Data Generaton IEEE Trans. Softw., Eng., vol. 17, 1991, pp [5] A. J. Offutt, Z. Jn and J. Pan, The dynamc doman reducton approach to test data generaton, Softw. Pract. Exper., vol. 29, 1999, pp [6] A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, C. Zapf An expermental Determnaton of Suffcent Mutaton Operators, ACM Trans. Software Eng. Method. vol. 5, 1996 pp [7] D. F. Yates, N. Malevrs, Reducng the effects of nfeasble paths n branch testng, ACM SIGSOFT Software Engneerng Notes, vol. 14, 1989, pp [8] N. Malevrs, D. F. Yates, The Collateral Coverage of data flow crtera when branch testng, J. of Informaton & Software Technology, vol. 48, 2006, pp [9] D. F. Yates, N. Malevrs, An objectve comparson of the cost effectveness of three testng methods, J. of Informaton & Software Technology, vol. 49, 2007, pp [10] A. J. Offutt and J. Pan, Detectng equvalent mutants and the feasble path problem, Softw. Test., Verf. Relab., vol. 7, 1997, pp [11] A. J. Offutt and D. S. Lee, An Emprcal Evaluaton of Weak Mutaton, IEEE Trans. Software Eng., vol. 20, 1994, pp [12] H. N. Gabow, S. N. Maheshwar, and L. J. Osterwel, On two problems n the generaton of program test paths, IEEE Trans. on Softw. Eng., vol. SE-2, 1976, pp [13] D. F. Yates and M. A. Hennell, An approach to branch testng In: 11th Workshop on Graph Theoretc Technques n Computer Scence, Wurtzburg, West Germany 1985, pp [14] C. Meudec, ATGen: Automatc Test Data Generaton usng Constrant Logc Programmng and Symbolc Executon, J. Software Testng, Verfcaton and Relablty, vol. 11, 2001, pp [15] P.D. Coward and D. Ince, The symbolc executon of software: The SYM-BOL system, 1995, London: Chapman & Hall. [16] R. A. DeMllo, D. S. Gund, W. M. McCracken, A. J. Offutt, K. N. Kng, An extended overvew of the Mothra software testng envronment, In proceedngs of Symposum on Software Testng, Analyss and Verfcaton 2, July 1988, pp [17] J.C. Kng, Symbolc executon and program testng, Comm. ACM vol. 19, 1976, pp [18] A. A. Sofokleous, A. S. Andreou, Automatc, evolutonary test data generaton for dynamc software testng, J. of Systems and Software, vol. 81, 2008, pp [19] M. Papadaks and N. Malevrs, Improvng Evolutonary Test Data Generaton wth the Ad of Symbolc Executon, AIAI nd Artfcal Intellgence Technques n Software Engneerng Workshop (AISEW 09), Aprl [20] M. Polo, M. Pattn, I.G. Rodrguez, Decreasng the cost of mutaton testng wth second-order mutants J of Software Testng, Verfcaton, and Relablty, vol. 19, 2009, pp [21] D. Schuler, V. Dallmeer, and A. Zeller, Effcent mutaton testng by checkng nvarant volatons. In Proceedngs of the Eghteenth nternatonal Symposum on Software Testng and Analyss (ISSTA '09), July 2009, pp [22] J. H. Andrews, L. C. Brand, and Y. Labche. Is Mutaton an Approprate Tool for Testng Exeperments? In Proceedngs of the 27 th Internatonal Conference on Software Engneerng (ICSE 05), May pp