RCA on Residual defects - Techniques for adaptive Regression testing

Transcription

1 RCA on Residual defects - Techniques for adaptive Regression testing

2 Abstract Oneoftheimportantissuesassociated withasystem lifespanview thatwehaveignoredinpastyearsisthe efects of enduring defects defects that persist undetected acrossseveralreleasesofasystem.many studiesperformed to date have evaluated regression testingtechniquesunderthelimitedcontextsuchasshort term assessmentwhich do notfuly accountforthe industrybasedsolutions. Reportsestimate thatregression testing consumesas much as80% ofthe overaltesting budgetand can consumeupto50% ofthecostofsoftwaremaintenance. Researchershavedeveloped techniquesto addressthe numberofissuesrelated to regression testing.in this paperwehaveconcentratedonthethreeareas. First, techniquesthatcanhelptoreusethetestsuitescreated foronebuildandcategorizingthosetestcasesthatneed toberepeatedfortestingsubsequentreleasesandthe strategyforefectiveorderingforrunningthetestcases. Secondly techniques that wilreduce the regression testingtimebycreatingefectiveregressiontestsuites thattestthealteredmodules,byidentifyingtestcasesin theregressionsuitethatarenotneededtorerunonthe new buildandremovingobsoletetestcases. Finaly,techniquestorecovertestcasesbyidentifying, manipulating,andtransformingobsoletetestcases,by writingnew testcasesfrom oldones,andbyrepairing/ rewritingtestcaseswhenthesoftwareevolves. 2

3 Overview Software products undergo several processes like design,development, testingetc.beforetheygetintroduced into the market.even afteralthe phases,therearechanceswhereone ormoredefectspersistundetectedor missedduringthetestingphasefora longtimeandgetsleakedouttothe customers.when these defects are encountered by the customers this ultimately becomes a failure.such defectsaretermedresidualdefectas onemightknow. Withthesuccessoftesting,measured by the number of such Residual defects, the only solution being regression,how doesone go about doingitinthemostefectiveway? RootCauseAnalysisisperformedon those ResidualDefects to find the actualcauseofthefailure,thearea thatwasmissed out,the area that requiresmoreimportance,reasonfor failureetc.afterthisinvestigation,the result/inferencecanfeedinasthe scopeforregressiontesting.adaptive regression techniques can then be appliedtomaketestingandinturnthe productsuccessful. ResidualDefect Residualdefectisanimportantfactor thatdecidesthequalityofareleased product.incommon,itisimpossibleto findalthedefectswithinaperiodof time.residualdefectsmightafectthe successofthesoftwareproductsbased oneachorganization sbusinessgoals andthetypeofsystem.thiswilleave behindaverydissatisfiedcustomer. Post-release defect density ofa software product cannot be measureduntilithasbeenputinto production and used by the customers. This information becomesavailabletoolatetotake corective actions to software quality.softwaredefectscorection is significantly more expensive whenthedefectsareidentifiedby thecustomercomparedtodefects corection in the earlier developmentprocess. ResidualDefect Density(RDD) RDD is used to measure the Percentage ofapplication defects found in production.itinfluences thequalityoftheendproductto the customers. RDD gives the percentageofdefectsleakedtothe enduser. Formula TotalNoofPostProduction Defects TotalNoofTestCases Executed Aspertheformula Example =20 =250 =20/250*100 =8% (Ratio=0.08) Intheaboveformula, TotalNoof PostProductionDefects refersto althedefectsfoundafterdeploymentbutmissedduringtesting.residualdefectdensityisa Lower thebeter metric.itevidencesthe EfectivenessofTesting. 2

4 TheneedforRCA BasicRCA (RootCauseAnalysis)can often be extremelyiluminating.for example,onemightbeabletoariveat reasonsfordefectsandwilbeableto categorizethem asenvironmentissue, InsuficientTestCoverage,DataIssue, Oversight of Requirements, InadequateInstruction,etc.Itwould alsobeadvisabletomapdefectswith theirrespectiveareas. RCA,thoughagenericterm maynot havehardandfastrulesofaspecific template orstandards.thiscan be customisedinawaythatwilhelpthe tester arive at solid inferences / solutionsthatwilhelp in the next stages of defining your regression tests.thiswilin turn help reduce Residualdefectsthatare leaked to customers. EfectiveRCAdocumentationcanalso serveasoneofthevaluabletestartefactsandcanbebundledtogetheras partofthe testsummary /results report. Root Cause Analysis (RCA) is addressingcausesratherthantreating thesymptoms.ifsomethingisbroken, insteadofjustfixingitatthepointof discovery,itisbeterto investigate and fixtheunderlying causeatthe point of origin itself. Careful applicationofrcacanuncoverserious inadequaciesin software testing.as quoted-thecauseofdefectscanbe sketchedtotheoriginalrequirements, thedesign,thecodeimplementation, theverification,thetestplanning,or eventhefinalqaitself.byaddressing theissueatroot,wecandrasticaly improvethefinalsoftwareandsave moneyatthesametime.wecan alsofixfundamentalproblemswith theprocessesthatwilbenefitnot only the curentproject,butal futureprojectsaswel. Afew knownrcatechniques: Fish-bonediagram Orthogonaldefectclassification 5W2H RegressionTesting Asdiscussedinearliersections,the outcomeoftherca wilserveas thebasisforyourregressionsuite. Forexample,ifyoufindaspecific areatobemostbugsprone,you canpicktherelevanttestcasesfor regression when you form your suite. Basicaly,therearetwosoftwareregressiontestingstrategiesfolowed byorganisations.thefirststrategy is complete regression where al previoustestcasesare executed again.thesecondstrategyispartial regressionwheresoftwarechanges areanalyzed qualitativelyand affectedpartsareretested.themerit ofusingthefirststrategyisthatthe afectedpartsarefoundasmuchas possible,andthedemeritisthatthe organizationcannotmeettheexpensespentontimeandresources needed as itis very huge.the second strategyhasthe meritof lesscostandthedemeritispooraccuracyofqualitativeanalysisand theinadequacyofthetest. 3

5 ProposedProcess Traditionalmethodsmayhaveitsown advantages and disadvantages. A generalapproachtowardsregression testingwasonlytofocusontestcases. Normaltestersmaythinkthatresults wilalwaysbepositiveandlessnumber ofdefectscapturedduringthisphase. Anotherperspectiveisthatrepeated teststepsare executed and testers wouldevenskiponsuchteststeps. In our exclusive 3R approach Residual,RCAandRegressionwithits own techniques we have tried to tweakthetraditionalmethodstosuit ourpracticalneeds. AssimilationofSME MechanizedChecks InnovativeTests By folowing these approaches and techniqueswewereabletodelivera qualityproductwithmorefocusonthe application as a whole. Tester s perspective towards Regression testing haswidely changed.in this paper,ourresultsshow thatthisisthe bestpractice to reduce the residual defectrate. 4

6 AssimilationofSME PerformingRegressionTestingonalmodulesofanapplicationisatedious processandlotsofhoursgetwastedwhenmanyresourcesinvolveinthesame modules.thusweincorporatedandassignedsmetoeachmoduleofthe applicationwhowiltakegoodcontrolofit.eachofthem wilbeindulged deeplyinmonitoringtheapplication. TheSMEwilberecordingalthechangesofthemoduleswithtimeandmaintainareport.Thiscanberecoveredonneedbasis.Anynew featureintroduced orchangesmadeinthemoduleisindentifiedandrecordedinstantly.thedata colectedisupdatedinawel-timedmanner. Eachrecordcolectedfrom thesmeissharedwithintheteam.asaresult,each oneofthem isawareofthelatestupdatesandchangesinthemodulesofthe application. Clarificationsregardingthemodulesinspecificorknowledge transfercan beobtained from them.theseconsolidated SMEreportsare valuabletestevidenceswhenconvertedintochecklists. ByalotingeachSMEforamodule,wereducethetimespentonregression testingandrca ofanynew issueisidentifiedeasilybythesme.thedata colectedbythesmecanbereusedinfutureaswel. 5

7 MechanizedChecks Mechanized Checks is another approachthatweadoptedtosavetime byusingthereuse,reduceandrecover approaches.ratherthan writing the samescriptsagain,itisalwaysbeter toautomatealthetestcasessothatit can be reused later from the maintainedrecords.itisalsoeasyto make changes when necessary by alteringjusttherequiredsectionofthe scriptinstead ofwriting the entire scriptswhichsavesvaluabletimeand manpower by just recovering new scriptsfrom theoldones. Mechanized Checks approach when used efectively results in a beter qualitybyusing the3r approach.it also reduces the cost spent on resourceseventualy. Mechanized Checks has it big advantage when its reports are designed withcare.theinitialefort putintodesigningthesereportswil realy pay ofasitisgoing to be consistentandprecise.whatmoretest evidence than an execution report! These are the reports that the customer is finaly going to be ininterestedin. MechanizedChecks Itisvirtualyimpossibletowritetest casesthatcovereveryscenario that canoccur.touncoverhiddenrisks,you needtogoofscript,andthatiswhere usinginnovativetestscanhelp. Letussayaclientwantshislegacy application revisited buthaveno formal documentation, testers could employexploratorytesting tolearnhow thesystem curently works,and simultaneouslygather informationfornextsteps. InInthisphaseasuggestionwouldbe conducting yourinnovative Tests not completely adhoc.folow a scenariobasedchecklistthatcanbe easilyreviewedbytheclient.once thesearefinalized,youcanconvert them toformaltestcases.thiswil reducebackandforthreview communication drasticaly.since this was not done adhoc and a documentation process was folowed,thiscanbereusedinthe designphase.whileinformationis colected,itisalsowisetocapture asmanypictorialreferencesthat canberecoveredfrom acommon storagepoint.thiswilhelpwhen youwanttogobackversionsfor crcrossverification.notonlydothese scenario based checklist and pictorialreferencesgoingtoserve ascheckpointsbutarealsogoing tohelpbuildboththetestplanning forthenextstagesand thefinal reportsofsummary.thesereports helpthecustomerhaveasafeand convinced feeling aboutwhatis covered and wilgive them an opportunitytowidenthescopeif theyseeaneed. 6

8 Traditionalytestexecutionswerefocused onlyonscriptsthatlimitedthescopeof testers.wheninnovativetestsisfolowedit wil let testers to think out of box. InnovativeTestsresultsinanalmostwhole testcoverageandthecountofresidualdefectsgodown. WeusedInnovativeTestsforthecurent yearandcomparedtheresidualdefectrate withlastyear.datacolectedwereanalyzed and results were obtained.our results clearlyshowedthatresidualdefectratehas decreased rapidly due to the implementationofinnovativetests. Casestudy ProjectType :Javabasedwebapplication Resources :Atotalof15resourcesemployedfortesting TestType :Testscriptsareautomatedandalsorunmanualy ReleaseFrequency :Monthly NumberofTestModules :250(Foreachcycle) Resourcesare primarily categorized into Manualtestersand Automation testers.manualtestersarefurthergroupedintotestersusinginnovativetests& SMEs.Buildfrequencyis5to6permonth.Nearlyaround2500testcasesare presentinthelibraryfortheapplication. GeneralChalenges Minimizere-testingefortsandachieving100% testingcoverage Identificationofsoftwarechangesanditsimpacts Identificationofafectedsoftwaretestcases Reducingthere-testsuites Wiseselectionoftestcasesfrom testsuiteforregressiontesting 7

9 Metrics GeneralChalenges Figs.1,2,3and4show,how thedefectsleakedtocustomershavegradualy decreasedafterimplementingthe3rapproachalsofig.2showshow the curvehasgradualygonedownfortheresidualdefectswhichisagoodsignresultinginasatisfiedcustomer.fig.4 6% lessdefectshaveleakedintoproductionafterimplementingthe3rapproach. Fig1 Fig2 8

10 Fig3 Fig4 DefectDistributionusingInnovativeTests Fig.5shows,how thedefectratehasincreasedbyperforminginnovativetests thantheusualtestcasebasedtesting.thereisidealyabout18% moredefects detectedintheinnovativetestapproach. Fig5 9

11 DefectDistributionTestervs.SME Fig.6shows,defectsidentifiedbytheSMEsarehigherbyabout17% compared totesters.threevitalconceptsofsmeinourprocessareformalscenariodoc- umentationasavalueaddition,widerscopeandsuggestionsonenhance- Fig6 Conclusion Onusingthe3Rapproachfortheregressiontesting,weobservethatthereis anefectivechangeoverintheresults.italsosolvesthechalengesthatare imposedbycontinualregressioncycles.duetotheinclusionofsmeconcept, theendresultnotonlyeliminatestheresidualdefectbutalsoprovideavalue additionintheform offindingoutofscopeissues,suggestionstoimprovethe product,etc.italsoprovestobeabestpractiseforregressiontestingenabling maximum testcoverage and reducing timeline issues.therefore aim for delightedcustomerswithalthepossibletestevidenceswithour3rapproach! 10

12 AuthorBiography NareshJayabalancomeswith4+yearsofexperienceinsoftwaretestingandhe isintocompatibilitytestingprojectformorethan4years.heholdsab.techin InformationTechnologyfrom S.A.EngineeringColege,ChennaiandaMBAin HumanResourceManagementfrom UniversityofMadras.Hestriveshimself intoanalyzingandexperimentingdefectswhichhasleadthewaytothispaper. Hespendshisidletimeinlisteningmusic& freaksouttoadventureplaces. SenthilKumarancomeswith2+yearsofexperienceinsoftwaretestingandhe isfrom thecasteam andhandlingwithcompatibilitytestingandfunctional testingproject.heisalsopartofhandlingthesystem instalationandissue fixing.heholdsamasterofscienceininformationtechnologyatthenew Colege,Chennai.Hehascontributedmanypresentationsinternalywhichlead thewaytothispaper.hishobbiesareplaying& watchingcricket.heisinterestedindatabasetesting. KeyTakeaway Efectivetestdatamanagementisanessentialprocessforcost-efective continuoustesting. 11

13 AboutIndium Software AtIndium Software,we vebeenentrenchedintheworldofsoftwaretestingsince1999. We vebuiltateam of450+ softwareandtestprofessionalsinouroficesinchennai, Bengaluru,New Jersey,Sunnyvale,LondonandKualaLumpur. The core of Indium s objective to servicing our global customers can be explainedwiththissimpleline: We resmalenoughtocare,largeenoughtodeliver. We areapreferedtestingvendorforenterpriseandisvcustomersrangingfrom Fortune100 to5000companiesandsmaltomedium enterprises. Tildate,we veservedover250clientsintheu.s.,andrestoftheworld. ContactUs USA SUNNYVALE Suite 210, 1250 Oakmead Parkway Sunnyvale, CA Phone: +1(408) Fax: +1(408) ATLANTA Crown Office Suites 1870 The Exchange Suite 100 Atlanta, GA Phone: +1 (770) PRINCETON Carnegie Center Suite 150, 300 Carnegie Center Princeton, NJ Phone: +1 (609) UNITED KINGDOM LONDON Indium Software Shelton Street London, WC2H 9JQ. INDIA CHENNAI No.64 (Old N.143), Eldams Road Ganesh Chambers Teynampet, Chennai Phone: BENGALURU No.100, Kay ARR Royal Stone Tech Park, 5th Floor, Pai layout, Benniganahalli, Bengaluru, Karnataka Phone: MALAYSIA SALES INQUIRIES GENERAL ENQUIRIES KAULA LUMPUR Suite 8-1 & 8-2, Level 8, Menara CIMB, No.1, Jalan Stesen Sentral 2 Kuala Lumpur Phone: +60 (3) Fax: +60 (3) americas.sales@indiumsoft.com apac.sales@indiumsoft.com emea.sales@indiumsoft.com india.sales@indiumsoft.com sales@indiumsoft.com careers@indiumsoft.com info@indiumsoft.com 12