Evaluation and Comparison of Different Machine Learning Methods to Predict Outcome of Tuberculosis Treatment Course

Transcription

1 Journal of Intellgent Learnng Systems and Applcatons, 2013, 5, Publshed Onlne August 2013 ( Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course Sharareh R. Nakan Kalhor 1,2*, Xao-Jun Zeng 3 1 Department of Publc Health, School of Health, Ahvaz Jundshapur Unversty of Medcal Scences, Ahvaz, Iran; 2 Socal Determnants of Health Research Center, School of Health, Ahvaz Jundshapur Unversty of Medcal Scences, Ahvaz, Iran; 3 Department of Machne Learnng & Optmzaton, School of Computer Scence, Unversty of Manchester, Manchester, England. Emal: * nakan.sh@aums.ac.r Receved December 24 th, 2012; revsed January 24 th, 2013; accepted February 4 th, 2013 Copyrght 2013 Sharareh R. Nakan Kalhor, Xao-Jun Zeng. Ths s an open access artcle dstrbuted under the Creatve Commons Attrbuton Lcense, whch permts unrestrcted use, dstrbuton, and reproducton n any medum, provded the orgnal work s properly cted. ABSTRACT Tuberculoss treatment course completon s crucal to protect patents aganst prolonged nfectousness, relapse, lengthened and more expensve therapy due to multdrug resstance TB. Up to 50% of all patents do not complete treatment course. To solve ths problem, TB treatment wth patent supervson and support as an element of the global plan to stop TB was consdered by the World Health Organzaton. The plan may requre a model to predct the outcome of DOTS therapy; then, ths tool may be used to determne how ntensve the level of provdng servces and supports should be. Ths work appled and compared machne learnng technques ntally to predct the outcome of TB therapy. After feature analyss, models by sx algorthms ncludng decson tree (DT), artfcal neural network (ANN), logstc regresson (LR), radal bass functon (RBF), Bayesan networks (BN), and support vector machne (SVM) developed and valdated. Data of tranng (N = 4515) and testng (N = 1935) sets were appled and models evaluated by predcton accuracy, F-measure and recall. Seventeen sgnfcantly correlated features were dentfed (P <= 0.004; 95% CI = ); DT (C 4.5) was found to be the best algorthm wth %74.21 predcton accuracy n comparng wth ANN, BN, LR, RBF, and SVM wth 62.06%, 57.88%, 57.31%, 53.74%, and 51.36% respectvely. Data and dstrbuton may create the opportunty for DT out performance. The predcted class for each TB case mght be useful for mprovng the qualty of care through makng patents supervson and support more case senstve n order to enhance the qualty of DOTS therapy. Keywords: Tuberculoss; Machne Learnng; Predcton; Classfcaton; DOTS 1. Introducton Over recent years, Tuberculoss has been consdered, partcularly n low and mddle-ncome countres as a global publc health concern wth the estmated two mllon deaths annually [1,2]. Up to 50% of all patents wth TB do not complete treatment and fal to adhere to ther therapy [3]. It has been estmated that n ndustralzed countres non-completon of treatment s around 20% [4] and accordng to the Centre of Dsease Control and Preventon n the Unted States, 25% of patents fal to complete ther chemotherapy [5]; n other words, the proporton of patents wth actve dsease who completes therapy under standard condtons ranges from as lttle as * Correspondng author. 20% - 40% n developng countres to 70% - 75% n the USA [6]. Noncomplance s a sgnfcant factor leadng to the persstence of tuberculoss n many countres and the consequences of ths well recognzed fact are prolonged nfectousness, relapse, prolonged and more expensve therapy due to multdrug resstance TB and death [7]. It has been revealed that noncomplance s assocated wth a 10-fold ncrease n the ncdents of poor results from treatment and accounted for most treatment falures [8]. It s the most serous problem hamperng tuberculoss treatment and control as patents wth actve dsease who are non-complant, sputum converson to smearnegatve wll be delayed and relapse rates wll be 5-6 tmes hgher and drug resstance may develop [6]. That s, TB patents remaned n the pool of actve cases wll n- Copyrght 2013 ScRes.

2 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course 185 crease the development of TB among latent cases who are prone to be nfected or affected. Ths threats publc health and makes huge costs that can be spent to mprove publc health mprovement and promoton nstead. DOTS (drectly observed treatment, short course) whch s current nternatonal control strategy for TB control nvolves the case detecton and completes the entre course of treatment successfully. In 2006, n order to mprove DOTS qualty, World Health Organzaton (WHO) has desgned Stop TB Plan [2]. In ths plan, t has been hghlghted that health care servces should dentfy and concentrate on nterrupton factors that halt TB treatment. Supervson whch plays mportant role n patent treatment adherence and drug resstance preventon must be carred out n a context-specfc and patent-senstve manner. Although WHO has hghlghted the necessty of mprovng the qualty of DOTS n terms of supervson and patent support n the Stop TB plan, there s no specfc way to measure how ntensve health provders support and supervson should be and to whom of TB cases t should be provded. To make ths clear, we may requre a tool to predct the patent destnaton regardng TB treatment course completon. The tool may dentfy hgh rsk TB patents for treatment course non-complance as t s mpossble to serve all TB patents wth actve supervson and support due to the cost consderaton and lmted resources. Ths may be used to defne the level of supervson and support each patent needs based on the predcted outcome by an accurate predctve model. Currently, no system s avalable to estmate TB treatment course through usng features of TB patents and desgnng a systematc method to predct the gven outcome. For the predcton of tuberculoss treatment course completon, the defned outcome related to each record of TB patent contans fve potental classes: cure and completed treatment (desrable outcomes), falure, qut, and death (undesrable outcomes) [9]. Patents who are classfed wth undesred outcomes need more supervson and support. Publc health nterventons for TB control need to be set accordng to how many of TB patents would shft to unwelcome outcomes. Makng sure that TB patents fnsh the treatment course entrely s a man step to TB control and publc health promoton. Actually, here the man queston of concern s can new TB cases at rsk of falng n treatment course completon dentfed from ther early regstraton? For ths purpose, machne learnng methods have been already appled and they worked properly accordng to prevous studes [10,11]. They nfer a model through beng traned by a set of hstorcal examples n frame of data categores; thus, new examples wth unknown classes could be assgned to one or more classes by pattern matchng wthn the developed vald model [12]. Ths work used machne learnng algorthms to acheve sx predctve models of a crucal real-world problem whch offer the medcal research communty a vald alternatve to manual estmaton of rsky TB patents to qut or fal treatment course completon. Ths recognzes TB patents whom supervson n frame of DOTS therapy needs to be more actve as they are rsky for treatment course nterrupton. 2. Methods and Algorthms 2.1. Correlaton Analyss In the case of a large dataset, learnng the dataset s not useful unless the unwanted features are removed snce an rrelevant and redundant feature does not add anythng postve and new to the target concept [13]. To dentfyng nfluental predctors, a bvarate correlaton s used whch s a correlaton between two varables ncludng ndependent and dependent parameters by calculatng the correlaton coeffcent r [14]. As there s no specfc predcton and hypothess two-taled test s carred out. Suppose we have got two varables X (lst of features) and (class), an optmal subset s always relatve to a certan evaluaton functon. To dscover the degree to whch the varables are related, correlaton crtera are appled [15]. It reflects the degree of lnear relatonshp between two varables rangng from +1 to 1. A perfect postve lnear relatonshp between varables s shown by +1; however, 1 mples an entre negatve lnear assocaton. The followng formula (1) s used to calculate the value of r: r n 1 X X n1 S S where there are two varables X and and ther means X, and standard devatons ncludng S X and S respectvely; n s the number of TB nstances. The correlaton coeffcent can be tested for statstcal sgnfcance usng specal t-test through followng formula. x (1) t r n r (2) Degree of freedom for correlaton coeffcent calculaton s equal to n 2. From a t-table, we would fnd sgnfcant relatonshp between each of varables X and (P < 0.05) Problem Modelng To acheve the am of ths work, we evaluate a number of well-known classfcaton algorthms on TB treatment course completon. In fact, predcton can be vewed as Copyrght 2013 ScRes.

3 186 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course the constructon and use of a model to assess the class of an unlabeled sample, or to evaluate the value or value ranges of an attrbute that a gven sample s lkely to have [16]. To tran every of appled algorthms, we set a bg matrces as follows: 1 17 x1 x 1 y 1 X : : : : (3) 1 x x y where X s a bg matrce whch s used to tran the gven algorthm by usng tran set n whch would be the th number of samples and s the th number of predctor. x n our tranng set would be x and n appled testng set has been x whch s the dependent varable of TB treatment course n both tranng and testng sets addresses fve classes where a label of 1 mples that TB case got cured and 2 means that s/he has completed the treatment course entrely; whlst a label of 3, 4, and 5 means that the patent belongs to the undesrable class such as quttng, falng the treatment course, or dead respectvely. Havng compared the pros and cons of machne learnng methods, to carry out the consdered predcton task n ths study sx classfers ncludng decson tree (DT), Bayesan network (BN), logstc regresson (LR), multlayer perceptron (MLP), radal bass functon (RBF) and support vector machne (SVM) were selected. Next secton s bref explanaton about each of these selected classfers. As shown n Fgure 1, the dataset was splt Whole database [6,450*18] Tranng set [4,515*18] DT SVM LR BN Testng set [1,935*18] Data Learnng & Model Buldng RBF Checkng Model Accuracy& Valdty ANN Fgure 1. Schematc presentaton of appled methodology of model development and valdaton process. In order to develop accurate predctve or classfcaton models, frstly data of TB patents (n a huge matce wth 6450 rows and 18 vectors) are dvded as tranng and testng sets. Tranng set s used to learn data and buld models through applyng 6 algorthms ncludng decson tree (DT), support vector machne (SVM), logstc regresson (LR), Bayesan network (BN), radal bass functon (RBF), and artfcal neural network (ANN). To check the valdty and accuracy of developed models, testng set s appled. nto tranng (two-thrd) and testng (the other one-thrd) datasets each contanng seventeen sgnfcantly correlated attrbutes and the outcome varables for every record wthout any mssng data. The sx above named classfers were appled to the tranng dataset to estmate the relatonshp among the attrbutes and to buld predctve models. Afterwards, the testng dataset whch was not used to model nference was utlzed to calculate the predcted classes and compare the predcted values wth the realones avalable n the testng dataset. The model whch s the most ftted and accurate one wll be selected to predct the outcome of Tb treatment course for new TB cases Modelng Algorthms Decson Tree The decson tree s a nonparametrc estmaton algorthm that nput space s dvded nto local regons defned by a dstance measure lke the Eucldean norm; t s a flowchart-lke tree structure where the nternal node, branches, and leaf node means concepts assocated wth our tranng tuples. In ths herarchcal data structure, the local regon s dentfed n a sequence of recursve splts that n a smaller number of steps by mplementng dvdeand-conquer strategy. Ths powerful classfer s famous for ts ntutve explanablty [17]. In ths research, C4.5 classfcaton task-orented algorthm has been appled through calculatng: m I s1, s2,, sm log p 1 2p (4) p s s (5) where p s the probablty that an nstance belongs to class C. Havng calculated the entropy E A, whch addresses the expected nformaton accordng to parttonng by attrbute A we have: q s s sm EA I 1 s,, s m (6) s Is1, 2,, m s s m p log 1 2p (7) s where p and s s the number of samples n s subset s. In the next step, the encodng nformaton that would be ganed by branchng on A s: 1 2 m Gan A I s, s,, s E A (8) The attrbute A wth the hghest nformaton gan s chosen as the root node, the branches of the root node s formed accordng to varous dstnctve values of a. The tree grows untl f all the samples are all of the same class, and then the node becomes a leaf and s labeled Copyrght 2013 ScRes.

4 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course 187 wth that class. From the tree, understandable rules can be extracted n a quck processng [17]. Here, the task of decson tree development s conducted usng C4.5 classfcaton algorthm where each tree leaf s allocated to a class of chemotherapy outcome along wth number of msclassfed cases Bayesan Networks Generally, Bayesan classfers are statstcal approaches capable of predctng class membershp lkelhoods lke the probablty of the tranng set belongng to a specfc class. It s based on Bayes theorem and well known for ts hgh accuracy and speed when appled to a large data collecton. Here, smple estmator Bayesan network has been used. Let X be a sample whose class s to be defned and let H be a hypothess that X belongs to class C. In ths classfcaton task, PH X needs to be defned whch s the probablty that the hypothess H holds based on the data sample X. It can be calculated through: PH X PX where PH and and X respectvely. P X H P X H P H (9) P X are the pror probablty of H s the posteror probablty of X gven on the H. In model tranng these three values are calculated and the probablty for sample X to be n hypothess H can be determned. There are two essental features that defne a Bayesan network: a drected cyclc graph n whch each node denotes a random varable, ether dscrete or contnuous values, as well as a set of condtonal probablty tables. Let x x 1,, xn be a data tuple related to the correspondent attrbutes y1, y 2,, yn. Gven that every attrbute s condtonally ndependent on ts non-descendent and parents n the network, the followng equaton s a representaton of ont probablty dstrbuton.,,, n parents P x x x P x y (10) 1 2 n 1 where the values for P x parentsy the entres for ; P x x x are related to y 1, 2,, n s the probablty of a specfc combnaton of values of X. Probablty dstrbuton, wth the probablty of each class may be the output of ths classfcaton process [18] Logstc Regresson Logstc regresson s used prmarly for predctng bnary or mult-class dependent varables. Ths algorthm s response varable s dscrete and t bulds the model to predct the odds of ts occurrence. Ths method s restrctve assumptons on normalty and ndependence nduced an ncreased applcaton and popularty of machne learn- ng technques for real-world predcton problems. In ths study, multnomal logstc regresson s appled whch s an algorthm that constructs a separatng hyperplane between two sets of data by usng the logstc functon to express dstance from the hyperplane as a probablty of dchotomous class membershp: 1 (11) x x x x P n n 1 e In ths equaton, X symbolzes dscrete or contnuous predctor varables wth numerc values; n the case of dependng varable () beng dchotomous, we use ths algorthm. The constants 0, 1, 2,, n are the regresson coeffcents estmated from tranng data whch are typcally computed by usng an teratve maxmum lkelhood technque. Normally, ths formula s ustfcaton s that the log of the odds, a number that goes from to +, s a lnear functon. Partcularly by usng ths model, stepwse selecton of the varables can be made and the related coeffcents calculated. In producng the LR equaton, the statstcal sgnfcance of the varables used to be determned by the maxmum-lkelhood rato [19] Artfcal Neural Networks Artfcal Neural Network (ANN) s bologcally nspred analytcal method whch s able of modelng extremely complex non-lnear functons. Here, a common archtecture named mult-layer perceptron (MLP) wth learnng by back-propagaton algorthm s bult. A neural network s a compound of lnked nput/output unts n whch every lnk has an assocated weght. Adustng the weghts s the core phase for predctng the correct class label of nput through teratve learnng. Ths method s popularly used n classfcaton and predcton tasks wth hgh tolerance to nose and the ablty to classfy unseen patterns [19]. Here, algorthm has conducted the learnng process 500 tmes and the network structure s shown n a smple way when only two attrbutes have been appled as nputs. The structure of a two-nput, one hdden layer neural network for our task wth fve outputs has been presented n Fgure Radal Bass Functon A radal bass functon network (RBFN) s an artfcal ntellgence network n whch ts actvaton functon s smply radal bass n a lnear combnaton. Ths type of network was desgned to vew a problem n curve-fttng (approxmaton) and hgh dmensonal space. The real nspraton behnd the RBF technque s fndng amultdmensonal functon that offers the best ft to tranng tuple and then apples ths multdmensonal surface to nterpolate the test data through regularzaton [20]. Gaus- Copyrght 2013 ScRes.

5 188 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course Recent TB nfecton HIV F F W 11 W 22 H W W Input Layer Hdden Output H Cured Completed Qutted Faled Dead Fgure 2. The smplfed structure of appled neural network only wth two of our attrbutes HIV and recent TB nfecton, one hdden layer and fve classes n output layer. The structure s much smplfed neural networks are popularly appled n classfcaton and predcton, as they have advantages such as hgh tolerance to nose, and the ablty to classfy unseen patterns. san radal bass functon s appled algorthm n ths study Support Vector Machne Support vector machne (SVM) s a new classfcaton method for both lnear and non-lnear data. SVM apples nonlnear mappng to transform the orgnal tranng tuple nto a hgher dmenson. It seeks the optmal lnear separaton hyperplane whch s the decson boundares separatng the tuple based on ther class labels.polykernel support vector machne has been appled n ths nvestgaton.let the dataset Abe consdered as x1, y 1, x2x2, y2, x A, y D where x s the set of learnng data wth correspondent class label y. For a two-class related tranng dataset, for nstance, every y can take ether +1 or 1. Ths could also be generalzed to n dmensons and the SVM duty s to fnd the best dvdng lnes that can be drawn and dvde all of the tuples of every class from the others. For multdmensonal classes the hyperplanes should be found asdecson boundares. Ths can be arranged by defnng the maxmum margnal hyperplane (MMH) snce models wth larger hyperplane are more accurate at classfcaton [18] Accuracy Measurements In order to evaluate the predcton rate, the followng related parameters need to be measured. Predcton accuracy percentage (model Accuracy), model ftness, recall, Precson, F-measure, and ROC area are consdered crtera used to assess the models valdty. In confuson matrx (Fgure 3) n whch the columns denote the actual cases and the rows denote the predcted cases, the accu- racy of model obtaned from tranng set (model ftness) and testng set (predcton accuracy) are calculated. The predcton accuracy s the percentage of correct predcton (true postve + true negatve) dvded by the total number of predctons. In machne learnng, senstvty s smply termed recall (r) and precson s the postve predcted value. F-measure s a harmonc means of precson and recall and the hgher value of t merely reveals the better performance of a predcton task. Another comparatve crteron s Roc curve whch s a two-dmensonal graph where the true postve (TP) rate on the axs s plotted on the false postve (FP) rate on the X axs [21]. TP Senstvty TP FN TN Specfcty TN FP Predcton Accuracy % TP TN TP TN FP FN Precson % Recall % 2.5. The Database TP TP FP TP TP FN (12) (13) (14) (15) (16) The avalable dataset has been drven from gathered records by health practtoners, nurses, and physcans at local TB control centres throughout Iran n In tuberculoss control centres, health deputes of each provnce n a network system collected data n every appontment. By usng Stop TB software, more than 35 parameters were collected and through applyng bvarate Predcted Class Negatve (-) Postve (+) Postve (P) True Postve Count (TP) False Negatve Count (FN) True Class Negatve (N) False Postve Count (FP) True Negatve Count (TN) Fgure 3. A smple confuson matrx, a two-by-two table, values n the man dagonal (TP and TN) represent nstances correctly classfed, whle the other cells represent nstances ncorrectly classfed (FP and FN errors). Copyrght 2013 ScRes.

6 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course 189 correlaton method; those ndependent varables whch are sgnfcantly correlated wth target outcome are selected as predctors (P <= 0.05). They are presented and defned n Table 1. The dstrbuton of correspondng outcomes of TB treatment course based on ther frequency n tranng and testng sets are shown n Table 2. Due to normal and non-normal dstrbuton varables whch were tested by both Kolmogorov-Smrnov test and vsual shape of attrbute s dstrbuton, Pearson and Spearman rho methods were appled to fnd hghly correlated factors. 3. Experments and Results 3.1. Feature Selecton Sgnfcant Kolmogorov-Smrnov test confrmed wth the shape of ther dstrbuton showed the dstrbuton of our varables; Spearman rho was appled for those wth non- normal dstrbuton. Havng consdered the results, t s revealed that males are more lkely to not get cure and complete the course of TB treatment (r = 0.082, P < ). As patents get older there s more probablty to get undesrable result from treatment course (r = 0.158, P < ). The more under-weght the patents are, they are more lkely at the rsk of unwelcome outcome (r = 0.056, P < ). TB cases wth extra-pulmonary TB are more lkely at the rsk of outcomes lke quttng, falng or death (r = 0.066, P < ). There s more probablty that mmgrants from Iraq, Pakstan, and Afghanstan to qut, get faled n treatment course (r = 0.127, P < ). TB patents who are lvng n abroad as well as moble cases are 0.027% more possble to get undesrable outcome compared wth Table 1. Patents attrbutons appled for experments, ther defntons and ther range of values. Varable Varable Defnton Categores of Values Demographc Characterstcs sex Gender of TB case Male (1)/Female (2) Age Age of TB patent (Contnuous var.) Weght Weght of TB patent (Contnuous var.) Natonalty Natonalty of TB patent Iranan (1), Central Asans (2), Iraq (3), Pakstan (4), Afghan (5) Area Area of resdence Abroad (1), Moble (2), Rural (3), Urban (4) Prson Current stay n prson No (1) /es (2) Clncal Features Case type Type of TB that patent s belonged to New (1), mported (2), cure after absence (3), returned (4) Treat cat Category of treatment that s conductng A (1)/B (2) TB type The part of body that has been affected Pulmonary (1)/extra-pulmonary (2) RTBnf Whether patent has recently TB affected No (1)/yes (2) Dabetes Whether TB patent saffected by dabete No (1)/yes (2) HIV Whether TB patent has been known as HIV+ No (1)/suspected (2)/yes (3) Length Length (month) of beng affected by TB (Contnuous var.) LBW Low Body Weght No (1)/yes (2) Socal Rsk Factors Imprsonment The hstory of lvng n prson No (1)/suspected (2)/yes (3) IV drug usng Whether patent s usng the (IV) drugs No (1)/suspected (2)/yes (3) Rsky sex Whether patent has hstory of rsky sex No (1)/suspected (2)/yes (3) Table 2. The real outcomes happened for Tuberculoss patents for ntal, tranng and testng sets wth ther correspondng outcomes. Data set Cured Completed Qut Faled Dead Total Tranng set Testng set Intal dataset Copyrght 2013 ScRes.

7 190 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course those who are lvng n urban and rural areas (r = 0.027, P < ). Prson resdency and treatment outcome completon are sgnfcantly correlated (r = 0.026, P < ). The more tme TB patents spend affected wth ths nfectous dsease, there s tmes more chance of non-desrable outcomes of tuberculoss treatment course. Dabetc or HIV + TB cases are postvely prone to have worse result of treatment course (rho dabetes = 0.029, P < & rho HIV = 0.045, P < 0.05). Also, the probabltes of havng mprsonment hstory n hs/her lfe, consumng drugs through ntravenous, or havng unprotected sex ncrease the lkelhood of unwanted outcome wth r = 0.157, , and 0.16 respectvely (P < 0.000) Classfcaton Algorthms Analyss In ths work, sx classfers ncludng DT, BN, LR, MLP, RBF and SVM were appled to the patent dataset. Model development s conducted n two man steps ncludng model ftness and model accuracy. To calculate the model ftness crtera we used the data of tranng set; however, to compute the model accuracy measurements, data of testng set s appled whch s merely much more valuable to udge about our models accuracy. Related results of these experments are demonstrated n Table 3. Model ftness assessment by evaluatng tranng accuracy s 84.45% for C4.5 decson trees; t s consderably less for Bayesan net where the value s 58.56%. The values of model ftness for logstc regresson (56.5%), MLP (64.93%), RBF (50.65%), and SVM (53.04%) have been close. Havng compared the Roc curves reveal that the area under curve for C4.5 has the most value for model ftness and accuracy wth 0.96 and 0.97 respectvely. Ths measurement s less for Bayesan net (0.85), logstc regresson (0.82), MLP (0.81), RBF (0.79), and SVM (0.76) n terms of model accuracy. C4.5 decson tree has been able to buld a model wth greatest accuracy snce the model ftness and predcton accuracy are 84.45% and 74.21% respectvely. Predcton accuracy for Bayesan networks has been calculated by 62.06%. Model accuraces obtaned from other classfers are dfferent as ths value for LR, MLP, RBF, and SVM have been 57.88%, 57.31%, 53.74%, and 51.36% respectvely. Fgure 4 s the comparatve Roc curves based on the gven outcome of tuberculoss treatment ncludng cure, complete, qut, faled or dead. Ths fgure shows sx Roc curves for sx developed models based on the gven outcome. For the outcome cure, C4.5 has outperformed than other classfers wth area under curve Smlarly, the most accurate result for outcomes completed and qut obtaned for C4.5 wth and respectvely. C4.5 has also performed the best for the outcome faled and dead by classfyng and of cases correctly. Overall, these results of area under curve reveals better performance of C4.5 decson tree classfcaton algorthm. Table 3 and Fgure 4 present obtaned results ncludng model ftness and accuracy as well as produced area under ROC. 4. Dscusson Of the sx nvestgated methods, decson tree has acheved the best performance whle other classfers have gven relatvely close results n lower ranks. Accordng to prevous studes [22,23], the technque wth the best classfcaton performance mght behave dfferently from another one and there s no sngle best method for every crcumstance. Decson trees that classfy nstances by sortng them based on feature values has outperformed other methods. C4.5 performs varously n dfferent condtons. It has been reported that there s an assocaton between the performance of appled tools and followng ssues ncludng the type of problem we are analyzng, the type of nput data (dscrete or contnuous), and fnally emergng overlappng n outcome classes. Due to the fact that our avalable dataset s a large volume of data wth dmensonal structure, normally t s expected that SVMs, neural networks, and decson tree outperform others. However, the dataset s manly com- Table 3. Comparson on model ftness and model accuracy of sx varous appled machne learnng algorthms. Model Ftness (through usng tranng set) Model Accuracy (through usng testng set) Classfers PA * Recall Precson F-measure ROC Area PA ** Recall Precson F-measure ROC Area C BN LR MLP RBF SVM * tranng accuracy (%); ** predcton accuracy percentage (%). Copyrght 2013 ScRes.

8 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course 191 Fgure 4. ROC curve for C4.5 decson tree, logstc regresson (LR), Bayesan networks (BN), radal bass functon (RBF), multlayer perceptron neural networks(mlp), and support vector machne on the task of classfyng outcome of TB treatment course based on the target outcome ncludng cured, completed, qut, faled, and dead. In all cases decson tree has outperformed other methods wth area under curve 0.92 for cured, 0.94 for completed, 0.88 for qut, 0.96 for faled, and 0.85 for dead. posed of fourteen dscrete varables and three contnuous attrbutons (age, weght, and length of dsease); n ths case, decson tree has produced the most promsng results due to ts dual ablty to tackle both contnuous and dscrete/categorcal predctors whch s superor to other aforementoned technques that are good at handlng only contnuous varables. In the case of emergng relatonshp among attrbutons, BNs don t perform well to manage learnng properly. There have been sgnfcant relatonshps between seventeen predctors and outcome class whch may cause the weaker performance of Bayesan networks rather than C4.5; furthermore, dscrmnant algorthms lke logstc regresson also fal on ths type of data wth hgh correlaton between the attrbutes. In ths study, there are many correlatons among varables, lke weght and natonalty (r = 0.052, P < 0.001), LBW and sex (r = 0.047, P < 0.001), mprsonment and sex (r = 0.156, P < 0.001), prson and weght (r = 0.065, P < 0.001), length and natonalty (r = 0.099, P < 0.001). Those correlatons n addton to applyng fourteen dscrete nputs mght cause weaker results from BN and LR rather than C4.5. s good at copng wth rrelevant data. Ths mght be the case n ths partcular study snce there are some varables wth very low correlaton coeffcent that decson tree has not taken them very much to bult the model and not at all n the man root nodes. For example, area ( 0.027), prson ( 0.026), dabetes (0.029) have low correlaton coeffcent, where ths value for recent TB nfecton, mprsonment, IV drug usng, sex are 0.250, 0.151, 0.172, respectvely. The varables wth hgh correlaton coeffcent lke recent TB nfecton, length, mprsonment, and treatment category have played a maor role as root nodes whereas the varables wth small correlaton coeffcent have been recognzed as less mportant factors placed n very sub-nodes close to leaves whch can be even pruned. Decson tree s ablty of utlzng sgnfcant nput factors n bass of ther degree of contrbuton to estmate outcome of tuberculoss treatment course creates greater predctve model than those classfers, such as MLP, RBF, and SVM whch count every of nput varable unformly by weghtenng affectng the results transparency. The hgher values for Kurtoss (>7) and skew (>1) denote that our varables are far from normalty and decson tree and other symbolc methods (nonparametrc Copyrght 2013 ScRes.

9 192 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course schemes) tackle robustly wth dstrbutons wth large kurtoss and skew [22]. In ths research s dataset, the average values of skew and kurtoss are and respectvely (P < 0.05) confrmng the non-normal dstrbuton. Hence, the only avalable nonparametrc symbolc learnng algorthm n the current study s decson tree whch performed well to partton the nput space. In actualty, hgh skew (>1) or kurtoss (>7) along wth the presence of bnary/categorcal varables, usng relevant and correlated predctors wthout any mssed nstances or nosed data have prepared the best opportunty for decson tree to predct more accurately than other appled algorthms. In vew of the rank of other employed classfers, BN outperforms others and four remanng classfers work relatvely smlar wth predcton accuracy percentage rangng from 53.74% to 57.82%. In a study [24], Tu has revewed a number of researches concludng that LR & MLP perform closely; t s the case here where they performed wth dentcal predcton accuracy (57.82). RBF s actually a type of neural network and t mght be a postulaton that based on ther algorthm smlartes and data type entty results are comparable. Decson tree wth flowchart-type structure s more lkely method to be understandable for general users wth low level of specalzed knowledge about TB. Produced results of decson tree can be smply nterpretable and applcable; ther rules can be understood ether by doctors or health practtoners who mplement DOTS n rural area and make decson alone. 5. Concluson To sum up, avalable bg body of real data related to TB patents has created an opportunty to generate accurate models whch can predct outcome of DOTS therapy. Ths provdes us nformaton about outcome of treatment course for each patent and defnes who needs hgh level of supervson and support; ths s valuable as t s not possble to gve every sngle of patents a full supervson and support dstnctly. The decson tree model can be used to screen rsky patents for fal n treatment course completon n populaton usng general data gatherng n routne general practce. Ths wll help healthcare practtoners especally n rural regons to evaluate the rsks of MDR-TB among ther patents quckly, nexpensvely, and nonnvasvely. TB control through totally mplemented DOTS therapy s such a crucal stage n publc health mprovement and promoton. 6. Acknowledgements We would lke to express our grattude to Iranan Mnstry of Health and Medcal Educaton for fundng and data access. Furthermore, the authors declare that they have no conflct of nterest. REFERENCES [1] A. D. Harres and C. Dye, Tuberculoss, Annals of Tropcal Medcne and Parastology, Vol. 100, No. 5, 2006, pp do: / x91477 [2] World Health Organzaton, The Stop TB Strategy, Buldng on and Enhancng DOTS to Meet the TB-Related Mllennum Development Goals, [3] W. D. Cuneo and D. J. Snder, Enhancng Patent Complance wth Tuberculoss Therapy, Clncs n Chest Medcne, Vol. 10, No. 3, 1989, pp [4] H. G. Tangüs, J. A. Caylà, P. García, J. M. Jansà and M. T. Brugal, Factors Predctng Non-Completon of Tuberculoss Treatment among HIV-Infected Patents n Barcelona ( ), The Internatonal Journal of Tuberculoss and Lung Dsease, Vol. 4, No. 1, 2000, pp [5] W. W. ew, Drectly Observed Therapy Short-Course: The Best Way to Prevent Multdrug-Resstant Tuberculoss, Chemotherapy, Vol. 45, No. 2, 1999, pp do: / [6] J. Legrand, A. Sanchez, F. Le Pont, L. Camacho and B. Larouze, Modelng the Impact of Tuberculoss Control Strateges n Hghly Endemc Overcrowded Prsons, Plos One, Vol. 3, No. 5, 2008, Artcle ID: e2100. do: /ournal.pone [7] S. Tham, A. M. Le Fevre and F. Hane, Effectveness of a Strategy to Improve Adherence to tuberculoss Treatment n a Resource-Poor Settng: A Cluster Randomzed Controlled Tral, Journal of the Amercan Medcal Assocaton, Vol. 297, No. 4, 2007, pp do: /ama [8] W. J. Burman, D. L. Cohn, C. A. Retmeer, F. N. Judson, J. A. Sbarbaro and R. R. Reves, Noncomplance wth Drectly Observed Therapy for Tuberculoss. Epdemology and Effect on the Outcome of Treatment Chest, Vol. 111, No. 5, 1997, pp do: /chest [9] P. D. O. Daves, The Role of DOTS n Tuberculoss Treatment and Control, Amercan Journal of Respratory Medcne, Vol. 2, No. 3, 2003, pp do: /bf [10] A. V. Star-Taut, D. Zdrenghea, D. Pop and D. A. Star- Taut, Usng Machne Learnng Algorthms n Cardovascular Dsease Rsk Evaluaton, Journal of Appled Computer Scence & Mathematcs, Vol. 5, No. 3, 2009, pp [11] M. Lazarescu, A. Turpn and S. Venkatesh, An Applcaton of Machne Learnng Technques for the Classfcaton of Glaucomatous Progresson, Vol. 2396, Sprnger-Verlag, Berln, [12] J. I. Serrano, M. Tomécková and J. Zvárová, Machne Learnng Methods for Knowledge Dscovery n Medcal Data on Atheroscleross, European Journal of Bomedcal Informatcs, [13] I. Guyon and A. Elssee, An Introducton to Varable and Copyrght 2013 ScRes.

10 Evaluaton and Comparson of Dfferent Machne Learnng Methods to Predct Outcome of Tuberculoss Treatment Course 193 Feature Selecton, Journal of Machne Learnng Research, Vol. 3, 2003, pp [14] M. Dash and H. Lu, Feature Selecton for Classfcaton, Intellgent Data Analyss, Vol. 1, No. 3, 1997, pp do: /s x(97) [15] A. Feld, Dscoverng Statstcs Usng SPSS, 2nd Edton, SAGE Publcaton LTD, London, [16] J. Han and M. Kamber, Data Mnng: Concepts and Technques, 2nd Edton, Morgan Kaufmann Publshers, Burlngton, [17] E. Alpaydn, Introducton to Machne Learnng, 1th Edton, The MIT Press, Cambrdge, [18] S. B. Kotsants, Supervsed Machne Learnng: A Revew of Classfcaton Technques, Informatca, Vol. 31, No. 3, 2007, pp [19] E. Vttnghoff, S. C. Shbosk, D. V. Gldden and C. E. McCulloch, Regresson Methods n Bostatstcs, Lnear, Logstc, Survval, and Repeated Measures Models, Sprnger, Berln, [20] S. Marsland, Machne Learnng: An Algorthmc Perspectve, 1st Edton, Chapman and Hall, London, [21] L. Olson and D. Delen, Advanced Data Mnng Technques, Sprnger, Berln, [22] R. D. Kng, C. Feng and A. Sutherland, Statlog: Comparson of Classfcaton Algorthms on Large Real- World Problems, Appled Artfcal Intellgence, Vol. 9, No. 3, 1995, pp [23] I. Kurt, M. True and A. T. Kurum, Comparng Performances of Logstc Regresson, Classfcaton and Regresson Tree, and Neural Networks for Predctng Coronary Artery Dsease, Expert System Applcaton, Vol. 34, 2008, pp do: /.eswa [24] J. V. Tu, Advantages and Dsadvantages of Usng Artfcal Neural Networks Versus Logstc Regresson for Predctng Medcal Outcomes, Journal of Clncal Epdemology, Vol. 49, No. 11, 1996, pp do: /s (96) Copyrght 2013 ScRes.