A Web-Based E-Testing System Supporting Test Quality Improvement

Transcription

1 A Web-Based E-Testing System Supprting Test Quality Imprvement Gennar Cstaglila, Filmena Ferrucci, Vittri Fuccella Dipartiment di Matematica e Infrmatica Università degli Studi di Salern, Fiscian(SA),Italy {gcstaglila,fferrucci,vfuccella}@unisa.it ABSTRACT In e-testing it is imprtant t administer tests cmpsed f gd quality questin items. By the term quality we intend the ptential f an item in effectively discriminating between strng and weak students and in btaining tutr s desired difficulty level. Since preparing items is a difficult and timecnsuming task, gd items can be re-used fr future tests. Amng items with lwer perfrmances, instead, sme shuld be discarded, while sme can be mdified and then re-used. This paper presents a Web-based e-testing system which detects defective questin items and, when pssible, prvides the tutrs with advice t imprve their quality. The system detects defective items by firing rules. Rules are evaluated by a fuzzy lgic inference engine. The prpsed system has been used in a curse at the University f Salern. Keywrds e-testing, Cmputer Aided Assessment, CAA, item, item quality, questins, ewrkbk, Item Respnse Thery, IRT, Test Analysis, nline testing, difficulty, discriminatin, multiple chice, distractr. 1. INTRODUCTION E-testing systems are mre and mre widely adpted in academic envirnments cmbined with ther assessment means. Thrugh these systems, tests cmpsed f several questin types can be presented t the students in rder t assess their knwledge. Multiple Chice questin type is extremely ppular, since, amng ther advantages, a large number f its utcmes can be easily crrected autmatically. The experience gained by educatrs and the results btained frm several experiments [22] prvide sme guidelines fr writing gd multiple chice questins (items, in the sequel), such as: use the right language, avid a big number f unlikely distractrs fr an item, etc. It is pssible t evaluate the effectiveness f the items, thrugh the use f several statistical mdels, such as Item Analysis (IA) and Item Respnse thery (IRT) [8]. They are bth based n the interpretatin f statistical indicatrs calculated n test utcmes. The mst imprtant f them are the difficulty indicatr, which measures the difficulty f the items, and the discriminatin indicatr, which represents the infrmatin f hw well an item discriminates between strng and weak students. Mre statistical indicatrs are related t the distractrs (wrng ptins) f an item. An item with a high value fr discriminatin is a gd item, that is, an item that is answered crrectly by strng students and incrrectly by weak nes, n average. Furthermre, in this study we regard as mre efficient thse items whse calculated difficulty tends t be clser t the difficulty guessed by the tutr. In a test, in rder t better assess a hetergeneus class with different levels f knwledge, it is imprtant t balance the difficulty f the items: tests shuld be cmpsed f given percentages f difficult (25%), medium (50%) and easy (25%) items. If the tutr succeeds in giving the desired difficulty level t an item, he/she can mre easily cnstruct balanced tests which assess students n the desired knwledge. Despite the availability f guidelines fr writing gd items and statistical mdels t analyze their quality, nly a few tutrs are aware f the guidelines and even fewer are used with statistics. The result is that the quality f the tests used fr exams r admissins is smetimes pr and in sme cases culd be imprved. Althugh it is almst impssible t cmpel the tutrs t read manuals fr writing gd test assessment, it is pssible t give them feedback n their items quality, allwing them t discard defective items r t mdify them in rder t imprve their quality fr next use and, at the same time, t learn hw t write gd items frm experience. This paper presents a Web-based e-testing system which helps the tutrs t btain gd quality assessment items. By item quality we intend the ptential f an item in effectively discriminating between strng and weak students and in btaining a tutr s desired difficulty level. After a test sessin, the system marks the items: gd items are marked with a green light. Fr pr quality items there are tw different levels f alarm: severe (red light), fr items which shuld be discarded, and warning (yellw light) fr items whse quality culd be imprved. Fr the latter nes, the system prvides the tutr with suggestins fr imprving item quality. Aware f defective items, and helped by the suggestins f the system, the tutr can discard r mdify pr items. Imprvable items can be re-used fr future tests.

2 Quality level and eventual suggestins are decided thrugh a rule-based classificatin [23]. Fuzzy lgic has been used in rder t btain a degree f fulfillment f each rule. Rules have been preferred ver ther frequently used classificatin methds, such as hierarchical methds [1], K-means methds [15] and crrelatin methds due t the fllwing reasns: Knwledge availability. Mst f the knwledge is already available, as witnessed by the presence f numerus theries and manuals n psychmetrics. Lack f data. Other types f classificatin based n data wuld require the availability f large data sets. Once they have gathered, in such a way t have statistically significant classes t perfrm data analysis, such methds might be explited. The system has been carried ut by adding the frmerly described features t an existing Web-based e-testing system: ewrkbk [5], develped at University f Salern, which has been equipped with an Item Quality Mdule. A first experiment has been carried ut in a curse at the University f Salern. The paper is rganized as fllws: sectin 2 gives sme cncepts abut the knwledge n which the system is based. In sectin 3, the system is defined, fllwing the steps f a classical methdlgy fr fuzzy systems definitin. In sectin 4, we briefly discuss the implementatin f the quality mdule and its integratin in the existing e-testing system. Finally, sectin 5 presents an experiment and a discussin f its results. The paper cncludes with a brief survey n wrk related t urs, several final remarks and a discussin n future wrk. 2. THE KNOWLEDGE-BASE Our system makes use f multiple chice items fr the assessment f students knwledge. Thse items are cmpsed f a stem and a list f ptins. The stem is the text that states the questin. The nly crrect answer is called the key, whilst the incrrect answers are called distractrs [22]. Test results can be statistically analyzed t check item quality. As mentined in the previus sectin, tw main statistical mdels are available: IA and IRT. Several studies, such as the ne in [20], make a cmparisn between the tw mdels, ften cncluding that they can bth be effective in evaluating the quality f the items. Fr ur study, IA has been preferred t IRT fr the fllwing main reasns: it needs a smaller sample size fr btaining statistically significant indicatrs and it is easier t use IA indicatrs t cmpse rule cnditins. The fllwing statistical indicatrs are calculated by ur system fr each item answered by a significant number f students: difficulty: a real number between 0 and 1 which expresses a measure f the difficulty f the item, intended as the prprtin f learners wh get the item crrect. discriminatin: a real number between -1 and 1 which expresses a measure f hw well the item discriminates between gd and bad learners. Discriminatin is calculated as the pint biserial crrelatin cefficient between the scre btained n the item and the ttal scre btained n the test. frequency(i): a real number between 0 and 1 which expresses the frequency f the i-th ptin f the item. Its value is calculated as the percentage f learners wh chse the i-th ptin. discriminatin(i): a real number between -1 and 1 which expresses the discriminatin f the i-th ptin. Its value is the pint biserial crrelatin cefficient between the result btained by the learner n the whle test and a dichtmus variable that says whether the i-th ptin was chsen (yes=1, n=0) by the learner r nt. abstained_freq: a real number between 0 and 1 which expresses the frequency f the abstentin (n answers given) n the item. Its value is calculated as the percentage f learners wh didn t give any answer t the item, where allwed. abstained_discr: a real number between -1 and 1 which expresses the discriminatin f the abstentin n the item. Its value is the pint biserial crrelatin cefficient between the result btained by the learner n the whle test and a dichtmus variable that says whether the learner refrained r nt (yes=1, n=0) n the item. Discriminatin and difficulty are the mst imprtant indicatrs. They can be used fr bth determining item quality and chsing advice fr tutrs. As experts suggest [16], a gd value fr discriminatin is abut 0.5. A psitive value lwer than 0.2 indicates an item which des nt discriminate well. This can be due t several reasns, including: the questin des nt assess learners n the desired knwledge; the stem r the ptins are badly/ambiguusly expressed; etc. It is usually difficult t understand what is wrng with these items and mre difficult t prvide a suggestin t imprve them, s, if the tutr cannt understand the prblem her(him)self, the suggestin is t discard the item. A negative value fr

3 discriminatin, especially if jined with a psitive value fr the discriminatin f a distractr, is a sign f a pssible mistake in chsing the key (a data entry errr ccurred). In this case it is easy t recver the item by changing the key. If the difficulty level is t high (>0.85) r t lw (<0.15), there is the risk f nt crrectly evaluating n the desired knwledge. This is particularly true when such values fr the difficulty are sught tgether with medium-lw values fr discriminatin. Furthermre, ur system allws the tutr t define the freseen difficulty fr an item. Thus, the clser a tutr s estimatin f item difficulty is t the actual calculated difficulty fr that item, the mre reliable that item is cnsidered t be. When difficulty is t high r underestimated, this can be due t the presence f a distractr (nticed fr its high frequency) which is t plausible (it tends t mislead a lt f students, even strng nes). Remving r substituting that distractr can help in btaining a better item. Smetimes, the item has its intrinsic difficulty and it can be difficult t adjust it, s the suggestin can be t mdify the tutr s estimatin. As fr distractrs, they can cntribute t a gd item when they are selected by a significant number f students. When the frequency f the distractr is t high, there culd be an ambiguity in the frmulatin f the stem r f the distractr. A gd indicatr f distractrs quality is their discriminatin, which shuld be negative, denting that the distractr was selected by weak students. In cnclusin, a gd distractr is the ne which is selected by a small but significant number f weak students. High abstentin is always a symptm f high difficulty fr the item. When it is accmpanied by a high (nt negative r next t 0) value fr its discriminatin and a lw value fr item discriminatin, it can tell that the questin has a bad quality and it is difficult t imprve it. 3. THE FUZZY SYSTEM The system fr the evaluatin f item quality is rulebased: the rules use, as linguistic variables, statistical indicatrs calculated after a test sessin. By this term we mean the time necessary t administer several items t a statistically significant number f students. The value f this number is set in the cnfiguratin f the system. The system wrks by perfrming a classificatin f the items. Several classes f items have been identified, and each class is assciated t a prductin rule. The degree f fulfillment f a rule tells the membership f the item t the crrespnding class. The classificatin is perfrmed by selecting the class fr which the degree f fulfilment is the highest. 3.1 Variables and Fuzzyficatin The set f variables used are reprted, tgether with an explanatin f their meaning and the set f pssible values they can assume (terms), in table 1. These variables are directly chsen frm the statistical indicatrs presented in sectin 2 r derived frm them. Table 1. Variables and Terms Variable Explanatin Terms discriminatin Item s discriminatin (see sec. 2) Negative, lw, high difficulty Item s difficulty (see sec. 2) Very_lw, medium, very_high difficulty_gap max_distr_discr max_distr_freq min_distr_freq distr_freq abst_frequency abst_discriminati n The difference between the tutr s estimatin f item s difficulty and the difficulty calculated by the system The maximum discriminatin fr the distractrs f an item The maximum (relative) frequency fr the distractrs f an item. The minimum (relative) frequency fr the distractrs f an item The (relative) frequency f the distractr with maximum discriminatin fr an item The frequency f the abstentins fr an item The discriminatin f the abstentins fr an item Underestimated, crrect, verestimated Negative, psitive Lw, high Lw, high Lw, high Lw, high Negative, psitive The variables discriminatin and difficulty are the same indicatrs fr item discriminatin and difficulty defined in sectin 2. The same discurse is valid fr the variables related t the abstentin, abst_frequency and abst_discriminatin. difficulty_gap is a variable representing the errr in tutr s estimatin f item difficulty. Thrugh the system interface, the tutr can assign ne n three difficulty level t an item (easy = 0.3; medium = 0.5; difficult = 0.7). difficulty_gap is calculated as the difference between the tutr estimatin and the actual difficulty calculated by the system. Three variables representing the frequency f the distractrs fr an item have been cnsidered: max_distr_freq, min_distr_freq, distr_freq. Their value is nt an abslute frequency, but relative t the frequency f the ther distractrs: it is btained by dividing the abslute frequency by the mean frequency f the distractrs f the item. In the case f items with five ptins, as ur system has been tested, their value is a real number varying frm 0 t 4.

4 3.2 Membership Functins As fr the membership functins f fuzzy sets assciated t each term, triangular and trapezidal shapes have been used. Mst f the values fr the bases and the peaks have been established using the expertise. Only fr sme variables, the membership functins have been defined n an experimental basis. While we already had clear ideas n hw t define sme membership functins, we did nt have enugh infrmatin frm the knwledge-base n hw t mdel membership functins fr the variables related t abstentin (abst_frequency and abst_discriminatin). A calibratin phase was required in rder t refine the values fr the bases and peaks f their membership functins. As a calibratin set, test results frm the Science Faculty Admissin Test f the last year (2006) were used. The calibratin set was cmpsed f 64 items with 5 ptins each. Fr each item, abut ne thusand recrds (students answers) were available, even if nly a randm sample f seventy f them was cnsidered. Test items and their results were inspected by a human expert wh identified items which shuld have been discarded due t lw discriminatin and anmalus values fr the variables related t abstentin. We have fund 5 items satisfying the cnditins abve: the mean values fr abst_discriminatin and abst_frequency were, respectively, 0.12 and Figure 1. Membership Functins f the Fuzzy Sets Due t the limited size f the calibratin set, the simple methd f chsing the peaks f the functins at the mean value, as shwn in [1], has been used. When mre data will be available, a mre sphisticated methd will be used fr the definitin f membership functins, such as the ne prpsed in [4]. Charts fr the membership functins are shwn in figure Rules Frm the verbal descriptin f the knwledge presented in sectin 2, the rules summarized in table 2 have been inferred. The first three clumns in the table cntain, respectively, the class f the item, the rule used fr classificatin and the item state. Fr items whse state is yellw, the furth clumn cntains the prblem affecting the item and the suggestin t imprve its quality. Cnditins in the rules are cnnected using AND and OR lgic peratrs. The cmmnly-used min-max inference methd has been used t establish the degree f fulfillment f the rules. All the rules were given the same weight, except fr the first ne. By mdifying the weight f the first rule, we can tune the sensitivity f the system: the lwer this value, the higher the prbability that anmalies will be detected in the items. Sme rules suggest t perfrm an peratin n a distractr. The distractr t mdify r eliminate (in case f rules 4, 7 and 10) r t select as crrect answer (rule 9) is signaled by

5 Table 2. Rules fr Item Classificatin Class Rule State Prblem and Suggestin 1 discriminatin IS high AND abst_discriminatin Green / IS negative WITH discriminatin IS lw AND abst_frequency IS high AND abst_discriminatin IS psitive 3 difficulty IS very_lw AND discriminatin IS lw 4 difficulty IS very_high AND discriminatin IS lw AND max_distr_freq IS high 5 difficulty_gap IS verestimated AND discriminatin IS lw 6 difficulty_gap IS verestimated AND discriminatin IS NOT lw 7 difficulty_gap IS underestimated AND max_distr_freq IS high 8 difficulty_gap IS underestimated AND max_distr_freq IS NOT high 9 max_distr_discr IS psitive AND discriminatin IS negative 10 discriminatin IS high AND max_distr_discr IS psitive AND distr_freq IS NOT lw Red / Red / Yellw Yellw Yellw Yellw Yellw Yellw Yellw Item t difficult due t a t plausible distractr, delete r substitute distractr x. Item difficulty verestimated, avid t plausible distractrs and t bvius answers. Item difficulty verestimated, mdify the estimated difficulty. Item difficulty underestimated due t a t plausible distractr, delete r substitute distractr x. Item difficulty underestimated, mdify the estimated difficulty. Wrng key (data entry errr), select ptin x as the crrect answer. T plausible distractr, delete r substitute distractr x. the system. An utput variable x has been added t the system t keep the identifier f the distractr. 4. DEVELOPMENT OF THE ITEM QUALITY MODULE AND INTEGRATION IN THE EWORKBOOK SYSTEM A sftware mdule fr the evaluatin f item quality has been implemented as a Java Object Oriented framewrk. In this way, it wuld have been easily integrated in any e- testing java-based system. Fr each item, the mdule perfrms the classificatin, by implementing the fllwing functinalities: Implementatin f an Applicatin Prgramming Interface (API) fr the cnstructin f a data matrix cntaining all the students respnses t the item. Calculatin f the statistical indicatrs, as described in sectin 2. Substitutin f the variables, evaluatin f the rules and chice f the class which the item belngs t. Implementatin f a suitable API fr btaining the state f an item (green, yellw, red) and, in case f yellw, f the suggestins fr imprving the item. It is wrth nting that suggestins can be internatinalized, that is, they can easily be translated int any language by editing a text file. A free java library implementing a cmplete Fuzzy inference system, named jfuzzylgic [10] has been used. The system variables, fuzzyficatin, inference methds and the rules have been defined using Fuzzy Cntrl Language (FCL) [7], supprted by the jfuzzylgic library. The advantage f this apprach, cmpared t a hard-cded slutin, is that membership functins and rules can be changed nly by editing a cnfiguratin file, thus aviding t build the system again. Data can be imprted frm varius surces and exprted t several frmats, such as spreadsheets r relatinal databases. The data matrix and the results can be saved in persistent tables, in rder t avid t perfrm calculatins every time they must be visualized. 4.1 ewrkbk ewrkbk is a Web-based e-testing system that can be used fr evaluating learner s knwledge by creating (the tutr) and taking (the learner) n-line tests based n multiple chice questin types. The questins are kept in a hierarchical repsitry. The tests are cmpsed f ne r mre sectins. There are tw kinds f sectins: static and dynamic. The difference between them is in the way they allw questin selectin: fr a static sectin, the questins are chsen by the tutr. Fr a dynamic sectin, sme selectin parameters must be specified, such as the difficulty, leaving the system t chse the questins randmly whenever a learner takes a test. In this way, it is pssible with ewrkbk t make a test with banks f items f different difficulties, thus balancing test difficulty, in rder t better assess a hetergeneus set f

6 students. ewrkbk adpts the classical three-tier architecture f the mst cmmn J2EE Web-applicatins. The Jakarta Struts framewrk has been used t supprt the Mdel 2 design paradigm, a variatin f the classic Mdel View Cntrller (MVC) apprach. In ur design chice, Struts wrks with JSP, fr the View, while it interacts with Hibernate [9], a pwerful framewrk fr bject/relatinal persistence and query service fr Java, fr the Mdel. The applicatin is fully accessible with a Web Brwser. N brwser plug-in installatins are needed, since its pages are cmpsed f standard HTML and ECMAScript [6] cde. 4.2 Integratin The integratin f the new functinalities in ewrkbk has required the develpment f a new mdule, named Item Quality Mdule, respnsible fr instantiating the framewrk and prviding imprt, exprt and visualizatin functinalities. Imprt f data was perfrmed by reading data frm ewrkbk s database and by calling the API t fill the data matrix f the framewrk. The interface fr brwsing the item repsitry in ewrkbk has been updated in rder t shw item s perfrmances (difficulty and discriminatin) and state (green, yellw r red). In this way, defective items are immediately visible t the tutr, wh can undertake the pprtune actins (delete r mdify). A screensht f the item reprt is shwn in figure 2a. Furthermre, the system has been given a versining functinality: nce an item is mdified, a newer versin f it is generated. Thrugh this functinality, the tutr can analyze the entire lifecycle f an item. In this way, the tutr can have feedback n the trend f statistical indicatrs ver time, making sure that the changes he/she made t the items psitively affected their quality. Figure 2b shws the chart f an item imprved acrss tw sessins f tests. The imprvement is visible bth frm the increase in the item discriminatin (the green line), and in the cnvergence f the calculated difficulty with the tutr s estimatin f the difficulty (the cntinuus and dashed red lines, respectively). 5. USE CASE IN A UNIVERSITY COURSE A first experiment has cnsisted f using the system acrss tw test sessins in a university curse, and measuring the verall imprvement f the items in terms f discriminatin capacity and matching t a tutr s desired difficulty. A database f 50 items was arranged fr the experiment. In the first sessin, an n-line test, cntaining a set f 25 randmly chsen items, was administered t 60 students. After, items were inspected thrugh the system interface in rder t check thse t substitute r mdify. Once the substitutins and mdificatins were perfrmed, the mdified test was administered t 60 ther students. Figure 3a shws a table, exprted in a spreadsheet, cntaining a reprt f the items presented in the first test sessin and their perfrmances. The item t eliminate are highlighted in red, while thse t mdify are highlighted in yellw. Accrding t the system analysis, 5 ut f 25 items must be discarded, while 4 f them must be mdified. Figure 2. Screenshts Frm the ewrkbk System Interface Actually, amng the items t mdify, fr tw f them (thse with id 1-F-4 and 1-E-1) the difficulty was underestimated due t a distractr that was t plausible (class 7), which was substituted with a new distractr. In anther case (1-B-16), the difficulty was different frm that estimated by the tutr, due t the intrinsic difficulty f the item (class 8). The actin undertaken was t adjust tutr s estimatin f the difficulty. Lastly, the item with id 1-F-1, with a negative discriminatin, presented a suspect errr in the chice f the key (class 9). By inspecting the item, the tutr verified that the chsen key was nt crrect, even thugh the distractr labeled crrect by the system was nt the right answer: simply, the item did nt have any crrect answer. The text f the key was mdified t prvide the right answer t the stem.

7 A new test was prepared, cntaining the same items f the previus, except fr the 5 discarded nes, substituted by 5 unused items, and fr the 4 mdified nes, which were substituted by a newer versin f themselves. A new set f sixty students participated in this test. In the analysis f test utcmes, ur attentin was mre fcused n the eventual imprvement btained than n the discvery f new defective items. Figure 3. Reprt f the Test Sessins Figure 3b shws the reprt f the secnd test sessin. The values f discriminatin and difficulty, changed in respect t the same rws f the sessin 1 table, are highlighted in yellw. T measure the verall imprvement f the new test, in respect t the previus ne, the fllwing parameters were calculated fr each f the tw tests: the mean f the discriminatins fr the items; the mean f the differences tutr_difficulty difficulty fr the items f the tests; As fr parameter 1, we have bserved an imprvement frm a value f 0,375, btained in the first sessin, t a value f 0,459, btained in the secnd sessin. The percentage f increment is 22,4%. As fr parameter 2, we had a decrement in the mean difference between the difficulty estimated by the tutr and the ne calculated by the system f 17,8%, passing frm a value f 0,19 t 0,156 acrss the tw sessins. 6. RELATED WORK Several different assessment tls and applicatins t supprt blended learning have been analyzed, starting frm the mst cmmn Web-based e-learning platfrms, such as Mdle [17], Blackbard [2], and Questinmark [18]. These systems generate and shw item statistics parameters but they d nt interpret them, s they d nt advise r help the tutr in imprving items erasing anmalies revealed by statistics. A mdel fr presenting test statistics, analysis, and t cllect students learning behavirs fr generating analysis result and feedback t tutrs is described in [12]. IRT has been applied in sme systems [11] and experiments [3, 21] t select the mst apprpriate items fr examinees based n individual ability. In [3], the fuzzy thery is cmbined with the riginal IRT t mdel uncertainly learning respnse. The result f this cmbinatin is called Fuzzy Item Respnse Thery. A wrk clsely related t urs is presented in [13]. It prpses an e-testing system, where rules can detect defective items, which are signaled using traffic lights. It prpses an analysis mdel based n IA. Statistics are calculated by the system bth n the items and n the whle test. Unfrtunately, the fur rules n which the system is based seem t be insufficient t cver all f the pssible defects which can affect an item. Mrever, these rules are nt inferred frm a slid knwledge-base and use crisp values (i.e., ne f them, states that an ptin must be discarded if its frequency is 0, independently frm the size f the sample). Furthermre, it des nt cntain any experiment which demnstrates the effectiveness f the system in imprving assessment. Nevertheless, this wrk has given us many ideas, and ur wrk can be cnsidered a cntinuatin f it. 7. CONCLUSION In this paper we have presented an e-testing system, capable f imprving the verall quality f the items used by the tutrs, thrugh the re-use f the items acrss subsequent n-line test sessins. Our system s rules use statistical indicatrs frm the IA mdel t measure item quality, t detect anmalies n the items, and t give advise fr their imprvement. Obviusly, the system can

8 nly detect defects which are visible analyzing results f item and distractr analysis indicatrs. The strength f ur system is in the pssibility fr all the tutrs, and nt nly experts f assessment r statistics, t imprve test quality, by discarding r, where pssible, by mdifying defective items. The system has been used at the University f Salern, t assess the students f a curse. This initial experiment has prduced encuraging results, shwing that the system can effectively help the tutrs t btain items which better discriminate between strng and weak students and better match the difficulty estimated by the tutr. Mre accurate experiments, invlving a larger set f items and students, are necessary t effectively measure the system capabilities. Our system perfrms a classificatin f items, carried ut by evaluating fuzzy rules. At present, we are cllecting data n test utcmes. Once a large database f items and learner s answers is available, there will be the pssibility f expliting ther methds f classificatin, based n data, such as hierarchical methds, K-means methds, and crrelatin methds. 8. REFERENCES [1]. Bardssy, A., Duckstein, L., Fuzzy Rule-Based Mdeling with Applicatins t Gephysical, Bilgical, and Engineering Systems. CRC Press [2]. Blackbard [3]. Chen, C.M., Duh, L.J., Liu, C.Y. A Persnalized Curseware Recmmendatin System Based n Fuzzy Item Respnse Thery. In Prceedings f IEEE Int. Cnf. n e-technlgy, e-cmmerce and e-service, 2004, Taipei, Taiwan, pp [4]. Civanlar, M. R., Trussel, H. J. Cnstructing membership functins using statistical data. Fuzzy Sets and Systems , pp [5]. Cstaglila, G., Ferrucci, F., Fuccella, V., Giviale, F., A Web-based Tl fr Assessment and Self- Assessment, in Prceedings f 2nd Internatinal Cnference n Infrmatin Technlgy: Research and Educatin, ITRE 04, pp [6]. ECMAScript. Standard ECMA-262, ECMAScript Language Specificatin, /publicatins/files/ecma- ST/Ecma-262.pdf, 2005 [7]. FCL. Fuzzy Cntrl Prg. Cmmittee Draft CD 1.0 (Rel. 19 Jan 97). [8]. Hambletn R. K., Swaminathan, H. Item Respnse Thery-- Principles und Applicatins, Netherlands: Kluwer Academic Publishers Grup, [9]. Hibernate [10]. jfuzzylgic. Open Surce Fuzzy Lgic (Java) [11]. H, R.G., Yen, Y.C. Design and Evaluatin f an XML-Based Platfrm-Independent Cmputerized Adaptive TestingSystem. IEEE Transactins n Educatin, 2005, VOL. 48, NO. 2,, pp [12]. Hsieh, C.T., Shih, T.K., Chang, W.C., K, W.C. Feedback and Analysis frm Assessment Metadata in E-learning. In Prceedings f 17th Internatinal Cnference n Advanced Infrmatin Netwrking and Applicatins, 2003, Xi'an, China, pp [13]. Hung, J.C., Lin, L.J., Chang, W.C., Shih, T.K., Hsu, H.H., Chang, H.B., Chang, H.P., Huang, K.H., A Cgnitin Assessment Authring System fr E- Learning. In Prc. f 24th Int. Cnf. n Distributed Cmputing Systems Wrkshps 04, pp [14]. Jhnsn, S.C. Hierarchical Clustering Schemes. Psychmetrika 32, 1967, pp [15]. Llyd, S. Least Squares Quantizatin in PCM. IEEE Transactins n Infrmatin Thery. 28, 2. pp , 1982 [16]. Massey. The Relatinship Between the Ppularity f Questins and Their Difficulty Level in Examinatins Which Allw a Chice f Questin. Occasinal Publicatin f The Test Dev. and Res. Unit, Cambridge. [17]. Mdle [18]. Questinmark [19]. Schwartz, M., Task Frce n Bias-Free Language. Guidelines fr Bias-Free Writing. Indiana University Press, Blmingtn, IN, 1995 [20]. Stage, C. A Cmparisn Between Item Analysis Based n Item Respnse Thery and Classical Test Thery. A Study f the SweSAT Subtest READ, www. umu. se/ edmeas/ publikatiner/ pdf/ enr3098sec.pdf, 1999 [21]. Sun, K. T., An Effective Item Selectin Methd fr Educatinal Measurement, In Prceedings f Internatinal Wrkshp n Advanced Learning Technlgies, 2000, pp [22]. Wdfrd, K., Bancrft, P. Multiple Chice Items Nt Cnsidered Harmful, in Prceedings f the 7th Australian cnference n Cmputing educatin, Newcastle, Australia, 2005, pp [23]. Zadeh, L. A. Fuzzy sets and their applicatins t classificatin and clustering. Academic Press (1977).