An Integrated Decision Support Model to Assess Reviewers for Research Project Selection

Transcription

1 202 45th Hawaii International Conference on System Sciences An Integrated Support Model to Assess Reviewers for Research Project Selection Wei Xu School of Information, Renmin University of China, Beijing, 00872, China Wei Du School of Information, Renmin University of China, Beijing, 00872, China Jian Ma Department of Information Systems, City University of Hong Kong, Hong Kong, China Wei Wang School of Information, Renmin University of China, Beijing, 00872, China Qian Liu Division of Finance, Agricultural University of Hebei, Baoding, 0700, China Abstract Reviewers play a significant role in research project selection, because their opinions will have great influence on the decision of the projects which will be funded. Current methods mainly focus on the evaluation of reviewers competitiveness in a certain research areas, while ignoring the performance of reviewers whether is suitable to review a proposal. In this paper, a group decision support model is proposed to assess reviewers performance in peer review process. In the proposed model, competitiveness and relevance of reviewers are considered to evaluate reviewers. It integrates analytic hierarchy process (AHP), scoring method and fuzzy linguistic processing. An illuminating example is used to show the effectiveness of the proposed method. The results find that the proposed method is a reasonable method to assess reviewers, and it can be potentially applied to reviewer evaluation in the National Natural Science Foundation of China (NSFC).. Introduction Research project selection is an important and recurring activity in many organizations, and it is also a challenge task that takes place in a complicated and multi-stage decision-making process. It begins with a call for proposals (CFP), which is distributed to the relevant communities, such as universities and research institutions. Proposals are then submitted to the body (e.g., funding agencies) that issued the CFP. Then these proposals are sent to reviewers for peer review. Reviewers normally review the proposals according to the instructions on the rules and criteria of the funding agency. The review results are collected, and ranked based on the aggregation methods []. For example, there are the following stages for research project selection in the Natural Science Foundation of China (NSFC): proposal submission, proposal grouping, proposal assignment to reviewers, peer review, aggregation of review results, panel evaluation, and final awarding decision [2]. As can be seen from the research project selection process, reviewers play a significant role in research project selection, because their opinions will have great influence on the decision of the projects which will be funded. Previous research deals with general project evaluation and selection process and several formal methods and models are available for this purpose. support methods/systems have been developed for project selection process [2-6]. A fuzzy AHP method is applied to government-sponsored project selection [7]. A fuzzy DEA and knapsack formulation integrated model is offered for project selection [8], while a hybrid fuzzy DEA and Bayesian belief network evaluation model is suggested [9]. Similarly, an AHP and fuzzy TOPSIS method is proposed for project selection [0], and an integrated DEA and balanced scorecard approach is suggested []. In recent years, some specific certain phases to support research project selection are proposed. A multilingual ontology framework is suggested to support research project selection [2]. A hybrid knowledge-based and modeling approach is offered to assign reviewers to proposals for research project selection [3], while a decision support approach is presented for assigning reviewers to grouped proposals, and genetic algorithm is employed to solve the assignment problem [4]. An integrated method for /2 $ IEEE DOI 0.09/HICSS

2 collaborative research project selection is proposed according to competitiveness and collaboration of candidates [5]. A method of optimal allocation of proposals to reviewers is presented in order to facilitate the selection process []. Finally, a group decision support method is suggested to evaluate experts for research project selection [6]. Among the literatures, previous studies mainly focus on general project selection process but not specific process such as expert evaluation. Furthermore, specific certain phases to evaluate reviewers mainly focus on the reviewers competitiveness in certain research areas [6], while ignoring the performance of reviewers whether is suitable to review a proposal. So, this paper contributes an integrated decision support model to assess reviewers in research project selection to cover this research gap. 2. Research background In China, in order to improve scientific or social development and cultivate talents, government sets up a lot of funding agencies, such as National Natural Science Foundation of China (NSFC, http: // NSFC, as one of the largest funding agencies, was set up in 986 with the aim to promote the reform of natural science and technology and push ahead economic and social development in China. As the main way to support the national strategic development of basic and applied basic research, NSFC has received over 5 thousand proposes and funded over 3 thousand projects in 200. According to its functions, NSFC is structured by seven scientific departments, five bureaus and two offices, as well as four units directly under its jurisdiction. The seven departments are responsible for the evaluation and management of projects in various categories, while bureaus, offices and associated units are in charge of policy making, administration and other related affairs. NSFC receives thousands of research proposals from universities and institutions all over the country each year, and then prepares peer reviews and panel evaluation, and selects the proposals of higher potential to grant. There is a strict evaluation system to decide whether fund a proposal or not. First, applicants submit applications to their own institutions for review after the committee release the guides to projects and the notice for accepting applications. Institutions make united reports to NSFC via the Internet-based Science Information System (ISIS, Second, seven scientific departments of NSFC would separately carry out a preliminary examination for proposals. After the examination, the fund management agency would choose several experts as reviewers for peer review. Then, the scientific departments analyze the comments and opinions of the peer review and select the valuable projects. Finally, funding management agency would set up a committee composed of responsible reviewers with high academic achievements and good professional ethics to discuss the approved proposals from the peer review. The qualified projects to be funded by NSFC would be published in this stage. As can be seen from the process, peer review is critical stages, and selecting appropriate reviewers to review their familiar proposals is critically important. However, for a certain proposal, how to measure the performance of reviewers and select some proper experts to review is not well studied. Therefore, this paper proposes a group decision support model to evaluate experts performance in reviewer assignment process. 3. The proposed integrated decision support model In this section, an integrated decision support model is proposed to assess reviewers performance. The overall performance of reviewers can be measured by two basic indicators, namely competitiveness and relevance. On one hand, reviewers competitiveness reveals their professional level. It s undoubted that a reviewer with high professional level and rich experience in project selection will make more fair and scientific judgments on proposals. There are two criteria to measure a reviewer s competitiveness, named objective competitiveness and subjective competitiveness. Objective competitiveness can be measured by three sub-criteria including publications, research projects and historical performance in project selection. Objective information can be collected from expert s database. Subjective information like other experts opinions is used to supplement the insufficiency of objective information. If the reviewer is more competitive, he/she may be more reliable in research project selection. On the other hand, to assign a reviewer to a proposal, the relevance between the research areas of a reviewer and the content of a proposal should be also considered. There are two criteria to measure a reviewer s relevance, named objective relevance and subjective relevance. Objective relevance is determined by two sub-criteria, namely the similarity of publications between a reviewer and an applicant submitting a proposal, and the similarity of a proposal and projects the reviewer funded. Subjective relevance 45

3 consists of three sub-criteria including the similarity of discipline codes between research areas of a reviewer and a proposal, the similarity of abstract between a proposal and projects a reviewer funded, and the similarity of keywords in a reviewer s information and a proposal. The overall performance of relevance can be achieved from the similarity of subjective and objective relevance. The measurement of text similarity can be done using text mining technique and cosine coefficient of text vectors [7-8]. If the research areas of reviewers are higher relevance with the content of proposals, it will be preferred to make more accurate in research project selection. Hence, a hierarchy structure with four levels for assessing reviewer can be shown in Figure. Objective competitiveness Reviewerscompetitiveness Reviewersperformance Subjective competitiveness Reviewersrelevance Objective relevance Subjective relevance Figure The hierarchy structure for reviewer evaluation Research data used in our model can be collected from Scholarmate ( With the collected information, reviewer s performances of competitiveness and relevance can be measured by analytical hierarchy progress (AHP) and fuzzy linguistic representation. Each reviewer s overall performance integrated competitiveness and relevance can be achieved by weighted geometric average method. There are different formats of information collected. Some information like numerical data can be used to calculating directly. Other information like natural language should be converted to the calculable format by appropriate algorithms. Performances of competitiveness and relevance can be then aggregated by using the processed data. The overall performance can be achieved from equilibrium between performances of competitiveness and relevance. Fund management agency would select appropriate reviewers according to their overall performances. The proposed group decision support method to evaluate potential reviewers is summarized as follows. 3.. Measuring a reviewer s competitiveness A reviewer s competitiveness can be divided into objective competitiveness and subjective competitiveness. The calculation process of the reviewer s competitiveness can be decomposed in the following subsections. There are three sub-criteria to measure a reviewer s objective competitiveness including publications, research projects and historical performance in project selection. The details to calculate the reviewer s objective competitiveness are as follows. ) Objective competitiveness in publications Generally, reviewers publish their research results in different academic journals. Their research publications reflect their academic contribution and activity in a certain degree. Three attributes are taken into consideration. The first one is the quantity of publications. It is an important indicator to evaluate a reviewer s performance in publications. The number of publications in one disciplinary field indicates a reviewer s contribution to this field. The second attribute is the quality of the expert s publications. Academic journals are not only classified into different disciplines but also different grades, such as grade A, B and C. The grade of the journal reflects the quality of the article a reviewer publishes. The third one is the time distribution of publications. Since the nearer the date of article an expert published, the more active researcher he/she is. The active reviewer with more publications in recent years will be preferred to select in project evaluation. There are three attributes listed above. The quantity of publications is numerical value. Time distribution, which is presented in the format of date, can be represented by the time interval between publication date and current date. The smaller the value of time interval, the nearer the date of article a reviewer published. The quality of an article can be measured by the grade of the journal. Journals are generally classified into grade A, B and C in China. A represents the top-level journal. A, B or C are not computable values, so it s necessary to determine the weight of each grade. AHP is used to deal with this task for several reasons. First, it is user friendly since users can directly input judgment data without the in depth knowledge of mathematics [9]. Second, relevant inconsistency in managerial judgments is dealt with appropriately [20]. Last, the power of AHP has been validated by empirical application in diverse areas [2]. Group decision making is used in order to minimize the dominance by a strong member. comparisons are made by the group with the grades raging from -9. The ranking scales and explanation are shown in Table. The detailed algorithms of AHP be found in [6, 9-20]. Table The scales for comparison judgments [9] Absolute values Definitions Equal importance 3 Moderate importance of one over another 5 Strong or essential importance of one over 3... Objective competitiveness. 46

4 another 7 Very strong or demonstrated importance of one over another 9 Extreme importance of one over another 2, 4, 6, 8 Intermediate values Reciprocals Reciprocals for inverse comparison Let P represent the reviewer performance in publications. w i (i=, 2, 3) denotes the weight of journal grade (A, B or C). t j (j=, 2,, n )denotes the interval time j between publication time and current time. We use b ij to represent the quantity of academic publications a reviewer publishes in journals with grade w i at time t j. Then each reviewer s performance in publications can be represented by p w b n 3 ( ) i ij j i t j () 2) Objective competitiveness in research projects Various research projects that a reviewer undertakes reveal his/her academic ability. Like reviewer s publications, there are also three attributes to measure a reviewer s performance in research projects he/she had undertaken. One hand, the quantity of research projects reflects a reviewer s experience in R&D projects. The other hand, research projects can also be graded into different levels like journals, so quality of research projects should be taken into consideration. According to the different funding agencies of central government, ministries, provinces and local cities, R&D projects can be classified into the national (N) level, ministry (M) level, provincial (P) level and local city (C) level. Projects with different levels make different contribution and values. Generally, a project with national level funded by central government would make more contribution to the whole country in comparison with projects with other levels. Besides the quantity and quality of research projects, the time distribution of projects a reviewer undertakes is also important. A reviewer undertakes projects in recent years indicate that he/she is still active in research areas, which is helpful in evaluating projects. Processing of information is similar to that of reviewer performance in publications. Let P 2 represent the reviewer performance in research projects. w 2 i (i=, 2, 3, 4) denotes the weight of project level (N, M, P or C). t 2 j (j=, 2,, n )denotes the interval time j between project completion time and current time. We use b 2 ij to represent the quantity of research projects a reviewer undertakes with level w 2 i at time t 2 j. Then each reviewer s performance in research projects can be represented by n p 2 ( w ) 2 i bij j i t j (2) 3) Objective competitiveness in historical performance in project selection A reviewer s historical performance in project selection can reflect his/her evaluation ability in a certain degree. A reviewer may give different evaluation results to various proposed projects at different time, which can be obtained from proposal review database. A reviewer s historical performance in projects evaluation can be measured by three attributes: reviewer s evaluation grade, quality and time distribution. In projects evaluation, a reviewer would choose a final result like A, B, C or D to each project as his/her judgment, which means that the project is excellent, good, moderate or poor. Funding management agency decides whether to fund a project or not according to reviewers evaluation. A reviewer s evaluation grade can be acquired by his/her judgment and final result of project evaluation. If the project is approved finally, then a reviewer who gives grade A will get score 4. It means he/she has high evaluation ability. Reviewers who give grade B, C and D will respectively get score 3, 2,. If the proposed project fails, reviewers score is zero. Projects quality represented by levels of N, M, P and C has listed above. The time interval between the evaluation date and today negatively reflects the reviewer s performance. Let P 3 represent the reviewer s historical experience in project selection. W 3 i (i=, 2, 3, 4) denotes the weight of project level (N, M, P or C). t 3 j (j=, 2,, n)denotes the interval time j between project evaluation time and current time. M denotes the number of projects that a reviewer evaluated in a given period. We use b 3 ijk to represent the score of evaluation on project k (k=, 2,,m )with level w 3 i at time t 3 j. Then each reviewer s historical performance in m projects can be obtained. A reviewer s historical performance in project evaluation can be represented by average score, the formula is m 3 3 p 3 ( w ) 3 i bijk m k tj (3) 4) Objective competitiveness integration The performance of objective competitiveness P can be concluded from above three sub-criteria. The normalization of criterion values is used as for commensurability between various criteria. According to the existing method [22], the normalization of each sub-criterion is obtained by following formulas: For benefit criteria, p p, p 0; i,2,, l ' i i i(max) p i(max) For cost criteria, p p, p 0; i,2,, l ' i(min) i i p i 47

5 AHP is used to determine the relative importance of these three sub-criteria. Let v i denotes the weight of sub-criterion i (i=, 2, 3), P can be computed by the formula 3 ' p ( v ipi) (4) i Subjective competitiveness. Objective information listed above is still not enough to measure a reviewer s overall performance. Other indicators like citation times and other expert s evaluation, which are important to access a reviewer s academic level, have not been used when measure a reviewer s performance in publications. Besides, a reviewer s performance in research projects he/she participated has not been considered. However, it s not easy to collect information for these indicators in China. So, subjective information like other peer experts opinions is a necessary supplement. Other experts opinions can be represented by their judgment to a reviewer like low, middle, high or other linguistic values. In order to quantify other experts opinions, we will use 2-tuple fuzzy linguistic representation model to convert other experts linguistic comments to numerical values [23]. Suppose S is an ordered set with seven scales to grade a reviewer, namely S={s :VL(very low), s 2 :L(low), s 3 :ML(more or less low), s 4 :M(middle), s 5 :MH(more or less high), s 6 :H(high), s 7 :VH(very high)}. The ranks and grades of importance are listed in Table 2. The detailed algorithms can be found in [6, 23]. Table 2 The seven scales and grade of importance [23] Seven scales of reviewer grading S =VL: very low S 2 =L: low S 3 =ML: more or less low S 4 =M: middle S 5 =MH: more or less high S 6 =H: high S 7 =VH: very high Suppose m experts have participated in the evaluation of a reviewer. 2-tuple (s i, а i ) (i=, 2,, m)is used to represent the linguistic evaluation of expert i. Then a group of 2-tuples (s i, а i ) can be transformed into numerical values by function Δ -. The average score of a reviewer grading by other experts can be computed by m p2 ( si, ai); (5) m i s S, a [ 0.5,0.5) i i Calculating a reviewer s overall competitiveness Since the performance of objective competitiveness and subjective competitiveness are measured, a reviewer s overall competitiveness can be obtained from the integration of these two criteria. The normalization of criterion is stated above. AHP is used to determine weight of each criteria v i (i=, 2). The overall evaluation of reviewer s competitiveness p (com) can be represented by 2 ( com) ' p ( v ipi) (6) i 3.2. Measuring a reviewer relevance A reviewer s relevance can be divided into objective relevance and subjective relevance. The calculation process of the reviewer s relevance can be decomposed in the following subsections Objective Relevance There are two sub-criteria to measure a reviewer s objective relevance including the similarity of publications between a reviewer and an applicant submitting a proposal, and the similarity of a proposal and projects the reviewer funded. The details to calculate the reviewer s objective relevance are as follows. ) The similarity of publications between a reviewer and an applicant Publications that a reviewer published reveal his/her research areas. Meanwhile, publications in applicant s proposal also show his/her research areas. The similarity of publications between a reviewer and an applicant can represent the consistency of their research areas. Let P 2 represent the similarity of publications between a reviewer and an applicant. Suppose n publications a reviewer published, and l publications an applicant published. We use c ij to represent the similarity of ith publication by a reviewer and jth publication by an applicant. Then the similarity can be calculated by p2 max{ ij},,...,,,..., c i n j l (7) 2) The similarity of a proposal and projects the reviewer funded Like the similarity of the publications, the similarity of a proposal and projects the reviewer funded can also show the consistency of their research areas. Processing of information is similar to that of 48

6 reviewer relevance in publications. Let P 22 represent the similarity of a proposal and projects the reviewer funded. Suppose m projects a reviewer funded. We use c 2 i to represent the similarity of ith project a reviewer funded and a proposal. Then the similarity can be calculated by 2 p22 max{ i },,..., c i m (8) AHP is used to determine the relative importance of these two sub-criteria. Let v 2i denotes the weight of sub-criterion i (i=, 2, 3), P 2 can be computed by the formula 2 p2 ( v2 ip2i) (9) i Subjective Relevance There are three sub-criteria to measure a reviewer s subjective relevance including the similarity of discipline codes between research areas of a reviewer and a proposal, the similarity of abstract between a proposal and projects a reviewer funded, and the similarity of keywords in a reviewer s information and a proposal. The details to calculate the reviewer s subjective relevance are as follows. ) The similarity of discipline codes between research areas of a reviewer and a proposal When an expert is as a reviewer, expert can fills in two or more familiar discipline codes. Also, applicant fills in two discipline codes in submitted proposal. The similarity of discipline codes between a reviewer and a proposal can represent the consistency of their research areas. Let P 22 represent the similarity of discipline codes between research areas of a reviewer and a proposal. Suppose dr the first discipline code and dr 2 the second discipline code filled in by reviewers, and dp the first discipline code and dp 2 the second discipline code in proposal. Then the similarity can be calculated by dr dp, dr2 dp2 0.8 dr dp2, dr2 dp 0.7 dr dp, dr2 dp2 (0) p dr dp2, dr2 dp2 0.6 dr2 dp, dr dp2 0.5 dr2 dp2, dr dp 0 otherwise 2) The similarity of abstract between a proposal and projects a reviewer funded The similarity of abstract between a proposal and projects the reviewer funded can also show the consistency of their research areas. Let P 222 represent the similarity of a proposal and projects the reviewer funded. Suppose m projects a reviewer funded. We use c 2 i to represent the similarity of ith project a reviewer funded and a proposal. Then the similarity can be calculated by 3 p222 max{ i },,..., c i m () 3) The similarity of keywords in a reviewer s information and a proposal Let P 223 represent the similarity of keywords in a reviewer s information and a proposal. Suppose the keyword set in reviewer s information S, and the keyword set in proposal S 2. Then the similarity can be calculated by # S S p223 2 (2) # S S 2 where # represent the number of the keywords in each keyword set. AHP is used to determine the relative importance of these three sub-criteria. Let v 22i denotes the weight of sub-criterion i (i=, 2, 3), P 22 can be computed by the formula 3 p22 ( v22ip22i) (3) i Calculating a reviewer s overall relevance Since the performance of objective relevance and subjective relevance are measured, a reviewer s overall relevance can be obtained from the integration of these two criteria. AHP is used to determine weight of each criteria v 2i (i=, 2). The overall evaluation of reviewer s relevance p (rel) can be represented by 2 ( rel) p ( v2ip2i) (4) i 3.3. A Reviewer s Overall Performance The performances of competitiveness and relevance of a reviewer can be measured respectively by above steps and formulas. Thus we can obtain the overall performance of a reviewer by weighted geometric mean method. The formula is as follows: p p p ( com) ( rel) (5) 4. An illuminating example As stated above, applicants submit proposals to NSFC via ISIS. NSFC focuses on the project selection, but informal reviewer selection throw doubt on the credibility of projects selection. Therefore, we 49

7 proposed an integrated model to solve this problem in this paper. The validation and potential application of the proposed model will be illustrated. Research data can be acquired from Scholarmate ( Other information like experts opinions has been gathered. Since there are different emphases on subjective and objective information in the measurement of competitiveness and relevance, we will select three decision (DMs) respectively to determine the weights of these two criteria in each dimension. Besides, different disciplines have different focuses, it is necessary to organize panels with different disciplines to determine the different weights of sub-criteria. Priorities of journal grade and project level are also determined by AHP. 4.. Determining the weights of objective criterion and subjective criterion A reviewer s overall performance can be obtained from the equilibrium between competitiveness and relevance. Each indicator can be measured respectively by two criteria: objective criterion and subjective criterion. Different disciplines have different focuses on these two criteria of each indicator in NSFC. So the first step is to determine the relative importance of these two criteria in different disciplines. As stated above, v i (i=, 2) denote the weights of objective competitiveness and subjective competitiveness, v 2i (i=, 2) denote the weights of objective relevance and subjective relevance. A panel with three DMs in a certain discipline has been organized to give pair-wise comparison matrices for the criteria. Two sets of matrices for two indicators are shown in Table 3 and Table 4. The final results are obtained from the three DMs judgments by weighted geometric mean method. According to the algorithm of AHP, the weights of criterion on objective competitiveness and subjective competitiveness can be obtained as v = and v 2 = respectively, the consistency ratio (CR) is equal to 0. Similarly, v 2 =0.3020, v 22 =0.6980, CR =0. Table 3 comparison matrices on objective and subjective competitiveness DM DM2 DM3 comparison matrices Table 4 comparison matrices on objective and subjective relevance DM DM2 DM3 comparison matrices Determining the weights of sub-criteria for each criterion According to the hierarchy structure, there are several sub-criteria for each criterion. Objective competitiveness can be measured by three sub-criteria: publications, research projects and historical performance in project selection. Other experts opinions represent subjective competitiveness. Objective relevance is determined by publication similarity and project similarity. Subjective relevance consists of three sub-criteria: discipline area, abstract and keywords. The same DMs give three sets of pairwise comparison matrices for various sub-criteria for each criterion as shown in Table 5, Table 6 and Table 7. The weights of publications, research projects and historical performance can be derived as v =0.4284, v 2 = and v 3 =0.3250, CR is equal to less than 0..The importance of publication similarity and project similarity can be derived as v 2 = and v 22 =0.4425, CR=0. The importance of discipline area, abstract and keywords can be derived as v 22 =0.7494, v 222 =0.474 and v 223 =0.03, CR=0.0. Table 5 comparison matrices on publications, projects and historical performance DM DM2 DM3 comparison matrices Table 6 comparison matrices on publication similarity and project similarity DM DM2 DM3 comparison matrices 2 2 Table 7 comparison matrices on discipline area, abstract and keywords DM DM2 DM comparison 5 2 matrices

8 4.3. Determining the weights of journal grades and project levels Similarly, DMs give pair-wise comparison matrices for the different grades of journals and different levels of projects as shown in Table 8 and Table 9. w i (i=, 2, 3) denotes the weight of journal grade (A, B or C) while w 2 i (i=, 2, 3, 4) denotes the weight of project level (N, M, P or C). The relative importance of journal grade A, B and C can be derived as w =0.6946, w 2= and w 3=0.09. CR is equal to The relative importance of nation, ministry, province and city level of projects can be derived as w 2 =0.608, w 2 2=0.2422, w 2 3=0.093 and w 2 4= CR is equal to Table 8 comparison matrices on journal grade A, B and C DM DM2 DM comparison matrices Table 9 comparison matrices on project level N, M, P and C DM DM2 DM comparison matrices Collecting information of the reviewer and proposal with its applicant The information of the candidate reviewer in recent five years and the proposal with its applicant will be used to evaluate the reviewer s overall performance. We select an expert as a candidate reviewer from the database in NSFC. The detailed information can be collected as follows. The reviewer s publication information including time distribution, quality and quantity is listed in Table 0. Information of research proposal the reviewer undertook is presented in Table. Table 2 contains the historical performance of the reviewer in project evaluation. There are five other experts giving their opinions to this reviewer as shown in Table 3. The publication similarities as well as project similarities between the reviewer and the applicant are listed in Table 4 and Table 5. Table 6 contains their discipline codes and similarities. Information of abstract similarity has been listed in Table 7. The similarity of keywords between the reviewer and the applicant are shown in Table 8. Table 0 The reviewer s publication information Time distribution 2007 / / /3 200 /2 20 Grade A 0 0 Grade B Grade C 0 0 Table The information of projects reviewer had undertaken Time distribution 2007 / / /3 200 /2 Nation Ministry Province 0 0 City Table 2 The reviewer s historical performance in project evaluation Time distribution 2007 / / /3 200 /2 20 Nation -- (A,Y) (B,Y) Ministry (B,Y) -- (A,Y) Province (C,N) (B,Y) -- (B,Y) (D,N) City (C,Y) (D,N) -- Table 3. Other experts opinions Experts E E2 E3 E4 E5 Their opinions MH H VH M H Publications of reviewer Table 4 The publication similarity matrix Publications of applicant A A 2 A 3 A 4 A 5 A 6 R R R R R R R R R R Table 5 The project similarity matrix Proposal R R R R R R R R R R projects reviewer had undertaken 42

9 Table 6 The relevance in discipline area Discipline codes of the reviewer R R 2 Applicant A 0 A 2 0 Reviewer s historical proposals Table 7 The abstract similarity matrix Proposal P P P P P P P P P P Table 8 The reviewer s research areas and proposal s keywords keywords electronic commerce, information reviewer technology, economic management, enterprises information information technology, computer proposal science, enterprises information, data mining 4.5. Evaluating reviewers According to formulas in section 3, the relevant performance of each sub-criterion is P =.683, P 2 =.780, P 3 =0.3659, P 2 =4.6, P 2 =0.6388, P 22 =0.489, P 22 =, P 222 =0.5623, P 223 = Then the reviewer s performance in competitiveness can be obtained as P (com) = (0<P (com) <) through normalization and AHP method. The numerical value of relevance can be computed as P (rel) = (0<P (rel) <). Therefore, his/her overall performance is derived as P=0.59. Then other candidate reviewers overall performance can be calculated similarly. According to the ranking order of reviewers, we can select appropriate reviewers attending the project evaluation. 5. Conclusions This paper presents an integrated decision support model for evaluating reviewer s performance. In the proposed model, competitiveness and relevance of reviewers are used to evaluate reviewers, and AHP, scoring method and fuzzy linguistic processing are employed to measure reviewers competitiveness and relevance. The experimental results showed that the proposed method improved the quality of reviewer evaluation. Also, the proposed method can be used to expedite the proposal assignment process in the NSFC and elsewhere. (It is an extension of the Internet-based Science Information System ISIS, The proposed method can also be used in other government research funding agencies. Future work can be done to empirically validate the results of reviewer evaluation. Also, there is a need to assign research proposals to reviewers systematically. Finally, the method can be expanded to help finding a better match between proposals and reviewers. Acknowledgement The authors would like to thank the minitrack chair and the anonymous reviewers for their valuable comments and suggestions which have helped immensely in improving the quality of the paper. This work was supported in part by National Natural Science Foundation of China under Grant (No ), and the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China. References [] W.D. Cook, B. Golany, M. Kress, M. Penn, and T. Raviv, Optimal allocation of proposals to reviewers to facilitate effective ranking, Management Science, Vol. 5, No. 4, pp , Apr [2] Q. Tian, J. Ma, and O. Liu, A hybrid knowledge and model system for R&D project selection, Expert Systems with Applications, Vol. 23, No. 3, pp , Oct [3] Q. Tian, J. Ma, J. Liang, R. Kowk, O. Liu, and Q. Zhang, An organizational decision support system for effective R&D project selection, Support Systems, Vol. 39, No. 3, pp , May [4] F. Ghasemzadeh and N. P. Archer, Project portfolio selection through decision support, Support Systems, Vol. 29, No., pp , Jul [5] P.V. Chu, Y.Hsu, and M. Fehling, A decision support system for project portfolio selection, Computers in Industry, Vol. 32, pp. 4-49, 996. [6] J. Klapka, and P. Pinos, support system for multicriterial R&D and information systems projects selection, European Journal of Operational Research, Vol. 40, pp , [7] C. Huang, P. Chub, and Y. Chiang, A fuzzy AHP application in government-sponsored R&D project selection, Omega, Vol. 36, pp , [8] P. Chang, and J. Lee, A fuzzy DEA and knapsack formulation integrated model for project selection, Appeared in Computers & Operations Research, 20. [9] T. Chiang, and Z.H. Che, A fuzzy robust evaluation model for selecting and ranking NPD projects using Bayesian 422

10 belief network and weight-restricted DEA, Expert Systems with Applications, Vol. 37, pp , 200. [0] M.P. Amiri, Project selection for oil-fields development by using the AHP and fuzzy TOPSIS methods, Expert Systems with Applications, Vol. 37, pp , 200. [] H. Eilat, B. Golany, and A. Shtub, R&D project evaluation: an integrated DEA and balanced scorecard approach, Omega, Vol. 36, pp , [2] O. Liu, and J. Ma, A multilingual ontology framework for R&D project management systems, Expert Systems with Applications, Vol. 37, pp , 200. [3] Y. Sun, J. Ma, Z. Fan, J. Wang, A hybrid knowledge and model approach for reviewer assignment, Proceedings of the 40th Hawaii International Conference on System Sciences, pp.-0, [4] Y. Xu, J. Ma, Y. Sun, G. Hao, W. Xu, and D. Zhao, A decision support approach for assigning reviewers to proposals, Expert Systems with Applications, Vol. 37, pp , 200. [5] B. Feng, J. Ma, and Z. Fan, An integrated method for collaborative R&D project selection: supporting innovative research teams, Expert Systems with Applications, Vol. 38, pp , 20. [6] Y. Sun, J. Ma, Z. Fan, and J. Wang, A group decision support approach to evaluate experts for R&D project selection, IEEE Transactions on Engineering Management, Vol. 55, No., pp , Feb [7] Feldman, R. and J. Sanger, The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data. Cambridge, New York, [8] S. Hettich and M. Pazzani, Mining for proposal reviewers: lessons learned at the national science foundation, in Proc. 2th Int. Conf. Knowl. Discov. and Data Mining, pp , [9] T.L. Saaty and L.G. Vargas, Prediction, Projection and Forecasting. Kluwer Academic Publishers, Dordrecht, 99. [20] T. L. Saaty, A scaling method for priorities in hierarchical structures, Journal of Mathematical Psychology, Vol. 5, No. 3, pp , June 977. [2] J. Aczel, and T.L. Saaty, Procedures for synthesizing ratio judgements, Journal of Mathematical Psychology, Vol. 27, No., pp , March 983. [22] C.L. Hwang, and K. Yoon, Multiple Attribute Making: Methods and Applications. Springer, Berlin, 98. [23] R. C. Wang and S. J. Chuu, Group decision-making using a fuzzy linguistic approach for evaluating the flexibility in a manufacturing system, European Journal of Operational Research, Vol. 54, No. 3, pp ,