Indicators for Measuring University Research Performance in Thailand

Transcription

1 Indicators for Measuring University Research Performance in Thailand Somchai Numprasertchai Department of Computer Engineering, Faculty of Engineering, Kasetsart University, Chatujak, Bangkok, THAILAND Abstract Organizations have established many R&D performance indicators for their unique needs. Large research organizations in the developed countries use publication counts, citations, patent counts, and patent citations as key R&D performance indicators. In developing countries research organizations have different research objectives. The appropriate R&D performance indicators should serve several factors such as research objectives, funding agencies requirements, industries needs, and national research policy. The university research units in developing countries have many limitations compared to those in developed countries. The objective of this study is to determine the appropriate R&D performance indicators for university science and technology (S&T) research units in Thailand, where are the main organizations to conduct S&T research. Key elements of measurement indicators are investigated through in-depth interviews with researchers, funding agencies and experts and through a national survey. This study found that researchers and funding agencies use different indicators to evaluate the performance of research programs. Most Thai researchers, especially young researchers value highly those research projects that lead to publication in international peer reviewed journals. They perceive the impact of their research on the society and economy as less important, whereas, research funding agencies value highly the impact of projects and support on the country s development. Barbara Igel School of Management, Asian Institute of Technology Klong Luang, Pathumthani, THAILAND. igel@ait.ac.th Keywords R&D measurement indicators, University R&D. INTRODUCTION Research organizations have established many R&D performance indicators for their unique needs. In the developed countries, publication counts, citations, patent counts, and patent citations as key indicators are generally used in the large research organizations. In developing countries the indicators are established for different purposes such as national development issues or national problem solving. Therefore, appropriate R&D indicators should reflect several factors such as research objectives, funding agency requirements, industry needs, country s needs, and national research policy. In the developing countries the research organizations have more limitations than those of the developed countries and newly industrialized countries. For example, small budgets are allocated for R&D. Latest statistic available shows that in 2001, the total R&D budgets of Indonesia, Philippines, Thailand, and Malaysia were 0.04, 0.08, 0.27, and 0.49 percent of the GDP respectively. While the corresponding rates in newly industrialized countries were 2.92 percent in Korea, 2.21 percent in Singapore, 2.16 percent in Taiwan. Leading economies invested 2.80 percent in the US, 2.98 percent in Japan, and 3.65 percent in Sweden (IMD, 2003). As different research resources, to measure the research performance should be used the different indicators. In this study, we aim at determining of R&D performance indicators for university research organizations in a developing country, such as Thailand. The results of this study should be useful suggestions and guidelines for research funding agencies, researchers, and research evaluators to design appropriate R&D performance indicators. LITERATURE REVIEW The research performance indicators suggested in previous studies can be categorized into indicators to evaluate the technological and commercial results. The technological results consist of awards, honors, patents, products, publications, citations, and time saving [Kim and Oh, 2002; Cordero, 1990; Hauser and Zettelmeyer, 1997; Werner and Souder, 1997; Liebowitz and Suen, 2000;]. Publication is a sign of innovation, while a patent is a sign that innovation has a market potential. Citation analysis is used to measure the frequency of citation of publications and patents. Honors and awards to researchers from national academies and government committees indicate a high potential of researchers and excellent research performance [Werner and Souder, 1997; Cordero, 1990]. The commercial results consist of the number of new products [Liebowitz and Suen, 2000], product success rate, sales volume and market share [Cordero, 1990; Hauser and Zettelmeyer, 1997; Ahmed, 1998;], and cost saving ratio [Dressler, Wood, and Alvarez, 1999].

2 Most R&D performance indicators proposed in the literature were constructed for large companies and leading research institutes in developed countries, that have research resources, objectives and goals quite different from research organizations in the developing countries. Unlike in developed country the different performance indicators would be selected to serve economic and social needs of the developing country. In Thailand, the R&D performance indicators are determined by universities and funding organizations. It is found that most indicators are similar to those used by companies and research institutes in developed countries. However some funding organizations in Thailand established performance indicators based on their objectives and research program types. For example, the performance of basic research is measured in terms of the number of publications in leading journals. The performance of development and engineering research is measured by patents and contribution to practical use. Unfortunately, some indicators, which include the summation of impact factors of paper published in scientific journals in ISI database, cannot predict the achievement of research organizations that operate under time and resource limitations. METHODOLOGY This study focuses on determination of appropriate indicators to measure the performance of university research organizations. It was initiated by collecting data on the research performance indicators currently used by three main research funding agencies and four leading public universities in Thailand. The commonly applied indicators obtained from these organizations are analyzed and compared with indicators proposed in the literature. To prevent bias from data collection, the performance indicators were discussed and analyzed with four researchers from large public universities and four staff members and executives of the three biggest funding agencies. The selected indicators were evaluated in a national survey with a large number of university researchers working in science and technology department. The survey questionnaires were distributed to researchers who attended a national information technology conference in Chonburi and two national research seminars in Bangkok. In addition, the survey questionnaire was sent directly to researchers in the large public universities by electronic mails through snowball technique. The survey results were assessed and validated with two experts who have long experience in conducting research and evaluating research performance in universities and funding agencies. RESULTS AND DISCUSSION Performance indicators currently used by research funding agencies and universities In Thailand the evaluation of the research performance has three phases, namely (1) pre-audit research evaluation for selecting proposals with potential and interesting projects, (2) formative evaluation for tracking the status and progress of research projects, and (3) post-audit research evaluation for evaluating the overall performance in terms of research output and its usefulness. This study focuses on the postaudit evaluation of research projects. There are two types of the post-audit indicators used by the three main funding agencies and four leading public universities, qualitative and quantitative indicators. Qualitative indicators measure the research performance in terms of output quality and contribution, while quantitative indicators measure the immediate research results that can be counted such as the number of publications, patents, prototypes, and new products, processes, or services. Generally, universities and funding agencies in Thailand use a combination of qualitative and quantitative measures for assessing research performance. The qualitative measurement is based on opinion of experts who are university professors, senior researchers, and board members of the funding agencies or universities. Some universities and funding agencies apply only quantitative measures to eliminate the potential bias of expert opinions in the final evaluation system. These quantitative indicators used in the different funding agencies are of different significance. The level of significance depends on the organizational policy and specific research objectives as shown in the Table 1. Table 1. Performance indicators currently used in research funding agency & universities in Thailand Research results F1 U1 U2 U3 U4 Publications - National Proceedings National Journals Region & Int. Proceedings Int. Journals in ISI lists Patents/ Copyrights - Patents/ Copyrights in Thailand - Patents/ Copyrights in US, Germany, Japan New products, processes, and services 30 N/A N/A N/A N/A 50 N/A N/A N/A N/A 25 N/A N/A N/A 40 The quantitative measurement indicators consist of publication counts, patents/copyrights counts, and number of new products, processes, and services. It shows that publications in leading scientific journals, especially publications in journals lists by the Insti-

3 tute for Scientific Information (ISI) are considered as more important. Patents and copyright are also considered as the most important indicators. However, patents and copyrights are not included in quantitative indicator lists of universities. This is because most universities have no major interest in the commercial performance of their research results. The research performance in terms of new products, processes, and services cannot be compared across different projects. Some projects may produce several new small products, process, and/or services, while others achieve only one product, but it might have a high contribution to the country s development agenda. As a results the performance of new products, processes, and services are measured in terms of quality rather than the numbers. Neither funding agencies nor universities define official performance indicator for honors and awards. However, researchers can use the honors and awards obtained from universities, national academies, government committee, and international research organizations or foundations to demonstrate high potential and quality of their research achievement. All public funding agencies have designed research policies to assess the research contributions. Measuring the research contribution is difficult because it is time consuming and costly to apply several dimensions of measures such as contribution to education, economic and social development, environmental, and health. Thus some funding agencies were not interested in measuring the wider contribution of research programs. The latest information available about the evaluation research project contribution, financed by the national research budget in 1999 and evaluated by the National Research Council of Thailand in 2003 is shown in Table 2. Table 2. Research contribution from R&D in 1999 Rank Research contribution percents 1 Education Environment Social Economic Organizational management Other (technology transfer, cultural support, agricultural support) No impact Source: Research Contribution Evaluation Report (NRCT, 2003) In general the research contribution from university research in Thailand is quite low because many projects were not publicized and implemented. According to the study the research contribution to education is highest since university professors usually integrate their research with educational duties. Indicators from different perspectives Evaluation of funding agencies perspective Based in Thailand, a developing country, all funding agencies focus on research contributions that serve the country s needs such as solving national problems, improving the quality of life, increasing national competitiveness, and so on. Since most funding agencies encounter several problems in the measurement of research outcomes, indicators measuring the direct output are used instead. Especially, publishing research results in high-ranking journals is a very important indicator for assessing university research performance. Table 3. Indicators and objectives defined by the funding agencies in Thailand Indicators Indicator lists Objectives Outputs Outcomes & Impact Publications, prototypes, patents, new products, processes, services Contribution to the country in terms of cultural, economic, social, educational goals, the environment, health, and so on. Measuring the quality of research outputs in the short term. Solve the country s problems, support and create national competitiveness in R&D. Some funding agencies have measured the research quality through publication counts and the number of impact factors of journals on the ISI lists. Quality of research is usually measured by ranking of journal and the number of citations by other researchers. Measuring by public in high ranking journals had rather widely for use in advanced research programs because it is hard to find other experts to evaluate the performance of advanced research in term of quality and results Some funding agencies have established the different specific performance indicators for different research program types. For example industrial R&D is measured by commercial potential of prototypes, patents, financial returns from selling patents. Long-term basic research is measured by publication counts, the quality of scientific journals, and the number of citations. Researchers perspective The researchers perspective is analyzed based on the 116 completed questionnaires returned: 90 from participants in the national conference and two national research seminars and 26 from university researchers who replied by electronic mail. The survey result is presented in Table 4. Most Thai researchers prefer the number and quality of publication as the main research performance measures. Especially young researchers value highly those research projects that lead to publications in leading international journals and conference proceedings. This performance can be contributed to a better academic reputation and higher job positions. On the contrary they perceive the impact of their

4 research on the society and economy as less important. This is one of several reasons why Thai researchers concentrate their efforts on publications rather than other research results. Table 4. Performance Indicators preferred by university researchers Respondentcents per- Rank Indicators 1 Publications Quality of publication (indicated by impact factor) 3 Award & honor New products,processes, or services Patents Commercial Products Publication citations Contribution to economic development Contribution to environment conservation Contribution to social development Total respondents Some senior researchers argued: Writing research for publishing in international journals is not suitable for Thai researchers. It requires time and a bigger budget for writing publications and only few benefits from these results return to the country. We consider as more appropriate publications those in national conference proceedings and journals. It will help to improve the quality of research in Thailand. Less than forty-four percents of Thai researchers focus their research on contribution to economic and social development, and the environment. This is because most researchers do not have sufficient potential to publicize their research in journals that are assessed by ISI for their research impacts. As most researchers do not believe that funding agencies have set up policies and processes to evaluate the research impact, they prefer as performance measure the publications in famous conference proceedings and journals, research prototypes, and new commercialized products, rather than measuring research contributions to the country. Assessment of survey results by Experts It is hard to determine appropriate R&D performance indicators for university research organizations in Thailand since Thai researchers, universities, and funding agencies have different research objectives. If contribution is used as performance indicator, only performance from experience researchers and researchers who collaborate with the industry can be evaluated due to lack of clear measures for research impact. However, the contribution measure is also suitable for high investment and long-term research. Although some funding agencies try to use a set of different indicators, such as patents and commercial potential, publications is still used as a key performance indicator. Thus it was not surprising for the 2 experts that Thai researchers focus only on publication counts and quality of journals rather than other research results. Most university researchers do not focus their efforts on creating new commercial products, processes, and services, but usually conduct research for exploring new knowledge for academic purposes. Most research projects stop when they have completed research prototypes or published in international journals. Therefore, the research indicators in terms of prototypes are more frequently used than number of patents and commercialized products, processes, and services. Nevertheless, Thai researchers consider that citation of publications as one of the important indicators for measuring their research performance. Only experienced and senior researchers have excellent publications that are referred to by other scientist and researchers. By using citations as the performance indicator the percentage of researchers is more limited when using of number of publications. However, the citation is also used as one of important indicators for selecting the excellent Thai researchers and scientist for the National Researcher Award and the National Scientist Award. Research experience, budgets, and time are also considered as important factors to evaluate research performance. Experienced researchers have high advantages compared with young researchers. Projects which have limited resources can not compete with projects which have generous support. Besides the quantitative measurement, the research performance should be evaluated through self-evaluation, peerevaluation, and other experts. Although some researchers did not have own publications, they had developed new knowledge and created ideas that contributed to other Thai researchers projects. Researcher should evaluate their R&D performance in comparing with the project s objectives, their own goals, and other factors such as supporting university s mission. Moreover, the performance of a researcher is also evaluated by other researchers and experts in related research areas. CONCLUSION & RECOMMENDATIONS The study shows that the research objectives of Thai researchers and funding agencies are quite different. Many researchers did the research to achieve publications but did not concentrate on the country s needs. Many did not even know the importance of R&D performance indicators used by funding agencies, and some had complaints regarding these indicators. Most Thai researchers consider publications in the quantitative dimension as a reliable performance indicator, while the universities and funding agencies prefer to use research contribution that is a qualitative measure. To achieve the research goals, the funding agencies, universities and researcher

5 should have shared research objectives and common measurement dimensions. To be effective, both qualitative and quantitative measures should be integrated and applied in a composite set of R&D performance indicators used by universities and funding agencies. This approach may help researchers, universities, and funding agencies to identify common research objectives and achieve research goals that contribute to the country s needs (see Table 5). Table 5. Proposed composite R&D performance indicators for university R&D Dimension Qualitative measurement Quantitative measurement Indicator Research quality evaluated by researcher (self-evaluation) Research quality evaluated by broad of research committee (peer-evaluation and experts) Research impacts to economic and social development, education, and the environment Publications - Technical/ Research manuals - National Proceedings/ Journals - Region & Int. Proceedings - Region & Int. Journals - Int. Journals in ISI lists Patents/ Copyrights - Patents/ Copyrights in Thailand - Patents/ Copyrights in the US, Germany, Japan Qualitative measurement is the process to evaluate the quality of research results such as prototypes, potential of new commercialized products, processes, and services. Final results are also benchmarked with other related research programs in this dimension. Qualitative measurement should be done on three levels: self-evaluation by researchers, peerevaluation by other researchers with expertise in the related fields, and experts opinions from distinguish researchers in the same and different research fields together within executives of the funding organizations. In addition, research contributions to economic and social development, education, and the environment should be evaluated by experts in each area. Useful quantitative performance indicators, which evaluate the direct research outputs, include the number of publications and patents. Especially, universities and funding agencies should assign a higher level of significance to patents. This is because it will motivate researchers to design and conduct research to meet market needs. However, each university can include different priorities in each composite R&D performance indicator based on the university s mission. This is because different universities serve the country s needs in the different ways. For example, some universities concentrate on the socio-economic contribution, and thus may assign a higher priority to qualitative indicators. While a research university, which aims at creating new knowledge, can measure research performance through publications and patents. Therefore, they may choose publications and patents as the most significant R&D performance indicators. However, universities have to use, evaluate, and redesign suitable performance indicators to match their research missions. ACKNOWLEDGMENTS The authors are grateful for helpful comments and suggestions received from distinguish researchers and executives of research funding agencies. We would like to thank Prof. A. Krawtrakul and her team for providing updated information about research in Thailand. We would like to thank all funding agencies, staff, and researchers who gave us information, suggestions, and comments. REFERENCES [1] Ahmed, P. K. (1998). Benchmarking innovation best practice. Benchmarking for Quality Management and Technology, 5(1): [2] Cordero, R. (1990). The measurement of innovation performance in the firm: An overview. Research Policy, 19(2): [3] Dressler, R., R. S. Wood, and V. Alvarez. (1999). Evaluating R&D performance using the cost savings metric.research Technology Management, 42(2): [4] Hauser, J. R. and F. Zettelmeyer. (1997). Metrics to Evaluate R, D &E. Research Technology Management, 40(4): [5] IMD (2003). The World Competitiveness Yearbook Switzerland, Institute for Management Development (IMD). [6] Kim, B. and H. Oh (2002). An effective R&D performance measurement system: survey of Korean R&D researchers. Omega, 30(1): [7] Liebowitz, J. and C.Y. Suen. (2000). Developing knowledge management metrics for measuring intellectual capital. Journal of Intellectual Capital, 1(1): [8] National Research Council of Thailand (2003). Report of the Project evaluation of Government agencies and State enterprises from the R&D budget in the year 1999, Department of Project Analysis and Evaluation, The National Research Council of Thailand: 40 [9] Werner, B. M. and W. E. Souder (1997). Measuring R&D Performance -State of the Art. Research Technology Management, 40(2): [10] Thailand Research Fund. (2004) Critical! Basic Research in Science in Thailand. Available on