Key Performance Indicators for Quality Assurance in Higher Education the Case of the Department of Informatics at the Technological Educational Institute of Thessaloniki, Greece Kerstin V. Siakas, Aristea-Alexandra Prigkou, Stergios Draganidis Technological Educational Institute of Thessaloniki, Department of Informatics, P.O. Box 141, GR-57400 Thessaloniki, Greece E-mail: siaka@it.teithe.gr, praral@it.teithe.gr, sdragan@it.teithe.gr Abstract The challenge of a united Europe has reached to the roots of European societies, namely to education. European countries have entered the process of aligning not only their economies, their legal and institutional systems, but also of creating an open European higher education area, a framework that is expected to enable closer cooperation between higher education institutions, facilitate student and staff mobility and increase both the competitiveness of Europeans in the international labour market and the attractiveness of European higher education in the world. An important objective of the Bologna Process is to move higher education in Europe towards a more transparent and mutually recognised system and furthermore to place the diversified national systems into a common frame. One of the objectives of the Bologna Process is the establishment of quality assurance including evaluation of programmes or institutions in the form of internal assessment together with external reviews, with the participation of students and the publication of results. Within this frame, the department of Informatics at the Technological Educational Institute of Thessaloniki in Greece decided to establish a pilot internal assessment in order to measure the current situation and to initiate a quality assurance program aiming to the continuous improvement of the quality of education. The purpose is not only to meet the European challenge but also to improve our competitiveness. The implementation of this process will be utilised by a software tool based on the Balanced ScoreCard methodology, which provides an integrated perspective on goals, targets and measures of progress. This paper provides a brief description of the Balanced Scorecard methodology and of the design of the appropriate key performance indicators suiting our needs.
1 Introduction Educational institutions need to have a clear mission, vision and objectives. When linked to a specific strategy and follow-up of the performance through communicated understandable indicators and measurements, both internal and external stakeholders are given the opportunity to understand what the objectives are and how well those objectives are achieved. A broader societal view is required. It does not concern if students are ready for the educational institutions and processes but whether the institutions and the processes are ready for the students. Targets for success clearly quantifying the desired level of performance necessary for current and future stakeholder satisfaction need to be developed. Subsequently, educational institutions need to assess other activities than finance, in order to provide a balance of tangible and intangible assets, and to measure future capability as well as past performance. Such measures are among others, stakeholder satisfaction, institutional, leadership and teaching effectiveness. A holistic consideration of learner, learning and learning context together with a broader societal reality that forms the backgrounds and perspectives of each learner are required as well as build-in quality assurance criteria and procedures during the whole process. 2 Towards the European Higher Education Area The Bologna Declaration [1, 2, 3], signed in June 1999 by 29 European ministers in charge of higher education, is the foundation for establishing a European Higher Education Area by 2010 and for promoting the European system of higher education world-wide. The basic aims of the Bologna Declaration are mobility, employability, and competitiveness. In particular, the objectives are the as follows: "adopting of a system of easily readable and comparable degrees"; "adopting of a system essentially based on two main cycles, undergraduate and graduate"; "establishing of a system of credits such as in the ECTS system - as a means to promote the most widespread student mobility"; "promoting mobility by overcoming obstacles to the effective exercise of free movement"; "promoting European co-operation in quality assurance. In May 2001, the ministers in charge of higher education of 33 European countries met in Prague to follow up the Bologna Process and to set directions and priorities for the following years. The following meeting was in Berlin in 2003. The objective was to review progress and set directions and priorities for the next stages of the process towards the European Higher Education Area. Three intermediate priorities were decided for the following two years, namely: quality assurance, the two-cycle degree system
and recognition of degrees and periods of studies. Specific goals were set for each of these action lines. Quality assurance Ministers stressed the need to develop mutually shared criteria and methodologies and agreed that by 2005, national quality assurance systems should include: A definition of the responsibilities of the bodies and institutions involved Evaluation of programmes or institutions, including internal assessment, external review, participation of students and the publication of results A system of accreditation, certification or comparable procedures, international participation, co-operation and networking 3 The Balanced ScoreCard Methodology Knowing that quality and performance are measurable entities it can be said that management tool Balanced ScoreCard (BSC) is their dynamic measuring system [4, 5, 6, 7]. It is valuable to be able to quantify entities, such as the performance of employees, the quality of results, customer satisfaction in an organisation, in order not only to increase the profit, but also to ensure transparency and efficiency in every aspect and to identify the weak points and improve them. These measurements influence essentially the human behaviour inside and outside the organisation. Unfortunately, many organisations adopt strategies that concern the clients, their core competencies and the abilities of the organisation, while simultaneously they motivate and estimate the performance based exclusively on financial measures [8]. On the contrary, BSC not only maintains the measurement of the financial entities as a resume of the performance, but also emphasizes a more holistic and complete set of measurement indicators. These indicators connect the customer (students) orientation with the internal procedures and the employees (lecturers & other staff) performance, as well as the whole system s (department) performance with the long-term achievement of the organisation s (educational institute) financial targets [8, 9, 10]. The application of a BSC system presupposes a thorough analysis of the processes and procedures used by the organisation. Assuming that each organisation is divided into four parts that all together indicate α complete image: Financial, Customer (students), Internal Procedures (the internal procedures of the organisation), Education and Development (ability of constant education aiming at continuous improvement and competitiveness). If these parts can operate separately, the organisation will operate also properly. These parts are transformed by the BSC to four performance indicators (Financial, Customer, Internal Processes, Education and Development). Thus, the organisation sets targets for each of these perspectives, collects the evidences of interest so as to verify the performance and quality level by using questionnaires, databases etc.
The BSC highlights the strategy and brings together the key indicators for management to run the organisation in order to achieve its goals. The graphical environment enables managers to understand the linkages between measures and objectives and helps them to focus on efforts. The visual environment enables cross-level communication between stakeholders (everyone in the organisation or unit, and their customers, funding providers, suppliers, the community etc.) and helps them to understand the objectives and how well they are achieved [8, 11]. Nowadays, when organisations compete in complex environments a holistic understanding of dynamics in order to achieve the goals is required. The BSC translates the vision, the mission and the strategy into measurable goals throughout a comprehensive set of performance measures aiming at leading an organisation into competitive success in the future [12, 13]. 4 Design of Key Performance Indicators Considering that the Balanced ScoreCard suggests that the organisation is viewed from four perspectives to develop metrics, collect data and analyse it relative to each of these perspectives we apply this methodology for the department of Informatics at the Technological Educational Institute of Thessaloniki in Greece. The main objective of this paper is to define the critical success factors for achieving our strategic goals in order to be able in the future to assess the actual performance against the measures. In the following paragraphs we analyse each perspective by using the GoalQuestion-Metric (GQM) methodology [14, 15]. The goals are determined and subsequently questions emerge to clarify the goals. These questions lead, after investigation of the use of BSC in higher education [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27], to the measures we will use in order to indicate our strategic direction. After the general description of goals, questions and metrics for each perspective, a more detailed diagram, depicted according the Ishikawa fishbone model [28] of the chosen performance indicators, follows.
Customer (Student) Perspective Goals I. High level of students II. Satisfied students III. Satisfied graduates Question: What is the level of students? Metrics: Demography of students, students grades, number of students publications, rate of attendance in theoretical courses, use of modern technologies Question: What is the level of educational personnel? Metrics: Demography of lecturers, educational personnel s pressure, number of publications, attendance rate of theoretical classes, number of final year projects, course evaluation Question: What is the level of the department? Metrics: Number of publications, number of new students per year, number of graduates per year, study duration, rate of students abandoning studies Question: How satisfied are students? Metrics: Courses, teaching conditions, notes - aids and tools supply, industrial placement, final year project, on-line support, secretarial service Question: How satisfied are employers? Metrics: Graduate s cognitive level, team work, initiatives Question: How satisfied is the educational personnel? Metrics: Courses, teaching conditions, industrial placement, on-line support, secretarial service Question: How satisfied are graduates? Metrics: Professional establishment, value (usefulness) of acquired knowledge during studies, postgraduate possibilities
Figure 1: Key indicators for the customer (student) perspective
Financial Perspective Goals I. Sufficient material infrastructure II. Accurate educational support Question: How accurate are the conditions of education? Metrics: Students per lecturer, students per course, teaching environment Question: What is the quality of educational material? Metrics: Number of new books in the library per year, sufficiency of instructive resources (e.g. video projectors, computers, etc) Figure 2: Key indicators for the financial perspective
Internal Process Perspective Goals I. II. III. IV. Active Learning Competitive program of studies Accurate secretarial support Accurate communication (electronic services) Question: How adequate are the teaching methods applied? Metrics: Projects assignments, motivation, active attendance of students in courses, use of examples from reality, application of course schedule, sufficiency of contact time, students support, marking scheme Question: How accurate is the study program (curriculum)? Metrics: Curriculum level, clear curriculum objectives, curriculum effectiveness Question: How accurate is the secretarial support? Metrics: Secretarial support Question: How accurate is the communication? Metrics: Accurate electronic support Figure 3: Key indicators for Internal Process Perspective
Learning / improvement perspective Goal I. Continuous briefing on new products / technologies Question: How many presentations / informative seminars are realised? Metrics: Seminars organised by the educational personnel, companies, department Question: What is the participation rate in research? Metrics: Research programs per year, participation of educational personnel in research programs, participation of students in research programs, number of publications Figure 4: Key indicators for Learning and Improvement Perspective
5 Conclusion The aims of this work were to investigate the possibilities of introducing a measurement-based management tool for assessing the current situation in the department of Informatics at the Technological Educational Institute of Thessaloniki, Greece, and for continuous use. The final objective is to improve the quality of the department within the frame of the Bologna Process. Transparency and competitiveness are requirements we have concentrated on. In order to obtain our targets, we chose the Balanced ScoreCard methodology, which is aligns goals, strategies and measurements of performance. In this paper the design of the key indicators chosen to fit our needs is presented. Future development will concentrate on applying our metrics to the principles and concepts of the Balanced ScoreCard. A pilot study will be implemented by using a Balanced ScoreCard software tool. We have already started by collecting information from different databases, interviews and questionnaires. The results of this procedure will in the first place be the quantification of the situation in the department, in order to focus on the weaknesses and to act proportionally. Finally, the department aims to use this tool for continuous improvement. References 1. http://europa.eu.int/comm/education/policies/educ/bologna/bologna_en.html 2. http://www.coe.int/t/e/cultural_cooperation/education/higher_education/activities/bo logna_process/default.asp ad 3. http://www.cep.org.hu/teachandlearn/bologna.html 4. Kaplan, R.S. & Norton, D.P. (1996), The Balanced Scorecard: Translating strategy into action, Harvard Business School Press, Boston, MA. 5. Kaplan, R.S. (1994): Devising a balanced scorecard matched to business strategy, Planning Review, September-October, pp. 15-19, 48. 6. Kaplan, R.S. & Norton, D.P. (1993): Putting the balanced scorecard to work, Harvard Business Review, Vol. 71, No. 5, September/October, pp. 134-147 7. Kaplan, R.S. & Norton, D.P. (1996): Using the balanced scorecard as a strategic management system, Harvard Business Review, Vol. 74, No. 1, Jan.-Feb., pp. 75-85. 8. http://socrates.lib.unipi.gr:81/dienst/repository/2.0/body/pa.pei/dt2003-0099/pdf 9. Olve, N.-G., Roy, J. & Wetter, M. (2000), Performance Drivers: A Practical Guide to Using the Balanced Scorecard, John Wiley, Chichester. 10. http://www.balancedscorecardsurvival.com/keyperformanceindicators.html 11. Thor, C.G. (1991), "Performance measurement in a research organization", National Productivity Review, Autumn, pp. 499-507. 12. Procurement Executives' Association (1999), Guide to a Balanced Scorecard Performance Management Methodology: Moving from Performance Measurement to Performance Management, Procurement Executives' Association. 13. Wade, D. (1997): Measuring performance with a balanced scorecard, Managers Handbook, Vol. 2, No. 7, pp. 2-9.
14. Basili Victor R. (1992): Software modeling and measurement: The goal / question / metric paradigm (Computer science technical report series / University of Maryland) 15. Basili Victor R. (1995): Applying the Goal / Question / Metric Paradigm in the Experience Factory in Fenton, Whitty, Robin, and Iizuka, Yoshinori (eds.), Software Quality Assurance and Measurement A Worldwide Perspective, International Thomson Computer Press, London 16. http://www.balancedscorecardsurvival.com/settingobjectiveswithbalancedscorecard.h tml 17. http://www.balancedscorecardsurvival.com/balancedscorecardobjectivessamples. html 18. http://oaa.osu.edu/irp/balancedscorecard.pdf 19. http://odl.rutgers.edu/pdf/score.pdf 20. http://www. balancedscorecardsurvival.com/balancedscorecardeducation.html 21. http://www.lib.virginia.edu/bsc/ 22. http://www-vcba.ucsd.edu/pa/performancearchitectureapril%2098_files/frame.htm 23. http://www-vcba.ucsd.edu/bsctelnor/bsc%20telnor%201-25-02_files/frame.htm 24. http://www-vcba.ucsd.edu/bscucsd/ 25. http://bas-avc.berkeley.edu/balancedscorecard/ 26. http://www.ais.ucla.edu/scorecard/ 27. http://web.uflib.ufl.edu/toolbox/stratplan.html 28. http://www.skymark.com/resources/leaders/ishikawa.asp