Development of evaluation Criteria. Design and Implementation of a database of

Transcription

1 9 th TOULON-VERONA CONFERENCE PAISLEY-SCOTLAND, SCOTLAND, 7-8 SEPTEMBER 2006 Identification of the effective practices in academic support services in Greek Public Universities. i i Preliminary findings from Library, Information Technology and Career Services

2 Research Team George Tsiotras, Professor, Secretary General of the Region of Central Macedonia, Greece Katerina Gotzamani, Lecturer, Project Director Antigoni Papadimitriou, Researcher Elias Ntinas, Researcher Acknowledgements "The project is co-funded by the European Social Fund & National Resources - (EPEAEK II) - PYTHAGORAS".

3 Further goals of this research Investigation of current application Development of evaluation Criteria Identification of Best Practices Design and Implementation of a database of Best Practices

4 Project framework Catalog of Greek universities iti and a presentation of the selected administrative and organizational functions Record of the current situation of the chosen administrative and organizational functions Evaluation of the applied procedures Collection and analysis of the information Development of the data base Results of the research project

5 Chosen organizational and administrative functions Library Information Technology Center Research Committee Career s Office Department of Public and International Relation

6 Defining the evaluation criteria Education Criteria i for Performance e Excellence by the Baldrige National Quality Program (2004) EFQM Excellence Model Higher Education Version by Sheffield Hallam University in collaboration with EFQM (2003) Academic Scorecard evaluation of academic organizations ( balanced scorecard )

7

8

9

10 EVALUATION CRITERIA QUANTITATIVE Financial i and other Leadership resources Customers-Users Strategy QUALITATIVE Personnel User satisfaction i Processes (function- specific) Personnel Society Processes Other data (not included in previous categories) Resources

11 BEST PRACTICES Quantitatively high performing practices High score achievements in qualitative factors

12 Problems: Pilot study - reluctance by most functions head officers to participate in the project in the form of evaluation so as to record best practices was observed - questionnaire related comprehension difficulties during completion by the representatives

13 Solutions - Development of second qualitative questionnaire which accompanies the interviews - identify and record best, good, or effective practices

14 Looking for best practices These practices are not directly associated with the terminology (definition) of best practices, as these are found in the literature, but they are based on the personal experience of the participants and the experience and subjective judgement of the research team

15 Comments from the 1 st International Conference Professor Brent Ruben, Rutgers University USA and Professor Al Shagana Kadim, Sheffield Hallam University, UK this project is a challenge for the research team

16 Project Schedule Phase 1 Phase 2 March June 2004 Catalog of Greek Universities Presentation of the selected administrative and organizational functions July October 2004 Investigation of current practices Phase 7 NOV- DEC 2006 Results of the Research Project Phase 3 Phase 4 Phase 5 Phase 6 Nov Feb.2005 Development of evaluation Criteria March Jun 2005 Best Practices from foreigner Universities Jul Jun 2006 Pilot study, Collection & analysis of selected data July- Oct 2006 Development of the Database

17 Geographical distribution ib ti of the Greek Universities

18 1. National and Kapodistrian University of Athens, 1837, 2. National Technical University of Athens (Metsovio), 1836, 3. Athens University of Economy and Business, 1920, 4. Pantion University, 1927, 5. Agriculture University of Athens, 1920, 6. University of Piraeus, 1938, 7. Athens School of Fine Arts, 1837, 8. Harokopio University, 1929, 9. Aristotle University of Thessaloniki, 1925, 10. University of Macedonia, 1948, 11. University of Patras, 1964, 12. University of Ioannina, 1964, 13. Democritus University of Thrace, 1973, 14. University of Crete, 1973, 15.Technical University of Crete, 1977, 16. University of the Aegean, 1920, 17. Ionian University, 1984, 18. University of Thessaly, 1984, 19. University of Western Macedonia, 2002, 20. University of Peloponnese, 2000, 21. University of Continent Greece, 2003 Source: A. Papadimitriou, Quality Assurance in Greek Higher Education, Ph. D. Dissertation, University of Macedonia, Thessaloniki, Greece, in process.

19 Preliminary DATA ANALYSIS For the above reasons the first questionnaire was regarded as not appropriate for the use of evaluating and identifying of best practices and it will only be used for an explorative recording of the current quality management practices in the specific academic functions services

20 Data collection until June 2006 Academic Support Services Quest Interv. Library Information Technology Center Career s Office 7 10

21 Population Administrators 18 Library Information Technology Center Career s Office Faculty 15 Library Information Technology Center Career s Office

22 Up to now, 28 out of 63 questionnaires have been completed (44.4 %). This percentage is somewhat higher for the in-depth interviews, since we have already implemented 33 (52.4%).

23 University A Effective practices LIBRARY 1. Adaptation of students with library services 2. Decrease of waiting time for books 3. Upgrading of computerisation 4. Internal decoration6. Users satisfaction questionnaire e

24 University B 1. Taking users feedback 2. Enhancement of library material security 3. Conduct of seminars about how to make literature review research 4. Enhancement of digital collection 5. Users s satisfaction questionnaire e

25 University C 1. Internal evaluation process of service s sections (reports) 2. Process recording in documentation section 3.Webpage design 4. Specific policies- Loan/interloan policies Policies for material Digitalisation so as to be accessed by visually impaired users (material manipulation, course analysis, application of conspectus technique) 5. Staff interview i for the confrontation ti of problems and the intensification of the relations between the personnel and the direction 6. Creation of a new electronic exhibition place for new material entries 7. Service provision to students and other users who are visually impaired

26 University D 1. Information and sensibilisation of public and library members regarding certain issues such as the forwarding of Christmas wishing cards 2. Implementation of activities to attract people regarding various art and culture issues 3. Updating of Information to users with both electronic and printed selected material

27 University E 1. Application of quality management system for university libraries: Α. Collection management policy Β. Loan policy C. Interloan and presumption delivery policy

28 CAREER SERVICES UNIVERSITY A 1. Integrated information system 2. e-learning, elea e-mentoring 3. Mapping of job and professional grounding opportunities 4. Diffusion of information for public sector vacancies UNIVERSITY B 1.Electronic recording of the office s daily activities for MIS 2. Electronic member card

29 UNIVERSITY C 1. Promotion of university students and graduates by organising g several activities, publishing newsletters and distributing leaflets in various places in the country UNIVERSITY D 1. Function of faculties and services aiming to inform students about their professional opportunities 2. Realisation of job seminars and workshops 3. Tele-consultancy

30 UNIVERSITY E 1. Small concentrated research for the identification of certain fields where graduates can find a job 2. Invitation of companies representatives ese es to present their professional plans and offer opportunities to the graduates of the institute 3. Organisation of Seminars in collaboration with European Poverty Network for graduates education in subjects of a social science following the ILO methodology so as to combat these problems 4. Particular efforts by business consultants t for the information of students on issues of self- employment and business activities

31 INFORMATION TECHNOLOGY University A 1. Coordination on issues of information technology UNIVERSITY B 1. Automatic information of academic staff about several university issues1.information technology support (aiming to save resources and techniques) 2. Work distribution 3. News: Since the home page of the service s website (user-friendly, has been evaluated as 10th in an international competition) 4. Help Desk and technical support to users (CNC Process)

32 1. Users information UNIVERSITY C e-class development UNIVERSITY D 1. Job grouping based on staff satisfaction Annual effectiveness ect e ess evaluation offering bonus UNIVERSITY E 1. Offering services to following projects: Gea Radio, Weather forecasting station, Applied hydrobiology Laboratory

33 EVALUATION CRITERIA Library Career s Office I/T 1 European Committee Criteria Press and Information Unit TQM evaluation criteria for Greek Academic Libraries Users satisfaction Increasing number of users Empirical but not organized criteria, feedback (orally, by e- mail)

34 2 TQM evaluation criteria for Greek Academic Libraries Simplicity criterion Simple and friendly handling High level of users satisfaction Number of users Students and teachers satisfaction of use based on questionnaires Direct supervision 3 Personnel and users Degrees particularity i of satisfaction TQM evaluation Social Administration, Social Politics and criteria for Greek Academic Libraries Psychology alumni. We are trying to invent appropriate and special ways of alumni absorbance b for better results.

35 4 TQM evaluation criteria i for Greek Academic Libraries i 5 TQM evaluation criteria for Greek Academic Libraries Users satisfaction since 1997 Number of users ( students) Enterprise participation i i in Career Week and interviews Number of finterviews i students / company Users Satisfaction

36 11 TQM evaluation criteria QUANTITATIVE for Greek Academic users Libraries Quality Assurance Systems Policies Number of visiting the Office Number of users visiting the website web site exploitation Number of participants in the Office events Number of events Printing and handing out of printed advertisement and information material Number of enrollments in the data bases Annual comparative results for all the above quantitative data.

37 User's Satisfaction Library Career s Office Information Technology 1 Yes/Questi onnaire Yes Questionnaires (temporary) Printed material from companies for hiring (annual) Informal Orally (temporary) 2 Yes (Questionnaire) Yes (Utility of service Quality of service (number of users and rate, consultancy from the staff to the users. Support to the users.

38 3 Yes Yes (Visitors satisfaction if i of the service on a continuous basis ) 4 Yes Yes Yes Quick response to the users. Level of services for users satisfaction. Prediction of users needs

39 11 Yes Yes (without (In the framework of f3 rd indexes and Support from E.E. questionnaires, During periods of direct service for seminars and meetings. users from Dil Daily Visitors i satisfaction i 730 7:30a.m. to of the service on a 8:30p.m. daily. continuous basis Stif Satisfaction of requests for all services that tthe network provides.

40 QUALITY TOOLS Library Career s Office I/T 1. Participation in TQM procedures for Greek Academic Libraries Users Satisfaction Questionnaire No 2. Participation in TQM procedures for Greek Academic Libraries Questionnaires 1.Tool to record the requests and the response time of the services. Users Stif Satisfaction 2. Measurement of response time for requests = Value of Utility

41 3 Participation in TQM procedures for Greek Academic Libraries Low range researches with questionnaires 4 Participation in Printed material Statistics of services, only TQM procedures for Greek for Users Satisfaction measurements. Performance of each service. Academic Libraries

42 5. Users Satisfaction Questionnaire Regulation of Internal Operations Users Satisfaction Questionnaire Regulation of Internal Operations Self assessment (though reviewing the statistic data for each year). Frequency of occurrence and confrontation of problems. Internal Regulation

43 12. Participation in Assessment of office services TQM procedures based on indexes. for Greek Academic Recording of the projects Libraries outputs that the Career Services Quality Assurance Systems ( ISO 9000 Office handles for each year and totally for the project. in process) Assessment of users needs through researches Recorded policies and strategies of the service. Through the Technical Bulletin of the Project recorded processes: Defined through the Technical Bulletin and redefined each year.

44 Before moving further to the analysis of the data, it is worth-noting that all academic libraries take part in MOPAB (Academic Libraries Total Quality Unit) and that all Career Offices use self- evaluation procedures.

45 Data analysis Leadership 12 indicators Strategic planning 9 Personnel 13 Processes 9 Resources 9 User s Satisfaction 8

46 Criteria Importance rate Application rate Leadership Strategic Planning Users satisfaction Personnel Processes Resources.68.75

47 Table 5: Mean values for importance and application rates in 6 quality criteria for academic libraries N=11 Importance Rate Application Rate (1-5 scale) (1-5 scale) Leadership Strategic planning Users Satisfaction Personnel Processes Resources Total Mean

48 Table 6: Mean values for importance and application rates in 6 quality criteria for career offices N=7 Importance Rate Application Rate (1-5 scale) (1-5 scale) Leadership Strategict plann Users Satisfact Personnel Processes Resources TotalMean

49 Table 7: Mean values for importance and application rates in 6 quality criteria for information technology centres N=10 Importance Rate Application Rate (1-5 scale) (1-5 scale) Leadership Strategict plann Users Satisfact Personnel Processes Resources TotalMean

50 1) leadership Scale -items Appplication rate Importance rate Scaled response - Score

51 Leadership (app. & imp. rates) scales distribution 6 Leadership 10 Leadership Std. Dev =,37 Mean = 4,57 2 Std. Dev =,81 Mean = 4,02 0 4,00 4,25 4,13 4,38 4,50 4,75 4,63 4,88 5,00 N = 21,00 0 1,00 2,00 3,00 4,00 5,00 1,50 2,50 3,50 4,50 N = 22,00 LEADIMP importance rate LEADAPP Application rate

52 2) Strategy Scaled response - score Importance rate Application rate Q3: Do you collect and estimate within your strategy planning, data concerning corresponding Services of other Constitutions or Foundations of Education? App:3.45/imp.4.18

53 Strategy (app. * imp. rates) scales distribution Strategy Strategy Std. Dev =,82 1 Std. Dev =,43 Mean = 4,40 0 Mean = 3,83 N = 22,00 0 N = 19,00 1,00 2,00 3,00 4,00 5,00 3,50 3,75 4,00 4,25 4,50 4,75 5,00 1,50 2,50 3,50 4,50 STRTIMP importance rate STRTAPP application rate

54 User Satisfaction Scaled Response - Score Importance rate Application rate Q6:Do you apply a comparative evaluation method for Users Satisfaction in relation to the satisfaction provided by corresponding Model-Services (Benchmarking) imp:3.78/app.2.2 Q7:Do you establish relationships (eg. collaborations with corresponding Services of other Educational Institutions) aiming to satisfy the Service Users? imp: 3.54/app:2.56

55 User satisfaction (app. & imp. Rates) scales distribution 6 User satisfaction 8 User satisfaction ,25 3,50 3,75 4,00 4,25 4,50 4,75 5,00 Std. Dev =,51 Mean = 4,17 N = 21, ,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 Std. Dev =,75 Mean = 3,30 N = 22,00 USERSIMP Information rate USERSAPP application rate

56 Personnel Scaled response - score Importance rate Application rate Q3: Do you apply an official system of recognition of Service Staff? imp:3.6/app:2.3

57 Personnel (app. & imp. rates) scales distribution 5 Personnel 10 Presonnel Std. Dev =,44 Mean = 4,44 0 N = 18,00 3,63 3,88 4,13 4,38 4,63 4,88 3,75 4,00 4,25 4,50 4,75 5, ,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00 Std. Dev =,68 Mean = 3,76 N = 18,00 STAFFIMP importance rate STAFFAPP application rate

58 Processes Scaled response - score Importance rate application rate Q3:Do you apply specific measurement indicators for the control and improvement of procedures? imp:4.1/app:2.5 Q4:Do you apply some official procedures evaluation method? imp:4/app:2.5

59 Processes (app. & imp. rates) scales distribution 6 Processes 6 Processes Std. Dev =,48 Mean = 4,34 1 Std. Dev =,96 Mean = 3,17 0 3,25 3,50 3,75 4,00 4,25 4,50 4,75 5,00 N = 21,00 0 1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 N = 21,00 MANGEIMP importance rate MANGEAPP application rate

60 Resources Scaled response -scored Importance rate Application rate Q4: Do you apply programs of mutual development and instruction (eg. mutual exchanges) imp:3.6/app:2.1

61 Resources (app. & imp. Rates) distribution 6 Resources 6 Resources Std. Dev =,48 Mean = 4,47 1 Std. Dev =,82 Mean = 3,53 0 3,50 3,75 4,00 4,25 4,50 4,75 5,00 N = 15,00 0 1,50 2,00 2,50 3,00 3,50 4,00 4,50 N = 17,00 RSRSIMP Importance rate RSRSAPP Application rate

62 Scale reliability Table1: Scale reliability: Reliability coefficients (Cronbach s α) ) of six criteria Leadimp Strtimp Usersimp Staffimp Mangeimp Rsrcsimp Leadapp Strtapp Usersapp Staffapp Mangeapp Rsrcsapp Application rates show, in general, greater reliability; Resources scale is, in both cases, the most problematic

63 Table 2: Descriptive statistics from EFQM variables (importance & application) N Minimum Maximum Mean Std.Devation LEADIMP STRTIMP USERSIMP STAFFIMP MANGEIMP RSRCSIMP N Minimum Maximum Mean Std.Devation LEADAPP STRTAPP USERSAPP STAFFAPP MANGEAPP RSRCSAPP

64 Average scores (app. & imp. rates) Mean responses Scaled response - score Importance rate Application rate In general, mean responses are greater for importance rates rather than for application rates

65 Average scores (app. & imp. rates) Std. Deviations 1,2 1 0,8 06 0,6 0,4 0, Importance rate Application rate In general, application scales show more variability than importance scales (as was also indicated by each criterion histogram)

66 Are these Differences statistically significant? Table 3: Resutls from difference of means tests (two-tailed) tailed) between the key EFQM variables. Leadimp Strtimp Usersimp Staffimp Mangeimp rsrcsimp Leadimp - Yes Yes No Yes No Strtimp - Yes No No No Usersimp - Yes Yes No Staffimp - No No Mangeimp - No Rsrcsimp - In most cases (9/15), they are not.

67 Correlation between the six criteria Table 4: Pearson correlations between the 6 criteria Leadimp Strtimp Usersimp Staffimp Mangeimp rsrcsimp Leadimp Strtimp Usersimp Staffimp Mangeimp -.79 As it was expected, all criteria are intercorelated (imp. Rates) although greater correlation is observed between User satisfaction & personnel/processes Table 5: Correlation between importance and application rate Leadimp Strtimp Usersimp Staffimp Mangeimp Rsrcsimp Leadapp.63* Strtapp.56* Usersapp.41* Staffapp.18 Mangeapp -.06 Rsrcsapp.238 * p<.05 Although suspected, importance rates responses are not driven by application rates responses (and vice versa)

68 Presentation ti of effective practices

69 Practice reported by: Source to the practice: Name of services that apply practice: Contact person for practice: General Manager Information Technology Centre University of Ioannina Information Technology Centre of University of Ioannina Professor Ioannis Rizos Title for practice: e-rental Key words: Practice focus on: Short description of practice: Electronic search of home, renting a home by the university s webpage Users satisfaction The information technology service inspired, organised and implemented an electronic service called e-rental (e-νοικιάζεται) whereby all students who look for a place can find important information on the university s webpage. Everyone who is interested in renting a house can simply call the centre, provide all necessary information and in this way plenty of houses are entered in the university s s webpage and can be accessed by those interested in finding a home. Being the first time that such a practice has been initiated by a university information technology centre; this initiative has been often mentioned in the press. In order the whole project to function efficiently, the centre has been collaborating with the university s public relations office whose main responsibility is to make this project known to new students and to sponsor it within the local society. Context of use of practice and actors involved: Results and benefits of practice: Key success factors: Transferability of practice: Requirements for transferability: Reference materials: To the students as well as people who either look for lodgers and tenants or for a place to rent. Users satisfaction, improvement of the university s public image Personnel initiative, innovation and know-how Yes Professional work, connection with agents from the local community, willingness to provide and realise new effective ideas Webpage: www uoi gr

70 Conclusions Taking into account that in Greece the implementation of Quality Assurance is in its early phases the responses of GRPUs to this research was very positive. Administrators from academic support services are not well experienced with quality and evaluation issues and need time in order to response accordingly. The university activities often permeate the local economic, social and cultural environment.

71 The research team believes that the results of this research project and especially the presentation of effective practices and the ways that these practices can adapt and transfer (Jarrar and Zairi, 2000) will contribute to the improvement of the practices of the functions of the universities. This information will be beneficial i in order to create a healthy communication system, exchange of knowledge and collaboration, aimed at improving these procedures.

72 Questions, comments and suggestions are welcomed THANK YOU gr