PROS AND CONS OF USING QUANTITATIVE METHODS IN EVALUATING THE PERFORMANCE OF AN INFORMATION SERVICE

Transcription

1 PROS AND CONS OF USING QUANTITATIVE METHODS IN EVALUATING THE PERFORMANCE OF AN INFORMATION SERVICE By Richard Kirui The Library, Kabianga University College, Kabianga and Japhet Otike Department of Library and Information Studies Moi University, Eldoret Abstract The article discusses the pros and cons of evaluating a library service using statistical methods. It argues that while quantitative evaluation has a number of advantages particularly in supporting the librarian in justifying the need for additional funding, demonstrating the use of the services; and preventing closure, among others, it poses a number of weaknesses. That statiscal evaluation does not show whether the library s resources are effectively used. It is more concerned about pleasing the policy makers and not users who are the ultimate beneficiaries of the service. In short, it compromises quality in favour of quantity. Keywords: Performance Measurement; Library Evaluation; Quantitative methods: Library Statistics. 1

2 Introduction According to Alemna (1994), evaluation is basically a judgement of worth. It means assessing the worth or value of the unit to the people for whom it is meant. It is the assessing of the performance against user s expectations. In other words, it demonstrates whether the output is commensurate with the input. He goes on to say that in the context of a system, it means the degree of usefulness of the set-up in meeting various objectives the system has to achieve. It is testing the service for effectiveness and efficiency. Lancaster (1988) points out that evaluation simply establishes a type of a benchmark to show at what level of the performance the service is now operating. If changes are subsequently made to the services, the effects can then be measured against the benchmark previously established. There are two approaches to library performance measurement: subjective and objective methods. The objective method is quantitative while the subjective method is qualitative. Quantitative method According to Saleemi (1997), quantitative techniques refer to mathematical or statistical method used by organizations to make decisions or formulate business plans. According to Abbott (1994), quantitative method involves simple quantification of activity such as the number of book issues, the number of registered users of the library, the total annual visits to the library for a given period of time, while Zittel (1987) views it simply as use of library statistics. This involves assessing the library stock in quantitative terms: the number of users, loans, visits, exhibitions, discussions, staffing, financial provision and so forth. These are stastical indicators or variables. Alemna (1999), argues that quantitative method in performance measurement adopts the tools of the management science. He states that for a complete and objective evaluation of library s effectiveness, three important factors need to be taken into consideration. These are: Specification of purpose or goal of the system and the parts studied. Selection of a measure or measure reflecting this purpose and specification of the units of measure. 2

3 Specification or construction of measuring instruments and the implementation of measuring method. In an attempt to satisfy the objectives, each library provides: physical facilities for study; information needed and demanded by readers; and core and auxiliary readers services. From the foregoing, it can be noted that quantitative method puts a lot of emphasis on inputs. It is more concerned about the immediate results. The rationale for performance measurement Establish whether the objectives have been achieved Every library has set objectives which have to be achieved within a given period of time. The objectives of the library should be in line with those of the parent organization since the library exists to support the organizational goal. According to Alemna (1999), it is evaluation that testifies whether the objectives of the library have been achieved and if so, to what extent. According to Cossette (1987), performance measurement is carried out to ascertain whether the library is achieving its objectives or meets the needs of its users. Performance measurement establishes the library s effectiveness. For example, if the library has set an objective to register 1500 users every year and registers 1600, the library is considered effective in its performance. Accountability to the parent organisation Performance measurement is carried out for purposes of accountability to the funding authorities. Abbott (1994) argues that libraries in the public and private sector are increasingly required to justify the resources they consume, and demonstrate that the benefits desired are worth the expenditure. Performance measurement helps to show the range, extent and importance of the services provided. Libraries are expected to show that the services they provide are both efficient and effective. They must demonstrate that they offer not just value, but value for money. On account of stiff competition from sister departments in an organisation for scarce resources, libraries must convince the funders the need to continue funding their activities. According to Cossette (1987), the reason prompting librarians to measure library performance systematically is the restricted 3

4 nature of financial resources. With the ever-rising costs of public services and the scarcity of financial resources, libraries have to be accountable to the taxpayers and their representatives. Libraries must justify their continued funding in terms of target results and must show that the products and services funded by these allocations match up to the expected results. Quality management and assurance Libraries employ performance measurement to ascertain the quality of their services. Abbott (1994) points out that performance indicators have value and relevance in the context of quality assurance. Quality assurance mechanism attempts to define how a process will operate in order to ensure a given quality of product or service. Performance indicators attempt to measure the extent to which the quality has been achieved. To establish possible sources of failure Performance measurement assists to establish possible causes of failure or inefficiency in the service with a view to raising the level of performance at some future date. When the library staff have adequate knowledge of a situation they are in, then, they are in a better position to decide on judicious corrective measures to improve the quality of the services. The ultimate aim of any performance measurement is not, to monitor with the purpose of rewarding or punishing any one, but rather, to improve the performance of a library service. The essential aim of performance measurement should therefore be to evaluate in order to evolve. Decision support Performance indicators are important to the library manager as aids to decision making in areas such as resource allocation, priority setting, and planning. Abbott (1994) observes that the use of performance indicators helps the librarian to: Understand what the library is achieving Monitor the library s progress year on year, and in comparison with other libraries or sites. Gauge how efficiently and effectively library operations and services are performing 4

5 Take decision about future resourcing of service areas on the basis of hard information on past performance, rather than on intuition Estimate the consequences of increasing or decreasing resources for the performance of different service areas Check that the objectives of the library as a whole are accurately reflected in the balance of activity in the library. Criteria for performance measurement According to Wijasuriya (1987), there are a number of objective standpoints from which performance can be measured. These include: a) Effectiveness - how well a service satisfies the demand placed upon it and is meeting the objectives of the organization b) Cost-effectiveness - how effectively a service meets its objectives in relation to the costs of maintaining that service c) Cost-benefit - whether the value (worth or benefit) of a service is more or less than the cost of providing it. This establishes whether the cost of providing a service is justified by the value or benefits derived from it. Merits of quantitative method A tool for planning Saleemi (1997), argues that statistics provide a base for future planning. In the absence of relevant data, no one can plan properly. Plans prepared without accurate and relevant data cannot achieve better results. Statistics present accurate data which help the librarian to plan. Checking on the number of seats in relation to users during evaluation assists the librarian to plan seating arrangement in the library. Annual increase in the number of users can assist the librarian to decide on the number of materials to add annually. Statistics are eyes of the librarian According to Saleemi (1997), statistics are the eyes of a manager. Basing on Saleem s understanding, statistics are required by a librarian to study the causes and find out the 5

6 remedies of various problems in the library. Measuring performance using quantitative technique clearly shows the difference between the achieved results and the expected results. Quantitative method involves simple quantification of activity such as the number of book issues, the number of registered users of the library, the total annual visits to the library, etc. These are important figures for a librarian to know since they provide some insight into the size and scope of the library operations. This insight helps the librarian to understand the library performance better. He/she is able to identify possible sources of failure or inefficiency in the service with a view to raising the level of performance at a future date. Knowing the number of users helps the librarian to predict the information materials needed now and in a future date. It helps the librarian to know the subjects and books which are heavily used. This in turn assists the librarian in book acquisition. Allows objective analysis of the quality of library s services According to Cossette (1987), by use of statistical techniques, it is possible to analyse objectively the quality of the library s services. Unless it is backed up by statistics, an evaluation study amounts to no more than a statement of opinion and is devoid of any scientific value. A methodical analysis of the results of a library service calls for valid models and instruments of measurements. The use of analytical models in library science makes the scientific management of libraries a possibility. Quantitative method of performance measurement is objective. This means that success is measured in quantitative terms and not on opinions of users. It demonstrates the value of the library in measurable terms, for example, annual budget. In harsher economic climate where competition for resources is fierce, there is an increasing acceptance that libraries must demonstrate their value in measurable ways. Useful in decision making De Gennaro (1987) argues that the quality of many decisions could be significantly improved if there were more and better data. The collection and analysis of quantitative data in libraries assist in rational decision making. It helps in knowing more about the library, its resources, and how they are actually used. Libraries have relied historically upon input data, for example, the number of books acquired, the number of serials 6

7 subscribed to, the number of books circulated, the money spent, etc. Quantitative data help librarians to think clearly and creatively about library management and to sound decisions based on valid data. Statistics acts as a technique of comparison To Saleemi (1994), certain facts may be meaningless until and unless they are capable of being compared with similar facts at other places or other periods of time. We estimate the book per capita, budget, etc, not essentially for the value of that fact itself, but mainly to compare the book per capita or budget of today with that of the past and draw conclusions as to whether they are on the increase, decrease, or stationery and how they have affected the services. Because of its measurability it allows comparison to be made. For example, figures of different years can be compared. Financial allocation to the library in a given year can be compared with those of the previous years hence judgement can be made if the increase in funding solved the problems in the library in that given year. Endeavour to interpret conditions Use of statistics renders useful service in presenting an attractive picture of a situation under investigation. Close study of data enables us to interpret conditions of the phenomena under study. For example, increase in the number of titles in all the subjects in the library can be used to interpret the improved student performance. An aid to supervision Saleemi (1997) states that measuring performance using quantitative method will aid in supervision. Statistics will reveal areas where users frequently complain about the services, for example, staff punctuality. The librarian will therefore strengthen supervision in the affected areas. Since statistics are used to evaluate the library, they are useful in controlling the affairs of the library. The librarian can decide on the basis of the statistics whether the policies are being implemented effectively or not. Demerits 7

8 It cannot assess effectiveness and efficiency Cossette (1987), argues that traditional statistics compiled by libraries are inadequate in measuring performance of a library service. This is so because they focus exclusively on resources (inputs) rather than the results (output). It deals with measurement of resources. This tells us nothing about the effectiveness of the services. It is unable to assess the efficiency or effectiveness of activities provided. Abbott (1994), observes that the materials, financial and human resources express no more than its capacity to produce results and not the results; products and services (outputs) actually achieved. It does not show the extent to which the activities satisfy the user needs (effectiveness). Do not indicate value and quality of the library services De Gennaro (1987) argues that statistics on the number of readers, loans, library visits, or exhibitions give virtually no indication of value and quality of activity. The quantitative indicators do not reflect quality of service offered. He further goes on to state that quantitative method is concerned with constant pursuit of the biggest possible number of loans and readers, but the question is, are the readers really satisfied with the services the library offers? An example is when a library collection is evaluated in terms of its size. Lancaster (1988), points out that one would expect that the larger the collection the greater the chances that it will contain a particular item sought by a user. In this case book per capita is a measure used. This is only useful if the books referred are likely to be of use or of interest to the users. The disadvantage with this approach is that there could be many books but of low quality. For example, a library can buy cheap books of low quality, indiscriminately accepting many donations or by never discarding books that are old and unused. The value of this collection is not considered. Abbott (1994) argues that quantitative method does not facilitate any judgement as to the quality or performance of the library service. They are simply statistics: the library budget, the number of staff employed, the number of books purchased or the size of the acquisition, etc. All these do not reflect on quality of the services the library offers. Such statistics only tell the reader how the library is resourced. They reveal nothing about the library s performance. 8

9 Cossette (1987) states that a quantifiable datum has no significance by itself, yields no direct information regarding quality of information products and services. Does not relate activity to needs, or services to demand According to Abbott (1994), quantitative method does not attempt to relate activity to needs, or services to demand. This method is not meaningful since it is not focused on user needs and does not reveal the extent to which they satisfy those needs. It is concerned with, for example, the number of collection but does not show concern if these items are the right balance. It does not put concern on the needs of the users. It only looks at the quantity without regard to the user needs or demands. To satisfy the users, a library should dispense services with due regard to the needs of the user. According to Cossette (1987), to be valid, performance measurement of a library s effectiveness should be useroriented; it should consider the library in relation to its environment. An example here, is the use of standards which are essentially quantitative rules defining the minimal resources in terms of documents, staff and material facilities relating to the functioning of a library. The main disadvantage of using standards to measure performance of an information service lies in the fact that standards are not user-oriented but are focused mainly on resources rather than on results, products and services. According to Alemna (1994), this method does not take into account the library user, who is the ultimate authority. The user is the most logical source of answer. This method does not take into consideration the user whose opinion remains a valid and potent measure of user satisfaction. Demanding It is a demanding task to ascertain the level of reader and library user satisfaction. It becomes demanding if the indicator specifying the number of readers for a given year was further broken down into fully satisfied readers, partially satisfied readers and those dissatisfied with the library s services; and the indicator for exhibitions and discussions organized by the library not only indicated the number they attracted but also, reflected the extent to which they were satisfied with these events. It becomes difficult to ascertain the level of reader and library user satisfaction 9

10 Can be misleading According to De Gennaro (1987), quantitative method can be misleading. In the hands of amateurs the quantitative method can frequently produce misleading and wrong conclusions. For example, the larger the number of books in the library does not mean that they are the right books. Also, the total number of registered users does not mean all of them are using the library. If the total number of registered users is used in measuring performance it can give misleading conclusion that all of them are using the library. Some users are registered but never use the library. Statistics taken about the daily use of the library are also misleading since the total number per day includes repetition. It is the number of times a user enters the library but not the individual user. Quantitative method is descriptive and not analytical Abbott (1994), points out that quantitative method emphasises on description, not analysis. For example, sometimes library annual reports contain information on library outputs: number of items catalogued, number of information skills seminars conducted and so forth. Here the emphasis is on decription, not analysis. It does not explain why there has been an increase or decrease in a given activity. The rise in percentage does not alone demonstrate an improvement in performance. Conclusion This article has endeavoured to provide a balanced view of the strengths and weaknesses of the quantitative approach as a method for evaluating library performance. From this account, it appears that there is no best or fool proof method of doing this. Whether to or not adopt quantitative method, the ultimate decision should be determined by the purpose for evaluating the library. If the ultimate purpose is to provide an account of the performance of the library for decision makers, then, quantitative approach would be appropriate. However, if the report is destined to the users, it is most likely that the users will not be impressed by mere statistics. They may be more interested in establishing whether the library services are effective in meeting their needs - something that 10

11 quantitative approach can not do. BIBLIOGRAPHY Abbott, C.(1994) Performance measurement in library and information services. London: ASLIB. Alemna, A. (1999) Library evaluation: some options for academic libraries ASLIB Proceedings, vol. 51 no. 7: Cossette, A. (1987) Evaluating the effectiveness of a library: a theoretical and methodological framework International Reader in the Management of Library, Information and Archive Service compiled by Anthony Vaughan. Paris: UNESCO. Cram, J.(1995) "Demonstrating value for money: issues for Libraries and Librarians" Singapore Libraries vol.24, Cram, J (1999) "Fishing with grenades or greening the mind: value, values and municipal Libraries for the New Millennium" De Gennaro, (1987) Library Administration & new Management systems International Reader in the Management of Library, Information and Archive Service compiled by Anthony Vaughan. Paris: UNESCO. Edward Evans et al (1972) Review of Criteria Used to Measure Library Effectiveness. Bulletin of Medical Library Association. vol.60(1): Lancaster, F. W.(1988) If you want to evaluate your library, London: Library Association. Library effectiveness. Masaru ITOGA (2001) Statistics and Performance Evaluation in Academic Libraries 11

12 Saleemi, N.A (1997) Quantitative methods. Nairobi : Saleemi Publishers. School Library Standards and Evaluation. Shin IRIE (2001) Towards the business statistics as a management target Wijasuriya, (1987) Performance measures for public libraries International Reader in the Management of Library, Information and Archive Service compiled by Anthony Vaughan. Paris: UNESCO. Zittel, (1987) Specialized problems of practical librarianship: planning International Reader in the Management of Library, Information and Archive Service compiled by Anthony Vaughan. Paris: UNESCO. 12