The Difficulty of Indicating Transformation: The Challenge for Library Statistics and Surveys

Similar documents
Why DDA is Here to Stay: An Analysis of the Demand-Driven Acquisition Model for Libraries

BENCHMARKING IN THE FORM OF PERFORMANCE INDICATORS AND BALANCED SCORECARD. Roswitha Poll Chair of ISO TC 46 SC 8

Chapter -3. Research Methodology

FOCAL POINT AGREEMENTS : KEY PERFORMANCE INDICATORS

General review of National Digital Library Development in the National Library of China

PROS AND CONS OF USING QUANTITATIVE METHODS IN EVALUATING THE PERFORMANCE OF AN INFORMATION SERVICE

Support for energy and climate technology

Transformation in Royal Mail

information exchange. To achieve this, the process must be managed through a partnership between the European Commission and the member states.

MODULE 1: INTRODUCTION TO STRATEGIC MANAGEMENT ACCOUNTING

Participatory planning and monitoring

Performance indicators at TUM library

RIA GUIDELINES INTRODUCTION

Developing a Library Public Relations Program

Quality Management System Guidance. ISO 9001:2015 Clause-by-clause Interpretation

O ver recent years the desire for HR to act as a strategic partner within organizations

BEREC Opinion on. Phase II investigation. pursuant to Article 7 of Directive 2002/21/EC as amended by Directive 2009/140/EC: Case FI/2013/1498

Preprint.

MULTICHANNEL MARKETING FOR PHARMA

Evaluating Implementations of GIS

STRATEGIC TRACKING BENEFITS OF STRATEGIC TRACKING

THE INWENT CAPACITY BUILDING CONCEPT

Identifying Cultural Variables in Information-Seeking

User s Satisfaction of Kuwait E-Government Portal; Organization of Information in Particular

The 28th IFLA Pre Conference of Library and Research Services for Parliaments 8 to 10 August 2012

Dear friends and colleagues,

Enhancement of International/Regional Partnership

Terms of Reference for a Gender Analysis

TARGET MARKET AND MARKET SEGMENTATION

THE IMPACTS OF OPEN DATA: TOWARDS EX POST ASSESSMENT

POLICY RECOMMENDATIONS FOR SUSTAINABLE EQUINE INDUSTRIES

Understanding the Current Situation of E-Government in Saudi Arabia: A Model for Implementation and Sustainability

VICTORIAN COMMUNITY LEGAL SECTOR OUTCOMES MEASUREMENT FRAMEWORK

Developing a Global Vision Through Marketing Research. Chapter 8

CHALABI CERIST, ALGERIA

VICTIMS SUPPORT PROGRAMME PEACE IV. Guidance Note on Monitoring & Evaluation G6/VSS

How to effectively commission research

1) Introduction to Information Systems

RAMP Study Executive Summary EXECUTIVE SUMMARY

CCU 2010 / Identifying User Needs and Establishing Requirements. Lesson 7. (Part1 Requirements & Data Collection)

Procurement system for Harley Davidson Motor Company Project Proposal

ELECTRONIC SERVICES, A REAL SUPPORT FOR CITIZENS AND BUSINESS COMMUNITY

THE COSTS AND BENEFITS OF DIVERSITY

OPQ Universal Competency Report OPQ. > Universal Competency Report. Name Ms Sample Candidate

AUDIT Where are we now? ONGOING MEASUREMENT Are we getting there?

Case Study: Netlife. Timeline. 85 employees 2016 Started to use CV Partner > 50% less time spent on CVs

Customers : Reaching, servicing, and learning

DEFINING CRITERIA FOR EVALUATING THE EFFECTIVENESS OF EU ENVIRONMENTAL MEASURES

Building Life Cycle and Tacit Knowledge Base of Best Practice

Vacancy Details. Vacancy code: VA/2018/B5109/14797

PROGRAM EVALUATIONS METAEVALUATION CHECKLIST (Based on The Program Evaluation Standards) Daniel L. Stufflebeam, 1999

II. INFORMATION NEEDS ASSESSMENT: A TOP-DOWN APPROACH

ACTION PLAN FOR CULTURAL STRATEGY

CHAPTER 4: RESEARCH METHODOLOGY

THE FUTURE OF VFM. A consideration of the challenges and potential solutions for improving its measurement and application.

European Capitals of Culture (ECoC) Guidelines for the cities' own evaluations of the results of each ECoC

Darshan Institute of Engineering & Technology for Diploma Studies

INFORMATION BOOKLET. September Library Assistant Donegal County Council

Switching Behavior in Social Networking Sites:

Online information services: present plans and future projects. The "Pergamon InfoLine view"

A SYSTEMATIC AND STRATEGIC SYSTEM FOR DATA COLLECTION MANAGEMENT AT STATISTICS NORWAY

Measuring e-government

Marketing Research as a Tool for Finding Library Users' Needs and Demands: Application of Three Party Theory

Identifying Important Regional Industries: A Quantitative Approach

CHAPTER 3: RESEARCH METHODOLOGY

ISO 2018 COPYRIGHT PROTECTED DOCUMENT All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of th

Monitoring and Evaluation: A Logical Frame. Evaluating the Impact of Projects and Programs Beijing, China, April 10-14, 2006 Shahid Khandker, WBI

THE MANAGEMENT'S ROLE IN THE SUCCESS OF THE INTRODUCTION OF MANAGEMENT INFORMATION SYSTEMS

Customer Service Strategy. Adelaide City Council. Contents

COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

Exchange is the basis of marketing, with both parties to the exchange receiving something of value.

DEVELOPMENT OF AHP BASED MODELS AND DSS FOR STRATEGIC PLANNING OF LOCAL SUSTAINABLE DEVELOPMENT

Has there been a structural improvement in US productivity?

COMPONENTS AND ASSESSMENTS

People s Report Card Technical Note

Interactive Forest Planning with NIPF Owners

Monitoring, evaluation and impact assessment of innovationenhancing

INFORMATION MANAGEMENT (IM)

ACTIONPLAN FOR THE CULTURE STRATEGY

CRITICAL SUCCESS FACTORS FOR ITIL BEST PRACTICES USAGE

Economists Online: user requirements for a subject repository

Qm 2 nonprofits

The Information System Development Research Based on the Middle Managers Competency Evaluation

Economic Opportunity Act (EOA) of 1964:

A Public Interest Framework for the Accountancy Profession

Direct-to-Consumer (D2C): A Change in Consumer Behaviour? Issued: Contact Information. Julian Rice. Julian Rice, Head of Marketing

CONSULTANCY SERVICES FOR EVALUATION OF THE ECOWAS PROGRAMME ON GENDER MAINSTREAMING IN ENERGY ACCESS (ECOW-GEN) IN ECOWAS COUNTRIES

WHATS NEW IN ISO 9001:2015

The Development of the Critical Factor Index Method

Lecture 1: Introduction to Marketing; The Marketing Environment and Market Analysis Chapters 1.

Is your PMO what it should be? A model to define which functions a PMO should perform, taking into consideration the expected benefits of its clients.

Key Competences Review 2017

Generating Value from Investments in Business Analytics. Rajeev Sharma, Tim Coltman, Abhijith Anand, University of Wollongong

BCS HIGHER EDUCATION QUALIFICATIONS Level 6 Professional Graduate Diploma in IT. October 2012 EXAMINERS REPORT. Software Engineering 2

Revealing Semantic Quality Characteristics of Free and Open Source Software

To achieve this, the public sector must modernise internally as well as in its relations with the public.

5th European Microfinance Award Microfinance and the Environment Application form

Lesson 3: Monitoring, evaluation and indicators

Successful service delivery ways of working with employers

THE ADDED VALUE OF HOUSEHOLD LEVEL ANALYSIS IN ACTIVE DEMAND: USING PSYCHOLOGICAL CONCEPTS IN UNDERSTANDING CHANGING ELECTRICITY CONSUMPTION BEHAVIOUR

Transcription:

Purdue University Purdue e-pubs Proceedings of the IATUL Conferences 2014 IATUL Proceedings The Difficulty of Indicating Transformation: The Challenge for Library Statistics and Surveys Heli Kautonen The National Library of Finland, heli.kautonen@helsinki.fi Markku Laitinen The National Library of Finland, markku.laitinen@helsinki.fi Anna Niemelä The National Library of Finland, anna.niemela@helsinki.fi Heli Kautonen, Markku Laitinen, and Anna Niemelä, "The Difficulty of Indicating Transformation: The Challenge for Library Statistics and Surveys." Proceedings of the IATUL Conferences. Paper 2. http://docs.lib.purdue.edu/iatul/2014/performance/2 This document has been made available through Purdue e-pubs, a service of the Purdue University Libraries. Please contact epubs@purdue.edu for additional information.

The Difficulty of Indicating Transformation: The Challenge for Library Statistics and Surveys Heli Kautonen, Markku Laitinen and Anna Niemelä The National Library of Finland Abstract Academic libraries have long refined the tools and skills in order to show their positive impact to research, to science and to the overall society. The aim is to provide essential information for library managers and decision makers. However, the qualitative and quantitative analyses showing libraries impact should also provide indications about change, i.e. the transformation of such social practices that challenge the essence of libraries. This paper presents two sources of information for analysing libraries impact on the national level. The first is the Finnish Research Library Statistics Database and the second is the National Library End-User Survey of Finnish Libraries. The Database includes the annual statistics of all Finnish higher education institution libraries and also several special libraries. The End-User Survey is an online questionnaire aimed at the customers of all library sectors. These sources of information have been collected for several years and they can therefore reveal changes in time. They have been adjusted to the changing information needs and the transforming functions of libraries. Still, the society seems to transfer faster than these mechanisms can capture. This paper discusses reasons why the transformation of social practices may be out of the reach of traditional information sources for analysing libraries impact. There are methodological issues to be considered. Furthermore, the paper examines the newly-published ISO 16439 standard on impact evaluation. The aim is to discuss how the standard comports with the recent practices of nationallevel data collection. The paper ends with discussion on complementary information sources which can be useful in tracing societal transformation. 1 Introduction Academic libraries have long refined the tools and skills in order to show their positive impact to research, to science, and to the overall society. The aim of this work is to provide essential information for library managers and decision makers. However, the qualitative and quantitative analyses showing libraries impact should also provide indications about change, i.e. the

transformation of such circumstances that challenge the essence of libraries. This concern becomes graver in the times of economic stress, when organizations have to justify their existence and prove their ability to serve their customers. Alongside with the development of digital services, more effort is put on gathering qualitative and quantitative data about library patrons behaviour and preferences (e.g. Connaway & Dickey 2010), as well as their problems dealing with library services (Kress et el. 2010). This customer-centred approach is essential, but it doesn't fully take into consideration the social aspects of peoples behaviour or the motivation behind the behaviour. The reasons and ways people use libraries can be viewed as a manifestation of social practices. As Shove et al. (2012) argue, people construct their everyday life, not only based on individual choices but, in relation to shared social conventions. The existence of social practices involves the elements of materiality, (such as premises or library systems,) competence, (such as information literacy and information retrieval skills,) and meaning (i.e., the variety of motivations urging people to spend their effort on doing something). Consequently, Shove et al. point out that any emerging transformation in a society takes time because it implies changes in these elements and their interconnections to other social practices. The transformation of social practices is usually slow, and therefore challenging to detect. The concept of impact is not less complicated to define. Even though it is easy to understand the matter intuitively, it is difficult and challenging to show the societal impact of the libraries in practice - not only because of the wide meaning of the concept but also because the exact indicators describing the impact of libraries have not been available. Adapting Nagarajan & Vanheukelen (1997:87-104), impact can be explained as broad, general societal changes that indicate, for example, the extent to which the impacts of an organization (a university or another educational institution, museum, research institution, a public library) have promoted the achievement of goals set, either general or specific. Yet, the definition above is too wide to the needs of measuring in practice because showing causality (e.g. library s role in social welfare) on the macro level may be impossible or at least controversial. So, there clearly is a need for a more restricted definition of impact. In 2010, a work group nominated by ISO (the International Organization for Standardization) started its work to formulate a new international standard that will give the terms of reference for 2

assessing the impact of libraries. The result of the work of the group, the international standard ISO 16439, Methods and procedures for assessing the impact of libraries, has been completed recently and it will be published these days. In the new standard, the utilization of various methods from interviews cost-benefit analysis is recommended. In this paper, the focus is on two sources of information for analysing libraries impact. The National Library of Finland offers two evaluation infrastructure services for libraries in Finland: The Finnish Research Library Statistics Database and the National Library End-User Survey of Finnish Libraries. Their purpose is to support libraries evaluation and development work. This paper examines the possibilities and limitations of these data types from viewpoint of a national service provider with organizational customer base, but an interest towards end user insight, too. 2 ISO 16439 standard as a tool for evaluation The new standard ISO 16439, Methods and procedures for assessing the impact of libraries, is the first documentation concerning the evaluation of the impact of libraries. In the standard, the concept of impact and other focal terms are defined and the standard is intended as a tool for the evaluation of impact of all types of libraries. It is suggested to be used for the following purposes: - Strategic planning and management of libraries - Comparison of library impact in the course of time and between libraries of similar type and mission - Promoting the libraries role and value for learning and research, education and culture, social and economic life - Supporting political decision making that concern the services and strategic goals of libraries Though all the methods described in the standard are not applicable to all libraries, the standard in general is applicable to all types of libraries in all countries. There are many different methods for evaluating the impact depending on which of the dimensions of the impact is desired for measuring. In a nutshell, the library s mission can be defined as supporting the strategy of its mother organization, whether it is a municipality, school, an institution of higher education, research 3

institute etc. So, the most evident target for measuring the impact of the library is to evaluate the desired impact of the library, to evaluate how the objectives of the library as the support in the carrying out of the strategy of its frame organization have been achieved, and what was the share of the library in the reaching of the objectives of its frame organization. Illustration of the idea of impact and description of focal terms is shown in Fig 1. Fig 1 Connections between input, output, outcome and impact. Input to the library turns on the service processes as a result of which outputs and outcomes are created. Turning of any of the cogs affects all the others. To get the desired impact, one must define the number and quality of the needed outputs and outcomes. Inputs must be adapted to produce the needed outputs and outcomes. Quality is the oil that makes the processes more effective. Having the library operations positive impact, societal or monetary value may be detected. The standard is intended to be taken into use in all countries, and it affords a uniform framework to the evaluation of the impact of libraries making a comparable evaluation feasible. 3 National data gathering 3.1 National surveys The national user survey of Finnish libraries has been conducted three times. The latest was carried out in March and April 2013. The survey is carried out through an open online questionnaire aimed at the customers of all library sectors: the public libraries, the university libraries and the libraries of universities of applied sciences, as well as the special libraries that are in e.g. research institutions, foundations or other organisations. The questionnaire consists of several topics: purposes of using 4

the library; use of electronic services; opinions on the library as a service environment, and views on quality, importance and benefits of library services. The contents and the structure of the survey are developed in cooperation with the different library sectors while the National Library has taken care of the costs, implementation and outsourcing of the technical execution of the survey. Regular, national-level data collection enables long-term monitoring of customer satisfaction and needs. In the light on customer satisfaction trends it is possible to evaluate e.g. the success of certain development operations from customers point of view. The survey also provides comparable data based on which individual libraries can put their results in a national or regional perspective. The library users have been active in giving feedback: the latest user survey received a total of 35 000 responses. The majority (54 %) of the respondents represented scientific libraries; 46 % of the responses became from public libraries. The information that is displayed in the surveys is customer-oriented and qualitative by nature. In terms of indicating the transformation, the development of customer satisfaction is not the only question that matters; regular surveys can bring forth changes e.g. in customers priorities and their habits of library use. Through experience-based information it is also possible to make visible some dimensions of libraries impact and value. There are certain limitations that regard survey data in general, and also some that are related to this survey and its data collection practices. One essential limitation results from the gap between survey participants and real library users. Various user studies suggest that e.g. age and gender have a significant role in library customers behaviour and expectations (Pors 2010). Yet, the national survey does not reach all the user groups and thus the results represent only a limited section of clients: the clear majority were women, highly educated and working- aged. Second, in the light of the more or less flattering results, it can be assumed that the critical clients are not very eager to participate in surveys (Serola & Vakkari 2012). If surveys are taken only by users who are active visitors and to some extent committed clients, a lot of useful information may be lost. Another issue to consider is setting the right questions. Challenges may arise in searching for balance between long-term monitoring of customer satisfaction and the need to continuously customize the content of the questionnaire up-to-date. When customers conditions of work or 5

study, as well as their information behaviour and needs may change rapidly, survey questions may lose their relevance in a couple of years or less. So, the picture of libraries benefits provided by survey results is restricted and somewhat traditional. Library users changing behaviour and needs, and thus libraries impacts overall stay at least partly out of reach. Some of these limitations can be solved by developing the methods and practices of conducting the survey: by creating new practices of data collection in order to involve also less regular and typical users, and by formulating the questionnaire more flexible based on an individual library s needs. 3.2 National statistics The annual statistics of Finnish scientific libraries are collected in the statistics database of scientific libraries according to the standard ISO 2789, International library statistics. The database is maintained by the National Library of Finland. It can be accessed through the Internet and it is open for all interested without logging in. The database can be browsed using the two official languages of Finland (Finnish and Swedish) plus English. The statistics of the scientific libraries of Finland give a general view on the resources, collections and services of Finnish scientific libraries. The statistics contain information about the national library, about the university libraries, about the libraries of applied sciences and about a few special libraries. Traditionally, the librarians have been good at collating statistics and other information but poor at utilizing the results effectively. Neither the way of collecting statistics nor the contents of statistics has supported proactive way of planning. The traditional library statistics merely quantified the extent of the collections and everyday performance of the library but information about quality or impact of the library performance was either hidden behind the huge number of data or not available at all. However, the rhetoric of using the library statistics is little by little changing from simply measuring resources to conducting an analysis of the library s impact. In the current economic climate, it is not enough for the library simply to quantify the extent of the resources or how they are used but one must also be able to show that investment in the library and availability of resources produced better results and value to the patrons and that these services were being delivered in an efficient manner. 6

In Fig 2, a comparison of developments of the number of full-text article downloads of electronic journals and cost of electronic journals in the Finnish universities of applied sciences is shown. Fig 2 The growth of e-journal article downloads in compared with the costs has been manifold in the Finnish universities of applied sciences; while the annual number of downloads has grown from 360 000 in 2006 to 1.2 million in 2013 (3.3-fold), the annual cost of e-journals has grown from 0.5 million euro in 2006 to 0.8 million euro in 2013 (1.6-fold). Information from 2013 is preliminary. Source: Finnish Research Library Statistics. Evidently, these materials are cost-efficient because the use of the material has increased faster than the money used for them. The strong increase of use with a much slower increase in the cost could implicate the successful information and successful choices of materials, but increasing need of clientele as well. Finding an answer to the last mentioned matter goes with the territory of user surveys. 4. Combining data from different sources In order to obtain full benefit from the statistics and e.g. user surveys, the information collected needs to be refined. It also shall be linked with the information collected in other ways, for example with information from the user surveys, accounting material, or even from other statistics, such as with information obtained from the demographic statistics, geographic information etc. to help in the evaluation and future planning. In order to obtain full benefit from the statistics and e.g. user surveys, the information collected needs to be refined. It also shall be linked with the information collected in other ways, for example with information from the user surveys, accounting material, or even from other statistics, such as 7

with information obtained from the demographic statistics, geographic information etc. to help in the evaluation and future planning. When explored concurrently, these different data types can complement each other for assessment purposes and together provide stronger evidence than if they were handled separately. Certain changes in library services that are displayed in statistics can be compared with the survey results in order to explore if users views somehow reflect the development. Statistics on physical library visits, for instance, combined to survey results on the visiting purposes or significance of e-resources can display changes in customer expectations. On the other hand, combined data sets may suggest the transformation of the everyday practices of academic work as well as citizens information retrieval. In the standard ISO 16439, the term mixed methods describing the combination and utilization of different types of statistical data is used. The mixed methods give new tools for measuring the efficiency of the work being done in libraries and what possible impact they may have in their parent organizations and in the society (Error! Reference source not found.virhe. Viitteen lähdettä ei löytynyt.). Fig 3 Combining data from different sources produces new tools for the analysis of the impact of library. These mixed methods are a step towards achieving a more comprehensive impact assessment of the library. The mixed methods may help to acquire a better understanding how library services contribute and support user success best and by this way to apply that information for improvement and advocacy 8

for library. In library practice, the mixed methods may be applied in iterative processes where the results of one study are followed up by another to gain a better understanding of issues and impact. In Error! Reference source not found.virhe. Viitteen lähdettä ei löytynyt., a follow-up study of user satisfaction (qualitative data) and use and input in electronic journals (quantitative data) in the Finnish universities of applied sciences is shown. The use of electronic journals has grown fast while the costs and number of titles in use have increased moderately. Respectively, the user satisfaction with the materials is high. Fig 4 Follow-up of user satisfaction, number of use and titles and cost-efficiency of electronic journals in the Finnish universities of applied sciences. The users evaluated their satisfaction on scale 1 to 5, the numerical values being 3.69 in 2006, 3.73 in 2007, 3.95 in 2010 and 3.91 in 2013. Statistical information from 2013 is preliminary, user survey results are final. Source: Finnish Research Library Statistics and the National User Survey. 5. Discussion The changing world sets challenges to library services as well as to evaluation data regarding these services. Besides monitoring things like the number of library visits, the number of loans or level of customer satisfaction, the evaluation data should also be able to help the libraries in showing impact and recognising end users changing needs and behaviour. A cost-effective service may turn into ineffective if the motives of customer behaviour are not reflected. 9

Thus, there is a need to move beyond the traditional library performance measurement and to focus on the contribution and value that library services provide to their users. Besides, it is essential to consider that this value is not necessarily constant. Libraries desired impacts and expected benefits change in time. The collection and utilization of evaluation data should keep up with these changes. In order to stay up-to-date, statistical data and survey results can be complemented by various data types. In addition to data on library customers interviews, log analysis also other data sources may be worth considering. Statistics regarding academic performance within library s frame organisation, population statistics, non-user studies, studies on information behaviour, urban studies, or leisure time studies may all provide viewpoints that can help in identifying change and thus figuring out customers needs and understanding the impact. References Broady-Preston, J. & Lobo, A. (2011) Measuring the quality, value and impact of academic libraries: the role of external standards. Performance Measurement and Metrics, 12, 122 135 Connaway, L.S. & Dickey, T.J. (2010) The Digital Information Seeker. Report of findings from selected OCLC, RIN and JISC user behavior projects. OCLC Research. Kress, N., Bosque D.D., & Ipri, T. (2010) User failure to find known library items. New Library World, 112, 3/4, 150 170. Maceviciute, E. (2014) Research libraries in a modern environment. Journal of Documentation 70, 282 302. Nagarajan, N. & Vanheukelen, M. (1997): Evaluating EU expenditure programmes: A guide, ex post and intermediate evaluation. Luxembourg 1997: European Commission. Pors, N. (2010).Burning platforms and melting icebergs. An exploratory analysis of present strategic challenges and cross-pressures in the public libraries. Performance Measurement and Metrics, 11, 9 24. Seppänen, A-P. & Laitinen, M. (2012): Libraries do make a difference: Common principles in showing the impact of different types of libraries. QQML Journal 1, 3-11. http://www.qqml.net/papers/july_issue/1qqml_journal_2012_seppanen_laitinen_1_3-11.pdf (accessed 24th Apr 2014). 10

Serola, S. & Vakkari, P. (2012). Perceived outcomes of public libraries. Library & Information Science Research 36, 37 44. Shove, E., Pantzar, M. & Watson, M. (2012). The Dynamics of Social Practice. Everyday Life and How it Changes. London, UK: SAGE Publications. 11