Societal quality of research in a hospital context

Transcription

1 Societal quality of research in a hospital context Ingeborg Meijer Centre for Science and Technology Studies, Leiden University SURFacademy Seminar February 18th, 2016

2 Outline Conceptual framework for research assessment Societal quality in a Dutch hospital Connection to SEP? 1

3 Conceptual framework 2

4 Scientific & societal quality Scientific returns Social returns Scientific interactions Professional interactions Public interactions Private interactions Cultural returns Economic returns 3

5 BUT: measuring societal relevance is complicated Lack of idea of the stakeholders; what kind of interactions. Also: lack of awareness Lack of systematic recording, and therefore lack of sufficient datasets. Many interactions are still implicit, instead of explicit. Methods to retrieve information from the Internet immature. Lack of in-depth research into the value of these interactions, quantitative approaches and visualisation methods. 4

6 Approaches Qualitative approach A case study on societal quality in a hospital Quantitative approach Other datasources, such as University - Industry co-publications Patents Altmetrics Question: what does it tell you about the societal quality of research?? 5

7 Societal Quality in a Dutch hospital a case study 6

8 Methodology (1) Step-by-step procedure following a bottom-up approach Both product and process indicators Combined with standardised scientific quality measurement (CWTS) 1. Open questionnaire to all departments 2. Qualitative dataset with societal interactions 3. Reduction and/or clearance of dataset 4. Clustering into main categories (indicators) 5. Quantifying data 7

9 Operationalisation of societal quality 8

10 Methodology (2) Step 1 Indicator occurrences Number of occurrences of each indicator for every department is related to the average of that indicator for all departments Step 2 Weighting system Ranking of indicators in relation to each other for each target group Calculation of weighting factors (within each societal target group and between target groups) Step 3 Societal quality per indicator Result of steps 1 and 2 per department and per indicator Relative weighting scores per indicator Step 1 Relative occurences per indicator per department Step 2 Step 3 Societal quality per indicator per department Step 4 Relative weighting factors per indicator Step 4 Societal quality per target group and total societal quality Total general public societal quality score per department Total healthcare professionals societal quality score per department Total private sector societal quality score per department Calculation of societal quality based on the indicator societal quality scores (step 3) Total societal quality score per department 9 9

11 Societal quality score per target group 10

12 Discussion methodology Robustness of method Weighting factors connecting to objectives? Results coincided with the perceptions Quantitative vs. qualitative indicators Societal quality vs. societal impact No correlation with scientific quality! Turning it into a pragmatic method that is easy to repeat Not taken into account/improving the method: Policy related issues/mission & objectives Number of patients Size of departments/number of researchers Stakeholders 11

13 Connection to SEP? 12

14 Evaluation model of societal quality measurement Practical guide: based on logical models and productive interaction concepts Step 1 Define societal mission & objectives Step 4 Reflection on realising societal quality Step 2 Describe interactions with societal stakeholders/sector Step 3 Describe results & added value of societal interactions 13

15 New SEP: triple focus Research Quality (no production) Relevance to society Viability (the extent to which the unit is equipped for the future) 14

16 Universiteit Leiden 15

17 Bevindingen uit onderzoek Scientists see valorisation and societal impact as separate from methods to assess sciences (i.e. publication and citations) Activities for society are not necessarily perceived as positive Societal output is part of other obligations; not related to science Basic research vs Long term societal impact; is not my responsibility University management see science assessment and third mission of universities as separate Development of scientific careers Valorisation is task of separate policy officer Stakeholder roles still unclear absorption of science no societal mission and objectives at research group level Research management information systems Qualitative versus quantitative discussion Counting beans vs contextualisation WoS of societal relevance 16

18 Thank you! Want to know more...contact me at 17