9 Principles of (alt)metrics

Transcription

1 1 9 Principles of (alt)metrics Special Libraries Association Vancouver, BC June 10, 2014 Michael Habib, MSLS Sr. Product Manager, Scopus habib@elsevier.com twitter.com/habib orcid.org/

2 PRINCIPLE 1: Metrics should be used alongside, not as a replacement for peer review and expert opinion 2

3 PRINCIPLE 2: Be clear on the question that is being asked 3

4 4 PRINCIPLE 2: Be clear on the question that is being asked Top-down evaluation? - A publisher assessing a journals performance - An editor seeking authors for a special issue Bottom-up showcase? - An author determining which journal to publish in - A researcher determining which articles to read

5 PRINCIPLE 3: Multiple metrics tell the most complete story 5

6 6 PRINCIPLE 3: Multiple metrics tell the most complete story 1. There is no single perfect metric to answer any question 2. Optimal decision-making requires input from at least two metrics 3. Alternative Metrics are not a replacement for citations 4. Alternative Metrics are not a replacement for usage metrics 5. Alternative Metrics are not one thing

7 7 6 classes of alternative metrics 1. Scholarly activity Counts of activity in scholarly platforms eg, Mendeley, Citeulike, Zotero, etc (in use). 3. News/Mass Media Counts and links to scholarly works found in the mass media, eg, NY Times, BBC, HuffPo 5. Re-use Indicators for the re-use and citation of scholarly artifacts, primarily data, 2. Scholarly commentary Counts (and potentially links) to where scholars have annotated, discussed and reviewed articles, eg, recognized scholarly blogs, F1000 Prime reviews, Publons, Peerpub, Mendeley annotations. Potentially qualitative. 4. Social activity Counts of links to research material found on generic social networks, eg, Twitter, Facebook, Pinterest, Delicious 6. Legal, commerce and governance Indicators for the use and citation of scholarly output by government and governance, eg, policy documents, law, patents

8 8 Mendeley Readership Statistics Beta launched Mar

9 Scholarly activity indirect measurement of activity by people using scholarly platforms, e.g., Mendeley, Zotero, Citeulike. 9

10 10 Altmetric for Scopus Released June

11 11 Scholarly activity Social activity Scholarly commentary Mass media coverage

12 12 Scholarly Impact Scholarly activity Social activity Scholarly commentary Mass media coverage

13 PRINCIPLE 4: Metrics should be as simple as possible 13

14 Composite metrics should be avoided! 14

15 15 PRINCIPLE 5: No methodological black boxes

16 All calculation methods must be freely available, and not proprietary 16

17 17 PRINCIPLE 6: Metrics should be agnostic

18 PRINCIPLE 7: Disciplinary differences and other factors must be taken into account 18

19 19 PRINCIPLE 7: Disciplinary differences and other factors must be taken into account Normalize for differences in research behavior between fields - Source-Normalized Impact Per-Paper (SNIP) for altmetrics? - Scopus books expansion project - Zotero vs. Mendeley? Account for differences in size and age of the entities being measured - Scopus to add cited references for pre-1996 content Distinct disciplines will balance the contributions of metric and peer review input differently Cultural differences need to be accounted for - Twitter vs. Facebook vs. Sina Weibo - interview with Juan Pablo Alperin (@juancommander)

20 20 PRINCIPLE 8: Narratives help to interpret metrics (case studies, etc.)

21 21 PRINCIPLE 9: Ownership and acceptance of metrics by communities

22 22 Snowball Metrics the essentials Global standards for institutional benchmarking Endorsed by a group of distinguished UK universities to support their strategic decision making Tried and tested, non-proprietary methodologies that are available free-ofcharge to the higher education sector Absolutely clear, unambiguous definitions enable apples-to-apples comparisons so universities can benchmark themselves against their peers to judge the excellence of their performance Snowball Metrics are unique because: Universities drive this bottom up Academia industry collaboration

23 23 Snowball Metrics approach Vision: Snowball Metrics enable benchmarking by driving quality and efficiency across higher education s research and enterprise activities, regardless of system and supplier Universities endorse metrics to generate a strategic dashboard Draw on all data: university, proprietary and public Ensure that the metrics are system-agnostic that they can be calculated regardless of systems and data structures Snowball Metrics Project Partners

24 25 Snowball Metrics are feasible Note: this pilot was built for the UK project partners, and is not available more widely. 25

25 26 Metrics can be size-normalized Note: this pilot was built for the UK project partners, and is not available more widely. 26

26 27 Metrics can be sliced and diced Note: this pilot was built for the UK project partners, and is not available more widely. 27

27 Scholarly Activity 28

28 Scholarly Activity per FTE 29

29 Scholarly Activity per publication 30

30 Scholarly Commentary per publication 31

31 Social Activity 32

32 Mass Media per publication 33

33 34 Focus on societal-economic impact Snowball Metrics endorse alternative metrics to be shared in Recipe Book v2 Develop Strategy Inputs: Enabling Research Processes: Doing Research Outputs & Outcomes: Sharing Research Secure & Recruit/evaluate Manage Establish researchers Funding partnerships Manage facilities Search, discover, read, Collaborate & Synthesize/ review network Experiment Analyze Manage Data Publish and disseminate Promote and Commercialize showcase (esteem) Benefit? $! Esteem Scholarly Activity Number of times that an institution s output has been posted in online tools that are typically used by academic scholars e.g. Mendeley, CiteULike, Academia.edu, ResearchGate Scholarly Commentary Number of times that an institution s output has been commented on in online tools that are typically used by academic scholars e.g. science blogs, video posts, peer reviews, post-publication comments, Wikipedia! Impact Mass Media Number of times that an institution s output has been referred to by press clippings and news websites Social Activity Number of times that an institution s output has stimulated social media posts e.g. Facebook, Twitter, Reddit, LinkedIn Societal-economic Impact Altmetrics

34 35 Euan Adie, Founder of Altmetric.com "We're very supportive of Snowball Metrics and of the worthy principles behind it, so I'm very happy to see the project partners take the imitative and put forward practical standards for altmetrics in the context of benchmarking. The approach taken is a smart one - structuring the way you look at altmetrics like this is a good way of maximizing the insights that the data can provide. We're working towards including the recipe in our own reporting tools, to make it as easy as possible to put into practice."

35 36 PRINCIPLE 9: Ownership and acceptance of metrics by communities 1. That will both use them to assess performance 2. Whose performance will be assessed

36 37 9 Principles of (alt)metrics 1. Metrics should be used alongside, not as a replacement for, peer review and expert opinion 2. Be clear on the question that is being asked 3. Multiple metrics tell the most complete story 4. Metrics should be as simple as possible 5. No methodological black boxes 6. Metrics should be agnostic 7. Disciplinary differences and other factors must be taken into account 8. Narratives help to interpret metrics (case studies, etc.) 9. Ownership and acceptance of metrics by communities

37 38 Thank you! Michael Habib, MSLS Sr. Product Manager, Scopus twitter.com/habib orcid.org/ Issue 37 June