SciVal Metrics Guidebook

Size: px

Start display at page:

Download "SciVal Metrics Guidebook"

Sharlene Patrick
6 years ago
Views:

1 TITLE OF PRESENTATION 11 SciVal Metrics Guidebook Dr Lisa Colledge Senior Manager for Strategic Alliances March 27 th, 2014 #SciVal

2 TITLE OF PRESENTATION 22 SciVal Metrics Guidebook

3 TITLE OF PRESENTATION 33 Agenda 1. SciVal: a quick overview 2. Entities: what you can generate metrics for 3. Metrics in SciVal 4. Selection of appropriate metrics 5. Think through a couple of questions 3

4 SciVal: a quick overview SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 44

5 TITLE OF PRESENTATION 55 Elsevier Research Intelligence portfolio 5

TITLE OF PRESENTATION 66 The layers of SciVal Using advanced data analytics super-computer technology, SciVal allows you to instantly process an enormous

6 TITLE OF PRESENTATION 66 The layers of SciVal Using advanced data analytics super-computer technology, SciVal allows you to instantly process an enormous amount of data to generate powerful data visualizations on-demand, in seconds. Query around 75 trillion metric values Scopus data only 1996 onwards Weekly update 6

TITLE OF PRESENTATION 77 SciVal in a nutshell SciVal offers quick, easy

collaborative partnerships Ready-made-at a glance snapshots of any

7 TITLE OF PRESENTATION 77 SciVal in a nutshell SciVal offers quick, easy access to the research performance of 220 nations and 4,600 research institutions worldwide. Visualize research performance Benchmark your progress Develop collaborative partnerships Ready-made-at a glance snapshots of any selected entity Flexibility to create and compare any research groups Identify and analyze existing and potential collaboration opportunities 7

8 Entities what you can generate metrics for SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 88

TITLE OF PRESENTATION 99 Scopus content and

Algorithm: 99% precision, 93% recall Manual

authority of affiliation for 100% precision

95% recall Manual reassignment on feedback

9 TITLE OF PRESENTATION 99 Scopus content and entities in SciVal Affiliation Profile Algorithm: 99% precision, 93% recall Manual reassignment on feedback from official authority of affiliation for 100% precision Author Profile Algorithm: 99% precision, 95% recall Manual reassignment on feedback for 100% precision ~30 million publications in SciVal

10 TITLE OF PRESENTATION Metrics in SciVal

TITLE OF PRESENTATION 11 11 Groups of metrics in SciVal Broad range of metrics is

11 TITLE OF PRESENTATION Groups of metrics in SciVal Broad range of metrics is essential for you to be able to address your many questions, and to play by the only rule

12 TITLE OF PRESENTATION The only metrics rule: triangulate! No single metric is perfect. Always use at least 2 metrics to give insight into your question No data set is perfect. Always reinforce your evidence-based conclusions with at least one of, and ideally both, peer review and expert opinion

13 TITLE OF PRESENTATION Selection of appropriate metrics

14 TITLE OF PRESENTATION Step 1: know your question Evaluation of performance Typically top-down performed by an entity with authority Essential to account for variables that can affect metrics values besides differences in performance, so that the evaluation is fair Demonstration of excellence Typically bottom-up performed by an entity requesting resource Generally aiming to find a way to ensure the entity looks strong compared to peers; non-performance variables may be used to advantage Scenario modeling Non-performance variables may or may not be important depending on scenario e.g. if modelling recruitment in nanoscience, might not need to worry about different citation rates between fields because only looking at one field!

15 TITLE OF PRESENTATION Non-performance variables? 1. Size 2. Discipline 3. Publication-type 4. Database coverage 5. Manipulation 6. Time

TITLE OF PRESENTATION 16 16 Non-performance variable 1: size Some metrics tend to have high

16 TITLE OF PRESENTATION Non-performance variable 1: size Some metrics tend to have higher values for bigger entities e.g. Scholarly Output Evaluation: probably usesize-normalized metrics, and the percentage (not count) option when it is available Demonstrating excellence: might use a power metric if you are relatively big

TITLE OF PRESENTATION 17 17 Non-performance variable 2: discipline Some disciplines tend to have high

17 TITLE OF PRESENTATION Non-performance variable 2: discipline Some disciplines tend to have higher values due to their researcher behavior e.g. Neuroscience Evaluation: be careful when comparing entities in different fields or with different disciplinary focuses. Consider using field-normalized metrics OR filter on a discipline Demonstrating excellence: you might ignore if you are in life sciences, but fieldnormalize of filter if you are in Social Sciences

TITLE OF PRESENTATION 18 18 Non-performance variable 3: publication-type Some publication-types tend to receive more citations Evaluation: use

18 TITLE OF PRESENTATION Non-performance variable 3: publication-type Some publication-types tend to receive more citations Evaluation: use publication-type-normalized metrics OR filter on publication types if this matters to your evaluation Demonstrating excellence: you might ignore if you are a journal editor (lots of editorials!)

19 Non-performance variable 4: database coverage SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 19 19

differences probably affect all large entities similarly.

20 Non-performance variable 4: database coverage The publication behavior of some disciplines enables a higher coverage in Scopus than others SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION Evaluation: coverage differences probably affect all large entities similarly. Be careful if evaluating entities with different disciplinary focuses or small entities Demonstrating excellence: be vigilant! Consider focusing on a subset of total output

TITLE OF PRESENTATION 21 21 Non-performance variable 5: manipulation Manipulation is rare, but, if you suspect it, you can check for unscrupulous behavior in SciVal Examples of manipulation: Combine

21 TITLE OF PRESENTATION Non-performance variable 5: manipulation Manipulation is rare, but, if you suspect it, you can check for unscrupulous behavior in SciVal Examples of manipulation: Combine research units so that their performance using power metrics appears stronger If you suspect this is going on: try using a size-normalized metric and see whether the relative performance changes, or a metric that is difficult to manipulate Self-citation This is normal and responsible academic behavior It is open to abuse, but abuse is extremely rare If you suspect this is going on: use a metric where you can exclude selfcitations, or a metric that is difficult to manipulate

TITLE OF PRESENTATION 22 22 Non-performance variable 6: time Some metrics need time to pass before they provide really useful information e.g.

22 TITLE OF PRESENTATION Non-performance variable 6: time Some metrics need time to pass before they provide really useful information e.g. those based on counting citations Evaluation and demonstrating excellence: if you are looking at the early stages after a strategic decision, or an early-career researcher, consider using time-independent metrics

23 TITLE OF PRESENTATION Factors that can affect metrics values 1. Size 2. Discipline 3. Publication-type 4. Database coverage 5. Manipulation 6. Time They may not all matter for your question, but think about them. Accounting for these 7. Performance reveals this

24 TITLE OF PRESENTATION Think through a couple of questions

25 Question 1: how does Cambridge perform relative to Harvard? SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION Aspect Performance Size Discipline Publication-type Database coverage Manipulation Time Metric to take into account? Yes Yes Yes No No No No

26 Question 2: how does Cambridge perform relative to Harvard in Medicine? SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION Aspect Performance Size Discipline Publication-type Database coverage Manipulation Time Metric to take into account? Yes Yes Yes No No No No No Taking the different disciplinary focuses into account by using the filter gives me other options for metrics to use. But why would I want to do this?

27 TITLE OF PRESENTATION Field-Weighted Citation Impact Field-Weighted Citation Impact is a very sophisticated and useful metric it normalizes for size, field and publication-type Suitable for: Fair comparison of entities regardless of size, disciplinary profile, and publication-types Avoiding the crash in recent years Immediate understanding of extent of brilliance (1 = average) Selection in case of uncertainty However. You lose the idea of the magnitude of your impact People might not like to see low numbers Complicated calculation that the users won t be able to validate themselves Always using it FWCI as default severely restricts richness of information

28 Guidebook has worked examples for all metrics SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 28 28

29 How does Cambridge perform relative to Harvard in Medicine? SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION Aspect Performance Size Discipline Publication-type Database coverage Manipulation Time Metric to take into account? Yes Yes No No No No No Taking the different disciplinary focuses into account by using the filter gives me other options for metrics to use. But why would I want to do this?

30 TITLE OF PRESENTATION Questions For product information, please visit:

31 TITLE OF PRESENTATION Thank you! For product information, please visit:

Research Metrics: use and abuse

Research Metrics: use and abuse Lisa Colledge, Director of Research Metrics, Elsevier COPE European Seminar 2015 17 th April 2015 Research metrics indicate many different ways of being good 2 Research