TITLE OF PRESENTATION 11 SciVal Metrics Guidebook Dr Lisa Colledge Senior Manager for Strategic Alliances March 27 th, 2014 #SciVal
TITLE OF PRESENTATION 22 SciVal Metrics Guidebook http://bit.ly/scivalmetricsguidebook
TITLE OF PRESENTATION 33 Agenda 1. SciVal: a quick overview 2. Entities: what you can generate metrics for 3. Metrics in SciVal 4. Selection of appropriate metrics 5. Think through a couple of questions 3
SciVal: a quick overview SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 44
TITLE OF PRESENTATION 55 Elsevier Research Intelligence portfolio 5
TITLE OF PRESENTATION 66 The layers of SciVal Using advanced data analytics super-computer technology, SciVal allows you to instantly process an enormous amount of data to generate powerful data visualizations on-demand, in seconds. Query around 75 trillion metric values Scopus data only 1996 onwards Weekly update 6
TITLE OF PRESENTATION 77 SciVal in a nutshell SciVal offers quick, easy access to the research performance of 220 nations and 4,600 research institutions worldwide. Visualize research performance Benchmark your progress Develop collaborative partnerships Ready-made-at a glance snapshots of any selected entity Flexibility to create and compare any research groups Identify and analyze existing and potential collaboration opportunities 7
Entities what you can generate metrics for SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 88
TITLE OF PRESENTATION 99 Scopus content and entities in SciVal Affiliation Profile Algorithm: 99% precision, 93% recall Manual reassignment on feedback from official authority of affiliation for 100% precision Author Profile Algorithm: 99% precision, 95% recall Manual reassignment on feedback for 100% precision ~30 million publications in SciVal
TITLE OF PRESENTATION 10 10 Metrics in SciVal
TITLE OF PRESENTATION 11 11 Groups of metrics in SciVal Broad range of metrics is essential for you to be able to address your many questions, and to play by the only rule
TITLE OF PRESENTATION 12 12 The only metrics rule: triangulate! No single metric is perfect. Always use at least 2 metrics to give insight into your question No data set is perfect. Always reinforce your evidence-based conclusions with at least one of, and ideally both, peer review and expert opinion
TITLE OF PRESENTATION 13 13 Selection of appropriate metrics
TITLE OF PRESENTATION 14 14 Step 1: know your question Evaluation of performance Typically top-down performed by an entity with authority Essential to account for variables that can affect metrics values besides differences in performance, so that the evaluation is fair Demonstration of excellence Typically bottom-up performed by an entity requesting resource Generally aiming to find a way to ensure the entity looks strong compared to peers; non-performance variables may be used to advantage Scenario modeling Non-performance variables may or may not be important depending on scenario e.g. if modelling recruitment in nanoscience, might not need to worry about different citation rates between fields because only looking at one field!
TITLE OF PRESENTATION 15 15 Non-performance variables? 1. Size 2. Discipline 3. Publication-type 4. Database coverage 5. Manipulation 6. Time
TITLE OF PRESENTATION 16 16 Non-performance variable 1: size Some metrics tend to have higher values for bigger entities e.g. Scholarly Output Evaluation: probably usesize-normalized metrics, and the percentage (not count) option when it is available Demonstrating excellence: might use a power metric if you are relatively big
TITLE OF PRESENTATION 17 17 Non-performance variable 2: discipline Some disciplines tend to have higher values due to their researcher behavior e.g. Neuroscience Evaluation: be careful when comparing entities in different fields or with different disciplinary focuses. Consider using field-normalized metrics OR filter on a discipline Demonstrating excellence: you might ignore if you are in life sciences, but fieldnormalize of filter if you are in Social Sciences
TITLE OF PRESENTATION 18 18 Non-performance variable 3: publication-type Some publication-types tend to receive more citations Evaluation: use publication-type-normalized metrics OR filter on publication types if this matters to your evaluation Demonstrating excellence: you might ignore if you are a journal editor (lots of editorials!)
Non-performance variable 4: database coverage SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 19 19
Non-performance variable 4: database coverage The publication behavior of some disciplines enables a higher coverage in Scopus than others SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 20 20 Evaluation: coverage differences probably affect all large entities similarly. Be careful if evaluating entities with different disciplinary focuses or small entities Demonstrating excellence: be vigilant! Consider focusing on a subset of total output
TITLE OF PRESENTATION 21 21 Non-performance variable 5: manipulation Manipulation is rare, but, if you suspect it, you can check for unscrupulous behavior in SciVal Examples of manipulation: Combine research units so that their performance using power metrics appears stronger If you suspect this is going on: try using a size-normalized metric and see whether the relative performance changes, or a metric that is difficult to manipulate Self-citation This is normal and responsible academic behavior It is open to abuse, but abuse is extremely rare If you suspect this is going on: use a metric where you can exclude selfcitations, or a metric that is difficult to manipulate
TITLE OF PRESENTATION 22 22 Non-performance variable 6: time Some metrics need time to pass before they provide really useful information e.g. those based on counting citations Evaluation and demonstrating excellence: if you are looking at the early stages after a strategic decision, or an early-career researcher, consider using time-independent metrics
TITLE OF PRESENTATION 23 23 Factors that can affect metrics values 1. Size 2. Discipline 3. Publication-type 4. Database coverage 5. Manipulation 6. Time They may not all matter for your question, but think about them. Accounting for these 7. Performance reveals this
TITLE OF PRESENTATION 24 24 Think through a couple of questions
Question 1: how does Cambridge perform relative to Harvard? SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 25 25 Aspect Performance Size Discipline Publication-type Database coverage Manipulation Time Metric to take into account? Yes Yes Yes No No No No
Question 2: how does Cambridge perform relative to Harvard in Medicine? SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 26 26 Aspect Performance Size Discipline Publication-type Database coverage Manipulation Time Metric to take into account? Yes Yes Yes No No No No No Taking the different disciplinary focuses into account by using the filter gives me other options for metrics to use. But why would I want to do this?
TITLE OF PRESENTATION 27 27 Field-Weighted Citation Impact Field-Weighted Citation Impact is a very sophisticated and useful metric it normalizes for size, field and publication-type Suitable for: Fair comparison of entities regardless of size, disciplinary profile, and publication-types Avoiding the crash in recent years Immediate understanding of extent of brilliance (1 = average) Selection in case of uncertainty However. You lose the idea of the magnitude of your impact People might not like to see low numbers Complicated calculation that the users won t be able to validate themselves Always using it FWCI as default severely restricts richness of information
Guidebook has worked examples for all metrics SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 28 28
How does Cambridge perform relative to Harvard in Medicine? SciVal Metrics Guidebook Webinar TITLE OF PRESENTATION 29 29 Aspect Performance Size Discipline Publication-type Database coverage Manipulation Time Metric to take into account? Yes Yes No No No No No Taking the different disciplinary focuses into account by using the filter gives me other options for metrics to use. But why would I want to do this?
TITLE OF PRESENTATION 30 30 Questions For product information, please visit: www.elsevier.com/research-intelligence
TITLE OF PRESENTATION 31 31 Thank you! For product information, please visit: www.elsevier.com/research-intelligence