Evidence-based Metrics for Research Performance Strategies

Transcription

1

2 Evidence-based Metrics for Research Performance Strategies Jeff Horon Consultant Pre-NORDP 2014 Research Performance Strategies Workshop

3 Agenda What are metrics? +How to develop good metrics Metrics for research +How to develop good research metrics Expanding research dashboard metrics to benchmarking and collaboration Drilling beneath research dashboard metrics for advanced use

4 What are metrics?

5 What are metrics? Quantitative values Measure, distill real-world a.k.a. Key Performance Indicators (KPI s), performance measures, etc. Connotation of monitoring, control

6 Why use metrics? The most critically-constrained resource in any organization is managerial attention Eric Svaan Distillation is necessary to focus on what is important, what can and should be managed, and to understand which areas are most in need of attention

7 Types of analyses and metrics Experimental / Explorational Descriptive / Historical Current / Real-time Predictive / Forecasting Prescriptive / Imperative

8 Experimental / Explorational Calculated out of curiosity or interest in what can be learned from existing information resources (highly situation-specific)

9 Descriptive / Historical What happened? Did we do well? e.g. last rolling month of web traffic:

10 Current / Real-time State of a variable, e.g. days cash on hand How are we doing? Real-time What is happening? e.g. real-time web analytics:

11 Predictive / Forecasting What will happen? e.g. extrapolation:

12 Prescriptive / Imperative What should I do? (covered in case study)

13 Reporting vs Decision Support Reporting connotations: open ended, periodic, requirement Challenge: Only continue if there are consequences for external non-compliance Decision support connotation: addresses a specific need, matches the metric to the question at hand Ad hoc analysis explicitly supports a decision Metrics often implicitly support decisions, specifically: How are we doing with respect to?

14 Reporting Maturity Backward-looking reporting asks How did we do? and might imply corrective action in situations where it pays to correct your mistakes. Better reporting provides context about the present and assists in decision support The best reporting includes forward-looking views that enable proactive decision-making and/or prescriptive decision guidance (Decision Support! Decision Support! D-e-c-i-s-i-o-n S-u-p-p-o-r-t!)

15 Good Metrics Metrics and reporting repeatedly draw and refocus managerial attention over time: How are we doing with respect to? How much attention do I need to pay to?

16 Implications: Good Metrics Matter Conserve managerial attention Are transparent and can be drilled into Look forward; Are as prescriptive as possible

17 Bad Metrics Most metrics describe the past or present, so you may miss opportunities for course correction or you may find yourself in trouble before you detect it!

18 Implications: Bad Metrics Don t Matter Waste managerial attention Are opaque Look backward

19 How to Develop Good Metrics

20 An organization measures what it values. what it must control:. and values what it measures i.e. incentives, goal effects

21 An organization measures what it values Subject to constraints: Managerial attention Talent -Applied time -Training/education

22 So why do organizations get stuck with bad metrics? Lack of managerial attention -Leads to low investment in data resources and/or talent -Which in turn leads to using what is at hand Following -Accepting market / industry standards

23 What does it look like when you re stuck with bad metrics? A business unit wants to be forward-looking Reporting only provides a backward-looking view; after all we only have data about events that have already happened

24 How do you break out of this cycle? Breaking out requires creating and developing forward-looking views, using available data to create expectations about the future

25 Implications Forward-looking views may allow for course correction

26 Implementation How? Unfortunately, it s not easy, but let s make it tractable It will require sustained commitment across several critical steps in the development life cycle: Diagnose and fix bad metrics Create good metrics Build awareness and acceptance Standardize and refine Integrate with existing reporting

27 Diagnose and Fix Bad Metrics Test, test, test! Do the metrics reward desired outcomes? Are the metrics comprehensive? Stable? Balanced?

28 Create Good Metrics Ask: What do we want to achieve / reward? (Not: What data do we have available?) Set good goals (a topic unto itself) SMART Framework: -Specific -Measurable -Achievable -Relevant -Time-Bound

29 Build Awareness and Acceptance Sell the idea Unwind defective strategies Introduce new ideas

30 Standardize and Refine How much refinement?

31 Standardize and Refine Cost / Benefit Analysis Value of Information vs Cost of Information

32 Standardize and Refine

33 Integrate with Existing Reporting Add into existing reporting Add alongside existing reporting

34 How to be a good metrics consumer Insist on answers to the questions at hand Is this really answering what I m asking? Can I see the this or that decision?

35 How to be a good metrics producer Meet your audience where they are

36 Metrics for Research

37 Research Dashboard

38 About the State of Research Report Required some buy-in building Dense, but everything you need to answer is everything on track? in one summary page Followed visual heatmap:

39 Understanding ROI in research The Return is the benefit to society, the economy, the institution and the researcher ROI can be difficult to measure ROI has a long time horizon Research idea Team formation Applications for external sponsorship Project Publication (or other research outputs, like Patents) Citation (or other outcomes)

40 Types of metrics for research Influence Recognition Output Rank Financial sustainability Efficiency Inputs

41 Why most research metrics are terrible Influence Recognition Output Rank Financial sustainability Efficiency Inputs These are often measured because they keep the lights on and salaries paid, but these are fundamentally inputs to the research process

42 Why most research metrics are terrible Influence Recognition Output Rank Financial sustainability Efficiency Inputs Also, these are easily visible and measurable. but read backwards up the list and you ll see that is the way inputs are converted to outputs

43 Influence (i.e. Return) Did this change: -the field of study -the law -public opinions -peer opinions -the way we do things -clinical practice -technology -the author s standing in the field -the university s reputation -(propensity to get another grant)

44 Influence, Outputs, and Recognition

45 Rank U.S. News & World Report Rankings NIH Market Share

46 Financial Sustainability

47 Efficiency

48 Inputs

49 How to Develop Good Research Metrics

50 Case Study Awards Pipeline Designed to be forward-looking view of financial sustainability Original construction: All future award commitments for the next fiscal year or multiple years divided by historical commitments This can be visualized by awards stretching across a timeline:

51 Case Study Awards Pipeline. being divided up by fiscal year:

52 Case Study Awards Pipeline. stacked:

53 Case Study Awards Pipeline. and considered as historical and future commitments:

54 Case Study Awards Pipeline Upon testing, the metric proved to be idiosyncratic departments tended to meet the goal in alternating flip-flop patterns

55 Case Study Awards Pipeline Diagnosis: By stating the goal relative to the prior year, the Pipeline Ratio could reward poor performance in the prior year and punish good performance in the prior year Mechanics: When a department performs well, the goal for next year becomes more difficult. When a department performs poorly, the goal for next year becomes easier These effects combine to flip-flop likely payout structures each year (confirmed by statistical testing):

56 Case Study Awards Pipeline And there was an additional idiosyncrasy: Underlying event patterns will cause the pipeline metric to move in cyclical patterns Mechanics: Awards are often funded for multiple years, with abrupt endpoints; Renewal does not occur until an award is expiring These effects produce sawtooth patterns in the pipeline metric, with rapid increases driven by large new awards or renewals, followed by gradual decline over the life of the awards:

57 Case Study Awards Pipeline Problematically, the highest performing department by absolute measures fell below goal with respect to the ratio:

58 Case Study Awards Pipeline Refinement sold by presentation, quasi-white paper Packaged with a solution the pipeline metric was refined to: -Utilize one future year of data to mitigate the sawtooth pattern -Compare this one future year against the same observation one year earlier (to recognize single- year growth) -Balanced with a longer-term growth measure (to recognize multiple-year growth, mitigating the effect of punishing good historical performance).

59 Case Study Awards Pipeline Sustained commitment and insights from metric testing, combined with forecasting techniques, awareness-building, refinement, and standardization have led to related new metrics and reporting. Adding awards projected from proposals (submitted but without a funding decision yet):

60 Case Study Awards Pipeline. again dividing by fiscal year:

61 Case Study Awards Pipeline. and stacking:

62 Case Study Awards Pipeline. current data has been used to create expectations about the future:

63 Case Study Awards Pipeline. which can then be compared against historical growth trends and/or target growth trajectories:

64 Case Study Awards Pipeline. Made available on demand in a production reporting environment, this analysis has become predictive and prescriptive:

65 Case Study Awards Pipeline Prescription: Gap Success Rate = Proposals needed Success Rate Proposals needed

66 Best Practices Sustained commitment across several critical steps in the development life cycle: Diagnose and fix bad metrics Create good metrics Build awareness and acceptance Standardize and refine Integrate with existing reporting

67 Strategies for translating thought into action Engage in true Decision Support Put the right metrics in front of the right audiences Trouble indicators are helpful Give metrics teeth (link to financial compensation)

68 Costs Managerial attention Information -Collection -Systems Talent

69 Costs of not playing Zero sum game ( my win is your loss ) for research funding We are spending a smaller percentage of GDP on research:

70 Costs of not playing Numbers of applications remain high; success rates low NIH Research Project Grants Competing applications, awards, and success rates NIH Data Book ( Data provided by the Division of Statistical Analysis and Reporting Branch

71 Costs of not playing Federal R&D Budgets are not improving (anytime soon)

72 Implications Who? What? When? Where? Why? So what? What if? Who cares?

73 Implications Who? What? When? Where? Why? So what? What if? Who cares? Research managers and research development professionals need better metrics as soon as possible in their research reporting infrastructure to make better decisions about policy and procedure (incentives), efficiency, recruitment and retention, resource allocation, and the identification of other critical needs or they ll lose to more competitive institutions at the expense of their own constituents

74 Expanding Research Dashboard Metrics to Benchmarking and Collaboration

75 Snowball Metrics Recipe Book

76 Snowball Metrics

77 Snowball Metrics

78 SciVal Research metrics for 4,600 research institutions worldwide; Analyze by country, state, and research group or individual Overview Benchmark Collaboration At-a-glance snapshots of any selected entity or group Compare research institutions and groups Identify and analyze collaboration, recruitment targets 77

79 SciVal - Benchmarking

80 SciVal - Benchmarking University of ABC University of DEF Home State of University DEF

81 SciVal Collaboration

82 SciVal Collaboration

83 SciVal Collaboration

84 Funding benchmarking Who is being funded for what? Who is in this same space? (for Medicine: NIH mechanism benchmarking)

85 Drilling Beneath Research Dashboard Metrics for Advanced Use

86 Individual productivity and financial sustainability

87 Individual productivity and financial sustainability Gap between award end date and next proposal s begin date

88 Individual productivity and financial sustainability Financially unsustainable (lack of proposals)

89 Individual productivity in context At MD Anderson: Is an h-index of good or bad for a faculty member of the same rank and in the same department? 16

90 Network Metrics

91 Highest Degree

92 Highest In-Degree Popular

93 Highest Out-Degree Gregarious

94 Highest Betweenness Bridge Commonalities

95 Highest Closeness Who could spread a rumor?

96

97 Highest Eigenvector Centrality Importance

98 Increasing Eigenvector Centrality

99 Degree (undirected): Number of connections In- / Out-Degree (directed): Popular / Gregarious Betweenness: Bridges / Commonalities Closeness: Rumor starting point Eigenvector Centrality: Importance jhoron@umich.edu

100 Who can we build a program around?

101

102 Retention risks Not connected to intellectual community Transferrable federal funding (visible in public data sets)

103 Recruitment targets Elsewhere, not connected to intellectual community Elsewhere, transferrable federal funding (visible in public data sets)

104 Q&A Contact: elsevier.com Materials posted at:

105