Introducing the. Performance Based Research Fund (PBRF)

Transcription

1 Introducing the Performance Based Research Fund (PBRF) and Completing the Evidence Portfolio 1 May 2015 Doug MacLeod, AUT PBRF Manager

2 PBRF first introduced in Quick Review of the PBRF Intent of the PBRF is to increase; the average quality of research, public information about research, ensure research continues to support degree and postgrad teaching amongst other things. PBRF is based on the UK RAE, now the REF (RAE first in 1986). Other countries have related performance systems: e.g. Australia (ERA), UK (REF), Sth Africa (NRF), Hong Kong, Israel, Netherlands Replaced previous top ups scheme based on EFTS. Is not new money, we must now compete for funds removed from the EFTS funding. Institutional and subject area performance is made public (rankings etc). Overall performance (income) is based on three assessment components.

3 Importance of the PBRF 1. Funding Total PBRF fund in 2014 = $275 million AUT s PBRF income 2014 = 12.9 million Total PBRF fund budgeted for 2016 = 300 million 2. Rankings See the Report resulting in various rankings and related marketing 3. Other Can be used as a proxy for other things e.g. Accreditations!! Also

4 PBRF Components The PBRF system is described as being a mixed model with three components: External Research Income (15% of PBRF fund) Frequency: Annual Calculated: rolling 3 year average Research Degree Completions (25% of fund) Frequency: Annual Calculated: rolling 3 year average - Items Masters by Thesis/Project, Doctorates (>0.75 efts or 90+ credits with a wholly research component) Quality Evaluation of staff (60% of fund) Frequency: 2003, 2006 (Partial), 2012, 2018 Calculated: on the basis of the last assessment

5 The Money - PBRF Funding for Faculties 2014, total AUT PBRF income of 12.9 million (from total of 275m) Includes revenue from all 3 PBRF measures: 1. Staff Quality Assessment (60%* of total pool) 2. Research Degree Completions (25%) 3. External Research Income (15%*) *55% from 2016 *20% from 2016 All PBRF income goes to Faculties, either: Directly (9.3m in 2014) on the basis of what was actually earned: or Indirectly (3.6m): 2.0 million, VC Scholarships 1.4 million, Strategic Research Fund 0.2 million, near miss & other strategic support

6 Staff Quality Assessment The staff quality evaluation is designed to assess the quality of an individual s research contribution through the external peer review of their research as presented in their evidence portfolio (EP) The result of the assessment is a quality category of either: A, B, C, C(NE New Emerging), R, or R(NE). The quality categories have funding weightings attached to reflect their PBRF funding value: A=5, B=3, C(NE)=2, C = 1 and R or R(NE) = 0 Funding in 2008: A = $33,500 - $83,000 B = $20,067- $50,155 C or C(NE) = $6,600-$16,700 depending on subject weighting (e.g. Education 1 vs Public Health 2.5) and FTE

7 Staff quality assessment - who to assess? 1. You teach on courses that lead to a degree or related qualification OR 2. Your employment functions include research OR 3. Both 1 and 2 AND 4. You have substantial involvement in EITHER research OR teaching AND 5. You're employed on paid contract/s of 0.2 FTE or more AND 6. Your employment contract/s are for 12 months or more AND 7. You will be employed (by a participating TEO) on the 14th of June 2018 (census date) and meet the above criteria

8 The New and Emerging Researcher C(NE) or R(NE) New and Emerging researcher Eligibility: - First appointment to a PBRF eligible or equivalent position in NZ or overseas was on or after January 2012 Criteria for receiving a C(NE): 1. Successful completion of a Doctoral degree or equivalent* within the period (item is listed as one of outputs) AND 2. Other research outputs of an adequate quality and quantity, a minimum of 2 quality assured outputs is normally expected OR 3. Research Outputs equivalent to both 1 and 2 *might be a Masters for Public Health

9 The Evidence Portfolio What is an evidence portfolio or EP? A highly structured form, developed by the TEC, for capturing and reporting on an individual s contribution to research over a given period. It is in effect a highly structured, time and volume limited, academic CV. What is the period of coverage for the 2018 assessment? Research outputs listed in the EP must be produced (published, presented, exhibited etc.) within the assessment period 1st January 2012 to 31st December In general this also applies to other items. It does not matter where you were when you produced the outputs, even if you were at another NZ University or overseas, the EP covers what you did, not what you did while at AUT.

10 The Evidence Portfolio (EP) cont d 1. Research Outputs (70% of score) 4 best outputs a) b) and up to 12 other outputs 2. Research Contributions (30% of score) Up to 15 entries. plus The panel and subject area for assessment. and A Research platform outline to explain context of your research (400 words) optional A section for special circumstances. This is for explaining how some issue impacted on the quantity, NOT quality, of research outputs or other items.

11 The Panels You need to identify the Panel and Subject area that you believe is most appropriate for assessing your EP (evidence portfolio). REMEMBER: The Panel is the audience for your portfolio, so keep your audience in mind when writing your portfolio There are 12 panels with specific guidelines for each panel (42 subject areas): Biological Sciences (e.g. Agriculture and other applied biological sciences) Business and Economics (e.g. Marketing and Tourism) Creative and Performing Arts (e.g. Design) Education Engineering, Technology and Architecture (e.g. Engineering and technology) Health (e.g Nursing) Humanities and Law (English language and literature, Foreign languages and linguistics) Mäori Knowledge and Development Mathematical and Information Sciences and Technology (Statistics) Medicine and Public Health (e.g. Clinical medicine, Public health, ) Physical Sciences (e.g. Earth sciences) Social Sciences and Other Cultural/Social Studies (e.g. Political science, public policy, Psychology, Sociology, social policy, social work, criminology and gender studies)

12 The Evidence Portfolio cont d Research Outputs Section Two parts First part is for up to four Nominated Research Outputs (NROs) your best four outputs over the assessment period THIS IS THE MOST IMPORTANT PART OF THE EP It is the section of the EP the assessment panels will review most thoroughly and have the most impact on your grade. Key Points: All of your NROs will be audited by the TEC (checking for accuracy of category, publication year, author numbers, etc.) Some or all of your NROs may be sought by the assessment panel for further investigation and quality review. So these need to be available, ideally electronically (upload into Research Elements).

13 The Evidence Portfolio cont d For each of your four NROs you need to indicate: 1. Bibliographic details including the type of output 2. If the item has been quality-assured (refereed) A quality-assured research output is defined as any research output that, prior to its publication has successfully completed a formal quality-assurance process. Successful completion of a formal quality-assurance process means the output must have been subject to formal, independent scrutiny by those with the necessary expertise and/or skills to assess its quality. 3. The nature and significance of your contribution to the output if there is more than one author/contributor. 4. Any comments relevant to the output e.g. why the output has been selected, any impact seen as a result of the output e.g. shift in paradigms, high citation rate, low acceptance rate, impact of the item, removal of any potential ambiguity such as the nature of the quality assured process if not typical, or information that might give the assessors a more complete picture of the item.

14 The Evidence Portfolio quick revision 4 x Nominated Research Outputs or your Best four outputs For each you need to explain your contribution, and why it is one of your best. Also need to provide a copy of each item (electronic) 12x Up to 12 other research outputs These are just cited, no other info required. Is your research platform. Research Outputs Section: 70% of the total score 15 x Up to 15 Research Contributions. These are text boxes, (approx 200 words**) so may contain numerous items in one entry. 18 categories including: Awards, appointments, research funding, refereeing, editorial board work, student supervision, keynotes, etc etc. Research Contributions comprise 30% of the total score - Publication/activity period: Jan 2012 to Dec 2017 (6 years) - Research can be done at AUT or elsewhere, or before you joined AUT if in the period.

15 Research Contributions The new Research Contributions - essentially the old Peer Esteem and Contribution to the Research Environment merged into 10 categories: 1. Contribution to research discipline and environment (e.g. ethics committee) 2. Facilitation, networking and collaboration (e.g. organise conference) 3. Invitations to present research or similar (e.g. Keynote at conference) 4. Other evidence of research contribution 5. Recognition of research outputs (e.g. citations, good reviews) 6. Research funding and support 7. Research prizes, fellowships, awards and appointments 8. Researcher development (e.g. mentoring) 9. Reviewing, refereeing, judging, evaluating and examining 10. Student factors (e.g. supervision) Plus 2 new categories: 11. Outreach and engagement (wider community) 12. Uptake and impact (beyond academia)

16 The Assessment Process in brief 1. EPs that are likely to receive a quality category of C or C (NE) or greater are submitted to the TEC 2. The TEC receives and distributes EPs to the relevant panels 3. The panel chair assigns each EP to a pair of panel members, one of the pair being the primary reviewer 4. The panel members assess the EP, scoring the two components of the EP from 0-7 (scoring is based upon a descriptive ranked list). They may read the full-text NRO at this point. The panel members first score EPs individually, but then compare scores and discuss differences, trying to seek consensus on the final scores 5. A provisional quality category is given on the basis of the total score 6. The full panel meets, reviews the scores, and awards a quality category based on a final holistic assessment 7. Scoring is moderated by a moderation panel (so all panels score similarly) 8. Grades are returned the process is over until next time!

17 The Assessment Process - Long 1. Score each of the two component from 0 to 7 2. Apply the component weighting to the scores to get the overall numerical score e.g. Research Outputs (70%) 3 x 70 = 210 Research Contributions (30%) 2 x 30 = 60 Total Score: Match score with numerical band for quality category: = A = B = C or C(NE) e.g. is C <200 = R or R (NE) 4. Apply holistic quality category based on all information within the EP e.g. any special circumstances

18 Specifics: Selecting your Best Four outputs Your top four (NROs) dominates scoring. Select the 4 best outputs based on: The quality of the research. Which is? Originality, novel discovery, publication outlet, impact factor, impact, cites, etc. Whatever story you re able to tell about the research. Your ownership of the intellectual elements of the research (authorship order OR robust explanation of why you are placed lower down in the list of authors). Output type in most cases journal articles are preferred over every other type (always exceptions) see Panel Guidelines. Do NOT: Attempt show the spread of your research select the best four even if similar. Select your personal favourite items (PhD alert!). Do: Fully describe the key novel/discovery aspects of the research. Be very careful when considering/selecting: Reviews (how much research?) and Books (textbooks?).

19 - End - Thanks for your time. Good luck with your Portfolio!