Overview of Panel N 1. The work of Panel N was facilitated by the excellence of the support from Helen Dowson and James Birkett, who were under considerable pressure at times and not least at the end of the process. They dealt with the large volume of work with great efficiency and good humour, and everyone involved wishes to express their gratitude to them. The chairs of the sub-panels also wish to express their appreciation of the input of Sarah Tyacke as the user representative who was able to provide an impartial and informed assessment on many issues. The international adviser, Professor Frank Turner, was able to assess a sample of work in detail, and confirmed the standards being applied. As Chairman of the Panel, I would like to thank the sub-panel chairs for their constructive attitude which meant that we were able to discuss issues in an open manner which allowed us to move towards consensus. 2. At the start of the process, Panel N expressed concern about the document delivery system which had been problematic in 2001. Although there were delays at the start of the process, action was taken and by the end of the process we found that the system worked well and did not cause any difficulties. We would like to thank Ed Hughes and his team for responding to the concerns and indeed for being generally available for discussion of issues as they arose. He and his team responded promptly and constructively, and helped to minimise any potential problems. 3. Nevertheless, there were some difficulties which we would like to note. More time, and perhaps an extra sub-panel meeting, would have been helpful at the end of the process to handle the large amount of data and the writing of reasoned cases. We are confident that the work was done to a very high standard, but only by dint of the Herculean efforts of the panel secretariat and the fact that the chair of the History sub-panel one of the very largest in the whole exercise was retired and able to devote all of her time to the RAE. 4. There were difficulties with the system of cross-referral. The timetable was not adhered to by other sub-panels, so that we had to chase other sub-panels for grades during the final meetings. In some cases, items were held by the cross-referral reader so that we were unable to secure the item to read. As far as incoming cross-referrals were concerned, and especially with material sent to Specialist Advisers, the absence of full bibliographic details for crossreferred items made it difficult to access items in other ways. We would recommend that in future exercises more attention is given to cross-referrals, in providing full bibliographic details at the outset and ensuring that readers adhere to the time-table. The use of Specialist Advisers was vital to the success of the exercise, but in any future exercise clearer and more formal lines of direct communication should be established between them and subpanel chairs. 1
5. In 2008, unlike 2001, members of sub-panels were not allowed to claim for expenses to visit major and especially copyright libraries where much of the material would have been readily available. We regret this decision, which would have made the task of readers easier and reduced the burden on the central repository. 6. We noticed a step change in research income and consequently in the scale and scope of some of the research that was assessed. The achievement of full research council status by the AHRC has had a major impact on research in the areas covered by Panel N. All of the sub-panels noted the importance of the AHRC s research leave scheme, which they commend for allowing scholars to complete projects, and to support the individual scholar who needs time. We hope that it will be possible to continue this scheme in the future. 7. Although early career researchers were allowed to return fewer than four outputs on a sliding scale according to their date of appointment, we did feel that a more generous relief from overly swift publication would have been desirable. 8. The sub-panels were confident in their assessment of outputs, which relied on their accumulated experience as academics in reading and grading theses, advising publishers and journals on manuscripts, commenting on research grants, and reviewing books. It is not accurate to characterise peer review as in some sense dangerously subjective. In fact, the sub-panels were more aware of the difficulties of interpretation of supposedly more objective data provided under research environment and esteem. Data needed to be interpreted with care, to ensure that due attention was paid, for example, to the uses of research income and the results of its acquisition. 9. We accept that it is necessary to assess environment in order to ensure that units give proper attention to the needs of the scholarly community and the future of the discipline, apart from producing outputs. We felt that the current division between 80 per cent of the profile for outputs and 20 per cent for environment and esteem was right. However, we were even more strongly of the view at the end of the process than at the beginning that esteem should not be separately measured, for it was largely co-terminous with elements of environment and contributions to the scholarly community. 10. There will inevitably be attempts to draw comparisons between 2008 and 2001. We would caution that there are important differences between the two exercises. In 2001, departments were awarded a single grade to cover all areas of their activities which was not the case in 2008. The assessment of outputs used different criteria for the various levels of international excellence; and the assessment of environment and esteem were different. Direct comparison 2
between the results of the two exercises should therefore be treated with the greatest caution. 11. Our sense at the end of the process was that all four disciplines are performing very well and have in many cases gone through a process of renewal. It will be vital that the new REF system continues to encourage the best work, and that the measurements used are not perverse in outcomes, for example by producing work for the sake of citations; or giving more weight to one form and place of publication (for example articles in particular journals) over another (such as books). 3
Subject overview report for Sub-panel 61, Theology, Divinity and Religious Studies (2008.) The process In total sub-panel 61 received submissions from 38 institutions covering 498 Category A staff (470.69 FTE) and 43 Category C staff. This compares to 43 submissions and 438.83 Category A and A* staff in 2001. There were fewer institutions than in 2001. A number of self standing units have disappeared or been amalgamated into new units. Some new former church-related colleges submitted this time, representing in most cases sensible consolidations - the latter now made up a third of the total. These changes could have an effect on outputs. Some units that submitted in 2001 had submitted material to cognate subject panels this time. Outputs were allocated according to the expertise of sub-panel members and taking into account declarations of interest. Virtually all submitted outputs for each institution were read (almost 2000 in all.) The sub-panel also used cross-referral and a number of specialist advisors where outputs were outwith its expertise or language competence or where conflicts of interest existed. Outputs to be weighted and those to be unclassified were confirmed by an arrangement of weighting partners. Additional moderation was undertaken to check the consistency of grading across assessors. Environment and esteem sections were read by all sub-panel members and considered at three meetings of the sub-panel, with two sub-panel members leading the discussion of each institution. The standard analyses were available to support decision-making. The sub-panel process began with detailed examination of each institution, based on the material in the assessment documents. All panellists read each institution s complete submission. There was discussion of the allocation of percentage grades. Following the advice of Main Panel N, it was agreed to allow 15% for research environment, 5% for esteem and 80% for the research outputs. Lead panellists introduced the examination and discussion of each institutional submission, considering first the research environment and the assessment of esteem. Careful note was taken of the views of the sub-panel members. Submitted works were distributed among the panellists on the basis of their academic specialities. Practically all submitted works were read by panel members, by outside assessors from other panels, or by specialist advisors. The weighting of outputs of significant scale and scope and the awarding of unclassified grades was agreed by at least two sub-panel members. Provisional grades for environment and esteem were made separately from outputs. Grades for outputs were then given to the leads. Final grades for environment and esteem, divided by percentage in the given four categories of starring for each of the sections were agreed at two panel meetings. Final grades for outputs were added, moderated and agreed. Issues of moderation were considered at Main panel N meetings and applied to sub-panel 61 results. There will inevitably be attempts to draw comparisons with the results of the 2001 exercise, but such efforts will need to recognise the significant differences between 4
the two exercises. In 2001 departments received a single grade for all aspects of their activity from which it is not straightforward to calculate a precise figure for international or national elements, whereas in 2008 more detailed profiles have been established. The criteria for defining the different levels of international in 2008 were also somewhat different, inevitably, from those used in 2001 for the single, broader, international element, and the balance between Outputs and other contributing factors was significantly different. The discipline The discipline was found to be flourishing; 80% of the submissions were of quality internationally recognised. (The majority of institutions were returning at least some 4* work and nearly all were returning a significant amount of 3* work.) There is a strong public interest in the subject. All of the recognised areas of Theology, Divinity and Religious Studies (TDRS) were represented in the submissions, though not all in each submission. The Arts and Humanities Research Council (AHRC) funded projects are a classic example of the impact of the Humanities on culture and society. Some of the fastest growing parts of the UK economy sit within the AHRC's subject domains. Arts and Humanities research can create social and economic benefits directly and indirectly through improvements in social and intellectual capital, social networking, community identity, learning and skills and quality of life. Knowledge of religion is very important at the present time and record numbers are taking RS at GCSE and A level. Strengths/Weaknesses The traditional theological sub-disciplines remained strong. The religious studies component was less strong than in 2001. Units were most successful when playing to their strengths (without avoiding new developments) focused units were more successful. There was a good balance between established academics and early career researchers. The sub-panel had the impression of several areas of growth, e.g. historical theology, alternative spirituality and Pentecostal studies. There was decline, within the material submitted to this panel, in areas of gender studies and in the emancipatory theologies, in analytic philosophy of religion, and in work on religions whose sources are not in English (it was noted that it is not easily possible to develop competence in original languages within a 3 year AHRC studentship.) Some submissions laid stress on religion in the contemporary world. The intention was sound, though some of the programmes appeared to lack focus. There were a good number of substantial monographs, notably in medieval theology and in modern church history, and these have been rewarded. The panel noted an increase in study of Islam post-9/11. However, a number of units were only now planning to appoint. There was uncertainty about the quality of some outputs: it was recognised that excellent work in this field will have been submitted to the Middle Eastern Studies panel. It was possible that many of those submitting in the area of third world religion have been appointed since 2001. This is a new area of research, and to be welcomed, 5
always bearing in mind that all religions have a history, and this historical background is significant for understanding contemporary manifestations. As before, there were instances in which the RAE may have encouraged premature publication. As in 2001, the question of textbooks arose: their research content varied considerably. Close attention to the guidelines on submissions is crucial. There was concern about overlapping outputs, not all of which were effective submissions. The sub-panel considered the state of publishing in this area. There was early career pressure to publish. Some submissions did not follow the guidance on early career researchers too many submitted more than necessary. Pressure to publish was perhaps generating too many edited volumes. The issue of weighting outputs of significant scale and scope was evaluated. Rules for weighting should be established at the start of the period of assessment, so that units are clearly aware of the position. It is vital that works of a substantial nature are encouraged. The sub-panel thinks it highly desirable to retain the possibility of weighting in a future RAE. Web publications: there were few, and there were some reservations about the most effective use of this route. Subject overview. The global picture is of a discipline which is again strongly flourishing. A very high proportion of outputs were rated to be of international or world-leading significance. There were also a substantial number of units with outstanding or excellent environment and esteem indicators. We noted the recruitment of some excellent new researchers. Some panel members reiterated the concern expressed in 2001 that recruitment, especially at the end of the period, appeared to favour established scholars rather than early career researchers. The submissions suggested a need for more post-doctoral places in the units. Postgraduate student numbers had increased markedly. There was some concern about career prospects for such a large number of PhDs in TDRS. Much reliance was placed on recruiting overseas research students - potentially a volatile market and a problem in the future. The figures reflect an increase in post graduate research students because of metrics, but there are not enough UK postgraduate research students. There was a welcome increase in the achievement of funding resources. It was noted that the AHRC has become a fully-fledged research council since 2001. Nevertheless the AHRC Research leave scheme has enabled scholars to produce important individual studies, and has provided young scholars with valuable opportunities for temporary employment. Liberating academic time is a most useful way to encourage research excellence. It would be helpful to have clarification of what was included as a studentship: this is difficult to measure. 6
Institutional strategies for fostering research excellence varied greatly - there was no single optimum approach. Much high value output was contributed by individuals working on their own. Finally, observation of the spreadsheets shows a marked decrease in environment and esteem ratings for the smaller units. This suggests that in the overall ratings the metrics work strongly to the advantage of the larger units. Analysis of the role of metrics in this period should provide important indicators of their scope and limitations for 2014. George M Newlands 7