Guidance for Staff on Moderation

Transcription

1 Guidance for Staff on Moderation 1. Introduction a. This guidance is to support staff implementing the Code of Practice on Assessment of Taught Programmes and Modules. b. This guidance should be read in conjunction with section 4.3 of the Code of Practice on Assessment of Taught Programmes and Modules. c. All Principal Academic Units (PAUs) should have in place staff development and guidance procedures for all marking processes. 2. What is moderation? a. Moderation refers to a range of processes conducted by an academic member of staff (i.e. an Internal Examiner) to ensure that assessment tasks and marking are accurate, appropriate to the level of the assessment and comparable with equivalent assessments. It is additional to the checking of the accuracy of marks recorded. b. It is necessary to have a process of internal moderation carried out by academic staff of the University 1, and a subsequent process of external moderation carried out by External Examiners. 3. Who can moderate? a. The Head of PAU (or nominee) appoint internal examiners on an annual basis b. The moderator(s) should have a good understanding of the general discipline, but may not necessarily be an expert in the subject of the assessment being moderated. 4. When is moderation needed? a. An assessment should be moderated if: i. It is weighted as 10% or more of a module mark; and ii. It contributes to the calculation of a final award. b. For those assessments that contribute to less than 10% of module marks, PAUs must ensure that: i. At least 60% of the module is moderated (e.g. if a module has multiple small assessments (<10% each) amounting to 70% and one assessment of 30%, a further 30% of the module needs to be moderated, even though the assessments are below 10% of the module mark). ii. All marks between 30-39% for first year undergraduate students are moderated (PAUs can define how this moderation is undertaken). 1 Or other appropriately qualified individuals, e.g. Part-Time Visiting Lecturers or Honorary Teaching Staff 1

2 5. Which pieces of work should be moderated? Guidance for Staff a. PAUs should define which components of assessment within modules should be subject to moderation, in consultation with the External Examiner(s). In some circumstances it may be appropriate to moderate marks for components of assessment which fall below the minimum threshold requirement of 10%. b. Examinations When moderating marks from examinations, Schools should determine whether moderation should be carried out either: i. At the level of individual questions within an examination paper (i.e. the mark awarded for each separate, substantive question); or ii. At the level of the paper as a whole (i.e. the overall mark for the examination). c. Where different questions within an examination paper are marked by different markers, it is necessary for moderation to take place at the level of the question. Where there is a single marker for the examination paper, it may be appropriate to moderate the marking for the paper as a whole. d. Coursework The same principles apply to moderation of coursework assessment: if the overall mark for the coursework element of a module is derived from the aggregation of marks for a number of different, distinct components which have been marked by different markers, each component mark should be moderated separately, unless each individual component of assessment does not contribute more than 10% of the mark for the coursework element and provided that at least 60% of the assessment for the module is moderated. If the components of the coursework assessment are all marked by the same marker, it may be appropriate to moderate the overall mark for the coursework element. e. Practical assessments For practical assessments such as presentations, oral examinations, musical or dramatic performances etc which individually contribute more than 10% to the overall module mark and where marking takes place at the time of the assessment, moderation should take place at the time of the assessment, by having more than one Internal Examiner present, and, where appropriate, the External Examiner(s). Where this is not feasible, there should be a formal record of how the mark was arrived at, with reference to the marking criteria, and also, wherever possible, an audio/visual recording of the assessment, which can be used for moderation purposes. 6. How to carry out sampling a. Although only a sample of work will be reviewed, it is necessary that the moderator has access to ALL the pieces of assessment from the cohort. 2 2 Within this context, a cohort is defined as a group of students who have taken the assessment in question for a particular module, thus ensuring that students who take the 2

3 b. In order to select a sample for review: Guidance for Staff i. Stage 1 1. Review the range of marks provisionally awarded for the assessment. (Other relevant statistical information may also be considered, if available, such as the mean mark, some indication of variation (e.g. standard deviation), and comparative data for previous years and for other similar types of assessment at the same level within the programme). 2. Determine the total number of pieces of work submitted for the assessment which is subject to moderation. 3. Determine the level of the assessment (e.g. Undergraduate Certificate, Intermediate, Honours, or Master s level). ii. Stage 2 1. Determine the sample for review in accordance with the following criteria: a. The sample must be representative of the full range of marks, including some fails, where they occur. b. The sample must meet the minimum sample size, as follows : Number of pieces of work in the cohort Minimum sample to be reviewed 100 or more Square root of the total number, rounded up Between pieces of work Below 10 All pieces of work 2. Note: a greater sample size than the minimum may be appropriate in the following circumstances: a. If the statistical information indicates a significant disparity between the marks awarded by different markers for a particular assessment or within a module, or where the marks awarded by a single marker appear to be unusual in any way (e.g. a particularly high or low mean mark; marks out of line with the normal distribution for the assessment / module etc.) b. Where there is a large number of first markers same assessment but are registered on different modules, and are therefore subject to different learning outcomes, are not regarded as a single homogeneous cohort. 3

4 c. Where the marker is a new or inexperienced member of staff d. If the assessment is taken by students from a range of programmes, in order to include examples from students on the full range of programmes iii. Stage 3 what to look for 1. When reviewing the sample of work, the moderator should be looking for trends or anomalies in how the marker has marked and should not make adjustments to the marks awarded for individual pieces of work. It is inequitable to change the marks for only the sample reviewed. 7. Outcomes of all methods of moderation: a. The outcomes of the review of marks awarded by the first and second marker or moderator/s, and the action which should be taken, should normally fall into one of the following categories: Outcome of moderation a) The marks of the first and second marker/moderator/s are consistently in agreement, differing by no more than the agreed difference as stipulated in the PAU moderation policy for all of the reviewed work; or by no more than the agreed difference as stipulated in the PAU moderation policy for the large majority of the reviewed work and by no more than 10% for a small number of pieces of assessment (e.g. 1-2 in a sample of 20). ) b) The marks of the first and second marker/moderator/s differ by 10% or more for a larger number of the pieces of assessment which have been reviewed (e.g. 5 or more in a sample of 20). Action to be taken Where sampling has been carried out: No further action is required and the marks of the first marker are approved as the confirmed marks for the sample and the rest of the cohort Where double-marking has been carried out: The marker and second marker / moderator/s should discuss the reasons for the marks they have awarded, and agree that the confirmed marks will be: (a) the full set of marks awarded by the first marker; (b) the full set of marks awarded by the second marker; or (c) an agreed set of alternative marks (e.g. the average or a weighted average of the two marks). Where sampling has been carried out: The marker and moderator/s should discuss the reasons for the marks they have awarded, with reference to the marking criteria. This may lead to one of the following outcomes: (a) If the marks of the first marker are agreed to be appropriate, they may be adopted as the confirmed marks for the whole cohort; (b) If the differences between the 4

5 marks of the first marker and moderator are consistently in the same direction and of a similar amount, it may be decided to adjust the marks of the whole cohort by an agreed proportion or number of marks; (c) If the first marker and moderator are unable to reach an agreement on the marks to be awarded, or if the scale and direction of differences of marks awarded by first marker and moderator vary across the sample the full set of work should be marked by the moderator, and the marks then agreed via the process for agreeing the outcomes of double-marking (below). Where double-marking has been carried out: The first marker and second marker should discuss the reasons for the marks they have awarded, with reference to the marking criteria, and agree one of the following outcomes: (a) that the full set of marks awarded by the first marker be adopted as the confirmed marks; or (b) the full set of marks awarded by the second marker be adopted as the confirmed marks; (c) that the average or a weighted average of the marks awarded by the first and second marker be adopted as the confirmed marks; or (d) the marks of the whole cohort may be adjusted by an agreed proportion or number of marks; or (e) a mark is agreed for each piece of assessment. Exceptionally, if the first and second marker are unable to agree on a course of action, then a third (internal) marker or moderator should be consulted. Only in very rare circumstances should an External Examiner be invited to consider the issue. 5