Quality Assurance during Individual Assessment QUALITY ASSURANCE. Key Action 2: Strategic Partnerships

Transcription

1 In this briefing sheet, detail is provided in relation to the various Quality Assurance measures that can be implemented by NA staff in relation to the assessment of KA2 strategic partnership applications. Quality Assurance during Individual Assessment As confirmed by the Guide for NAs, KA2 applications are normally required to be assessed by at least two experts, with at least one external expert involved in the assessment process. An exception to this is Schools Exchange for which the number of experts involved in quality assessment depends on the size of the grant being requested - two experts are required only where the grant request is greater than 60,000 otherwise a single expert assessment is sufficient - and for which there is no requirement for external expert involvement. In all cases, experts (internal/external) must produce a unique set of scores and comments, in line with the award criteria for the selected action, with the total score automatically generated within the Online Expert Evaluation Tool (OEET). As shown in Figure 1, quality assurance activity begins by ensuring that all participating experts (internal/external) are provided with an expert briefing during which the core goals of the assessment exercise are confirmed alongside national, European and programme level priorities, assessment criteria and thresholds, a schedule for assessment, and all or any tools or materials that need to be used or consulted during assessment. In this respect, experts should be presented with the 2018 Erasmus+ Guide for Experts on Quality Assessment as well as relevant information on the fields and actions that they will assess (Erasmus+ Programme Guide and/or targeted s). Where not already addressed, it is important to confirm during the expert briefing that no Conflict of Interest exists among participating experts. «a conflict of interests exists where the impartial and objective exercise of the functions of a financial actor or other person is compromised for reasons involving family, emotional life, political or national affinity, economic interest or any other shared interest with a recipient» [Financial Regulation Art. 57(2)] In practical terms, this goes one step beyond the signing of a Declaration on the Prevention of Conflicts of Interest and Disclosure of Information (as per the model provided in Annex I of the 2018 Erasmus+ Guide for Experts on Quality Assessment) and requires that all participating experts work methodically through the list of attributed applications, checking the names of individual applicants, coordinators and partner organisations, ensuring no conflict of interest. Page 1

2 In this briefing sheet, detail is provided in relation to the various Quality Assurance measures that can be implemented by NA staff in relation to the assessment of KA2 strategic partnership applications. Figure 1: Overview of Key Assessment and Quality Assurance Phases for KA2 : Model 1a applies to School Exchange requesting 60,000 whereas Model 1b applies to all other types of KA2 Partnership. Expert Assessment 1 Expert Assessment 1 Expert Assessment 2 Difference of 30 or more points and thresholds passed by both experts? Briefing of Experts, Conflict of Interest Check and Random Quality Checks Assessment Scores and Comments Consolidation (where 3 experts this involves only 2 experts with closest scores) Expert Assessment 3 The Five Cs Ranked List of Assessments Consolidated Assessment Scores and Comments The Six Cs Ranked List of Assessments Completeness and Correctness Model 1a Model 1b Quality Assurance Page 2

3 In this briefing sheet, detail is provided in relation to the various Quality Assurance measures that can be implemented by NA staff in relation to the assessment of KA2 strategic partnership applications. At the point of submitting their assessment, within the OEET, experts will also be required to officially declare that they have no known conflict of interest with the application being assessed. NAs might additionally opt to include the previouslyreferenced Declaration on the Prevention of Conflicts of Interest and Disclosure of Information as a part of the formal contracting process for experts, as this also includes the required confidentiality commitment from experts. Where a potential or actual conflict of interest is identified, NA staff need to decide on one of the following courses of action: exclude the expert from the whole assessment process: usually only required where there is a direct conflict of interest such as the expert s own organisation participating in one or more applications; replace the application that is causing the conflict with another application: usually the case where an expert is familiar with, or has had previous relations with, an individual or organisation but has no direct link with any project application; continue as planned: this might be the case where a perceived connection is felt to be sufficiently in the past, or sufficiently at a distance, with little potential for influence over the assessment process. Once the briefing process is completed, and potential conflicts have been checked, experts can begin to assess the applications that they have been attributed producing a unique set of assessment scores and comments for each assessed application. At this stage, the NA is responsible for ensuring that assessments are undertaken to the required or expected standard and that the resulting scores and comments meet NA and higher-level expectations. Depending on the total number of assessments being undertaken, and the time available to NA staff, either a FULL or a PARTIAL (e.g. random sample of files for each expert) check should be undertaken, ensuring that each assessment is Coherent, Comprehensive, Consistent, Courteous and Concise (the Five Cs): Coherent comments should be easy to understand (even for a reader that has not read the application) and should provide feedback that the applicant will understand and can learn from Comprehensive comments should be provided for each of the award criteria and should incorporate most, if not all, of the composite elements Consistent comments should be consistent with the scores that have been awarded for each criterion and should be aligned with the predefined scoring bands for each action Courteous comments should always be polite and respectful, and should avoid first person reference (e.g. I think that ) Concise whilst exceptions exist, comments should be of a standard size, as determined by NA staff (e.g. 1-2 paragraphs per criterion); experts should avoid repeating that which is written in the application Page 3

4 NA staff are responsible for ensuring that all assessments meet the above standard, allowing assessment data (scores and comments) to be used to inform the national selection committee and to provide feedback to individual applicants. In cases where more than one expert is involved in quality assessment (Figure 1: Model 1b), it is important to remember that INDIVIDUAL ASSESSMENT is only the first stage in the process. In such cases, it is not necessary to check every assessment to ensure that comments are courteous or concise - especially as it is only the CONSOLIDATED scores and comments that will be accessed by wider audiences. It can, however, be useful to undertake a random check of 2-3 files for each expert ensuring that they have understood the need to be coherent, comprehensive and consistent in their assessment. In all cases, NA staff can request that an individual expert revisits or revises an assessment where the Five Cs are not felt to be satisfactorily achieved. However, in no situation should NA staff propose changes to the scores attributed by experts, asking instead that experts ensure consistency between scores and comments. Quality Assurance during Consolidation Once both experts have completed their individual assessment, the NA has additional responsibility for ensuring the complementarity of scores and comments and for potentially launching a third assessment where experts have a significant difference of opinion (i.e. where at least one expert has passed all thresholds and there is a scoring difference of 30 points or more). In all cases, and for all applications and assessments, NA staff need to consider a series of questions relating to differences in scores, the achievement of assessment thresholds and the potential for consolidation, with a view to deciding on the correct course of action (i.e. launch third assessment, launch consolidation): as can be seen in Figure 2 overleaf. Where a third assessment is launched, NA staff have the same responsibility for quality assurance as with earlier assessments (expert assessment 1, expert assessment 2) with the same individual assessment process followed, resulting in a unique set of assessment scores and comments. Upon completion of the third assessment, NA staff need to select the two experts with the closest scores, with only these two experts invited to work together to produce a final set of scores and comments (consolidation process). At whichever point the consolidation process is launched, it will only ever involve two experts coming together to produce a single set of scores and comments (i.e. in no case will the comments and scores of three experts be used). In all cases, the NA is responsible for nominating a lead expert that will review the two individual assessments, discuss with their fellow expert and produce a single consolidated assessment (scores and comments) that both experts have agreed on. Page 4

5 Figure 2: Overview of Decision-making Process on the Need for a Third Assessment In cases where more than one expert is involved in quality assessment (Figure 1: Model 1b), it is the CONSOLIDATED results that will be used to inform the national Selection Committee and to provide feedback to individual applicants. In all such cases, there is a need for NA staff to quality-assure the results of this process with, wherever possible, a FULL check undertaken, ensuring that each assessment is Coherent, Comprehensive, Consistent, Courteous, Consolidated and Concise (the Six Cs): Individual Assessment and Consolidation Processes Coherent comments should be easy to understand (even for a reader that has not read the application) and should provide feedback that the applicant will understand and can learn from Comprehensive comments should be provided for each of the award criteria and should incorporate most, if not all, of the composite elements Consistent comments should be consistent with the scores that have been awarded for each criterion and should be aligned with the predefined scoring bands for each action Consolidation Process Only Courteous comments should always be polite and respectful, and should avoid first person reference (e.g. I think that ) Consolidated comments should read as single texts and should be harmonised (not contradictory) Concise whilst exceptions exist, comments should be of a standard size, as determined by NA staff (e.g. 1-2 paragraphs per criterion); experts should avoid repeating that which is written in the application Page 5

6 In all cases, NA staff can request that an individual expert revisits or revises an assessment where the Six Cs are not felt to be satisfactorily achieved. However, in no situation should NA staff propose changes to the scores attributed by experts, asking instead that experts ensure consistency between scores and comments. Final Data Check Whilst experts in all programme countries will continue to use a single online assessment tool (OEET), this use of technology is no guarantee of accurate final data. Previously-unidentified software glitches, or simple human error during data entry, might easily result in missing applications, missing comments and/or zero-entry scores. With this in mind, a final series of checks can be considered by NA staff with a view to ensuring the completeness and correctness of assessment data: Completeness confirming that the total number of individual assessments matches the number of applications attributed to experts, with any anomalies able to be easily explained (e.g. applications deemed ineligible during assessment) also, confirming that any zero entries or scores are meant to be (e.g. relaying a total lack of data for one or more award criteria) Correctness ensuring that scores and comments are consistent with those contained in the printed and signed assessments, through undertaking a quick random sample of assessments (i.e. visiting a small number of applications in the OEET and ensuring that the required data is present in all fields of the assessment) Page 6