Key questions about assessment in the QCF. June 2010 QCDA/10/5202

Transcription

1 Key questions about assessment in the QCF June 2010 QCDA/10/5202

2 Contents Introduction... 4 Questions about assessment development Can we specify the assessment methods in the units we develop? Is there a requirement for external assessment? Can skills and knowledge be assessed in separate units? Can skills-based learning outcomes and knowledge-based learning outcomes be in separate units? Can we grade our qualifications?... 6 Questions about assessment delivery Can we use multiple-choice tests to assess units? Can we assess units through written examinations? Can we assess units through online tests? Can we set a percentage pass mark for a unit? If so, what should it be? We plan to use a portfolio of evidence for assessment. Is this okay? We plan to assess all units through work-based observation. Is this okay? Can we assess a sample of learning outcomes in a unit? Do all assessment criteria have to be met for credit to be awarded? How does the professional judgement of the assessor relate to a set of assessment criteria? What if an assessor thinks that learners are required to produce unnecessary evidence to demonstrate that a learning outcome has been achieved? Can learners be compensated for poor performance in one unit through excellent performance in another unit? Can learners be compensated for poor performance in one learning outcome through excellent performance in other learning outcomes in the same unit? Do we have to assess each unit separately? Do we have to assess each unit in the same way in each centre? QCDA

3 20. Can a centre use different methods of assessment for the same unit offered to different groups of learners? Can we ignore the assessment guidance field in the unit template? We plan to base our approach to assessment on the NVQ Code of Practice. Is this okay? QCDA

4 Introduction This guidance covers key questions that awarding organisations might have about assessment in the Qualifications and Credit Framework (QCF). It is organised into two main sections questions about assessment development and questions about assessment delivery. Note that in this document an assessor may be a marker or moderator of an examination script. In this document, the Regulatory Arrangements for the Qualifications and Credit Framework (Ofqual et al 2008) are referred to as the Regulatory Arrangements. If you have any further questions after reading this guidance, please info@qcda.gov.uk. Questions about assessment development 1. Can we specify the assessment methods in the units we develop? Yes. The additional information fields in the unit template allow assessment requirements to be identified, and these could include the use of a specific assessment method, or the application of a particular assessment instrument. The method must enable a valid assessment of the unit against the specified assessment criteria, and the assessment criteria themselves should not include references to a method of assessment. Evidence from the sampling of good quality units in the QCF databank shows that wellwritten assessment criteria will actually point towards valid methods of assessment. For example, where assessment criteria demand that evidence is produced through performance of a particular task, assessment through a multiple-choice test will not be valid. Unit-submitting organisations that make use of the assessment requirements field should ensure that it relates only to the delivery of assessment and does not set any requirements that have an impact on the assessment standard, which must be defined solely by the learning outcomes and assessment criteria of the unit. 2. Is there a requirement for external assessment? No. Unlike the National Qualifications Framework (NQF), there is no requirement that a qualification, or a designated part of a qualification, should be externally assessed, meaning where assessment tasks are set, and candidates work assessed, by the awarding body (2004 Statutory Regulations). Some awarding organisations will choose to QCDA

5 use external assessment in the QCF and this is perfectly acceptable. Despite the absence of an external assessment requirement in the Regulatory Arrangements, assessment developers should note the regulatory requirement that assessment instruments and tasks are produced to the required quality standards' (5.6a). So, where responsibility for developing an assessment instrument or task is undertaken by a centre, the awarding organisation itself will still be held responsible for the quality standard of such instruments or tasks. 3. Can skills and knowledge be assessed in separate units? There is nothing in the Regulatory Arrangements that prevents the separation of skills assessment and knowledge assessment through separate units. However, sampling of units from the QCF databank suggests that this approach increases the risk of developing units that do not meet QCF regulatory requirements. One apparent reason for separating the assessment of skills and knowledge is so that the performance criteria and underpinning knowledge of a single unit in the NQF can be reproduced as closely as possible in two QCF units. It should be noted that this separation is not a requirement of the QCF and can lead to distortion in the conduct of assessment. For example, where skills-based and knowledge-based units are separated this may lead to unnecessary duplication of assessment tasks, or to an increased assessment burden for both learners and assessors. The original intention of the NQF unit approach that there should be a relationship between skills and knowledge in the assessment process may be lost where separate units are developed. 4. Can skills-based learning outcomes and knowledge-based learning outcomes be in separate units? The development of separate learning outcomes for skills and knowledge within a unit is consistent with the design requirements for units in the QCF. However, this approach may reproduce some of the problems of duplication and/or over-assessment in separating skills-based and knowledge-based units (see question 3 above). Much will depend on the advice given by awarding organisations for the conduct of assessment in such units. For example, it may be possible for an assessor to infer evidence of knowledge through observation and questioning about the performance of a particular skill. However, if skills and knowledge are assessed through separate assessment activities, such inferences may not be made, and the burden of assessment may grow unnecessarily. The mechanistic separation of learning outcomes into know how to... and be able to... does not tend to produce units of good quality. QCDA

6 5. Can we grade our qualifications? Yes. The Regulatory Arrangements explicitly permit qualifications to be graded, and set out a minimum set of regulatory requirements for such qualifications. As some of the issues related to graded qualifications are quite complex, guidance on developing graded qualifications in the QCF is set out in a separate document Grading qualifications in the QCF (QCDA/10/5201). Questions about assessment delivery 6. Can we use multiple-choice tests to assess units? Yes. There is nothing in the Regulatory Arrangements that states you cannot assess a unit (or part of a unit) through a multiple-choice test. The QCF regulations require awarding organisations to validly assess the skills, knowledge and/or understanding of all the learning outcomes against the stated assessment criteria (5.3c), so a multiple-choice test is a valid way of assessing a unit against the given assessment criteria. One important thing to note is that a test of this kind is an assessment instrument it does not define the content of a unit. The questions set in the test must enable an assessor to make a judgement about the achievement of the learning outcomes of the unit, but there is no requirement that the test questions should reproduce verbatim these learning outcomes. Indeed, the separation of test questions from learning outcomes will be essential where test questions are drawn from a question bank or are randomised to prevent plagiarism. 7. Can we assess units through written examinations? Yes. Written examinations, like multiple-choice tests, are simply a type of assessment instrument and the Regulatory Arrangements do not exclude any type of assessment instrument. Some of the issues identified above in question 6 in relation to multiple-choice tests will also apply to written examinations. Again, there is no requirement that the learning outcomes of a unit should form the actual questions in any such examination. The examination questions must be designed in such a way as to enable an assessor to make an assessment judgement about whether or not the learner has achieved the outcomes of a unit, but the questions themselves may be much more explicit than the learning outcomes of the unit. Indeed, where learners are offered a choice of examination questions, this separation of learning outcomes and examination questions will be essential. QCDA

7 8. Can we assess units through online tests? Yes. The QCF does not exclude any assessment instruments. The only condition here is that the test must be a valid form of assessment in relation to the content of a unit. Of course a test may relate to only some of the learning outcomes of a unit, or to some of the learning outcomes of two or more units. The validity requirement applies in all cases. 9. Can we set a percentage pass mark for a unit? If so, what should it be? Yes, but the percentage pass mark must relate to all the learning outcomes of a unit. A learner cannot be awarded credit for a unit if only a proportion of the learning outcomes have been achieved. Again, the separation of test or examination questions from the learning outcomes of a unit enables such assessment judgements to be made easily. In effect, a percentage pass mark reflects the level of confidence that the assessor has in the outcomes of the assessment process, not the proportion of the learning outcomes that a learner has successfully completed. The QCF sets no particular requirements about pass marks in such circumstances. Providing the assessor is confident that all the learning outcomes of a unit have been achieved against the stated assessment criteria, credit can be awarded for the unit. Of course in many cases percentage pass marks will be linked to grading arrangements for a qualification and not just to the award of credit. Assessment developers should refer to the separate guidance, Grading qualifications in the QCF (QCDA/10/5201), to ensure that the application of percentage pass marks is consistent with the regulatory requirements for graded qualifications in the QCF. 10. We plan to use a portfolio of evidence for assessment. Is this okay? Yes. A portfolio of evidence (either paper-based or electronic) can be used for assessment in the QCF. The awarding organisation must determine the relationship between a portfolio and the units to be assessed. These units may or may not relate to a single qualification, and the portfolio may relate to only some of the units to be assessed, or to some of the learning outcomes of these units. Awarding organisations need to take notice of the QCF requirement that credits should be awarded to learners quickly on completion of a unit. An assessment approach for a large qualification that delays the award of credits until the final submission of a portfolio at the end of the programme is not consistent with the QCF's intention to be responsive to the needs of individual learners. 11. We plan to assess all units through work-based observation. Is this okay? Although there is nothing to prevent an awarding organisation deciding to establish one single approach to assessment for all units, the potential limitations of this strategy need QCDA

8 to be noted. If the awarding organisation was solely responsible for submitting to the QCF all the units that it assessed, then this might be practically possible. However, it may prevent the awarding organisation using units developed by other submitting organisations. Indeed it may prevent it from using units developed by a sector skills council in its sector. Pre-selecting a single approach to assessment and offering qualifications based on units that are all assessed through this one approach is an unnecessarily restrictive approach to the conduct of assessment by an awarding organisation. 12. Can we assess a sample of learning outcomes in a unit? No. All learning outcomes in a unit must be achieved for credit to be awarded. Valid assessment judgements leading to the award of credit cannot be made through sampling only some of the learning outcomes through the assessment process. Of course it is permissible for a single assessment activity to relate to only one or some of the learning outcomes of a unit, but as whole, all assessment activities must cover all learning outcomes. 13. Do all assessment criteria have to be met for credit to be awarded? The Regulatory Arrangements make it clear that learning outcomes and their related assessment criteria 'set a clear assessment standard for the unit' (1.4d). There is therefore a clear expectation that the evidence produced through assessment of a learning outcome will relate to all the assessment criteria linked to that outcome. Whether or not the evidence produced by a learner meets the assessment standard established by a learning outcome and its related assessment criteria will, in many instances, be a matter of professional judgement by the person responsible for assessing the evidence of learner achievement. If the assessor is satisfied that the assessment standard of the unit has been met, then credit can be awarded for that unit. 14. How does the professional judgement of the assessor relate to a set of assessment criteria? The assessment criteria related to a learning outcome should not be seen as a simple checklist to be applied uniformly to all evidence of learner achievement. There is no assumption that assessment criteria are equally weighted in terms of the generation of assessment evidence. In some cases an assessor may well be able to infer from the production of a small amount of evidence that an assessment criterion has been met. In other cases a more substantial amount of evidence may be required in order to satisfy the assessor that the assessment standard has been achieved. QCDA

9 A well-written unit should enable an assessor to make reliable and consistent judgements about the evidence produced by learners to meet the assessment standard, without generating unnecessary evidence that is extraneous to the exercise of this judgement. One of the skills of an assessor will be to ensure that there is sufficient evidence to meet the standard set by all the assessment criteria for a particular learning outcome. In many cases this may well involve the assessor selecting from a range of available evidence. Providing the assessor is satisfied that the assessment standard has been met, there is no assumption that this evidence will be weighted equally against each individual assessment criterion. 15. What if an assessor thinks that learners are required to produce unnecessary evidence to demonstrate that a learning outcome has been achieved? Then their views should be fed back into a review of the unit. All awarding organisations are required to have arrangements in place for the ongoing review of units, based on feedback from the delivery of these units in practice. One of the purposes of this review process is to ensure that assessment is 'manageable and cost-effective' (5.3e) and does not create 'an undue assessment burden for learners or assessors' (1.5c). One of the benefits of a unit-based framework is that individual units can be reviewed, amended and replaced in qualifications very quickly. The Regulatory Arrangements enable such updates with a minimum of bureaucracy, and the regulators anticipate that, especially in the early phase of implementation, units will be updated frequently, based on feedback from learners, assessors and verifiers or moderators. In the context of the QCF the professional responsibilities of assessors relate not just to the application of the assessment standard in a unit, but to improving the quality of assessment through the ongoing review of units over time. 16. Can learners be compensated for poor performance in one unit through excellent performance in another unit? Although such compensatory assessment could work within the grading arrangements for a qualification, it is not possible in the QCF for credit to be awarded where a learner has not achieved all the outcomes of a unit. There is therefore no facility to use compensation in assessment in relation to the award of credit. However, providing this principle is observed, it would be possible for excellent performance in one unit to compensate for a simple pass grade in another unit when the overall grade is calculated for the qualification. This compensation would operate QCDA

10 through the grading arrangements for the qualification and not through the QCF credit system. Please refer to the separate guidance in Grading qualifications in the QCF (QCDA/10/5201). 17. Can learners be compensated for poor performance in one learning outcome through excellent performance in other learning outcomes in the same unit? No. Here the requirements of the Regulatory Arrangements are straightforward all learning outcomes must be achieved against the stated assessment criteria for credit to be awarded. There is no facility for compensatory assessment to operate within a unit. 18. Do we have to assess each unit separately? No. The Regulatory Arrangements require that units should be designed to be capable of individual assessment. They also require awarding organisations to demonstrate that they have the capacity and expertise to be able to assess individual units. The assessment requirements of some units in the QCF databank mean that they have to be assessed separately from any other unit. However, there is no requirement that awarding organisations have to assess each unit separately in all cases. Assessment activities may be developed that relate to more than one unit, or to some of the learning outcomes of two or more units. Awarding organisations will need to balance the expectation that learners should have access to the award of credits at regular intervals with the manageability of the assessment process. 19. Do we have to assess each unit in the same way in each centre? No. The QCF enables different centres to assess the same unit in different ways. There is no requirement for an awarding organisation to establish a standard approach to unit assessment across all centres (though of course some awarding organisations will choose to do so). It should however be emphasised that in all instances, the assessment of a unit must produce sufficient evidence from learners to enable reliable and consistent judgements to be made about all the learning outcomes against the stated assessment criteria (5.3d) and that the awarding organisation will be held accountable by the regulators for the quality of assessment through all its centres. 20. Can a centre use different methods of assessment for the same unit offered to different groups of learners? Yes. The flexibility of assessment in the QCF allows a centre to use different assessment methods and/or instruments for the same unit offered to different groups of learners. It QCDA

11 may be that these differences arise because the unit is located in two or more different qualifications. In all instances, assessment must be valid, reliable and consistent against the assessment standard of the unit, and the awarding organisation will be responsible for assuring the regulators of this in all cases. 21. Can we ignore the assessment guidance field in the unit template? The QCF unit template includes a separate field where assessment requirements may be identified. Unless an explicit requirement is set by the unit-submitting organisation, the awarding organisation conducting assessment for that unit may choose to ignore the guidance. However, the awarding organisation must be confident that ignoring such guidance has no detrimental impact on the quality of the assessment process. 22. We plan to base our approach to assessment on the NVQ Code of Practice. Is this okay? An awarding organisation may choose to establish an approach to assessment based on the National Vocational Qualifications (NVQ) Code of Practice. However, the code is explicitly based on the 2004 Statutory Regulations for the National Qualifications Framework (NQF) and therefore has no statutory force in the QCF. The regulators will still monitor and review awarding organisation assessment practices against the Regulatory Arrangements and not the NVQ Code of Practice. As it stands, the code cannot meet QCF requirements because it relates explicitly to qualifications and not to individual units. However, awarding organisations may adapt elements of the code, relate these to individual units, and use this as guidance or requirements for assessment through the 'additional information fields' in the unit template. QCDA