Interim Statement on Statistical Quality and Quality Assurance of Administrative Data Sources

Transcription

1 Interim Statement on Statistical Quality and Quality Assurance of Administrative Data Sources August 2018 Version Control and Review This Interim Statement on Statistical Quality and Quality Assurance of Administrative Data Sources was approved by the Management Board on 13 June This is an interim statement while Qualifications Wales further develops its approach to the publication of statistics. Feedback on this policy is welcomed at any time. Please send any comments to Please note that only comments regarding the generality of this document, rather than specific situations will be considered as part of that review. 1

2 Statement on Statistical Quality and Quality Assurance of Administrative Data Sources Introduction Qualifications Wales is the regulator of qualifications other than degrees and the qualification system in Wales. Under the Qualifications Wales Act 2015, our principal aims are to ensure that qualifications, and the Welsh qualification system, are effective for meeting the reasonable needs of learners in Wales and to promote public confidence in qualifications and in the Welsh qualifications system. This policy document outlines our statement on statistical quality which details the principles and procedures that we adhere to when producing statistical outputs. We also outline our quality assurance processes for the administrative data sources we use in our statistical outputs. The quality of our statistics is important to ensure that users have confidence in our statistical services and to ensure that we are positively recognised for: the quality of our advice; the quality of our data; the relevance and impact of our analysis. The quality of our statistics has an impact on the trust our users have in them and their trust in us. When making decisions, users need to be clear about the quality of the data they are using, and we strive to provide high quality statistics. We ensure our statistics are fit for purpose; we use appropriate processes and are transparent about our methods. We also ensure the factual and presentational quality of our statistics meets the requirements of our users. When using data from administrative sources we will work with data providers to improve quality and to provide users with appropriate levels of information on processes. We operate in compliance with the UK Statistics Authority Code of Practice for Statistics, which provides producers of official statistics with the detailed practices they must commit to when producing and releasing statistics. The Code is based around three pillars: trustworthiness, quality and value. The quality pillar provides guidance for ensuring sound statistical practices. Background to the statistics Our statistical outputs present statistics relating to qualifications that are regulated by us. In previous years, Ofqual produced statistics at an England, Wales and Northern Ireland level and at individual country level for the 2016/17 academic year 2

3 only. Caution is needed when making comparisons between statistics relating to these three nations from 2017/18 onwards. These statistics are likely to be of interest to us, Ofqual, other regulators, awarding bodies and the Welsh Government. We welcome feedback from users, so expect to gain a better understanding of who our users are over time. The data sources and suppliers Our regular statistical publications are produced using administrative data collected from awarding bodies that we recognise. The data are only for centres located in Wales, regardless of the type of centre. This document outlines the processes regarding data sent to us directly from awarding bodies for general qualifications statistics. As a regulator of qualifications, we need to balance data quality with our regulatory principles, for example being proportionate in the requirements placed on awarding bodies during our data collections. As far as possible, we aim to collect similar data to Ofqual using similar timelines, with the intention of reducing additional requirements. Ofqual provide data to us detailing the number of certificates awarded for vocational qualifications for centres in Wales, which is presented in our quarterly vocational qualifications statistical release. Following the transition period, we aim to collect our own vocational qualifications directly from awarding bodies from April 2019 and this statement will be updated accordingly. The Statistical Quality Objectives In line with the quality pillar of the UKSA Code of Practice for Statistics, our statistical team has four strategic objectives for quality: 1. Ensure staff know about the quality assurance methods in place for our statistical products and what is expected of them; 2. Ensure staff are trained in effective quality assurance and quality management; 3. Publish appropriate quality information for our statistics; and 4. Conduct regular reviews of our processes and the outputs we produce, ensuring we work with data providers to understand the end-to-end processes and the meaning and quality of the data provided to us. 3

4 What is Quality? The Code of Practice for Statistics defines quality in the following way: Quality means that statistics fit their intended uses, are based on appropriate data and methods, and are not materially misleading. Quality requires skilled professional judgement about collecting, preparing, analysing and publishing statistics and data in ways that meet the needs of people who want to use the statistics. 1 There are three principles within the quality pillar of the Code, each with its own associated practices: Q1 Suitable data sources Q2 Sound methods Q3 Assured quality Quality and Our Users In order to be a responsible producer of statistics we provide information that enables our users to know and understand the quality of our statistics. This will allow them to decide if our outputs meet their needs and to understand the limitations of our data when they are using that data to make decisions, measure performance or allocate resources. In turn, this facilitates better conversations with our users about their ongoing needs. Identifying and responding to user needs is a key component of the Code of Practice. Table 1: Delivery of statistical quality pillar Quality Principles Q1 Suitable data sources Statistics should be based on the most appropriate data to meet intended uses. The impact of any data limitations for use should be assessed, minimised and explained. What users can expect from us All our statistical releases contain background notes which include an explanation of the strengths and limitations of the statistics which are considered with the potential uses of the statistics. Any changes to our data or possible reasons for breaks in the series will be explained in these notes contained in the release. Our statistical releases are based on data collected from awarding bodies. This is the most appropriate data source to use for a number of reasons, such as: Data comes directly from the source; 1 4

5 It is consistent with other UK statistics producers for qualifications, e.g. Ofqual; It is consistent with the methodology used by Ofqual when they previously published statistics for Wales we use the same templates as Ofqual. This helps to meet user consistency needs and minimises limitations of historical comparisons. Awarding bodies submit their data via QiW 2, which is a database owned and managed by us. Automated validation rules are run on the data files submitted and these must be passed before the data is accepted into QiW. Our statistical team carries out further checks and query the data with the awarding bodies when necessary. Q2 Sound methods Producers of statistics and data should use the best available methods and recognised standards and be open about their decisions. Our data collection schedule is published online and sent to the relevant individuals from awarding bodies ahead of collections. A requirement notice is sent ahead of collections that includes how and why the data will be used. A template and validation rules are also shared with awarding bodies ahead of collection. Where possible we keep our data coherent across different levels of aggregation, consistent over time, and comparable between geographical areas. All our statistical releases contain background notes and a glossary which contain information about the data, definitions and the methods used. We also take the following steps: Attend regular data forum and data consultation group meetings with awarding bodies and other UK regulators to discuss data matters. Work closely with Ofqual, who previously produced many of the statistics that we now produce for Wales on our behalf. This is to ensure a smooth transition and keep our methods consistent with theirs wherever possible. If our methods differ from those previously used for Wales data, or if they are different to the methods used by Ofqual for equivalent data, then we clearly explain how and why in our release/background notes. We also inform users to be cautious when making interpretations and make clear the impact any methodological changes have had on the statistics. We 2 For a detailed description of QiW, see 5

6 are also transparent about any limitations to the data to aid user understanding. We provide a glossary in each statistical release published which provide clear definitions to aid interpretation. Staff are recruited openly and with appropriate qualifications from relevant disciplines. Staff regularly attend training and events with the wider GSS statistical community to share best practice. Validation rules are run on all of our regular data collections. We use the Joint Council of Qualifications (JCQ) subject groupings in our statistical outputs. Q3 Assured quality Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely. All our statistical releases include the following information, either within the release or as accompanying notes on our website: Details of how the underlying data are collected to allow users to understand the strengths and limitations. A description of quality issues and any impact this may have on analysing changes over time. Compliance with, and specific details of, our Interim Revisions and Corrections Policy for Statistical Releases. Details of our quality assurance checklist and process e.g. independent checking. Comparisons to other data sources. QiW validation. Quality assurance of administrative data. As standard practice, we release related statistical publications on the same day and at the same time to aid user understanding, unless this would mean significant delay to present the coherent set of releases. Where related statistics are published across several publications we make it clear to users where the related information can be found. We publish statistical releases as soon as possible after the relevant time period. We also pre-announce our statistics on our website and Welsh Government s website in line with the code of practice. 6

7 We conduct periodic reviews on the strengths and limitations of the data and methods used. If we publish corrections or revisions we ensure that we explain clearly the scale, cause and impact these changes will have on the data and the messages from the data. We have provided a statistical process flowchart which outlines the quality assurance taken at each stage of the process on page 10. Implementing the Statistical Quality Objectives Ensure staff know about the quality assurance methods in place for our statistical products All staff have a (differing) responsibility for ensuring the quality of the statistics we produce. By establishing a culture of continual improvement, the quality of outputs will improve as all those involved in producing statistics implement best practice. We communicate guidance clearly and hold regular meetings to share best practice and common issues. Ensure staff are trained in effective quality assurance and quality management. To maximise the culture of continual improvement and self-assessment, staff are trained in effective quality assurance and data checking approaches according to their roles and responsibilities. Statistical quality is fundamental to the induction process for our new staff, with all new staff receiving training in quality management and data checking. We have a quality guidance and quality assurance checklist document for use by statistical staff. Publish appropriate quality reports or indicators for our statistics. To increase transparency and trust in our statistics, we publish quality reports or indicators for all our statistics, in the most appropriate manner. Quality information is either included in individual outputs or published as a Background Quality Report (available on our website as a separate document alongside our statistical tables/report for the relevant output). These include details of how our statistics meet and respond to users needs and details of appropriate use of our statistics, including any limitations to their use. One element of quality management is ensuring appropriate and transparent handling of revisions and errors. We use our Interim Revisions and Corrections Policy for Statistical Releases policy for this, which 7

8 contains information for handling revisions. Conduct regular reviews of the processes and outputs we produce, ensuring we work with data providers to understand the end-to-end processes. To continually improve our processes and outputs, we conduct peer reviews and critical self-assessments, for example using Government Statistical Service (GSS) quality management tools and guidance such as the GSS Quality, Methods and Harmonisation Tool. 8

9 Quality Assurance of Administrative Data With regards to statistics that use administrative sources we aim to follow the UK Statistics Authority guidance 3 for the quality assurance of administrative data. We will ensure that we work with data providers to understand the quality of the underlying data and related administrative issues that might impact on the data, and to continuously seek ways to improve the end-to-end process. The following primarily relates to general qualification data that is collected by us. Currently, statistical releases for vocational qualifications published by us are based on data collected by Ofqual. In the production of the statistical release we follow the same principles that we outline below but for further guidance on the quality assurance carried out on the data prior to Ofqual sending the data to us, please refer to Ofqual s quality assurance framework for statistical publications 4 and their statement of administrative sources 5. For further detailed information on the source of administrative data used by us please see our list published as a separate document. Outline of our quality assurance approach for administrative data 1. Determine the level of assurance for each data source by deciding the degree of data quality risk and level of public interest. 2. Gather information on the four practice areas for each data source from available material held by the statistical team and document the range of activities undertaken to assure the data. 3. Contact data suppliers for information about their approaches to data quality assurance. 4. Identify the strengths and limitations of the statistics taking into account the quality issues. 5. Regular contact with other qualification statistics producers, e.g. Ofqual to share insight about quality issues and method developments

10 Statistical Output Production Flowchart 1. Project planning 8. Evalute/ Lessons Learned 2. Data Collection 7. Publication and final sign off 3. Data Delivery and Validation 6. Disseminate 4. Data Transformation/ Process 5. Analyse The above flowchart provides the high-level stages involved in the production of our statistical outputs. Quality assurance checks are conducted at each stage in the process for all statistical releases, and the general process is outlined in further detail below. Stages of Production 1. Project Planning The first stage of producing statistics is project planning. Effective planning is essential to the quality assurance process. It is important that lessons learned from previous publications are implemented into the project planning for the next publication. We will assess the main risks to quality, which may have changed since the last publication: these risks could involve resource, staff availability over the period, and 10

11 so on. Lessons learned from previous publications could help manage these risks and it is therefore vital to include these in the initial phase of planning. We will ensure that quality assurance documents, guidance notes and process plans are updated following the production of the previous publication and lessons learned are incorporated into these documents. Risk and issue logs will take account of new data items, revisions and system changes and kept up to date to ensure that further quality assurance checks take place after each incident. We will assign roles to tasks and identify the stages where sign off is required in order to mitigate potential risks during the process. This will enable time for robust quality assurance during the whole process and should allow for as much independent checking as possible. When the project is being planned we will determine user requirements to ensure that the output is relevant and meets the user needs and follows the values and principles of the Code of Practice for Statistics. 2. Data Collection Data collection is often carried out by an external organisation on our behalf, most often by an awarding body. It is vital that we perform checks when we receive data from the awarding bodies and that the awarding body checks the data received from centres to ensure that the data meets our needs. Under Condition B4 of the Standard Conditions of Recognition we require awarding bodies to ensure that any information provided under this condition is accurate and complete. Therefore, we require awarding bodies to have sufficient quality assurance processes in place to check the data received from centres before this is sent to us. When we require data from an awarding body, we will issue them with a notice that describes the data required, the format that it must be submitted in and when it must be submitted. We will carry out quality assurance checks at this stage to ensure that the data definitions for collection are consistent with previous years and in some instances consistent with relevant organisations across the United Kingdom, e.g. Ofqual. Any inconsistencies will be reviewed and there will be a determination if the time series is going to be affected by these inconsistencies. Once complete, a decision will be made to either find a solution or provide an explanation of the differences between the two publications. There may be other changes to the data collection process which would need to be identified and assessed in the same way. 11

12 We will check the data templates provided to the awarding bodies for the same reasons as above. Checks will be made prior to data collection with templates provided by similar organisation across the UK, e.g. Ofqual. Where required, discussions will be held to determine the rationale for any differences and whether the data template needs to be altered to account for this. The raw dataset provided by the awarding bodies for each collection will be checked by us at the initial stage to ensure that missing rows, columns or individual cells are explained or addressed by the data awarding body. The dataset will also be checked to ensure that fundamental errors are eradicated, e.g. whether the totals match the sum of the components. We will raise any issues with this data with an awarding body at the earliest opportunity to allow time for errors to be resolved. 3. Data Delivery and input validation It is vital when data is received from awarding bodies that the data appears as we would expect. It is possible that the wrong file may be sent, the incorrect data template used or perhaps data corruption during the process of data transfer. Therefore, we will run checks to ensure that the file is in the format we would expect. This is a critical stage and can save time by ensuring that the data is correct at the point of collection. Further common-sense checks will be carried out on the data at this early stage. An example of a type of these checks could involve data visualisation where scatterplots are created to identify outliers, or a time series is created to identify particular year on year trends. We would then follow up with the awarding body/bodies where data looks peculiar and investigate the rationale for this. At this point we would also perform similar checks that should have been completed by the awarding bodies, e.g. ensuring that totals match the sum of the components. Checks will also be made at this stage to ensure that code is functioning correctly when transferring the data from QiW 6 into our data repository for transformation and analysis. If there are any concerns at this point, then production will not proceed until a solution has been implemented. 4. Data transformation/processing 6 Qualifications in Wales (QiW) is a database owned and managed by Qualifications Wales. For further information, see: 12

13 This stage involves us checking the process of converting the data provided by awarding bodies into tables or subsets used in the publication of the statistical release. When transforming the data from individual candidate/centre to an aggregated format there is a high risk to quality if thorough quality assurance is not in place. Therefore, we use two methods and two different software packages independently when transforming data to ensure that the same result is achieved. When code is used, this is checked independently by another statistician to ensure the process and logic are correct. At this stage automated processes will be checked for consistency with previous publications and checks are implemented such as year on year trend analysis, along with spot checks for various data breakdowns. This is important especially if there are any changes to the data collection process which may introduce errors, e.g. new data fields, reordering of data fields, etc. At this stage summary statistics will be produced for each awarding body and sent back to them for quality assurance. 5. Analyse When analysing the data, we will ensure that two methods are used to confirm consistency. This is the same for writing the commentary for the statistical release and producing the tables and charts in the release. Where applicable, code will be independently checked to ensure logic and processes are correct. The data tables produced may be included in the statistical release and/or available as supplementary tables to coincide. Tables will be compared against historical data and previous publications and large differences are investigated with rationale provided in the commentary of the release. Tables that summarise the same information in various breakdowns will be compared to ensure consistency. If possible, tables are compared to external sources. When drawing conclusions to the data, statisticians will hold discussions with colleagues to understand relevant context for data movements and to aid quality assurance. 6. Data Dissemination We may disseminate data in various ways, such as through commentary in the statistical release, tables, charts and infographics. We will ensure that every table, chart and infographic has been sourced correctly, along with the source of the data 13

14 for commentary in the release stated in the introduction/background of the publication. We will clearly explain any data limitations to users using plain English to enable them to draw appropriate and accurate conclusions in the data. Limitations may include changes in measures across years affecting comparability over time. Regular users who receive pre-release access will be ed to identify if they continue to require pre-release access to the statistics. A list is maintained and published alongside the release on the day of publication. All checks are completed prior to the pre-release date and when the document is being translated (where applicable). Checks will be carried out to ensure that only the correct individuals receive the pre-release access and that s carry the correct security markings, so that recipients are aware not to share the statistics with others. Briefing will be provided to Communication colleagues and checks are carried out to ensure that trends and numbers are consistent with the statistical release. The wording will be checked to ensure that it is in plain and clear language, whilst ensuring that appropriate definitions are provided so lines cannot be misinterpreted. These briefing lines are cleared by the responsible statistician or the Head of Research and Statistics. 7. Publication and final sign off Often the statistical release is the main/only document that users will consult. It is important that enough resource has been made available to thoroughly review the document for sense checks, statistical checks and grammar checks. These checks will include ensuring that the messages in the data are accurate and consistent throughout, that any unusual data movements have been explained, that the methodology has been outlined, that the charts and tables are referring to the most up to date data, titles and labels are correct and the correct time period is being referred. All checks will be included on a statistical release checklist. Two statisticians will ensure that each check has been conducted independently and then signed off by the Head of Research prior to publication. Where applicable, once a translated release has been received, all numbers will be rechecked, and the formatting will be completed again. During the planning stage a publication timetable will have been agreed with the Communications team and will be followed on the day of publication. This will include the use of social media where applicable. The leading statistician will liaise 14

15 with the Communications team throughout the process and check any briefing or social media that is produced. 8. Evaluate/Lessons Learned During the process of production, we will keep a log of the lessons learned throughout each stage. These will be updated frequently and by all statisticians who work on producing the statistics. A wash-up meeting will be held to discuss what lessons have been learned from the production of the latest release. This will include what went well, so that good practice guidance can be compiled, along with areas that could have gone better and suggestions for further improvements. It is important that any actions from this are allocated and given enough time and resource to be implemented prior to the next release. The lessons learned will also be included in the planning stage of the next release. Along with this, user feedback will be collected and used to identify areas for future improvements. 15

16 Assessment of General Qualification (GQ) Statistics We follow guidance provided by UKSA in the Administrative Data Quality Assurance Toolkit 7 in assessing GQ statistics, which is set out below. This assessment is only based on statistics collected by us, which are all GQ statistical releases. Currently Ofqual still collect vocational qualifications (VQ) data 8 on our behalf and we produce the statistical release. We aim to collect our own VQ from April 2019 and will update this assessment to account for this. Level of risk of data quality concerns We have followed the UKSA Administrative Data Quality Assurance Toolkit to determine whether the administrative data we publish should be regarded as low, medium or high risk of having quality concerns. We have decided that the data quality concern for all our statistical series are either low or medium. Low and Medium risk categories are outlined by the UKSA as: Low risk the data may have a low risk of data quality concerns in situations in which there is a clear agreement about what data will be provided, when, how, and by whom; when there is a good appreciation of the context in which the data are collected, and the producer accepts that the quality standards being applied meet the statistical needs. Medium risk the data may be regarded as having a medium risk of data quality concerns when high risk factors have been moderated through the use of safeguards, for example, integrated financial audit and operational checks, and effective communication arrangements. It is also appropriate to consider the extent of the contribution of the administrative data to the official statistics, for example, in cases where the statistics are produced in combination with other data types, such as survey or census data. High risk - the data may have a high risk of data quality issues when there are many different data collection bodies, intermediary data supplier bodies, and complex data collection processes with limited independent verification or oversight For further information about Ofqual s quality assurance procedures, see: 16

17 Public interest profile of the statistics We have used the UKSA toolkit to determine that the general qualifications statistics we publish have a low or medium public interest profile. This is described in the toolkit as: Low profile politically neutral subject; interest limited to niche user base, and limited media interest. Medium profile wider user and media interest, with moderate economic and/or political sensitivity. High Profile- economically important, reflected in market sensitivity; high political sensitivity, reflected by Select Committee hearings; substantial media coverage of policies and statistics; important public health issues; collection required by legislation. Risk/Profile matrix Table 1: Risk/Profile Matrix for General Qualifications statistics Statistical series Administrative source Data quality concern Public interest Matrix classification November QiW Low Medium A2 entries Appeals QiW Low Low A1 Review of QiW Medium Medium A2 marking Summer QiW Low Medium A2 entries Malpractice QiW Medium Low A2 Special consideration Access arrangements Entries and late entries for GCSE and A level QiW Low Low A1 QiW Low Low A1 QiW Low Medium A2 Table 1 outlines the data quality concern and level of public interest for each statistical series relating to general qualifications published by us. We have determined that the data quality of each statistical series is either low or medium and 17

18 the public interest is low or medium. Therefore, the risk/profile matrix classification for each data series is either A1 or A2. We have decided to provide enhanced (A2) assurance for all our statistical series within this policy, background notes and methodology in our statistical outputs. Quality management actions There are three quality management actions outlined below: Investigate: Statistics producers should investigate, for example: the types of checks carried out by data collectors and suppliers, as well as the operational circumstances in which the data are produced identify any coverage issues and potential sources of bias in the data collection and supply process Manage: Producers should also manage: their relationships with suppliers by establishing clear processes for data provision and for managing change maintain regular quality assurance checks of the data and use other data sources where possible to corroborate their findings Communicate: Producers should communicate effectively: with their data suppliers and others to ensure users are provided with clear explanations of the strengths and limitations of the data work closely with other statistical producers using the administrative data to ensure a common understanding of any quality issues When considering the quality management actions, note that the majority of data for general qualifications for Wales comes from one awarding body. Table 2 outlines the quality management actions we will be undertaking, these are reviewed continuously throughout the process. Table 2: Quality management actions undertaken for quality assurance of GQ statistics Quality Actions Management Area Investigate Quality assurance carried out by awarding bodies, including the operational circumstances in which data are produced. Investigate through visits to awarding bodies. We validate outputs against data previously produced by Ofqual. 18

19 We plan to visit some awarding bodies to further understand their data processes. Manage We define clear roles and responsibilities across the statistics team, IT team and data suppliers, and separate internal and external quality assurance checks. We provide data submission guidance to data suppliers outlining input validation rules. Communicate We have regular dialogue with data suppliers. We have documented quality assurance processes for creating general qualifications statistics and will update these accordingly. We provide a description of data methodology included in the background notes of each statistical output. We provide a description of data limitations included in the background notes of each statistical output. We plan to engage further with users of statistical outputs. We have regular dialogue with Ofqual to ensure a common understanding of any quality issues. We are in the process of creating a system to send automated s to awarding bodies to request data, send remainders, etc via our Central Records Management system (CRM). Quality assurance of administrative data is an ongoing, iterative process to assess the data s fitness to serve their purpose. This reflects the ongoing use of the data and the dynamic native of operational environments. Consequently, these quality management actions will be continuously reviewed and will be updated when this policy is reviewed. If flaws are found in the administrative data, we will: evaluate the likely impact on the statistics establish whether the issue can be resolved, or whether there is any other action they can take to mitigate the risks determine whether the level of impact is such that users should be notified. 19

20 Strengths and weaknesses Table 3 outlines the strengths and limitations of the administrative data used to produce Qualifications Wales general qualifications statistics. These will be reviewed continuously, and any updates will be included in the next review of this policy or in the background/methodology notes of statistical releases. Table 3: Strengths and limitations of GQ statistics Strengths Data has been submitted via QiW through fully automated input validation rules. General qualification data from awarding bodies covers all centres in Wales offering these qualifications. Data guidance is provided to awarding bodies to ensure consistent data definitions are used. Data suppliers and producers work in close proximity aiding understanding of processes and facilitating resolution of issues. This includes awarding bodies being fully consulted during the initial design phase and any subsequent change phases. As a condition of awarding bodies being regulated, all data supplied to Qualifications Wales must be completely accurate. QW works closely with Ofqual to ensure consistency with data definitions and procedures whilst limiting the burden on data suppliers, to ensure geographical comparisons are possible. Limitations Potentially many data collectors (centres) providing their data to awarding bodies to then supply to Qualifications Wales. There is potential error in the information provided by awarding bodies. However, this is limited by comparison checks, input validation checks and quality assurance procedures implemented prior to data suppliers being challenged or questioned about the data. Qualifications Wales have to rely on awarding bodies ensuring that centres are using consistent data definitions when supplying data to them. Comparability issues may arise in future publications due to the various qualifications reforms taking place in both England and Wales. We mainly collect aggregated data for statistical releases such as summer entries, this is not the lowest level of data available from the awarding bodies. We do collect candidate level data for releases such as review of marking. 20