Using Evaluation to Strengthen Organisational Self-Assessment. A supplement for industry training organisations

Transcription

1 Using Evaluation to Strengthen Organisational Self-Assessment A supplement for industry training organisations 1 July 2010

2 Contents Introduction 3 Self-assessment 4 External evaluation and review 5 Evaluation tools for ITOs 6 Appendix 1: How the ITO indicators relate to the key evaluation questions 8 Appendix 2: Evaluation Indicators for ITOs an Overview 10 Appendix 3: Plan of Enquiry for ITOs 12 2

3 Introduction This document provides supplementary material to support industry training organisations apply the evaluative approach to their particular role and functions in arranging training, setting standards and providing industry leadership. It is to be used in conjunction with NZQA s Policy and Guidelines for the Conduct of External Evaluation and Review. This document is one of three prepared to support the evaluative approach to quality assurance in industry training organisations (ITOs). The others are Policy and Guidelines for the Conduct of External Evaluation and Review A supplement for Industry Training Organisations and Evaluation Indicators for Industry Training Organisations. ITOs and evaluators should consider these documents in total when planning and undertaking self-assessment in ITOs. Quality Assurance Environment The NZQA Board set policies and guidelines for the conduct of external evaluation and review of ITOs on 27 May 2010, pursuant to NZQA s statutory role under Sections 6(a), 7(f) and 10 of the Industry Training Act (1992) to provide advice to the Tertiary Education Commission on the registration and re-registration of ITOs. In the past, quality audits were usually conducted at five-yearly intervals to coincide with NZQA s advice to the Tertiary Education Commission about the re-recognition of each ITO. Self assessment and external evaluation and review replace quality audits. In addition, the Policy and guidelines for conducting external evaluation and review replaces Quality Assurance Standard for Industry Training Organisations. NZQA is looking to make statements of confidence in ITO performance and capability in self-assessment. However, where compliance issues are identified, the ITO will be responsible for taking any appropriate corrective action. In serious cases more definitive action may be required, and the current levers may be used to manage any identified non-compliance with the relevant policy and regulatory positions. The key levers are: withdrawal of consent to assess unit standards on the Directory of Assessment Standards 1 and reporting non-compliance to the Tertiary Education Commission. 1 Unit standards are now registered on the Directory of Assessment Standards. 3

4 Self-assessment Self-assessment includes the ongoing processes an ITO uses to engage in systematic, data-driven, self-reflection based on their mission or purpose, leading to actions to improve and develop. To be effective, self-assessment needs organisational ownership and leadership so that it is a routine part of all business and strategic planning. Key features of effective self-assessment in an ITO NZQA does not prescribe how an ITO should conduct their self-assessment. However, any process used should be comprehensive, authentic, transparent, and robust, and focused on: Needs assessment extent to which the ITO systematically determines and addresses learner and industry needs. Processes and practices impact of ITO processes and practices on achieving these outcomes. Learner achievement effectiveness of learning and teaching/training and the impact of education and training on learner progress and achievement. Outcomes targeted outcomes and their value for learners and other stakeholders. Using what is learned use of what is learned from self-evaluation for planning and improvement purposes across the organisation. Actual improvement the extent to which relevant, worthwhile improvements actually occur. If self-assessment is robust, and record keeping is meticulous, then the external evaluation and review process can be relatively non-intrusive. 4

5 External evaluation and review External evaluation and review is a periodic evaluation of an ITO to provide a level of confidence (judgement) about the ITO s performance in delivering relevant outcomes and its capability in using self-assessment to improve its performance. External evaluation and review uses a systematic process to make independent judgements about ITO performance and capability in self-assessment. ITO performance is the extent to which the training outcomes influenced by an ITO represent quality and value for employers, industry, trainees, and government. An evaluation of ITO performance involves answering questions focused primarily on the quality, relevance and value of training and how well the ITO meets the needs of these groups. Capability in self-assessment is the extent to which the ITO systematically uses self-assessment information to understand performance and bring about improvement. It reflects the extent to which the ITO effectively manages its accountability and improvement responsibilities. External evaluation focuses most directly on the: extent to which the ITO systematically determines and addresses employer, industry, trainee and government needs key processes contributing to the achievement of outcomes for learners (through the ITO s responsibilities for industry leadership, standards setting and arranging training) quality and relevance of training and its impact on trainee progress and achievement achievement of outcomes for employers, industry, trainees, and government effectiveness of the ITO s self-assessment in understanding its own performance and using this for improvement. 5

6 Evaluation tools for ITOs Key evaluation questions and evaluation indicators are the main tools of external evaluation and review, along with performance criteria used to reach judgements about ITO performance and capability in self-assessment. Key evaluation questions The key evaluation questions for ITOs provide a common platform for the external evaluation and review of the ITO regardless of their size and complexity. These high-level open ended questions focus on the outcomes achieved or the key processes contributing to the outcomes. Outcome questions 1. How well does the ITO understand and meet the needs of industry, learners and government? Successful ITOs identify and balance current and future needs of their industry/ies and use this information to maintain the utility of training and standards for employers and learners. Systematic needs analysis helps to ensure that industry, learner and government needs are well understood and enables standards and qualifications to be designed to meet those needs. 2. What is the value of the outcomes for employers and their trainees? Who are the important stakeholders, and how does the ITO know that their needs are being met through the achievement of identified, relevant and important outcomes. Process questions 3. How well do trainees achieve? Evidence of actual trainee achievement, including where possible the progress or value added component, is a primary indicator of an effective ITO. The achievement of trainees is important for ensuring trainees meet the needs of employers and industry. 4. How effective is the training arranged by the ITO? ITOs play an important role in arranging training (on- and/or off-job) and the quality of these arrangements is important in achieving outcomes. The effectiveness of offjob training is the primary responsibility of providers. However, the ITO has an interest in the quality of provision through direct engagement with the tertiary education provider as well as through the relevant quality assurance body. 6

7 5. What is the quality of the assessment being undertaken towards industry standards and qualifications? The maintenance of national standards is an important role of ITOs. The integrity of assessment is a major contributor ensuring consistency of standards within and between industries. 6. How well does governance and management support trainee achievement? ITOs organisational purpose and directions should be clear with strategic and business plans relevant to meeting identified needs of industry and learners. Evaluation indicators for ITOs The ITO evaluation indicators systematically set out the kinds of evidence that evaluators might use to answer the individual key evaluation questions, and are organised into the five key areas relating to ITO performance (including the ITO statutory responsibilities): ITO purpose and direction industry skills leadership setting skill standards and qualifications managing arrangements for training, and assessment. Performance criteria Performance criteria set out the definitions or explanatory guides for external evaluation and review teams to make judgements about ITO performance and capability on self-assessment. 7

8 Appendix 1: How the ITO indicators relate to the key evaluation questions Key evaluation question How well does the ITO understand and meet the needs of industry, learners and government? What is the value of the outcomes for employers and their trainees? How well do trainees achieve? How effective is the training arranged by the ITO? Types of indicators that may be used to help answer the key evaluation questions The ITO: provides valid and/or effective advice to industry, employers and trainees regarding the skill development needs of its industry/ies provides valid/and/or effective advice to government and crown agencies, and tertiary education providers on matters relating to skill shortages, supply, and demand supports and is supported by its industry contributes to a satisfactory supply of skilled workforce. The ITO: ensures that skill development and training reflect skill needs and career paths in the industry trainees are gaining credible industry standards and qualifications industries served by the ITO retain and advance skills and capability within their industry/ies. Trainees: achieve standards and qualifications that are current, credible, and meaningful, and reflect the skill requirements of industry standards and qualifications provide for pathways in employment and/or further training progress is actively monitored and trainees are supported to succeed. Training: is an appropriate mix to ensure that trainees acquire relevant skills for the workplace meets the skills needs of employers and employees. 8

9 What is the quality of the assessment being undertaken towards industry standards and qualifications? How effective are governance and management in supporting trainee achievement? Assessment: in the workplace supports learning and provides evidence of achievement to the required standard conducted in the workplace and/or by accredited providers is valid, reliable, fair and transparent. The senior managers and governors of the ITO: articulate a clear purpose and direction undertake activities consistent with the purpose and direction of the ITO effectively anticipate and respond to change allocate resources efficiently to support the development of a skilled and qualified industry. 9

10 Appendix 2: Evaluation Indicators for ITOs an Overview OUTCOME INDICATORS Trainees gain credible industry standards and qualifications. Industry the industry/ies served by the ITO retain, and advance skills and capability within their industry/ies PROCESS INDICATORS ITO supports and is supported by its industry contribute to a satisfactory supply of skilled workforce. ITO Purpose and Direction Governance and management systems and processes are effective and enable it to: articulate a clear purpose and direction undertake activities consistent with its purpose and direction effectively anticipate and respond to change allocate resources efficiently to support the development of a skilled and qualified industry. Industry skills leadership valid and/or effective advice to industry, employers and trainees regarding the skill development needs of its industry/ies valid and/or effective advice to government and crown agencies, and tertiary education providers on matters relating to skill shortages, supply, and demand ensures that skill development and training reflect skill needs and career paths in the industry. Setting skill standards and qualifications standards and qualifications are current, credible, and meaningful, and reflect the skill requirements of industry standards and qualifications provide for pathways in employment and/or further training. 10

11 Making arrangements for training develops an appropriate mix of training to ensure that trainees acquire relevant skills for the workplace actively monitors the progress of trainees and supports them to succeed ensures that the training meets the skills needs of employers and employees. Assessment in the workplace supports learning and provides evidence of achievement to the required standard conducted in the workplace and/or by accredited providers is valid, reliable, fair and transparent. 11

12 Appendix 3: Plan of Enquiry for ITOs Below is a worksheet you may like to use for planning. It has been partially filled in to show how to use the document. Key evaluation questions Enquiry questions Sources of self-assessment information (Are sufficient information sources identified to demonstrate the likelihood of credible and reliable findings?) How well do the ITOs standards and qualifications match the needs of industry? Reports of qualifications reviews Reports of unit standards Conversations with industry representatives/ employers/ management How well do trainees achieve? Continue as required 12