Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review Transpower New Zealand Ltd trading as Transpower Grid Skills Highly Confident in educational performance Highly Confident in capability in self-assessment Date of report: 26 March 2015

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review... 5 Summary of Results... 6 Findings... 8 Recommendations Appendix MoE Number: 8351 NZQA Reference: C16751 Date of EER visit: 25 and 26 November

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: Transpower New Zealand Ltd trading as Transpower Grid Skills Private training establishment (PTE) Level 5, IRD Building, Corner Ashley and Ferguson Streets, Palmerston North. Staff are now located at the three regional offices in Auckland, Palmerston North and Christchurch. There are five active delivery sites in Blenheim, Christchurch, Wellington, Palmerston North and Huntly. First registered: 1 July 1996 Courses currently delivered: Code of Practice signatory: Domain accreditations: Core Electrical (to level 2) Electricity Networks (to level 5) (Expiring) Electricity Supply Core Skills (to level 5) Electricity Supply Design and Commissioning (to level 5) Electricity Supply Distribution Networks (to level 5) Electricity Supply Live Work (to level 5) Electricity Supply Power system Maintenance (to level 5) Electricity Supply Testing (to level 5) Electricity Supply Transmission Networks (to level 5) No, all domestic students 3

4 Number of students: Domestic: 2,723; Māori, 50 and Pasifika, 14 Number of staff: Scope of active accreditation: Distinctive characteristics: Eight full-time equivalents As above Transpower Grid Skills is the training arm of Transpower, and provides training for all Transpower lines and sub-station staff, its service providers and sub-contractors. Transpower requires that all of its service providers and contractors have competent staff. Transpower Grid Skills is a monopoly and receives no government funding. There are two sets of competencies required within the organisation: lines and sub-stations. There are national qualifications available for lines work, but not for sub-stations, although work on this is being undertaken through Connexis, the relevant industry training organisation. Learners are all in employment. Those undertaking the Earthed Line Mechanic 1 and Earthed Line Mechanic 2 courses are funded for the training by their employer, Transpower. Recent significant changes: Previous quality assurance history: Since the most recent external evaluation and review (EER) in 2011 the Grid Skills team has been relocated from the Omaka training facility in Blenheim to Transpower s three regional offices in Auckland, Palmerston North and Christchurch. This allows the training team to work in the same office and environment as the business the training is provided for. This location allows feedback to be received from the maintenance managers and project managers on the effectiveness of the training in the workforce. Transpower Grid Skills has undergone a recent restructure which has led to staff and structural changes. At the most recent EER in 2011, Transpower Grid Skills was found to be Highly Confident in educational performance and Confident in capability in self-assessment. External moderation results from Connexis show that assessment materials and assessor judgments are at the 4

5 national standard. 2. Scope of external evaluation and review The focus areas selected were the mandatory focus area of governance, management and strategy, and the Earthed Line Mechanic 1 and Earthed Line Mechanic 2 courses, also called compliance training. These courses are delivered over two years and are recognised by Connexis as meeting a major part of the National Certificate in Electricity Supply (Line Mechanism Transmission). Transpower Grid Skills is a small training provider, and the two focus areas were sufficient to give the evaluators an accurate representation of the whole organisation. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. The evaluation team consisted of two evaluators who visited the Palmerston North head office of Transpower Grid Skills over 1.5 days. The evaluators met with the Grid Skills manager, technical training lead, trainer and verification lead, Grid Skills delivery team leader, Grid Skills delivery analyst and the Grid Skills delivery coordinator. The evaluators also visited the Bunnythorpe site where they spoke with a small group of learners and their trainer and viewed the on-site training facilities. After the EER on-site visit, the lead evaluator spoke with a sample of learners and external stakeholders, including the national operations manager (Electrical/Telco) at Connexis. The evaluators also sighted a range of documentation including the self-assessment, the quality management system, feedback from learners and trainers, minutes of the Technical Training Governance Board and the Technical Advisory Forums, a list of trainers and their qualifications, the strategic plan, reviews of training manuals and comprehensive documentation around the recent restructure and the competency training strategy. 5

6 Summary of Results Statement of confidence on educational performance NZQA is Highly Confident in the educational performance of Transpower New Zealand Ltd trading as Transpower Grid Skills. Key reasons for this include the following: Compliance and refresher training has high learner numbers and achievement has been very high over the last three years, with most learners achieving (refer findings 1.1). Since 2012 Grid Skills has worked more collaboratively with service providers to ensure that training is more responsive to current and future needs. This has required reviewing and updating assessment materials and a planned schedule of training to proactively meet the needs of learners and their employers (refer findings 1.3). All trainers are well-qualified practitioners with many years of industry experience. Trainers performance is observed regularly and upskilling of trainers is supported by management (refer findings 1.4). The support offered to learners is appropriate for short courses, often lasting only several days. Employers are always informed of learner progress, and reasons for not achieving are discussed with the learner and the employer. Opportunities to re-sit assessments are offered where appropriate (refer findings 1.5). Genuine consultation with, and feedback from, a wide range of stakeholders, including learners, has led to training that is more flexible, more consistent and informed by current training practices (refer findings 1.6). 6

7 Statement of confidence on capability in self-assessment NZQA is Highly Confident in the capability in self-assessment of Transpower New Zealand Ltd trading as Transpower Grid Skills. Key reasons for this include the following: Grid Skills staff have successfully used their combined skills, knowledge and experience to engage with and encourage wide consultation across the electricity supply industry. This has led to respectful, collaborative relationships and the development of shared goals that have informed Grid Skills restructure. Effective governance structures have been developed to respond to industry needs. They comprise the Technical Training Governance Board which provides strategic guidance with input from all parts of Transpower, service providers and Connexis, and Technical Advisory Forums which provide the specific technical competency requirements, to ensure the workforce remains current with technological and other changes. A comprehensive analysis of the needs of all stakeholders informed the review of Grid Skills processes and practices. The outcomes included an updated quality management system, new staff appointments, clarification of roles and a new organisational structure effective from October Grid Skills has worked closely with Connexis in the review and future development of electricity supply qualifications that are aligned to the longer-term skills required across the industry, through active participation in the Targeted Review of Qualifications. 7

8 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Achievement in compliance courses (often lasting only two days) in the years has been consistently close to 100%, which is not uncommon for courses of this level and duration. Grid Skills learners are all in employment and their training is required to meet ongoing competency requirements (including refresher training every two years) for their jobs. Table 1. Achievement rates for compliance courses, Compliance courses Achieved 2,627 2,556 2,698 Not achieved Over the years a total of 78 learners did not achieve. Grid Skills selfassessment shows that learners who did not achieve were distributed across a range of courses (23 in total) and trainers (28 in total). Over a three-year period there were six learners who did not achieve and who had the same tutor. This was the greatest number of non-achieved results for any one trainer (from a total of about 40 trainers). The other learners who did not achieve were spread across different courses and different trainers, and Grid Skills have not yet identified any common cause for this outcome. The Grid Skills team responds directly to any learner feedback on these courses, and refines delivery and material, where appropriate. The learner management system has been recently overhauled and a new online system introduced in The new system will enable Grid Skills staff to extract data and report on learner achievement more efficiently than previously. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 8

9 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. The competency framework developed by Grid Skills, in consultation with stakeholders, maps out requirements for individual jobs and identifies clear pathways for workers to develop skills, qualifications and experience that can improve the quality and safety of industry performance and enable them to carry out project and maintenance work. It also identifies industry-wide gaps in certain roles, and future skills shortages. The success of the framework is measured through key performance indicators, such as the number of competency pathways introduced and/or the proportion of the workforce who are competent to maintain new equipment before it is commissioned. Early indications are that the competency framework contributes to identifying skill shortages, helps individuals to map out career options, and assists Transpower to identify future training requirements. The overall value of training outcomes is measured by Grid Skills from feedback by learners and their employers about whether the training has led to greater competence in the workplace. In 2011 and 2012 the training delivery model was reassessed and new goals developed. Feedback indicates that the training has become more proactive (fewer learners waiting for a course), better understood and appreciated by the wider Transpower organisation, and more integrated with the training and skills development of electricity transmission workers across the wider industry. Other evidence of outcomes of value includes learners skills being sought overseas, the Transmission Trainer of the Year awards that acknowledge excellent performance, and longer-serving workers skills and experience being recognised through recognition of current competency and the award of formal qualifications. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Grid Skills is responsible and accountable to Transpower to provide training that closely matches industry needs. It has successfully achieved this by undertaking consultation with stakeholders over a number of years. This has required excellent relationship-building skills and patience to ensure agreement between various stakeholders, and has resulted in an agreed vision and strategy to meet that vision. 9

10 As a result of the agreed vision and strategy, Grid Skills has reviewed all aspects of their training, including relocating to three regional offices to be closer to the Transpower business team, employing new staff and instituting more efficient administrative systems involving a new learning management system and rewriting the quality management system. Grid Skills has also researched electricity transmission industries in other countries to look for models of good practice and had active involvement through Connexis with the Targeted Review of Qualifications. The shared goals and values in the partnership between Transpower and Grid Skills provides a balance between the theory learnt in training and in practice, and verification of good practice on site. Grid Skills self-assessment indicates that training has led to improvements in learner attitudes and behaviours, particularly around issues of safety, and has increased their competence on the job. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Grid Skills trainers have many years experience in the electricity transmission skills industry. The trainers performance is tracked through evaluations from learners, and any issues that arise are dealt with promptly. Formal performance reviews of trainers occur every two years and feedback is provided face-to-face. Some team teaching and peer review also occur. The competency model and pathways for training include the competencies required. They are comprehensive and include safety and health, technical leadership, adult education and training skills and knowledge of assessment and moderation. Trainers meet face-to-face each year to discuss training issues, review moderation of assessment materials and assessors judgements, and prepare for the following year s training. Trainers are well supported by Grid Skills to upskill and undertake training opportunities. Trainers are spread across New Zealand, so they have limited opportunities to meet face-to-face. Trainers were keen to meet more often to share common concerns and provide mutual support. Learners are given individual attention and assistance, including the opportunity to be assessed verbally if literacy and numeracy issues are identified. They are also given re-sit opportunities to pass assessments. There is close contact between the trainers and their managers as to progress made, and occasionally a learner realises that they may not be in the best job for them and is counselled to pursue other options. Most training groups are small, with six to eight learners, and satisfaction levels with the training are high. Monthly learner evaluations and learners comments confirmed that the trainers are well informed, very experienced and are highly responsive to learners learning needs. The learners commented 10

11 that some of the best learning comes from the stories the trainers can relate on the course, explaining how the trainers dealt with various situations in the field. Some learners considered that they were unsuccessful at school and were pleasantly surprised when they could manage the theory involved in the training, a win-win for learners, trainers, employers and the wider community. Learners who were interviewed also welcomed the opportunities to gain qualifications across different areas of the electricity transmission industry and had some understanding of the function of the competency framework in this regard. Grid Skills has also developed an internal moderation strategy to assist with consistency of assessment and has employed instructional designers to ensure training and assessment material is written consistently. Pre-assessment and post-assessment moderation indicates that assessments are at the correct level and that the marking is fair and consistent. One issue identified by Grid Skills self-assessment was that evaluations of learner performance, which are currently collected by the trainers, needed to be more confidential to encourage learners to comment more frankly. The evaluation form provides contact details of the Grid Skills manager to enable the learner to contact this person directly, but the evaluators understood from Grid Skills that this option was rarely used. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Guidance and support is appropriate for learners already in employment who are undertaking training to comply with job requirements. Grid Skills trainers work hard to ensure the learning is provided in an accessible and relevant manner, including the provision of training equipment that mimics real-life equipment. The need to undertake regular refresher training allows learners to apply the knowledge and skills gained at training, on an ongoing basis, in their workplaces. Issues around workplace behaviour and competency are Transpower s responsibility. However, as noted in Findings 1.4 above, Grid Skills works with Transpower to assist individuals to find other options, if it becomes apparent in the training context that the learner will not be able to meet training requirements successfully. While theoretically sound, the strategy to develop blended training programmes across a range of modes, to appeal to a wide age range and different learning styles, was not met with great enthusiasm by the small group of learners interviewed. These learners enjoyed the block course system as it offered a break from the work routine, encouraged socialising with people from other parts of the country, and allowed them to concentrate on study without the distraction of work 11

12 commitments. The evaluators provided this feedback to the Grid Skills manager who was very interested and appreciative and indicated that the issue would be followed up. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. The outcome of the change process at Grid Skills means that Transpower is some way towards meeting its vision: to provide a qualified and competent workforce that meets current and future electricity transmission industry needs. It has achieved this through a governance and management structure that provides guidance for the electricity transmission industry on required competencies and the training and qualifications required currently, and to meet future industry demands. This guidance has involved two distinct groups: the Technical Training Governance Board and the Technical Advisory Forums. The board has stakeholders from Transpower s core business, service providers and Connexis. The forums provide the technical competency requirements in specific technical areas. Members are subject experts from the electricity transmission industry and Connexis. The needs identified have led to the development of the competency framework, which guides training programmes aligned to Transpower s service specifications. It has also required extensive consultation within Transpower and with external service providers: Connexis, the Electrical Workers Registration Board and the Electricity Engineers Association. Feedback is captured, actioned and responded to by the Grid Skills team. The competency framework, Transpower service specifications and on-the-job training provide the skills required in the workforce. The mapping of generic competencies, identification of desired learner core behaviours, and mapping of unit standards to programmes put Grid Skills in a very strong position regarding the post-targeted Review of Qualifications development of qualifications. The clear identification by Grid Skills of its learners needs ensures that these needs will not be overlooked and puts it in a strong position to influence the qualifications that are developed. It will take several years for all the changes and improvements Grid Skills has initiated across technical training to be completed. However, the evaluation team is convinced that Grid Skills has demonstrated, through its comprehensive selfassessment, that it has the respect of stakeholders (including trainers and learners). The organisation also has competent personnel, suitable processes and procedures (including very clear, comprehensive documentation), and the ability to continue to access the resources required to guide these developments. 12

13 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management and strategy The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 2.2 Focus area: Compliance programmes The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 13

14 Recommendations NZQA recommends that Transpower Grid Skills: Identify the best way/s to collect evaluations from learners on trainer performance to maximise opportunities for learner feedback. Investigate, with Transpower, the advantages and disadvantages to learners of proposed blended learning approaches. 14

15 Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( The External Evaluation and Review (EER) Rules 2013 are available at while information about the conduct and methodology for external evaluation and review can be found at 15

16 NZQA Ph E qaadmin@nzqa.govt.nz 16