Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review M C Consultancies Services Associates Limited trading as M C Training Highly Confident in educational performance Confident in capability in self-assessment Date of report: 31 August 2015

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review... 5 Summary of Results... 6 Findings... 8 Recommendations Appendix MoE Number: 7617 NZQA Reference: C18215 Date of EER visit: 3 February

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: M C Consultancies Services Associates Limited trading as M C Training Private training establishment (PTE) First registered: 16 August 2003 Location: Delivery sites: Courses currently delivered: Code of Practice signatory: 14 Aratonga Avenue, Greenlane, Auckland M C Training delivers training nationally at a range of sites. M C Training delivers a range of short courses in temporary traffic control, supervision and management; occupational health and safety; and working at heights and in confined spaces. M C Training is not a signatory and has no international students. Number of students: Domestic: 433 trainees in 2014 Number of staff: Scope of active accreditation: 1.5 full-time equivalents (one full-time owner/director/trainer and one part-time administrator/liaison person) M C Training does not assess against unit standards, although it has the consents to do so. These are in the domains of Temporary Traffic Control and Management and Occupational Health and Safety (to level 5), and First Aid and Interpersonal Communications (to level 2). M C Training has NZQA training scheme approval for Temporary Traffic Management Operate as a 3

4 Traffic Controller (level 3). It also has consents to assess unit standards related to firefighting, cranes, forklifts, and heights and confined spaces (up to level 4). For details, see: Distinctive characteristics: Recent significant changes: Previous quality assurance history: M C Training delivers mainly one and two-day and refresher employer-funded courses throughout New Zealand. M C Training delivers its Working on the Roads courses under the auspices of the New Zealand Transport Agency (NZTA) which has a Code of Practice for related traffic management. M C Training s follow-up of traffic management and occupational safety training in particular is sometimes connected to the consultancy advice and audits provided by M C Consultancies Services Associates Limited. NA At the previous external evaluation and review (EER) in 2011, NZQA was Highly Confident in M C Training s educational performance and Confident in its capability in self-assessment. M C Training meets NZTA s quality assurance requirements for training delivery, assessment and reporting. 2. Scope of external evaluation and review The following courses focus area was selected: Working on the Road courses including NZTA Working on the Road (Basic Traffic Controller), Site Traffic Management Supervisor, and NZQA Training Scheme Traffic Controller courses as appropriate The NZTA Working on the Road Basic Traffic Controller and Site Traffic Management Supervisor one and two-day courses and refresher courses were selected because they had the highest demand and highest number of attendees in 2014 (153, 75, and 53 respectively). The Temporary Traffic Management Operate as a Traffic Controller (level 3) course was also included as a focus area because it was approved as a training scheme by NZQA and is at a higher level, but its attendee numbers were in fact significantly lower. The PTE was advised in advance of the EER visit that the one and two-day and refresher courses would be the main focus of the EER, with particular, but not 4

5 exclusive, reference to Working on the Road courses. The PTE accepted and agreed to this approach, which included other areas such as occupational safety and enabled the EER to provide an overall and fair representative view of M C Training, how well it was performing, and how MC Training knows this. M C Training does not train or assess First Aid and refers trainees to other providers for this. In accordance with NZQA policy, the scope also included the following mandatory focus area: Governance, management and strategy. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. Two evaluators conducted the EER over one day at the Auckland office. They interviewed the owner/director/trainer and the administrator/liaison person. Subsequent to the visit, the evaluators spoke by telephone to trainees and stakeholders including company employers, managers, NZTA and Connexis (a merger of the Electricity Supply Industry Training Organisation (ESITO) and InfraTrain). The evaluators sighted a range of documents, which included evaluations, training resources, assessments, attendee data spreadsheets and summaries, analyses, and a technological tool under development. 5

6 Summary of Results Statement of confidence on educational performance NZQA is Highly Confident in the educational performance of M C Consultancies Services Associates Limited trading as M C Training. M C Training has very high achievement rates in the Working on the Roads courses (traffic control and management and occupational safety). At least 95 per cent of trainees successfully complete the courses by passing test assessments during and at the end of the courses. These results have been sustained over several years since at least 2010/2011. M C Training follows up trainees who are not successful, explains why they are not to their employer, and helps support them to achieve later. Courses are delivered effectively, engage the trainees, and assess their knowledge and skills during and throughout each course. Courses are adapted to align to the workplaces of those who are attending. Each course usually intentionally includes trainees from different workplaces who learn from the similarities and differences in their work practices. Trainees and their employers report that attendees are able to apply newly acquired knowledge and skills immediately upon their return to work. Relevant and up-to-date resource manuals support attendees who take them with them when they complete the courses and refer to them on site when they need to. Trainees and employers report raised awareness, knowledge and understanding as well as increased confidence and improved practice on site after completing the courses. Consultancy and audit visits reinforce the learnings gained in training and applied in practice in the workplace. The administrator liaison person and/or owner/director/trainer gather anecdotal feedback, respond to any questions from attendees or employers, provide advice and/or audit workplace practices, and point out future training possibilities when they visit. Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of M C Consultancies Services Associates Limited trading as M C Training. M C Training has collected detailed training data since at least 2010/2011. These include the number of attendees on courses, details of their employers, and individual achievement records and course locations. This enables M C Training to summarise individual course provision and trainee completions, and demand for different courses, total attendee numbers and trends. Data shows that total trainee numbers have dropped since 2012, but also that repeat business has been steady and numbers consistent in 2013 and There is limited evidence of changes that have been made as the result of such data analysis. However, the 6

7 owner/director/trainer is now considering the possibility of increasing the extent of the training in light of the administration and liaison role being carried out by the second staff member. The completion of the development and roll-out of an information technology application will enable easier capture of trainee and employer information, better support for trainees and employers in real time on site, and further free up the owner/director/trainer to do more training. The director/owner/trainer gains a wealth of informal and anecdotal feedback based on observations made as a trainer during the training. This is valid and useful feedback from both trainees and employers who also often attend the training, and is supplemented by visits made by the owner/director/trainer as a consultant/auditor and the administrator liaison person after the training. Trainee evaluations have been collated annually from at least 2011, and analysis of the responses to the same questions each year has enabled M C Training to identify satisfaction levels, similarities, differences and trends in trainee satisfaction course by course. Because the satisfaction levels are consistently positive, and variations from course to course and year to year are so slight, the value of the data is confined to confirming consistently high satisfaction. There is limited evidence of direct use of any satisfaction data variations to make changes. Completion of evaluations is voluntary and M C Training does what it can to encourage trainees to complete them before they leave, but this does not often happen. In an attempt to improve evaluation completion rates, evaluations sheets are now included in the manual that attendees take away with them, although M C Training expects that response rates will most likely remain quite low. According to M C Training, attendee comments in addition to merely ticking boxes are rare and are acted on as appropriate case by case. The post-training visits and the application, which are yet to be rolled out in full, will provide the opportunity to better capture all feedback and further strengthen selfassessment, which may contribute to further improving the quality and value of the training. M C Training could also strengthen feedback in areas such as literacy and numeracy and different learning styles and current trainee evaluations. While M C Training has the capability to assess unit standards, such assessment is not mandatory. M C Training argues that it is not currently prepared to carry out assessments in a competitive environment where training and assessment practice is driven by demand and cost rather than allowing for sufficient time and quality in some cases. M C Training is open to reconsidering this situation, and may have to eventually. Not assessing unit standards also deprives M C Training of an additional layer of external feedback on assessment and the training it is based on. 7

8 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Trainee achievement in Working on the Road courses has been consistently high since at least 2010/2011. At least 95 per cent of trainees complete the courses and pass the quiz assessments provided or verified by NZTA throughout the courses. Trainees need to gain 80 per cent to pass. The assessment validates trainee knowledge and enables trainees to register with NZTA to work on the roads with specified levels of responsibility. The high level of achievement is confirmed by M C Training s database and training records which are verified by NZTA. Occupational safety trainee achievement is similarly high and externally verified by trainees and employers. Because M C Training does not assess unit standards, it is not currently quality assured through external moderation of assessment by the industry training organisation. Visits by the administrator liaison person and consultancy and/or audit visits by the trainer provide some important additional quality assurance of the implementation of what is learned at the training. The quantitative data recorded by M C Training each year is analysed and identifies course demand and achievement trends, but there is little evidence of how this data is used to make changes beyond scheduling sufficient courses and deciding where and when they might be best offered and delivered. 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Trainees and key stakeholders such as employers and NZTA confirm that course attendees learn useful knowledge and skills that they are able to use immediately upon their return to their workplace. This enables them to carry out safe practices themselves and inform, control and supervise others. Awareness of practices in a range of different work environments helps reinforce their own learning and practice. The trainer, trainees and employers also observe that successful trainees gain confidence. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 8

9 Trainees take the fully up-to-date manuals back from the training to their workplaces. They are then able to refer to them which is useful to reinforce their learning, knowledge of requirements and practices on site. Follow-up visits by the administrator liaison person and/or trainer, who sometimes audits practice as part of the M C Consultancy Services, support the trainee and employer and help determine the success of the training. The visits also enable MC Training to provide practical and business advice which includes providing information about future training and refresher courses. Completion and increased use of the technological application currently being developed will support training and practice. Such visits also inform M C Training about what is going on in the sector beyond the valuable updates received from NZTA and other external sources. M C Training gains valuable informal and anecdotal feedback. It is conscious of making improvements through its own self-assessment and is considering how such feedback could be better captured. This could contribute to identifying any additional changes that could lead to further improving the value of the outcomes of the training for trainees, employers and other key stakeholders. Review of the trainee evaluations with regard to response rates, the questions asked, and ways of obtaining more trainee comments would add to the value of the feedback M C Training gathers. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. M C Training s Working on the Roads courses have maintained their relevance to trainees, their employers and other key stakeholders such as NZTA. The courses comply with the requirements of NZTA which include using the most recent legislated policies, procedures and practices and NZTA s own Code of Practice and quiz assessments as the basis of the training and resources. Good practice enables trainees to take back to the workplace the manuals used at training. Trainees report that they refer to their manuals to inform their traffic management plans or guide their occupational safety practices. Successful trainees attend refresher courses as required. They can also progress from the level 1 course to a level 3 course, and some do. The refresher courses and progression to higherlevel courses are important for working on the roads effectively and safely. They are based on practical experience and reinforce learning and ensure it is up to date the turnover rate of traffic controllers and supervisors is quite high, with such employment typically lasting about two years. The short courses are planned and structured for the benefit of the trainees and to match their needs as effectively as possible. PowerPoint presentations, photographs, videos, diagrams and manuals are used flexibly and effectively. Course and resource planning takes into account the needs of each individual 9

10 organisation. These are balanced by the deliberate inclusion of trainees from different organisations which helps diversify and add to the stories and experiences the trainer and attendees provide, which enhance learning. This does not compromise quality as course numbers are usually relatively small (and training does not take place if numbers fall to three or less). M C Training is usually but not always able to exercise the flexibility such a small PTE can provide as to when, where, why and how it delivers training. The process of signup, attendance, being assessed, follow-up if unsuccessful, and notification of future courses is efficiently and effectively administered by the administrator liaison person in consultation with the trainer. The use of an M C Training information technology application will further enhance these processes and help provide the trainer with more time and flexibility for training. M C Training attendees present a range of previous educational experiences and learning styles and abilities which challenge the trainer, particularly within the constraints of one and two-day short courses. The proportion of trainees who do not successfully complete their course is small, and M C Training works with unsuccessful trainees and their employers post-training to enable the trainees to succeed later wherever practicable. Notwithstanding, M C Training may find it helpful to further consider how resources and delivery could help address the challenges of varied previous learning experiences and successes, different preferred learning styles, the levels of literacy and numeracy albeit within the context of the short courses. Enhanced trainee, employer and other key stakeholder feedback and comments would also contribute to this process. M C Training used to conduct employer and key stakeholder surveys but it has abandoned them, and they may not be the most effective and suitable way for a small PTE to self-assess. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Trainees are consistently satisfied with the quality of the training, its effectiveness and the approachability of the very experienced trainer. Trainees find that the trainer engages them in learning and report that relevant examples of practice, shared discussions based on attendees different work environments, and question and answer sessions support their learning well. The current and relevant manuals are taken back and used at the worksite, which reinforces learning and practice. Notwithstanding, while the manuals are rich in content, it is possible some trainees would benefit if the presentation was improved to accommodate the needs of all trainees, including those with literacy and numeracy challenges. Post-training support for trainees who are not successful with quiz and other assessments throughout the training the first time round, and their employers, is positive practice which is appreciated. 10

11 The effectiveness of the teaching and learning is reinforced by the fact that successful trainees are able to put into practice their learning immediately upon their return to the workplace and worksite. It is also enhanced by the follow-up visits made by the administrator liaison person and/or the owner/director/trainer to clients as part of the consultancy part of M C Training visits include providing advice, getting feedback from trainees and employers, and audits of safe practice. Improvements to programme delivery, content, resources and the effectiveness of the teaching are made based on the trainer s observations, trainee evaluations and informal and anecdotal feedback gathered at training and during client visits. As mentioned, while the trainee evaluations confirm that most trainees are highly satisfied with their training overall, a higher response rate, reviewed questions and encouragement to make more comments may help identify any further changes that could be made. In addition, improved capture of employer and stakeholder feedback would also contribute to self-assessment for improvements. Should M C Training decide to assess standards in the future, external moderation would provide another source of feedback which would contribute to self-assessment. As far as M C Training is concerned, the main issues to consider in regard to assessing standards are whether it is mandatory, the influence of market forces, and how to maintain the integrity of such assessment balanced against time and cost. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. The most effective guidance and support, given that the courses are only one or two days in duration, is provided by the trainer. The M C Training evaluations and the trainees and stakeholders the evaluators spoke to confirmed this. The trainer engages trainees in their learning, uses relevant examples provided by the trainer and attendees, encourages questions, and helps those who need it most one-toone. When an attendee cannot answer the quiz or part of the quiz successfully at a first or second attempt, the trainer follows up with their employer, explains why the attendee has not passed, and provides the attendee and their employer supervisor with support and additional resources. This often results in the trainee eventually successfully completing the assessments. Some pre-course information about trainees is completed on the enrolment form, but how helpful this is depends on how comprehensively the form is completed. Any barriers to learning, such as literacy or numeracy (the trainer considers that numeracy is a more common challenge than literacy), or other barriers to learning are often not identified until trainees arrive at the course. Then the trainer provides more one-to-one support for those who require it and ensures that the presentations, situations and activities reflect and meet trainees needs. 11

12 The level of guidance and support, which includes initial information, the application/enrolment process, pre-course information, follow-up, and the roll-out of the M C Training application, is very appropriate considering the short length of the courses. The only area it could be strengthened is around addressing literacy and numeracy and educational experience, and the learning style challenges provided by some trainees, by obtaining more information about this as a starting point. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. The PTE s training accounts for approximately 65 per cent of the total business and work of M C Consultancies Services Associates Limited. The remaining 35 per cent includes consultancy and providing on-site occupational safety and hazard planning, advice and auditing. More recently, M C Consultancies has identified and begun consultancy work in accident reporting and its role in relation to improving health and safety practice and accident prevention. The connection between the small PTE delivering mainly theory short courses and the consultancy work is a strength. It connects the training with insight into the application of the knowledge and skills on site, and the value and impact of the training, which in turn also identifies further training opportunities. It also provides ongoing understanding of sector business practices and changes, and the health of the sector. The leadership of M C Training is effective and the manuals are regularly updated and relevant resources provided to support training as well as the application of knowledge and skills gained during training when trainees return to their workplace. M C Training s capability to anticipate and respond to change and balance innovation and continuity should help it to continue to do so. External advice and support with regard to innovations such as the application of information technology may also support the leadership s effectiveness. M C Training s future purpose and direction are yet to be determined. The owner/director/trainer has identified that the balance of the training and consultancy work may change as it maintains consistent but lower training numbers than previously. The owner/director/trainer has also identified that the current role and practice of the administrator liaison person and introduction of the technology application which has the potential to provide on-site and on-job support as well as training records and related information may give the owner/director/trainer more time to consider and implement further integration of the mainly theory training with the practical application. In addition, M C Training is still open to the possibility of assessing standards and realises that this will depend on several factors: market forces and demand; ability to compete with other providers based 12

13 on time, cost and maintaining the quality and integrity of such assessment practices; and any changes in the expectations and requirements of key external stakeholders such as NZTA and employers. To some extent, M C Training is waiting to see what changes in practice and assessment may be mandated externally. Its self-assessment is effective, useful and appropriate for a small organisation. However, it could be strengthened as previously mentioned. Strengthened processes, analysis and use of information, based realistically on what M C Training already does, would at least verify the valid and valuable informal and anecdotal feedback and may identify and contribute to further changes and improvements to benefit M C Training, its trainees, employers and key stakeholders. 13

14 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management and strategy The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 2.2 Focus area: One, two-day and refresher courses with particular but not exclusive reference to Working on the Road courses including NZTA Working on the Road (Basic TC), refresher and NZQA Training Scheme The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 14

15 Recommendations NZQA recommends that M C Training: Based on what is realistic for a small organisation, further develop selfassessment, including trainee evaluations, employer and other key stakeholder feedback, to provide even better information to help identify any further changes it could make to the volume and quality of its training. Consider more future scanning, and the possibility of independent external advice, to help determine the organisation s future purpose and direction. Further rationalise the roles of owner/director/trainer and administration/liaison person to maximise the benefits of the strengths of both in current and future operations. 15

16 Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( The External Evaluation and Review (EER) Rules 2013 are available at while information about the conduct and methodology for external evaluation and review can be found at NZQA Ph E qaadmin@nzqa.govt.nz 16