Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review New Zealand Institute of Management trading as Institute of Management New Zealand Not Yet Confident in educational performance Not Yet Confident in capability in self-assessment Date of report: 15 January 2016

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review... 6 Summary of Results... 7 Findings... 9 Recommendations Appendix MoE Number: 7479 NZQA Reference: C19449 Dates of EER visit: 22 and 23 September

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: New Zealand Institute of Management trading as Institute of Management New Zealand (IMNZ) Private training establishment (PTE) 4th Floor, Phillip Fox Tower, 209 Queen Street, Auckland As above and Level 7, Lumley House, 3-11 Hunter Street, Wellington First registered: 22 March 2002 Courses currently delivered: National Certificate in Business (Level 3) National Certificate in Business (Level 4) National Certificate in Adult Education and Training (Level 4) NZIM Diploma in Frontline Management (Level 5) NZIM Diploma in Project Management (Level 5) NZIM Diploma in Management (Level 5) NZIM Diploma in Management (Advanced) (Level 6) NZIM Diploma in Health and Safety Management (Level 6) NZIM Diploma in Supply Chain 3

4 Management (Level 6) NZIM Diploma in Managerial Excellence in Engineering and Construction (Level 6) Code of Practice signatory: N/A Number of students: 179 learners in Māori learners in 2014 (41 over the last four years) Nil Pasifika learners in 2014 (six over last four years) Number of staff: Scope of consent to assess: Distinctive characteristics: Ten full-time and three part-time staff, 17 contract facilitators IMNZ is a not-for-profit organisation established in 1946 to equip and support management professionals to build their capability and understanding of their management and leadership abilities. It is a membership-based organisation with a voluntary elected Board. IMNZ offers a range of services, networking opportunities, seminars, short courses, and formal education and training programmes. Learners are all employed in related management and leadership roles and largely study at a distance from IMNZ, but also have short but regular face-to-face sessions. IMNZ also accredits other tertiary education organisations to deliver its programmes, which IMNZ quality assures. Recent significant changes: In the past months IMNZ has undertaken an organisational review and subsequently restructured its operations. In this period it has gained the input of a number of external specialist organisations. This has resulted in a number of changes designed to improve the organisation for future success, including reducing the number of qualifications offered; creating a national education manager role (currently vacant); refreshing the brand and look and feel of its delivery material; changing its approach to assessment from many assessors to a single independent assessor and a 4

5 single moderator; and reducing the regions in which it operates, now including Auckland and Wellington only. IMNZ is currently establishing an arrangement with Open Polytechnic for delivery of IMNZ programmes via the polytechnic s online distance learning platform. Previous quality assurance history: IMNZ was most recently evaluated by NZQA in 2011, when NZQA was Confident in both the educational performance and capability in selfassessment of the organisation. IMNZ is engaging, as required, with NZQA Tertiary Assessment and Moderation as the standardsetting body for the external moderation of assessment material and assessors judgements. Over the last three years a number of moderated unit standards identified areas requiring improvements to assessment material and to assessor judgements. While there have been a number of concerns raised over this time, IMNZ has more recently made changes to improve the quality and consistency of its assessment practices. 2. Scope of external evaluation and review This evaluation included the mandatory focus area of governance, management and strategy, which provides an overview of the strategic and operational levels of the organisation. The other two focus areas were the certificate level 3 and 4 programmes and the diploma level 5 and 6 programmes, as listed above. These focus areas provided a review across the structured training offering, and included approximately half the training and development delivered. The short courses delivered by IMNZ were not included in the evaluation because they are not formally assessed programmes of learning. 5

6 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. A team of two evaluators visited the Wellington site over two full days. Prior to the site visit, the lead evaluator met with the general manager strategy and performance, and IMNZ supplied the evaluation team with a self-assessment summary and wide range of supporting appendices, which contributed to the development of the evaluation scope and plan of enquiry. During the site visit, the evaluators interviewed the chair of the Board face to face and other Board members by phone. The evaluators interviewed the management team, learning development personnel, facilitators, students, the external assessor, external moderator, and client company representatives. A wide range of the organisation s documents, records, meeting minutes and database reports, and correspondence from the Tertiary Assessment and Moderation unit of NZQA regarding moderation of students assessments, were reviewed. Overall, this material contributed to an appropriate and accurate basis for this evaluation. 6

7 Summary of Results Statement of confidence on educational performance NZQA is Not Yet Confident in the educational performance of the Institute of Management New Zealand. Statement of confidence on capability in self-assessment NZQA is Not Yet Confident in the capability in self-assessment of Institute of Management New Zealand. The following key factors have contributed to the two not yet confident judgements reached: The evaluation team heard directly from management, learners and other stakeholders about good support and follow-up with students to ensure they complete their qualifications, high pass rates, considerable value being gained from studying with IMNZ, and graduates being promoted in their workplaces. This information provided a solid picture supporting an understanding of the quality of education provision at IMNZ. However, this data is currently not being fully collected, collated or used in the organisation s self-assessment to draw overall conclusions or lead to ongoing improvements. IMNZ learners are achieving at high rates in some courses, but achievement varies somewhat, and self-assessment and review processes are less than comprehensive and complete to effectively demonstrate the reasons for uneven achievement. No firm plans were evident showing how uneven achievement was to be addressed. Achievement data year on year is not currently reviewed to establish patterns or trends across programmes or across programme levels. This is partly due to the organisation having incompatible databases at its two sites, making the extraction of data for self-assessment purposes particularly problematic. There is some uncertainty about the validity of achievement data. In past years and up until recently, assessment and moderation have been less than robust, with insufficient guidance provided to students and facilitators to ensure they were fully aware of the programme learning outcomes, or that assessment matched the learning outcomes. There have not been strong processes in place for the moderation of assessment tools prior to use, or of students scripts to fully ensure the quality of assessment processes or the validity of achievement rates. These issues with assessment and moderation have been recognised by management and appropriate steps have been taken to strengthen related 7

8 education provision at IMNZ. The organisation is currently in the process of appointing a national education manager to take overall responsibility for education matters, to ensure the implementation or embedding of key processes to oversee expected improvements. There are still vulnerabilities to address; for example, there is only one contracted assessor covering all subject areas except finance. Management is aware of this issue and is planning to address it. IMNZ has been in existence since 1946, and has established a national reputation for providing management and leadership programmes. In recent years management identified a weakening position and has taken appropriate steps through advice from external consultants to revitalise the organisation and address the identified areas of weakness. The evaluation team considers that the steps undertaken by IMNZ are appropriate and are likely to result in the organisation being in a stronger position to deliver quality education and to build capability in self-assessment. 8

9 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Adequate. The rating for capability in self-assessment for this key evaluation question is Adequate. The organisation was able to show overall achievement rates spanning 2012 to the time of this evaluation (see Table 1). The achievement rates across all programmes range between 46 per cent and 100 per cent, with some evidence to show that many students achieve with high marks. While these figures show a reasonable level of achievement, the organisation does not track patterns or trends across programmes, tutors or programme levels over time, or benchmark the achievements against other similar tertiary education organisations. A key issue with this lack of systematic tracking is that the organisation s understanding of factors affecting achievement is variable and therefore also affects how the organisation responds. For example, the organisation s selfassessment summary identified patterns of lower achievement in some certificate programmes which were attributable to programme flexibility. However, revised data presented at the site visit showed these trends instead related to only a couple of higher-level programmes. Identifying trends may help the organisation s understanding of student achievement patterns to identify areas of strength to build on, or identify areas of poor achievement to investigate causes and put remedies in place. Achievement data is difficult to extract due to the organisation having two incompatible databases. However, there are plans to rectify this. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 9

10 Table 1. Achievement Certificate-level programmes Diploma-level programmes Overall enrolment numbers and achievement rates All students enrolled All graduated % 63% 74% 71% Māori students enrolled Māori graduated % 2 %NA (small numbers) 36 92% 38 86% Pasifika students enrolled Pasifika graduated % 0 0% 1 %NA (small numbers) 1 %NA (small numbers) The organisation s student surveys seen by the evaluators indicate a high level of satisfaction, generally above 4 on a 1-5 scale, and most students comments are complimentary. IMNZ indicated that this was the predominant result across learners and programmes. While IMNZ did not have a summary of this overall picture, students interviewed at this evaluation confirmed this level of satisfaction. As shown in Table 1, Māori are achieving well. Pasifika numbers are too small over the 2012 to 2015 years to make any judgements on Pasifika achievement, and enrolment numbers are very small in some programmes, with none in 2015, distorting some programme percentage achievement rates. However, achievement rates indicate that Māori and Pasifika students are appropriately supported and have their needs meet. Weaknesses in assessment and the moderation of assessment and persistent gaps in national moderation in recent years (see Findings 1.4), raise questions about the validity of student achievement. While much work has gone into addressing this area, changes are quite recent. Therefore, much of the achievement data predates revised assessment and moderation practices. Organisational review of achievement is not yet well planned or embedded, but the planned single database is likely to help improve this. 2 Figures as supplied by IMNZ. No year-by-year analysis is available. 10

11 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Adequate. The organisation has a good level of knowledge of the value of its programmes to the learners and client companies. This is partly a result of being established since 1946, but also because current knowledge is maintained through ongoing, close contact with learners and corporate clients who bring repeat business. Most learners are employed in management or leadership positions, and engage in study with IMNZ to improve their career opportunities and business knowledge. The organisation regularly surveys its learners, gaining consistently high satisfaction ratings and positive comments, as noted. Graduates are engaged about the impact of the training and the information summarised and presented at graduation. While insightful, this information is not collated across learners, programmes and over time. Processes to gain feedback from corporate clients is less systematic and less well documented than for learners. Staff interviewed at this evaluation showed an understanding of the value of study to learners and client companies. The evaluators consider that further formalising engagement and e-documentation of input from stakeholders on met needs may improve IMNZ s ability to identify and build on strengths or to close gaps. The evaluators interviews with learners and stakeholders identified examples of value added through studying with IMNZ, including learners improving their understanding and ability to apply management principles, with some learners being promoted to positions of increased responsibility. Many anecdotes were given of the value of studying while in employment and learners gaining confidence and being able to apply the theory learned in their current roles, as well as using current business activities as case studies, for example strategic and project planning. Personnel interviewed at this evaluation commented that they felt the organisation is particularly well positioned in the new environment of reviewed New Zealand qualifications to show evidence of meeting Graduate Profile Outcomes 3 from learners being in employment and applying their new knowledge and skills on a day-to-day basis. 3 This is a key component of the newly listed New Zealand qualifications; 11

12 While the evaluators found good evidence that learners are gaining value from study, the organisation s review of this is not yet well documented or the information consistently analysed and used to show evidence of ongoing improvement. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. The organisation s self-assessment is stronger in this area. This was evidenced by client company needs being well matched through close engagement with businesses to customise training, specifically addressing business and organisational culture needs. Client companies interviewed at this evaluation confirmed that IMNZ actively reviews how well it has matched their needs and makes changes as required. Programmes and activities are reviewed after their first year and every three years after that, which involves appropriate consultation with all stakeholders to ensure currency. Student survey data reviewed showed strong levels of satisfaction, with their needs being matched by the design and delivery of the programme and activities. This data is regularly collected, collated and analysed, and used for ongoing improvements to programmes and activities. Changes include, for example, renewing course material, including business examples, and incorporating client company language and specific company scenarios. However, there was some evidence of concern in the past about facilitators using material that was not up to date, or facilitator styles that were old school or tired. IMNZ management provided evidence that these matters were known and being addressed. Appropriate processes are used at enrolment to ensure only learners who are likely to succeed are enrolled. However, staff acknowledged that there are still some issues with their learners who are, as noted, in employment and enrolling in study but consequently finding themselves over-committed, and so not completing the study. This reduces overall achievement rates. The organisation is involved in and up to date with the current Targeted Review of Qualifications, and aware of the future need for developing new programmes to lead to the achievement of these qualifications. Overall, IMNZ has strong processes to ensure its programmes and activities match learners and other stakeholders needs. 12

13 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Adequate. The rating for capability in self-assessment for this key evaluation question is Adequate. While learners find the teaching to be effective, and learning materials are mainly fit for purpose and up to date, the primary concern is with the historic quality of assessment and the moderation of assessment. There have been gaps in matching assessments to learning outcomes, with not all facilitators having access to the programme design including the intended learning outcomes. This has resulted in occasions where facilitators have taught and assessed what they know, rather than strictly to the syllabus. In fairness, the evaluators acknowledge that there is clear evidence that learners have gained value and have been able to apply their learning in the workplace. The main issue is that there may have been occasions where the programme, as approved, has not been fully communicated or available to facilitators in all cases, so potentially not all learning outcomes may have been fully taught and assessed. The evaluators understand that these issues have been identified and addressed by the organisation. However, it is too early to determine that all changes have resulted in significant improvements, and that all assessments are now appropriate, to therefore provide assurance that achievement rates are valid. The organisation has appointed an external assessor and an external moderator who were interviewed at this evaluation, with both commenting on the improvements that are now in place, providing more robustness to the teaching and learning. The evaluators consider that these appointments are likely to result in needed improvements. There have been ongoing concerns identified through the national external moderation processes carried out by NZQA s Tertiary Assessment and Moderation unit that have not been fully addressed. The organisation is somewhat vulnerable in having a single assessor. Management agrees and is now exploring options for contracting more than the one current assessor. Facilitators teach in their specialist areas and have appropriate adult teaching qualifications. The organisation has been relying primarily on student evaluations for guidance about the effectiveness of teaching and learning. This is not considered robust self-assessment. Management is aware of this, and currently uses a coach on the go model while the organisation progresses through its current refresh and rebuilding processes. IMNZ has contracted the appropriate input from external specialists to address identified concerns, and evidence presented at this evaluation indicates that the changes implemented to address the above concerns have been well considered and are likely to lead to significant improvements in educational performance. These factors indicate that the organisation is actively developing its capability in self-assessment. The evaluation team considers that it is too soon to see sufficient 13

14 evidence of effective, consistent and robust self-assessment and performance at this stage. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Adequate. The rating for capability in self-assessment for this key evaluation question is Adequate. IMNZ students appear to be appropriately supported for the style and nature of the study they undertake (a mixed mode of delivery, with some face-to-face and some workplace-related assignments). There are no obvious or apparent gaps in student support, and student satisfaction is high in student surveys. However, there is a lack of evidence to clearly show this. As already noted, the organisation s selfassessment processes primarily use the student surveys to identify students satisfaction with the training, but the survey does not include any questions regarding student information, guidance or support. In the absence of any other processes or data, the evaluation team considers that they cannot confirm that there are strong, consistent and reliable student support mechanisms in place. Students interviewed at this evaluation said they felt well supported, and in one case the student noted that when they moved to new employers, IMNZ chased me up and helped me complete my qualification I was amazed. Staff have also noted that they keep in close contact with students to offer support. For example, one staff member noted that there are small groups of students and, I know them all. The organisation operates a flexible approach to students completing qualifications as and when they are able, resulting in some students taking a longer time than expected to complete their qualification. This approach is based on meeting students needs. While this informal and anecdotal approach to the self-assessment practices around student support is common in small private training establishments, the evaluation team considers INMZ is of a size where it is harder to keep an accurate track on personal interactions, and a more planned and documented approach may be more appropriate. 14

15 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Adequate. The rating for capability in self-assessment for this key evaluation question is Adequate. IMNZ is a well-established private training establishment and a member-based organisation; it has both a professional management and leadership member support and networking function, as well as an education and training function. The organisation became aware that its membership base was reducing and its education programmes were attracting less support than previously. IMNZ contracted three separate organisations to review it from an external perspective as well as to develop improved internal processes. This has led to a refreshing of the organisation s brand with an internal restructure, including the creation of a national education manager position, and a major review of teaching and learning material to better meet client company needs, and to improve its programmes relevance. The Board has carried out an effectiveness review, resulting in changes to the structure of the Board from regionally based elected members to include two coopted members based on governance skills. The organisation considers that these changes are making significant improvements in how it is perceived in the business community and among its member organisations, as well as making improvements to the quality and relevance of its programmes. The evaluators consider that there is sufficient evidence to show that these changes are gaining traction and making the improvements noted above. However, the organisation has experienced challenges appointing a national education manager, resulting in a continuing significant knowledge and skills gap in the management team, to ensure all the areas identified in the external reviews have sufficient policy and procedures developed and implemented to ensure the learning and teaching are consistently and reliably implemented, and outcomes achieved, as well as tracked and monitored for ongoing improvements. The implementation or embedding of some key processes have had to be put on hold. The organisation has some way to go to have an embedded culture of self-assessment and sufficient processes in place to ensure all programmes and activities are reviewed with sufficient rigor to identify areas of strength and weakness. 15

16 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management and strategy The rating in this focus area for educational performance is Adequate. The rating for capability in self-assessment for this focus area is Adequate. 2.2 Focus area: Certificate programmes levels 3 and 4 The rating in this focus area for educational performance is Adequate. The rating for capability in self-assessment for this focus area is Adequate. 2.3 Focus area: Diploma programmes levels 5 and 6 The rating in this focus area for educational performance is Adequate. The rating for capability in self-assessment for this focus area is Adequate. 16

17 Recommendations NZQA recommends that IMNZ: Develop an organisational culture that has a clearer focus on the educational side of the organisation as a matter of priority. Continue to strengthen assessment and moderation knowledge and skills within staff to ensure robust internal pre-assessment and post-assessment moderation. Develop structured and documented processes for reviewing the organisation across all education and capability-building functions (programmes and short courses), as well as management and governance functions, i.e. strengthen self-assessment. Proceed with the planned implementation of a single data management system with a view to better data analysis for organisational selfassessment. Develop more systematic stakeholder engagement to determine how well their needs are being met. Improve processes for the self-review of guidance and support. 17

18 Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( The External Evaluation and Review (EER) Rules 2013 are available at while information about the conduct and methodology for external evaluation and review can be found at NZQA Ph E qaadmin@nzqa.govt.nz 18