Follow Up Data Quality Review of Data Collection Systems:

Transcription

1 Follow Up Data Quality Review of Data Collection Systems: Vocational Education and Training Management Information System (VETMIS) Final Draft 9 October 2014 Nils Riemenschneider, Jesse McConnell, David Megill Oxford Policy Management Commissioned by the Millennium Challenge Account Namibia with funding from the Millennium Challenge Corporation

2 Table of Contents Acronyms... 3 Preface Introduction VETMIS Features and Implementation Status Modules NETSYS Additional features Potential data usage User manuals Implementation Training Challenges Qualifications structure System capacity NTA Capacity TP infrastructure Conclusions and Recommendations Annex 1: Sources consulted Stakeholders consulted List of documents consulted Annex 2: VETMIS interface Main Menu VTPs listed Trainees listed

3 Acronyms BAR CBET DQR MCA N M&E MIS NTA NQA NTF SME ToR TP VET VETMIS VTP Bi Annual Review Competency Based Education Training Data Quality Review Millennium Challenge Account Namibia Monitoring and Evaluation Management Information System Namibia Training Authority Namibia Qualifications Authority National Training Fund Subject Matter Expert Terms of Reference Training Provider Vocational Education and Training Vocational Education and Training Management Information System Vocational Training Provider 3

4 Preface The Millennium Challenge Account Namibia (MCA N) has contracted Oxford Policy Management (OPM) to provide consultancy services for the Data Quality Review (DQR) of different MCA N monitoring and evaluation (M&E) related activities. One of these DQR tasks is to carry out ex post reviews of selected projects or components of MCA N s activities during the life of Millennium Challenge Corporation s compact with the Government of Namibia. This report contains the findings of the follow up on data collection systems for vocational training data, specifically the Vocational Education Training Management Information System (VETMIS), developed and implemented by the Namibia Training Authority (NTA). The findings are based primarily on consultations with the Manager of Policy, Planning and Research at NTA, conducted during the DQR team s mission to Namibia in May 2014, as well as an overview of the various user manuals provided alongside VETMIS. The VETMIS has seen significant growth and development over the past year. Though the system is still to be finalised in its development, it shows considerable usefulness in the way that it is able to manage data from across the vocational education training sector. In spite of the significant gains that have been made in developing VETMIS, significant challenges still remain. The largest challenge to NTA in the completion and roll out of VETMIS is in the area of capacity, both within NTA as well as the capacity amongst the TPs. In order to address these critical challenges, the following recommendations are offered: Increased resources are needed firstly at NTA by way of the financial resources to be allocated for the creation of a single staff position that is wholly dedicated to maintaining and rolling out VETMIS. This is especially important towards the completion of VETMIS, given the anticipated lengthy process that was indicated as still being needed. Further to this position, a staff member wholly dedicated to training and building capacity amongst vocational training providers (TPs) on the use of VETMIS is needed, as this requirement is too great to be a simple additional component of other staffing roles. In support of this role, component tasks of the VETMIS system should be delegated across the divisions at NTA, to allow for various staff to carry portions of the responsibilities needed by VETMIS. Furthermore, NTA and the VETMIS developer should provide both data producer and data user with a comprehensive training programme, including orientation and training (including refresher courses) to relevant stakeholders (including, for example, staff of the National Planning Commission s Monitoring and Evaluation directorate who may use VETMIS derived statistics in their national M&E work), ensuring that all are conversant and confident in the various functionalities of VETMIS. 4

5 1. Introduction The Vocational Education Training Management Information System (VETMIS) has been development by the Namibia Training Authority (NTA) as a mechanism with which to manage and monitor data from across the vocational education and training (VET) sector, as well as manage projects implemented and managed by NTA. 1 The development of VETMIS has been a lengthy process over two years at this point with a number of service providers having started on the system and not satisfactorily completed the task. The long standing need to develop VETMIS was mentioned in the second of the DQR Team s Bi Annual Ex Post Data Quality Review Reports as a likely solution to many of the data quality challenges that the NTA had experienced over the course of its work with MCA N. This pertains largely to the challenge of training providers reporting on needed data biographical information of trainees, number of trainees enrolled or registered at different times, courses and qualifications enrolled in by trainees, academic performance levels, etc by NTA in good time, and doing so with reasonable quality. A further challenge that VETMIS has possibilities of overcoming is tracking individual trainees, especially tracking them in the way they perform against specific qualification requirements. 2 The full system development, as it stands currently, began in August 2013, and started with developing modules that were most directly related to internal NTA staff. Since then, the roll out of VETMIS, as well as training on the various modules and the further development and refinement of the modules has been ongoing. The VETMIS system was bought as an off the shelf system, meaning that it was not designed from scratch, but derived from the basis of an existing template, and modified according to the needs of NTA. At the point in time of this review, NTA had developed all of the necessary modules except for two outstanding modules, which include an assessment module and a skills planning module. As the full system is still in the process of being developed, NTA is also working on modifying already developed modules as they reach levels of usability. 1 See especially RemoteNet VETMIS NETSYS User Manual and RemoteNet Project Management User Manual: all projects 2 OPM, Bi Annual Ex Post Data Quality Review (Round 2), 29 June 2012, p21 5

6 2. VETMIS Features and Implementation Status This section provides a basic narrative on how VETMIS works in terms of its different modules and the manner that the Training Providers (TPs) participate in data collection. The NTA has identified various user types which are allowed access to the VETMIS system. These user types include, but are not limited to, NTA staff, training providers, and subject matter experts (SMEs). 3 Within the different user types, then, NTA has categorised the users into different roles with NTA to determine their level of access to data available through the VETMIS. All users are then able to register on the system with their own password Modules VETMIS is comprised of multiple modules, all of which serve different functions, both to internal NTA staff, as well as external stakeholders. TPs enter data into NETSYS an altogether separate component of VETMIS designed specifically for the functional needs of the TPs which is then uploaded periodically into the broader VETMIS system. The Main Menu provides a portal for all the different modules that make up the system. The main components on the menu are the following: 4 Core processes o VTP Gateway Module (TP interface) NTF (training fund) Reports Project Management Supply Change Client Liaison Governance MIS Access Levels Some of these modules are described in greater detail below: Project Management: this module allows for the tracking of all aspects of project management for NTA, such as income and expenses, disbursements, payments, etc. This is then linked to the strategic objectives of NTA, ensuring a degree of accountability in the way that projects are implemented and align with the main objectives of NTA. Product Development: the products that NTA offers include the different levels of qualifications that trainees are able to attain through its standardised system, as well as unit standards for course curriculum and to which the TPs should adhere. This module is then linked with the NETSYS, which TPs complete for each trainee. Trainees are then registered for specific qualifications as they occur in the product development module. 3 This list of users is not exhaustive as it is still being developed together with the broader VETMIS, but reflects the key users noted during the stakeholder consultations during the DQR Team s field mission. 4 Please see Annex 2.1 for a screenshot of the VETMIS Main Menu webpage. 6

7 Procurement: the procurement module is intended to make this component of NTA s operations function more efficiently. This allows NTA to manage the manner that vendors are registered in their system and how communication is dispersed. Assessment: this module is still being developed, and NTA is struggling to allocate sufficient resources to complete this module. However, once complete, it is envisioned that the assessment module will help monitor the levels of achievement and performance attained by trainees against, for example, the qualification standards that they are enrolled in. Skills Planning: this module is also still being developed, but it will be able to report on priority skills areas both in supply and demand across various sectors throughout the country NETSYS One of the visions of NTA for VETMIS is that it will allow training providers to input data directly themselves, which would greatly streamline the data collection process for NTA in monitoring their associated training providers, as well as ensuring a greater degree of data quality, as the system allows for different built in quality checks. The NETSYS module is developed specifically for the TPs to use for entering data, which then gets uploaded to the larger VETMIS system. The module is currently being rolled out through trainings offered to the various TPs. The training is being implemented gradually as NTA continues to struggle with adequate staff capacity to manage their full scope of work. Thus, the NETSYS is a stand alone component that connects to VETMIS which is specifically focused for use by the TPs. The TPs enter all trainee data, such as biographical data, courses enrolled in and completed, and any other similarly relevant personal data. They are not able to track trainees yet, which will rely on the assessment module, which is not yet fully developed. But once the assessment module is complete, it will tie into the NETSYS and allow for trainees to be tracked according to their performance and results achieved in their courses. The assessment module will help to effectively close the loop of monitoring trainee progress. The information that VETMIS includes for the TPs specifically includes a correspondence history, links to a Subject Matter Expert (SME), and the specific qualifications offered by the TP. 5 NTA is also able to correspond with TPs or their SME through VETMIS. The NTA currently has 777 trainees registered in the system. 6 They (NTA) are now primarily looking at new trainees to get them entered into the system. Once this is seen as largely completed, they will also have to go back several years as well, and capture previous or 5 Please see Annex 2.2 for a screenshot example of the data for TPs in the data base. 6 At the time of the discussions held with NTA for this review, on 15 July

8 currently enrolled trainees that are not new into the vocational training system. This is seen as a secondary priority to updating the system with the newest trainees. 7 Moreover, the major concentration for NTA for the moment is with the full and complete development of the system before its full roll out and implementation, provided through initial and on going training and support Additional features Some additional features that the system currently is able to perform are listed below: The system is able to signal when a unit standard is up for review, and it will send a notification about this. When data are uploaded from the TPs to VETMIS, they get checked against the qualifications levels as a quality assurance measure, ensuring uniformity and accurate data entry for trainees. There is also a verification and quality assurance process that NTA uses before uploading new data from the TPs into VETMIS. The system also includes tracking information on registration with NTA and accreditation with the Namibia Qualifications Authority (NQA) for TPs. Notification around the accreditation of the TPs, flagging when this is due for renewal or notification when accreditation has expired. NTA is also exploring options of linking with NQA in order to automatically upload new qualifications into VETMIS. 8 Employers (companies) are also registered with NTA on this system, as long as they are paying the Vocational Education Training (VET) levy. These are listed in the system, which currently includes over 2,800 registered companies. NTA is exploring the potential for using VETMIS as a platform for conducting surveys of various kinds, such as amongst trainees, TPs, and registered employers. VETMIS is able to generate reports of various sorts as well, such as data on the providers that are registered, trainee profiles and achievements, etc. NTA has a monitoring and evaluation (M&E) framework, and expects that VETMIS will be able to calculate values for the different indicators that NTA monitors Potential data usage Given the large amount of data that the VETMIS incorporates and manages, as well as the diverse types of data, the potential usage of this data is wide and varied. Nevertheless, the following sub section provides a brief overview of possible examples of how the data might be used, and the potential analytical power that it presents. 7 Please see Annex 2.3 for a screenshot example of the data for trainees currently listed in the data base. 8 This specific functionality is not yet operational in the system, but the necessary modules for this are being modified to allow for these features. 8

9 Trainee data Because the TPs at the most local levels upload trainee data into the NETSYS (as described above), which is in turn uploaded to VETMIS, these data include fairly standard biographical and contact data, which, while standard, can provide powerful analytical potential, such as the following: The average age of trainees entering vocational training programmes, or the average amount of time spent in a training programme by age, including trends around ages most likely to complete higher levels of certification, etc; Statistics on drop out and completion trends, gender distribution, or subject preferences by gender/age/region/etc; The presence on VETMIS of a module for the National Training Fund (NTF) even offers the possibility of tracking graduated trainees and monitoring success levels after the completion of training programmes, as well as market needs as presented by the industry members participating in the VET Levy a highly valuable source of information for trainees seeking to enter the job market at the completion of their qualification; These various analytics present the possibility of NTA proving more targeted training according to the different trends that can be analysed, such as those mentioned above, such as targeting according to gender specific trends in subject matter, or regional and industrial based needs. Training Providers A significant function for NTA in monitoring the vocational training sector includes monitoring and regulating those who provide training (TPs). Two methods through which this is done include TPs being registered with NTA or accredited with NQA. As a result, some functionalities of the data in these regards including the following: Both of these memberships registration and accreditation have periods of validity and points of expiration, which VETMIS can monitor and flag at various points before expiration occurs, while also serving to disseminate information to the TPs as reminders about upcoming renewal periods. VETMIS might even consider creating an incentive system based on TP performance that would relate to statistics on trainee completion and graduation a high graduate rate indicates a strong support system offered by the TP, which makes it a more sought after institute to receive training from; high placement rates for graduating trainees, which indicates relevance and sought after qualifications by the industry. Such an incentivising system could be bolstered by various grants offered to TPs that perform and achieve in such key performance areas. Educational products The NTA provides various training and educational products, such as training modules in various sectors, at various levels of certification. As a result, the development and monitoring of these modules can be conducted through VETMIS, including some of the following examples: Modules are registered in VETMIS and include unit standards and course curriculum details. Their registration information would include a date at which the module is 9

10 first included in VETMIS. From this point, it would need to be reviewed after a certain period of time (i.e. every few years) to ensure it remains relevant and that its curriculum is not outdated. These periods of renewal can be flagged by VETMIS, which can automatically signal when modules are up for review or renewal. The various unit standards that constitute the different modules and subject matter courses in VETMIS can also be linked with actual trainee registration information, ensuring that a trainee registers for a course that is formally registered in VETMIS, linking the way that the trainee s progress is monitored against the different unit standards that constitute a specific course. Furthermore, any updates made to courses or modules at NTA can then be automatically updated and linked to a trainee s profile, with the trainee being alerted of the changes. This provides a strong quality control measure for trainees progress, ensuring accurate progress for a trainee and sound monitoring for the TP and NTA. Project management The Project Management module of VETMIS promises to offer very powerful monitoring tools for the administrative aspects of NTA, allowing for greater efficiency and transparency. Some of these data functionalities might include the following: Budget tracking for different projects that NTA implements, such as income received, payments made to vendors or service providers, categorised expenses; and from these data one can run analyses that indicate margins of efficiency with income to expense ratios, profit and loss, as well as budget forecasting for future planning purposes; Project timelines can be included in the project management module, which would include inputting details on when tasks that pertain to a project s implementation would need to be completed, organised against the specific service providers that would be working on those tasks, forecasted associated costs, and other details that would be needed for effective planning and timing in the implementation of a project. Dissemination platform Because contact information is a vital component of the data that is collected by VETMIS, whether from trainees, TPs, industry members, or subject matter experts, this allows VETMIS to have a broad reach of access to highly relevant stakeholders. As a result, VETMIS is able to serve as a platform from which NTA can disseminate information or conduct surveys or otherwise send data back to providers and stakeholders, not simply only receive inputted data. Examples of this might include the following: Surveys distributed to trainees via about their levels of satisfaction with the training they are currently engaged in; Messages to TPs about upcoming training programmes being offered on new course material; Notifications to subject matter experts about upcoming reviews of modules and unit standards due for updating, requiring their input; Surveys distributed to industry members (companies enrolled in the NTF) distributed through contact information around their current and forecasted employment 10

11 needs, market trends, and request for input on potential training alterations based on industry and market needs User manuals Various user manuals have been developed for VETMIS and the various user types that have access to the system. These are all generally very user friendly, providing screenshots along with clear step by step instructions on how to make use of the system s functionality. The list of currently available user manuals for the system is provided below. The first manual the user manual for NTA is the most comprehensive, as it covers the majority of the system. The additional manuals are all generally created around single modules, directed to specific users. The following is a list of primary manuals provided by NTA at the time of this review: 1. NETSYS User Manual, NTA 2. VETMIS User Manual, CRM Call Centre 3. VETMIS User Manual, Meeting Management 4. VETMIS User Manual, Project Management all projects 5. VETMIS User Manual, Project Management set budget and period 6. VETMIS User Manual, Qualifications 7. VETMIS User Manual, receive new invoice 8. VETMIS User Manual, VTP Learners 9. VETMIS User Manual, VTP Quality Control 10. VETMIS User Manual, VTP SME Registrations 11. VETMIS User Manual, VTP Training Provider 2.6. Implementation The general implementation of the VETMIS system is occurring slowly, having started in August 2013 and still being incomplete (two modules require development). This is attributed primarily to NTA s capacity challenges. The institution is unable to allocate sufficient staff resources for the full implementation of the system, in terms of the complete modular development and roll out to staff and TPs. NTA currently has no internal staff equipped to run VETMIS, and draws on a staff member in their quality assurance department to help with coordinating the system, provide training on it, help build capacity for the TPs, etc. These are all significant tasks to be undertaken by any one person, let alone to be undertaken by a single employee whose primary task is not related to VETMIS. The estimated amount of time required for the full roll out of the system was a further two years from the time of this review. This is the estimated time that would include the full 11

12 development of the system and staff training, as well as complete data uploaded into the system and stakeholders well versed in the system. 9 Thus, while there are 777 trainees currently enrolled in the system, it is estimated that there are over 13,000 trainees currently enrolled with VTPs. This means that only about 6% of enrolled trainees have been entered into the VETMIS data base. Thus, considerable work is yet to be done in the complete roll out of the system with complete data entry Training For training on the system, NTA held a week long introduction for all the TPs in the early part of the VETMIS development, but this was noted as being poorly attended. Now, NTA s approach to the training is to go to specific TPs as deemed necessary in order to build specific capacity. Such targeted training is also a function of NTA s capacity shortages, where they are not yet able to provide a broad scale implementation. Nevertheless, NTA has noted that more and more data are being entered and uploaded from the TPs, and are being provided more regularly. NTA also notes needing to formulate a more systematic training programme to put in place that would provide better on site support for the TPs in addition to servicing the full implementation of the system. Again, this is hampered by capacity shortages. 9 Taken from a discussion with Ernst Mbangula, the Manager for Policy, Planning and Research at NTA, noted in Annex

13 3. Challenges A number of key challenges remain for NTA to overcome in its completion of the VETMIS development and its roll out amongst the various training providers Qualifications structure The previous qualifications structure is creating a challenge to the way that the newer system is set up in the MIS. The previous qualification structure used a modular system, which simply required passing a level (module) in order to move into the next. The current system Competency Based Education Training (CBET) is competency based, where a trainee gets qualified as competent or not for a whole qualification. The CBET system is also unit based, rather than modular. This is a somewhat confounding component in the development of the assessment module too, as NTA is considering how to reconcile the two methods into the system, or whether they should even do so (compared to only keeping the more recent CBET framework) System capacity During the demonstration of VETMIS provided by NTA to the DQR Team, the system itself was noted as already functioning slowly, even though such a small portion of the required data has actually been uploaded. This is concerning at so early a stage in the development of the system, and portends the need to increase the capacity of the system to more efficiently manage the level of data that it is ultimately intended to process NTA Capacity As stated earlier in the report, NTA has insufficient staff capacity to manage the VETMIS system. Their suggested solution to this, overall, is to have one staff member who is fully dedicated to the entire system, and additionally to have staff members in each NTA division responsible for inputting the relevant data for NTA. This is a delegation of the more minor tasks across all the divisions in NTA for basic data monitoring of the system. NTA also currently has a maintenance contract with a developer that has worked on the system thus far. More ideally though, they should have a fulltime staff member who is dedicated to maintaining VETMIS. A person dedicated to training on the system is also a key need, especially given the staff turnover amongst the TPs that NTA has noted as a critical challenge for them as they seek to roll out the system. Finally, provision has been made in the Policy, Planning and Research division for a data analyst to run analyses and queries with the data. Ultimately NTA wishes to produce annual reports with the data provided and managed by VETMIS. 13

14 3.4. TP infrastructure A further challenge and need relates to the infrastructure at the levels of the TPs. Internet service is needed for the TPs to upload data that they have input into NETSYS. This is not always available or reliable. Furthermore, even basic computer skills are noted as often being lacking or at best minimal for many of the TPs. And as a result, even computers that have been provided by NTA to many of the TPs were seen still in boxes well after they were delivered. Thus even occasional infrastructural challenges are compounded by capacity deficiencies on the part of the TPs to make adequate use of available infrastructure. 14

15 4. Conclusions and Recommendations The VETMIS has seen significant growth and development over the past year, considering the laborious and painstaking process that NTA underwent to arrive at its commencement. Though the system is still to be finalised in its development, it shows considerable usefulness in the way that it is able to manage data from across the vocational education training sector. There is little doubt that it will be able to aid NTA in its efforts to monitor and manage the sector in a more timely and poignant fashion. In spite of the significant gains that have been made in developing VETMIS, significant challenges still remain. The largest challenge to NTA in the completion and roll out of VETMIS is in the area of capacity, both within NTA as well as the capacity amongst the TPs to be able to be fully competent with the system. Within NTA, the organisation does not have sufficient staff for them to allocate adequate human resources to finish developing and then to maintain the system, nor to sufficiently roll out the system amongst the TPs through targeted training. The TPs themselves are faced with capacity constraints firstly in the area of having sufficient staff to manage the data requirements that NTA needs for adequate data monitoring, nor do they generally have the basic technical skills needed for data input. Moreover, NTA s challenge in providing capacity building at the TP level is challenged by the fact that the TPs experience high staff turnover, compounding the challenge of building and maintaining staff capacity. In order for these challenges to be overcome, the following recommendations are offered: Increased resources are needed firstly at NTA by way of the financial resources to be allocated for the creation of a single staff position that is wholly dedicated to maintaining and rolling out VETMIS. This is especially important towards the completion of VETMIS, given the anticipated lengthy process that was indicated as still being needed. Further to this position, a staff member wholly dedicated to training and building capacity amongst TPs on the use of VETMIS is needed, as this requirement is too great to be a simple additional component of other staffing roles. In support of this role, component tasks of the VETMIS system should be delegated across the divisions at NTA, to allow for various staff to carry portions of the responsibilities needed by VETMIS. Furthermore, NTA and the VETMIS developer should provide both data producer and data user with a comprehensive training programme, including orientation and training (including refresher courses) to relevant stakeholders (including, for example, staff of the National Planning Commission s Monitoring and Evaluation directorate who may use VETMIS derived statistics in their national M&E work), ensuring that all are conversant and confident in the various functionalities of VETMIS. 15

16 VETMIS has the clear potential to offer all the necessary components of a sound management information system, and presents NTA with the opportunity to efficiently manage data from across the sector. However, if these challenges are not met with sufficient political will and financial resources, VETMIS could just as easily languish in diminishing use as much as act as a vehicle for sound sectoral management. 16

17 Annex 1: Sources consulted 1.1 Stakeholders consulted Institution Capacity Date People present NTA Manager: Policy, Planning & Research 15 July Ernst Mbangula 1.2 List of documents consulted OPM (2012), Bi Annual Ex Post Data Quality Review (Round 2), 29 June 2012 RemoteNet (2014), VETMIS User Manual CRM Call Centre RemoteNet (2014), VETMIS User Manual Meeting Management RemoteNet (2014), VETMIS User Manual Project Management: all projects RemoteNet (2014), VETMIS User Manual Project Management: set budget and period RemoteNet (2014), VETMIS User Manual Qualifications RemoteNet (2014), VETMIS User Manual receive new invoice RemoteNet (2014), VETMIS User Manual VTP Learners RemoteNet (2014), VETMIS User Manual VTP Quality Control RemoteNet (2014), VETMIS User Manual VTP SME Registrations RemoteNet (2014), VETMIS User Manual VTP Training Provider RemoteNet (2014), VETMIS NETSYS User Manual NTA 17

18 Annex 2: VETMIS interface 2.1 Main Menu 18

19 2.2 VTPs listed The data listed in the provider code column have been redacted for anonymity purposes. 19

20 2.3 Trainees listed The data listed in the learner record, registration number, national ID and learner last name columns have been redacted for anonymity purposes. 20