ESCO quality management plan December 2015

Transcription

1 ESCO quality management plan December 2015

2 ESCO (2015) SEC 101 FINAL Document Date: 20/11/2015 Last update: 04/11/ Purpose of this document With this document, the ESCO Secretariat (SEC) provides an overview of the quality assurance processes for the initial development of ESCO as well as of the quality management processes during the continuous improvement of ESCO after release of the initial version. The SEC has developed this document on the basis of previous documents 1 on this topic and discussions in the ESCO Maintenance Committee (MAI). With this document the SEC consults the MAI on the approach. 2. ESCO guidelines and existing classifications The SEC has developed together with the MAI a set of guidelines describing the development of ESCO and its main features. The guidelines ensure that all parties involved in the development and maintenance of ESCO have a common understanding of the requirements of the classification and the process for its development. The guidelines are the basis for the whole ESCO development process and guarantee a consistent outcome for ESCO v1. The SEC has also provided the people involved in the development of ESCO with a list of high-quality documents and existing classifications. The ESCO concept developers have used these existing sources in order to build their knowledge and cross-check their conclusions for the classification Quality management during the development process The SEC has organised the development of the 27 economic sectors of the ESCO classification in 2 phases. In the first phase, the SEC focussed on delivering the content of a set of 11 economic sectors. In the second phase the SEC focused on the creation of the content for the remaining 16 sectors. 1 Overview of quality management in the ESCO project (ESCO (2012) SEC 065 FINAL); Quality Assurance for ESCO (ESCO (2014) SEC 008 FINAL). 2 Although the SEC has not yet formally published the ESCO guidelines in its entirety, the SEC has discussed all sections relevant for the developers with the MAI, and distributed them to all people involved in the creation of the classification. The SEC expects to formally publish the guidelines in December

3 Figure 1. ESCO content development Quality management for the development of the first 11 ESCO sectors Together with stakeholders the Commission completed the classification for 11 out of 27 sectors. Groups of sector experts (SREFs) assisted by taxonomy experts (TEG) developed the concepts for the sectors on the basis of relevant source materials, existing classifications and the ESCO guidelines. During the development of the content, the system ensured technical compliance of the content by continuous application of automated validation based on the data model and the ESCO guidelines. Furthermore, the system provides statistical information to the developers on the basis of a set of defined key performance indicators (KPIs). Once the content was created, the SEC assessed its quality using a checklist that is part of guidelines, and reported its state to the MAI. The MAI evaluated the content and verified if the SEC applied the quality process during the development Quality management for the development of the last 16 ESCO sectors For the remaining 16 sectors (plus some subsectors of healthcare), the SEC adapted the process for developing ESCO. In the new process, taxonomy experts draft the content using existing classifications and other quality sources. During this process the December

4 system supported the taxonomy experts with the aforementioned compliance checks and KPIs. The taxonomy experts further verified their work with regard to completeness by comparing it with with the concepts available in well-known and mature classifications such as: ROME (Répertoire Opérationnel des Métiers et des Emplois, by PES France) and KldB (Klassifikation der Berufe, by PES Germany). The SEC assessed the quality of the draft content using the quality assurance checklist from the guidelines and presented the findings to the MAI. After these quality assessment steps, the SEC involved the sector experts. Currently, experts from all over Europe review the content online through a consultation platform. The experts have the possibility to rate occupations (OCC) and skills (KSC), and comment and discuss the content that the taxonomy experts have developed. After the consultation, the TEG will refine the content by taking into consideration the ratings and comments that the domain experts have provided. Once these quality improvement steps have been completed, the SEC will discuss the comments, and (if needed) remediation actions with the MAI, while in parallel, the SEC will share the content once more with the domain experts for final validation. 4. Quality management during the clean-up process After the finalisation of the content for the whole set of 27 sectors, the SEC will integrate and clean-up the different parts of the classification. The SEC will organise the clean-up process for the OCC separately from the clean-up process of KSC. During this process, the system supports the people involved by verifying continuously the compliance with the data model and by providing feedback on various characteristics of the data through KPIs. December

5 Figure 2. ESCO content clean-up and publishing process 4.1. Quality management for the clean-up process for the ESCO occupations The SEC will start to clean-up OCC by having an ISCO expert validate if the OCC are correctly mapped to ISCO-08. The SEC will also evaluate if ISCO is properly covered by the ESCO OCC. After this, the SEC will map the ESCO occupations to various national classifications, in order to compare the coverage of each of these national classifications with ESCO Quality management for the clean-up process for the ESCO knowledge, skill and competences In parallel, the SEC will start the clean-up process for the KSC. During this process, the SEC considers inter alia to organise the skills in collections, remove redundant skills and ensure that transversal skills are properly represented. At the same time, the SEC will liaise with Cedefop, to build on their expertise and to align the application of action verbs. The Commission considers organising an additional step after this: Mapping the ESCO skills pillar to a national, regional or sectoral skills classification could help to better understand the coverage of the ESCO skills pillar. Based on this mapping, the TEG can remediate possible omissions. December

6 5. Quality management during the term formulation process The Commission initially develops ESCO concepts in the English language. Once the concepts are final, the SEC will first organise a thorough quality check for the English terms. Secondly, the SEC will organise that the terms are formulated as well for the other 23 official languages of the EU. Figure 3. ESCO term formulation process Validation of the defined terms in the reference language (EN) In this process, an English terminologist and market experts ensure that the defined terms properly reflect the meaning of the concept as captured by the description and the scope note (if available). The SEC has implemented the English term formulation process in two steps: a. A linguistic check encompassing the compliance of the terms with grammar rules - vocabulary check; b. A market reality check from the viewpoint of pragmatics and relevance of the terms used for a given economic sector done by market experts. The validation will take place for both the preferred terms and any other existing nonpreferred terms in English. In this stage, the terminologist adds any other nonpreferred terms. In case of doubts, the service provider contacts the experts who are competent for a given sector. The terminologist validates the English terms against practice in the market in cooperation with selected experts Term formulation for the target languages After the finalisation of the reference language (EN), the terminologists for the other languages will formulate the terms for the target languages following the principles laid down in the guidelines available for all EU languages. Translators will check the formulated terms in order to see if they are relevant for the labour market. For this purpose, the SEC will provide them with a list of contacts and sources for all languages, including websites, classifications, contact persons at the public employment services etc. The SEC will obtain information for this list through ministries, permanent representations and/or the ESCO expert groups (BOA, MAI, MSWG, CSREF). Terminologists will create terms in all the languages for each concept and capture these as a preferred term, non-preferred terms or hidden terms. They will capture the terms which are used in practice in job posts, by employment services, employers and December

7 job seekers. They will also add widely used non-standard terms such as misspellings and colloquial formulations as hidden terms. For choosing the right terms, the terminologist will take into consideration the description as well as other available metadata on the concept. The terminologist will aim to capture terms used in national classifications or by employment services as preferred terms. In the case of any doubts concerning the formulation of the terms that pertains to either aspect of the validity check, the service provider will contact the experts who are in charge and have the knowledge for a given sector. In order to minimise the gaps between the ESCO terms and the terms of the national classifications, the SEC will seek close cooperation with the employment services of the Member States. Additionally, the SEC will use a versioning strategy for ESCO that allows updating the terms in one specific language without affecting the rest of the ESCO classification. 6. Quality management building the collection of classification deliverables At the moment that the SEC considers the ESCO v1 classification to be complete, the SEC will extract the ESCO v1 content from the database and: ensure technical consistency and data integrity; ensure the DEURI compliance of the Uniform Resource Identifiers (URI) 3 ; annotate URIs for occupations with history information to support backward compatibility for ESCO v0 occupations; publish URIs in a persistent manner; build and package the versioned classification modules; create and test the various technical interfaces for: o accessing the definition pages for the concepts; o downloading the classification; o using the classification over the Internet with: the service API (application interface); the RDF query language (SPARQL); the local API (application interface) test the dynamic integration functionality for qualifications provided by the qualification integration framework. Additionally, the SEC will ensure completeness of the ESCO guidelines. To ensure that these processes can be executed in an efficient manner, the SEC will automate and test most of them before. 7. Quality management during the publication process The SEC will present the ESCO v1 classification deliverables, the ESCO guidelines, the qualification integration framework, the ESCO s technical interfaces, and their test 3 URIs are the linked open data identifiers used for the concepts in the classification; DEURI is the method for expressing URIs published by the European institutions. December

8 quality assessment reports to the ESCO Board (BOA) and the MAI. Based on the discussion the Commission will take the decision on publishing the ESCO v1 classification. 8. Quality management after publication After the publication and implementation of ESCO, the process of managing its quality will change significantly. At that moment, the quality of ESCO can be redefined in context of its actual use, i.e. how well it does what it is designed for. That is why the Commission will also involve users in the quality processes for ESCO. Domain experts can continue to provide information on new emerging occupations and skills and on the quality of the ESCO concepts, while users can provide information on how well ESCO supports the functions on their systems. The Commission will use the feedback of both of these groups in order to further improve ESCO after the publication of the ESCO v1. Quality Measurement The ESCO classification has many features, use cases and potential user groups. We need to take this into consideration when evaluating ESCO. For instance, it might be the case that ESCO could have a perfect fit for the labour market, while having a less than optimal fit for the educational domain (or the other way round). Likewise, the terms captured in ESCO for a given language could have a very good match with what is needed for the interpretation of plain text curricula vitae, whereas the same set of terms could require improvement for effective use at employment services. This is why the Commission will need to design key performance indicators for, e.g.: - Coverage and relevancy of the OCC (from the perspectives of the different use cases for ESCO 4 ) - Coverage and relevancy of the KSC (from the perspectives of the different use cases for ESCO) - Coverage and relevancy of the Qualifications (from the perspectives of the different use cases for ESCO) - Coverage and relevancy of the Work-Context (from the perspectives of the different use cases for ESCO) - Quality of the OCC profiles (from the perspectives of the different use cases for ESCO) - Aptness of the terms provided for OCC (e.g. for the employment services, and for text recognition from curriculum vitae, and job vacancy descriptions; per language) - Aptness of the terms provided for KSC (e.g. for the employment services, for learning outcome descriptions, for text recognition from curriculum vitae, and job vacancy descriptions; per language) - Aptness of the terms provided for work-context (e.g. for the employment services, and text recognition from curriculum vitae, and job vacancy descriptions; per language) December

9 - Supporting hierarchies It will not always be possible to measure these various dimensions of quality in a simple manner. The SEC therefore expects to implement these assessments by combining quantitative indicators, surveyed opinions as well as feedback from experts and users. 9. Next Steps After the consultation with the MAI, the SEC will finalise the quality management plan taking into account the opinion of the MAI. The SEC intends to regularly update the MAI on the implementation of the quality management plan. 4 The basic ESCO use cases are described in the ESCO guidelines. December