ESCO Mapping Pilot workshop 4

Transcription

1 ESCO Mapping Pilot workshop 4 03/11/2015 Brussels 1/

2 Agenda Time Agenda item Time 10:00 Opening of the meeting 12:45 General conclusions mapping classifications 10:20 Status update mapping pilot 13:45 Coffee break 11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching 11:15 Country specific conclusions mapping classifications 14:45 Next steps & planning 2/ 12:00 Lunch 15:00 End of the meeting

3 Opening of the meeting 3/

4 Approval of meeting minutes July 2nd Meeting minutes of the July 2nd workshop Feedback of Julius de Zeeuw on section Dutch data Feedback of Alain Dupuch on section French data Feedback of María José Arias Fernández on section Spain data Other corrections 4/

5 Agenda Time Agenda item Time 10:00 Opening of the meeting 12:45 General conclusions mapping classifications 10:20 Status update mapping pilot 13:45 Coffee break 11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching 11:15 Country specific conclusions mapping classifications 14:45 Next steps & planning 5/ 12:00 Lunch 15:00 End of the meeting

6 Status update mapping pilot 6/

7 Mapping pilot objectives Define a straightforward process to create a mapping To prototype tooling to support the mapping process To create draft mappings for NL, FR, CZ, and ES (Hospitality and Tourism) To validate the draft mappings to learn What type of information is retained and what is lost What is going well, and what needs improvement 7/ To test the mappings with some CVs and Job Vacancies

8 Mapping process 8/

9 Actions from previous workshop Task 1 Provide an updated mapping tool User validation interface Enhanced ROME occupation proposals based on appélations Enhanced skill mapping proposals based on skill/occupation relations 9/ Who TenForce 2 Participants provide feedback on mappings PESs 3 Select & translate curriculum vitae and job vacancies for matching TenForce 4 Process curriculum vitae & job vacancies PESs 5 Process cross-border curriculum vitae and job vacancy matching TenForce

10 Updated mapping tool strategies to improve proposed mapping 1. SolR based text matching [new things below this line] 2. Automatically add synonyms 3. Better filtering using ISCO Match concepts based on context 5. Use hierarchy 6. Use skill relations 10/

11 Updated mapping tool - subject based clustering Using open source Mallet NLP engine (UMASS) Use extracted subjects from Wikipedia to increase number of mapping proposals Subject based clustering provides Extra context, not solely based on textual similarity Boost clarity on mapping proposals Can increase the number of false positives 11/

12 Update mapping tool - combining different solutions Need a normalized total score, so used weighted approach. Weights should be configurable Should depend which metric performs best, e.g. text based matching is better than context based matching and receives a higher weight But need expert knowledge to make that call 12/

13 Updated mapping tool - Concept relations and flooding (explained) 1. For every proposed mapping, e.g. between two occupations 2. These occupations have relations, like hierarchical or occupation-skill relations 3. Look at nearby mappings of concepts connected through these relations 4. If these nearby mappings have a high value, flood some of this value to the target mapping being considered 5. This is iterative, if the user accepts or rejects a proposal, the certainty of the suggestion changes to 1 (accepted) or 0 (rejected) and the flooding can be redone. 13/

14 ❶ Concept relations and flooding 14/

15 ❷ Concept relations and flooding 15/

16 ❸ Concept relations and flooding 16/

17 ❹ Concept relations and flooding 17/

18 ❺ Concept relations and flooding 18/

19 Updated mapping tool enhanced clarity on mapping proposals 19/

20 Updated mapping tool enhanced clarity on mapping proposals 20/

21 Updated mapping tool enhanced clarity on mapping proposals 21/ Reduce uncertainty for proposed mappings

22 Mapping tool workflow 2 Step approach: first basic mapping, second refined mapping + Faster to functional result + Earlier gap analysis + Allow perfecting the mapping at a later stage - Good proposals visited twice Experts can manually add/correct mappings 22/

23 Mapping tool - general Two step approach towards mapping Focus on efficiency in the workflow Need for usability improvements Features to be added (lessons learned): Function to add additional mappings Filter and search capabilities Side-by-side concept browsing Ignore stop words/words without meaning Configuration of scoring parameters Automatically check mappings on logical errors 23/

24 Feedback received from participants on data and mapping tool On the data : expert involvement facilitates Validate the mapping outcome Assure the quality of the translations (not an objective of the mapping pilot) On the process: no comments On the mapping tool: overall positive, however: Approach by the tool differs from current practice Need more functionalities in validation interface Persist user status in workflow Noise level in mapping proposals is sometimes too high ISCO filtering: differences due to differences in how concepts are classified (irrelevant when mapping complete classifications) 24/

25 Agenda Time Agenda item Time 10:00 Opening of the meeting 12:45 General conclusions mapping classifications 10:20 Status update mapping pilot 13:45 Coffee break 11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching 11:15 Country specific conclusions mapping classifications 14:45 Next steps & planning 25/ 12:00 Lunch 15:00 End of the meeting

26 Country specific conclusion classifications mapping 26/

27 NOC specific results Detailed analysis per NOC Numbers represent what can be done purely using tool No manual addition of mappings TEG completed mappings Noise reduction by tool: 90% 27/

28 Common vocabulary Parameter Number NOC & NSC Concepts Exact Matches Close Matches Broad Match Narrow Match No Match from ESCO (54 353) No Match from NOC PES TEG Description Total number of OCC and KSC from PES to be mapped Transitive, symmetric, interchangeable Similar, sometimes interchangeable Hierarchical relation Hierarchical relation Absolute number of percentage of concepts without match Absolute number of percentage of concepts without match 28/ # days to complete mapping HOSP # days to complete mapping HOSP

29 29/

30 Mapping results NL NOC Received feedback on the use of ISCO, different views on ISCO tagging resulted in need of manual adding of matches outside of HOSP Note this is due to the restriction of the mapping pilot to the HOSP sector, the effect of this restriction on a full mapping should be of lesser impact. 30/

31 ESCO: 54 OCC & 353 KSC Number of NOC and NSC Concepts OCC KSC Exact Matches Close Matches Broad Match (ESCO>NOC) Narrow Match (ESCO<NOC) No Match from ESCO No Match from NOC 113/ PES TEG More NOC concepts, finer granularity 50% of ESCO occupations has exact match many of NOC occupations are not matched: Different approach towards cross-sectoral occupations Different interpretation on the scope of the sector (94 are not ESCO HOSP) ISCO filtering 50% of NOC skills 31/ are not matched Occupations are broader while skills are narrower

32 32/

33 Mapping results FR NOC fiche métier (Occupation group) appélations (occupations) Granularity mismatch Granularity mismatch Many terms Few terms a balance of both? Significant words decreased in value Mapping base too small 33/

34 ESCO: 54 OCC & 353 KSC Number of NOC and NSC Concepts FR OCC FR KSC Exact Matches 2 8 Close Matches Broad Match Narrow Match No Match from ESCO No Match from NOC 534/ Granularity difference with ESCO No NPTs at level of appélations ISCO filtering (364 out of 534 unmatched NOC appélations are not ESCO HOSP) 33% additional mapping because subject based clustering (7k) PES TEG /

35 35/

36 ESCO: 54 OCC & 353 KSC OCC KSC Number of NOC and NSC Concepts Exact Matches 34 9 Close Matches Broad Match (ESCO>NOC) Narrow Match (ESCO<NOC) No Match from ESCO Notes: Missing relation between skills and occupations Missing non-preferred terms Most ESCO occupation mapped, many with exact match 70% of NOC skills are mapped, but many ESCO skills are not No Match from NOC PES 36/ TEG 0.5 1

37 37/

38 No. Nat. Class. Concepts CZ OCC CZ KSC Exact Matches Close Matches Broad Match 4 93 Narrow Match No Match from ESCO (54 OCC 353 KSC) No Match from NOC PES High number of unmatched concepts ISCO filtering Solr rules to be improved Mappings done by PES, more restrictive Low compatibility of skill descriptions 38/ TEG 0 0

39 39/

40 Agenda Time Agenda item Time 10:00 Opening of the meeting 12:45 General conclusions mapping classifications 10:20 Status update mapping pilot 13:45 Coffee break 11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching 11:15 Country specific conclusions mapping classifications 14:45 Next steps & planning 40/ 12:00 Lunch 15:00 End of the meeting

41 General conclusions classification mapping 41/

42 Agenda Time Agenda item Time 10:00 Opening of the meeting 12:45 General conclusions mapping classifications 10:20 Status update mapping pilot 13:45 Coffee break 11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching 11:15 Country specific conclusions mapping classifications 14:45 Next steps & planning 42/ 12:00 Lunch 15:00 End of the meeting

43 Status update job vacancies and curriculum vitae matching 43/

44 CV-JV Matching Tagging Scenario s Evaluation grid 44/

45 Task Who Status Selection of CV/JV TenForce Done Translation of CV/JV TenForce Done Annotation of CV/JV PES In progress Evaluation grid TenForce To do 45/

46 46/

47 Matching scenario s 1 Input Actor Output MATCH Input Actor Output CV language A PES A CV tagged PES A JV language B PES B JV tagged PES B CV tagged PES A MAP CV tagged ESCO CV tagged ESCO MAP CV tagged PES B 2 Input Actor Output MATCH Input Actor Output CV language A PES A CV tagged PES A JV language B PES B JV tagged PES B CV tagged PES A MAP CV tagged ESCO JV tagged PES B MAP JV tagged ESCO 3 Input Actor Output MATCH Input Actor 47/ Output CV language A TF CV tagged ESCO JV language B TF JV tagged ESCO + additional trials to investigate effects and parameters

48 Agenda Time Agenda item Time 10:00 Opening of the meeting 12:45 General conclusions mapping classifications 10:20 Status update mapping pilot 13:45 Coffee break 11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching 11:15 Country specific conclusions mapping classifications 14:45 Next steps & planning 48/ 12:00 Lunch 15:00 End of the meeting

49 Next steps and planning 49/

50 Next steps and planning Homework: processing the CVs and JVs Before Fri 13 November 2015 Workshop Tuesday 15 December 2015 Mapping Matching results : grid Actions Conclusions 50/

51 Thanks for your attendance and participation! and your collaboration towards workshop 5! 51/ ESCO - ESCO European - Eurpean Skills, Skills, Competences, Qualifications and Occupations and Occupations November 3, 2015