EUROPEAN CIVIL AVIATION CONFERENCE

Transcription

1 EUROPEAN CIVIL AVIATION CONFERENCE BEST PRACTICES FOR THE CERTIFICATION OF TRAINERS ENDORSED AT DGCA/142 (6 May 2014) Introduction 1. The purpose of this document is to provide Member States and other relevant entities with best practices in the development and implementation of trainer certification (competency assessment), as developed by the ECAC Network of Training Organisations. Selection, training and certification 2. As it is necessary that trainers be properly selected and trained to perform at the standard required, the training process has to be clearly distinguished and formally separated from the certification process. 3. The selection process aims at identifying those persons having the aptitudes, and mental and physical abilities required to carry out their designated tasks effectively. This process is carried out before completion of any probationary period. It does not aim at assessing competencies. 4. Training is a means to deliver knowledge to, and enhance the competency of a selected group of persons. It is a common misconception that everyone who has ever learned anything as a participant in a training course is able to become a trainer as well. Instructors should complete train-the-trainer or instructors courses to become familiarized with educational principles, instructional techniques, and group facilitation. 5. The certification process is the means by which these competencies are assessed. Certification 6. The main aim of certification is to ensure that a person has the combination of skills, knowledge and attitudes required to perform a task to the prescribed standard. 7. The key benefit of certification is to reasonably ensure, by means of valid and reliable assessment methods, that adequate standards are consistently and reliably achieved. 8. Furthermore, certification can be regarded as a layer in the aviation quality control system. Last but not least, certification can also be used as a relevant tool to measure training effectiveness and to improve trainers motivation. In this regard, certification can be considered as a key instrument serving the purpose of increasing the efficiency of aviation systems. Challenges to certification 9. There are many challenges to be faced when developing and implementing certification procedures:

2 - 2 - In the development phase, what should be assessed has to be very clearly identified and the tests designed accordingly. A well-designed test should measure exactly what it purports to measure (e.g. knowledge of subject matter). Then comes the development of the test itself (i.e. structure and content), where careful consideration should be given to potential mistakes that may seriously affect the test value. The revision and validation process, involving the administration of the test to as large a group of trainers as possible, under realistic conditions, is another challenging task, and one with costs implications. Maintaining test material (e.g. theoretical questions) constantly up-todate is a demanding task, which requires proper organisation and knowledge of regulatory and operational developments. 10. In short, the key challenges in the implementation phase are the following: Setting pass marks, so as to find a trade-off between competence (as high as possible) and the need for a sufficient workforce for practical reasons; Technical aspects, such as the choice of the most appropriate test tools (e.g., tests taken on paper, by means of computer-based systems or by means of internet-based test platforms); Organisational and operational aspects, such as: Costs. Competencies of trainers Test facilities Exam sessions organisation (number of trainers per session, duration, monitoring, test rules) Feedback given to trainers: When? Where? By whom? 11. Certification should verify trainers have the minimum knowledge, skills and competencies required for their tasks. Examples of the competencies to be assessed during the certification procedure include: Knowledge of the topics being taught subject matter expertise Knowledge can be demonstrated by: proof of attendance at previous training courses/certificates of attendance, professional experience in relevant field/oral or written test. Instructional/teaching skills: These skills can be verified by: proof of previous experience as a trainer (certification in a different domain), practical test to show the ability to use instructional aids and technologies, oral presentation tests (see Appendix). Planning and organising: the ability to plan, organise, and carry out a lesson by managing time well and supporting the learning development individually. Assessment skills: the ability to assess trainees performance is needed before training, during training, and after training in order to fully support the learning process and fill in knowledge gaps on time.

3 - 3 - These skills can be verified by: proof of previous experience in assessment performance of trainees, test on evaluating oral or written tests. Certification process - Scientific criteria: reliability, validity and standardisation 12. Over the past fifty years, scientific criteria have been developed that are widely used in psychological testing and psychometrics. These criteria are essential for the development of tests for measuring human performance (whatever purpose the test serves, whatever the field considered). Amongst these criteria, the three most important concepts are: reliability, validity and standardisation. 13. Reliability is the extent to which a test gives the same result with repeated measurements. Reliability refers to the consistency or repeatability of measures and relates to the quality of measurement, or in other words the consistency between measurements. If, for example, an IQ test yields a score of 90 for an individual today and 125 a week later, it is likely to be not reliable. In brief, tests should give approximately (to an extent that is precisely known) the same result with repeated measurements, in order for individual performance to be measured reliably. 14. Validity is the extent to which the test does actually measure the competency it purports to measure. A certification process aims at ensuring that a person possesses the necessary competencies to perform assigned functions, i.e. to act as a trainer, to an acceptable level. Therefore, a properly designed test must ensure that it actually measures these competencies. 15. For a test to have content validity, it must demonstrate at least two qualities: The content of the test must be job related (i.e. resulting from a detailed and comprehensive job analysis, where a set of competencies relevant to each task is identified); and The test must cover areas where lack of knowledge would result in inability to perform the job. 16. As with reliability, there are different methods of estimating the validity of a test. 17. Standardisation involves administering the test to a representative group of people in order to establish a standard. When an individual takes the test, it then can be determined how far above or below the average is his or her score, relative to the normative group. It is important however, to know how the normative group was selected. For instance, for the standardisation of a test used to evaluate the instructional skills of trainers, a meaningful normative group has to consist of a large and representative sample of trainers. 18. Last but not least, it is also important to consider test difficulty, especially if results from different tests are compared to each other. For example, two tests can have similar properties in terms of reliability, but an easy test may not adequately assess the level of competency needed for the trainer job. If a test is made so easy that everyone passes it, there will inevitably be a number of individuals within the group who do not really have the competency to perform at the required standard. On the other hand, setting the test difficulty too high may result in only very few passing, maybe even too few to have the sufficient number of trainers needed for the training organisation. It is essential to carefully consider the level of difficulty as well as the pass mark level that is attributed to the components of the test.

4 Appendix APPENDIX EXAMPLES OF EVALUATION CRITERIA FOR ORAL PRESENTATION CERTIFICATION TESTS Delivery / Overall Impression Superior: Enthusiastic, poised, comprehensible, can be heard by all, interesting to audience Good: Moderately enthusiastic, comprehensible, generally can be heard, and moderately interesting Fair: Only mild enthusiasm, problems with comprehensibility, cannot be heard very well, not very interesting to audience Weak: No interest in presentation, barely comprehensible or incomprehensible Fluency Superior: Gets the idea across fully with little hesitation; goes beyond the minimum. Communicates with ease overall. Good: Communicates confidently using simple structures; some hesitation and false starts with more complex material. Evidence of fluency outweighs moments of uncertainty or stumbling. Fair: Consistently uses simple structures, vocabulary, etc. and avoids more complex material. Some signs of fluency, but hesitant performance and/or excessively simple language predominate. Weak. Use of simple structures is uncertain and hesitant. Little evidence of fluency despite moments of ease. Correctness Superior. Expression is as grammatically accurate as can be reasonably expected for this level. Good. Consistent accuracy with simple structures, few errors with more complex structures. Accuracy is in greater evidence than errors in grammar and usage. Fair. Minor errors with simple structures; less accurate use of more complex structures. Errors rarely impede comprehensibility. Weak. Inaccuracies in grammar throughout. Errors interfere with comprehensibility and force interpretation by the listener. Pronunciation Superior. Pronunciation is accurate throughout, with good rhythm and intonation for this level. Good. Accent is more "foreign" than that rated "15," but errors are rare. Fair. Pronunciation is generally comprehensible, with few errors. Rhythm and intonation deviate from that of fluent German, but do not interfere with communication. Weak. Major errors (e.g., terminal consonants, verb endings, etc.) are excessive for this level. Mispronunciations force the listener to interpret. Vocabulary Superior. Demonstrates broad command of vocabulary; knows and uses precise words for the situation. Good. Vocabulary is generally appropriate, despite limitations. No major inaccuracies in word choice. Fair. Vocabulary tends to be simple, no thought given to use of precise expressions for the context. Few or no major inaccuracies. Weak. Vocabulary is inadequate for this level. Length of Presentation Total: 100 Points Student Name: Total Score Text taken from Duke University

5 - 5 - Appendix CRITERIA Excellent Good Fair Needs Work Comments (1) ORGANISATION talk was well-prepared topic clearly stated structure & scope of talk clearly stated in introduction topic was developed in order stated in introduction speaker summed up main points in conclusion speaker formulated conclusions and discussed implications (2) CONTENT Speaker: was in control of subject matter covered what s/he intended to cover displayed knowledge of topic by referring to other sources (3) CONFIDENCE Speaker: looked relaxed and confident controlled nervous hand movements used appropriate body language established good eye contact initially maintained eye contact throughout talk was aware of the audience response

6 Appendix speaker s movement and posture were relaxed (4) SPEECH pace was varied and not too rushed volume was appropriate intonation varied & appropriate pronunciation articulation were clear and grammar accurate (5) VISUAL AIDS equipment handled with confidence introduced at appropriate times were legible did not contain too much information showed appropriate use of diagrams (6) TIMING/MODE OF DELIVERY talk was well-timed time was divided appropriately speaker spoke from notes speaker read text in spoken style (7) DISCUSSION MANAGEMENT Speaker used appropriate strategies to: initiate discussion control flow of questions and focus of discussion

7 - 7 - Appendix respond confidently deflect difficult or irrelevant questions END