Impact Assessment for Changes to Railway Industry Standards

Transcription

1 Impact Assessment for Changes to Railway Industry Standards Version: Final Date: June 2013 Relevant Railway Group Standard Title: Rail Industry Standard for Train Driver Selection Number: RIS-3751-TOM Issue: One Synopsis: The document is a voluntary standard, produced by RSSB at the request of the industry. It provides a standard on the selection process for train drivers who will drive on Network Rail managed infrastructure. Duty holders may choose to mandate it through internal instructions/procedures or contract conditions Executive Summary Issue one of RIS-3751-TOM Rail Industry Standard for Train Driver Selection, published in 2008, has undergone a review to evaluate whether the current requirements are fit for purpose as a result of research work undertaken by RSSB on train driver selection 1. The current train driver psychometric assessment process has been in place since the 1980s and does not take into account the demands of modern train driving or the anticipated future demands of new systems, such as European Rail Traffic Management System (ERTMS). A review of RIS-3751-TOM issue one has identified a number of weaknesses in the psychometric assessment process (detailed in Appendix A). The process does not at present cover in sufficient detail the ability for a driver to maintain attention and vigilance. These skills are becoming increasingly important due to the introduction of more automation and driver assistance systems into the train driving task. The current process does not cover all of the selection criteria that are required by the Train Driving Licences and Certificates Regulations (TDLCR, 2010). Railway undertakings (RUs) are already required to comply with TDLCR for new cross-border drivers and will be required to comply with TDLCR for all new domestic drivers from 29 October In order to respond to these changes, a review of the psychometric assessment process has resulted in the selection criteria and psychometric assessment methods in Appendix A being revised. At the same time, all other requirements have been reviewed. Some requirements in issue one of the RIS are not to be taken forward to issue two of the RIS and are withdrawn. Other requirements have been introduced or revised to assist the industry with a clear consistent approach more suitable for modern assessment centre practices. 1 T948 Train Driver Selection RSSB Impact Assessment for RIS-3751-TOM 1

2 In addition, some changes have been made to remove requirements which are duplicated unnecessarily and to improve the guidance and consistency of terminology. These changes do not require RUs to do anything differently. The proposed changes require some initial investment in railway assessment centres to transition from the current process to the new process. After this, the changes will have negligible impact on on-going costs and neutral impact on safety and are therefore considered to be in the long term best interests of the main line railway. 1 Initiators of change A standardised psychometric assessment process for train drivers was first introduced in 1988 by British Rail. The purpose of the psychometric assessment process is to measure whether candidates meet a safe minimum standard for train driving. The psychometric assessment process consists of a set of selection criteria which are considered relevant to train driving, methods of assessment and associated scoring rules for each criterion. The scoring rules have undergone minor changes on several occasions since 1988, but the overall process is still recognisable as the original. In 2005, RSSB undertook some research to look at the current train driver selection process, to evaluate if the current requirements set out in issue one of RIS-3751-TOM Rail Industry Standard for Train Driver Selection, are fit for purpose 2. The research identified that the current selection criteria do not fully take into account the demands of modern train driving or the anticipated future demands of new systems, such as ERTMS. The process does not at present cover in sufficient detail the ability for a driver to maintain attention and vigilance. These skills are becoming increasingly important due to the introduction of more automation and driver assistance systems into the train driving task. The more recent TDLCR (2010) legislation imposes particular selection criteria that the current process does not cover. The process therefore needs to be updated so that it is fully compliant. Railway undertakings (RUs) are already required to comply with TDLCR for new cross-border drivers and will be required to comply with TDLCR for all new domestic drivers from 29 October The 2005 research and subsequent reviews 3 have shown that the current methods of assessment could be improved to provide more effective measurement of the relevant skills and aptitudes. Two RSSB research projects on how the process should be updated have now been completed and provide recommendations for how to take the process forward 4. The new draft of RIS-3751-TOM incorporates these recommendations. 2 T340 Psychometric testing - a review of the train driver selection process 3 Annual health check review of the train driver assessment process. 4 T948 Train Driver Selection Implementation Phase RSSB Impact Assessment for RIS-3751-TOM 2

3 Section 2 of this impact assessment details the changes proposed from issue one to issue two of the RIS. A summary of the changes are included in Annex 1 of this impact assessment. 2 Summary of changed controls and implications for industry 2.1 New requirements Re-assessment of failed candidates Issue one of the RIS provides guidance on what is required when an applicant has failed the assessment. It is proposed to strengthen this guidance as a new requirement in issue two of the RIS. RUs that choose to adopt the RIS must not reassess a failed applicant within six months of sitting their first assessment and must not allow candidates more than one reassessment unless there are mitigating circumstances which have been considered and documented. Review of the psychometric assessment process In issue two, it is proposed to clarify the psychometric assessment review process by introducing a new requirement where RUs must consider the reliability, validity and fairness of the psychometric assessment methods. Previously, the content of the review process was not well defined. In addition, the scope of the review process has been clarified (see section 2.2 Revised requirements). Results of the psychometric assessment process Issue one of the RIS provides guidance stating that results from the assessment centre should be categorised as pass or fail against a minimum acceptable standard for recruitment as a potential train driver. It is proposed to introduce this requirement in issue two of the RIS where RUs that choose to adopt the RIS must assess each candidate against a standard scoring criterion to make a judgement on whether the candidate has passed or failed. 2.2 Revised requirements Evaluations of assessments The current requirement in issue one of the RIS sets out what railway undertakings should consider when periodically reviewing their recruitment and selection arrangements. It is proposed to revise the requirement in issue two of the RIS such that RUs must review only their psychometric assessment processes periodically to evaluate the suitability of the assessment methods. RSSB Impact Assessment for RIS-3751-TOM 3

4 Psychometric Assessments (Appendix A) In order to achieve compliance with TDLCR (2010) and deliver improvements to the assessment process, the selection criteria and assessment methods in Appendix A of issue two of the RIS have been revised. It is proposed to withdraw the following assessment methods from issue two of the RIS: Candidate demographic questionnaire. Criterion Based interview (CBI). Computerised DTG. Computerised Group Bourdon. It is proposed to introduce the following assessment methods in issue 2 of the RIS as the research project T948 Train Driver Selection has recommended that they will provide a more appropriate and effective assessment process: ATAVT (Adaptive Tachistoscopic Traffic Test) which measures the ability to anticipate elements in a traffic environment and make a correct decision about how to respond. WAFV (Vigilance test) which measures the ability to attend and respond to stimuli which occur relatively infrequently and over extended periods of time. 2HAND (Hand coordination test) which measures the ability to make appropriate and controlled movements in response to decisions about complex stimuli. TEA-Occ (Test of Everyday Attention Occupational) which measures the ability to switch attention between sources and information for example lineside information and in-cab information and perform different tasks in parallel, eg making rain announcements while on the move. SJE and MMI (Situational Judgement Exercise and Multimodal Interview) which measures conscientiousness, dealing with challenging situations and tolerance for low stimulation. Transfer of Safety Information (Appendix B) The current form in Appendix B in issue one of the RIS, may be used when requesting information on an applicant with previous experience as a train driver from a current or previous employer. The form is not very clear, and does not clearly set out how to complete it. RSSB Impact Assessment for RIS-3751-TOM 4

5 It is proposed to update the form in Appendix B in issue two of the RIS, by making some cosmetic changes to the structure and layout of the form, and providing additional instructions on how to complete it. 2.3 Withdrawn requirements Applying, reviewing and validating the selection criteria and selection process Issue one of the RIS requires RUs to employ persons chartered by the British Psychological Society, or possessing an equivalent European Union national standard qualification to apply, review and validate the selection criteria and selection process. It is proposed to withdraw this requirement in issue two of the RIS as the revised psychometric assessment process proposed in issue 2 has already been reviewed by individuals who are chartered by the British Psychological Society. Review of the selection process and feedback from training staff The current requirement in issue one of the RIS sets out what RUs must do when reviewing reports from training staff and train driver managers following the performance of drivers selected, to review their selection process. It is proposed to withdraw this requirement in issue two of the RIS as the requirement does not deliver its intention. 3 Impacts 3.1 New requirements Re-assessment of failed candidates Guidance provided in issue one of the RIS for railway undertakings not to assess an applicant who has failed the assessment within six months of the initial assessment and not to permit more than one reassessment, is proposed to be introduced as a requirement in issue two of the RIS. Currently, RUs who have adopted the RIS into their company management system (CMS) record the test results of applicants in the Rail Assessment Centre Forum (RACF) database. This allows other RUs to cross reference any applicant to ascertain if they have failed the process within the last six months or more than once. These guidance items are already treated by most RUs as mandatory and there is a clear desire for consistency. In addition, RSSB receives lots of enquiries about these clauses from assessment centres and candidates. This suggests that their current status as guidance creates some ambiguity in how they should be applied. Therefore, a clear way forward is to introduce them as requirements to avoid any confusion and to reinforce a consistent approach which will have long term benefits. RSSB Impact Assessment for RIS-3751-TOM 5

6 There are some positive financial impacts for RUs who comply with the requirements as they remove the unnecessary costs of assessing candidates who are not eligible to be reassessed through checking the database, knowing that the information of failed or passed candidates has been processed correctly. RUs will not be required to do anything differently as a result of choosing to mandate these requirements in issue two of the RIS, and there are no foreseen negative safety impacts as it is current industry practice to record and cross reference details using the database. Review of the psychometric assessment process It is proposed to introduce a new requirement c to in issue two of the RIS, where RUs must consider the reliability, validity and fairness of the psychometric assessment process. The new requirement provides clarification for what should be included in such a review according to good practice. There are no negative financial or operational impacts because RUs that follow the assessment process in the RIS can take advantage of information provided by RSSB to undertake the review. No negative safety impacts have been foreseen as a result of introducing this requirement in issue two of the RIS. Results of the psychometric assessment process Guidance is provided in issue one of the RIS for RUs to determine if a candidate has passed or failed according to if they have achieved a minimum acceptable standard. In practice, this is achieved using a standard scoring criteria sheet. This guidance item is already treated by most RUs as mandatory and there is a clear desire for consistency. The proposed new requirement in issue two of the RIS will not require RUs to do anything differently because the only change is to emphasise the importance of this by making it a requirement instead of guidance. There are no foreseen negative safety impacts as it is current industry practice to determine pass/fail against a minimum standard. 3.2 Revised requirements Evaluations of assessments The current requirements in issue one of the RIS set out what RUs must do when considering a review of their recruitment and selection arrangements and to periodically evaluate the validity of selection criteria and procedures for their operations which include advertising of roles through to selection. It sets out what RUs must consider when periodically reviewing their wider recruitment and selection arrangements. The requirement to periodically review recruitment and selection arrangements, in issue one, places additional burden on RUs as they are best placed to determine what their recruitment RSSB Impact Assessment for RIS-3751-TOM 6

7 and selection process should include, and how often those arrangements should be reviewed. The psychometric assessment process is the only aspect of recruitment and selection where there is benefit in standardisation across the industry. It is proposed to clarify in issue two of the RIS that RUs must review their psychometric assessment process periodically to evaluate the suitability of the assessment methods, which is only one element of the wider recruitment and selection process. The proposed revised requirement in issue two of the RIS has a neutral impact on safety as RUs will be required to review their psychometric assessment process and this is the key part of recruitment that relates to safety. The proposed revised requirement in issue two of the RIS has a positive impact on operations as the psychometric assessment process is an efficient way to measure underlying abilities which cannot be observed during a traditional interview. RUs will no longer be required to periodically review their recruitment and selection processes, but will now be required only to review their psychometric assessment processes. Psychometric Assessments (Appendix A) TDLCR (2010) will come into force for new domestic drivers on 29 October TDLCR requires additional selection criteria to be assessed. Therefore, in order to help prepare RUs to comply with this legislation, it is proposed to revise Appendix A in issue two of the RIS. There are no financial or safety impacts as a result of withdrawing the four psychometric assessment methods as highlighted in section 2 of this impact assessment because they will be replaced with an updated assessment process that is compliant with TDLCR, and is more comprehensive and more effective. Subject matter experts who are chartered by the British Psychological Society have been involved in the design and comprehensive evaluation of the new assessment methods proposed in issue two of the RIS, which cover the key skills required for modern train driving. As all potential drivers will be assessed against a standard psychometric assessment, this has a potential cost saving for RUs by not requiring them to employ a person chartered by the British Psychological Society to design their own psychometric assessment process. Assessment centres will require some additional hardware and software in order to transition to the new psychometric assessment process in Appendix A. There will be one-off capital costs for items such as computers, response panels and software licences. After the transition the running costs of the assessment centres are expected to be broadly comparable, examples of these transition costs are shown in Annex 2 of this impact assessment. RUs that use the services of an external assessment centre for the train driver psychometric assessment process may find that the price of this service changes as a result of the introduction of the new assessment process. RSSB Impact Assessment for RIS-3751-TOM 7

8 There are no foreseen negative safety impacts as a result of the changes to Appendix A of issue 2 of the RIS. On the contrary, the improvements that have been made to the psychometric assessment process should provide a benefit in terms of the quality of candidates who are eligible to be recruited as train drivers. Transfer of Safety Information (Appendix B) The form in Appendix B in issue one of the RIS, does not currently set out how to complete it, and the layout and structure is not very clear. Feedback from members indicates that the form is often completed incorrectly. The proposed revised form in Appendix B in issue two of the RIS contains additional instructions in how it should be completed. A number of cosmetic changes to the layout and structure have been made to make it clearer what information should be completed by each party. Where there is a significant overlap with the information contained on a train driving certificate, this is indicated so that candidates and RUs do not have to enter information more than once if the candidate in question already has a train driving certificate. The content of the form in terms of what information is requested has not changed significantly. RUs are not required to do anything differently except for using the new form as an alternative to the old form. There are no foreseen negative safety impacts as far as these changes are concerned. The redesign of the form should result in fewer errors in completing it. 3.3 Withdrawn requirements Applying, reviewing and validating the selection criteria and selection process The selection criteria and selection process in issue one of the RIS require RUs to employ persons chartered by the British Psychological Society, or possessing an equivalent European Union national standard qualification to apply, review, judge and validate the selection criteria and selection process. The criteria and processes including the psychometric assessment process set out in issue two of the RIS, have been validated by qualified persons chartered by the British Psychological Society. Therefore, it is proposed to withdraw the requirement in issue two of the RIS for RUs to employ someone chartered by the British Psychological Society to validate the selection criteria and selection process because it would be a duplication of work that has already been done. Therefore, the withdrawal of this requirement has a potential cost saving for RUs. There are no foreseen negative safety impacts as a result of the withdrawal from issue two of the RIS. RSSB Impact Assessment for RIS-3751-TOM 8

9 Review of the selection process and feedback from training staff The current requirements in issue one of the RIS state that RUs should take account of reports from training staff and train driver managers following the performance of drivers selected to review their selection process. The scope of the required review has now been reduced to cover only the psychometric assessment process. Therefore, it is proposed to withdraw the requirement to use reports from training staff and driver managers because they are not suitable sources of evidence to make a judgement about the quality of the psychometric assessment process. The withdrawal of this requirement has some potential operational impacts and cost savings through the time saved by no longer requiring RUs to review reports from training staff or train driver managers. There are no foreseen negative impacts as a result of the withdrawal of this requirement in issue two of the RIS. 4 Compliance with decision framework The proposed new and revised requirements in issue two of the RIS require railway undertakings to record details in a database and access records of applicants to help them identify if a candidate has previously failed the assessment and if they are eligible to be reassessed. This requires co-operation between two or more duty holders. The requirements proposed to be withdrawn comply with the Decision Framework and are justified. The way forward for these requirements therefore satisfies the relevant criteria in the Standards Manual. The requirements in issue two of the RIS are provided for adoption by members of RSSB. The RSSB member may adopt any or all of the requirements through company procedures or contract conditions. Where this is the case the member of RSSB will specify the nature and extent of application. The requirements do not fulfil the criteria to be either a National Technical Rule (NTR) or a National Safety Rule (NSR) and therefore cannot be mandated in the Railway Group Standard. The proposed revisions to requirements set out in section 3 of this document allow for operational benefits and have a positive or neutral impact on safety. RSSB Impact Assessment for RIS-3751-TOM 9

10 Annex 1 - Summary of the changes RIS-3751-TOM Issue 1 Clause c Way forward Withdrawn Withdrawn Withdrawn Withdrawn Withdrawn Withdrawn New Reason and way forward RUs adopting the RIS issue 2 will not be required to employ additional persons to evaluate, judge, validate selection criteria and selection processes because the process specified in issue 2 has already been sufficiently evaluated. Does not deliver its intention because the scope of the review has been changed. Reports from training staff and driver managers are not a suitable source of evidence to review the psychometric assessment process. New and which were previously given as guidance are proposed to be introduced as a requirement to make it clear that all candidates are assessed against the same scoring criteria and that RUs are required to record each score. - New New and which were previously given as guidance are proposed to be introduced as a requirement as part of the selection process to reinforce the consistency of practice across the industry Appendix A Appendix B New Revised Revised Revised Now in issue 2 A new bullet C has been introduced to provide clarity on what railway undertakings should include when reviewing their psychometric assessment process. Now in issue 2 Revised to clarify that RUs are not required to have their wider recruitment and selection arrangements periodically evaluated. RUs will now be required to have only their psychometric assessment process evaluated instead. The selection criteria and psychometric assessment methods have been enhanced to comply with TDLCR 2010 and to provide a more effective assessment process overall. Cosmetic changes to the structure and layout of the form. Some additional instructions added in how to complete it. RSSB Impact Assessment for RIS-3751-TOM 10

11 Annex 2 Example Implementation Costs for Railway Assessment Centres RUs that have in-house assessment centres will require some additional hardware and software in order to transition to the new psychometric assessment process. Estimates of these costs are provided below. Please note that these costs are based on initial estimates from potential suppliers rather than formal quotations and would be subject to change when the process is implemented. They should be treated as approximate but they provide enough information for assessment centres to calculate an indicative implementation cost. Fixed costs per assessment centre: All assessment centres who implement the new psychometric assessment process will need Vienna Test System software to administer the WAFV, ATAVT and 2HAND assessment methods. This costs approximately 937/ 750 (converted at a rate of 0.8 to nearest 1). All assessment centres will require a licence for the Situational Judgement Exercise Software. For assessment centres who are not members of RSSB this will cost approximately 2000 for five years. It will be free to assessment centres run by members of RSSB. Each assessment centre will require an internet connection and a printer in order to administer the Situational Judgement Exercise. The test licence required for the DTG costs approximately per year and will no longer be required. Fixed costs per assessment terminal: Each assessment centre will need internet connected and compatible computer terminals to administer the computerised tests to candidates. Each assessment centre will need to determine how many terminals they require depending on how many candidates they want to assess at one time, space, budget etc. Product Table 1 Approximate costs per assessment terminal Price ( ) converted at a rate of 0.8 to nearest Price ( ) WTS-D Test System Dongle Ue - Response Panel Universal H-Set-E Stereo USB Headset for adults Total RSSB Impact Assessment for RIS-3751-TOM 11

12 Upgrading computer hardware will only be necessary if the existing computers do not meet the specification required (See Table 2). Costs for replacement computer hardware are not provided because it is assumed that RUs have their own preferred suppliers for computer equipment and that, if necessary, assessment centres would be replacing existing equipment that does not currently meet a modern specification. Table 2 Required computer specification Computer technical specification PC or laptop with Pentium CPU or compatible (e.g. Celeron, Athlon), at least 2,5 Ghz At least 2 GBytes (MB) of RAM Display adapter with 24 or 32-bit color depth (16 million colours) DVD drive, hard disk, mouse, keyboard USB ports for license dongle and peripheral hardware (in case all USB ports on the PC are used up a USB hub with external power supply is required) Network connection (e.g. for the installation of a Vienna Test System Network) Windows XP (Service Pack 3 necessary), Windows 7 (32 or 64 bit) No programs which can interfere with the test execution (e.g. by heavy CPU usage or on-screen presentations) installed on the computer Monitor At least 19" CRT or LCD colour monitor with a resolution with at least 1280 x 720 pixels. A refresh frequency of at least 75 Hz has to be set for CRT monitors. Cost per assessor: Each assessor will be required to attend a four day training course in order to be able to administer the new Situational Judgement Exercise and Multimodal Interview. The cost would be in the region of 1500 per person but is yet to be finalised. Note that it is expected to be lower than the training which is required for the Criterion Based Interview which is being replaced. The number of assessors to train is a decision for each assessment centre to take. RSSB Impact Assessment for RIS-3751-TOM 12

13 Cost per candidate: The current assessment process costs the assessment centre approximately per candidate. The cost of the new assessment process per candidate will depend on the number of candidates assessed but is expected to be comparable to or lower than the current process. RSSB Impact Assessment for RIS-3751-TOM 13