Participant Workbook TAEDES505 Evaluate a training program 1 st Edition 2017 Part of a suite of support materials for the TAE Training and Education Training Package
Acknowledgement Innovation and Business Industry Skills Council (IBSA) would like to acknowledge Precision Consultancy for their assistance with the original development of the resource for TAEDES505A. This resource has been updated for TAEDES505 by IBSA. Copyright and Trade Mark Statement 2017 Innovation and Business Industry Skills Council Ltd All rights reserved. Apart from any use permitted under the Copyright Act 1968, no part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, or otherwise, without written permission from the publisher, Innovation and Business Industry Skills Council Ltd ( IBSA ). Use of this work for purposes other than those indicated above, requires the prior written permission of IBSA. Requests should be addressed to the Product Development Manager, IBSA, Level 11, 176 Wellington Pde, East Melbourne VIC 3002 or email products@ibsa.org.au. Disclaimer Care has been taken in the preparation of the material in this document, but, to the extent permitted by law, IBSA and the original developers do not warrant that any licensing or registration requirements specified in this document are either complete or up-to-date for your State or Territory or that the information contained in this document is error-free or fit for any particular purpose. To the extent permitted by law, IBSA and the original developers do not accept any liability for any damage or loss (including loss of profits, loss of revenue, indirect and consequential loss) incurred by any person as a result of relying on the information contained in this document. The information is provided on the basis that all persons accessing the information contained in this document undertake responsibility for assessing the relevance and accuracy of its content. If this information appears online, no responsibility is taken for any information or services which may appear on any linked websites, or other linked information sources, that are not controlled by IBSA. Use of versions of this document made available online or in other electronic formats is subject to the applicable terms of use. To the extent permitted by law, all implied terms are excluded from the arrangement under which this document is purchased from IBSA, and, if any term or condition that cannot lawfully be excluded is implied by law into, or deemed to apply to, that arrangement, then the liability of IBSA, and the purchaser s sole remedy, for a breach of the term or condition is limited, at IBSA s option, to any one of the following, as applicable: (a) (b) if the breach relates to goods: (i) repairing; (ii) replacing; or (iii) paying the cost of repairing or replacing, the goods; or if the breach relates to services: (i) re-supplying; or (ii) paying the cost of re-supplying, the services. Published by: Innovation and Business Industry Skills Council Ltd Level 11, 176 Wellington Pde East Melbourne VIC 3002 Phone: +61 3 9815 7000 Fax: +61 3 9815 7001 Email: products@ibsa.org.au www.ibsa.org.au ISBN: 978-1-925460-01-8 Stock code: TAEDES5051W First published: 2016 1 st edition version: 2 Release date: 2017
Table of Contents Welcome Introduction...1 Sections in this workbook...1 Using this workbook...2 Recommended reading...2 Learning Undertaking assessment...3 Topic 1: Plan the Evaluation...7 What is an evaluation?...7 Identify needs for the evaluation...7 Determine the evaluation s aims and scope...8 Indicators of success... 10 Data collection methods... 11 Consider the evaluation model... 16 Write the methodology... 21 Plan to communicate with stakeholders... 23 Determine milestones and a timeline... 25 Manage risk... 25 Topic 2: Collect Evaluation Evidence... 26 Develop data collection tools... 26 Access to data Workplace visits and meetings... 31 Collate and process data... 31 Rules of evidence... 32 Topic 3: Analyse Evaluation Evidence... 34 Organising the evidence... 34 Assessing the evidence... 36 Topic 4: Report on the Evaluation... 40 Preparing to write the report... 40 Drafting data, analysis and recommendations... 40 Drafting the remaining material... 41 Discuss with stakeholders... 42
Application Activity 1: The Aim and Scope of Your Evaluation... 47 Activity 2: Policy, Legislation, Codes of Practice and National Standards... 49 Activity 3: Gathering Qualitative and Quantitative Data... 51 Activity 4: Kirkpatrick Reaction and Learning... 53 Activity 5: Stakeholder Management Skills... 56 Activity 6: Evaluation Outcomes Identifying Other Issues... 57 Activity 7: Prepare an Evaluation Plan... 58 Activity 8: TTIC Training Solutions Case Study... 59 Activity 9: Impact on Workplace Capabilities... 62 Activity 10: Propose Changes to Improve Processes... 64 Activity 11: Evaluation Report... 66 Appendices Appendix 1: Simple Evaluation Plan... 69 Appendix 2: Detailed Evaluation Plan... 71 Appendix 3: Sample Evaluation Report... 75
Welcome
Participant Workbook Introduction This Participant Workbook supports TAEDES505 Evaluate a training program. This unit is an elective unit in the TAE50116 Diploma of Vocational Education and Training and a core unit in the TAE50216 Diploma of Training Design and Development. It covers the performance outcomes, skills and knowledge required to evaluate a training program in order to measure the effectiveness of training in meeting workforce performance needs and capability requirements. In a regulatory context, it contributes to the continuous improvement cycle within a registered training organisation (RTO). This unit will help you develop skills and knowledge to enable you to: plan and prepare evaluation collect and collate evaluation data and information for a training program analyse evaluation data and information and make conclusions report on the conclusions and recommendations of the evaluation. Sections in this workbook This Participant Workbook consists of the following sections. Learning This section provides background information to support this unit of competency, and where to go for further information. It also includes some case studies to illustrate the unit of competency in practice. Throughout this section, you will notice small text included at the sides of some pages. This side text provides definitions, notes, resources or it pinpoints the best time to carry out a learning activity. The activities referred to are included in the following section of the workbook, titled Application. Application This section provides activities to help you develop your skills. Your facilitator may also add some extra activities to support your learning as you are working through this unit. Some of the activities may contribute to assessment. Appendices The Appendix section of the workbook may provide templates, examples or other sample documents to help illustrate some aspects of the unit of competency. TAEDES505 Evaluate a training program 1 st edition version: 2 Innovation and Business Industry Skills Council Ltd Page 1 of 78
Introduction Using this workbook Templates Helpful templates from the appendices of your Participant Workbook are available in Word format online at <https://www.ibsa.org.au/tae-resources>. References Throughout the workbook you will be directed to websites for additional information. Given that web addresses can change, you may need to check the link and use search tools to find updated links. Glossary A glossary of terms is included on the TAE Resources website at <https://www.ibsa.org.au/tae-resources>. Use the glossary to clarify the meaning of terms used in this workbook. Context To complete this unit you need a learning and assessment environment where you can access information and apply your skills. This environment is called the practice environment. If you are already employed in an organisation that delivers training and assessment, your workplace will most likely be your practice environment. If not, your facilitator can help to find you a suitable practice environment. If you do not have direct access to a practice environment, your facilitator may need to help you by providing simulations or case studies that meet the assessment requirements for this unit. For example, you could practice applying your competencies on fellow learners in the classroom, or your facilitator may set up a virtual learning community online. Throughout this workbook, these environments will be referred to as your practice environment. Recommended reading Some recommended reading for this unit includes: Denzin, N., 2006, Sociological methods: a sourcebook, 5th edn, Transaction Publishers, Herndon, Virginia. Feuerstein, M., 1986, Partners in evaluation: evaluating development and community programmes with participants, Macmillan Education Ltd, London. Hayes, M., February 2003, Just who s talking ROI?, Information week. Kirkpatrick, D., 1994, Evaluating training programs: the four levels, Berett-Koehler Publishers Inc., San Francisco. 1 st edition version: 2 TAEDES505 Evaluate a training program Page 2 of 78 Innovation and Business Industry Skills Council Ltd
Participant Workbook Please note that any URLs contained in the recommended reading, learning content and learning activities of this publication were checked by IBSA and correct at the time of publication. Note, however, that IBSA cannot vouch for the ongoing currency of URLs. Every endeavour has been made to provide a full reference for all web links. Where URLs are not current, we recommend using the reference information provided to search for the source in your chosen search engine. Undertaking assessment Assessment of this unit must demonstrate that you can plan and undertake an evaluation of a training program and produce an evaluation report and a critical review of the evaluation process. You need to show that you have the skills and knowledge to: interpret training program requirements plan the evaluation study develop resources to support the evaluation process collect and analyse relevant data and information determine appropriate improvements to the evaluation study. You must provide evidence that you have: evaluated a training program against workforce performance needs and capability requirements produced an evaluation report that addresses all phases of the evaluation process including recommendations on areas of improvement critically reviewed the evaluation process and approaches taken, and proposed changes to improve the process. This unit might be assessed individually or together with other units in the TAE Training and Education Training Package. Taking a portfolio approach to assessment can make the assessment process more valuable for you and will often better reflect processes in the workplace. Using a portfolio approach, you will gather your evidence together for this unit, and perhaps for other units, and submit it all together as a portfolio. Activities to be undertaken as assessment for this unit of competency are located in a separate document. TAEDES505 Evaluate a training program 1 st edition version: 2 Innovation and Business Industry Skills Council Ltd Page 3 of 78
Learning
Participant Workbook Topic 1: Plan the Evaluation What is an evaluation? An evaluation is a systematic, planned effort to examine how effectively a program (such as a training program) has achieved its objectives. An evaluation is undertaken to provide insight into how a program is conducted, in order to see what works and what doesn t work. This provides assurances about how well money has been used in a program, and enables future programs to be better planned and implemented. Identify needs for the evaluation Trainers and assessors know the value of evaluation; they spend much time and effort considering methods, developing tools, and delivering training and assessment. They want to know what works, what doesn t work, and why. Ultimately, they want to know that learners actually learnt, what learners aimed to learn, and what they are able to apply in the workplace. Other stakeholders have just as much of an interest in evaluating a training and assessment program. At the outset, the evaluator should ensure they understand what stakeholders want from the evaluation. The table below shows some things that different stakeholders may want from an evaluation. Stakeholders interests in an evaluation Trainers, assessors and RTOs to find out if, how and why training and assessment processes, methods and tools were successful or unsuccessful ideas to improve their processes, methods and tools whether participants, managers and supervisors found the program to be effective and worthwhile, and whether participants are using what they learnt on the job for the evaluation to contribute to the RTO s continuous improvement program TAEDES505 Evaluate a training program 1 st edition version: 2 Innovation and Business Industry Skills Council Ltd Page 7 of 61
Topic 1: Plan the Evaluation Learners and graduates to find out how effective the training and assessment was to be involved in improving the training and assessment to have their voice heard in the evaluation to know how the evaluation has improved the training and assessment Workplace managers and supervisors Employer organisations and funding bodies to know what the training and assessment achieved, and if it is worthwhile for other employees to know what to expect from trained employees to use the results of training to make changes in the workplace to have evidence on which to base decisions about continuing, altering or stopping the training and assessment to know how funds for training and assessment were spent to know if the training and assessment met clients goals to know what the outcomes of the training were to see if the processes, methods and tools can be used in workplaces of different industries to improve how the industry measures return on training investment Determine the evaluation s aims and scope As we saw above, stakeholders want particular things from an evaluation. This is why you conduct it: to provide those involved with information about how well (or otherwise) things went, so they can consider the next steps on their particular journey. That might be: for RTOs, trainers and assessors to improve their processes, methods and tools for learners and graduates to improve the course and its reputation, and to improve their job and career prospects for workplace managers and supervisors to maximise the productivity and profit return on their training and assessment investment for funding bodies and employer organisations to justify funds spent and ensure training meets industry needs. 1 st edition version: 2 TAEDES505 Evaluate a training program Page 8 of 78 Innovation and Business Industry Skills Council Ltd
Participant Workbook At the broadest level, an evaluation examines: what the program achieved how well the program contributed to the organisation s goals and objectives what worked in the training program and what didn t, and why any unintentional outcomes, and why they occurred lessons for future programs. To dig deeper than this broad level, you should talk to stakeholders about exactly what they want from the evaluation, both in terms of outcomes and types of data and analysis. This will not only maximise the relevance and usefulness of your findings, but also make it easier to engage (and thus collect information from) stakeholders. You then derive the aims of the evaluation by listing, grouping and prioritising what stakeholders want from the evaluation, considering that: some stakeholders may want the same, or similar, things from the evaluation, while others might have unique interests some stakeholder needs may be more important than others it may be too costly or impractical to satisfy some of the needs some needs might be satisfied through methods other than a formal evaluation. You will find that this exercise creates groups of needs which can be summarised by a high-level statement (which becomes the aim). Common evaluation aims are to: determine how effectively the training met workforce needs estimate how cost-effective the training was identify improvements to training or assessment processes, methods or tools. But don t discard ideas you generated to develop the aim; they will be essential when you plan the evaluation. The scope of the evaluation might include: all or part of the RTO s training or assessment processes particular training or assessment processes, methods or tools arrangements with other organisations (such as enterprises, industry associations or public sector agencies) clients needs. Now is a good time to complete Activity 1: The aim and scope of your evaluation. TAEDES505 Evaluate a training program 1 st edition version: 2 Innovation and Business Industry Skills Council Ltd Page 9 of 61
Topic 1: Plan the Evaluation Indicators of success Look at the common evaluation aims in the section above; for example, the aim determine how effectively the training met workforce needs. What exactly would you be looking for here? Greater adherence to procedures? Less absenteeism? Increased productivity? Each of these things is an indicator of the success of the program. According to Feuerstein: An indicator is a marker. It can be compared to a road sign which shows whether you are on the right road, how far you have travelled and how far you have to travel to reach your destination. Indicators show progress and help measure change. 1 Indicators point you toward the data you need to gather during and after the program. Indicators can be categorised as: process indicators which address how well the training and assessment is conducted, in terms of: implementation (e.g. the effectiveness of the methods and tools) reach and scope (e.g. the number of trainees or the diversity of participants in terms of age, gender, ethnicity or industry) quality (e.g. the percentage of participants satisfied with the training and assessment) outcome indicators which address the extent to which the client s objectives for the training have been achieved by changes in the knowledge, skill and behaviour of participants. All of these indicators link back to broad ideas of what constitutes quality in the provision of training and assessment services. Quality is a key component of the regulations covering the operation of RTOs. In Australia, training and assessment organisations are regulated by the Australian Skills Quality Authority (ASQA) or, in some cases, state regulatory authorities. One of the standards that RTOs must meet is a systematic approach to continuous improvement of training and assessment strategies. The best way to develop and implement a continuous improvement strategy is to: 1. investigate mechanisms that could be implemented to attempt improvements 2. implement one or more of those mechanisms 3. evaluate the results of those mechanisms 4. determine what is happening now. 1 Feuerstein, M., 1986, Partners in evaluation: evaluating development and community programmes with participants, Macmillan Education Ltd, London. 1 st edition version: 2 TAEDES505 Evaluate a training program Page 10 of 78 Innovation and Business Industry Skills Council Ltd
Participant Workbook Evaluation may include research into the organisational environment as well as a specific training program. For instance, a program delivered in partnership with a company undergoing significant change may have a number of challenges for those trainers and assessors involved. Participants may be less likely to commit to the program if they are unsure about their job futures. Now is a good time to complete Activity 2: Policy, legislation, codes of practice and national standards. Data collection methods To plan your evaluation project, you need a working understanding of the various data collection methods you might be able to use. Qualitative and quantitative research There are two types of research: qualitative and quantitative. Qualitative research Involves the collection of verbal (oral or written) data Quantitative research Involves the collection of numerical data Qualitative research collects subjective data (such as opinions, beliefs and first-person accounts) that is of sufficient quality to be useful to the evaluation. Quantifying the data (for example, finding out what percentage of people agree with a particular opinion) is not immediately important in qualitative data. It is data derived from observation, not measurement. Quantitative research involves collecting numerical data and is useful for its ability to be statistically analysed (such as, 73% of people agree with opinion X ). These two types of research use different types of questions. Qualitative research Quantitative research Uses open-ended questions invites participants to respond freely (for example, What did you like most about the training program?, to which responses might be, The material was relevant to my work, The facilitator engaged everyone or It was interesting working with people from different industries ) Uses closed-ended questions limits participants responses to a range of pre-set options (which allows you to report, for example, 87% of respondents thought the training content was highly relevant to their work, 9% of respondents thought the training had no impact on their work and 4% of respondents thought the training negatively impacted their work). TAEDES505 Evaluate a training program 1 st edition version: 2 Innovation and Business Industry Skills Council Ltd Page 11 of 61
Topic 1: Plan the Evaluation The table below shows examples of qualitative and quantitative data. Examples of qualitative and quantitative data Qualitative (open-ended) Employees thoughts about their skills and abilities. Employees descriptions of workplace incidents and injuries. Quantitative (closed-ended) Answers on a survey with ratings from 1 5. Number of employee injuries. Now is a good time to complete Activity 3: Gathering qualitative and quantitative data. Employees feelings about a change in their workplace. Employees suggestions about their training needs. Number of sick days employees took in the last year. Retention rate of staff. It is common to combine qualitative and quantitative questioning. For example, you might conduct a group discussion from which 30 ideas might arise. You might decide that there are five ideas that you know can t be implemented and you don t want to report on; 10 ideas that are subsets of other ideas; and five ideas that are low priority. That would leave you with 10 ideas that you felt it was important to get statistics about. You might then survey the whole workforce to determine what percentage of employees supported each idea. Equally, it is uncommon to write a quantitative research tool (such as a questionnaire with multiple-choice questions or ranking scales) without first doing qualitative research. If you did start with quantitative research, where would you get your questions from? How would you assure yourself that they represented the key issues of concern to participants, rather than your own interests and concerns? Qualitative research aims to identify the possibilities; quantitative research aims to put numbers against the possibilities most related to the project goals. It is not uncommon to only do qualitative research. For example, you might conclude that an opinion strongly agreed to in a group discussion of a representative sample of people is widely held among the workforce. Common data collection methods The following table shows you the main ways that you can collect data, and some advantages and disadvantages of each. 1 st edition version: 2 TAEDES505 Evaluate a training program Page 12 of 78 Innovation and Business Industry Skills Council Ltd
Participant Workbook Data collection methods Method Description Advantages Disadvantages Surveys (qualitative and quantitative) Focus groups (qualitative) Surveys ask participants a set of pre-determined questions. Surveys can be completed in hard copy (to complete while you watch, or respondents given a deadline to complete them), by telephone, by email, or online via internet or intranet. For example, a survey can be completed by staff and managers to compare their perspectives on the same issues. A group of up to 10 participants meet, with a facilitator who guides discussions according to a discussion point schedule. Can be used with a few, or many, respondents. Can collect confidential information if you promise anonymity. Can get qualitative (open-ended questions) and quantitative (ranking scale, yes/no questions) data. Easy to set up using free software (such as SurveyMonkey). Email and internet surveys are quick and cheap (however, processing qualitative responses takes time) and respondents can be anywhere in the world (so long as they have internet access). A rich source of qualitative data; all main viewpoints can be identified. Allows for exploration of complex issues, including the effects of possible workplace changes. Can involve people with low English literacy. Helps people feel included and listened to. A percentage of responses will be incomplete and/or inaccurate. Might not identify causes for gaps or issues. Needs time to develop and test a survey. Getting people to participate may be difficult, resulting in a low response rate. Those with low English literacy are much less likely to respond. Responses might be skewed toward people who want to express their views or complain. May be difficult to convince people to attend. Powerful individuals might dominate the discussion. There is no anonymity, so people may not feel they can be totally honest, or can feel pressured to fit in with the crowd. TAEDES505 Evaluate a training program 1 st edition version: 2 Innovation and Business Industry Skills Council Ltd Page 13 of 78