Workplace Literacy and Essential Skills Research

Transcription

1 Canadian Manufacturing Network Workplace Literacy and Essential Skills Research MW Canada Ltd. Technical Skills Certificate Program Impact Analysis Prepared by: Canadian Manufacturing Network, Div. of Excellence in Manufacturing Consortium Research Authors: L. Gillis, Ph.D., and A. Bailey, January, 2014

2

3 Executive Summary MW Canada: Technical Skills Certificate Program MW Canada was selected as one of eight case studies examining the bottom line returns from investing in Literacy and Essential skills (LES) in manufacturing settings. MW Canada is an innovative textile manufacturing firm in Cambridge, Ontario. MW Canada's 65 employees manufacture window covering, filtration composite and energy-related fabrics. In an effort to adapt to increasingly competitive markets, MW Canada initiated a new learning initiative, the Technical Skills Certificate Program, designed to improve technical communications and problem solving of its production staff. Researchers implemented an evaluation strategy using the Gillis and Bailey High Impact Evaluation methodology to examine the key impact questions: How successfully did the training program develop new capabilities (learning and intentionality) in participants? How effectively were the new capabilities transferred to the job? To what extent (if any) did the new skills impact business outcomes and return on investment? Although the six-months training program covered a variety of technical areas, this evaluation focused on two segments Mechanical Power Transmission and Introduction to Pneumatics. The findings suggest that, at the Transfer stage, approximately three-quarters of those who were trained applied their learning, improved their performance on the job, and perceived the training as a worthwhile investment for MW Canada. Slightly more than half the participants reported that the training improved organizational outcomes such as quality and productivity. Although the Certificate Program may well have impacted a variety of different business metrics, key stakeholders chose to limit attribution of improvement to two business measures: reduction of downtime and improvement in product quality. The total annual savings from improvement in these measures was $20,400. The return on investment is calculated to be 117%. (i.e., every dollar spent on training was returned by improved productivity, plus an additional $0.17.) In reality, the returns to the organization can in fact be expected to be much higher since productivity benefits will continue to accrue to MW Canada in subsequent years. (In keeping with the conservative nature of the High Impact Evaluation methodology, however, only the benefits annualized for the first year after training are included in the return on investment calculations.) Page 1

4

5 Case Study MW Canada: Technical Skills Certificate Program Canadian Manufacturing Network with research & report contributed by Lynette Gillis Ph.D. & Allan Bailey, Centre for Learning Impact Study Background Overview of MW Canada Training Rationale MW Canada is an innovative textile manufacturing firm located in Cambridge, Ontario. MW Canada's 65 employees manufacture window covering, filtration composite, and energy- related fabrics that are sold to manufacturers and wholesalers. MW describes itself as a vertically integrated textile mill and attributes part of its success to its ability to quickly respond to shifting market changes by the design, testing and production of new textiles and products. This kind of flexibility is an essential survival prerequisite for the textiles industry which has for many years struggled against wind gusts of adversity technological change, highly competitive global market conditions, and unfavorable currency rates. For a number of years, MW has recognized that maintaining a high level of workforce skills and knowledge is critical to survival in current markets and will strengthen the company's position and its ability to adapt to future market dynamics. A skilled workforce is a key requirement for achieving maximum productivity and maintaining an innovative, high quality mix of products. To achieve this, the firm has adopted a series of work- related learning initiatives designed to establish a culture of continuous learning. Among other things, these efforts have resulted in several workplace learning awards including one from The Conference Board of Canada. As part of this strategy to adapt to rapidly changing technologies, MW initiated a new learning initiative in January 2013 called the Technical Skills Certificate Program. The 90 hours of training, which was delivered in two hour sessions twice a week, targeted basic technical manufacturing skills. The Training The overall objective of the Technical Skills Certificate Program (TSCP) training was to impact job performance at two levels. First, it was anticipated that, as a result of the training, employees would acquire more effective communication skills to better discuss technical issues/faults/problems with co- workers and managers. Secondly, it was expected that production staff would develop problem- solving skills to tackle technical issues, independently or in teams. Such skills are important to ensuring or maintaining high manufacturing productivity and quality and, importantly, high levels of customer satisfaction. The Technical Skills Certificate Program covered a variety of essential manufacturing/technical skills categories: Mechanical Power Transmission Approximately six weeks were devoted to a variety of topics related to power transmission including: blueprint reading, materials properties, lubrication, threads and fasteners, mechanical drives (belts, chains, gears), bearings, couplings and seals. Electrical Fundamentals The program spent two weeks covering electrical basics such as units of electrical measurement, electrical code, alternating currents, and an introduction to electrical transformers. Introduction to Pneumatics Participants studied the pneumatics basics for one month including basic theory of pneumatics, common uses of pneumatics in industry, pneumatics systems and components (cylinders, air preparation units, and control devices). Page 1

6 Hydraulics One month of training covered a range of hydraulics concepts and technologies such as pressure and flow, hydraulics fluids, piping and reservoirs, hydraulic pumps, cylinders, motors, and control valves. Introduction to Electronic/Computer Controls This four week session explored the control theory domain including electronic signals, solid state electronic control devices, computers and networks. Programmable Logic Controllers (PLCs) The final session gave participants insight into the development of PLCs and their advantages to manufacturing. These sessions included overview of control systems, installation and setup practices, and advanced topics such as ladder logic programming practices. The training was delivered to eight production employees from different parts of the MW Canada plant. The training format included a blended delivery utilizing on- line modules from the CMN Skills and Learning Portal; in- plant, instructor- led method for hands- on training, and on- the- job coaching activities with MW Canada production managers. A Conestoga College instructor and a MW Canada production manager provided classroom instruction supported by on- line learning modules. This case study focuses on evaluating the impact of the Mechanical Power Transmission and the Introduction to Pneumatics sections of the Technical Skills Certificate Program training. Prior to the training, a TOWES (Test of Workplace Essential Skills) assessment was conducted with the eight participants to measure numeracy, document uses and reading text. TOWES results were used as a guide to determine TOWES levels for each participant and the skills training required. After training a post- TOWES was implemented. In addition, pre and post Technical Skills assessments were developed to assess technical skills levels of the participants and to help create customized learning plans. Methodology: High Impact Evaluation The Case Study Methodology The Learning Value Chain The MW Canada case study adopts the Learning Value Chain evaluation approach the core evaluation framework and instrument- set develop for the Gillis and Bailey High Impact Evaluation model. The Learning Value Chain framework offers an uncomplicated, field- tested framework for evaluating the learning effectiveness and the business value of training and human resource investments (Figure 1). In the Learning Value Chain, the training program triggers a chain of critical outcomes. Monitoring this chain of outcomes adds an innovative and critically important predictive value to the evaluation process. As desired outcomes are achieved at each link along the chain, greater value is added and the likelihood increases that training will result in positive business outcomes and return on investment. Conversely, if training fails to meet outcomes at any link, value is diminished and the prospect of positive business results and return on investment is at risk. Using the Learning Value Chain framework, the training program is evaluated at each of four links (Capability, Transfer, Business Results and ROI). At each link, data is gathered to assess the extent to which the training has achieved key outcomes, added value and enabled the next critical event in the chain to occur. The Learning Value Chain model also incorporates a diagnostic strategy to investigate training practices and strategies that may strengthen or weaken outcomes at each link and subsequently influence business impact and return on investment. Page 2

7 Figure 1 The Learning Value Chain Training Activates a Chain of Critical Outcomes CAPABILITY Training Enhances Capability and Intentionality: As a result of training, participants acquire new knowledge, skills or attitudes enhancing their capability to take on new work roles and tasks or improving the way they currently do their work. They also develop the intention to apply their learning and begin to formulate plans or ideas for doing so. TRANSFER Enhanced Capability Improves Job Performance: The work context and learning strategy supports the transfer of learning to the workplace. As a result, participants successfully demonstrate new behaviours, perform new roles/tasks, or improve their job performance. BUSINESS ROI Enhanced Job Performance Improves Business Results: Changes or improvements in the way in which participants perform their work contribute to positive improvements in closely linked business or organizational outcomes. Positive ROI: Net benefits from business improvements exceed training costs, resulting in a positive return- on- investment (ROI). Page 3

8 Capability Results Evaluation Questions: Did training participants develop new capability: acquire new knowledge and skills, adopt new attitudes, improve existing skills, or discover new and more productive ways of doing work? Did participants also develop the intention to apply their learning or improve their job performance? Training enhances learning and has moderate impact on other Capability indicators To assess Capability the first link in the Learning Value ChainTM course participants were asked to complete the Capability Questionnaire at the conclusion of the training program. The findings suggest that the Technical Skills Certificate Program training enhanced participants' capability in the two sections evaluated, Mechanical Power Transmission and Introduction to Pneumatics. Participants' self assessments suggest overall learning gains are appreciable. The majority of learners also perceive the training as "moderately" valuable (68%) and express a similar level of confidence in their ability to apply their learning. Learners are "moderately" motivated (78%) to apply their learning and, to a similar degree, perceive their workplace as supportive. In the course of training, about one- third of the learners formulated plans for applying their learning; an additional 50% report that they had done a moderate level of planning. Key questionnaire results are described below and summarized for all program participants in the Capability Index (Figure 2). Skills & Knowledge: Before training, 49% of training participants rate their knowledge and skill level as high. After training, 69% rate their knowledge and skill level as high a gain of 20%. Confidence in Applying Learning: The largest proportion of participants (65%) report a "moderate" level of confidence in their ability to effectively apply their problem solving knowledge and skills to their jobs. Of those remaining, 21% report a "low" level of confidence and 14% a "high" level of confidence in their ability to Perceived Value: Most participants (68%) perceive their as "moderately" valuable (i.e., credible, practical, relevant, and essential). Of those remaining, 26% perceive the training as "highly" valuable and only 6% perceive it as having little or no value. Motivation to Apply Learning: Most (77%) report that they are "moderately" motivated to apply their learning; 23% are "highly" motivated. Workplace Readiness: The largest proportion (77%) report that they will be able to apply the material addressed in the session to a "moderate" extent in their specific situation. The remaining 16% of respondents report that they will be able to apply their learning from the Technical Skills Certificate Program to a "high" degree. Plans for Action: Half the group report having developed at least some plans or ideas for using their problem solving skills in their work. 34% report having a lot of plans or ideas for learning application. Risk Alert for Transfer As indicated by the yellow flag in the Risk Alert box of the Capability Index, more than two- thirds of participants assign "low" or "moderate" ratings on 5 key indicators of program impact: confidence to apply the learning (87%), perceived value of the training (75%), motivation to apply learning (78%), plans for action or application (67%), and the readiness of the workplace for learning transfer (84%). Taken together, the findings suggest a substantive risk for the transfer of learning to the job and performance enhancement. Page 4

9 Effective Practices The following strategies and practices were identified by participants as having contributed positively to the learning: Support from the top Support for the program and communication came from the top. MW Canada's President was a strong supporter of the training. All communication about the training to employees describing its purpose and importance came directly from the President. Production oriented The involvement of a production manager in classroom activities provided learners with a coach who is able to continuously relate the training to the participants' work situations. Line manager buy- in Additionally, involvement of line manager encouraged the buy- in of other managers resulting in an overall supportive environment for learning. Hands on practice In- class demonstrations and hands- on practice were seen as extremely engaging and generated substantial discussion. In house training room The availability of an in- house training room provided a quiet setting for learning. CFLI Capability Index CAPABILITY RISK ALERTS The majority of participants assign moderate ratings on 5 key indicators of program impact: confidence to apply the learning, perceived value of the training, motivation to apply learning, plans for action or application, and the readiness of the workplace for learning transfer. The yellow flag alert suggests that moderate ratings may hinder learning transfer. Employing strategies to strengthen these results will enhance program impact on job performance, organizational results and ROI. Page 5

10 Transfer Results Evaluation Questions: Did the work environment and learning strategy support the transfer of learning to the job? Did training participants apply their learning to their jobs, and did the application of learning impact their work or job performance? Training improves job performance and organizational outcomes including product quality and productivity Effective Practices Enabling Transfer To assess Transfer the second link in the Learning Value Chain TM participants in the Technical Skills Certificate Program were asked to complete the Capability Questionnaire at the completion of training. Further, all training participants completed a subset of questions on the Transfer Questionnaire (TQ) approximately two months following their TSCP training. The findings suggest that approximately three- quarters of those who were trained applied their learning, improved their performance on the job, and perceived the training as a worthwhile investment for MW Canada. Slightly more than half the participants reported that the training improved organizational outcomes such as quality and productivity. The Transfer Index that follows summarizes these key results: Learning Application: 71% of participants report a "high" degree of learning application to their jobs. Performance Improvement: 70% of respondents report a "high" level of performance improvement as a result of the training. Of those remaining, 12% report moderate improvement and 18% little or no improvement. Organizational Results: 53% of respondents report a "high" level of improvement in organizational results (product quality, productivity, teamwork, communications, and customer satisfaction) as a result of participating in the Technical Skills Certificate Program training. 26% report a "moderate" level of improvement in organizational results while 21% reported a "low" level of improvement. Perceived Value: 77% of respondents report that the Technical Skills Certificate Program was a 'worthwhile investment' for MW Canada. The following strategies and practices were identified by participants as having contributed to the training event s success: Daily problem- solving team meetings Manager/facilitator used daily team meetings to discuss specific problems or issues related to the training. This activity helped reinforce the training and provided opportunities for participants to use the new knowledge to problem solve as a group (half of trainees were on the manager's production team). Manager Support Management support is widely recognized as a critically important effective practice to ensure that training is effectively transferred from the classroom to the job. In addition to the reinforcement benefits derived from the daily problem- solving meetings, these sessions also helped drive home to participants the high level of importance management places on the technical skills training and competencies Page 6

11 Other Findings, Observations, and Insights Several participant responses and stakeholders interviews highlighted a problematic aspect of the training sheer size. It was proposed by some that the original curriculum 80 hours of training over six months might be somewhat overambitious. ("We wanted to eat the elephant whole not just begin with the toe.") It was recommended by some that a more practical and manageable training format might be considerably shorter three to five weeks rather than 80 hours over six months. Additionally, based on the observed effectiveness of hands- on segments and problem solving activities, stakeholders came to recognize the potential value of incorporating a workplace problem- solving project with the training. It was recommended by some that the training design should evolve to become a more problem- based learning experience. Participants might be offered the opportunity to strengthen newly acquired knowledge by applying it to real world situations. In particular, training could focus on such issues as preventative maintenance, breakdown of power transmission on a loom, or how to conduct systematic investigation and problem solving. It was also proposed that participants might use the MW Canada training room and resources to collaboratively research problems and engage in problem- solving activities. As a corollary to these considerations, it was suggested that future training programs might be rendered more effective by developing a problem- based design structure. Such a program structure would offer participants greater opportunity to apply theory and knowledge to real world, job- related scenarios and tasks. MW Canada Transfer Index TRANSFER RISK ALERTS No Risk Alerts. Favourable ratings on all key indicators suggest learning has transferred for most participants and has the potential to impact business results and ROI. Page 7

12 Organizational Results Evaluation Questions: Did the work environment and learning strategy support the transfer of learning to the job? Did training participants apply their learning to their jobs, and did the application of learning impact their work or job performance? Better preventative maintenance resulting from enhanced technical skills, reduced equipment downtime and lowered costs of parts replacement Organizational Outcomes: An initial rationale for implementing the Technical Skills Certificate Program was to help MW Canada improve or maintain high levels of manufacturing productivity and product quality. The feedback from training participants suggests that approximately three- quarters of those who were trained applied their learning, improved their performance on the job, and perceived the training as a worthwhile investment for MW Canada. Slightly more than half the participants reported that the training improved organizational outcomes such as quality and productivity. This evaluation determines the financial benefits derived by MW Canada as a result of improved business outcomes directly attributable to the technical skills training. Although the Certificate Program may well have impacted a variety of other business metrics, key stakeholders chose to limit quantifiable attribution of improvement to one business measure: the reduction of downtime and equipment replacement costs as a result of more effective preventative maintenance practices. More effective preventative maintenance was directly linked to the Mechanical Power Transmission and Pneumatics Certificate Program training modules. In the months following training, the senior production manager observed substantial improvement in several distinct areas of operations including more effective preventative maintenance, improved product quality and improved customer satisfaction. The most readily measureable (tangible) improvement, however, was reduced equipment downtime as a result of the production staff's abilities to provide more effective preventative maintenance on the textile looms. The production manager attributed this improvement to the increased levels of technical knowledge and skills in the production staff a direct result of the training. Business Benefit from Training Loom Arm: Typically, the textile loom arm equipment experiences five mechanical breakdowns annually as a normal consequence of the stress of continuous operations. Each loom breakdown requires a two- day shutdown to identify problems and to repair and replace defective components. Importantly, each day of shutdown represents one lost day of production output to MW Canada. The combined tally of lost production and parts replacement from a single breakdown has been calculated to be $6,800. In addition, each unplanned shutdown triggers an additional lost opportunity cost as loom technicians are diverted from their normal productive activities. As a consequence of more effective preventative maintenance, loom breakdowns were reduced from five incidents to two per year. Total annual business benefit from the reduced downtime, therefore, is the cost saved by having 3 fewer loom shutdowns per year. Total Annual Benefit: 3 shutdowns/year X $6,800/Shutdown = $20,400 Note: In keeping with the conservative nature of the Centre for Learning Impacts' ROI methodology, only the benefits annualized for the first year after the training are included in return on investment calculations. There is every reason to suspect, of course, that the MW Canada will continue to reap the benefits from this training for years to come. Page 8

13 ROI Evaluation Questions: Did the net business benefits of training exceed its total costs (Benefit/ Cost Ratio)? What was the return on investment (ROI)? Return on Investment (ROI) of Training A meaningful way to judge the success of the Technical Skills Certificate Program is to evaluate its impact on the organization s bottom line. To do this we can compare the quantifiable business benefits directly attributable to the program with the total cost of the delivering the training the return on investment (ROI). Cost of Training MW Canada management calculated the following costs related to implementation and delivery of the technical skills training. External course acquisition and facilitation costs. Cost of Instructional materials and facilitation from Conestoga College: $15,000 Internal costs MW Canada Facilitator salaries: (class time, prep time, meetings) Participants: (salaries, overtime wages to cover production demands) Pilot Supplies: (laptops, binders, copying) Administration: ( s, letters, meetings, other from Feb 2012 until June months) Total internal costs: $20,000 Total cost of all training: $35,000 Pro- rating training costs: This case study evaluates the organizational impact of only two components of the training program the Mechanical Power Transmission and the Introduction to Pneumatics components. In order to obtain a meaningful ROI value, therefore, only the training costs associated with those two course components are include in benefit/costs ratio (BCR) and return on investment (ROI) calculations. Total training: 90 hours Portion of training for Mechanical Power Transmission and the Introduction to Pneumatics: 45 hours Percentage of training of the two components: 50% Relevant TSCP cost = 50% X $35,000 = $17,500 Benefit Cost Ratio BCR = Total Business Benefits = $20,400 = 1:1.17 Total Costs $17,500 Return on Investment (ROI) ROI = Total Benefits Total Cost X 100% Total Cost = $20,400 - $17,500 X 100% = $2,900 X 100% = 17% $17,500 $17,500 As a result of MW Canada investing resources and time (course acquisition, facilitators and participants salaries, etc.) in Technical Skills Certificate, the organization received substantial, quantifiable business benefits. For every dollar invested, MW Canada received that dollar back and an additional $0.17 in profit as a result of reducing equipment downtime. Note: It is worth noting that for most private organizations, the typical hurdle rate (the minimum acceptable rate of return for giving the go- ahead to a planned project) is between 10% and 20%. Also, very few organizations hold their Training and Human Resource functions to the rigorous financial scrutiny reserved for the operational functions production, information technology, marketing, etc. Page 9

14 Intangible Benefits In addition to the quantifiable benefits addressed in this ROI calculation, there are often non- quantifiable, intangible outcomes. Intangible outcomes from training are those benefits that are not easily convertible to monetary metrics. (Typical training intangibles might include metrics such as enhanced employee engagement, reduced workplace stress, reduced staff turnover, etc.) The intangible benefits resulting from the MW Canada Technical Skills Certificate Program include improved product quality, problem solving capabilities, on- the- job confidence, and workforce collaboration. Improved Product Quality As a result of the improved preventative maintenance, production managers have noticed a sharp reduction in calls from the Mending Department. This suggests a savings in time required to repair defective product as well as cost savings from reducing amount of product that ends up as scrap. There was almost zero scrap from the production lines in the six months following the training. However, management was unable to specifically attribute these efficiency improvements to the training due to the diverse variety of variables influencing the quality and the production of the fabric. To remain conservative, therefore, this ROI calculation does not factor in the business benefits from reduced fabric damage. Improved Problem Solving Capabilities in Production Workforce The design of the Technical Skills Certificate Program training included considerable hands- on instruction which provided participants with the opportunity to problem- solve independently and as a group. The pneumatics exercise, for example, included display boards that the participants had to use to design a process. In one situation, the problem- solving skills and routines acquired using the display board helped production staff resolve a complex mechanical loom problem that would normally have required outside expertise. According to a MW Canada production manager, two loom mechanics, using their new insights into pneumatics along with new problem- solving skills, were able to devise a solution. In the past, explained the manager, "We would have brought in outside assistance much earlier in the process to resolve the problem. This kind of problem solving speaks well to our ability to effectively identify and resolve future issues." Confidence On- the- Job and Independence Production managers have observed an increased level of independent thinking in the workplace as a result of the technical skills training. For example, according to one manager, "Loom mechanics now start jobs without requiring supervisors to coax or push them. In the morning the tasks of the day are reviewed employees pick their jobs and independently start them." Previously, it would be necessary for supervisors to assign tasks and to be more directive. Now it is a great deal easier to "attack more complex problems" since staff feel they have more knowledge and more confidence. Enhanced Workforce Collaboration Following training, MW Canada managers observed a marked improvement in the willingness of production staff to share information and generally collaborate with their co- workers. Supervisors reported that, following training, production staff were more confident in their own capabilities: they found it easier to work with others and were more comfortable sharing ideas. Importantly, this enhanced workplace collaboration /collegiality on the weave room floor resulted in additional, unquantifiable, gains. Improved teamwork reduced the supervisory time required on the production line. As a result, supervisors had more available time to contribute to other tasks and projects. Page 10

15 Table of Contents Capability at a Glance Capability summary Transfer at a Glance Transfer summary Page 1

16 MW Canada Technical Skills Certification Program Detailed Results: CFLI Capability Index RISK ALERTS: The majority of participants assign moderate ratings on 5 key indicators of program impact: confidence to apply the learning, perceived value of the training, motivation to apply learning, plans for action or application, and the readiness of the workplace for learning transfer. The yellow flag alert suggests that moderate ratings may hinder learning transfer. Employing strategies to strengthen these results will enhance program impact on job performance, organizational results and ROI. Page 2

17 MW Canada Technical Skills Certification Program Transfer Results at a Glance: CFLI Transfer Index Transfer Alerts: No Risk Alerts. Favourable ratings on all key indicators suggest learning has transferred for most participants and has the potential to impact business results and ROI. Most Frequently Mentioned Barriers to Transfer: Require time to practice4 new skill or adopt new ways of doing things (33%). Forgotten much of what I learned (33%). Content was not applicable in my work situation. Page 3

18 Capability Results: Technical Skills Certificate Program Response Rate: Eight participants responded to the Capability Questionnaires for Module 1 (Mechanical Power Transmission) a 100% response. This data was collected January Seven of 8 training participants responded to the Capability Questionnaires for Module 2 (Introduction to Pneumatics) resulting in a response rate of 88%. This data was collected March Learning: Table 1: Please rate your knowledge and skill level before training in each of the following areas. The table below reports the percent of respondents rating their knowledge and skill on a 5- point scale as 'low' (a rating of 1 or 2), 'average' (a rating of 3), or 'high' (a rating of 4 or 5). Mechanical Power Transmission LOW % AVERAGE % HIGH % LEARNING OBJECTIVE Explain the importance of the three standard views of a blueprint Identify and understand the different compositions of material used in mechanical power transmission Demonstrate a basic understanding of shop measuring and hand tools Recognize appropriate uses of threads and fasteners Demonstrate basic understanding of lubricants and the method of application means 32% 20% 49% Total Number of Respondents = 8 Introduction to Pneumatics LOW % AVERAGE % HIGH % LEARNING OBJECTIVE Identify common pneumatic symbols Identify and understand different schematics circuits Demonstrate basic understanding of air treatment for shops Identify proper formulas for the working cylinders Relate to practical applications of pneumatic circuits in the plant means 32% 20% 49% Total Number of Respondents = 7 overall mean rating before course 32% 20% 49% Page 4

19 Table 2: Please rate your knowledge and skill level after training in each of the following areas. Mechanical Power transmission LOW % AVERAGE % HIGH % LEARNING OBJECTIVE Explain the importance of the three standard views of a blueprint Identify and understand the different compositions of material used in mechanical power transmission Demonstrate a basic understanding of shop measuring and hand tools Recognize appropriate uses of threads and fasteners Demonstrate basic understanding of lubricants and the method of application means 11% 19% 70% Total Number of Respondents = 8 Introduction to Pneumatics LOW % AVERAGE % HIGH % LEARNING OBJECTIVE Identify common pneumatic symbols Identify and understand different schematics circuits Demonstrate basic understanding of air treatment for shops Identify proper formulas for the working cylinders Relate to practical applications of pneumatic circuits in the plant means 11% 20% 68% Total Number of Respondents = 7 overall means rating after course 11% 20% 69% Table 3: Comparison of mean ratings on each objective before and after training. Mechanical Power Transmission Before Training After Training Change Score LEARNING OBJECTIVE Explain the importance of the three standard views of a blueprint Identify and understand the different compositions of material used in mechanical power transmission Demonstrate a basic understanding of shop measuring and hand tools Recognize appropriate uses of threads and fasteners Demonstrate basic understanding of lubricants and the method of application means Page 5

20 Introduction to Pneumatics Before Training After Training Change Score LEARNING OBJECTIVE Identify common pneumatic symbols Identify and understand different schematics circuits Demonstrate basic understanding of air treatment for shops Identify proper formulas for the working cylinders Relate to practical applications of pneumatic circuits in the plant means overall change score (5- point scale).6 Table 4: To what extent did the program meet your personal skills gap (in the topic area)? Module LOW % MODERATE % HIGH % Mechanical Power Transmission Introduction to Pneumatics Total Number of Respondents for each module = 6 Table 5: Mean pre and post test scores on the Technical Assessment. Module Before Training % After Training % Change Score Mechanical Power Transmission % Introduction to Pneumatics % Overall mean change score for course 34% 36% +2% Table 6: Participants' TOWES levels pre and post training (October 2012 to June 2013). Table describes number of participants who demonstrated a decrease in level, increase in level or no change. Decrease Increase No Change Reading 1 7 Document Use 2 6 Numeracy 3 5 Page 6

21 Confidence in Applying Learning Table 7: How confident are you in your capability to effectively apply knowledge and skills on the job? The table reports the percent of respondents rating their confidence level as low, moderate, or high following training. Mechanical Power Transmission LOW % AVERAGE % HIGH % LEARNING OBJECTIVE Explain the importance of the three standard views of a blueprint Identify and understand the different compositions of material used in mechanical power transmission Demonstrate a basic understanding of shop measuring and hand tools Recognize appropriate uses of threads and fasteners Demonstrate basic understanding of lubricants and the method of application Means 21% 58% 22% Total Number of Respondents = 6-8 Introduction to Pneumatics LOW % AVERAGE % HIGH % LEARNING OBJECTIVE Identify common pneumatic symbols Identify and understand different schematics circuits Demonstrate basic understanding of air treatment for shops Identify proper formulas for the working cylinders Relate to practical applications of pneumatic circuits in the plant means 22% 72% 6% Total Number of Respondents = 6-7 overall means for course 22% 65% 14% Page 7

22 Workplace Readiness Table 8: How confident are you that it will be possible to effectively apply this learning given your specific workplace situation, conditions, or setting (e.g., have the time, manager's support, resources, or appropriate opportunities)? The table reports the percent of respondents rating their confidence level as low, moderate, or high following training. Module LOW % MODERATE % HIGH % Mechanical Power Transmission Introduction to Pneumatics means 7% 77% 16% Total Number of Respondents =); 6 (Mechanical Power Transmission); 7 (Pneumatics Motivation to Apply Learning Table 9: How personally motivated are your to apply this learning to your job? The table reports the percent of respondents rating their motivation level as low, moderate, or high following training. Module LOW % MODERATE % HIGH % Mechanical Power Transmission Introduction to Pneumatics means 0% 78% 23% Total Number of Respondents = 8 (Mechanical Power Transmission);7 (Pneumatics) Plans for Action Table 10: To what extent have you developed plans and ideas for applying your learning to your job? Module To a small extent or not at all To a moderate extent To a great or very great extent Mechanical Power Transmission Introduction to Pneumatics means 17% 50% 34% Total Number of Respondents = 8 (Mechanical Power Transmission); 7 (Pneumatics) Page 8

23 Perceived Value Table 11: How would you describe this program? Participants used a 5- point scale to rate how credible, practical, relevant or essential they perceived the program. For each descriptor, the table reports the percent of respondents assigning low, moderate, and high ratings. Tables report response to each module. Mechanical Power Transmission LOW % MODERATE % HIGH % Credible Practical Relevant Essential Means 0 75% 25% Total Number of Respondents = 4-8 Introduction to Pneumatics LOW % MODERATE % HIGH % Credible Practical Relevant Essential Means 13% 60% 27% Total Number of Respondents = 3-6 overall Means for course 7% 68% 26% Other Findings Contents: 88% of participants reported that they liked the contents of the "Mechanical Power Transmission" module to a 'moderate degree.' The reaction to "Introduction to Pneumatics" was more favourable: 73% reported that they liked the module contents. Delivery Mode (face- to- face, webinar, e- learning, etc) 88% of participants rated the delivery mode of the "Mechanical Power Transmission" module as 'moderately' effective and conducive to learning. In contrast, 71% of participants rated the delivery mode of "Introduction to Pneumatics" as 'highly' effective and conducive to learning. Page 9

24 Transfer Results: Technical Skills Certificate Program Response Rate: Nine participants responded to the Transfer questionnaires on "Mechanical Power Transmission" and "Introduction to Pneumatics" a response rate of 100%. Participants completed the first questionnaire in May 2013 and the second in July Learning Application Table 12: Select the rating that best reflects the extent to which you applied the knowledge and skills acquired in each module. Participants rated their degree of learning application on a 5- point scale where a rating of 1 or 2 was considered 'low' impact, a rating of 3 'moderate' impact, and 4 or 5 'high.' Mechanical Power Transmission LOW % MODERATE % HIGH % Understanding of the three standard views of a blueprint Understanding the different compositions of material used in mechanical power transmission Understanding of shop measuring and hand tools Using threads and fasteners appropriately Understanding lubricants and the method of application means 9% 20% 71 % Total Number of Respondents = 9 Introduction to Pneumatics LOW % MODERATE % HIGH % Identifying common pneumatic symbols Identifying and understanding different schematics circuits Understanding of air treatment for shops Identifying proper formulas for the working cylinders Practical applications of pneumatic circuits in the plant means 9% 20% 71% Total Number of Respondents = 9 overall course means 9% 20% 71% Page 10

25 Job Performance Table 13: To what extent have each of the modules improved your performance in the following areas. If you have not had opportunity to apply your learning, indicate not applicable. Course participants rated the degree of improved job performance using a five- point scale ranging from Not at All" to "To a Very Great Extent." The table reports the percent of respondents rating their performance as 'low,' (rating of 1 or 2), 'moderate' (rating of 3), or 'high' (rating of 4 or 5). Introduction to pneumatics LOW % MODERATE % HIGH % Identifying opportunities to improve quality Identifying opportunities to improve productivity Leadership engagement of a technical team Effective communication of mechanical transmission challenge/opportunities and management means 19% 12% 70% Mechanical Power transmission LOW % MODERATE % HIGH % Demonstrating effective technical skills and teamwork Identifying opportunities to improve quality and productivity through the use of mechanical transmission skills Leadership/engagement for mechanical power transmission issues Effective communication of mechanical power transmission challenges/opportunities and potential solutions to management means 19% 12% 70% overall course means 19% 12% 70% Page 11

26 Manager's Perceptions of Learning Application and Performance Improvement Mechanical Power Transmission Three managers assessed the extent to which their employees had applied their learning and enhanced their performance. All indicated that application had occurred and for the most part, participants' performance in most targeted areas had improved. Managers varied in their assessment, however, how much application and improvement had occurred as a result of training. Given the small sample of managers, the results are inconclusive. Introduction to Pneumatics Managers varied considerably in the extent to which they perceived their employees as applying their learning and improving and their performance. For the most part, learning application and performance improvement was low to moderate. Table 14: Manager's Comments Mechanical Power Transmission Most of the work that would be performed in these areas (learning areas) is performed by maintenance. There may be some possibility of a lead hand/operator performing some preventative maintenance. It is too soon to say if my employee will use the information from this program. The mechanical knowledge certainly is useful but most mechanical work is performed by maintenance. Blue prints power transmissions are not frequently looked by my employee. My employee has taken on more of the mechanical work at his machine and is doing so frequently and with confidence. Introduction to Pneumatics Very few pneumatics are used in my departments. However, I very much like the fact that I have extended the knowledge base for my employees which in the future could be greatly needed. So while training has little immediate impact, it is excellent for the future I'm glad they have it. [Suggested improvements] Slower pace. Too much information in a short period of time. [Suggested improvements] Need a computer layout that is operable. Page 12

27 Enablers and Barriers to Learning Application Enablers Table 14: Which factors, if any, helped you to apply your learning from this training to your job? Check all that apply. Factors Enabling Application: Mechanical power transmission % Selecting the Factor Clear performance expectations 33 Adequate time and resources 44 Adequate motivation and incentive 44 Feedback on my performance 22 Support from my manager or supervisor 33 Follow- up discussions or coaching 33 Support from my colleagues or peers 33 Sufficient level of knowledge and skill 55 Information, reference materials, tools, or job aids 55 Other; please describe below 0 Total Number of Respondents = 9 Factors Enabling Application: introduction to Pneumatics % Selecting the Factor Clear performance expectations 33 Adequate time and resources 44 Adequate motivation and incentive 66 Feedback on my performance 11 Support from my manager or supervisor 33 Follow- up discussions or coaching 33 Support from my colleagues or peers 44 Sufficient level of knowledge and skill 44 Information, reference materials, tools, or job aids 44 Other; please describe below 0 Total Number of Respondents = 9 Page 13

28 Barriers Table 15: Which of the following barriers, if any, hindered or prevented you from applying what you learned in training? Check all that apply. Barriers to Application: Mechanical Power Transmission % Selecting the Factor Too difficult to break old habits 11 Lack support from colleagues or peers 0 Unable to dedicate the time required to practice new skills or adopt new ways of doing things 33 Lack feedback on performance 11 Lack relevant situations or opportunities to apply my learning 0 Lack information, reference material, tools, or job aids 0 Forgotten much of what I learned 33 Require more knowledge and skill to be able to apply the learning on my own 22 Performance expectations were unclear 22 Content was not applicable in my work situation 33 Too little motivation or incentive 0 Policies, procedures, rules, work processes 0 Lack the interest and support of management 0 Other; Please describe: 0 Total Number of Respondents = 9 Barriers to Application: Introduction to Pneumatics % Selecting the Factor Too difficult to break old habits 11 Lack support from colleagues or peers 0 Unable to dedicate the time required to practice new skills or adopt new ways of doing things 33 Lack feedback on performance 11 Lack relevant situations or opportunities to apply my learning 0 Lack information, reference material, tools, or job aids 0 Forgotten much of what I learned 33 Require more knowledge and skill to be able to apply the learning on my own 22 Performance expectations were unclear 22 Content was not applicable in my work situation 33 Too little motivation or incentive 0 Policies, procedures, rules, work processes 0 Lack the interest and support of management 0 Other; Please describe 0 Total Number of Respondents = 9 Page 14

29 Perceived Organizational Results (Participants) Table 16: To what extent has this training helped you to achieve the following business outcomes? Participants rated the impact on business outcomes on a 5- point scale: 'low' (rating 1 or 2), 'moderate' (rating of 3), or 'high' (rating of 4 or 5). Business outcomes: Mechanical power transmission LOW % MODERATE % HIGH % Improve product quality Improve productivity Enhance teamwork Enhance communication Improve customer satisfaction (7) means 21 % 26% 53 % Total Number of Respondents = 9 Business outcomes: Introduction to Pneumatics LOW % MODERATE % HIGH % Improve product quality Improve productivity Enhance teamwork Enhance communication Improve customer satisfaction (7) means 21% 26% 53% Total Number of Respondents = 9 overall means for course 21% 26% 53% Page 15

30 Perceived Organizational Results (Managers) Table 16: To what extent has this training helped you to achieve the following business outcomes? The managers of participants rated the impact on business outcomes on a 5- point scale: 'low' (rating 1 or 2), 'moderate' (rating of 3), or 'high' (rating of 4 or 5). Business outcomes: Introduction to Pneumatics LOW NO. MODERATE NO. HIGH NO. NA Improve product quality Improve productivity Enhance teamwork 1 2 Enhance communication Improve customer satisfaction 2 1 Total Number of Respondents = 3 Business outcomes: Mechanical power transmission LOW % MODERATE % HIGH % NA Improve product quality 1 1 Improve productivity 1 1 Enhance teamwork 1 1 Enhance communication 1 1 Total Number of Respondents = 3 Related Comments: Too early to decide (the impact on the business). Page 16