Evaluating Training and Development Evidence from Ireland. Anna Lavelle Alma McCarthy NUI, Ireland

Transcription

1 Evaluating Training and Development Evidence from Ireland Anna Lavelle Alma McCarthy NUI, Ireland Developments at an economic and societal level over the past number of years such as the knowledge society, the growing importance of human capital, and the focus on lifelong learning have heightened the attention given to investment in employee training and development. As a firm s intellectual capital generally resides within individuals, the organisation must create internal practices to leverage their expertise (Lee, 1999; Fritz-enz, 2002) and the process of Human Resource Development (HRD) facilitates the continuous updating of employee competencies within organisations (Garavan et al., 1995a). Indeed, there has been somewhat of a renewed interest of late, in linking human capital investments with business performance (Holton, 1996; Phillips, 1996; Walker, 2001). In order to maximise the benefits of training, researchers and practitioners must understand what makes training work effectively (Mathieu et al., 1992) and training evaluation is central in determining value and effectiveness. In essence, evaluation assesses the total value of a learning event and thereby puts the event in its wider context and provides information which is imperative for future planning and investment (Harrison, 1997). Evaluation of aims to ensure that training delivers optimum value and that training resources are effectively deployed in order to address the strategic needs of the organisation. The dominant perspective on training evaluation derives from the Four Stages Model consisting of reaction, learning, behaviour and results as proposed by Kirkpatrick (1959). Research regarding training evaluation holds that trainee reactions are an inadequate measure of training success (e.g. Alliger and Janak, 1989; Tannenbaum and Yulk, 1992) and further research underlines the importance of transfer of learning (Alliger et al., 1997; Facteau et al., 1995; Herold et al., 2002; Mathieu et al., 1992). Although such research has led to more sophisticated evaluation practices in recent years, evaluation still remains a neglected stage in the training cycle (Garavan et al., 1995a; Holton, 1996; Sadler-Smith et al., 1999). Although training evaluation is perhaps the most important stage in the entire training cycle, some argue that it still remains the most neglected stage and is generally conducted in a piecemeal and half-hearted fashion focusing mainly on surface level reaction criteria (Bassi et al., 1996; Bramley, 1999; Bramley and Kitson, 1994; Garavan et al., 1995a; Mann and Robertson, 1996; Sadler-Smith et al., 1999). However, relatively few studies have investigated the reasons why organisations do or do not engage in training evaluation. This study reviews the literature on training evaluation methods, practice, and philosophy. The study reports on the evaluation practices in five large organisations in Ireland. Thus, the objectives of the study are two-fold. Firstly, the study explores to what extent the case companies are evaluating training and development investment and what methods these organisations use to capture this information.

2 Secondly, the study reports on the perceptions of training and development effectiveness among the five HR managers and Training Specialists interviewed. The paper also sets out recommendations for research and practice going forward. Training and development evaluation The process of training and development evaluation ensures that a level of control is maintained over the various HRD activities (Heraty and Morley, 2000). Great amounts of time, money and energy tend to be invested in the design and delivery of training and less on the latter stage incorporating evaluation. This neglect may largely be associated with the perceived difficulties of evaluation, which render it the subject of much rhetoric but considerable practical neglect (Sadler-Smith et al. 1999). Scarbrough and Elias (2002: 4) highlight the difficulty of evaluation stating on the one hand, employee skills and competencies make a critical contribution to business performance. On the other hand, the features, which make human capital so critical, are the very same features that inhibit evaluation. However, organisations are attempting to engage in more evaluation of training efforts. In Ireland, a recent report commissioned by CIPD, indicates that a significant 84% of large organisations evaluate their training activities in Ireland in comparison to only 38% of small and micro firms. Although this appears to be promising, it is also revealed that some 78% of large organisations evaluate reaction-level criteria, with few organisations comprehensively measuring return on training investment. The training evaluation process has been like a reptile which has evolved over the years, making incremental changes whilst at the same time retaining its essential features (Canning, 1996) and today there are better tools to conduct evaluations and consequently better experimental designs have emerged (Salas and Cannon-Bowers, 2002). Approaches to evaluation may range from those that adopt an objective and scientific approach to those that are subjective and interpretative in orientation (Garavan et al., 1999). The traditional models of evaluation can be placed in one of two schools of thought: the goal based or naturalistic approaches. The goal-based approaches are the best known and consist of Phillips (1997) Return on Investment, Kirkpatrick (1959) Stages of Evaluation, Hamblin (1974) Cycle of Evaluation, and Warr (1970) Needs Assessment. In contrast to such goal/outcome based approaches the naturalistic school focuses on training delivery (Canning, 1996). Canning goes on to further explain that: although both schools may appear worlds apart, once the rhetoric is put to one side, we are left with the same methodological approach, seen through a different set of lenses (1996:5). This would appear to imply that both perspectives should be viewed as mutually inclusive and complementary. Donald Kirkpatrick s (1959) model remains one of the most comprehensive and widely referenced models of training evaluation to date (Alliger and Janak, 1989; Blanchard et al., 2001; Salas and Cannon-Bowers, 2001; Warr et al., 1999). Essentially, the power of Kirkpatrick s typology rests in its simplicity (Alliger and Janak, 1989) and its ability to assist HRD practitioners in comprehending training evaluation criteria (Sadler-Smith et al., 1999). Kirkpatrick s framework consists of

3 four inter-linked levels (reaction, learning, behaviour and results), which will each be explored in turn. Reaction is the term that Kirkpatrick uses to refer to how well the participants like a particular training programme. Clearly reaction is a critical decider in the continuation of any training initiative (Mann and Robertson, 1996), yet reaction-level evaluation alone tends to be a poor determinant of overall training effectiveness. Reaction measures remain one of the most popular methods of evaluation in the field of HRD (Ruona et al., 2002) and many organisations continue to adopt a reactiononly strategy. In fact the inclusion of trainee reactions as a primary outcome is perhaps the greatest failing of Kirkpatrick s model (Holton, 1996). Traditionally reaction-level criteria have been gathered by means of some form of post-course questionnaire or more commonly the happy sheet. Smith warns of the ineffectiveness of the happy sheet arguing that the happy-sheet in its current form does not provide any more that a superficial evaluation of the course which is subjected to a considerable amount of bias and influence (2001:12). Ideally reaction level measures should be used in conjunction with other more objective levels of evaluation. As it stands, whether a participant feels that a programme was interesting, challenging or stimulating, tells little about the impact and effectiveness of the training (Bramley, 1999). Evaluation at the learning level is essentially concerned with the principles, facts and techniques understood and absorbed by the trainees. The nature of learning activity is a complex and multi-faceted phenomenon and oftentimes this fails to be acknowledged by the majority of organisations, thus resulting in a one-dimensional perspective being adopted. Learning may be perceived as an individual process or in contrast as a process embedded within social-cultural practices (Clarke, 2004) and depending on which perspective is adopted different evaluation techniques are appropriate. Gerber and Oaklief (2003) distinguish between three main types of learning processes namely: transmissible, transactive and transformative. Transformative learning is considered to be the most sophisticated form of learning: In addressing dilemmas that they encounter, workers will use previously learned skills (2000:179). According to Kirkpatrick s learning criterion level, measuring the change in knowledge as a method of evaluating the effectiveness of training is of value. It is an important stage as it measures the extent to which training objectives have been achieved and the use of pre and post tests are a useful way to capture trainee learning (Bee and Bee, 1999). Evaluation that occurs at the behavioural level refers to learning application and implies that learning has transferred from the training environment to the job. Training is considered a means of changing behaviour and therefore evaluation of its effectiveness should rest on measuring change (Burden and Proctar, 2000). For training transfer to successfully occur, learned behaviour and principles must be generalised to the job context and maintained over time (Mann and Robertson, 1996). However, due to the inherent subjective nature of training evaluation, behavioural objectives are difficult to define in concrete terms and this requires more initial observation and planning (McClelland, 1994). Surveys, interviews and observations are some of the more common tools used in measuring behavioural change (Endres and Kleiner, 1990).

4 The results level of Kirkpatrick s framework is concerned with the achievement of end-goals and desired performance outcomes. This is the least evaluated level and indeed the most complex level at which to conduct evaluation. Alliger et al., (1997: 348) stress the importance of evaluation at this level stating organisational results criteria are the most distal from training, and often perceived as the most fundamental to judging training success (1997:348). The difficulty in obtaining solid measures of training impact makes systematic level four evaluations extremely complex. In addition, the problem of identifying the actual cause of any changes observed is logically dubious (Warr et al., 1999) and isolation of the effect of training from other factors occurring simultaneously in the organisational environment poses a challenge (Phillips, 1996). It may equally be that non-training factors have also contributed to performance improvements. Measuring the Return On Investment (ROI) has appeal as it assesses all the major components of profitability in one measure (Phillips and Phillips, 2001). The lack of ability to measure ROI, explains the inevitable abandonment of numerous training programmes especially during times of economic recession (Plant and Ryan, 1992). The American Society of Training and Development (ASTD) reports that 78% of organisations assess reaction criteria in comparison with a mere 7% assessing organisational results (Van Buren and Erskine, 2002). Another major reason why companies fail to conduct evaluation at the results level is the high cost associated. Although the actual costs of training are generally known, the benefits are often soft, subjective and difficult to quantify (Parry, 1996). The subjective nature associated with results level evaluation does not fare well with the management of most companies who desire hard facts and concrete evidence. It is increasingly difficult to measure soft data items such as customer satisfaction or employee turnover. Evaluation at the results level may be likened to an advanced evaluation strategy, which Tannenbaum and Woods describe as: Evaluation based on experimental designs.yielding the greatest confidence in interpretation (1992:69). Organisations should strive to reach level four in the Kirkpatrick typology as this indicates a genuine and articulated attempt to measure the true effectiveness of training within an organisation. Nonetheless, in selecting programmes to be evaluated at the various levels, especially level four, several criteria must be considered e.g. life-cycle of programme, linkage of programme to operational goals, strategic objectives, programme cost and many more (Phillips and Phillips, 2001). In essence, the connection between organisational results and training intervention depends upon effective management decision-making and the setting of strategic directions (Plant and Ryan, 1992). A key assumption inherent in the goals-based evaluation models is the hierarchical nature of the taxonomy and the link from one level to the next (Leach and Liu, 2004). Hamblin (1974) further emphasises this hierarchical structure, however he suggests that the link may break down at any stage. Essentially there are five levels of objectives (reaction, learning, job, organisational, behaviour and ultimate level value), which impact on five effects and in turn this impacts on the initial objectives. With the exception of the ultimate value level Hamblin s model bears a striking resemblance to Kirkpatrick s framework. However, Clement (1982), through the use of a pre-test/post-test control group design, tested this assumption and found only partial support for the hierarchy of training outcomes. Reactions were positively

5 related to learning and learning to improvement in behaviour, but none of the other expected relationships were reported (Clement, 1982). Therefore, trainee learning appears to be a necessary but not essential prerequisite for behaviour change (Tannenbaum and Yukl, as cited in Arthur et al., 2003). In a meta-analectic investigation conducted by Alliger et al., (1997) of the relationships between the first three stages of Kirkpatrick s model, they found only modest correlations between the various training criteria. In an earlier study Alliger and Janak (1989) found there to be no relationship between reaction criteria and the other three levels (Bramley and Kitson, 1994). Returning to the age-old concern of cause and effect, Holton (1996) conceives any attempt to draw linkages as being a futile process because by definition taxonomies classify rather than define causal constructs (1996:6). Kirkpatrick s model has weathered well but some argue that it has limited our thinking regarding evaluation (Albernathy, 1999) and this has attributed to a lack of progress in the field of training evaluation (Holton, 1996). Consistent evaluation at all levels of the hierarchy is implicitly difficult and evaluation at the top two levels is complex due to other intervening variables impacting upon performance (Stokking, 1996). In addition, Dionne (1996) identifies the lack of a unifying model as a principal barrier to the overall effectiveness of training procedures. There is a call for greater objectivity to be incorporated into the entire process of evaluation. But how such scientific rigor can be applied is unknown especially when costs and managers reactions to them are considered (Dionne, 1996). It may equally be argued that subjectivity is fundamental to the evaluation process and should be addressed as a positive aspect (Smith, 2001). Although the goal-based evaluation typologies have served as a good foundation for training evaluation over the years, Salas and Cannon-Bowers (2001:487) stress that a richer more sophisticated typology is needed. The problem with such taxonomies is their failure to identify all underlying variables impinging upon the phenomena in question, thus rendering true evaluation impossible (Holton, 1996). Thus, there is a pressing need to account for the various contextual influences within an organisation through adopting a more holistic, relational perspective (Doyle, 1995). The notion of utility analysis is offered by that of Honeycutt et al., (2001) as one means to strengthen the objectivity of training evaluation, as is the concept of formative evaluation offered by Brown and Gerhardt (2002). Method Semi-structured interviews were conducted with individuals responsible for training and development in the five organisations with questions generated to assess the objectives outlined above. The organisations were chosen from the Top 150 companies in Ireland (Irish Times, 2004) across a number of industry sectors. Two of the organisations were medium-sized and three were large sized organisations following the EU definition of organisational size (EU Commission, 1996). Small organisations were not considered for investigation as they typically do not invest as much as their larger counterparts on training and development (Carey, 2000; Hill and Stewart, 1999; Savery and Luks, 2004; Westhead and Storey, 1997) and therefore it is

6 expected that training evaluation would not receive as much attention compared with companies who have a larger spend. The focus of the research is on getting an in-depth understanding of the nature and practice of training and development evaluation and therefore, interviews were deemed the most appropriate research method to facilitate in-depth understanding of the key issues. The first company is a large American-Owned, Multinational, Information Technology (IT) Company, and henceforth will be referred to as InfoTech. The organisation engages in IT infrastructure, personal computing and access devices, global services and imaging and printing. InfoTech first established its business in Ireland over thirty years ago and today it employs over 4,000 employees throughout four separate outlets nationwide. The second company, FinanceCo, is a large publicly owned financial services organisation. The organisation has extensive retail, corporate, and investment banking divisions. There are 5,500 employees employed in branches throughout the country and has a well resourced learning centre in located in Dublin supporting employee training and development needs. The third large company under investigation, Medicom, is a global healthcare company employing approximately 51,000 employees with subsidiaries worldwide. Medicom applies its expertise in medical devices, pharmaceuticals and biotechnology. Medicom employs over 1,100 employees between two western locations in Ireland and produces a range of medical solutions. Two medium-sized companies were also included in the study. The first is a US owned multinational health care company, CardiCare, which primarily focuses on pharmaceuticals, nutritionals and diagnostics. There are 55,000 employees worldwide who devote their expertise to CardiaCare, over 1,700 of these are employed over five manufacturing plants in Ireland. The particular manufacturing plant in question is located in the west of Ireland, is dedicated to the cause of vascular disease treatment and employs around 214 employees. The second, VisionCo, is a medium sized multinational company with joint parent companies in America and France. This company is located in the west of Ireland and employs 230. VisionCo produces photo-chromic lenses, which protect the eye against UVA and UVB radiation. Findings Role and Importance of the Training Function An important determinant of training and development activity and sophistication is the role the nature of the training and development function. Organisational size is a chief determinant of whether an organisation has the resources and hence the expertise to have a separate training and development function. The three large organisations report that there is a dedicated training and development function in the business while the two medium sized organisations indicate that training and development is subsumed into the wider HR function. Many commentators argue that HRD must become more strategically oriented (Boxhall and Purcell, 2003; Becker and Gerhart, 1996; Garavan, 1991; Garavan et al., 1995b; Hallier and Butts, 1999; Storey, 2001). The organisations in this study report conflicting viewpoints in relation to the strategic role importance of training and

7 development in their company. InfoTech purports to have the most strategically focused training function stating that: In this company from a corporate perspective, training and development is viewed as vital a department as any other one. We ve had times in the past four or five years when we ve had demands to cut back from corporate but we would have taken the opposite view that this is probably the best time to invest in training and development (Training Manager, InfoTech). The other two large organisations report that training and development plans are related to the strategic objectives of the business. For example, Medicom report that: Training is an important support function. Its chief aim is to support business goals. The business goals are typically three-year goals and the training objectives flow from this (Training Specialist, Medicom). The medium organisations in this study report that there is less strategic integration of the training and development activity in the organisation and CardiaCare expressly states that: Training is not up there at the strategic level yet (Training Officer, CardiaCare). The problem of training and development being cut back in times of financial difficulty is highlighted by VisionCo when they state that: Training is of strategic importance however, if we had to cut back on training we would look to see which training is the most essential at the time i.e. the business needs (HR Generalist, VisionCo). These findings indicate that the larger organisations report greater levels of strategic integration of training and development with the wider organisational context compared with the medium sized companies. Training Expenditure The investment in training and development is another important factor when assessing evaluation practice. The literature suggests that organisations that spend more on training and development are more obliged to conduct evaluation justifying the return on such expenditure. Table 1 presents the percentage of payroll spend on training and development for each of the five companies studied here. Table 1: Annual Spend on Training and Development as a Percentage of Payroll Company % of Payroll Spend on Training Medicom 2-3% FinanceCo 4% or slightly above CardiaCare 2-3.6% InfoTech 2% VisionCo Refused to provide such information Garavan et al. (2003) report that average expenditure on training and development in Ireland rests at 3.55% of payroll. From the above table it is evident that with the exception of CardiaCare and FinanceCo, all other establishments are lagging behind the national average. Type of Training and Development Activity The companies engaged in a variety of different types of training activities aimed at technical and soft skill development. The training manager of InfoTech indcates an even balance between technical and soft skills stating: The focus is not just on technical skills; it s half-and-half. In this site over half of the employees are software developer engineers, at least 80% have a primary degree. We value all soft-skill and management-type courses.

8 In a similar vein the HR Generalist at VisionCo reveals that: There is generally a mix between technical skills training and soft skills such as time management. The remaining establishments also indicate that depending on the particular needs of the organisation there is equal importance attached to both soft skills and technical skills training. Importance of Training Evaluation The study aims to explore the relative importance that training and development evaluation plays in relation to other dimensions of the training function such as needs analysis, design, and delivery. The five organisations examined reveal that training evaluation needs to be improved and is not awarded the same significance as the other stages of design and delivery. For example, VisionCo states that: Evaluation tends to be forgotten often and it is extremely important to keep evaluating. In reality so much focus goes into setting up courses and the actual logistics side, and the evaluation process is neglected (VisionCo). InfoTech reports a similar trend: The evaluation side of training and development is probably not given as much focus as all other stages. We avail of different courses and the one thing you notice pretty quickly is delivery and the quality of the tutors (InfoTech) And FinanceCo again highlight the attention given to the other parts of the training and development cycle other than evaluation stating: We tend to spend a lot of time focusing on what the business wants and then designing the programme around that and piloting the programme (FinanceCo). Evaluation Techniques and Practices The findings highlight that with the exception of InfoTech (which used the Phillips model) none of the organisations follow any frameworks or models. Although there is a general knowledge of the existence of such frameworks, the majority of establishments envisage them as being too complex and time consuming to conduct in their entirety. The first level at which training can be evaluated at is the reaction level generally assessing participant reactions to the design of the programme, the appropriateness of the content, and satisfaction with the trainers and training facilities. The findings from the interviews indicate that all organisations engage in evaluation at this level. The following statement is indicative of how the companies assess reactions through the use of happy sheets and what this information is used for: Happy sheets measure the relative change between different trainers and reflect the various strengths and weaknesses of trainers. If you throw in the content of training then they are of some use. However, they only measure an employee s opinion on the day (Medicom). The manager in VisionCo points to the reluctance on the part of some training participants to complete satisfaction questionnaires stating that: Some people don t bother filling happy sheets out; they don t see the value in it as the feedback is not going to them (VisionCo). Evaluation at the learning level is useful for assessing what participants actually understood and internalised from the training. FinanceCo considers it too difficult in practice to gather information at the learning level: It s quite difficult to measure how much employees have learned in practice, unless you put it in a quiz or something and we don t necessarily do that.

9 In contrast, Medicom attempt to measure learning some time after the training event: Post course evaluation occurs two to three months after the training with managers of the area in question this is usually by phone-call and discusses how satisfied they are about the skills and abilities acquired by the learner (Training Specialist, Medicom). Evaluation at the behavioural level, where learning transfer to the workplace is explored, in a structured and formal format is minimal in the organisations investigated as displayed by the following statements: The trainer does not conduct performance appraisal on a formal basis to measure changes in behaviour yet, informally, line-managers do appraise their immediate subordinates (Training Specialist, Medicom). The role of the line manager in reviewing change as part of the appraisal process is also used in VisionCo to assess behaviour change: We don t really measure changes in behaviour as a result of training but we would be looking out for changes. The line manager would be able to see if there was an improvement due to training (HR Generalist, VisionCo). InfoTech indicate that the evaluation of behaviour occurs as part of the performance appraisal system in an informal and indirect way stating: As a training and development group, we don t use performance appraisal to measure how employee behaviour has changed as a result of training and development. But we would expect that as a line manager if you are investing in someone or sending them on a training course that you would follow up and try to measure how much learning occurred (Training Manager, InfoTech). The Training Specialist at FinanceCo argues that cause and effect acts as a barrier to effective evaluation of behaviour change stating that: Training is a big part of changes in behaviour at work but it could equally be other elements, so the problem is how do you separate them out? The employees change in behaviour is not directly linked to performance appraisal because it would be too difficult (Training Specialist, FinanceCo). And CaridCare indicate that they want to improve evaluation at this level stating: We don t measure behaviour level yet. We have some ideas on measuring the success of some courses in an easy manner i.e. supervisory development programmes (has there been a reduction in absenteeism? higher productivity or morale?) We plan to trend this to see if we need to make improvements (Training Officer, CardiaCare). The most extensive level of evaluation investigates the impact of training and development on results and determining the return on investment. Not surprisingly, following from the lack of engagement in training at the behavioural level, the majority of organisations studied report that they do no conduct evaluation at the results level. The complexity of evaluation at this level is the reason given by Medicom for not engaging in evaluation at this level stating that: Measuring the results of training on organisational performance is too complex. (Training Specialist, Medicom) The training specialist at FinanceCo goes on to explain the complexities of measuring what are often intangible and non-quantifiable aspects of impact rendering resultsbased evaluation difficult: I think that such models are useful because they give a structure to training evaluation but how useful they are, I would have my reservations You cannot tangibly work out what some of the soft benefits are worth e.g. how much networking or innovation will bring to the company. It s all a little bit wishy-washy. I think companies would get more value if they started evaluating every programme after three months than they would in carrying out level five evaluations.

10 InfoTech forward the costs involved as the reason for not conducting results-based evaluation stating: The main reason we don t measure results is the cost involved, whether it would be cost effective to do it, I m not entirely sure (Training Manager, InfoTech) and limited resources is also cited by InfoTech as problematic stating that: Aside from the happy sheet which focuses on the course content and how the instructors performed on the day, we don t go beyond that. With a department of only one, doing anything else would require an army of staff (Training Manager, InfoTech). FinanceCo have used the return on investment level of Phillips model in the past to evaluate a 360-degree programme introduced but highlight that evaluation at this level is not the norm, again citing costs as the issue: The reason we did the evaluation of the 360-degree programme was because there was a significant investment in the process. So we calculated a 457% increase in performance. We don t do that with all training programmes; the amount of investment involved would determine that (Training Specialist, FinanceCo). Perceptions of Training Effectiveness The interviewees were asked to comment on their perceptions of the overall effectiveness of their training and development activities. The majority of managers believe training is successful regardless of the fact they do not systematically measure its effectiveness. The following selection of statements is indicative of perceptions of effectiveness: Training has been very successful and if it wasn t successful or delivering what we said it would deliver then it would be pulled (Training Specialist, FinanceCo). Training has been extremely successful. Aside from our in-house courses and the whole e learning thing, we spend a lot of money on further education for our staff (Training Manager, InfoTech). Training has been successful. It has been proactive and is linked to the business and individual goals (HR Generalist, VisionCo). Discussion and conclusions Training is considered one of the most pervasive methods for enhancing productivity and achieving organisational goals (Arthur et al., 2003). The ability to track and report regularly on the effectiveness of such training programmes can help cement organisational recognition of the value of training from a strategic standpoint and can help ensure continued support from all organisational levels. The present study was designed to investigate the overall nature, practice, and effectiveness of training evaluation practices in five organisational settings. The evidence suggests that few organisations truly understand the importance of not only conducting an evaluation. Furthermore, the main objective for conducting evaluation in the organisations studied is to determine whether or not the same course or vendor will be used in future. Practitioners must ensure that evaluation results are used in a manner which will be of strategic benefit other than merely using the information to make decisions on the sustainability of programmes and training providers.

11 The collective findings suggest that whilst substantial investment is being made in the process of training and development, evaluation of such activities is being conducted largely only at the reaction level. Fewer organisations focus on the learning level when evaluating training and evaluation focused on measuring behaviour and impact on results is not used in most of the organisations investigated. These findings lend support to Phillips (1996) assertion that practice of training evaluation still challenges even the most sophisticated HR departments. In reality, trainee reactions have limited value in predicting the actual transfer of learning to the work environment (Ruona et al., 2002) and positive reactions do not necessarily determine improvements in learning or behaviour (Alliger et al., 1997). The limitation of engaging in evaluation only at the reaction level is highlighted by Holton (1996) who argues that positive reactions to training act as an intervening variable impacting on learning rather than as a primary outcome of evaluation (Holton, 1996). One of the key challenges in the HRD profession is that there is no all embracing and widely accepted methodology for measuring training transfer and effectiveness (Hortwitz, 1999). The managers in the organisations investigated in this study believe training evaluation models to be overly complex and too time consuming to implement. Theoreticians need to engage with practitioners in a more effective manner to ensure that conceptual models such as the Kirkpatrick model can effectively be operationalised in practice. Currently, the divide between evaluation theory and practice is quite extensive. Practitioners have not succeeded in translating theoretical models into practical tools that can be easily accessed and employed to generate meaningful and useful evaluation information. The role of the line manager in training evaluation practice needs attention. The findings here indicate that there is an implicit expectation that line managers, either formally through performance appraisal or informally, play an important role in training evaluation. In the literature, little is currently known about the role line managers play in training and development evaluation. Further research should explore line manager perceptions of training evaluation and training effectiveness. The findings of this study must be interpreted while having regard to the limitations of the research. Five research sites are studied and therefore the generalisability of the findings is not extensive. Further studies should focus on investigating training evaluation using research designs which allow for larger samples. Also, it is worth exploring line management attitudes and perceptions of training and development evaluation as they seem to play an important role. References Alliger, G. M., and Janak, E. A., (1989), Kirkpatrick s level of training criteria: thirty years later, Personnel Psychology, Vol. 42, pp Alliger, G. M., Tannenbaum, S. I., Bennett Jr, W., Traver, H., and Shortland, A. (1997), A meta-analysis of the relations among training criteria, Personnel Psychology, Vol. 50, No. 2, pp

12 Arthur, W., Bennett Jr. W., Edens, P. S., and Bell, S. T., (2003), Effectiveness of training in organisations: A meta-analysis of design and evaluation features, Journal of Applied Psychology, Vol. 88, No. 2, pp Baldwin, T. T., and Ford, J. K., (1988), Transfer of training: A review and directions for future research, Personnel Psychology, Vol. 44, pp Bassi, L. J., Benson, G., and Cheney, S., (1996), The top ten trends, Training and Development Journal, Vol. 50. pp Becker, B., and Gerhart, B., (1996), The impact of Human Resource Management on organisational performance: progress and prospects, Academy of Management Review, Vol. 1, No. 3., pp Bee, F., and Bee, R., (1999), Training Needs Analysis and Evaluation, Institute of Personnel and Development, London. Blanchard, P. N., Thacker, J. W., and Way, S. A., (2001), Training evaluation: Perspectives and evidence from Canada, International Journal of Training and Development, Vol. 4, No. 4, pp Boxhall, P. and Purcell, J. (2003), Strategy and Human Resource Management, Palgrave, Macmillan: London. Bramley, P., (1999), Evaluating effective management learning, Journal of European Industrial Training, Vol. 23, No. 3, pp Bramley, P., and Kitson, B., (1994), Evaluating training against business criteria, Journal of European Industrial Training, Vol. 18, No. 1, pp Brown, K. G., and Gerhardt, M. W., (2002), Formative evaluation: An integrative practice model and case study, Personnel Psychology, Vol. 55, No. 4, pp Burrow, J., and Berardinelli, P., (2003), Systematic performance improvement refining the space between learning and results, Journal of Workplace Learning, Vol. 15, No.1, pp Canning, R., (1996), Enhancing the quality of learning in human resource development, Journal of European Industrial Training, Vol. 20, No. 2, pp Carey, S., (2000), The organisation of the training function in large firms, pp in H. Rainbird (eds): Learning in the workplace, Macmillan Press Ltd: London. Clarke, N., (2004), HRD and the challenges of assessing learning, International Journal of Training and Development, Vol. 8, No. 2, pp Clement, R. W., (1982), Testing the hierarchy theory of training evaluation: an expanded role for trainee reactions, Public Personnel Management Journal, pp Dionne, P., (1996), The evaluation of training activities: A complex issue involving different stakes, Human Resource Development Quarterly, Vol. 7, No. 3, pp Doyle, M., (1995), Organizational transformation and renewal a case for reframing management development?, Personnel Review, Vol. 24, No. 6, pp Endres, G. J., and Kleiner, B. H., (1990), How to measure management training and development effectiveness, Journal of European Industrial Training, Vol. 14, No. 9, pp Facteau, J. D., Dobbins, G. H., Russel, J. A., Ladd, R. T., and Kudisch, J. D., (1995), The influence of general perceptions of the training environment on pretraining motivation and perceived training transfer, Journal of Management, Vol. 21, No.1, pp

13 Garavan, T. N., (1991), Strategic human resource development, Journal of European Industrial Training, Vol. 15, No. 1, pp Garavan, T. N., (1995), HRD stakeholders: their philosophies values, expectations and evaluation criteria, Journal of European Industrial Training, Vol. 19, No. 10, pp Garavan, T. N., Costine, P., and Heraty, N., (1995a), Training and Development in Ireland: context, policy and practice, Oak Tree Press: Dublin. Garavan, T. N., Costine, P., and Heraty, N., (1995b), The emergence of strategic human resource development, Journal of European Industrial Training, Vol. 19, No. 10, pp Garavan, T., Heraty, N., and Barnicle, B., (1999), Human Resource Development literature: current issues priorities and dilemmas, Journal of European Industrial Training, Vol. 23, No. 4/5. Garavan, T. N., Collins, E., and Brady, S., (2003), Results of the 2003 National Survey of Benchmarks, CIPD, Ireland. Gerber, R., and Oaklief, C., (2003), Transfer of learning to strengthen workplace training, pp , in R. Gerber and C. Lankshear (eds): Training for a SMART workforce, Routledge: London. Hallier, J., and Butts, S., (1999), Employers discovery of training: self-development, employability, and the rhetoric of partnership, Employee Relations, Vol. 21, No. 1, pp Hamblin, A. C., (1974), Evaluation and Control of Training, McGraw-Hill: New York, NY. Harrison, R., (1997), Employee Development, Institute of Personnel and Development: London. Herold, D. M., Davis, W., Fedor, D. B., and Parsons, C. K., (2002), Dispositional influences on transfer of learning in multi-stage training programs, Personnel Psychology, Vol. 55, pp Kirkpatrick, D., (1959), Techniques for evaluating training programmes, Journal of the American Society of Training Directions, Vol. 13, pp. 3-9 and 21-26; 14, and Leach, M. P., and Liu, A. H., (2004), Investigating interrelationships among sales training evaluation methods, Journal of Personal Selling and Sales Management, Vol. 23, No. 4, pp Lee, S., (1999), Taxonomy of HRM policies of knowledge based firms: A resourcebased explanation, International Journal of Innovation Management, Vol. 3, No. 4, pp Mann, S., and Robertson, I. T., (1996), What should training evaluations evaluate?, Journal of European Industrial Training, Vol. 20, No. 9, pp Mathieu, J. E., Tannenbaum, S. I., and Salas, E. (1992) Influences of individual and situational characteristics on measures of training effectiveness, Academy of Management Review, Vol. 35, No. 4, pp McClelland, S., (1994), A model for designing objective-oriented training evaluations, Industrial and Commercial Training, Vol. 26, No.1, pp Noe, R. A., (1986), Trainees attributes and attitudes: Neglected influences on training effectiveness, Academy of Management Review, Vol. 11, No. 4, pp Orpen, C., (1999), The influence of the training environment on trainee motivation and perceived training quality, International Journal of Training and Development, Vol. 3, No. 1, pp

14 Ostroff, C., (1991), Training effectiveness measures and scoring schemes: A comparison, Personnel Psychology,?, pp Parry, S. B., (1996), Measuring Training s ROI, Training and Development, pp Phillips, (1997), Return on Investment in Training and Performance Improvement Programmes, Gulf Publishing: Texas. Phillips, J., (1996), ROI: The search for best practices, Training and Development, Vol. 50, No.2, pp Phillips, P., and Phillips, J., (2001), Symposium on the evaluation of training, International Journal of Training and Development, Vol. 4, No. 4, pp Plant, R. A., and Ryan, R. J., (1992), Training evaluation: A procedure for validating an organisation s investment in training, Journal of European Industrial Training, Vol. 16, No. 10, pp Rainbird, H., (2000), Training in the workplace and workplace learning, pp. 1-18, in H. Rainbird (eds): Training in the workplace, Macmillan Press Ltd: London. Ruona, W., Leimbach, M., Holton, E.F., and Bates, R., (2002), The relationship between learner utility reactions and predicted learning transfer among trainees, International Journal of training and development, Vol. 6, No. 4, pp Sadler-Smith, E., Down, S., and Field, J., (1999), Adding value to HRD: evaluation, investors in people and small firm training, Human Resource Development International, Vol. 4, No. 2, pp Salas, E., and Cannon-Bowers, J. A., (2001), The science of training: A decade of progress, Annual Review Psychology, Vol. 52, pp Salas, E., Kosarzycki, M. P., Burke, C. S., Fiore, S. M., and Stone, D. L., (2002), Emerging themes in distance research and practice: some food for thought, International Journal of Management Reviews, Vol. 4, No. 2, pp Savery, L. K., and Luks, J. A., (2004), Does training influence outcomes of organizations? Some Australian evidence, Journal of Management Development, Vol. 23, No. 2, pp Sayer, A., (1984), Method in Social Science a realist approach, Hutchinson. Scarbrough, H., and Elias, J. (2002) Evaluating Human Capital - Research Report, CIPD: London. Skinner, D. (2004), Evaluation and change management: rhetoric and reality, Human Resource Management Journal, Vol. 14, No. 3, pp Smith, A., (2001), Evaluation of Management training subjectivity and the individual, Journal of European Industrial Training, Vol. 14, No. 1, pp Stewart, J., (1999), Employee Development Practice, Financial Times Management: London. Stokking, K., (1996), Levels of evaluation: Kirkpatrick, Kaufman and Keller, and beyond, Human Resource Development Quarterly, Vol. 7, No. 2, pp Storey, J., (2001), Human Resource Management Today: An Assessment, pp. 3-21, in J. Storey (eds): Human Resource Management: A Critical Text, Thomson Learning. Tannenbaum, S. I., and Woods, S. B., (1992), Determining a strategy for evaluating training: Operating within organisational constraints, Human Resource Planning, Vol. 15, No. 2, pp

15 Tannenbaum, S. L., and Yukl, G., (1992), Training and development in work organisations, Annual Review of Psychology, Vol. 43, pp Van Buren, M. E., and Erskine, W., (2002), The 2002 ASTD State of The Industry Report, Alexandria, USA: American Society of Training and Development. Van der Klink, M., Gielen, E., and Nauta, C., (2001), Supervisory support as a major condition to enhance transfer, International Journal of Training and Development, Vol. 5, No. 1, pp Walker, J. W., (2001), Human Capital: Beyond HR?, Vol. 24, No. 2, Human Resource Planning, Vol. 24, No. 2, pp Warr, P., Allan, C., and Birdi, K., (1999), Predicting three levels of training outcome, Journal of Occupational and Organisational Psychology, Vol. 72, pp Warr, P., Bird, M., and Rackham, N., (1970), The Evaluation of Management Training, Gower, Aldershot. Westhead, P., and Storey, D. J., (1997), Management training in small firm performance: why is the link so weak, International Small Business Journal, Vol. 14, No. 4, pp