Evaluation: What is the True Framework? Chervon M. Moore James Madison University

Transcription

1 Article Critique: Evaluation 1 Running head: Article Critique; Evaluation Evaluation: What is the True Framework? Chervon M. Moore James Madison University

2 Article Critique: Evaluation 2 Abstract In looking at the training evaluation model of Donald Kirkpatrick it appears that the reason for evaluating is to determine how effective program the program was after it has been completed. In addition it seems as though trainers should hope for the results to present to positive and have appropriate information for those responsible for the program. In order to evaluate training programs Kirkpatrick created a model that involves a four level sequence (Reaction, Learning, Behavior, Results) of how to effectively evaluate training of all forms. Since that conception in 1959 there have been revisions and additions to his model who speak against his model as well as they feel his model is complete. In this article critique it that issue specifically will be addressed.

3 Article Critique: Evaluation 3 Kirkpatrick, D. (1959a). Techniques for evaluating training programs. Journal of the American Society for Training and Development, 13, 3-9. Content In the article titled Techniques for Evaluation Training Programs, (Kirkpatrick, 1959), is mainly focusing on the focusing on introducing the first step of the Four Level Model of Evaluation which is Reaction. Kirkpatrick defines the reaction level of his model as, how well trainees liked a particular training program. Evaluating in terms of reaction is the same as measuring the feelings of the conferees (pg. 3). The article emphasizes that this level is purely seeing what the participants liked and not evaluating whether or not in learning actually tool place. In addition the article gives a guide for evaluating the reaction level. The article makes clear that Reaction evaluations seek to see how the participants of the training felt and their personal reactions to the training. Examples of this evaluation tool are the use of happy sheets. In addition feedback forms are utilized in order to gauge their personal thoughts. Post-surveys or questionnaires are also used. These can be done immediately after the training is concluded. According to Kirkpatrick (2009), he states 1. Determine what you want to find out, 2. Use a written comment sheet covering, 3. Design the form so that reactions can be tabulated and specified, 4. Obtain honest reactions by making the forms anonymous, 5. Allow the conferees to write in additional comments not covered by the questions that were designed to be tabulated and quantified (pg.4). This guide it emphasizes the need to formulate the evaluation of reaction to be given in a way in order for the information and data to be easily tabulated and summarized. In addition, the article goes on to argue that in many other evaluations participants are being asked too many questions that require write in responses. To counteract the article suggests that using

4 Article Critique: Evaluation 4 the technique of write in responses hinders the ability to gather data on comments of reaction. To conclude the article it suggests that the reaction level of evaluation is important and that it helps the decisions that are being made by those responsible. Kirkpatrick, D. (1959b). Techniques for evaluating training programs: Part 2 Learning. Journal of the American Society for Training and Development, 13, Content In the article Techniques for Evaluating Training Programs: Part 2-Learning (Kirkpatrick, 1959), it focuses on level two of Kirkpatrick s evaluation model. This level is the learning level. The article aims to build upon the aforementioned Techniques for Evaluation Training Programs, (Kirkpatrick, 1959). Level two of the evaluation model seeks to evaluate whether or not the learner had an increase in knowledge of the material in the training. It also seeks to evaluate whether the learners learned what was intended to be taught as well as did they experience what was intended for the learner to experience. The tool used to asses this is an assessment given before and after the training. This step takes more thought than what was required in the reaction evaluation. The article defines learning as what principles, facts, and techniques were understood and absorbed by the conferees? (Kirkpatrick, 2009). The article gives guideposts for that Kirkpatrick suggests. 1) The learning should be able to be measured so that quantitative data can be determined. 2) The learning should be measured objectively. 3) A control should be used. 4) Statistics should be used in order to make correlation inferences. The article emphasizes that it is more difficult measure learning as it reaction.

5 Article Critique: Evaluation 5 Kirkpatrick, D. (1960a). Techniques for evaluating training programs: Part 3 Behavior. Journal of the American Society for Training and Development, 14, In the article Techniques for Evaluating Training Programs: Part 3 Behavior (Kirkpatrick, 1959), it focuses on the third level of Kirkpatrick s model which is behavior in which the main emphasis is before and after data collection in order to assess whether or not learning has taken place. Within level 3 of the evaluation behaviors of the learner are evaluated. This portion of the evaluation seeks to understand if the learners applied what was learned and changed their behavior accordingly. The article cites Robert Katz based on an that was written in 1956 entitled Skills Can Be Sharpened and it states that if a person wishes to change his job behavior 1) He must want to improve themselves (strictly voluntary), 2) He must acknowledge the weakness and know where there is a weakness, 3) His work environment must be able to allow him to do so 4) He must have help 5) He must have the opportunity. The article suggests that a more scientific approach is needed and many factors must be considered as well as the use of guideposts. Those guide posts are 1) a systematic appraisal should be used before and after the training, 2) the appraisal should be used by either the person receiving the training, supervisor, superiors, or subordinates, 3) a statistical analysis should be done in order to compare the before and after, 4) the post training should be made three or more months after 5) a control group should be used. The article also gives information on many evaluation studies that used before and after approaches. These studies are Fleishman-Harris Studies, Survey Research Center Studies, Lindholm Study, The Blocker Study, The Tarnopol Approach, The Moon-Hariton Study, The Buchanan-Brunstetter Study, The Stroud Study, and the Sorensen Study.

6 Article Critique: Evaluation 6 Kirkpatrick, D. (1960b). Techniques for evaluating training programs: Part 4 Results. Journal of the American Society for Training and Development, 14, Content In the article, Techniques for Evaluating Training Programs: Part 4 Results the main\ focus is on the fourth level of Kirkpatrick s model which is results. According to Kirkpatrick, types of results can be classified as reduction in costs, reduction of turnover and absenteeism, reduction of grievances, increase in quality and quantity of production (pg. 28). 4. Level 4 of evaluations seeks to assess the effect on the organization or environment. Measures would be an organization s key performance indicators. This portion of evaluation allows the designer and the organization to see if the instruction effected the organization overall. This information would also be useful in order to find the Return on Investment. The article suggests that there a complex factors affecting the results level of evaluation and it is recommended to start with the aforementioned levels, reaction, learning, and behavior first. In addition the article attempts to describe other training program evaluations. The first was on safety programs. In the study the purpose was to reduce the amount of accidents. The program proved to not have desirable results and a new approach was then taken for future training efforts. Another that was used was a Postal Carrier Training, which they used control groups in order to emphasize the results. An Insurance Company Study was also discussed, they utilized a before and after approach in which they measured six months after in order to compare. Lastly a Cost Reduction Study was also discussed in which to get at the results level they utilized interviews, and questionnaires.

7 Article Critique: Evaluation 7 Holton, E. F. (1996). The flawed Four-level evaluation model. Human Resource Development Quarterly, 7, Content In the article titled The Flawed Four-Level Evaluation Model (Holton, 1996), the main focus is that the four-level evaluation model of Kirkpatrick is flawed and missing components. Holton proceeds to establish a model that meets the model criteria and is not merely taxonomy. Holton argues that the model is truly a taxonomy as opposed to a model. He suggests that companies are ill-equipped against global competitors and that the evaluation tool of Kirkpatrick is not working and therefore not allowing companies to be as successful as its global competitors. Holton also suggests that taxonomies are the classifications of the stages between the stages of evaluation. He also argues that Kirkpatrick s model is a taxonomy of outcomes and causal conclusions (pg. 6). He suggests a more complete model should be used if causal conclusions are to be made. He states by way of Kilmoski (1991), that a model should have the following criteria, 1) elements represented as constructs, 2) the relationship between constructs 3) there are boundaries of generalization, 4) the system changes are described, 5) deductions are made as hypotheses, 6) predictions are made. He argues that the model of Kirkpatrick does not have any of the aforementioned criteria. He also feels that the variables that affect the transfer of learning process are omitted. His proposed model includes three primary outcomes, learning, individual performance, and organizational results. To further his argument he states that there are important differences between his and Kirkpatrick s model. The differences are 1)the exclusion of reaction as a primary outcome, 2) individual performance is used instead of the

8 Article Critique: Evaluation 8 behavior level because it is a broader construct as well as the fact that individuals choose to learn3) inclusion of primary and secondary influences on outcome. He further states that based on previous research there seems to be no correlation between reactions and learning. In addition he includes motivation to learn in his model to state that learning has a direct relationship with the motivation to learn. Motivation to transfer is another variable included in his model. Transfer design and transfer conditions are other variables utilized in the model as well. Lastly, he believes that there must be a link to organizational goals as well. Kaufman, R. & Keller, J. M. (1994). Levels of evaluation: Beyond Kirkpatrick. Human Resource Development Quarterly, 5, Content In the article, Levels of Evaluation: Beyond Kirkpatrick (Kaufman and Keller, 1994), the main focus is extending Kirkpatrick s model in order to answer specific questions of evaluation. They suggest that these questions should answer what the results are that are being sought, the processes that must be undergone, as the resources to be used. The results questions that Kaufman and Keller presented are as follows: 1) Are you concerned with the impact, consequences and payoffs of what your organization will or should deliver to external clients? 2) Are you concerned with the quality of what your organization will or should deliver to external clients? 3) Are you concerned with the quality of what an individual or a small group within your organization will or should produce for others within the organization? (pg.374). When discussing these questions they are discussed in terms of mega, macro, and micro level planning. In the mega level the primary beneficiary is society. In the macro level the primary beneficiary is

9 Article Critique: Evaluation 9 a client or small group. In the micro level it is concerned with internal results within an organization. The process and resource questions are as followed: 4) Are you concerned with the efficiency of the methods and procedures that will or should be used by an individual or small group, 5) Are you concerned with the availability and/quality if resources that will or should by an individual or small group (pg. 375). Kaufman and Keller feel that these questions must be answered in order to define the ends that need to be achieved and align the processes and resources toward the desired result. To further extend Kirkpatrick s framework society impact is believed by Kaufman and Keller to be important to include. This is urged to be considered in the mega level. They encourage organizations to consider societal consequences. Kaufman and Keller propose a newer model that expands learning to be defined as acquisition and behavior to be defined as application instead of transfer. Results is unchanged in terms of purpose but they suggest that results should be expanded to include benefits, organizational performance improvements and cost-result assessment. Relation to AHRD In reading the articles it is clear that all of the authors feel that in order for training programs to be effective those programs can only be effective when utilizing some form of model of evaluation. In general evaluation is relative to this course in that we as students are posed with the tasks of seeing how effective self-selected programs are in order to then recommend to our clients whether or not their programs need adjustment or not. In addition it also is helpful in that it gives the student a tool to evaluate their programs. The articles provide a wide range of viewpoints and perspectives on what should be included in the evaluation. The practical application is relative in that the main purpose of evaluation is to provide information for decision making and learning. For practical application each article was relevant to what

10 Article Critique: Evaluation 10 organizations need in order to be effective. Most organizations want to have a positive impact on the communities they serve. By applying an evaluation as a way to quantify and improve that impact a greater result is more than likely going to arise. The application of evaluation from a funders prospective clarifies the organization's goals. Most important, however, is what the organization does with the data from the evaluation. Critique Evaluation involves the collection of formal and informal data and information for organizations to achieve their goals. In this section the critic will group Techniques for Evaluation Training Programs, (Kirkpatrick, 1959), Techniques for Evaluating Training Programs: Part 2-Learning (Kirkpatrick, 1959), Techniques for Evaluating Training Programs: Part 3 Behavior (Kirkpatrick, 1959) Techniques for Evaluating Training Programs: Part 4 Results (Kirkpatrick, 1959) together as one for the purposes of referral to Kirkpatrick s Four- Level Evaluation Model. This model has been used since 1959 which is evident. My opinion of this model is solely as a foundation and not a model that is still proven to hold true in every form today. The critic did find that a lot has changed since 1959 in regards to the model. In the article The Kirkpatrick Four Levels: A Fresh Look After 50 Years (Kirkpatrick, 2009), it states It is now time to pull the wraps off not only the true Kirkpatrick model, but also the complete model. The end is the beginning because you are working from Level 4 to Level 1. You will determine what success looks like (L4) and then subsequent critical behaviors and organizational drivers (L3). Next we deal with the required KSAs or competencies needed for participants to be able to perform their on the job behaviors (L2). Finally we address the learning environment and conditions that will support effectiveness and enjoyment (pg. 7). The critic can

11 Article Critique: Evaluation 11 see that there has been a great change as the years have progressed and thus provided to possibly be more useful. The new model that this article argues is as followed: (Phillips, 2002)The performance analysis [included in results] uncovers several points of view on a problem or opportunity, determining any factors that include barriers to successful performance and the proposal to a solution discovered. According to Phillips (2002), when carrying out a performance analysis evaluation it is best accomplished when linking them with Kirkpatrick s Four Levels of Reaction. The three levels most relative are job performance needs linked to behavior (level 3), business needs linked to impact (level 4) and individual needs linked to reaction (level 1). In tying in Kirkpatrick s four levels. In looking at the supporting information for this model I am not in complete agreement. There is a strong point in that the levels are not clearly defined so it leaves things open ended which can be good, but that also is the weak point that there is not defined outcomes of each level. In an article by Watkins, Leigh, Foshay, and Kaufman (1998), they state that Kirkpatrick s model is a basic model for identifying training interventions. In addition they state that the model has provided a solid basis proposed changes have not been frequently adopted. It is therefore likely time for professionals to reevaluate the utility and responsiveness of the Kirkpatrick framework to meet the value-added requirements of today s organizations (pg.90). In looking at these statements it further supports the idea that Kirkpatrick s model is more of a foundation than a model. The

12 Article Critique: Evaluation 12 critic believes that it provides a basis and structural foundation of each phase as an opposed to it actually being an inclusive model Which Kaufman, Keller, and Holton all in their articles feel that the model should in fact be expanded as well. The critic does agree that the model should be expanded. In Holton s article, the critic feels that his ideas were pertinent in that Kirkpatrick s model is somewhat flawed. There are strong points to his article with his ideas to use individual performance instead of the behavior level because it is a broader construct as well as the fact that individuals choose to learn as well as the inclusion of primary and secondary influences on outcome. It can also be agreed in the opinion of the critic that there really is not a correlation between reactions and learning. Just based on a reaction level evaluation which usually include things such as how did you like the snacks? how were the materials? Etc. You cannot firmly say whether or not a participant has truly learned based on those types of questions. The critic however does feel that his article proves to have weak areas and is not inclusive of organizational outcomes as what Keller and Kaufman (1994) suggest. In Kaufman and Keller s article (1994), the critic finds there are strong points in this article that include the mega level, macro level, and micro level. The critic feels these are vital components to look at in the scope of an evaluation tool. In the opinion of the critic it is important to look at these levels because each level gives a beneficiary that can also be seen as a stakeholder because they are affected by the training program. The critic feels that this should be utilized in addition to the model of Kirkpatrick. This should merely be an extension.

13 Article Critique: Evaluation 13 In the critics opinion all of the articles make valid points however, Kirkpatrick s model should be used as the foundation and the other models should build upon it and not to be used alone. Citations Holton, E. F. (1996). The flawed Four-level evaluation model. Human Resource Development Quarterly, 7, Kaufman, R. & Keller, J. M. (1994). Levels of evaluation: Beyond Kirkpatrick. Human Resource Development Quarterly, 5, Kirkpatrick, D. (1959a). Techniques for evaluating training programs. Journal of the American Society for Training and Development, 13, 3-9. Kirkpatrick, D. (1959b). Techniques for evaluating training programs: Part 2 Learning. Journal of the American Society for Training and Development, 13, Kirkpatrick, D. (1960a). Techniques for evaluating training programs: Part 3 Behavior. Journal of the American Society for Training and Development, 14, Kirkpatrick, D. (1960b). Techniques for evaluating training programs: Part 4 Results. Journal of the American Society for Training and Development, 14, Kirkpatrick, J. (2009). The Kirkpatrick four Levels: A fresh look after 50 Years Training Magazine Phillips, J. & and Phillips, P. (2002). "Reasons why training & development fails...and what you can do about it." Training Magazine

14 Article Critique: Evaluation 14 Watkins, R., Leigh, D., Foshay, R., Kaufman, R. (1998). Kirkpatrick plus: Evaluation and continuous improvement with a community focus. Educational Technology Research and Development