HRD Evaluation and the Public Interest 1

Transcription

1 HRD Evaluation and the Public Interest 1 Darlene Russ-Eft Professor & Chair Adult Education & Higher Education Leadership Oregon State University 210 Education Hall Corvallis, OR U.S.A. darlene.russeft@oregonstate.edu Phone: Stream: Constructing and Deconstructing: Insights on Corporate Social Responsibility and Ethics within HRD Submission Type: Refereed Key Words: Evaluation, ethics, HRD, social responsibility. 1

2 HRD Evaluation and the Public Interest This paper examines the issues involved in the use of ethical standards related to social responsibility using the two ethical codes: the American Evaluation Association Guiding Principles for Evaluators and the Academy of Human Resource Development Standards on Ethics and Integrity. This examination will take the perspective of an internal evaluator working within a private organization. Using that perspective, the discussion will consider the complex issues involved in following such standards, particularly when undertaking evaluations within organizational settings. Suggestions for future research are described. Evaluations within organizational settings and evaluations of HRD interventions in particular are guided by two or more sets of ethical standards. Such ethical standards emerge from the separate fields of evaluation and from HRD. Specifically, the field of evaluation possesses a number of different ethical statements, and these various statements, reviewed in Russ-Eft (2004), include the American Evaluation Association (AEA) Guiding Principles for Evaluators (2004), the Program Evaluation Standards (Sanders, 1994), and the Canadian Evaluation Society (CES) Guidelines for Ethical Conduct (n.d.). The CES Professional Designation Core Committee (2008) undertook an analysis of the ethical standards emerging from AEA, CES, and the Australasian Evaluation Society (2000). They identified three major domains: Competence, Integrity, and Accountability. Russ-Eft, Bober, de la Teja, Foxon, and Koszalka (2008) presented a similar but more extensive analysis, including these same ethical codes as well as some other national evaluation codes and the AHRD Standards on Ethics and Integrity (AHRD, 1999). This latter analysis revealed the following major domains: Professional Foundations and Competence, Professional Responsibility, Integrity, Accountability, Respect for People, and Social Responsibility. Only the AEA Guiding Principles for Evaluators and the AHRD Standards on Ethics and Integrity contain specific statements focused on social responsibility. The purpose of the present paper is to examine the issues involved in the use of ethical standards related to social responsibility (or Responsibilities for General and Public Welfare in the AEA Guiding Principles) using the two ethical codes that promote such practices. The perspective of this examination will be that of an internal evaluator working within a private organization. Using that perspective, the discussion will then turn to a dissection of these statements and a consideration of the issues involved in following such standards, particularly when undertaking evaluations within organizational settings. Some questions for this discussion include: Why do we (or why don't we) need principles like these? To what extent is this a realistic expectation of evaluators in the rough-and-tumble world of practice? What are ways that the principles could be manifested in our practice? Codes Promoting Social Responsibility The AHRD Standards state HRD professionals are aware of their professional responsibilities to the community, the society, in which they work and live, and the planet. They work to minimize adverse efforts on individuals, groups, organizations, societies, and the environment (p. 3). Similarly, one of AEA s Guiding Principles for Evaluators states Evaluators articulate and take into account the diversity of general and public interests and 2

3 values that may be related to the evaluation (p. 3). This statement further specifies that evaluators should include relevant perspectives and interests of the full range of stakeholders, should consider not only the immediate operations and outcomes of whatever is being evaluated, but also its broad assumptions, implications and potential side effects, should allow all relevant stakeholders access to evaluative information, should maintain a balance between client needs and other needs, have obligations that encompass the public interest and good (p. 4). The next sections provide greater detail on each of the codes. This discussion will begin with the AHRD Standards on Ethics and Integrity and then it will turn to the AEA Guiding Principles for Evaluators. AHRD Standards on Ethics and Integrity. HRD evaluators working within organizations recognize their professional responsibilities to the organization, but an awareness of any responsibilities to the community, the society, and the planet seem may seem somewhat outside of those evaluators sphere. Indeed, Swanson (2001) and Swanson and Holton (1999) focused their discussions of assessing HRD results on financial benefits, knowledge and expertise gains, and participant and stakeholder perceptions. These would appear to be the kinds of results that might be expected from an HRD intervention or initiative and would be of greatest interest to the organization and its employees. Furthermore, as an internal evaluator examining the processes and the outcomes related to a new sales approach or some line-worker training program, one might find it a stretch to see the work as having some effect on society in general or on the planet. Preskill and Russ-Eft (2003), however, using a systems approach to evaluation, argued that such evaluators must look beyond the immediate program in order to take into account the environment surrounding the evaluand. By doing so, the evaluator should recognize the influence of the community, the society, and the planet on the evaluand and the evaluation, as well as the influence of the evaluand and the evaluation on the community, the society, and the planet. As an example, an internal evaluator might undertake an evaluation of the customer service process within a global organization. This evaluation might reveal that processing orders in the United States would be more efficiently (and inexpensively) undertaken in India, after examining the organization s infrastructure and the global competition). The evaluation findings might, then, result in workers within a particular U.S. community losing their jobs with a related significant decline in the tax base for that community. At the same time, the workers in India may now realize financially and professionally rewarding employment. (See Friedman, 2007 for more such examples.) Thus, the evaluator in this case should be aware of the ripple effects of the evaluation. Furthermore, whether information regarding the evaluand and the evaluation should be shared beyond the organization, however, would seem to depend upon the specific situation and the kinds of impacts and ripple effects. Another aspect of the AHRD Standards concerns working to reduce adverse effects on individuals, groups, organizations, societies, and the environment (p. 3). Again, such concerns may seem beyond the scope of evaluation work within an organization. After all, if an evaluation of an organizational process suggests a more efficient approach, is it not the evaluator s responsibility to provide some recommendations to that effect, even though that may lead to personnel reductions? The general public will dramatically be impacted [by an evaluation effort] (especially in an economy such as today) and depending on the community may change the entire way of living for some segments of the population (M. Pankey, personal communication, January 31, 2009). Within this context, the evaluator needs to consider the survival of the organization as well as the impacts on the community or the general public. After 3

4 all, if the organization does not survive because of outmoded and inefficient processes, the community will still lose that employer. In this case, the evaluator may need to consider the sustainability of the organization and the sustainability of the community. Russ-Eft (2008), for example, argued for the connection among evaluation, ethics, and sustainability both in terms of programmatic sustainability (e.g., Scheirer, 2005; Scheirer, Hartling, & Hagerman, 2008) as well as environmental, economic, and social sustainability (e.g., Goodland, 2002; Krizek & Power, 1996). Yet another part of the AHRD Standards suggests that HRD practitioners and evaluators should be concerned about and work to mitigate the causes of human suffering (p. 3). Certainly evaluations that are focused on international development efforts or community development initiatives may address such issues. Mitigating the causes of human suffering is a laudable goal, but it is not clear that that goal can be accomplished in all evaluation work. Turning to the previous example of an evaluation that potentially could result in downsizing and layoffs, we might argue that the evaluation would then lead to an increase in human suffering. At the same time, if the evaluator did not make the needed recommendation, might we claim that the evaluator violated the following Program Evaluation Standards? A 10 Justified Conclusions. The conclusions reached in an evaluation should be explicitly: A11 justified, so that stakeholders can assess them. Impartial Reporting. Reporting procedures should guard against distortion caused by personal feelings and biases of any party to the evaluation, so that evaluation reports fairly reflect the evaluation findings. It would seem reasonable, however, to have the evaluator consider the multiple impacts of any evaluation undertaken during times of economic stress for the organization, for the community, and for the society. The report to the client should consider the various impacts on the organization, the community, and the society that might result from any of the recommendations. This section has considered some of the statements within the AHRD Standards on Ethics and Integrity that emphasis the public interest and that may pose difficulties for HRD evaluators and practitioners. Now the discussion will turn to the AEA Guiding Principles for Evaluators and some of the similar issues that can arise. AEA Guiding Principles for Evaluators. Turning to the AEA Guiding Principles for Evaluators, we see that there are several separate aspects or principles related to Responsibilities for General and Public Welfare. The first of these focuses on involving the full range of stakeholders in the evaluation. Although this may seem quite reasonable, there can be instances when such involvement does not take place. First, high-level executives and administrators may simply be unavailable (M. Pankey & S. Star, personal communication, January 30, 2009). In some cases, these executives and administrators may want to delegate such involvement to others, because of either lack of time or lack of interest. A second issue may involve the infeasibility in engaging the full range of stakeholders. An organization may feel somewhat territorial about its clients and may be unwilling to have the evaluator contact certain clients or even any clients, even when that evaluator happens to be part of that organization. Another issue may relate to the politics of evaluation within organizations, in that a specific stakeholder may avoid involvement in order to discredit or subvert the evaluation. In all of these situations, an internal evaluator may want to raise questions regarding the appropriateness and value of the evaluation effort, given the political climate. It may also be helpful to point to the emphasis in the Guiding Principles for such involvement, to outline the evaluation literature that emphasizes stakeholder involvement (e.g., Patton, 2008; Stake, 1967, 4

5 1983; Torres, Preskill, & Piontek, 2005), and to suggest the following Program Evaluation Standard. U1 Stakeholder Identification. Persons involved in or affected by the evaluation should be identified, so that their needs can be addressed. None of these actions can necessarily ensure stakeholder involvement, but the evaluator should at least make every possible attempt to secure such involvement. A second aspect included in the AEA Guiding Principles concerns the issue of the evaluation considering the broad assumptions, implications, and potential side effects as well as the immediate operations and outcomes (p. 4). Again, this might be viewed as laudable when undertaking a large-scale evaluation of a publicly funded program. Certainly, even though it is likely to ultimately focus more narrowly, an evaluation of a federally funded program should at least start by considering the various aspects of a logic model, from the assumptions and resources to the activities and operations to the immediate and short-term outcomes to the longterm outcomes, ultimate impacts, and any side effects. Within the context of an individual organization, however, such a sweeping mandate may be unwarranted, counter-productive, and impractical. Through stakeholder involvement, the evaluator may have identified the focus of the evaluation (Russ-Eft & Preskill, 2005). Typically such a focusing effort results in just that a focusing of the evaluation on the most critical issues. For an internal evaluator undertaking an evaluation of a training program within the organization, the focus might be upon the results as shown by a learning test. The stakeholders may not be at all interested in long-term outcomes, ultimate impacts, and any side effects. To broaden that focus may lead to lengthy and complex evaluations that overrun the budget and the schedule, violating the following Program Evaluation Standards: U6 Report Timeliness and Dissemination. Significant interim findings and evaluation reports should be disseminated to intended users, so that they can be used in a timely fashion. F3 Cost Effectiveness. The evaluation should be efficient and produce information of sufficient value, so that the resources expended can be justified. Since many organizations avoid evaluations because they are viewed as wasting precious program resources, the evaluator must use sound judgment in deciding how broad and comprehensive the evaluation should be. A third aspect of the Guiding Principles for Evaluators emphasizes that freedom of information is essential in a democracy (p. 4). Again, within the context of an evaluation of a federal, state, or local program, the emphasis on freedom of information and the widespread sharing of information seems quite reasonable. Furthermore, this recommendation seems appropriate, given the literature on communication and dissemination (Torres, Preskill, & Piontek, 2005) and on utilization of evaluation findings (Patton, 2008). Within a private organization, however, this aspect may be difficult to accomplish. In some cases, upper management may want to hold the information confidential for a specified time period in order to develop specific implementation procedures. In other cases, the organization may view the information as proprietary and may be unwilling to share what might be considered trade secrets with the greater public. For example, an evaluation of a new sales process or a new approach to customer service may not be something that an organization would want to share with the public, since those efforts might provide the organization some competitive advantage. 5

6 Another part of this third aspect involves providing stakeholders with evaluation findings in forms that respect people and honor promises of confidentiality. Of all of the portions of the fifth Guiding Principle, this appears to be one that should be incorporated in all evaluations, regardless of the setting, the context, and the culture. As described by Russ-Eft, et al. (2008), this principle appears in all evaluation and research ethical codes. Furthermore, it forms the basis for one of the competencies validated by the ibstpi 2 work, namely Professional Foundations: 4. Observe ethical and legal standards. Not only should the communications be respectful, they should also present results clearly and simply so that clients and other stakeholders can easily understand the evaluation process and results (AEA, 2004, p. 4). Again, this aspect captures what might be considered good practice on the part of the evaluator, and it is included as one of the ibstpi evaluator competencies, namely Professional Foundations: 1. Communicate effectively in written, oral, and visual form. A fourth aspect in the AEA Guiding Principles concerns balancing the client needs and other needs. Within a private organization, it is not clear what these other needs might be, except for other possible stakeholders. Again, the other needs as separate from the client, seem relatively clear when considering a government program. For example, some national government agency may be the client, and the other needs may be those of the states, districts, counties, localities, and the general public. Certainly an evaluator examining a government program should be cognizant of the variety of needs that exist. In contrast, within a private, for-profit organization, the CEO and the VP of Sales may be the client for an evaluation of a new sales process, while the others may include the sales professionals, sales assistants, and even the customers. In addition to considering the needs of these other stakeholders (if possible), the evaluator should identify the needs that exist within the organization s leadership, infrastructure, communication processes, and so forth. It may be, however, that the time and budget constraints prohibit such a wide-ranging investigation. This particular principle further suggests that the evaluator should determine whether continued work on the evaluation is advisable and make clear any significant limitations As an internal evaluator, unless one is willing to leave one s employment and move into independent consulting, it is highly unlikely that the person would simply walk away from the evaluation project. At best, the evaluator could identify and report on the limitations and the possible consequences of those limitations. The final consideration involves obligations that encompass the public interest and good (AEA, 2004, p.4). Such obligations seem particularly clear when the evaluation focuses on a governmental program, being supported by public funds. Indeed, this Principle acknowledges the special responsibility when an evaluator is undertaking work that is supported by public funds. However, it continues with the statement that clear threats to the public good should never be ignored in any evaluation. This aspect appears to be somewhat similar to that included in the AHRD standards indicating that HRD professionals should attempt to reduce harm to individuals, groups, organizations, societies, and the environment. With evaluations taking place within organizations, one wonders what the public interest and good might represent and what harm might need to be avoided, other than the previous example of some potential downsizing effort arising from the evaluation. Thus, it is difficult to determine what threats to the public good might arise, however, with an evaluation of a supervisory training program, of a customer service training initiative, of a new sales process, or of a new strategic initiative. Conclusions and Implications 6

7 Let us return now to the questions asked at the beginning of this paper. First, why do we (or why don't we) need principles like these; and is a social responsibility principle a realistic expectation of evaluators? The previous discussion pointed to the need for the AHRD Standards on Ethics and Integrity and the AEA Guiding Principles for Evaluators. Specifically, it argued that evaluators working on publicly funded evaluations of publicly funded programs should adhere to this Guiding Principle of Responsibilities for General and Public Welfare and the related ethical standards appearing in the AHRD Standards on Ethics and Integrity. At the same time it suggested that there may be limitations to an evaluator s ability to adhere to each of the principles and standards in all situations and all contexts; in particular, it pointed to limitations that might exist for internal evaluators working within private organizations. Responding to the question about how the principles can be manifested in our practice, the above discussion also identified some of the situations and contexts in which the principles can apply. It furthermore suggested instances when the principles might be reasonable to use and when they could pose challenges for the evaluator and the evaluation. Several implications for research emerge from this examination. A survey of evaluators who are working within organizational settings should be undertaken to determine the level of awareness of this particular Guiding Principle and of the related standard in the AHRD Standards on Ethics and Integrity and the degree to which these are being followed. Such a survey should include some questioning as to the conflicts and challenges that such evaluators and HRD practitioners experience in attempting to use these principles. With some questioning as to demographics, the study could examine whether differences emerge based on such factors as (a) whether the person functions as an internal or external evaluator and (b) in which locations and which cultures the person operates. Might there appear to be more or less conflict encountered by the internal evaluators? And, what might be the reasons for such results? What might be the results when examining different cultures? After all, both sets of ethical standards were developed in the North American context. Might these issues simply not arise for HRD evaluators operating in organizational settings in Asia, Africa, and South or Central America? Another research effort might focus on gathering case studies or critical incidents showing the actual use of this Guiding Principle and of the related AHRD Standards by evaluators and HRD practitioners working in organizational settings and some of the outcomes and consequences of such use. The reporting, analysis, and interpretation of such cases and incidents may prove helpful to the profession and to novice evaluators. This effort might reveal that certain aspects of this principle or standard never actually appear in practice. If so, there may need to be a rethinking and refinement of the statement. In conclusion, as an internal evaluator working within a private firm, I found it extremely helpful to have these principles and standards to guide my practice. In particular, I was able to review them myself when trying to make certain decisions, and I shared them with my internal clients and stakeholders to indicate some of the rationale for certain actions undertaken as part of the evaluation. Such sharing of the AEA Guiding Principles for Evaluators and the AHRD Standards on Ethics and Integrity helps to inform and educate my clients and stakeholders and to build the organization s evaluation capacity. Furthermore, when potentially problematic questions or challenges arose in the course of an evaluation, I contacted colleagues within the American Evaluation Association or the Academy of Human Resource Development to help me identify some reasonable options and possible actions. The kinds of reflections presented in this paper provide an opportunity to consider how and when our professional guidelines fit into our practice. Looking at the AEA and AHRD social responsibility principles from the perspective of 7

8 an internal, organizational evaluator illuminates some of these nuances and some of the challenges. Examining the principles from this perspective and from other perspectives could strengthen our practice more broadly. References Academy of Human Resource Development (1999). Standards on ethics and integrity. Bowling Green, OH: Author. American Evaluation Association (2004, July). Guiding principles for evaluators. Retrieved January 30, 2009 from Canadian Evaluation Society (n.d.). CES guidelines for ethical conduct. Retrieved January 30, 2009 from CES Professional Designation Core Committee (2008, January 31). Professional designation project Comparison of evaluation ethics. Retrieved January 30, 2009 from: tte_kuji-shikatani_keiko.pdf Friedman, T. L. (2007). The world is flat: A brief history of the twenty-first century (Further updated and expanded). New York: Picador. Goodland, R. (2002). Sustainability: Human social, economic and environmental. In P. Timmerman and the faculty of environmental studies. York University (Eds.). Social and economic dimensions of global change: Vol. 5, Encyclopedia of global environmental change. Hoboken, NJ: Wiley. Krizek, K. & Power, J. (1996). A planner s guide to sustainable development (Planning Advisory Service Report No. 467). Chicago, IL: American Planning Association. Patton, M. Q. (2008). Utilization-focused evaluation (4 th ed.). Thousand Oaks, CA: Sage. Preskill, H., & Russ-Eft, D. (2003). A framework for reframing HRD evaluation practice and research. In A. M. Gilley, L. Bierema, & J. Callahan (Eds.), Critical issues in HRD (pp ). Cambridge, MA: Perseus. Russ-Eft, D. (2008, November). Sustainable development: Is there a role for HRD ethics and evaluation? Keynote Address, Presented at the 7 th Annual Conference on Asian HRD, Bangkok, Thailand. Sanders, J. R., & the Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: How to assess evaluations of educational programs (2 nd ed.). Thousand Oaks, CA: Sage. Scheirer, M. A. (2005). Is sustainability possible?: A review and commentary on empirical studies of program sustainability. American Journal of Evaluation, 26, Scheirer, M. A., Hartling, G., & Hagerman, D. (2008). Defining sustainability: Outcomes of health programs: Illustrations from an on-line survey. Evaluation and Program Planning. Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record, 68, Stake, R. E. (1983). Responsive evaluation. In International encyclopedia of education. Oxford: Pergamon. Swanson, R. A. (2001). Assessing the financial benefits of human resource development. Cambridge, MA: Perseus. Swanson, R. A., & Holton, E. F. III (1999). Results: How to assess performance, learning, and perceptions in organizations. San Francisco: Berrett-Koehler. 8

9 Torres, R. T., Preskill, H., & Piontek, M. E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations (2 nd ed.). Thousand Oaks, CA: Sage. Author Note I want to thank the students in the Oregon State University AHE 532 course focused on program evaluation for their lively discussion of the issues involved in this paper. It should be recognized that these students are employed in business and industry, governmental and nongovernmental agencies, and in educational institutions. In particular, I would like to acknowledge the contributions of the following individuals: Megan Pankey, Sequoia Star, Mary Amick, Liz Coleman, Bregetta Gutierrez, Erin Heim, Kristine Keizer, Heather Masterson, LeeAnn Mikkelson, Melissa Olson, Brianna Patterson, Shannon Sims, and Edward Woods. Footnotes 1 A shorter version of this paper will be published in the American Journal of Evaluation. 2 ibstpi refers to the International Board of Standards for Training, Performance, and Instruction. See 9