This article will identify and discuss the unique conceptualization. The CEFP as a Model for Integrating Evaluation Within Organizations

Transcription

1 Michael Quinn Patton, PHD The CEFP as a Model for Integrating Evaluation Within Organizations ABSTRACT: This article identifies and discusses the innovative contributions of the Collaborative Evaluation Fellows Project (CEFP) as a model for evaluation training and organizational development. The importance of the CEFP model to the effectiveness and accountability of nonprofit organizations is best understood within the larger context of the historical emergence of evaluation as a specialized field of professional practice and the documented challenges of integrating evaluation within organizations. Key elements of the model discussed include a strong commitment to evaluation use (utilization-focused evaluation); careful attention to evaluation standards; developing core evaluation competencies among fellows; supporting collaboration among evaluation fellows, facilitators, and faculty; working to infuse evaluative logic, values, and thinking into the planning and management processes of the American Cancer Society; modeling effective evaluation practice by intensively evaluating its own effectiveness, both internally and with external reviewers; and using the findings to incorporate lessons learned, overcome barriers, and adjust to emergent challenges. This article will identify and discuss the unique conceptualization and innovative contributions of the Collaborative Evaluation Fellows Project (CEFP) as a model for evaluation training and organizational development. To understand these contributions and their importance to the effectiveness and accountability of nonprofit organizations, it is helpful to understand the historical emergence of evaluation as a specialized field of professional practice and the particular challenges that the history of evaluation presents. After a brief historical overview, this article will identify key elements in the CEFP model that hold larger lessons for integrating evaluation within organizations. Evaluation in the United States has grown up in the projects, federal projects spawned by the Great Society legislation of the 1960s. When the federal government of the United States began to take a major role in alleviating poverty, hunger, and joblessness during the Depression of the 1930s, the closest thing to evaluation was the employment of a few jobless academics to write program histories. It was not until the massive federal expenditures on an awesome assortment of programs during the 1960s and 1970s that requirements for accountability began to be included in legislation. Program evaluation as a distinct field of professional practice was born of several lessons from that period of large-scale social experimentation, lessons about both programming and evaluation: 1) there are not enough resources to do all the things that need doing; 2) it takes more than money to solve complex human and social problems. As not everything can be done, it is necessary to evaluate what works and does not work to support decision making about which things are worth doing; 3) evaluation, to be useful and actually used, requires capacity building in organizations, so that they are ready and able to construct and use evaluation systems for program improvement and decision making; 4) evaluators need special training and skills, beyond the original focus on research methods and measurement, to facilitate and implement useful evaluations. The CEFP has incorporated these lessons that have been learned into its design and implementation in ways that put it on the cutting edge of evaluation training and capacity building. To appreciate the importance of this for the profession more generally, some additional historical context will be helpful. A Vision for Evaluation While pragmatists turned to evaluation as a commonsense way to figure out what works and is worth funding, visionaries were conceptualizing evaluation as the centerpiece of a new kind of society: The experimenting society. Donald T. Campbell gave voice to this vision in his 1971 address to the American Psychological Association: The experimenting society will be one which will vigorously try out proposed solutions to recurrent problems, which will make hard-headed and multidi- Michael Quinn Patton, PhD, Professor, The Union Institute and Director, Utilization-Focused Evaluation, South Minneapolis, Minnesota. Address for correspondence: Michael Quinn Patton, The Union Institute and Utilization-Focused Evaluation, th Avenue, South Minneapolis, MN CANCER PRACTICE March/April 2001, Vol. 9, Suppl. 1 S11 American Cancer Society /01/$15.00/S11 S11 S16

2 S12 CANCER PRACTICE March/April 2001, Vol. 9, Suppl. 1 mensional evaluations of the outcomes, and which will move on to other alternatives when evaluation shows one reform to have been ineffective or harmful. We do not have such a society today (Campbell, 1991, p. 223). Thus, evaluation emerged as a field of professional practice with high expectations for the contributions it would make. Edward Suchman (1967, p.1) began his seminal text on evaluation research with the observation that one of the most appealing ideas of our century is the notion that science can be put to work to provide solutions to social problems. Carol Weiss (1977) has captured the tenor of those times: There was much hoopla about the rationality that social science would bring (p. 4). It would provide hard data for decision making, identify which variables to alter to attain desired outcomes, assess relative costs and benefits so that decision makers could select the options with the highest payoff, and make objective judgments about program effectiveness to improve performance. Early visions for evaluation, then, focused on the expected role of evaluation in guiding funding decisions and differentiating the wheat from the chaff in federal programs (summative evaluation). But as evaluations were implemented, a new role emerged: helping to improve programs as they were implemented (formative evaluation). The Great Society programs floundered on a host of problems: management weaknesses, cultural issues, and failure to take into account the enormously complex systems that contributed to poverty. Wanting to help is not the same as knowing how to help; likewise, having the money to help is not the same as knowing how to spend money in a helpful way. Many War on Poverty programs turned out to be patronizing, controlling, dependency generating, insulting, inadequate, misguided, overpromised, wasteful, and mismanaged. Evaluators were called on not only to offer final judgments about the overall effectiveness of programs, but also to gather process data and provide feedback to help solve programming problems along the way. These different evaluation functions, formative and summative (both of which are important), turned out to involve different methods and communications skills. The challenge to both organizations and evaluators was to figure out what kind of evaluation was appropriate for a particular situation. Doing this would require skills in situation analysis, systems thinking, and group facilitation, which were original elements of evaluator training. Vision Unfulfilled Despite the early optimism about and hopes for an evaluation-based experimenting society, by the end of the 1960s it was becoming clear that evaluations of Great Society social programs were largely ignored or politicized. The utopian hopes for a scientific and rational society had somehow failed to be realized. While all types of applied social science suffered from underuse (Weiss, 1977), nonuse seemed to have been particularly characteristic of evaluation studies. Williams and Evans (1969, p. 453) wrote, in the final analysis, the test of the effectiveness of outcome data is its impact on implemented policy. By this standard, there is a dearth of successful evaluation studies. Wholey, Scanlon, Duffy, Fukumotu, & Vogt (1970, p. 46) concluded, the recent literature is unanimous in announcing the general failure of evaluation to affect decision-making in a significant way. They went on to note that their own study found the same absence of successful evaluations observed by other authors (Wholey et al., 1970, p. 48). In an article that has become a classic in the evaluation literature, Weiss (1972) identified underutilization as one of the foremost problems in evaluation research. An important related issue was misuse of findings. Ernest House (1990), one of the most astute observers of how the evaluation profession has developed, commented: Results from poorly conceived studies have frequently been given wide publicity, and findings from good studies have been improperly used (p 26). With the increased use of findings over the years, concern about increased misuse has grown within the evaluation profession (Patton, 1997, pp ). Thus, the dual challenges of attaining and enhancing appropriate uses while working to eliminate improper uses took center stage as program evaluation matured as a distinct field of professional practice. Increasing use poses particular challenges both for training evaluators who are utilization-focused and for increasing the capacity of institutions to use evaluations appropriately and well. The design and implementation of the CEFP addresses both of these historical challenges, and addresses them interdependently. It also addresses the more recent challenge of helping the American Cancer Society respond to increased demands for public accountability and enhanced effectiveness as a learning organization. Increasing Demands for Accountability In the 1990s, growing concerns about US federal budget deficits and runaway entitlement costs intensified the debate about the effectiveness of programs. Both conservatives and liberals were faced with public demands to know what had been achieved by all the programs that had been created and all the money that had been spent. The call for greater accountability became a watershed flowing at every level national, state, and local; public sector; nonprofit agencies; and the private sector. Every major organization attempting to serve society has been affected by the increased demands for accountability as well as the corollary pressure for increased effectiveness in achieving outcomes and results (as opposed to just delivering services). Clear responses to these pressures were slow in coming. Few programs could provide data on the results achieved and outcomes attained. Internal accountability had come to center on how funds were spent (ie, inputs monitoring), eligibility requirements (ie, who gets services and client characteristics), how many persons get services, what activities they participate in, and how many complete the program. These indicators of inputs, client characteris-

3 CEFP as Model / Patton S13 tics, activities, and outputs (program completion) were important but stopped short of documenting the outcomes that were attained. Few programs were results focused. Programs were rewarded for providing services rather than making a difference in the participants lives. Public skepticism turned to deep-seated cynicism. Polling data showed a widespread perception that nothing works. More damning still, in modern times, the perception has grown that no relationship exists between the amount of money spent on a problem and the results accomplished. In recent years, with the movement to downsize government, evaluation also has grown in importance in the private, nonprofit sector (Independent Sector, 1993). Corporations, philanthropic foundations, and nonprofit agencies are increasingly turning to evaluators for help in enhancing their organizational effectiveness. As a result, attention has moved from just doing occasional isolated and episodic evaluations to the challenge of institutionalizing evaluation in organizations. The CEFP was designed in this context with the explicit purpose of infusing evaluation into the culture of the American Cancer Society. Standards of Excellence for Evaluation Another important element of the historical context that illuminates what the CEFP offers as a model has been the professionalization of evaluation, including the articulation of standards for evaluation that have helped make evaluation distinct from basic and applied research. The standards make it clear that evaluations ought to be, first and foremost, useful. In the early years, many evaluation researchers took the position that their responsibility was merely to design studies, collect data, and publish findings; what decision makers did with those findings was not their problem. This stance removed from the evaluator any responsibility for fostering use and placed all the blame for nonuse or underutilization on decision makers. Moreover, before the field of evaluation identified and adopted its own standards, the criteria for judging evaluations could scarcely be differentiated from the criteria for judging research in the traditional social and behavioral sciences, namely, technical quality and methodological rigor. Utility was largely ignored. Methods decisions dominated the evaluation design process. Validity, reliability, measurability, and generalizability were the dimensions that received the greatest attention in judging evaluation research proposals and reports (Bernstein & Freeman, 1975). Indeed, evaluators concerned about increasing the usefulness of a study often called for ever more methodologically rigorous evaluations to increase the validity of findings, thereby supposedly compelling decision makers to take findings seriously. By the late 1970s, however, it was becoming clear that greater methodological rigor was not solving the use problem. Program staff and funders were becoming openly skeptical about spending scarce funds on evaluations that they could not understand, found irrelevant, or both. Evaluators were being asked to be accountable just as program staff were supposed to be accountable. The questions emerged with uncomfortable directness. Who will evaluate the evaluators? How will evaluation be evaluated? It was in this context that professional evaluators began discussing standards. The most comprehensive effort at developing standards was hammered out over 5 years by a 17-member committee, appointed by 12 professional organizations, with input from hundreds of practicing evaluation professionals. The standards, published by the Joint Committee on Standards for Educational Evaluation in 1981, dramatically reflected the ways in which the practice of evaluation had matured. Just before publication, Dan Stufflebeam, Chair of the Committee, summarized the committee s work as follows: The standards that will be published essentially call for evaluations that have four features. These are utility, feasibility, propriety and accuracy. And I think it is interesting that the Joint Committee decided on that particular order. Their rationale is that an evaluation should not be done at all if there is no prospect for its being useful to some audience. Second, it should not be done if it is not feasible to conduct it in political terms, or practicality terms, or cost effectiveness terms. Third, they do not think it should be done if we cannot demonstrate that it will be conducted fairly and ethically. Finally, if we can demonstrate that an evaluation will have utility, will be feasible and will be proper in its conduct, then they said we could turn to the difficult matters of the technical adequacy of the evaluation (Stufflebeam, 1980, p 90; emphasis in the original). In 1994, revised standards were published after an extensive review spanning several years (Joint Committee on Standards for Educational Evaluation, 1994). While some changes were made in the 30 individual standards, the overarching framework of four primary criteria remained unchanged: utility, feasibility, propriety, and accuracy. Taking the standards seriously has meant looking at the world quite differently. Unlike the traditionally aloof stance of basic researchers, evaluators are challenged to take responsibility for use. The implementation of a utility-focused, feasibilityconscious, propriety-oriented, and accuracy-based evaluation requires situational responsiveness, methodological flexibility, multiple evaluator roles, political sophistication, and substantial doses of creativity, all elements of utilization-focused evaluation and is the focus of the CEFP model, both in training and American Cancer Society organizational development. Lessons from Utilization-Focused Evaluation Utilization-focused evaluation (Patton, 1997) emphasizes working with the primary intended users of an evaluation to design an evaluation that meets their practical information needs and is relevant to the challenges they face. The focus in utilization-focused evaluation is on intended

4 S14 CANCER PRACTICE March/April 2001, Vol. 9, Suppl. 1 use by intended users. For evaluators to facilitate a utilization-focused process, they need not only methodological capabilities but also good communication and group facilitation skills, sometimes called people skills. Indeed, recent research on needed evaluator competencies shows the challenges of training evaluators to meet the standards of evaluation and to be effective in facilitating use. The reflections of experienced evaluation practitioners suggest that evaluators need the kind of broad range of competencies that are shown in Table 1. Developing this range of competencies is fundamental to acquiring the logic and values of evaluation; that is, learning how to think evaluatively. Table 2 presents examples of the logic and values that undergird evaluative thinking. This kind of thinking is far from natural; indeed, it is alien to many, but it must be learned. The CEFP provides a process for graduate students to acquire core evaluation competencies while, at the same time, helping to infuse evaluation logic, values, and thinking into American Cancer Society planning, program development, and ongoing evaluation. Learning by Doing Knowledge can be acquired in classroom settings. Learning research methods, evaluation theory, diverse models, evaluation standards, analytical techniques, and formal writing has long been included in the curriculum of university evaluation courses. People skills, however, are not so easily taught in the classroom. Learning to work with diverse stakeholders program directors, funders, program staff, participants, and volunteers requires real-world experience with real people. Learning communication and group process skills happens best through engagement that offers practice in negotiating among diverse interests, resolving conflicts, facilitating group decision making, and working collaboratively. Learning how to work with nonresearchers to understand and appreciate the logic and values of evaluation requires practice and experience. Nor can such skills be acquired by simply going out into the real world and doing evaluations. Direction and supervision are needed to ensure host agencies and programs that students can and will conduct themselves professionally and competently. Students need assistance in gaining access to key decision makers and insight into organizational politics and culture. They also have to learn to communicate effectively and to write reports that contribute to use and learning (Torres, Preskill, & Piontek, 1996), which are not skills traditionally learned in academic research courses. In essence, the training of competent evaluators requires opportunities to integrate theory with practice and to turn classroom knowledge into practice wisdom through concrete experience followed by feedback and reflection. The CEFP does just that for students with the support of their academic advisors and American Cancer Society evaluation facilitators. Moreover, both the academic advisors and American Cancer Society evaluation facilitators have the opportunity to deepen their own evaluation understandings through involvement with the students evaluation projects. Table 1. Evaluator Competencies 1. Systematic inquiry Able to conduct research Frame research questions Design research Review literature Sampling Instrument construction Data collection and management Analyze and interpret data Report results in a balanced fashion Computer application skills Knowledgeable about evaluation Know evaluation theory, models, and underlying philosophical assumptions Needs assessment tools Evaluation design options and implications Making judgments Developing recommendations Writing useful reports Meta-evaluation 2. Evaluation practice skills Able to identify and serve the information needs of intended users Able to do situational analysis Knowledgeable about organizational development, change, and politics Able to analyze the political context of an organization Respectful of uniqueness of evaluation site and client Open to others input Able to adapt and change a study as needed Able to organize and manage evaluation projects Respond to requests Able to negotiate and write formal agreements Able to budget an evaluation Able to access needed resources Able to train support people Able to conduct the evaluation in a nondisruptive manner Able to complete work in a timely manner Able to deal with stress during a project 3. Interpersonal skills to facilitate evaluation processes Logical and critical thinking skills Written communication skills Verbal communication skills Listening skills Translating between research and practice Interpersonal competence Negotiation skills Conflict resolution skills Group facilitation skills Collaboration skills Cross-cultural interaction skills 4. Evaluation professionalism Knowledge of professional evaluation standards and guiding principles Ethical conduct Ongoing professional development Source: Ghere, Minnema, Stevahn, & King, (1998).

5 CEFP as Model / Patton S15 Table 2. Examples of the Logic and Values of Evaluation Be clear Be clear about goals and purposes; be clear about what is being evaluated, what data will be collected, what judgments are to be made, how results will be used indeed, be as clear as possible about everything. Be specific A favorite evaluation clarifying question: What exactly do you mean by that? Focus and prioritize You cannot do or look at everything. Be intentional and purposeful in deciding what is worth doing and knowing. Be systematic Plan your work; work your plan. Carefully document what occurs at every stage of decision making and data collection. Make assumptions explicit Determine what can and cannot be subjected to empirical test. Operationalize program concepts, ideas and goals The fundamental evaluation challenge is determining how to measure and observe, quantitatively or qualitatively, what is important. Reality testing becomes real at this point. Distinguish inputs and processes from outcomes Confusing processes with outcomes is common. Have data to support allegations of fact; provide empirical support and logical explanations for conclusions This means a commitment to reality testing in which logic and evidence are valued over strength of belief and intensity of emotions. Separate data-based statements of fact from interpretations and judgments Interpretations go beyond the data and must be understood as what they are: interpretations. Judgments involve values about what is desirable or undesirable. Make criteria and standards for judgments explicit The logical mandates to be clear and specific apply to making criteria and standards explicit. Limit generalizations and causal explanations to what the data support Overgeneralizations and overly definitive attributions of causality are epidemic outside the culture of research and evaluation. Source: Patton (1997) Organizational Capacity and Institutionalizing Evaluation Just as students need experience and practice to learn to do evaluations, organizations need experience and practice to become adept at using evaluations for program improvement and organizational learning. The field of evaluation is paying more and more attention to the ways of building capacity for evaluation into programs and organizations (Patton, 1994). Openness to evaluation increases as organizations have positive experiences with evaluation and learn to reflect on and take lessons from those experiences (Schön, 1983, 1987). A common problem in introducing evaluation into organizations has been doing too much (ie, large scale efforts and universal mandates) before the capacity was sufficient to support useful evaluation. That capacity includes developing administrative and staff understanding of the logic and values of evaluation (Table 2), developing organization-specific processes for integrating evaluation into planning and program development, and connecting evaluation to the latest understandings about organizational learning (Preskill & Torres, 1999). Research on readiness for evaluation (Patton, 1997; Preskill & Torres, 2000; Seiden, 2000) has found that valuing evaluation and learning are necessary conditions for evaluation use. Valuing evaluation cannot be taken for granted, nor does it happen naturally. The commitment of users to evaluation is typically fragile, often whimsical, and must be cultivated like a hybrid plant that has the potential for enormous yields, but only if properly cared for, nourished, and appropriately managed. Nurturing evaluation often means overcoming barriers to institutionalizing evaluation in organizations. Common barriers include: inadequate or underdeveloped learning-oriented culture; failure to integrate evaluation with other major organizational processes (eg, planning and budgeting); thinking that data and information are the same as knowledge; paying lip service to evaluation without real leadership support; defining evaluation as a paperwork and external reporting process rather than as an organizational development and learning opportunity; having inadequate resources to support meaningful evaluation; fearing that evaluation will reveal flaws to the larger world; inadequately using situational analysis to design an appropriate evaluation; failing to clearly and explicitly specify the intended uses of an evaluation and designing studies that are relevant to those intended uses; and conducting evaluations without the meaningful involvement of the primary intended users. Recognition of these barriers has focused increased attention on the need to help programs and organizations get

6 S16 CANCER PRACTICE March/April 2001, Vol. 9, Suppl. 1 ready for evaluation while institutionalizing evaluation as an ongoing part of the culture and primary structures of that organization. As a result, evaluators need to display heightened evaluator sensitivity to client needs and interests while facilitating the increased use of evaluation in the organizational decision making. The CEFP epitomizes this approach by emphasizing the performance of useful evaluations, institutionalizing evaluation into the American Cancer Society organizational culture, increasing the capacity of the American Cancer Society to use evaluation, and increasing the capacity of collaborating universities to train evaluators who can facilitate utilization-focused processes in support of organizational learning and decision making. Looking forward, the CEFP appears to be anticipating the challenges that lie ahead for organizations. At least four significant developments will affect the future connection between evaluation and organizational effectiveness: 1) the increasing complexity of managing modern organizations; 2) the availability of sophisticated information-processing technology to gather and process voluminous data; 3) the ever-increasing demands for improved performance and service delivery by organizations; and 4) the need to be responsive to an increasingly diverse world. The combination of these factors creates a need for increased analytical and evaluative capacity in modern organizations (Sonnichsen, 2000, p. 2). Modeling Institutionalization of Evaluation The CEFP, as it has evolved and adapted over time, has modeled effective evaluation practice by intensively evaluating its own effectiveness, both internally and with external reviewers, and by using the findings to incorporate the lessons learned and to adjust to emergent challenges. In that sense, the CEFP has modeled how to institutionalize evaluation into a specific program even as it works to institutionalize evaluation into the larger American Cancer Society culture. References Bernstein, I., & Freeman, H. (1975). Academic and entrepreneurial research: Consequences of diversity in federal evaluation studies. New York: Russell Sage. Campbell, D.T. (1991). Methods for the experimenting society. Evaluation Practice, 12(3), (Reprinted from a 1971 presentation to the American Psychological Association) Ghere, G., Minnema, J., Stevahn, L., & King, J.A. (1998). Evaluator competencies. Paper presentated at the meeting of the American Evaluation Association, Chicago, IL. House, E.R. (1990). Trends in evaluation. Educational Researcher, 19, Independent Sector. (1993). A vision of evaluation. Washington, DC: Author. Joint Committee on Standards for Educational Evaluation. (1994). The standards for program evaluation. Thousand Oaks, CA: Sage Publications. Patton, M.Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), Patton, M.Q. (1997). Utilization-focused evaluation: The new century text (3rd ed). Thousand Oaks, CA: Sage Publications. Preskill, H., & Torres, R.T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage Publications. Preskill, H., & Torres, R.T. (2000). The readiness for organizational learning and evaluation instrument. Unpublished manuscript. (Available from H. Preskill: hpreskill@umn.edu) Schön, D.A. (1983). The reflective practitioner. New York: Basic Books. Schön, D.A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass. Seiden, K. (2000). Development and validation of the organizational readiness for evaluation survey instrument. Unpublished doctoral dissertation, University of Minnesota. Sonnichsen, R. (2000). High impact internal evaluation. Thousand Oaks, CA: Sage Publications. Stufflebeam, D.L. (1980). An interview with Daniel L. Stufflebeam. Educational Evaluation and Policy Analysis, 2(4), Suchman, E.A. (1967). Evaluative research: Principles and practice in public service and social action programs. New York: Russell Sage. Torres, R.T., Preskill, H., & Piontek, M. (1996). Evaluation strategies for communicating and reporting: enhancing learning in organizations. Thousand Oaks, CA: Sage Publications. Weiss, C.H. (Ed.) (1972). Evaluating educational and social action programs: A treeful of owls. In C.H. Weiss (Ed.), Evaluating action programs (pp. 3 27). Boston: Allyn & Bacon. Weiss, C.H. (1977). Introduction. In C.H. Weiss (Ed.), Using social research in public policy making (pp. 1 22). Lexington, MA: DC Heath. Wholey, J., Scanlon, J., Duffy, H., Fukumotu, J., & Vogt, L. (1970). Federal evaluation policy: Analyzing the effects of public programs. Washington, DC: Urban Institute. Williams, W., & Evans, J. (1969). The politics of evaluation: The case of Head Start. Annals of the American Academy Of Political And Social Science, 385,