Tips for Planning Your Evaluation

Transcription

1 Tips for Planning Your Evaluation Introduction Planning is the first phase of any evaluation. It is typically where you will get to know the program being evaluated, the clientele it serves, its goals and expected outcomes. This is also when you will find out what is needed from the evaluation and come up with a strategy to meet those needs. Planning usually beings with: having discussions with program stakeholders reviewing program materials considering supporting documents and literature A well thought out and realistic evaluation plan is the best way to ensure that your evaluation is as efficient, reliable and valuable as possible (Posavac & Carey, 1997). This document offers practical tips to consider when planning an evaluation. Communicate and Collaborate Planning a program evaluation is not a solitary exercise. It is important to consult with a variety of stakeholders, which may include: Administrators Managers funders service providers clients Speaking with stakeholders will help you gain a strong understanding of the program and expectations around the evaluation. Ideally, consultations will be ongoing and key stakeholders will be actively involved in planning the evaluation (e.g., from clarifying the objectives and purpose of the evaluation, providing input into data collection tools, and strategies for disseminating results). The benefit of this can be (Posavac & Carey, 1997): a strong commitment to the evaluation reduced potential for misunderstandings bringing multiple perspectives to the evaluation plan An opportunity for relationship building

2 Know Your Program A well-planned evaluation is rooted in a strong understanding of the program being evaluated. It is helpful to be familiar with the following: Program resources (both financial and staffing) The program s goals and objectives Activities or services provided Target clientele Anticipated outcomes (both short and long-term) This will help clarify the underlying rationale or logic behind the program (WHO, 2000). When a program does not have well documented goals and expected outcomes it is often useful to create a written description of the program that can be used for planning the evaluation. This is often called an evaluator s program description, which is: A statement prepared by the evaluator after working with stakeholders to amplify and clarify all aspects of the program, including goals, objectives, activities, and anticipated outcomes (Boulmetis & Dutwin, 2005, p. 58). Developing an evaluator s program description is useful for ensuring that all stakeholders have a similar understanding of the program and what it is expected to achieve. It can also provide clear and concise program goals and objects that may guide the development of the evaluation objectives and questions. If program descriptions are available they can be modified and updated for this purpose. It will also be helpful to know the current status of the program in terms of its development and implementation. For instance, is it still being planned or implemented or is it a long standing program? In this regard it might be useful to also learn how and why it was developed. Was it designed to meet a particular need? Or, who was involved in planning the program and what information was used? Some common strategies for getting to know your program include the following: Reviewing program documents, materials and administrative data Developing a logic model Consultations with stakeholders or key informant interviews Site visits Participating in program meetings Literature reviews and e-scans

3 Know What is Expected Having a clear understanding of the purpose of the evaluation, including how it will be used and by whom, is important for ensuring the results are useful and meet the needs of different stakeholder. It is often important to consider the needs of the following stakeholders: Funding bodies Administrators, managers Service providers Community members Other stakeholders Expectations around evaluations are often not clear or as concrete as we would like. Teasing out what stakeholders want from the evaluation can be an art. It requires sensitivity and recognition that many administrators, managers, service providers and other stakeholders do not have expertise in evaluation. As the evaluator, stakeholders will likely rely on you to provide advice on the scope and objectives of the evaluation. There are also often different, and sometimes, competing perspectives and expectations about the evaluation. It is important to consider all perspectives, but also to prioritize what is needed from the evaluation. It is unlikely that the evaluation will be able to meet every need or expectation. Managing expectations about what the evaluation can and cannot achieve is an important role for the evaluator. Keep it Realistic and Practical Evaluations can range from a simple survey to complex evaluation designs with multiple data sources and phases. Determining the best way to evaluate your program can be difficult. It is often tempting to design the ideal evaluation. However, evaluation resources are often scarce and staff commonly have to carry out evaluation activities in addition to their regular workload so it is important for the evaluation to be realistic and practical. Keep in mind the program resources as well as the resources available for the evaluation, including: Financial and material resources Human resources Time resources Ideally, scale of the evaluation should reflect the amount of resources invested in the program. At the same time, it is not useful to plan an elaborate evaluation if it cannot realistically be carried out with the resources available. Planning an evaluation will often involve finding a balance between what is the ideal way to evaluate the program and what is practical and realistic, both in terms of the resources available and the impact it will have on clients, service delivery, and staff.

4 Build on What is Available Considering what information is available and potential options for linking or building on existing projects is a useful starting place when deciding what information you will need for the evaluation. Many programs collect information on: clients they serve service(s) they deliver It is also common for programs to be involved in organizational performance measurement and quality improvement project or other activities that generate information about the program. These are all potential sources of data that can be used in an evaluation. Using or building on existing sources of information can: Minimize the amount of data that needs to be generated specifically for the evaluation Reduce the burden of data collection on service providers Limit the amount of data that is collected from clients Highlight avenues for integrating data collection into service delivery Making the most evaluation resources which are often limited Planning is Ongoing and Iterative Plans for evaluation often evolve as programs grow and change. This is particularly true in programs that are being implemented or are newly established. Changes in leadership or resources can also change the course of an evaluation. Because of this, planning an evaluation often requires: Flexibility Adaptability Time It is not uncommon to revise and update evaluation plans even after the evaluation is underway. As such, evaluation plans are typically considered living documents. This underscores the importance of ongoing communication and consultation. Any changes to the evaluation plan should be vetted through key stakeholders. Evaluation plans that appear sound on paper often have challenges or difficulties that were not anticipated when put into practice. Because of this, it is important to establish mechanisms for ongoing input from stakeholders, particularly those who are involved in implementing the evaluation (e.g., helping coordinate or carry out data collection).

5 References Boulmetis, J. & Dutwin, P. (2005). The ABCs of Evaluation (2nd Edition). San Francisco, CA: Jossey-Bass. Posavac, E.J. & Carey, R.G. (1997). Program Evaluation: Methods and Case Studies (5th Edition). Boston: Prentice Hall. World Health Organization (2000).Workbook 1: Planning Evaluations Evaluation of Psychoactive Substance Use Disorders Treatment Workbook Series. Retrieved from Prepared By Alberta Health Services Addiction and Mental Health Community Treatment and Support Knowledge and Strategy