Indiana Evaluation Association October 20, 2017

Transcription

1 Smile sheets are not enough: An effective, systematic approach to program and educational evaluation using CIPP Indiana Evaluation Association October 20, 2017

2 Introductions Name Affiliation (organization, role, etc.) What will make this a successful session for you? Key elements of evaluation

3 Program evaluation is most often defined as a process used to determine whether the design and delivery of a program were effective and whether the proposed outcomes were met. Caffarella, R.S., Daffron, S. R. (2013). Planning programs for adult learners: A practical guide. (3 rd Ed.). San Francisco, CA: Jossey-Bass. p. 233.

4 Evaluation is the systematic assessment of the operation and/or outcome of a program. Milanesi, L. (2016). In The Scholar-Practitioner s Guide to Research Design, G. Burkholder, K. Cox, & L. Crawford (Eds.). (p. 260). Baltimore, MD: Laureate Publishing.

5 Program evaluation is the systematic collection of information about the activities, characteristic, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.... Evaluation research, quite broadly, can include any effort to judge or enhance human effectiveness through systematic data-based inquiry. Patton, M. Q. (2002). Qualitative research and evaluation methods (3 rd ed.). Thousand Oaks, CA: Sage Publications. p. 10

6 I proposed that evaluation be redefined as a process of providing useful information for decision making. Stufflebeam, D. (1983). In Evaluation models: Viewpoints on educational and human services evaluation. G. F. Madaus, M. Scriven, D. L. Stufflebeam (Eds.). Hingham, MA: Kluwer Academic Publishers.

7 Evaluation the process of delineating, obtaining, and applying descriptive and judgmental information concerning the merit of an educational or training program s goals, plans, processes, and products in order to serve decision making and accountability relative to that program; using measurement data to judge the worth or merit of an object. Schoch, K. W. (1990). A comparison of management development evaluation methods in selected healthcare institutions with a recognized educational evaluation model. Doctoral dissertation. Indiana University, Bloomington, IN.

8 The concept of evaluation underlying the CIPP Model and this checklist is that evaluations should assess and report an entity s merit (i.e., its quality), worth (in meeting needs of targeted beneficiaries), probity (its integrity, honesty, and freedom from graft, fraud, and abuse), and significance (its importance beyond the entity s setting or time frame), and should also present lessons learned.... The model s main theme is that evaluation s most important purpose is not to prove, but to improve. Stufflebeam, D. L. (March, 2007). CIPP Model Evaluation Checklist, 2 nd Edition.

9 Key Characteristics Use of data, information: Quantitative Qualitative Focus on outcomes Focus on improvement Merit, Worth Assessment Accountability Systematic Drive decision-making What has been the driving force behind your evaluations? What do you do with the results?

10 Your program....

11 Context Intent: What needs to be done? Assess needs, assets, problems, and overall readiness within a defined environment. Considerations Outcome

12 Context Considerations Background information on intended beneficiaries needs Interview leaders and stakeholders perspectives on needs or problems to be solved Assess/determine program goals in light of assessed needs Identify program environment: related programs, resources, needs and problems, political dynamics Compare goals and priorities to needs Assess whether objectives are responsive to needs

13

14 Context Intent: What needs to be done? Assess needs, assets, problems, and overall readiness within a defined environment. Considerations Outcome Identification of beneficiaries, needs, goal/outcome, and potential issues or challenges

15 Your program... Which elements of C Context are, or could be, part of this program s planning and implementation?

16 Input Intent: How should it be done? Assess competing strategies and the work plans and budgets of the selected approach. Considerations Outcome

17 Input Considerations Existing programs that can serve as a model or template? Create and evaluate potentially relevant approaches How does the program s proposed strategy respond to assessed needs and feasibility? Assess strategy against research, literature; compare with strategies in similar programs Assess program budget for sufficiency Assess work plan and schedule for sufficiency, feasibility Formulate a responsive plan

18

19 Input Intent: How should it be done? Assess competing strategies and the work plans and budgets of the selected approach. Considerations Outcome A well-designed program or project to meet the identified need(s)

20 Your program... Which elements of I Input are, or could be, part of this program s planning and implementation?

21 Process Intent: Is it being done? Monitor, document, and assess program activities. Considerations Outcome

22 Process Considerations Monitor, observe, record (visually, audio) progress on program implementation Maintain ongoing record of program events, problems, costs, etc. Periodic interview or survey of beneficiaries, program leaders, staff for input on progress Ongoing up-to-date program profile Draft written reports on process evaluation findings for program leaders, other stakeholders; update meetings

23

24 Process Intent: Is it being done? Monitor, document, and assess program activities. Considerations Outcome Are planned activities being carried out? What revisions are needed?

25 Your program... Which elements of P Process are, or could be, part of this program s planning and implementation?

26 Product Intent: Did it succeed? Right beneficiaries? Needs met? Gains sustained? Transportable and adaptable? Considerations Outcome

27 Product Considerations Impact Correct beneficiaries? Affect or proper services for them? Affect on a non-targeted group? Effectiveness Stakeholders: positive and negative outcomes, affect on beneficiaries; case study approach Sustainability Interview beneficiaries; review program effectiveness; what can, should be sustained or eliminated Transportability Get feedback from potential adopters

28

29 Product Intent: Did it succeed? Right beneficiaries? Needs met? Gains sustained? Transportable and adaptable? Considerations Outcome Were beneficiaries needs met? Is the program sustainable and transportable? What improvements need to be made?

30 Your program... Which elements of P Product are, or could be, part of this program s planning and implementation? Does your Product evaluation consider: Impact? Sustainability? Effectiveness? Transportability?

31 CIPP Advantages? Challenges?

32 Other thoughts.... Cyclical process Contracting Meta-evaluation

33 Questions?

34 Evaluation Lessons learned what might you implement next week? Lingering questions What went well? Opportunities for improvement

35 Kurt Schoch, EdD Performance Improvement Consulting, LLC