ADVANCED MONITORING & EVALUATION: EVALUATION RESEARCH AND ANALYSIS 3 day training course
Introduction This 6-credit HEQF (2007) level 9 short course covers knowledge of a formal postgraduate programme, approved and quality controlled as required by the South African Qualification Authority (SAQA) and the Higher Education Qualification Committee (HEQC). Background Evaluation research and analysis require evaluators to select an appropriate design and responding methods for data collection and analysis. Evaluation is often defines as an applied social research, as it entails the application of social research methods to solve real-life developmental problems. As such, choosing an appropriate evaluation design for a study requires choices between quantitative, qualitative or mixed-methods approaches, the degree of evaluator independence or stakeholder participation and input in the study, and the intended use of the evaluation results (political, administrative, or public accountability). In designing studies, evaluators need to uphold professional standards, as expressed in the American Evaluation Association s Principles for Evaluators: Evaluators conduct systematic, data-based inquiries about whatever is being evaluated. Evaluators provide competent performance to stakeholders. Evaluators ensure the honesty and integrity of the entire evaluation process. Evaluators respect the security, dignity, and self-worth of the respondents, program participants, clients, and other stakeholders with whom they interact. Evaluators articulate and take into account the diversity of interests and values that may be related to the general and public welfare. Once the approach to the evaluation has been selected, evaluators need to make decisions on the methods for data collection and subsequent analysis of the findings. This process tries to balance the quality of the data and information gathered in the study, with the time, budget and political constraints to which evaluators are subjected. Target participants Public service departments, entities, municipalities and non-governmental managers dealing with the monitoring and evaluation of public sector policies, programmes and projects. 2
Minimum requirements Attendance of the Public Sector Monitoring and Evaluation Course or another appropriate M&E programme or Social Research course AND / OR relevant public sector M&E experience is a prerequisite for this course. Course objective, topics and outcomes Objective: To create capacity in the public sector to conduct evaluation research and systematically present findings and recommendations in a well-motivated report. Outcomes: After attending this course, participants should be able to: Critically discuss various evaluation designs or approaches to conduct evaluation studies. Select and motive an appropriate design for an evaluation study. Critically compare various quantitative and qualitative evaluation methods. Select and motivate appropriate methods for an evaluation study that will provide information that addresses the evaluation questions posed. Describe the requirements for commissioning evaluation studies. Compile an evaluation research report. Assessment and credits: The course is accredited with the 2007 HEQF on NQF level 9 and counts for six (6) credits. To obtain the certificate of competence, participants will be required to meet the following requirement and submit to the following assessments: Attend a sufficient part of the course contact session In-course group assessment(s) as determined by the course facilitator to test basic understanding (30% of the course mark); and A written post-course assignment to be submitted 30 days after the conclusion of the course (70% of the course mark) 3
Please note that the policy on submission of assignments for all Executive Development Programmes specifically states that assignments must be submitted four weeks after the course start date or on and alternative date agreed on by the facilitator and the participants. Assignments must be submitted by e-mail to Ms Tharia Uys: tl@sun.ac.za and should be accompanied by the completed assignment cover sheet included on the next page. Please also adhere to the instructions for naming the file containing your assignment. Please do not send assignments to course facilitators. submission related questions. You can contact Tharia at 021-918 4121 if you have Late submission of assignments will be accepted, subject to the following directive: In the light of the accreditation requirements, as well as the Short Course Policy of the School of Public Leadership (SPL) of Stellenbosch University (SU), a late submission of an assignment will be accepted within two weeks after the above submission date, subject to a 5% penalty per week. This penalty will not be applied if the comprehensive written request for the extension is accompanied by a letter from the manager of the participant or a medical certificate, and is accepted after consideration by the Project Manager. Thereafter, no further submission or assessment will be permitted. Assignment marks and final results for the course will be available as from 12 weeks after the commencement of the course. No marks, final results or certificates will be released until invoices have been settled in full. Please note that late settlement of invoices will delay the release of assignment marks, final results and the issuing of certificates. Certificates are printed by the Certificate Division of SU and are prepared and printed four weeks after all assignment marks and final results have been received from the SPL. Certificates will therefore, at the earliest, be dispatched 16 weeks after the course by SU. Certificates are sent by registered mail to the address noted on the registration form or to the coordinator who arranged the course with the SPL. Any certificates returned to the SU or the SPL due to non-collection will be retained and will only be sent or may be collected after suitable arrangements and payment have been made for collection from SU/SPL. Lost certificates: Please ensure the proper 4
safekeeping of certificates. Kindly note that, in order to prevent fraud, SU cannot reissue lost certificates. 5
Daily Programme of Activities: DAY 1 CHOOSING AN EVALUATION DESIGN 8:30 9:30 INTRODUCTION AND CONTEXTUALISATION Evaluation or Research? 9:30 10:30 FACTORS THAT INFLUENCE THE EVALUATION DESIGN Scope of the evaluation Evaluation theories and paradigms Data collection methods 10:30 11:00 MORNING TEA/COFFEE 11:00 13:00 SELECTED EVALUATION THEORIES, PARADIGMS AND MODELS Integrated evaluation vs input, process, output, outcome or impact evaluation Clarification evaluation, Realist evaluation Illuminative evaluation, Goal-free evaluation Responsive evaluation, Utilisation-focused evaluation Appreciative inquiry, Evaluative inquiry, Critical theory evaluation Empowerment evaluation, Democratic evaluation Experimental, Quasi-experimental, Qualitative and Mixed method designs 13:00 13:45 LUNCH 13:45 14:45 CHOOSING THE APPROPRIATE DESIGN 14:45 16:15 GROUP ACTIVITY AND FEEDBACK 6
Discuss and identify a specific evaluation study that you need to undertake. Given the discussion of various available theories, paradigms and models in the evaluation field, propose an appropriate philosophical and practical approach for your study. Motivate and explain your decision in the plenary session. 16:15 16:30 CONCLUSION DAY 1 DAY 2 SELECTED RESEARCH METHODS 8:30 10:30 QUANTITATIVE RESEARCH METHODS Structured or semi-structured Questionnaire Self-administered / observations checklists Secondary data analysis Standardised tests Stakeholder analysis Cost benefit & cost effectiveness analysis 10:30 11:00 MORNING TEA/COFFEE 11:00 13:00 QUANTITATIVE RESEARCH METHODS (cont) 13:00 13:45 LUNCH 13:45 15:15 QUALITATIVE RESEARCH METHODS Basic individual Interview/key informant interview Depth individual interview Focus group interview Community group interview 7
Observations and Participants observations Participatory rural appraisal/rapid rural appraisal methods Delphi technique Brainstorming Beneficiary assessment 15:15 16:15 GROUP ACTIVITY 16:15 16:30 CONCLUSION DAY 2 DAY 3 ANALYSING AND REPORTING EVALUATION FINDINGS 8:30 10:30 DATA ANASIS METHODS QUANTITATIVE DATA ANALYSIS: Elementary analysis Univariate analysis Bivariate analysis Multivariate analysis Social statistics Descriptive statistics Inferential statistics QUALITATIVE DATA ANALYSIS: Content analysis Discourse analysis Grounded theory 10:30 11:00 MORNING TEA/COFFEE 8
11:00 13:00 COMPUTER AIDED DATA ANALYSIS, SOME CONSIDERATIONS 13:00 13:45 LUNCH 13:45 15:15 STRUCTURE OF THE EVALUATION REPORT 15:15 16:15 ETHICAL CONSIDERATION IN CONDUCTING EVALUATIONS 16:15 16:30 CONCLUSION DAY 3 9
10
11