Thematic strand 3: Evaluation quality and values in evaluation the role of evaluation standards and professional associations

Transcription

1 Thematic strand 3: Evaluation quality and values in evaluation the role of evaluation standards and professional associations By Matthew Lubuulwa, Ministry of Finance, Planning and Economic Development Presentation 4 (PS4): Policies, standards and guidelines. Three presentations were based on Uganda Evaluation Association Standards, the USAID evaluation guidelines and Government of Uganda Evaluation Guidelines. Panel Discussion 2 (PD2): The role of evaluation societies in the professionalisation of evaluations. The panel discussion had four discussants namely: Margaret Kakande from Uganda, Antonio Hercules from South Africa, Derek Poate from England and Thania De la Garza from Mexico. Panel Discussion 3 (PD3): Safeguarding the independence of evaluations. Three panelists that is Rosetti Nabbumba from Uganda Evaluation Association, John Ogwanga an Independent Evaluator, and Derek Poate from United Kingdom Evaluation Society (UKES). The presentation outlined the following: The Uganda Evaluation Standards The Uganda Evaluation Association (UEA) launched the evaluation standards in November 2013 with the aim of building evaluation capacity; provide a platform for networking and sharing methods (awareness); professionalism; increase evaluation quality and use. The standards are aligned to the African Evaluation Standards and run under five themes namely; utility, feasibility, quality and precision, ethical conduct and capacity development. The association is engaging her stakeholders to disseminate and use the standards. The standards provide for a periodic review for assessment of use and improvement. They can be downloaded from USAID Evaluation Guidelines The United States Agency for International Development (USAID) is a lead federal government agency working to end extreme global poverty and enable resilient, democratic societies to realize their potential. USAID had operated in Uganda for 51 years. In 2011, there was a policy change to re-affirm the purpose of evaluation to learning and accountability to ensure credible, unbiased and transparent results. The focus of the guidelines is on utility, feasibility, propriety (ethics), accuracy and accountability. The guidelines were developed, capacity is being built through training, sharing resources using learning groups, local and international partnerships and embedding evaluation in the organization s culture. The guidelines are available at the learning labs website Government of Uganda Evaluation Guidelines

2 In order to operationalise the National M&E policy and Uganda Evaluation Standards in government, the Office of the Prime Minister (OPM) developed government of Uganda Evaluation Guidelines. The draft guidelines focus on appropriateness of policies/programs, coherency of government response, effectiveness and impact (value for money) of interventions. The intention of the guidelines is to align evaluation to government systems instead of commissioners and funders, consistence and easy management, quality control, provide a sense of purpose and promote informed decision making. Once approved, the guidelines shall be uploaded on the OPM website. The presenters and panellists addressed the following key questions: Questions: How can evaluation standards and guidelines help improve the quality of evaluations? Standards and guidelines give a sense of purpose through the planning, design, team selection, implementation, report writing and dissemination of evaluation results. Standards and guidelines demand that evaluations are feasible, accurate, usable and ethical. Standards and guidelines help evaluators to follow approaches that lead to credible results. By following standards and guidelines evaluations quality increases arising out of a consistent flow of events, reporting and feedback. Standards enable associations, commissioners of evaluations and financiers to identify the right professionals to undertake specific evaluation assignments. Ethics bring what unites the evaluators instead of professional background and with ethically moderated assignments, quality improves. Who should own evaluation standards and guidelines government and/or professional associations? Ownership of standards is a shared responsibility between associations, government, civil societies and the private sector. However, professional associations should take the lead because they have the mandate from practitioners to safe guard the profession and have the role of ensuring professionalism among the members. The professional association should partner with stakeholders to ensure use of standards. For impact to be felt, the Uganda Evaluation Association (UEA) should lobby government to enact a law establishing and recognising the profession of evaluation. The law should mandate the association or a board to accredit and certify members as well as enforce the evaluation standards. The legal framework should detail the ultimate users of evaluation reports like the case is with audit reports. It was observed that the UEA was concentrating on sensitising evaluators and promoting the standards through strategic partnerships with stakeholders and not focusing on who owns what? With time the standards shall be reviewed to provide for accreditation after developing the competencies and building the capacity of evaluators in the country.

3 What are the best ways to strengthen professional values in evaluation? In order to strengthen professional values, both evaluators and commissioners of evaluations must follow a code of ethics which detail values that help in managing expectations of evaluation users. Development and application of standards and guidelines help in strengthening professional values. Association should ensure that the standards are used through several approaches for example, through mentoring of younger evaluators, peer reviewing of evaluation results, attachment of professionals to ongoing evaluations, skilling evaluators through continuous professional development programs, networking, and regular participation in national and international evaluation events to benchmark best practices. It was noted that in Mexico, the focus is on certifying evaluations done instead of evaluators. In the United Kingdom, the Evaluation association was looking for means of designating a process of validating the evaluation professionals. The intention is to ensure accountability of evaluators. How can the culture of evaluations enhance the quality of evaluations? Standards will help to point out the competencies of evaluators and if applied as part of evaluation culture, commissioners of evaluations will be able to identify good (competent) from bad (incompetent) evaluators. Standards enable evaluators to express themselves independently, objectively and avoid conflict of interest in performing their roles. Discussions were reportedly ongoing in Mexico on certification and accreditation of evaluators. The general thinking is, after certifying an evaluator; there are no incentives to deliver quality work. The country was developing a culture of certifying evaluations conducted; assess the consistency of evaluations by a certain evaluator as a basis for future engagements. In South Africa, most of the public evaluations are tendered to private companies and universities using an outsourcing model. Majority of the pre-qualified service providers are experienced persons above 50 years of age and in most cases White females. Attempts to address the inequalities are made by encouraging young and middle aged persons including blacks to participate in evaluations. Like the case is with other professions, where upcoming individuals go through mentoring, peer reviews and attachments to experienced practitioners. Evaluator s performance and quality will greatly improve if the same culture is adopted. The application of appropriate methodologies, provision of an enabling environment, and empowerment of users to distinguish between good and bad evaluation enhances quality. The Uganda National Planning Authority requires that international consultants should work with local evaluators as part of knowledge transfer and avoidance of biases and unrealistic conclusions and recommendations. How can evaluation societies contribute to the professionalization of evaluation?

4 By ensuring that individuals participating in evaluations enrol as members, subscribe to, and use evaluation standards. Evaluation was noted to be multidisciplinary and is conducted in every field of life; organising training clinics/camps for evaluators enables professionalization of evaluation. Evaluation societies should organise internship and attachment programs for their members. Societies can contribute to professionalism through building synergies with all stakeholders to promote development, implementation and review of evaluation standards and guidelines. Societies can work with curriculum experts at universities and other tertiary institutions to develop post graduate programs for evaluation at masters and PHD level. It was noted that education alone is inadequate and experience and exposure was key in professionalizing evaluations. It is equally important for commissioners of evaluation to know what they do and what they want. Societies should skill their members on communicating results using effective and appropriate language to ensure use of evidence based results in decision making. Packaging of reports with balanced and objective language is critical in making the reports usable. In Uganda, the evaluation standards were commissioned in November 2013 (relatively new); the association is at the stage of disseminating them to users through websites, social media, professional events and engaging commissioners of evaluation to use them. Government of Uganda through the Office of the Prime Minister demands knowledge of national standards and guidelines as part of the terms of reference for undertaking government evaluations. In South Africa, the evaluation association is engaging the national planning body to plan for evaluation of government development plans at design stage; political parties provide key literature for use during evaluation of the plans. Independence of evaluation: why independence and what undermines it? How to foster independence in evaluations? Independence enables evaluators to avoid conflict of interest, remain objective (given the external eye) and reduce the degree of bias. Independence is expressed from the selection of evaluation topics, development of terms of references, evaluation designs, selection of methodologies, analysing findings and reporting of results. All evaluations should be independent to ensure credibility of results and usability of findings; however a degree of dependence is sometimes acceptable provided it does not materially distort the results. For example, an external evaluator may work with internal staff as part of learning and knowledge transfer. Over emphasizing independence can sometimes endanger the work of evaluators as users may turn against the findings or refuse to use evaluation results. Independence may not necessarily address the biases but control the degree. Self-evaluation is sometimes inevitable especially in organisations with large volumes of work and inadequate resources. Evaluation standards and guidelines play a key role in safeguarding the independence of evaluators.

5 In the public sector, independence is affected by financiers who in most cases wish to have positive results, and control the results by developing skewed terms of references with Do s and Don ts, inadequate scope, lack of guidelines and a code of ethics and withholding information by key respondents. Civil societies on the other hand depend on external support where results determine the next financing step. Managers therefore attempt to secure good evaluation results even when things are not well. Independence should be demonstrated in balanced findings showing both the positives and negatives, clear evidence based and well-drawn conclusions and recommendations and validation of results through peer reviews. Total independence is hard to achieve, however safeguards for independence must be adhered to through contracting experienced evaluators, persuasion, legal protocols including standards, insistence on application of ethics, transparency, evaluator s competencies and skills. To foster independence, there is need for code of ethics, a degree of experience to avoid undue Case studies: Experiences when independence is compromised In a donor funded project, a former employee was contracted as a team leader to evaluate a project he participated in its design. The evaluation design was objective with several specialists on the team; however the funders were having reservations on the choice of team leader. The final report provided amazing results which attracted bashing from the commissioners (management) and applauded by the financiers. What made the report tick were transparency, objectivity and ethical conduct. In monitoring government of Uganda projects/programs/policies; managers attempt to restrict the scope, provide very little information and irrelevant key informants. The place of work has a code of ethics with clear guidelines on expectations. M&E is conducted in teams (team work) with multiple competencies. The Monitoring Unit is independently financed and uses triangulation to validate findings. In an Asian Bank, the commissioners of evaluation only provided information asked for and made setting up appointments with key informants difficult. In the European Union, the team leader was called aside and told that the team was spending a lot of time speaking to the wrong people. The supplied copies of the final report were never disseminated. It is important to listen to issues raised but evaluators must remain objective. influence from commissioners and or financiers of evaluations, selection of balanced evaluation teams, use of peer reviews, use of mentoring to discuss case load with experienced individuals and address problems faced by young evaluators through sharing experiences, educating and preparing commissioners of evaluations to receive and accept bad news as a way of identifying weaknesses and developing strategies for improvement. Conclusion Evaluation units should be semi-autonomous with guaranteed budgets to attain independence. Independence does not necessarily mean quality evaluations. The UEA should continue to engage with

6 all stakeholders on dissemination and use of national evaluation standards, create an avenue for sharing evaluation results, lobby for recognition of the profession by government and multi-lateral organisations, and continue to develop the capacity of evaluators. The developed capacity should be put to practice to foster quality and development of evaluation as a profession and culture.