DOCTORATE in BUSINESS ADMINISTRATION. Abdourahmane BA

Transcription

1 DOCTORATE in BUSINESS ADMINISTRATION Abdourahmane BA HOW TO MEASURE THE EFFECTIVENESS OF DEVELOPMENT PROGRAM MONITORING AND EVALUATION SYSTEMS? THE CASE OF WEST AFRICA Thesis Director: Thesis jury: M. Paul Beaulieu Professor, Department of Strategy, Social and Environmental Responsibility, ESG-UQAM Canada Thesis Director Mme Francoise Chevalier Professor, HEC, Paris, France. Suffragant M. Emmanuel Josserand Professor, University of Technology Sidney UTS. Rapporteur M. Yvon Pesqueux Professor, Conservatoire national des arts et métiers, CNAM, Paris, France. Rapporteur

2 It has to be made clear that BSI and IAE Lyon do not approve or disapprove the opinions expressed in this thesis. These opinions are to be considered solely as the author s. September 2017

3 ACKNOWLEDGEMENT This work would not be possible without the patience and support of my wife and children; I am so grateful for their perseverance in providing me with the needed attention and support to perform this thesis research. I would like to thank first Professor Paul Beaulieu, Professor, Department of Strategy, Social and Environmental Responsibility, ESG-UQAM Canada, member of the Scientific Council of the Business Science Institute, who accepted to supervise this work and who spent lot of time with me in the different steps of the research, and whose treasured support made this work possible. I would like to thank Professor Michel Kalika, President of the Scientific Council of Business Science Institute, Scientific Advisor of the Business Science Institute, who put particular attention very early to my research agenda and provided me with the needed start-up recommendations and connections to the research field. He educated me on how to complete a successful Doctorate in Business Administration (DBA) combining effectively professional work and meaningful research to create knowledge in my expertise and share it. This work would not be possible without the logistic and administrative support of the delightful Caroline Hertz, Business Science Institute administrative and pedagogical assistant. Thanks goes to her. Special thanks goes to Karen Kent, Principal Investigator (PI) of the USAID West Africa Analytical Support Services and Evaluations for Sustainable Systems (ASSESS) project, Program Manager in the Coastal Resource Center (CRC) of the University of Rhode Island (URI). I would like to thank ASSESS partners the United States Department of Agriculture (USDA), the United States Agency for International Development (USAID) Mission in West Africa, Kwame Nkrumh University of Science and Technology (KNUST), Delaware State University (DSU), and all the ASSESS Staff in Accra. Special thanks to ASSESS Staff in Accra. Thanks goes to Pr Philip LeBel, University of Monclair US, my Professor in International Development Management (MDI), Illinois State University MDI program. I would like to thank my former colleagues in FAO, IESA project, Gregorio VelascoGil, Bertrand Dambre, Yacouba Maiga, and Ismaila Badji, my colleagues in Office du Niger Segou, in ASER Senegal, in PAC- Niger, Zakou Aminata, in PAIMSC Cote d Ivoire, Dr Awa Kamara, and in Nigeria NPFS, Christophe Okonjo, and Okorie Agwu. Thanks goes to my former colleagues, and partners, Ababacar Kane, Executive Director of MGP Africa, Dr. El Hadji Adama Toure, WB, Mamadou Kane, AfDB, Amelie Solal-Celigny, and Nourou Macky Tall, FAO. Special thanks to Dr. Zakari Lawal, head of the National Planning Commission of Nigeria. i

4 ABBREVIATIONS AAA AECID AfDB ASER ASSESS ATA AU BSI CIDA COMESA CRC DAC DBA DCEF EBDM ECCAS ECOWAS ENDA ESG-UQAM EU FAO GANTT GDP GOS IAE IESA ILO IMF IS KIM LDP LED LFA M&E M&E/SEE MDGs MfDR NDP NFSP NGOs : Accra Agenda for Action : Agency for Development Cooperation International of Spain : African Development Bank : Agence sénégalaise d'électrification rurale : Analytical Support Services and Evaluation for Sustainable Systems : Agriculture Transformation Agenda : African Union : Business Science Institute : Canadian International Development Agency : The Common Market for Eastern and Southern Africa : Coastal Resource Center : Development Assistance Committee : Doctorate in Business Administration : Direction of Economic and Development Cooperation, Senegal : Evidence-Based Decision-Making : The Economic Commission of Central African States : The Economic Commission for the West Africa States : Environment and Development Third World : École des sciences de la gestion de l'université du Québec à Montréal : European Union : Food and Agriculture Organization : Henry Gantt Charts : Gross Domestic Product : Government of Senegal : Institut d'administration des entreprises : Water and Food Security Initiative in Africa : International Labor Organization : International Monetary Fund : Information System : Knowledge and Information Management : Local Development Program : Local Economic Development : Logical Framework Approach : Monitoring and Evaluation : Monitoring and Evaluation System Effectiveness Evaluation : Millennium Development Goals : Managing for Development Results : National Development Program : National Food Security Program : Non-Governmental Organizations

5 NPM ODD OECD OMD ON PERT PCI PSE RBM RDP SADC SDGs TCSF TOC UN UNDESA UNDP UNESCO URI USA USAID USDA UTS WA WB WC ZOI ZOPP : New Public Management : Objectifs de développement durable : Organization for Economic Co-operation and Development : Objectifs du millénaire pour le développement : Office du Niger : Project Evaluation and Review Techniques : Practical Concepts Incorporated : Plan Senegal Emergent : Results-Based Management : Regional Development Program : Southern African Development Community : Sustainable Development Goals : Technical Office for Support to Integrated Food Security : Theory of Change : The United Nations : The United Nations Department of Economic and Social Affairs : The United Nations Development Programme : The United Nations Educational, Scientific and Cultural Organization : University of Rhode Island : United States of America : United States Agency for International Development : United States Department of Agriculture : University of Technology Sydney : West Africa : World Bank : Washington Consensus : Zone of Influence : Zielorientierte Projektplanung iii

6 CONTENT ACKNOWLEDGEMENT... I ABBREVIATIONS... II ABSTRACT... 1 RÉSUMÉ... 2 EXECUTIVE SUMMARY... 3 RESUME EXECUTIF... 9 CHAPTER 1. GENERAL INTRODUCTION CHAPTER 2. PURPOSE AND AIM OF THE RESEARCH CHAPTER 3. DEVELOPMENT MANAGEMENT AND PERFORMANCE MEASUREMENT SYSTEMS: A LITERATURE REVIEW CHAPTER 4. A REVIEW OF MONITORING AND EVALUATION SYSTEM HISTORY CHAPTER 5. THE RESEARCH APPROACH CHAPTER 6. RESEARCH METHODS AND TECHNIQUES CHAPTER 7. THE PROPOSED FRAMEWORK FOR EFFECTIVE M&E-SYSTEM CHAPTER 8. RESULTS AND DATA ANALYSIS OF THE CASE STUDIES CHAPTER 9. RESULTS AND DATA ANALYSIS FROM THE SURVEY (ITERATION 3) CHAPTER 10. KEY FINDINGS OF THE RESEARCH CHAPTER 11. THE WAY FORWARD FOR IMPROVED MONITORING AND EVALUATION PRACTICE 202 CHAPTER 12. CONCLUSION ANNEXES... I BIBLIOGRAPHY... L

7 LIST OF TABLES Table 1. Gross official development assistance disbursements to Africa by Sector (US$ Millions), (United Nations, 2014:70) 18 Table 2. Logical Framework Matrix, (Coleman, 1987) 52 Table 3. The general research approaches 68 Table 4. Mapping of relevant articles and reports 80 Table 5. Documents consulted in the literature review 81 Table 6. Criteria for selection of the case studies 83 Table 7. Sub-dimensions and measurements for System Quality dimension of M&E System effectiveness framework 121 Table 8. Tool to track funding mobilization in ASER programs 124 Table 9. Sub-dimensions of M&E Information Quality and measurements to be considered 135 Table 10. Sub-dimensions and measurements for Service Quality dimension of M&E System effectiveness framework 143 Table 11. Matrix of correlations for variables/measurements of System Quality Design Quality 146 Table 12. Matrix of correlations for variables/measurements of System Quality Set-Up Quality 148 Table 13. Matrix of correlations for variables/measurements of System Quality Operations Quality 148 Table 14. Matrix of correlations for variables/measurements of System Quality Maintenance Quality 149 Table 15. Matrix of correlations for variables/measurements of System Quality Resources Quality 150 Table 16. Descriptive Statistics for System Quality Synthesis 151 Table 17. System Quality Synthesis Relation with the Program/Project domain/sector 151 Table 18. System Quality Synthesis linkage with the Dimension of the program/project 152 Table 19. System Quality Synthesis Linkage with the Program/Project duration? 152 Table 20. Matrix of correlations for variables/measurements of Information Quality Input Information Quality 156 Table 21. Matrix of correlations for variables/measurements of Information Quality Output Information Quality 157 Table 22. Matrix of correlations for variables/measurements of Information Quality Outcome Information Quality 158 Table 23. Matrix of correlations for variables/measurements of Information Quality Performance Information Quality 159 Table 24. Variable/measurement of Information Quality Risk Management Information Quality 159 Table 25. Information Quality Synthesis Relation with the Program/Project domain/sector 160 Table 26. Information Quality Synthesis Linkage with the Program/Project duration? 160 Table 27. Information Quality Synthesis linkage with the Dimension of the program/project 161 Table 28. Matrix of correlations for variables/measurements of Service Quality Information availability 165 Table 29. Matrix of correlations for variables/measurements of Service Quality Information accessibility 165 Table 30. Matrix of correlations for variables/measurements of Service Quality System Responsiveness 166 Table 31. Matrix of correlations for variables/measurements of Service Quality System Sustainability 167 Table 32. Upper Levels (Development Objective and Program Objective) of ASER Logical Framework Matrix XV LIST OF FIGURES Figure 1. ECOWAS Member Countries (in blue color) 46 Figure 2. Development Program Activities/Components and Specific Objectives. 51 Figure 3. Strategic alignment to national and global development objectives. 53 Figure 4. Dimensions of effective M&E System at organizational level 90 Figure 5. DML IS Success Model, Figure 6. Results Chain, (Meier, 2003:14) 94 Figure 7. Measurements for the Results-Based Management dimension of effective M&E System, The Author, Figure 8. Knowledge and information enable effective action, (Kock, McQueen, & Baker, 1996:10) 97 Figure 9. The SECI process, (Nonaka et al., 2000:12) 99 Figure 10. Knowledge Transfer and Comparative Advantage, (Li & Gao, 2003:12) 101

8 Figure 11. Measurements for the Knowledge and Information Management dimension of effective M&E System, The Author, Figure 12. Organizations as decision-making systems, (Choo, 1996:332) 105 Figure 13. Dimensions to assess decision process effectiveness, (Schilling, Oeser, & Schaub, 2007:231) 105 Figure 14. Measurements for the Evidence-Based Decision-Making dimension of effective M&E System, The Author, Figure 15. M&E System Effectiveness Framework, version zero (to be tested) 109 Figure 16. R. Grun, Value for Money in development program (2006) 124 Figure 17. Development program risk assessment tool, the author, Figure 18. Linkages between the dimensions of M&E System 171 Figure 19. Relationships between RBM Practice and dimensions of M&E System 173 Figure 20. Relationships between RBM Practice, KIM and dimensions of M&E System 174 Figure 21. Relationships between RBM Practice, KIM, EBDM and dimensions of M&E System 175 Figure 22. Effective M&E System Framework, the status from the results of analyses with quantified coefficients (Standardized) 184 Figure 23. The M&E System Effectiveness Framework with Quantified relationships 186 Figure 24. M&E System Effectiveness Framework 196 Figure 25. Pathway for Improved Policy and Program Design in the M&E System Effectiveness Framework 198 Figure 26. Pathway for Improved Operational Decision-Making in the M&E System Effectiveness Framework 199 Figure 27. Pathway for Improved Tactical and Strategic Decision-Making in the M&E System Effectiveness Framework 200 Figure 28. Pathway for Improved Capability to Advance Development in the M&E System Effectiveness Framework 200 i

9 ABSTRACT This research examined the capabilities of development institutions and organizations to managing programs and projects through effective Monitoring and Evaluation (M&E) System. It looked at how to measure M&E System effectiveness within greater learning processes for improved evidence-based decision-making and measured risk-taking to advance development objectives towards greater welfare, and freedom of people. The research purpose focused on how to build a rigorous measurement framework that robustly assesses M&E System effectiveness as a critical development management approach that could improve significantly the capabilities of institutions and organizations to move development goals to success. The research focused on cases in West Africa, which is one of the regions facing the most critical development issues. The research approach and methods are organized in three main phases for data collection, analysis and building of meaningful academic knowledge and managerial recommendations. First, a systematic literature review was conducted to gauge the existing knowledge and map the different theories on effective M&E Systems and their expected outcomes, Information System (IS), Results Based Management, Knowledge and Information Management, and Evidence-Based Decision-Making, to draw an initial model. Second, three cases were selected in West Africa to study and operationalize the dimensions and sub-dimensions of the proposed model. Third, the detailed model and its initial measurements and benchmarks were submitted to a survey with a sample of fifty development programs in West Africa that cover the main development sectors at local, national and regional levels. Data from the survey was analyzed with multiple correlations and multiple regression methods. The main findings of the research led to a rigorous M&E System effectiveness framework, its dimensions, and sub-dimensions, how they interact through tangible pathways towards greater capability of organizations and institutions to move development actions to success. The findings also clarified how to manage M&E Systems better and how to effectively address the needs of different development organizations and institutions in assessing and improving their M&E Systems and learning platforms in the building of more effective demand-driven M&E supports services. Key words: Monitoring and Evaluation (M&E), Development Management, Development Institutions and Organizations, Results-Based Management, Knowledge and Information Management, Evidence- Based Decision Making, Development Policy. 1

10 RESUME Cette recherche a examiné les capabilités des institutions et organisations de développement à gérer des programmes et projets par des systèmes de Suivi et Evaluation (S&E) plus efficaces. La recherche a analysé comment mesurer l efficacité des Systèmes de S&E pour une meilleure prise de décisions basées sur des preuves et une prise de risques plus réfléchie dans la réalisation des objectifs de développement en direction du bien-être et de la liberté améliorés des peuples. L objet de la recherche s est focalisé sur comment construire un cadre de mesure rigoureux qui évalue l efficacité des systèmes de S&E comme outils critiques et approches de gestion du développement pouvant améliorer de manière significative les capabilités des institutions et organisations à faire avancer les objectifs de développement. La recherche a utilisé des cas en Afrique de l Ouest qui est une des régions qui fait face aux problèmes de développement les plus difficiles. L approche et les méthodes utilisées dans cette recherche s organisent autour de trois phases essentielles dans la collecte, l analyse des données et la construction de connaissance robuste ainsi que de recommandations managériales tangibles. Premièrement, une revue systématique de la littérature a été conduite pour analyser et cartographier les différentes théories sur l efficacité des systèmes de S&E et leurs résultats escomptés, les systèmes d information (IS), la gestion axée sur les résultats (GAR), la gestion des connaissances et de l information, la prise de décisions fondées sur des preuves, pour dessiner un model initial pour la recherche. Deuxièmement, trois cas ont été sélectionnés en Afrique de l Ouest pour étudier et opérationnaliser le modèle proposé. Troisièmement, le modèle détaillé avec ses mesures initiales et benchmarks a été soumis à une enquête à travers un échantillon de cinquante programmes en Afrique de l Ouest qui couvrent les principaux secteurs de développement au niveau local, national et régional. Les données de l enquête ont été analysées en utilisant les méthodes de corrélations multiples et de régressions multiples. Les résultats de la recherche ont mené vers le cadre de mesure de l efficacité des systèmes de S&E, ses dimensions, sous-dimensions et comment elles interagissent à travers des chemins qui guident vers des capabilités meilleures des institutions et organisations à conduire les actions de développement vers le succès. Les résultats ont aussi montré comment gérer les systèmes de S&E mieux et régler efficacement les besoins des institutions et des organisations de développement variées dans l évaluation et l amélioration de leur system de S&E et plateformes d apprentissage vers l élaboration et la mise en œuvre de services de S&E qui répondent mieux à la demande. Mots clés : Suivi et Evaluation (S&E), Management du développement, Institution et organisations du développement, Gestion axée sur les résultats, Gestion de la connaissance et de l information, Prise de décisions fondées sur des preuves, Politiques de développement. 2

11 EXECUTIVE SUMMARY The main question of this research is: How to measure the effectiveness of development programs Monitoring and Evaluation Systems for improved evidence-based decision-making and measured risktaking to advance development in the context of West Africa? Despite billions of funding spent in development strategy and policy design and implementation in Africa, the continent still faces tremendous development challenges ranging from food insecurity, high unemployment, poverty, and inequality, to commodity dependence, lack of economic transformation, environmental degradation, and low integration of the continent in the global economy. (United Nations, 2014:2). The main field of this research is Development Management and specifically Performance Assessment Systems for development policies and programs directly or indirectly designed and implemented by organizations and institutions such as governments, regional bodies, development partners, and other stakeholders. The research examined the role of institutions and organizations in the development process and analyzes how institutions, the rules of the game, and organizations, the game players, shape the development process toward the desired changes. It was highlighted that development is not only about economics and neoclassical approaches but should focus first on the creation of effective institutions, as Douglas North clarifies that institutions and organizations lead the set of standards for any given game including development management (North, 2013). We analyzed development management, as the attempt to moving institutions and organizations towards higher efficiency and effectiveness to advance development actions. Hence, effective management approaches are essential to support institutional and organizational capabilities to move development actions to success through improved efficiency and effectiveness (Beaulieu, 2015; Kemp, Parto, & Gibson, 2005; Mantzavinos, North, & Shariq, 2004; North, 1994, 2013; Roper & Pettit, 2002). One of the key management approaches in this instance in the development performance measurement approach is called Monitoring and Evaluation (M&E) System. We clarified how institutions and organizations use the M&E System in their role of managing development. The research focused on the West Africa region which is one of the regions where the most critical development challenges are concentrated as well as the related management constraints. West Africa is also the region where the Author spent his entire professional experience on M&E Systems of development programs. In the West Africa region, three levels of organizations exist, regional, national and local. At these different levels, institutions and organizations define development strategies and policies aligned with the global development objectives led by the United Nations organizations, the Sustainable Development Goals (SDGs). We noted that there is a growing need for accountability and efficiency in managing development, especially in Africa where development results are yet to be recorded, and highlighted the 3

12 Paris declaration in 2005 that brought back M&E Systems and results-based management at the crossings of institutional and organizational capability in better managing development actions. The research focused mainly on our real experiences in Monitoring and Evaluation System practice in the region, and on a sample survey on current M&E Systems of fifty development programs in West Africa. The research approach is trifold: Literature-Case Studies-Survey, and comprises three iterations or evidence-based building blocks, within each of which comprehensive, rigorous, and valid approaches were used to draw findings and conclusions on the research question and sub-questions, as well as managerial recommendations to advance Development Management practices. In the design of the detailed M&E System effectiveness framework, the two first iterations aimed at exploring the existing knowledge and to learn from specific well selected cases in finalizing the comprehensive M&E System effectiveness model ready to be tested through a sample survey, while the third iteration is a sample survey to verify the validity of measurements proposed in the framework and their interrelations. For each iteration, commensurate approaches were used to collect and analyze the data. In the first iteration which focuses on a review of the dimensions of effective M&E System, we reviewed systematically the existing knowledge in the M&E System literature and development practitioners approaches in a view to support the process of designing a robust framework. First, we revisited the history of M&E System in development management theories, from the 1886 Charles Booth s Life of the People of London (Gelase, 2011), to the recent approaches developed between the MDGs in 2000s and the advent of the SDGs in 2015, and looked at the various tendencies and analyses on the origin and development of the concept of M&E System. We then analyzed the core stages of the M&E System practices largely influenced by the development theories and priorities at different periods from the end of the World War II (WWII), the independence of African countries in the 50-60s, the A. O. Hirschmann's Development Projects Observed with their uncertainties (Hirschman, 1967), the advent of the Logical Framework Approach (LFA), and the Washington Consensus. We then analyzed and ranked the M&E practices in the last decades in three main stages: The Network-Chart stage (60-70s), the Impact Evaluation Clinical Trial stage (80-90s), the Web-based information system stage (the 2000s). We reviewed the M&E System effectiveness literature from the views of scholars and practitioners (J. Cameron, 1993; Cracknell, 1994; Kusek & Rist, 2008; K. Mackay, 2007; Mackay, 2008, 2010a, 2010b; Quesnel, 2010) and analyzed the viewing of M&E System as an information system (Barton, 1997; Crawford & Bryce, 2003; Wagner et al., 2005). We concluded with a presentation of the most meaningful IS success models (DeLone & McLean, 2003; Leclercq-Vandelannoitte, Isaac, & Kalika, 2014; McGuire, 2011; Petter, DeLone, & McLean, 2008; Soualhia & Mejbri, 2014), that would serve as the basis for the building of a framework for effective M&E System. 4

13 We then analyzed the knowledge in the development field related to effective M&E System and practices in various development programs in the West Africa region to fine tune the proposed model for effective M&E System measurement framework. We noted that the strength of effective M&E is in bridging the gap between the different stages of the M&E System history. The long route dispute between these sides of performance management and effective results-based M&E, hindered clear and appropriate conceptualization, development and set of effective M&E systems in the Aid Industry field mainly in developing countries. As we argue here, a strong results-based approach is a key to organizations and institutions in charge of development programs, and M&E Systems are important feeding elements to planning, implementing and managing development programs. An effective M&E System would then have a positive impact on the organizations practice of Results-Based Management. We found that Results- Based Management Practice, Knowledge and Information Management culture and Evidence-Based Decision-Making Process, are the dynamic organizational capabilities that would be improved via an effective M&E System set forth by the organizations to design, implement and manage development actions. In the iteration 2, we analyzed the Office du Niger (ON) M&E System case study in Mali; we looked at the System Quality dimension of the M&E System effectiveness framework to analyze and select the appropriate, relevant measurements to be considered for the framework. The study revealed that key subdimensions to be considered in analyzing the System Quality dimension include: 1) Design Quality, 2) Setup Quality, 3) Operations Quality, 4) Maintenance Quality, and 5) Resources Quality. We then analyzed the Senegalese Rural Electrification program (ASER) case in Senegal through the dimensions of Quality Information that revealed five key criteria to be considered: 1) Quality inputs information, 2) Quality outputs information, 3) Quality outcomes information, 4) Quality performance information, and 5) Quality risks management information. We finally analyzed the FAO Water and Food Security Initiative (IESA) M&E System case study, at regional level in West Africa; we looked at the Service Quality dimension of the M&E System effectiveness framework to analyze and select the appropriate, relevant measurements to be considered for the M&E System effectiveness framework. The study shown that key sub-dimensions to be considered in analyzing the Service Quality dimension include: 1) Information Availability, 2) Information Accessibility, 3) System Responsiveness, 4) System Flexibility and Adaptability, and 5) System Sustainability. The iteration 3 used a sample survey of fifty development programs in the West Africa region to analyze linkages between the different components of the proposed M&E System Effectiveness Framework that is detailed in the Iteration 2. The responses received total forty-six among which there are national and regional development programs in ten countries among the fifteen ECOWAS countries; there are development programs in all the priority development sectors, including agriculture, infrastructure, energy, environment, 5

14 democracy and governance, poverty reduction and social protection, education, health, etc.; the development programs in the sample have at least reached mid-course or are closed less than three years ago; M&E Managers, Program Managers, Technical and Financial Partners, other program components managers, responded to the questionnaire built using the Likert Scales and benchmarks from the results of iteration 2. Data collection, cleaning, and analysis were made using SPHINX; only a few secondary analyses required SPSS. The data analysis comprises multiple correlation analyses to gauge the linkages among the key proposed measurements of the three dimensions of M&E System. The approach also used multiple regression analyses to assess the linkages between the three dimensions of M&E System with the three dynamic capabilities in managing development programs, namely Results-Based Management (RBM), Knowledge and Information Management (KIM), and Evidence-Based Decision-Making (EBDM). In the same approach, multiple regression analyses were used to test the linkages between the Development Management capabilities and the five net benefits of Effective M&E System, namely 1) Improved Policy and Program Design, 2) Improved Operational Decisions, 3) Improved Tactical Decisions, 4) Improved Strategic Decisions, and 5) Improved Capability to Advance Development Objectives. The analyses led to the revised M&E System Effectiveness Framework showing the tangible quantified relationships between the dimensions of M&E System, the Development Management capabilities and the M&E System net benefits. We find that the pathway of Improved policy and program design starts with quality M&E System and quality M&E Information. The design, operations, resources and maintenance of the M&E System, when well performed by organizations, influence positively the quality of M&E information (quality inputs, outputs, outcomes, performance and risks management information), which, in turn, influences the effective practice of Results-Based Management in the organizations, which will contribute significantly to improved policy and program design for future planning. We find that Improved operational decisions are influenced directly mainly by the Evidence-Based Decision-Making (EBDM) practice within the organization. The pathway for improved operational decisions starts with M&E Service Quality, which is influenced positively by M&E Information Quality and System Quality. Service Quality influences directly the EBDM practice which in turn generates directly Improved operational decisions. The pathway is completed by the positive influence on EBDM from effective KIM. System Quality and Service Quality facilitate effective KIM through improved learning processes, which contributes greatly to effective EBDM. All these dimensions of effective M&E System should function effectively to ensure improved operational decisions. While Improved operational decisions need only effective EBDM pathway, Improved Tactical and Strategic Decisions necessitate both effective EBDM and effective RBM pathways. Tactical decisions are 6

15 decisions that will affect the programs intermediary results, and strategic decisions affect the higher levels of programs, specific objectives, and goals. Hence, EBDM is necessary but not sufficient. There is need to ensure effective RBM practice together with effective EBDM. Effective KIM contributes to RBM and EBDM to generate Improved tactical and strategic decisions. Improved Capability to Advance Development by organizations and institutions in charge of development actions is the highest objective of effective M&E System. Effective M&E System has a long-term positive impact on the capability of organizations and institutions to advance development objectives. We find that two dynamic capabilities are necessary to ensure the improved capability to advance development objectives, Results-Based Management (RBM) and Knowledge and Information Management (KIM). This is the singular effective M&E System net benefit directly influenced by KIM practice. Improved learning process is a key success factor for an organization to be able to move development actions to success. In short, in 1886, Charles Booth, a Statistician from London wrote: To effectively deal with poverty there was need to gather quantitative information on characteristic of poverty. (Booth, 1886:67), we find that to effectively deal with poverty or any other development objective, there is need to set effective M&E System with quality information, system, and service, within structured organizations and institutions which practice effectively Results-Based Management, and Knowledge and Information Management. Effective M&E System contributes to expanding the successful practice of the key Development Management approaches, Results-Based Management, Knowledge and Information Management, and Evidence-Based Decision-Making, to advance development objectives towards improved welfare and freedom of people. The research findings and conclusions could have a lot of managerial implications in the ways M&E Systems are created and operate. We put the focus on the two most strategic and valuable learnings for the way forward that would greatly add value to the M&E practices and their expected outcomes. First, there should be a paradigm change in the way M&E System is viewed. There is need to build the capacity of development practitioners on the effective M&E System comprehensive framework. There is critical need to improve the understanding of the comprehensive M&E framework and its objectives, so as development practitioners be aware of what is the contribution of effective M&E System in translating development plans into tangible changes on the welfare and freedom of people. Hence, the first important step for a development organization or institution is to ensure that there are capacity building actions and outcomes on the way their employees, beneficiaries, and various partners understand their framework for effective M&E System. There is need to review in many organizations and institutions the organizational charts that relate to M&E activities, learning, and decision-making processes. The M&E department should include the 7

16 required functions that create effective M&E System. In addition to having highly skilled M&E Experts who are playing the needed roles to make the M&E dimensions effective: System Quality, Information Quality, and Service Quality, the department should include expertise in RBM, KIM and EBDM, at least a Strategic Planning expert, and Knowledge and Information expert, and a high Caliber Program Analytics expert. Second, better understanding how to assess M&E System effectiveness and its outcomes. There is need to move from Data Quality Assessment - DQA practices to a broader M&E System Effectiveness Evaluation M&E/SEE culture. This would be one of the ways to ensure that organizations effective management practices are leading to advanced sustainable development goals. DQA and other assessments of specific aspects of M&E System are useful tools that should be integrated into a broad evaluation of M&E Systems effectiveness (M&E/SEE) that looks at the different dimensions of the M&E System, their interrelations with the organizations dynamic development management capabilities (RBM, KIM and EBDM) towards improved capacity to advance development objectives. Henceforward, M&E/SEE would help see whether the M&E System is operating along lines that increase organizations capability to advance development objectives. At this juncture, there are two cases to be considered: o In case the organization s M&E System does not face challenging technical weaknesses, and seems to work perfectly in a first view, over the implementation of the program, during annual reviews, mid-term evaluations, or routine supervision missions, the scopes could integrate a check list on the M&E System effectiveness that would greatly help quickly to check whether the M&E System presents the minimum characteristics of effectiveness, and run a Quick M&E/SEE. o In case there is need to carry out a deep assessment and take a particular look at the organization s M&E System, like the case of Office du Niger in the Helen project (A.3. Introduction to Office du Niger Contract Plan Case 1), organizations can run a Deep M&E/SEE. The framework also offers the perspective on how to efficiently manage resources for M&E Systems in looking at their effectiveness. If Quick M&E/SEE or Deep M&E/SEE are used effectively, there are ways to mobilize resources more efficiently for M&E activities. However, the study focuses only on the West Africa region, a systematic generalization of the results is not possible, but up-scaling might be considered with measured steps. There are a lot of productions of knowledge in the M&E System area that are not always published scientifically but only used internally by development organizations; those productions were then not used in the literature review in this research. To address that issue, a large review of the available literature was made as an attempt to cover all the relevant aspects of M&E System effectiveness. 8

17 RESUME EXECUTIF La principale question de cette recherche est : Comment mesurer l efficacité des systèmes de suivi et évaluation des programmes de développement dans l amélioration du processus de prise de décision basée sur des preuves et de prise de risques mesurés pour l atteinte des objectifs de développement dans le contexte de l Afrique de l ouest? Malgré les milliards en ressources financières dépensés dans les stratégies et les politiques de développement en Afrique, le continent fait toujours face à d énormes défis de développement comprenant : «l insécurité alimentaire, l important niveau de chômage, la pauvreté, les inégalités, la dépendance alimentaire, le manque de transformation de l économie et la faible intégration du continent à l économie globale.» (Les Nations unies, 2014:2). Le champ principal de cette recherche est le management du développement, spécifiquement, les systèmes de mesures de performance pour les politiques et programmes de développement directement ou indirectement préparés et mis en œuvre par des organisations et des institutions comme les gouvernements, les institutions régionales, les partenaires du développement et autres parties prenantes. La recherche a examiné le rôle des institutions et des organisations dans le processus de développement et analysé comment les institutions, «les règles du jeu», et les organisations, «les acteurs du jeu», façonnent le processus de développement en direction des changements escomptés. Il a été souligné que les théories du développement ne sont pas seulement relatives aux approches économiques et néo-classiques, mais doivent se focaliser en premier sur la création d institutions efficaces, comme Douglas North a clarifié que les institutions et les organisations dirigent la mise en place de normes dans n importe quel «jeu», y compris le management du développement. Nous avons analysé le management du développement comme un processus qui mène les organisations et les institutions vers plus d efficience et d efficacité dans l atteinte des objectifs de développement. Ainsi, des approches de management efficaces sont essentielles pour supporter les capacités des organisations et des institutions à diriger les actions de développement vers le succès à travers une efficience et une efficacité améliorées (Beaulieu, 2015; Kemp et al., 2005; Mantzavinos et al., 2004; North, 1994, 2013; Roper & Pettit, 2002). Une des approches des plus connues, dans ce domaine de la mesure de la performance du développement, est appelée Système de Suivi et Evaluation (S&E). Nous avons clarifié comment les institutions et les organisations utilisent le Système de S&E dans leur rôle de management du développement. L Afrique de l ouest a été choisie comme champ de cette recherche car c est une des régions du monde dans lesquelles sont concentrés la plupart des problèmes de développement et les contraintes de management y afférentes ; c est aussi le champ d expérience de l auteur dans le domaine de S&E de programmes de développement. En Afrique de l ouest, il existe trois niveaux d organisation territoriale : régional, national et local. Dans ces trois niveaux, les institutions et les organisations définissent des stratégies et des politiques de développement alignées aux objectives globaux de développement menées 9

18 par le Système des Nations-Unies, les Objectifs de développement durable (ODD). Nous avons noté qu il y a un intérêt grandissant pour la gestion efficiente et responsable du développement, spécialement en Afrique où les résultats du développement tardent à se montrer, et nous avons souligné la Déclaration de Paris de 2005 qui a ramené les Systèmes de S&E et la gestion axée sur les résultats à la croisée des chemins de la capacité des institutions et organisations à mieux gérer les actions de développement. La recherche a été centrée sur notre propre expérience dans le domaine de S&E dans la région ouest africaine et sur une enquête par échantillon menée sur cinquante programmes de développement en Afrique de l ouest. L approche utilisée pour la recherche est triphasée : Revue de la Littérature-Etudes de cas sur le terrain-enquête par échantillon, et a compris trois itérations ou blocs de travaux basées sur des analyses robustes, dans chacune desquelles, des approches rigoureuses et valides ont été utilisées pour arriver à des résultats et conclusions sur la question de recherche et les questions subséquentes, de même que des recommandations managériales pour faire avancer les pratiques de management du développement. Dans la conception du cadre de référence de la mesure de l efficacité du System de S&E, les deux premières itérations avaient pour objectifs d explorer l état de la connaissance existante et d apprendre de la pratique de S&E à travers de cas spécifiques dans la finalisation du cadre général de mesure de l efficacité du System de S&E à tester. La troisième itération est une étude quantitative basée sur une enquête par échantillon pour vérifier et valider les variables proposées dans le cadre de mesure de l efficacité des Systèmes de S&E et leurs interrelations. Pour chaque itération, des approches appropriées ont été utilisées pour la collecte et l analyse des données. Dans la première itération portant sur une revue des dimensions du Système de S&E efficace, nous avons revu systématiquement la connaissance existante dans la littérature et les approches des praticiens du développement dans une perspective de renforcer le processus de conception d un cadre de mesure robuste. Nous avons revisité l histoire des Systèmes de S&E dans la théorie de management du développement, depuis la publication en 1886 par Charles Booth de «La vie des populations de Londres» (Gelase, 2011), aux approches récentes entre les OMD en 2000 et l avènement des ODD en 2015 ; et nous avons consulté les tendances et analyses variées sur l origine et le développement du concept de Système de S&E. Nous avons ainsi analysé les étapes principales des pratiques de S&E largement influencées par les théories et priorités de développement à des périodes différentes entre la fin de la deuxième guerre mondiale, l indépendance des pays africain dans les années 50-60, le livre de A.O. Hirshmann «Les programmes de développement observés» et leurs incertitudes (Hirschman, 1967), l avènement du cadre logique, et du Consensus de Washington. Nous avons ainsi analysé et ordonné les pratiques de Systèmes de S&E dans les cinq dernières décennies dans trois principales étapes : la Phase des Réseaux et graphiques (années 60-70), la Phase des Tests cliniques d évaluation d impact (années 80-90), et la Phase des applications basées sur 10

19 le web (années 2000). Nous avons revu la littérature sur l efficacité des Systèmes de S&E à partir des vues des praticiens et académiciens (J. Cameron, 1993; Cracknell, 1994; Kusek & Rist, 2008; K. Mackay, 2007; Mackay, 2008, 2010a, 2010b; Quesnel, 2010), et analysé le Système de S&E dans la perspective de Système d information (Barton, 1997; Crawford & Bryce, 2003; Wagner et al., 2005). Nous avons conclu avec une présentation du plus important modèle d analyse du succès des SI (DeLone & McLean, 2003; Leclercq- Vandelannoitte et al., 2014; McGuire, 2011; Petter et al., 2008; Soualhia & Mejbri, 2014) qui pourrait servir de base dans la construction d un cadre de mesure de l efficacité du Système de S&E. Nous avons ainsi analysé la connaissance existante dans le champ du développement qui est relative au System de S&E et des pratiques dans les programmes de développement variés de la région de l Afrique de l ouest pour affiner le modèle proposé pour la mesure de l efficacité des Systèmes de S&E. Nous avons noté que la force du S&E efficace est dans la réalisation de ponts entre les différentes phases des systèmes de S&E. La longue et hardie dispute entre ces approches de management de la performance et de gestion effectivement axée sur les résultats, ont gêné la conceptualisation claire et appropriée, le développement et la mise en place de systèmes de S&E efficaces dans l industrie du développement principalement dans les pays en développement. Ainsi comme nous argumentons ici, une approche forte de gestion axée sur les résultats est importante pour les organisations et les institutions en charge des programmes de développement, et les Systèmes de S&E sont les systèmes qui alimentent les processus de planification, de mise en œuvre et de management des programmes de développement. Un Système de S&E efficace devrait ainsi avoir un impact positif sur les pratiques de la gestion axée sur les résultats par les organisations. Nous avons trouvé que la pratique de Gestion axée sur les résultats (GAR), la culture de Gestion de la connaissance et de l information (GCI), et le processus de Prise de décision basée sur des preuves (PDBP) sont les capacités dynamiques organisationnelles qui pourraient être améliorées à travers un Système de S&E efficace mis en place par les organisations pour concevoir, mettre en œuvre et gérer les actions de développement. Dans la deuxième itération ou phase d investigation du terrain, nous avons procédé à l étude de cas sur le System de S&E de l Office du Niger (ON) au Mali, nous avons examiné la dimension Système Qualité du cadre de mesure de l efficacité du Système de S&E pour analyser et sélectionner les variables de mesures importantes appropriées à considérer dans le modèle. Les résultats de l étude de cas ont montré que les sousdimensions pertinentes à considérer sont : 1) Qualité de la conception, 2) Qualité de l implantation, 3) Qualité des opérations, 4) Qualité de la maintenance, et 5) Qualité des ressources. Nous avons par la suite analysé les résultats de l étude de cas du Programme d électrification rurale (ASER) au Senegal, à travers la dimension Qualité de l information, ce qui a montré que les critères clés à considérer dans cette dimension sont : 1) Qualité de l information sur les intrants, 2) Qualité de l information sur les extrants, 3) Qualité de 11

20 l information sur les effets et impacts, 4) Qualité de l information sur la performance, et 5) Qualité de l information sur la gestion des risques. Finalement, nous avons analysé les résultats sur l étude de cas sur l Initiative eau et sécurité alimentaire en Afrique de la FAO (IESA) au niveau régional Afrique de l Ouest, nous avons examiné la dimension Qualité du Service du cadre de mesure de l efficacité du System de S&E pour analyser et sélectionner les mesures pertinentes de cette dimension. L étude de cas IESA de la FAO, a montré que les sous-dimensions essentielles à considérer dans l analyse de la dimension Qualité de service sont : 1) La disponibilité de l information, 2) L accessibilité à l information, 3) La réactivité du système, 4) La flexibilité et l adaptabilité du système, et 5) La durabilité du système. Dans la troisième itération ou phase d enquête par échantillon, une enquête été menée sur cinquante programmes de développement en Afrique de l ouest pour analyser les interrelations entre les différentes composantes du cadre de la mesure de l efficacité des systèmes de S&E proposé à partir de la première et de la deuxième itération. Le nombre total de réponses reçues est quarante-six (46) parmi lesquelles on peut comptabiliser des programmes régionaux et des programmes nationaux, dans dix pays parmi les quinze pays de la Commission économique des Etats de l Afrique de l ouest (CEDEAO) ; il y a dans l échantillon des programmes dans tous les secteurs prioritaires de développement, y compris l agriculture, les infrastructures, l énergie, l environnement, la démocratie et la gouvernance, la réduction de la pauvreté et la protection sociale, l éducation, la sante, etc. ; les programmes de développement ciblés dans l échantillon ont au moins atteint la mi-parcours ou sont clôturés il y a moins de trois ans. Les Spécialistes en S&E, les Gestionnaires de programmes, les Partenaires techniques et financiers, les Responsables de composantes, etc., ont répondu au questionnaire construit en utilisant les scores de Likert avec des benchmarks à partir des résultats de la deuxième itération. Les données ont été collectées, nettoyées avec le logiciel SPHINX, seulement quelques analyses mineures ont requis l utilisation de SPSS. L analyse des données a compris une analyse des corrélations multiples pour jauger les liens entre les mesures et variables proposées dans les trois dimensions du Système de S&E. L approche d analyse des données a aussi utilisé la régression multiple pour analyser les liens entre les trois dimensions du Système de S&E et les trois capabilités dynamiques dans la gestion des programmes de développement, que sont la GAR, la GCI et la PDBP. Dans la même approche, la régression multiple a été utilisée pour tester les relations entre les capabilités de gestion des programmes de développement et les cinq bénéfices nets d un Système de S&E efficace, que sont 1) La conception des politiques programmes est améliorée, 2) Les décisions opérationnelles sont améliorées, 3) Les décisions tactiques sont améliorées, 4) Les décisions stratégiques sont améliorées, et 5) La capacité à atteindre les objectifs de développement est améliorée. Les résultats des analyses ont mené vers la présentation du modèle révisé de mesure de l efficacité des Systèmes 12

21 de S&E et les relations quantifiées tangibles entre les dimensions du Système de S&E, les capabilités de gestion des programmes de développement, et les bénéfices nets du Système de S&E efficace. Nous avons trouvé que le cheminement pour «La conception des politiques et programmes est améliorée» commence avec le Système de qualité, et l Information de qualité. La conception du Système de S&E, ses opérations, ses ressources et sa maintenance, lorsque qu elles sont menées à bien par les organisations, influencent positivement l Information de qualité (qualité de l information sur les intrants, les extrants, les effets et impacts, la performance et la gestion des risques), qui va, en retour influencer positivement la pratique de la GAR dans l organisation, ce qui conduira à une amélioration significative de la conception future des politiques et programmes. Nous avons trouvé que l amélioration des décisions opérationnelles est influencée directement par surtout la pratique de la PDBP dans l organisation. Le cheminement pour l amélioration des décisions opérationnelles commence par le Service de qualité, qui est influencé positivement par l Information de qualité et le Système de qualité. Le Service de qualité influence positivement directement la pratique de la PDBP, qui en retour influence positivement l amélioration des décisions opérationnelles. Le cheminement est complété par l influence positive sur la PDBP de la part de la culture de la GCI à travers un processus d apprentissage et de collaboration amélioré, qui contribue grandement à la PDBP effective. Toutes ces dimensions du Système de S&E efficace devront fonctionner correctement pour favoriser des décisions opérationnelles améliorées. Au moment où, l amélioration des décisions opérationnelles a uniquement besoin du cheminement par la PDBP, l amélioration des décisions tactiques et stratégiques nécessitent à la fois des cheminements de la PDBP et la GAR effectives. Les décisions tactiques sont les décisions qui affectent les résultats intermédiaires des programmes, et les décisions stratégiques quant à elles, affectent les niveaux supérieurs de la chaine des résultats des programmes pour toucher les objectifs spécifiques et globaux. Ainsi, la PDBP est nécessaire, mais pas suffisante. Il y a besoin d assurer une GAR effective dans l organisation en plus de la PDBP. La GCI effective contribue à la fois à la PDBP et la GAR pour générer des décisions tactiques et stratégiques améliorées. L amélioration de la capacité à atteindre les objectifs de développement par les organisations et les institutions en charge des actions de développement est l objectif le plus élevé du Système de S&E efficace. Le Système de S&E efficace a des effets long-terme positifs sur la capacité des organisations et des institutions à atteindre les objectifs de développement. Nous avons trouvé que deux capabilités dynamiques sont nécessaires pour assurer l amélioration des capacités à atteindre les objectifs de développement, ce sont la GAR et la GCI. C est le seul bénéfice net du Système de S&E efficace directement influencé par la GCI. Un processus d apprentissage et de collaboration renforcé à travers une meilleure GCI est un facteur clé de 13

22 succès pour que les organisations soient capables d atteindre les résultats de développement plus efficacement. En résume, en 1886, Charles Booth, un Statisticien de Londres, a écrit : «Pour lutter efficacement contre la pauvreté, il y avait besoin de collecter de l information quantitative sur les caractéristiques de la pauvreté.» (Booth, 1886:67), nous avons trouvé que pour lutter efficacement contre la pauvreté ou pour atteindre n importe quel autre objectif de développement, il y a besoin d établir un Système de S&E efficace avec information, système et service de qualité, dans des organisations et institutions structurées qui pratiques effectivement la GAR, la GCI et la PDBP. Le Système de S&E efficace contribue à renforcer et répandre des pratiques réussies de gestion de développement majeures que sont la GAR, la GCI et la PDBP pour faire avancer plus efficacement les objectifs de développement en direction de meilleures conditions de vie et de liberté des populations. Les conclusions et résultats de la recherche pourraient avoir beaucoup d implications pour l amélioration des pratiques managériales dans une perspective dans laquelle les Systèmes de S&E sont créés et opèrent. Nous avons mis l accent ici, sur deux connaissances acquises des plus stratégiques pour la suite, qui pourraient ajouter de la valeur dans les pratiques de S&E et leur résultats escomptés. Premièrement, il devrait y avoir un changement de paradigme dans la compréhension du Système de S&E. Il y a besoin de renforcer les capacités des praticiens du développement dans la compréhension du cadre du Système de S&E efficace. Il y a un besoin critique d améliorer la compréhension du modèle complet de Système de S&E efficace dans la traduction des plans de développement en changements tangibles dans les conditions de vie et la liberté des populations. Ainsi, la première étape importante pour les organisations et institutions en charge de développement c est d assurer qu il y a des plans de renforcement des capacités et des résultats sur la façon dont leurs employés, bénéficiaires et partenaires variés comprennent leur cadre de Système de S&E efficace. Il y a besoin de revoir dans beaucoup d organisations et institutions l organigramme relatif aux activités de S&E, apprentissage et collaboration, et du processus de prise de décision. Le département en charge de S&E doit intégrer les fonctions qui créent et sous-tendent le Système de S&E efficace. En plus d avoir d experts de S&E de haute qualité, qui vont jouer le rôle de rendre effectives les dimensions du Système de S&E : qualité du système, qualité de l information, et qualité du service ; le département devrait intégrer de l expertise en GAR, GCI et PDBP, au moins un Expert en planification stratégique, un expert en gestion de la connaissance et de l information, et un expert de haut calibre Programme analyste. 14

23 Deuxièmement, une meilleure compréhension de comment évaluer l efficacité des Systèmes de S&E et leurs résultats. Il y a besoin de passer des pratiques de l «évaluation de la qualité des données», plus connue sous le nom de DQA, vers la culture de l «évaluation de l efficacité du Système de S&E» plus large, appelée ici M&E/SEE. Cela devrait être une des voies pour assurer que la gestion effective du développement par les organisations mène vers l atteinte des objectifs de développement durables. Les DQA et autres évaluations sur des aspects spécifiques des Systèmes de S&E sont des outils utiles qui devront être intégrés dans une évaluation de l efficacité des Systèmes de S&E plus large (M&E/SEE) qui regarde les différentes dimensions du Système de S&E, leurs interrelations aux capabilités dynamiques des organisations (GAR, GCI et PDBP) en direction de l amélioration de la capacité d atteindre les objectifs de développement. A ce titre, deux cas devront être considérés : o Dans le cas où le Système de S&E de l organisation ne fait pas face à des faiblesses techniques majeures, et semble fonctionner parfaitement dans une première vue, durant la mise en œuvre des programmes, durant les revues annuelles, l évaluation à mi-parcours, ou lors des missions de supervision de routine, les termes de références pourront intégrer une checklist pour vérifier si le Système S&E présente les caractéristiques minima d efficacité, ainsi mener lors de ces missions un Quick M&E/SEE. o Dans le cas où il y a besoin de mener une évaluation approfondie pour prendre un regard particulier au Système de S&E de l organisation, comme c était le cas de l Office du Niger dans le projet Helen (A.3. Introduction to Office du Niger Contract Plan Case 1), les organisations peuvent mener une Deep M&E/SEE. Le cadre offre aussi une perspective dans la gestion des ressources allouées aux Systèmes de S&E en regardant leur niveau d efficacité. Si Quick M&E/SEE ou Deep M&E/SEE sont menées à bien, il y a des possibilités de mobiliser des ressources pour les Systèmes de S&E de manière plus efficiente et effective. Cependant, la recherche s est focalisée seulement sur le cas de l Afrique de l Ouest, une généralisation systématique des résultats n est donc pas possible, une inférence pourrait être envisagée en suivant une approche rigoureuse. Il y a beaucoup de productions de connaissance sur les systèmes de S&E qui ne sont pas publiées scientifiquement, mais seulement utilisées à l interne par les Organizations, ces connaissances ne sont pas toujours considérées dans notre revue de la littérature. Une revue large de littérature disponible a été effectuée pour couvrir les aspects les plus pertinents des systèmes de S&E efficaces. 15

24 CHAPTER 1. GENERAL INTRODUCTION Development concerns occupy a central place in social sciences research as they are focused first of all on priority dimensions of human development, livelihood and welfare of people in a purely market-driven and freedom approach. From the local communities to the global society, development is an incredible challenge to mankind when more than 50% of the people in the world and countries are still considered "underdeveloped" despite outstanding science and technology progress, as they are lagging far behind in producing or accessing to the minimum basic needs in their daily existence. Human beings need, within a sustainable environment, to access to good and quality health, have free movement, access sufficient income, adequate and quality food, healthy entertainment, good education, decent housing, to ensure that a greater life is not threatened by factors that might hinder fundamental human rights, freedoms, and welfare. In September 2015, in the Seventeenth session of the United Nations, the Heads of States of the World and High-level Representatives, in the introductory statement of the 2030 Agenda for Sustainable Development Declaration, made commitment to the Sustainable Development Goals (SDGs): We envisage a world free of poverty, hunger, disease and want, where all life can thrive. We envisage a world free of fear and violence. A world with universal literacy. (United Nations, 2015:3-4). Douglas North analyzed the incapacity to achieve development objectives in the last decades and stressed that neoclassical theory is simply not appropriate to address development objective as it is first concerned with how markets operate, not how they develop (North, 1994:359). Development policy and programs defined and implemented by organizations and institutions 1 at different geographic levels (local, national, regional or global) need healthy economic approaches to set their goals and targets, while management is crucial to ensure that development policy preparation, design, implementation, and evaluation are well performed in ways that increase organizational and institutional capabilities to reach the desired outcomes and goals. Organizations and institutions in charge of development management have to set and use management tools and approaches in designing and implementing development policies and programs. These range from the definition and elaboration of development policies and programs, to the mobilization, allocation and management of financial, human and technical resources, the timely implementation of activities and delivery of development programs products and services, the distribution of development programs products and services in the areas of intervention and among beneficiaries, the responsibility to ensure the 1 Organizations and institutions here are the entities in charge of development actions, while institutions are the rules that interline the roles of organizations to achieve their mandate. In many cases institution is used is way it includes organization concept. 16

25 sustainability of development outcomes towards the desired goals of improved welfare and freedom of people. To play correctly their role in managing development programs, organizations and institutions need strong capabilities in mainly three core management areas: Results-Based Management, Knowledge and Information Management, and Evidence-Based Decision-Making. To ensure proper and effective development management, organizations set forth performance management frameworks and learning platforms to track development results and ensure that activities are leading to the desired outcomes. The most popular development performance measurement framework and learning platform used by organizations and institutions that tracks development results, and creates a continuous learning and decision-making process on development actions, is called Monitoring and Evaluation (M&E) System. M&E System is: 1. A set of activities for Data collection on development results and indicators, the organization and management of high-quality data on different aspects of the development actions linked to the programs and projects theory of change, 2. The scrutinizing and analysis of the data towards the creation of meaningful Information on development achievements and results, 3. The scientific use of the information in learning processes towards the generation of actionable Knowledge to improve development policies and programs success, and 4. The tailored delivery of the knowledge to the Wisdom of development Managers for improved organizational and institutional capability to advance development objectives through effective evidence-based decision-making and measured risk-taking. To understand the so-called "incapacity" of institutions and organizations to move development actions to success as pointed out by North, in many cases and contexts, especially in the West Africa region, different explanations are postulated that stress the weaknesses in the organizations and institutions capabilities to achieve development actions and results. Top down policy design approaches and neoclassical economic productivity models heavily used are highlighted as part of the first explanation of unsuccessful achievement of development programs results mainly in developing countries. Difficulty in timely mobilizing the needed resources is also another causal factor underlined to explain this difficulty to advance development objectives. Billions of US dollars were spent to design and implement development strategies and policies in Africa and massive amounts of funding were invested in development programs, including foreign assistance efforts (Table 1), with little success as highlighted by the United Nations in the 2014 world development status report, the African continent is still facing several development challenges from food insecurity to low integration of the continent to the global economy (United Nations, 2014:2). 17

26 Table 1. Gross official development assistance disbursements to Africa by Sector (US$ Millions), (United Nations, 2014:70) Challenges in achieving development objectives and results are not only due to factors related to resource scarcity or poor economic approaches, there are other socio-economic and managerial dimensions to be considered, as Claire Nelson stressed it: The lack of teleological effectiveness in much of the development project and policy prescriptions is in large part because of our inability to model the complexity of the socio-economic systems we attempt to address. (Nelson, 2014:476). This research looks first of all to the institutional and organizational capabilities in managing development policies and programs through effective Monitoring and Evaluation System within greater learning processes for improved evidence-based decision-making and measured risk-taking to advance development objectives. The research focuses mainly on Monitoring and Evaluation System effectiveness, and how it influences dynamic capabilities of development organizations and institutions in managing development programs and projects towards the desired outcomes and expected impacts on people s livelihood and welfare. The Chapter 2 (Purpose and aim of the research), presents the research field, the challenges in seeking effective M&E Systems, the gaps, and limits in assessing M&E Systems, and finally the research purpose and aim. As highlighted in Chapter 2, the primary field of the research is Development Management and Performance Assessment Systems, as far as Development Management is concerned first of all with the analysis of institutions and organizations roles in the process of development (Faguet, 2011). Few authors refer to M&E System as a comprehensive Information System that improves management practices in the development field through better learning and knowledge management, improved resultsbased management, and greater evidence-based decision-making. The merely viewing of M&E System as a simple data collection and management technique to inform development programs results frameworks, is quite problematic, as it makes it difficult to have an agreed upon meaningful scientific measurement framework for effective M&E System. 18

27 This research would not be possible through only experimental approaches mainly used in classical theses, or purely empirical research methods in social sciences. Thanks to the Doctorate of Business Administration (DBA) approach that combines academic scientific knowledge creation process with the experience of professionals and management concerns, it was possible to seek scientifically from the experience of the researcher, how to build a meaningful measurement framework that robustly assesses the M&E System effectiveness as a critical development management approach that could significantly impact on the capabilities of institutions and organizations to advance development objectives. The experiences capitalized in the West Africa region in different areas and contexts helped understand the importance of M&E Systems in development management in the context of under-developed countries. The West Africa region is one of the regions in the world facing the most critical development issues (United Nations, 2016), and is the region where we performed most of our development management experience the last two decades; it was selected as the area of research in this DBA thesis. In West Africa, development institutions and organizations are facing critical gaps in designing effective M&E Systems largely because of weak capacity and a dearth of resources for the building and operating of those systems. Also, and most important, there is a growing confusion and a lack of clarity on what would be the objective and framework for effective M&E Systems. Hence, development managers and practitioners in the region are increasingly vulnerable and exposed to different conflicting ineffective M&E approaches or guidelines sponsored by development gurus and prophets as simple data collection and management tools to respond to their sponsored development programs results frameworks and indicators needs. Because of lack of a meaningful M&E System framework, many experts, practitioners, and stakeholders are involved at some level of M&E Systems without even knowing clearly their roles and why they collect or assemble data or write cases and reports using put-forward formats and guidelines they usually don't understand. There are robust top-down approaches exercising intense coercion on the local research of effective ways to improve M&E Systems effectiveness to advance development objectives in the West Africa region, and under-developed countries in general, through improved managerial capabilities in learning, results-based management, knowledge and information management and evidence-based decisionmaking. As per the Business Science Institute guidelines for the research conduct approach, the objective of this research is to wisely enter into a process that will end with the writing of a meaningful quality thesis through the use of experiences practiced in the West Africa region and the utilization of strong conceptual knowledge creation process as guided by scientific approaches in management sciences research promoted in the Business Science Institute program. For the author, almost twenty years of experience in designing and piloting M&E Systems in the West Africa for national and international development organizations in 19

28 different contexts grounded the need to reveal this field as it relates to development management and institutional capability to advance development objectives. These experiences were fine-tuned through an in-depth scientific process and a large selection and review of literature in development policy designs, program and project management, M&E Systems, results-based management, knowledge and information management, evidence-based decision-making, research approaches in management, the various information and knowledge transmitted by the large high caliber scholars, and faculty members leveraged by the Business Science Institute and IAE-Lyon3 University during the last three years, the strategic insights from Pr. Paul Beaulieu, the supervisor of this research activity, helped significantly equip the researcher with the needed tools and approaches to perform this research agenda. While this research intends to contribute to the building of knowledge base in the field of development management and performance measurement frameworks, the key outcomes are expected to contribute to the development organizations and institutions capability to better perform their development agenda for the greater welfare of the people in the region. This research will not be successful if the outcomes do not provide actionable knowledge to development practitioners and managers through concrete, meaningful and realistic way forward insights. In short, the research rationale is trifold: 1. Personal, as it is originated from the researcher willing and enthusiasm to build on his experience and acquire higher education level at the same time, 2. Academic, as it is meant to be a scientific contribution to the critical knowledge building in development management that scholars can use and improve, 3. Managerial, most important, as the research intends to contribute to the construction of improved capability of development organizations and institutions to better move development programs and actions to success through a greater understanding of the requirements for effective M&E System framework. This research is a contribution to the building of knowledge base on the effectiveness of Monitoring and Evaluation System in the field of Development Management. The primary purpose of this research is to contribute to the clarification of what rigorous framework would apply to the understanding and meaningful measurement of Monitoring and Evaluation System effectiveness. 20

29 CHAPTER 2. PURPOSE AND AIM OF THE RESEARCH The main objective of the chapter is to present the research purpose and aim. This chapter describes the research field and clarifies that it pertains to the field of Development Management. While development involves economic approaches, which are relevant to all socioeconomic fields, it is mainly concerned with management issues as it is planned, implemented and assessed by development organizations and institutions. The main objective of the first section of this chapter is to clarify the field of the research and how it links to the main research subject. The second section of the chapter describes the challenges in seeking effective Monitoring and Evaluation Systems. It aims at showing that effective M&E Systems are yet in place in the development field mainly in developing countries. It seeks to justify the need for working on effective M&E System framework to improve this status. The third section seeks to highlight the gaps in assessing effective M&E System. The inexistence of an agreed upon robust and scientifically built M&E System effectiveness framework might hinder good assessment of effective M&E System. The third section clarifies the gaps and underlines the need to build the framework for this research. The last section presents subsequently the research purpose and aim. It announces and explains the main research question and its sub-questions. The sub-questions are specific aspects of the main question that progressively will help respond to the main research question. The objective of the last section is to clarify the main question and its sub-questions and the general measurements to consider that clarify the research approach and methods to be used and why THE RESEARCH FIELD This section clarifies how working on M&E System effectiveness pertains first to development management and broadly to the Management field. It clarifies that development management approaches including M&E Systems fall under the management practices and theories as used by organizations and institutions in the development field which go beyond only economic sciences considerations. The main field of this research is Development Management and specifically Performance Assessment Systems for development policies and programs directly or indirectly designed and implemented by organizations and institutions such as Governments, development partners, and other stakeholders 2. Faguet clarified here the concept of Development Management: Development management is concerned first of all with analyzing the role of 2 This includes the local organizations, beneficiaries based associations, the private sector involved in the program, the nongovernmental organizations and other civil society organizations interested with the program, financial institutions, other development partners involved in the technical or financial support to the program. 21

30 institutions and organizations in the process of development Development management is about actively fomenting institutional and organizational changes that increase human freedom and wealth, and hence the level of development. (Faguet, 2011:4). Governments and other development organizations and institutions have increasing responsibility in their role to advancing development objectives. The Paris Declaration 3 on development aid effectiveness in 2005 carried developing countries Governments back to the heart of Results-Based-Management-RBM of development policies and programs in ensuring greater commitment to better alignment, harmonization and appropriation of development actions and various strategies for implementation. The author spent almost twenty years in the development field in West Africa countries, working on Monitoring and Evaluation Systems and development programs management, both at regional and national levels. There are many learnings capitalized on how development programs are performing in West Africa and how development results are being achieved. Working in the building and operationalizing of Monitoring and Evaluation Systems in the West Africa region is an exciting experience. We supported several Governments, Organizations and development partners in this region in the design, implementation, and assessment of Monitoring and Evaluation Systems. From the early 2000s to current, we spent significant time and resources in learning on how development programs Monitoring and Evaluation System would succeed. While there is much information on M&E Systems in various professional documents and reports that are produced or sponsored by international development institutions and organizations working in the West Africa region, there are few scholar, academic and scientific productions, and publications in the Monitoring and Evaluation System area in West Africa. Our objective is to turn our experience into knowledge and share it (Business Science Institute, - In doing so, we will use the academic, scientific approach built on shared experience through knowledge creation, and the research platform in Management Science provided in the Business Science Institute-IAE-Lyon-UTS 4 Doctorate in Business Administration program. The primary field of this research as described is Development Management (See 3.1. Development Management, and the Learning and Decision-Making Processes), and Performance Assessment Systems (See 3.2. Development Policies and Performance Management Systems) linked to Knowledge and 3 The Paris Declaration, endorsed in 2005, commits international development aid donors and recipients to act in accordance with five principles: ownership, alignment, harmonization, results, and mutual accountability. This landmark international agreement was the culmination of several decades of attempts to improve the quality of aid and its impact on development. (Dabelstein & Patton, 2005:19)

31 Information Management, Evidence-Based Decision-Making to advance development objectives. While this research touches Management Science fields such as Performance Management, Knowledge and Information Management, and Evidence-Based Decision Making in development programs and projects, it remains originated first from field experience and practitioner view and is willing to creating and sharing rigorously established knowledge base information that would be useful for managers, decision-makers and development practitioners for improved measured risk-taking in managing development resources towards higher quality livelihood and greater welfare of the people CHALLENGES IN SEEKING EFFECTIVE MONITORING AND EVALUATION SYSTEMS In the M&E-System literature, few authors refer to Monitoring and Evaluation System as a comprehensive Information System. Effective M&E-System needs sound and reliable information for decision makers to generate and implement policies and actions through learning, knowledge base and evidence-based decision-making processes. Development programs and projects are designed and implemented to achieve greater freedom and improved welfare of the people mainly in developing countries. At different levels, growth targets are defined as well as strategies, policies, and actions are prepared and implemented to support and operationalize the development objectives. Specifically, at the country level, Governments are elected by the citizens based on their visions and goals to foster development. Designing and implementing development strategies and policies at regional, national or local levels for economic growth and the population well-being is an essential planning and learning process. An M&E-System is not a purely formal Information System while in many cases, Information Technology is used to set Monitoring and Evaluation Information Systems. As explained in next sections, Monitoring and Evaluation Systems are Information Systems outlined in the area of Development Management for improved decision-making and measured risk-taking to advance development objectives. Hence, M&E-System should be able to provide evidence-based information for program managers and stakeholders in their role of making decisions to achieve development goals. M&E-System should then be a mechanism or system that creates knowledge from information on the development programs implementation, achievements and results for ongoing and forward-looking decision-making. Mackay highlighted the significance of M&E-System in development performance measurement (World B. Mackay, 2010a). He established that effective M&E-System: can measure the performance of all government policies, programs, and project. It can identify what works, what does not and the reason why Additionally, it provides information on the performance of donors who support the work of the government. (Mackay, 2010:170). Clear development strategies and policies are necessary for 23

32 understanding and achieving the objectives and results, targets and milestones at the central level of regional institutions, as well as at national government, sectoral and local levels. Also, well-articulated programs and projects to implement these regional and national policies and strategies are key in achieving development objectives. As highlighted here by Segone: Many governments and organizations are moving from opinion-based policy towards evidence-based policy, and are in the stage of evidence-influenced policy (Segone, 2010a). In accelerating and deepening the implementation of the Paris Declaration on Aid Effectiveness, African countries committed to improve results-based management practices and to demonstrate that development actions translate into positive impacts on people s lives. An abundant literature has explained the relation between effective M&E-System and the clarity of the Strategic Intent at regional, national, local and sectoral levels (Quesnel, 2010). Quesnel recalled in 2010 that the very first requirement of any country-led M&E System is the strategic intent or objective of the development program with its logic and performances measures (Quesnel, 2010:72). This aspect is not a new finding in the literature, as, in the early 90s, Cracknell pointed out, in analyzing M&E-System of public investments in the United Kingdom, that the most serious difficulty in designing effective M&E System is the problem in clearly stating the programs objectives (Cracknell, 1994:227). In assessing M&E-Systems effectiveness worldwide in 2007, the World Bank group found few success cases. In Africa including Uganda (the only African World Bank champion in Monitoring and Evaluation in the early 2000s), effective M&E-Systems are yet to be found (Mackay, 2007:50). In the early 1990s, Maddock launched in the field of development and aid industry a thought-provoking question on the effectiveness of M&E-System: Has project monitoring and evaluation worked? (Maddock, 1993). Initially developed, according to a broad consensus (not exact), as a simple monitoring approach at beneficiaries level and promoted in the World Bank rural development projects in developing countries, Monitoring and Evaluation primary objective was to provide to researchers and implementers with data trends findings to assess and address field projects implementation pitfalls in the 1960s and 70s while taking its roots in the International Monetary Fund and World Bank perception of public-sector failure dating back to 1950s (J.Cameron, 1993:92). M&E has become progressively in the 1980s a development performance assessment system for managers, donors, NGOs and Government counterparts. Currently, in the increasing need for accountability, there is a growing attentiveness for evidence-based decision-making and measured risk-taking in Development Management for donors and developing countries governments that bring back effective M&E-Systems at the crossings of Aid-effectiveness concerns through the Results Based Management approach (Dabelstein & Patton, 2005). 24

33 Unachievable objectives, weak reporting and lack of timeliness among other factors had been identified as primary constraints that hinder Monitoring and Evaluation Systems effectiveness (Maddock, 1993) since its introduction and development in the development field and Aid-Industry in the 1970s and 1980s. In the 1990s, Barton in a view of an integrated Information System analysis, identified six weaknesses that undermine the effectiveness of Monitoring and Evaluation Systems: poor development program design, human resources development needs, quantitative bias, low priority for information system, involvement limited to data collection and poor feedback (Barton, 1997). If, The aim is to improve relevance, efficiency and effectiveness of policy reforms. Why then is monitoring and evaluation not playing its role to its full potential? asked Segone in 2008 (Segone, 2008:26). Then, the Maddock s question on Monitoring and Evaluation Systems effectiveness is yet to be addressed by both development practitioners and researchers in Development Management for improved decision-making in the area of Development Management and development Performance Measurement Systems GAPS AND LIMITS IN ASSESSING MONITORING AND EVALUATION SYSTEM EFFECTIVENESS While an abundant documentation and professional technical reports, as well as few publications, exist on how to build Monitoring and Evaluation System as well as evaluation findings on existing systems and their weaknesses, there are critical gaps in the literature on how to model and measure effective M&E-System at development programs level and how to foster organizations and institutions demand, service and use of quality Monitoring and Evaluation information for improved decision-making and measured risk-taking. Effective Monitoring and Evaluation System is core in improving development performance and the capacity to deliver high-quality development results. Mackay described four key benefits of an effective Monitoring and Evaluation System at country level: 1- to support policy making, 2- to improve policy analysis work, 3- to foster Ministries sector management, 4- to enhance transparency and support accountability (K. Mackay, 2010a). As such, the introduction of M&E-Systems in the developing world has increasingly improved accountability and Results-Based Management practices in the public sector. Nevertheless, Monitoring and Evaluation Systems are yet to be effective in development management field mainly in Africa to better support organizations and institutions capability to advance development. About evidence-based policymaking at the country level, Segone characterized the two sides of a Monitoring and Evaluation System: the demand and the supply sides. Abundant documentation is showing that donors are more frequent on the demand side than Governments (DAC, OECD). That is one of the factors that hinder country governments and regional institutions leadership in the building of effective M&E-System (Segone, 2010a:23-24). In How to build M&E-Systems to Support Better Governance?, 25

34 Mackay described key factors of success for an effective Monitoring and Evaluation System: 1- utilization of Monitoring and Evaluation information, 2- good quality of Monitoring and Evaluation information and, 3- sustainability of the system (K. Mackay, 2007). How to build and assess effective Monitoring and Evaluation System is a critical area of research to expand the use of better learning for improved decisionmaking to advance development policies and strategies. There are yet existing rigorous scientific and managerial standards in assessing Monitoring and Evaluation System effectiveness. The little assessments made so far used experts and consultants oriented insights in their approaches. Thus professional biases might influence the quality of the results obtained. This lack of evidence in the measurement of Monitoring and Evaluation System effectiveness may lead to greater confusion and weak successes in assessing Monitoring and Evaluation of development results and their achievements, primarily in Africa and low-income countries in general, hence endorsing the wastage of resources in non-successful learning systems that may lead to increases in confusion and inefficiencies in development management. In looking at the Monitoring and Evaluation System literature, data are showing that little cases have effectively worked, but limited evidence is showing why these systems have worked or not, or what are the evidence-based measurements underlying the conclusions on their effectiveness. This situation made possible the existence of a strong influence in the development field of Monitoring and Evaluation gurus and organizations that provide to the Monitoring and Evaluation Systems which follow their approaches or being implemented by their members positive rewards for to be considered effective Monitoring and Evaluation Systems by the public and beneficiaries of their supports, even though the conclusions on their effectiveness are generally objected by other development practitioners and scholars because of lack of evidence. The lack of evidence in assessing Monitoring and Evaluation Systems in the development field is mainly due to the absence of an underlying effective acceptable performance management framework built upon within a robust scientific approach to generate quality measurements and data that, when analyzed, the findings could lead to a rigorous and meaningful appreciation of whether systems in place are effective or not. The primary challenge is that the focus is put on whether the Monitoring and Evaluation Systems are operating and providing selected information as per the results frameworks defined by their sponsors, instead of looking at the broad picture. If there is no evidence and research findings on what would be the goal of a Monitoring and Evaluation System in the broad development agenda, there will be little chance to design a best rigorous way to assess Monitoring and Evaluation Systems effectiveness. There is abundant production of reports by various consultants and development practitioners as well as few scholars on what would be the role and operations 26

35 of a Monitoring and Evaluation System as a tool or technique, but little is clarifying the primary objective of a Monitoring and Evaluation System in development policy and programs management and what would be its impact in the process of attainment of development goals by organizations and institutions. Hence, assessing Monitoring and Evaluation Systems mainly on their technical role and functioning only, how they operate, the quality of data collected, will certainly not clearly unfold if higher objectives of Monitoring and Evaluation Systems and theirs impact on the decision-making process to advance development goals are leading to the desired outcomes. There is then a need to build a scientifically acceptable framework to assess Monitoring and Evaluation Systems against their higher objectives to support development achievements and results as to refer to what Charles Booth revealed in the 19 th century (1886): To effectively deal with poverty there was need to gather quantitative information on characteristic of poverty, including statistics on number of poor people, causes of poverty and measures to alleviate it. (Booth cited by Gelase, 2011:13). Monitoring and Evaluation System contributes to the effective attainment of development goals. How to measure that contribution is an important question yet to be addressed explicitly and adequately by development scholars and practitioners. Another challenge is that, while Monitoring and Evaluation Systems are defined and implemented by organizations and institutions that are in charge of the design and implementation of development actions, there is no framework clear enough on what would be an effective Monitoring and Evaluation System within those institutions and organizations, and what would be the effects on their Development Management capabilities. Many authors refer to effective Monitoring and Evaluation System as being a performant quality data collection system or a data analysis and presentation system. These frameworks look mainly to how Monitoring and Evaluation Systems work and operate and do not clearly capture what is an effective Monitoring and Evaluation System within organizations and institutions in charge of managing development actions. The Monitoring and Evaluation System effectiveness would then be analyzed in line with the internal and external institutional transformations of the organizations towards the higher capability to ensure proper evidence-based decision-making processes. Monitoring and Evaluation System effectiveness would then be better assessed through a Dynamic Capability outlook on organizations and institutions in charge of development policies and programs for greater impacts on people s improved livelihood and welfare through effective Development Management as a learning strategy. As to clarify that Monitoring and Evaluation System is part of the general organization strategy to achieve development objectives, as clarified here by Beaulieu: strategy is understood as the knowledge, determination, organization, decision and action processes of an entity for and into its environment. From the individuals, up to large collective 27

36 organizational entities finalized and coordinated, human actions constituted, through history the main object of the process dynamics that modern times named management. This refers to the general process of transforming ideas into objective realities. (Beaulieu, 2015:2). There is vibrant need to define a rigorous acceptable framework for the measurement of Monitoring and Evaluation System effectiveness that captures the dynamic institutional transformation of capabilities within the organizations in charge of development policies and programs, on aspects directly linked to a greater results-based approach, evidence-based decision-making and measured risk-taking. There is a critical need in the development field to move from the viewing of Monitoring and Evaluation System as a set of tools and techniques, to the considering it as a broader Development Management system that effectiveness would be found on the dynamic capabilities transformation of the organizations towards greater development impacts delivery as the outcome of investing resources in Monitoring and Evaluation Systems in a purely market-driven approach and Development Management framework. Even if there are data in the literature on how to describe the way Monitoring and Evaluation Systems are functioning and delivering their products and services, there are critical gaps in identifying what measurements and criteria were used to assess those aspects of Monitoring and Evaluation System. There are yet rigorous approaches based on clear Information System framework to measure the effectiveness of each component of the Monitoring and Evaluation System with a strong distinction between the Information Quality, the System Quality and Service Quality components of such information system (DeLone & McLean, 2003). In searching for relevant measurements on those aspects, little evidence is available to make robust judgements on the Monitoring and Evaluation System effectiveness as an information system, as to referring to Barton s definition: The M&E system is a form of information system, which is a broad term for information selection, gathering, analysis, and use. It can be described as a logical chain of linked ideas starting (and continuing) with information users. (Barton, 1997:10). There is need then to find ways to define clear criteria and measurements, approaches that would apply for successfully and rigorously assessing the different dimensions of a Monitoring and Evaluation System as being considered as a form of an information system for improved Development Management. In short, there are critical gaps in defining and measuring what would be the objective of a Monitoring and Evaluation System as being a management system that contributes to the smooth attainment of organizations and institutions goals in the development field in a purely market-driven approach through improved development management capabilities. In the same trace, there is need to define and clarify what is an effective Monitoring and Evaluation System framework in a view to assessing appropriate organizational and institutional capabilities linked to the greater evidence-based decision-making process and measured risk-taking in Development Management 28

37 field. Finally, there is need to define core components of Monitoring and Evaluation System as we refer to Barton s definition of such system as a form of Information System. Creating and validating a clear set of consistent, rigorously acceptable measurements and approaches in each of these dimensions of Monitoring and Evaluation System would greatly enhance the body of knowledge in this area and increase the quality of practices in assessing development programs Monitoring and Evaluation Systems. In short, Maddock s question (Maddock, 1993) on Monitoring and Evaluation Systems effectiveness is yet to be addressed by both development practitioners and researchers in Development Management as M&E Systems, including in the West Africa region, have shown little effectiveness. There are no existing rigorous scientific and managerial standards in assessing Monitoring and Evaluation System effectiveness, and the lack of evidence in assessing Monitoring and Evaluation Systems is mainly due to the absence of an effective, acceptable performance management framework with a robust approach to generate quality measurements and data. Because there is no evidence and research findings on what would be the goal of a Monitoring and Evaluation System in the development agenda, there is little chance to design best rigorous ways to assess Monitoring and Evaluation Systems effectiveness. Hence, there is a need to build a scientifically acceptable framework to assess Monitoring and Evaluation Systems against their higher objectives to support development achievements and results. As we argue here, there is a critical need in the development field to move from the viewing of Monitoring and Evaluation System as a set of tools and techniques for data collection, analysis and presentation in sponsored results-frameworks, to the considering it as a broader Development Management system that effectiveness would be found on the dynamic transformation of the organizations and institutions management capabilities towards greater development impacts delivery. The creation and validation of a clear set of consistent, rigorously acceptable measurements of Monitoring and Evaluation System effectiveness framework and dimensions, would greatly enhance the body of knowledge in this area and increase the quality of practices in designing, implementing assessing development programs Monitoring and Evaluation Systems THE RESEARCH PURPOSE AND QUESTIONS This research is a contribution to the building of knowledge base on the effectiveness of Monitoring and Evaluation System in the field of Development Management. The primary purpose of this research is to contribute to the clarification of what rigorous framework would apply to the understanding and meaningful measurement of Monitoring and Evaluation System effectiveness. Such framework would help to improve the learning and knowledge management for improved evidence-based decision-making in Development Management. This research intends to contribute to the scientific and the professional discussions on what 29

38 would be the rigorously acceptable meaningful framework to understand better the success factors underlying an effective M&E-System and how they unfold in a dynamic capability outlook for improved Development Management. This research intends to be the right rally of the experienced practitioner and the scientific researcher in Development Management to contribute to the building of knowledge in monitoring and evaluation field both at an academic level in Development Performance Management, and development practices at Programs, Organizations, Institutions, States and Regional levels. Combining these two categories of needs and questionings would be the strength of any probable success in working on How to measure the effectiveness of Monitoring and Evaluation System for improved evidence-based decision-making and measured risk-taking to advance development objectives? The case of West Africa, which is the proposed subject of this research. In line with Barton s definition of Monitoring and Evaluation System (Barton, 1997) as being an Information System, the research wishes to look to what is the most suitable existing Information System success models that might be aligned with the institutional and organizational capability in learning, evidence-based decision-making and measured risk-taking to advance development objectives. In looking to its aim, the research will analyze what would be the outcomes of effective M&E-System on the capability of institutions and organizations to advance development. Is there a direct link between M&E-System effectiveness and development results achievement? How to describe the relationship and how to explain the characteristics of an effective M&E-System that contribute to the achievement of development results? M&E-system is generally seen as a tool that measures the achievement of development results. This research is looking at M&E system as a function of Development Management strategy for improved decisionmaking to advance development objectives by institutions and organizations in charge. In analyzing M&E-System as an Information System and building on the tryptic System-Information- Service, one of the purpose of this research is to look critically and rigorously at the practices and experiences in the West Africa region built by existing organizations such as governments and regional bodies (West Africa is an interesting case as most of the development partners operate in the region, it also is the area where countries are facing the most difficult development challenges including food insecurity, weak health, and education systems, low-developed physical infrastructure and weak energy industry, etc.), and generate and test a model of what would be the measurements relevant to studying the triple aspects of the Information System, namely System, Information, and Service. Another objective is, not only analyzing M&E system as a tool for assessing of development results achievement and performance, but to look also critically at what are the institutional and organizations cultures and practices that might be improved by an existing effective M&E System as a dynamic capability. What are the managerial and performance tools 30

39 and behaviors that can enhance internally and externally and foster organizations institutional capacity to advance development objectives through effective M&E System? In addressing this research agenda, the central research question is: How to measure the effectiveness of development programs Monitoring and Evaluation Systems for improved evidence-based decision-making and measured risk-taking to advance development in the context of West Africa? How to contribute to the improvement of Development Management scholars and practitioners knowledge and view of effective Monitoring and Evaluation System measurement through this research by using agreed upon techniques and approaches in Management research, the knowing, the rationale of development programs evidence-based decision-making processes within effective Monitoring and Evaluation Systems, their broad objectives, and the ways their effectiveness should be measured. To address this central question, the following sub-questions will be answered in this research. a. How to build a commensurate framework in studying the effectiveness of development program M&E-System as part of Development Management strategy? This sub-question is to look through literature and relevant research, how to translate existing and new knowledge into a commensurate model that will serve the primary hypotheses to measure M&E System effectiveness. In analyzing the history and literature of M&E System, Development Management, and Information System theories, there might be the already existing knowledge that could guide in the building of a meaningful model for effective M&E System. This sub-question seeks to understand how an effective M&E System works and what would be the proper scientifically acceptable framework to track its effectiveness and success. The sub-question can be gauged through various learning questions as follows: i. What are the key relevant lessons and knowledge from the M&E System history in development management that may be recalled, analyzed, and used in the process of building a commensurate framework for effective M&E System? ii. What are the main steps and different models used for M&E System preparation and implementation that can help gather relevant M&E approaches utilized in the past and currently by institutions and organizations in managing development actions, that can contribute to the clarifying of the gaps and needs for effective M&E System framework? iii. From the existing knowledge on information system theories, what would be relevant IS success framework that might guide the building of the M&E system effectiveness framework, and by using responses in i. and ii., how can we draw the first model of M&E System Effectiveness measurement? 31

40 The primary output of this first sub-question would be the proposed initial M&E System effectiveness framework built upon through the findings from learning points i, ii, and iii. b. What are the key components (and variables) at different levels of the Monitoring and Evaluation System effectiveness model relevant in studying M&E-System role in the effective evidence-based decision-making in advancing development objectives in West Africa? Based on related cases and experiences and literature rigorously analyzed, the question will help operationalize the main components of an M&E System Effectiveness model as a form of information system identified in the Sub-question a-. These elements and variables built from experience, literature, and cases will be considered as the probable detailed sub-units and measurements of the M&E System model identified in the first Sub-question. This question seeks to give the accurate measurements for the different dimensions of the effective M&E System established in the first sub-question, by using specific important cases that have the full potential in the process of studying the dimensions and subdimensions selected. In the practices and experiences of identified organizations and institutions that host the cases to be studied well known by the researcher, this question will help analyze how the organizations process and operationalize their M&E System to make them effective, and from there, will capture what would be the proper measurements to be generalized and used. Though, this question will be answered through cases that will be selected individually based on what particular dimension of the M&E System is to be detailed, as closely linked to the institutional steps related technically to the selected dimensions and sub-dimensions. There is need then to decide on cases that are very well known by the researcher. Hence experience and practice are critical in this step. c. What is the meaningful scientific acceptable framework to measure the effectiveness of Monitoring and Evaluation System, through the operationalization and testing of the first proposed model? The comprehensive framework, its components, and variables generated as tangible hypotheses in the previous sub-questions will be tested under this research Sub-question c. Professionals and development programs operating in the West Africa region will be asked to provide feedback on the proposed model through a scientific approach. The analysis of the feedbacks will help review and finalize a rigorous acceptable framework to assess M&E-System effectiveness. The proposed framework built from findings in the first sub-question, and detailed with variables at different levels and meaningful measurement from results on the second sub-question, will be materialized through a detailed questionnaire with scientific measurements of each dimension and sub-dimension of the M&E System effectiveness framework that will be assessed with professionals in development with selected development programs that touch the most important development sectors in the region, at country and regional levels. Data will be analyzed using scientifically 32

41 statistical approaches to validate the relations in the framework, gauge relationships of the different sub dimensions. The aim of this question at the end is to find the meaningful, scientifically acceptable M&E System effectiveness framework that relationships and correlations between various dimensions and sub-dimensions are quantified, analyzed and explained for the future meaningful use of the model in academic and in development management practices. This sub-question will be gauged through the following learning questions. i. What are the robust sub-components and variables of the different dimensions of effective M&E System that are independently relevant to consider in the measurement of the quality of the dimension? In considering the sub-dimensions in each component of the model from the literature review and case studies, there might be overlaps and correlation between the variables and subcomponents proposed from the findings in sub-questions 1 and 2. Statistical approaches can help identify the overlaps and create a set of meaningful independent sub-dimensions and variables that will be proposed in the final framework. ii. What are the different quantifiable relationships between the dimensions of the effective M&E System? What is the particular function (relation) between the M&E System effectiveness dimensions and the other aspects of the M&E System framework as a tool that can help measure and predict at organization and institutional levels the needed characteristics that might influence the M&E System effectiveness and then the capabilities to advance development through better policy-making and evidence-based decision-making? The responses in this sub-question will determine the final M&E System effectiveness framework cleaned with probable overlapping in the measurements proposed. That will finally help set the final model for M&E System effectiveness measurement which is the main output of this research. The findings of this research are priorly addressed to decision-makers including State Governments and Regional Organizations, Donors, NGOs, other stakeholders, and to development practitioners in a learning process. They also would be addressed to the scientific community in the building of a comprehensive knowledge and science base in Monitoring and Evaluation. This research is also designed to contribute to the development of an evaluation approach of programs M&E-System effectiveness. Also, the result through a revised Monitoring and Evaluation Systems effectiveness and capacity assessment approach will help to better use investments in regional and national development strategies and policies. 33

42 CHAPTER 3. DEVELOPMENT MANAGEMENT AND PERFORMANCE MEASUREMENT SYSTEMS: A LITERATURE REVIEW This chapter clarifies and defines the key concepts utilized in this research as they pertain to the development field. It clarifies first the notion of development and how it links to development management through the organizations and institutions capabilities in designing, implementing and assessing development actions and results. It also clarifies how these concepts translate into the practice in West Africa region and presents the contexts in which they operate. The chapter introduces and analyzes the most relevant existing literature that explores Development Management and Monitoring and Evaluation (M&E) System for improved Results-Based Management (RBM), Knowledge and Information Management (KIM) and Evidenced- Based Decision-Making (EBDM). As we argue here, Development approach as a purely market-drive strategy for improved welfare of people is not limited only to economics approaches. Instead, it includes Performance Management with structured actions and coordination among organizations and institutions responsible for designing, governing and implementing development policies and programs. Gauthier examined and highlighted the role of institutions in development theory to improve organizations capability to advance development objectives (Gauthier, 2005:33-34). Development, as defined by Sen, is the expansion of freedoms that people enjoy for greater opportunities and improved welfare (Sen, 1999). As such, there is need to build effective frameworks that guide and coordinate development actions and programs. This section explores development policy and programs at every level that shows the effective need for coordination: local, national and regional. From the early cold-war to the recent period, a systematic analysis of how development theories and programs unfolded helped shape the linkages between development performance and the knowledge, the learning and the evidence-based decision-making processes to advance development objectives DEVELOPMENT MANAGEMENT, AND THE LEARNING AND DECISION-MAKING PROCESSES While the laisser-faire approach to economic development assumes that change will take place without structured action and coordination, Development Management adopts a different approach to counter the potential inefficiencies of a purely market-driven strategy. As such, according to Faguet Development Management is, instead, the deliberate, conscious and informed attempt to move institutions and organizations towards higher levels of efficiency and effectiveness in a way that is faster than an 34

43 autonomous rate of progressive change. (Faguet, 2011:22). Moreover, at its heart, Development Management is predicated on understanding and managing the rules of the game within and across organizations and institutions that are responsible for development policies and actions as well as their dynamic relationships. What is development? Nobel Economics Laureate Amartya Sen defines development as an expansion of real freedoms that people enjoy. He argues that Development requires the removal of major sources of unfreedom: poverty as well as tyranny, poor economic opportunities as well as systematic social deprivation, neglect of public facilities as well as intolerance or over-activity of repressive States. (Sen, 1999:3). In considering the relationship of development to economic growth, Davis lists three dimensions for an effective measurement of development from a macro economic view largely criticized by scholars: economic, human and social. Increases in real per capita income are a standard measure of development. Yet only through a decomposition of structural changes can we gain a fuller appreciation of how development takes place. Development of a country s economic system needs to be measured through: a number of structural changes, including urbanization, the rise in the size of firms, the relative decline of the agricultural sector in terms of employment and output with expansion of manufacturing and services, the geographic expansion of markets, and increases in the diversity of goods produced and traded. (L.S.Davis, 2011:3). The Distribution of the gains from economic growth is the second dimension of economic development. That can be measured by tracing the growth of the income of the poorest 20 to 40 percent of the population. That is also known as poverty weighted index (PWI) as the proposed measure. Development economists often have argued that growth is inherently biased against the deprived: the share of the variance of growth in social welfare due to growth in average incomes is smaller the greater is the weight that the social welfare function places on the poorest. (Dollar, Kleineberg, & Kraay, 2014:21). The third element of economic development is sustainability, defined by the UN (UN, 1987:12) as a development that meets the needs of the present generation without compromising the ability of future generations to meet their own needs. Its measurement requires adjusting income levels to account for the depletion of natural resources and degradation of environmental quality. (L.S.Davis, 2011:278). The human development dimension of development is linked to the capacity of the individual to satisfy their needs and fulfilled lives. Freedoms, as defined by the United Nations Development Programme can be measured through the use of a Human Development Index (HDI) that apportions weights to three dimensions of development that include measuring income, health, educational attainment, and gender 35

44 equality. But economic growth, as argued by Mahbub is not sufficient itself to achieve economic development (Haq, 2003:19-20). Development is not just about economic growth and economics models. Instead, it is a process guided by institutions, organizations, and individuals. As such, if any development policy is to succeed, it must be gauged by an assessment and improvement of linkages and operations across social, economic, and political agencies (LeBel, 2011). Development Management goes beyond understanding and analysis of development indicators to address the basis of active management of organizations and institutions in a process that promotes increased rates of growth, development, and individual freedom (Faguet, 2011). As such, we need a framework that understands the capacity of institutions, organizations, and individuals to adopt measures that promote efficiency and effectiveness in the learning and the decision-making processes. In short, institutions, organizations and individual matter, and only by taking them into closer consideration will development proceed along lines that result in successful outcomes. Many of the development scholars argued for a focus on the capability approach that goes beyond economic growth, to focus on individual and groups including organizations and institutions capabilities (Alkire, 2007a, 2007b; McNeill, 2007; Ranis, 2004; Ranis, Stewart, & Ramires, 2000; Sen, 1999; Stewart, 2004, 2013). Managing individual and groups capability development is key to any development policy. The most important for development management, decision-making and learnings in development actions is the capacity of the evaluation system to assess individual and groups capabilities to seize growth opportunities and translate them into well-being and people s welfare in a sustainable manner. Even with the kinds of adjustments, we note here, much of the development literature overlooks roles of development institutions, organizations, individuals, and management, even though they are highly elaborated in macro and micro economic theories and strategies. As we argue here, institutions are key components for development effectiveness. Some countries are rich while many others are poor not from a lack of resources or absence of economic growth, but from poorly governed institutions and counterproductive incentives. Institutions, The rules of the game according to North, are the underlying forces that govern economic, social and political matters (Douglass C North, 1991:98). Leftwich et Sen qualified two types of institution, formal (written) and informal (usually unwritten) and clarified that effective institutions are key in in shaping social, economic and political life. (Leftwich & Sen, 2010:13). While Institutions are the drivers of rules, Douglas C. North defines an operational and regulatory framework that includes individuals, households, local governments, government departments, regional bodies, that operates as a game. He notes that rules, both written and unwritten, derive from organizations that set the standards of performance of any given game, be that for a sporting context or for the allocation of scarce resources. Like teams of people, organizations operate within an institutional environment they 36

45 can also modify to achieve a certain number of goals and a certain level of performance (Faguet, 2011 and Leftwich & Sen, 2010). And any given organization, whether a firm, a government department, or a household, functions under a set of incentives created by multiple institutions, all of which affect their behaviors. (Faguet, 2011:4). As we argue here, Development Management requires an examination, monitoring and evaluation (M&E) of institutional change and organizational behavior that can affect the achievement and sustainability of development objectives. As such, institutions design, governance and their Monitoring and Evaluation System are key to development change. According to North, Institutions provide the incentive structure of an economy; as that structure evolves, it shapes the direction of economic change towards growth, stagnation, or decline. (Douglass C North, 1991:97). Development policy design necessarily starts with an understanding of institutions underlying policies and politics in each context. It then proceeds to define a certain number of goals and targets to improve the development status of countries or other entities and organizations in place. Development Management is focused on an analysis of institutional changes and organizational behaviors that may affect the attainment and sustainability of development objectives and includes the capacity of organizations to advance development objectives. Development and economic change can be explained by several parameters ranging from the quality of the policies and programs designed and implemented, the level of investment and priority areas that may add value, the efficiency and effectiveness of institutions and organizations in place, and how these instruments lead to changes in the livelihood of people. As Mantzavinos, North and Shariq put it, The greatest challenge for the social sciences is to explain change or more specifically, social, political, economic, and organizational change. (Mantzavinos, North, & Shariq, 2004:75). Development and economic change are not gained only through dreamlike economic theories or models. Instead, they occur under institutional and organizational choices. According to North: Economic change is a ubiquitous, ongoing, incremental process that is a consequence of the choices individual actors and entrepreneurs of organizations are making every day. (North, 1994:361). Decision-making and risk-taking within institutions and by organizations and people are key to any given development change. Learning and knowledge-sharing by individuals and management units within institutions and organizations affect how decisions are made in the presence of risk. Improved management of risk can accelerate development. North in Economic performance through time, established a linkage between the expansion of knowledge and the development process (North, 1994:362). 37

46 In the field of development management, learning and knowledge sharing are key elements to success. As learning and knowledge within institutions can improve the quality of decision-making process, how knowledge expands and is distributed and used for decision-making, is central to development management. Where organizations are more focused on development results in the policy-making, and the implementing processes, evidence-based decision-making, and measured risk-taking are necessary to advance development objectives through a transparent learning and knowledge sharing system or Monitoring and Evaluation (M&E) System. This practice goes beyond the components of what is called in public performance: better governance. As stated by Faguet: Institutional change can occur through the gradual accretion of small changes over time, or via sudden exogenous shocks. Both kinds of change can be either positive or negative, leading to higher or lower levels of efficiency and effectiveness. (Faguet, 2011:21) DEVELOPMENT POLICIES AND PERFORMANCE MANAGEMENT SYSTEMS In the early days of the Cold War in the 1950 s, the U.S. Government launched a foreign aid development initiative designed to thwart the expansion of Communist power in poorer areas of the world 5. The main objective was to avoid another China risk, by promoting investment in agriculture, health, education and water access in developing countries to generate increases in welfare via per growth in per capita incomes. As Alacevich states politics of productivity promoted by the US in Europe that time would help to tackle communism threat in other parts of the world (Alacevich, 2011:65). While foreign aid focused initially on the rebuilding of war-torn Europe, the expansion to developing countries took a broader view of the kinds of institutional transformations that would be needed to produce credible increases in per capita income. As the geographic focus expanded, much debating took place within development institutions such as the World Bank as to whether infrastructures building alone could produce expected increases in welfare. One evolving approach was put forth by John Williamson. What John Williamson (1990) called Washington Consensus (WC), was built around three major pillars of economic reforms in developing countries: 1) Fiscal discipline, 2) Re-ordering public expenditure priorities, and 3) Tax reform. Williamson summarized the WC as a response for a leading role of the state in initiating industrialization and import substitution. The flagship strategy of this Washington Consensus was undoubtedly the 1981 World Bank Berg report: Accelerated Development in Sub-Saharan Africa: An Agenda for Action (Berg, Amoaka, Rolf, Meerman, 5 The Bretton Woods Agreements created the IMF, the IBRD and the WB for this purpose and this was done on the basis of the original intent of the Marshall Plan which will later support the reconstruction / redevelopment of Europe. 38

47 & Tidrick, 1981). The Berg report gave a prescriptive list of structural reforms that given the team could put African countries on a sustained and accelerated path for economic growth and development. The Berg report was published in part as a response to the 1980 African Chiefs of State strategy, The Lagos Plan of Action (Organization Of African Unity, 1980), which called for greater independence from international financial institutions such as the World Bank and the International Monetary Fund. As the World Bank and IMF constituted a major source of development finance at the time, structural adjustment programs of the World Bank and conditionality loans by the IMF became the norm. The thrust of the Berg Report was a critical look at an over-expanded public sector that diluted some essential functions of government. According to Berg and his team what characterized African countries at that time, was underdeveloped human resources, political fragility, insecurely rooted and ill-suited institutions, a climate and geography hostile to development, and rapid population growth. (Berg et al., 1981:4). The WC and the Berg report fell short in bringing African countries to buy-in properly. There was no consensus on the Berg report as there was no consensus on the WC (Stiglitz, 2008a). Rather they were implemented as a necessary medicine to take by force though access to the needed funds, promoted by the WB and International Monetary Fund (IMF). As former World Bank-Chief Economist, Joseph Stiglitz put it, If there is a consensus today about what strategies are most likely to promote the development of the poorest countries in the world, it is this: there is no consensus except that the Washington Consensus did not provide the answer. Its recipes were neither necessary nor sufficient for successful growth, though each of its policies made sense for particular countries at particular times. (Stiglitz, 2008:1). Despite ongoing dissent within the ranks of development institutions and within developing countries themselves, some effort was made to paint a picture of success through structural reform (LeBel, 1999). The early 60s (1962), the called development decade, the UN Secretary General U. Thant described development as growth plus structural change. From that period to the Millennium Development Summit in 2000 organized by its Secretary General Kofi Annan, significant changes emerged in the field of development on how to define it and how to address it. J.F.E. Ohiorhenuan summarized this evolution in the development conception by: The MDGs 6 are the culmination of 50 years of thinking and policy about development. They also represent a shift from an original empirical grounding to a normative framing of the development question. (Ohiorhenuan, 2011:13). 6 Millennium Development Goals 39

48 The MDGs were expected to improve coordination among partners and developing countries for a better appropriation of development actions to reach targets. Have MDGs contributed to foster countries accountability and public performance? To that question, Davis (2011) answered that the MDGs fail to encompass all the elements of development, with implication for the effective use of resources. (T.W.D.Davis, 2011:568). In short, MDGs contributed in improving country-led accountability and performance measurement systems. Development dimensions as described by Sen, include: Economic facilities, political freedoms, social opportunities, transparency guarantees and protective security. (Sen, 1999:42). Development programs aim to address these fundamental dimensions to improve people wellbeing and livelihood. In West Africa, these components of development are yet to be addressed in many countries. This situation is aggravated by the weak institutional and organizational framework in designing and implementing development policies and programs in the region. Understanding the development program approach in Africa necessitates revisiting the roots of poverty and development in the continent. It should start from the colonialism and its impact on the present development condition of Africa (Mccord, Sachs, & Woo, 2005). The emergence of the MDGs in the 2000s introduced new managerial practices in the development field that led to increased awareness and results-based management and evaluations innovation for improved decision-making. But, the concept of results-based management, closely linked to effective Monitoring and Evaluation System (see sections below), is not new in the development field, it was highlighted in the management field by Peter Drucker in his Book Management (Drucker, 1954) through the concepts of Management Effectiveness in an internal view of the organization/firm, Spirit of Performance, Organizational Results, and Social Impacts in an external view of the organization/firm. In , the United Nations launched the Sustainable Development Goals-SDGs as a continuum and improvement of the MDGs, through the 2030 Development Agenda for Actions (United Nations, 2014b). Griggs (2013), summarized in a framework the objectives of the SDGs (D. Griggs, 2013). The United Nations Rio+20 summit in Brazil in 2012 committed governments to create a set of sustainable development goals (SDGs) that would be integrated into the follow-up to the Millennium Development Goals (MDGs) after their 2015 deadline. (D. Griggs, 2013:1). The concept of performance management that relates to results based management was and is still a priority in the development area in a context where many developing countries and development cooperation agencies were facing some common issues and challenges in the 80-90s. Many countries and development agencies engaged, in the 90s, in the process of reforming their performance management systems and methods of measuring development results. For example, the main concern was: how to set up an effective performance measurement system, address issues related to the attribution of development outcomes and 40

49 impacts, ensure a separate but complementary role to evaluation, and establish incentives and organizational processes that stimulate the use of performance information in management and decision-making? During the years , most countries undertook extensive public-sector reforms in response to economic, social and political pressures that included fiscal deficits, structural problems, increasing competitiveness and globalization. Even in the developed countries, all government executives and parliaments were guided towards public sector reforms. It was then noted, the passage in 1993 of the Performance and Results Act as the main driver of the US federal government reform in the United States (Robert & Ronald, 1996). For example, in the United Kingdom, the publication of a 1995 White Paper on Improving Accounting for Taxpayers' Money was a key step in committing the government to the introduction of resource accounting and budgeting (UNESCO, 2008). In Australia, the main driver of change was the introduction of outcome-based regularization and output budgeting. In Canada, the Office of the General Auditor and the Treasury Board Secretariat have been the main proponents of reforms within the federal government (Brown, Waterhouse, & Flynn, 2003; Mayne, 2006). For France, Galdamar et al., argued (Galdemar, Gilles, & Simon, 2012:4): «la recherche de la performance dans le secteur public, entendue comme l optimisation des services rendus aux citoyens, a connu différentes étapes depuis l après-guerre, la plus récente étant la Loi Organique Relative aux Lois de Finances (LOLF), votée en 2001 et mis en œuvre à partir de 2006.» to clarify that the French administration through this reform moved from the logic of resources and means to the logic of results in the New Public Management since In developing countries in early 2000, the Africa continent, in particular, was characterized by multiple challenges and crises in various development dimensions including economic, political, demographic, and environmental crises (World Bank, 2001:15). All these crises led to a lack of confidence in governments by the citizens and a growing demand for better public services and more appropriation of services by the civil society, a better resources management toward effective development results. Popular refrains such as "reinventing the state," "doing more with less," "value for money," etc. described the broad movement toward public sector reform. The African States facing severe economic difficulties and crises as well as pressures were characterized mainly by lack of transparency in the management of public resources, poor governance and lack of financial and legal rigor for agreed upon public management procedures including procurement and financial expenditures, lack of strategic and programmatic vision, multiplicity of stakeholders and lack of harmonization and control of development actions, highly centralized top-down planning systems without taking into account the basic needs of populations, structural adjustments and deficits, the extreme poverty of a large segment of the population, incomplete democracy system and poorly managed institutions, and 41

50 presence of dictatorial regimes, over union of employees in the public and private sectors, rebellion wars and movements in many countries (Bakare, 2014; Bijaoui, 2012; Bujra, 2004; Funke & Nsouli, 2003; Heldring & Robinson, 2012; Khapoya, 2012; Obeng-odoom, 2013; Ohiorhenuan, 2009, 2011). To overcome these difficulties and the democratic and governance challenges in the continent, there was a need to build robust and well-managed public services and institutions, public performance systems and better responsibility of government and public administration in managing development. Those reforms included a better consideration of democratic aspirations, the advent of some peaceful democratic changes, strengthening civil society organizations, equity and gender mainstreaming, better citizen awareness, introduction of new and more participatory planning processes, introduction of new programming and budget management instruments, better taking into account the different dimensions of poverty as per Sen s expansion of real freedoms, and strengthening governance, monitoring and evaluation systems. At the United Nations Millennium Summit in September 2000, world leaders pledged to achieve specific development goals, known as the Millennium Development Goals (MDGs) by 2015 in the following areas: reduction of poverty, hunger, illness, illiteracy, combating environmental degradation and discrimination against women (Roberts, 2005). Resources were inadequate, mainly in developing countries, to achieve the international development goals including those contained in the Millennium Declaration. In March 2002, Heads of States and Governments and international financial institutions met in Monterrey, Mexico in order to address challenges in financing development particularly in developing countries with the aim of eradicating poverty, achieving sustained economic growth and promoting sustainable development for progress towards a truly inclusive and equitable global economic system (UNDESA, 2003). Improving development management strategies was at the forefront of the Monterrey Conference (OECD, 2006:1). The Joint Memorandum on Managing for Result in Marrakech, 2004, came to repeat and highlight the commitment to better manage resources for development results at higher level, the African Development Bank, the Asian Development Bank, the Inter-American Development Bank, the European Bank for Reconstruction and Development, the World Bank, the Development Assistance Committee of the Organization for Economic Co-operation and Development (DAC/OECD) affirmed in Marrakech their commitment to promoting a global partnership for managing development results (OECD, 2006). The culmination of this global process towards a better partnership for improved development results management in the world is, of course, the 2005 Paris Declaration on Aid Effectiveness. Ministers from developed and developing countries in charge of promoting development and heads of bilateral and multilateral development assistance agencies met in Paris on March 2, 2005, committed to carrying out ambitious actions to improve the effectiveness of development aid management (Paris Declaration on Aid Effectiveness, 2005:2). 42

51 In the 2005 Paris Declaration, development partners and development countries representatives committed to advance development objective in developing countries through, 1.Ownership: Partner countries exercise effective leadership over their development policies, and strategies and coordinate development actions ; 2.Alignment: Donors base their overall support on partner countries national development strategies, institutions and procedures; 3.Harmonization: Donors actions are more harmonized, transparent and collectively effective; 4.Managing for Results: Managing resources and improving decision-making for results; 5.Mutual Accountability: Donors and partners are accountable for development results. (Paris Declaration on Aid Effectiveness, 2005:5). To strengthen and deepen the implementation of the Paris Declaration in Africa, in 2008, development partners and African countries met in Ghana to set and commit to the implementation of the Accra Agenda for Action (AAA) for aid and development effectiveness. The commitment 4) Managing for results is one of the cornerstones of the Paris Declaration that grounded the institutionalization of effective monitoring and evaluation to advance development objectives. Before the Paris Declaration, Monitoring and Evaluation was considered as an emerging good practice in development management that increases accountability. The Paris declaration has put managing for results and Monitoring and Evaluation System at the heart of development management as an institution for improved learning, risk-taking, and decision-making to advance development mainly in developing countries. The Results-Based approach re-introduced in the new development practices and endorsed by the international community in Paris in 2005 (Paris Declaration on Aid Effectiveness, 2005), is putting more attention on the so what? question for development actions. The Monitoring and Evaluation Systems need to be better positioned to providing evidence-based learnings in answering to the So what? question. Results-based Monitoring and Evaluation systems are designed to address the So what? question. So what that all the inputs have been gathered, so what that all the activities have taken place, and so what that the outputs from these activities have been counted? (J.Z.Kusek, Rist, & White, 2005:8) REGIONAL, NATIONAL AND LOCAL FRAMEWORKS FOR DEVELOPMENT POLICY AND PROGRAM A development program is illustrated as a set of actions and investments planned in a way to respond to a substantial need or aspiration and to achieve a development objective or goal. Primarily, the goal is stated clearly as well as its contribution to the regional, national or local framework. The policy approach seeks to 43

52 determine the relevance of the development actions and investments realized in line with the program goal or development objective. The program goal is defined in a way it is captured through tangible long-term development results. These long-term results are analyzed in a certain manner that shows clearly the development impact produced by the program outcomes. The development impact can be measured at macro, meso and micro levels of the program s zone of influence (ZOI) or development area. The program s direct beneficiaries will experience the impact directly at their level. For example, a national agricultural development program serving directly different actors at specific micro-levels of the value chain, where the development actions produce a positive impact on businesses and livelihood of the various actors at those levels of the industry. How the development program impact is aligned and contribute to the general development framework defined by the institutions and organizations responsible for policy-making, is key to assess its broad relevance. Whereas a development program is not a stand-alone action, it is originated from a broader policy framework that underlines the development objective at regional, national or local level; it is also elaborated within the development sector in which it originates. The development sector is an area of common sub-goals, activities generating different development programs translated to specific projects that constitute the bulk of actions defined to move the sector towards its potential development. A sector can be characterized by a comparative advantage and be subject to prioritization by the institutions, organizations at regional, national and local levels as the broad development driven scheme that necessitates major attention and coordination (planning, funding, management, etc.) from decision makers. Some of those sectors are considered productive, while others are designated as social sectors as their impact is not directly linked to economic growth (Alacevich, 2011). Development programs at country and regional levels are key components of national and regional policies to address development objectives. Quesnel expressed: strategic intent of intervention constitutes policy, strategy, institutional strengthening, program, project, activity, task, product or service (Quesnel, 2010). Monitoring and Evaluation Systems are therefore central to how this range of activities can be viewed as a coherent approach to the success of development programs. Mainly characterized by its strategic intent or general objective, development program focuses on institutions and organizations in a geographic coverage and with intended impact and outcomes as well as core stakeholders concerned with the proposed actions and changes. In our view, development program must be driven by demand and based on a substantial and sustained institutional ownership (Bakare, 2014). How Monitoring and Evaluation Systems addresses the various differing interests that incorporate the range of public sector intervention, underline the grounding rationale of these development actions and supports 44

53 depending on existing development strategies, policies and the level of and roles in the intervention: local, national and regional. In West Africa, the definition of the different levels localities may change from a nomenclatural view for various countries. It is worth noting that three levels of administrative and geographic contexts in defining localities exist in all countries: Communities level (the lowest), District level (the medium), and national level (the highest and unique level). When development programs are defined and implemented for the country-wide population benefit and the national coverage, they are called National Development Program (NDP). That means, the products and services of the development program go to different areas and cover the country. Whether are they defined and implemented at district or communities level, they are called Local Development Program (LDP). The products and service go only to beneficiaries in those areas. When regional bodies or other regional stakeholders along with country representatives, define and implement actions in the development field, activity is called Regional Development Program (RDP). In the introduction of the Africa Union Strategic Development Plan, Agenda 2063, the Continental organization stated its vision as follows: build an integrated, prosperous and peaceful Africa, an Africa driven and managed by its own citizens and representing a dynamic force in the international arena. (The African Union, 2014:10). Development projects and programs are defined under regional, national or local development strategy and policy frameworks. Regional bodies, national governments, and local authorities are responsible for the definition of development objectives through various participatory consultations and policy assessments. The most significant level in regard to political aspects is the country or national level, where development objectives are strictly driven by the political, social and economic vision of authorities in charge Regional Development Policies and Program In Africa, regional bodies that oversee the integration of the region s people and institutions are weak and hampered by bad governance, lack of funding and weak political will to the integration of some countries. The Economic Commission for West African States (ECOWAS) 7, with the diversity of the population, 7 ECOWAS, the Economic Community of West African States, was established by the Treaty of Lagos signed on May 28, Francophone countries in ECOWAS include: Benin, Burkina Faso, Côte d Ivoire, Guinea Conakry, Mali, Niger, Senegal, and Togo. Mauritania withdrew its membership in 2002 to join the Maghreb Arab Union States (UMA). Lusophone members of ECOWAS include Guinea-Bissau and Cape Verde. When added to the Anglophone countries of Gambia, Ghana, Liberia, and Nigeria, ECOWAS total country membership stands currently at fifteen (15).Headquartered in Abuja, Nigeria, data on ECOWAS programs and activities can be found at: ECOWAS constitutes one of five regional groups that constitute the African Economic Community (AEC), a subsidiary entity of the Africa Union (AU). The other regional groups are: COMESA (The Common Market for Eastern and Southern Africa), ECCAS (the Economic Community of Central African States), IGAD (the Intergovernmental Authority on Development), and SADC (The Southern African Development Community). A February 1998 protocol defines relations between the AEC and the five regional groups. 45

54 variety of democracies, is challenged by a lack of political will of some countries not yet able to fulfill their commitment to funding the institution, hence making the regional organization dependent on foreign investments and technical support in implementing regional development programs. The fifteen African ECOWAS countries (Figure 1) are Low-income and Dependent Countries (LDCs) and, as Fozzard et Foster (2001) put it: Poverty, limited technical capacity, economic instability, a narrow revenue base and weak systems of governance represent significant constraints on effective expenditure management in all LDCs. Aid-dependent countries, where aid inflows commonly account for over 10% of GDP, must also contend with multiple donors and reliance on aid inflows to finance a substantial part of their budget. (Fozzard & Foster, 2001:1). Figure 1. ECOWAS Member Countries (in blue color) Nigeria, the largest economy of the region hosts the organization. As shown by the UN-Africa Economic Commission Report (2015), while the economic growth rate declined slightly in 2015 (4.4%) compared to 2014 level (5.7%), the West African countries are moving to a critical structural change of improved Trade- Industrialization with weak institutions and huge fluctuations in major primary commodity prices. ECOWAS countries have growth margins and market opportunities to fund their development strategies. The UN economic outlook is showing a total average 21% of level debt compared to the regional convergence threshold of 70% of the GDP. Only Cape Verde, the Gambia, and Ghana are beyond that threshold with respectively 117%, 95.1%, and 72.8%. The regional average level of Public Debt is influenced by the lower level of Nigeria (78% of the entire ECOWAS economy), with only 11.9%. Nevertheless, Security conditions, which are still difficult in some countries of the Sahel (especially Mali 46

55 and Niger) and the north of Nigeria, will continue to be major risks for the economic outlook for (United Nations, 2016:10). To overcome development challenges and ensure the livelihood of the West African people, regional institutions such as ECOWAS set forth regional policies and strategies to advance the development of their communities. Built generally within a participatory approach by convening various stakeholders at the regional level, regional policy-making is primarily to address institutional, political and economic challenges and aims to foster integration and organizational capacity towards a performant regional market for improved economy and business enterprise to ultimately benefit to people. ECOWAS constitutes the regional umbrella under which policy and development programs are designed and implemented in the West Africa region. The importance of multi-party decision-making in policy design and implementation is wonderfully stressed by Clinton J.A. in his Rationality in Decision Making (2010) approach that shows the strong role of regional umbrella or institutions in moving policy at broad level for greater sustainability. In the same streak, Bakare stressed that the difficulty in establishing reliable and sustained governance institutions is one of the major development challenges faced by African States (Bakare, 2014). Thus far West African people, in general, have little understanding and knowledge of the development vision defined by their leaders (Bujra, 2004), the regional institution vision is considered as the principal reference for sustainable development planning and regional integration. In the African continent, there are four regional economic and political grouping institutions: The Common Market for Eastern and Southern Africa (COMESA) grouping nineteen countries, the Southern African Development Community (SADC) with fifteen member states, the Economic Commission of Central African States (ECCAS) counting ten countries, and the ECOWAS with fifteen countries. These integrating regional institutions according to Otto & Patino, have an ambitious objective to promote the continent s economic growth, political stability, and good governance. To achieve their objective of economic development and political and economic integration, regional institutions define and implement policy and programs in many areas purely linked to a regional free market and improved business opportunities that lead to increased people s well-being. In its Strategic Plan , ECOWAS stated its Vision as follows: To create a borderless, peaceful, prosperous and cohesive region, built on good governance, and where people have the capacity to access and harness its enormous resources through the creation of opportunities for sustainable development and environmental preservation. (ECOWAS- Commission, 2010:1). In 2014, the African Union (AU) defined a continental strategic plan called Agenda 2063 The Africa We Want - which presents the vision of the continental institution as to: build an 47

56 integrated, prosperous and peaceful Africa, an Africa driven and managed by its own citizens and representing a dynamic force in the international arena. (The African Union, 2014:10). Regional integration increases economic growth and incomes to result ultimately in welfare gains for the people through improved regional free trade and increases in the size of the markets and business opportunities (Kimenyi & Kuhlmann, 2012). The way decision-making is organized at the regional level and the approach for regional institutions in making development policy through improved results-based management, knowledge and information sharing, learning, and evidence-based decision-making, are key to advance the development and welfare of the people in the region. This evidence-based approach to regional priorities is key to overcome factors that hinder the achievement of development actions at regional level including, but not limited to, political factors, social and cultural barriers, conflict, weak infrastructures, limited economic diversification, and weak ownership of the process by the population, etc. (Kimenyi & Kuhlmann, 2012; Ott & Patino, 2009). The necessity of participation and ownership of a regional monitoring mechanism for development actions is stressed in the Blair s Our Common Interest. Report of the Commission for Africa as follows: Management requires measurement and monitoring. If citizens around the world are to hold their governments accountable and partners in development are to be mutually accountable, they must have reliable and reasonably objective ways of monitoring actions and outcomes. (Blair et al., 2005:381) National and Local Development Policies and Programs While the regional and global policies and programs are relevant to business and free market development in the context of globalization, at national and local levels, according to various authors, policy making is key to manage existing resources, to create jobs and stimulate the economy of a well-defined territory. Helmsing and Egziabher (2005:1) cited by C.M. Rogerson (C.M.Rogerson, 2011). These levels cover the local development practice in a general concept called Local Economic Development (LED). According to Rogerson, the International Labor Organization (ILO, a UN Agency) distinguishes four core features that characterize LED strategies: (1) participation and social dialogue; (2) a focus on territory; (3) the mobilization of local resources and competitive advantages; and, (4) the imperative for high levels of local ownership and management. (C.M.Rogerson, 2011:1). In West Africa, in a context of a weak nascent private sector and civil society, Government and Public actors are the single leaders of development policy and strategy making at national level. At State level, as democracy is consolidating in the West Africa region, political parties run to elections in general with a proposed strategic development plan to convince voters. For example, in Senegal, the President Wade, with 48

57 his strategic plan built around the SOPI 8 philosophy expanded a free-market policy, strengthened by proper infrastructures development, youth opportunities enhancement and women empowerment and participation. This approach, following Wade election in 2000, was implemented progressively through various national documents called successively National Poverty Reduction Strategy (joining the World Bank terminology), that shifted to National Program for Economic and Social Development following the renewal of his mandate in The fourth Senegalese President, Macky Sall, who defeated President Wade in 2012, invented the Yoonu Yokkute 9 strategy after having visited more than 1000 communities between 2009 and This strategy was a continuum of Wade vision of free market with more integration of vulnerable groups through social projects and improved integrity and better governance in public resources management. In 2015, the President Macky Sall upgraded the national development strategy to the notion of Emerging Economy: The Emerging Senegal (Plan Senegal Emergent PSE). This denoting of national strategies is adopted by many francophone countries and seeks to move from poverty reduction policy-making to more market-driven institutional and economic growth strategy. The President Goodluck Jonathan shaped the Nigeria 20:2020 Vision, portending by 2020, Nigeria seeks to be part of the 20 biggest economies of the world as a vision. That dream of President Jonathan translated into real programs in different strategic sectors of the Federal Republic in the 2010s. In the driving agriculture sector, it was translated into the Agriculture Transformation Agenda ATA-. The ATA introduced many innovations in Agriculture sector development approaches in Nigeria and West Africa: i) Ending the era of treating agriculture as a development project but rather as a business, ii) No more isolated projects without a strategic focus to drive agriculture growth and food security in a clear and measurable way, iii) Ending of "big government" crowding out the private sector, iv) Value chains focusing where Nigeria has comparative advantage, v) Strategic partnerships to stimulate investments, and vi) Putting sharp focus on youth and women. National development strategies are key in operationalizing the leaders political, economic and social vision for improved welfare of their people and enhanced development in reducing the gap North-South and launch African countries into economic growth paths. These strategies up till now necessitate lots of incentives and participatory approaches as they guide all policies and programs defined at lower levels and sectors. West African Governments put in place in their Constitutions and Legal frameworks the underlying 8 Change in Wolof 9 Sustainable Development Pathway in Wolof 49

58 laws and reforms that outline the institutions and organizations responsible for defining, implementing and assessing the results of development actions operationalized from the national development strategies. While the Government through the President, is responsible for defining the vision and development priorities in the strategy, many other stakeholders are involved in the operationalization. The National Assemblies - Parliaments - in charge of legislations are responsible for adopting the government proposed reforms or to initiate laws that will set forth the needed foundation to ground the development actions implementation and evaluations. On a yearly basis, through various presentations and discussions between the Parliaments and the government Ministries, work-plans and budgets are adopted in line with the available resources and the fundraising systems in place to gather national and foreign investments to move the action plans to achievements. Private sector and civil society organizations are also involved in the definition and implementation of national strategies. In West Africa, the local private sector is still nascent, weak and fragmented. Some prominent private sector organizations are currently moving to a growing status and capacity like the Dangote Company in Nigeria which is expanding its investments in other West African Countries. In infrastructures development, the Sow Family Company in Senegal, the Sahelienne d Entreprises (CSE) is expanding its activities in many countries, especially in roads construction. Without a strong local private sector, development action impacts will not last long and remain sustainable for West African people livelihood and welfare as benefits from investments and actions promoted will be keeping moving to the former colonialist countries and recently to China which in leading the building of infrastructures in Africa with its national private sector generally reluctant to erect partnerships with local firms. For both Government and national private sector along with civil society to master development policymaking and implementation for improved economic growth, job creation, and welfare, it is critical that reliable information flow within different interfaces, institutions, and organizations. For improved decisionmaking in risk-taking, evidence-based information should be gathered, analyzed for the conduct of policy making and business enterprises management to advance development. As development actions are decentralized at different levels of the countries, local territories gain political power and responsibilities in defining and managing their local development policies and by partnering with national and international organizations, private sector entities, local governments, bilateral agencies and UN agencies to implement their development programs: what is called, The Globalization of Local Development. In West Africa countries, different decentralization policies and institutions started being put in place early These policies defined local territories based on geographic, environmental, economic and cultural 50

59 standards. The conception and implementation of local development policy are under the responsibility of local government authorities elected by the population at the local level. Capacity building of local governments in defining and managing their policies and programs is still the main activities of national governments and development partners. Weak technical and financial capability and absence of local private sectors and strong civil society is the continuous trial to local development ownership and success in West Africa. Top-down policy-making is still unyielding as national government and entities possess and manage the essential of economic resources to implement development actions Synergy and Integration of Regional and National Development Programs Development programs define their activities and components to achieve their specific objectives. The different activities and components are leading to significant outputs that constitute the bulk of the development outcomes generation (Figure 2). In fact, development programs deliver products and services under the agreed activities/components to the beneficiaries and different stakeholders involved or parties prenantes. The relation Activity/Outputs-Specific Objective/Outcome will underline the development program effectiveness in leading to the desired results at the broad level objective (National, Global). Figure 2. Development Program Activities/Components and Specific Objectives. This approach used for development program planning, implementation, and assessment, is based on the Logical Framework Approach (LFA). The LFA tool (Table 2) origin, content, and development are about Monitoring and Evaluation practices that will be analyzed in next sections. Described as a planning, management and evaluation of Development Program tool designed by Practical Concepts Incorporated (PCI) in late 1960s for the United States Agency for International Development -USAID- (Coleman, 1987), 51

60 the Logical Framework has itself integrated the core elements of the Monitoring and Evaluation System: indicators, means of verification, results chain, assumptions. The authors of the LFA (PCI-USAID) were obviously very conversant with M&E-System needs as they provided consonant elements to it in the LFA. The LFM is built on three major hypotheses/logics: The Vertical Logic which identifies what the project intends to do, clarifies the relationship between project means and ends, and specifies the uncertainties concerning both the project itself and the social/physical/political environment within which the project is located. (Coleman, 1987:251). The Horizontal Logic objective is the MEASUREMENT of the resources and results of a project, through the identification of Objectively Verifiable Indicators and Means of Verification for these indicators.(cida) (Coleman, 1987:256). The Logic of Intervention which describes the general approach, conditions, and strategy for the project/program implementation, from the transformation of the resources/input to products and services/outputs to the changes, outcomes and then the impacts. The LFM provides the essentials for the basis of building a relevant M&E-System: 1) The results chain, 2) The objectively verifiable indicators, and 3) The mean of verification or sources of information. These columns represent the Horizontal Logic of the LFM. The results chain provides information on the various levels of the program from inputs to outputs, outcomes, and impacts. At each level, commensurate indicators are defined to capture the results to be achieved. Table 2. Logical Framework Matrix, (Coleman, 1987) 1-Narrative Summary 2- Objectively Verifiable Indicators (OVI) 3 -Means of Verification (MOV) 4 -Important Assumptions GOAL Measures of goals achievements Sources of information Methods used Assumption affecting Purpose-Goal Linkage PURPOSE End of project status Sources of information Methods used OUTPUTS Magnitude of outputs Sources of information Planned completion date Methods used INPUTS Nature and level of Sources of information resources necessary Costs Planned starting date Assumption affecting Outputs-Purpose Linkage Assumption affecting Inputs-Outputs Linkage Initial assumptions about the project The LFM shows the status of the project/program at the end of its implementation, a picture of the desired situation. Over its introduction in the development field through its development and improvement, the LFA 52

61 faced several criticisms. According to D. Cordingley one of the major criticisms leveled at the logical framework is its neutrality with respect to various development criteria. (Cordingley, 1995). That was one of the main gaps addressed through the design of the Zielorientierte Projektplanung (goal oriented projectplanning- ZOPP) in its unbiased heavy participatory approach. David Akroyd, in 1995 confirmed the central role of LFA in Monitoring and Evaluation development (Akroyd, 1995). The LFA, in fact, goes beyond the mere implementation aspects of projects and is a tool to aid projects (and programme) planning and management at strategic an institutional levels (Wiggins & Shields, 1995:3). In all developing countries, the MDGs were the reference for policy design the last decade, strategies and programs were assumed to show their contribution to the MDGs as an international commitment of the country to the global development framework set forth by the United Nations (UN) in When the designed national programs in their broad level objective ensure alignment to the MDGs, the second general strategic reference is the country national development strategy core priorities (Figure 3). Figure 3. Strategic alignment to national and global development objectives. The national development strategies have many objectives and sub-objectives as well as specific intermediate results for each sector or priority area of the country (Figure 3). The development program may contribute to a certain number of development objectives or priorities, and highlights one major area of focus in its results chain. In how the development program will effectively contribute to the underlined development objective, a Specific Objective or Program Intent is defined leading the development program to its contribution and performance to the achievement of the national and global development objectives. The M&E-System intends to provide relevant and meaningful feedback on development programs and projects to help stakeholders and decision makers learn from achievements, results, constraints, impacts, sustainability and to capitalize on learnings for adjustment of current programs and future development 53

62 policy design. The development intent generally summarized in a formal policy document is the basis for setting up the M&E-System. The development intent could be defined at various levels: regional, national, sub-national and local. The main challenge in building M&E-System is how to articulate the development Programs and Monitoring and Evaluation Frameworks to the national or regional results pathways and government performance management plan CONCLUSION ON LITERATURE REVIEW OF DEVELOPMENT MANAGEMENT AND PERFORMANCE MEASUREMENT SYSTEMS In short, this chapter 3, which is a systemic presentation of the development context and its requirements or needs for performance measurement frameworks, examined the role of institutions and organizations in the development process and analyzes how institutions, the rules of the game, and organizations, the game players, shape the development process toward the desired changes. While it was highlighted that development is not only about economics and neoclassical equations, the chapter reviewed Gauthier s idea that development, mainly development aid, should focus first on the creation of effective institutions, and recalled the Douglas North s clarification that institutions and organizations are leading in the set of standards for any given game, be that for the sporting game, so be for the development management field. It also analyzed the Faguet s definition of development management, as the attempt to moving institutions and organizations towards higher efficiency and effectiveness for increased capability to advance development actions. Because institutions and organizations are at the center of any given development policy and program, management approaches are essential to support institutional and organizational capabilities to move the development actions to success through improved efficiency and effectiveness. One of the key management approach in this instance is the development performance measurement system called Monitoring and Evaluation System. It was then clarified how institutions and organizations use the M&E System strategy in their role of managing development. In the West Africa region, three levels of organizations exist: regional, national and local. At these different levels, institutions and organizations in charge define development strategies and policies by using many approaches in policy design ranging from the new theory of change design and the logical framework methods. The development policies and programs at local, national and regional levels should align with the global development objectives led by the United Nations organizations. It was shown that there is a growing need for accountability and efficiency in managing development, especially in Africa where development results are yet to be recorded. The Paris Declaration in 2005 brought back M&E Systems and results-based management at the heart of institutional and organizational approaches in better managing development actions. Hence, effective M&E System is key to development managers and decision-makers 54

63 capability to ensure better learning, knowledge management for improved evidence-based decision-making and measured risk-taking to advance development objectives. This long way for better results management of development actions was revisited in this chapter, from the Washington Consensus in the 1980s, to the sustainable development approaches in the 1990s as well as the governance and accountability reforms by governments, the Monterrey Consensus on financing development, the Paris Declaration in 2005, the Accra Plan of action in While this chapter 3, which is a first section of the systematic literature review, focused on clarifying the role of intitutions and organizations in development management and the linkages with performance management systems including M&E System for incearsed efficiency and effectiveness, with a focus on practices in West Africa, the next chapter (A Review of Monitoring and Evaluation System) will focus on the Monitoring and Evaluation System systematic review from its history, to the gaps for effective M&E System, how the M&E System links to development management strategies and the critical gaps that need to be adressed in this research. 55

64 CHAPTER 4. A REVIEW OF MONITORING AND EVALUATION SYSTEM HISTORY While Chapter 3 introduces development management and performance assessment system, the Chapter 4, presented here, provides a systematic review of Monitoring and Evaluation System as a performance assessment framework for organizations and institutions in charge of development management. This chapter is a cornerstone of the research as it aims at reviewing systematically the existing knowledge in the M&E System literature and development practitioners approaches in a view to support the process of designing a robust framework for the next phases of the research. It revisited first the history of M&E System in development management theories. From the 1886 Charles Booth s Life of the People of London (Gelase, 2011), to the recent approaches developed after the MDGs in the 2000s, it looked at the various tendencies and analyses on the origin and development of the concept that will help better understand its framework. We analyzed the core stages of the M&E System development largely influenced by the development theories and priorities at different periods from the end of the WWII, the independence of African countries in the 50-60s, the A. O. Hirschmann's Development Projects Observed with their uncertainties (Hirschman, 1967), the advent of the LFA, and the Washington Consensus (Alacevich, 2011; Stiglitz, 1999, 2008a, 2008b, Williamson, 2004, 2008). The chapter then analyzed and ranked the M&E practices in the last decades in three main stages: The Network-Chart stage (60-70s), the Impact Evaluation Clinical Trial stage (80-90s), the Web-based information system stage (the 2000s). We then developed a section analyzing the M&E System effectiveness literature from the views of scholars and practitioners (J. Cameron, 1993; Cracknell, 1994; Kusek & Rist, 2008; K. Mackay, 2007; Mackay, 2008, 2010a, 2010b; Quesnel, 2010). It analyzes the viewing of M&E System as an information system (Barton, 1997; Crawford & Bryce, 2003; Wagner et al., 2005). The chapter concludes with a presentation of the most powerful IS success models (DeLone & McLean, 2003; Leclercq-Vandelannoitte et al., 2014; McGuire, 2011; Petter et al., 2008; Soualhia & Mejbri, 2014), that would serve as the basis for the building of a framework for effective M&E System HISTORICAL VIEW OF M&E-SYSTEM IN THE DEVELOPMENT FIELD Effective Monitoring and Evaluation is key to public policy planning and development program implementation mainly in developing countries context such as West African countries. Policy choices may 56

65 have a positive or adverse impact on human conditions, and should then be built in the context where good and reliable information is available for decision-makers in charge of development policy planning and implementation. Lack of quality data on socioeconomic issues in the countries or region can hinder proper planning process of public policies in the context of fragile countries and vulnerable population. Implementing development programs also needs a strong Information System that could guide decisionmakers towards right and tangible development targets through better evidence-based decision-making. As such a decision support and performance measurement system, M&E System plays a key role in public policy management (Andrews, 2010). Nevertheless, the origin of the concept of Monitoring and Evaluation (M&E) in development program management is yet to be clarified by the scientific community and development practitioners. Unquestionably, there is a strong need to dig out and specify the origin of the concept to understand better its comprehensiveness. Many researchers and development experts used to link the source of Monitoring and Evaluation to the development aid system and tried to relate it, in a historical view, to one or another donor or development partner. John Cameron (Project Appraisal, 1993, 91-96) situated the origin of Monitoring and Evaluation in World Bank programs, first as a project or quality assessment tool for program implementation in the 1970s that main role was to assist donors and countries in improving public and aid development investments effectiveness. He related the development of the concept both as a planning and implementation tool for decision-makers to the impacts on development practices of the Market-led Structural Adjustment conditionality in developing countries in the 1980s and their political disruption, poverty, and environmental implications. This theory linking Monitoring and Evaluation System origin to the World-Bank sponsored development programs, led by Cameron, justified the increased interest on formal Monitoring and Evaluation in development practices to the growing role of Non-Governmental Organizations (NGOs) in implementing development projects, hence replacing progressively public sector activities by formal and more performance-oriented private sector projects in the 1990s. Monitoring and Evaluation were then positioned between the project start-up and its closure describing it as an implementation performance assessment tool. That idea is driven by the network charts, outputs and efficiency measurements approach in the field of performance management of development projects in the 1960s and the 1970s. This theory in its general outline conflicts slightly with the Logical Framework Approach tool origin and history in the development field, its content, and deployment in relation to Monitoring and Evaluation practices. Described as planning, management, and evaluation of development activities tool designed by Practical Concepts Incorporated (PCI) in the late 1960s for USAID (Coleman, 1987), the Logical Framework has itself integrated the core elements of the Monitoring and Evaluation System: indicators, 57

66 means of verification, results chain, assumptions. The authors of the Logical Framework Approach (PCI- USAID) were obviously very conversant with Monitoring and Evaluation System needs as they provided consonant elements to it in the Logical Framework Approach. That shows clearly that Monitoring and Evaluation cannot be originated in the 1970s after the Logical Framework Approach development. Instead, Monitoring and Evaluation is originated from improved development performance management practices and is dated more than hundred years in the management history (Gelase, 2011). Charles Booth 10, cited by M. Gelasse, in The Life and Labor of the People of London ( ), believed that To effectively deal with poverty there was need to gather quantitative information on characteristic of poverty, including statistics on number of poor people, causes of poverty and measures to alleviate it (Gelase, 2011). That clearly shows that the Theory of Change approach in the Results Chain scheme linking outputs (measures to alleviate) to outcomes (causes of poverty) and then outcomes to impact (number of poor people) is an old knowledge in development management practices. Charles Booth linked clearly effective capability to advance development objectives, specifically poverty reduction objectives in the peri-urban areas of London in the 19 th century, to the effectiveness of information generation, analysis and use for management purpose, that is describing wonderfully the characteristics of M&E-System. This idea pointing that effective Monitoring and Evaluation System is key to improved evidence-based decision-making developed and clarified by C. Booth in the 19 th century shows clearly that the need for effective Monitoring and Evaluation for development management is not a recent concern in the development field, To effectively deal with poverty there was need to gather quantitative information on characteristic of poverty. (Booth, 1986:87). In ordering the needed information in such way that coincides with the different levels of the development programs in the LFM and Theory of Change approaches, Booth s work has shown that the vertical logic and the theory of change is not introduced only after the WWII in the development field. He pointed that to reduce poverty effectively; there was a need to gather, first, information on number of poor people. Number of poor people is the basic variable used with the Total number of the population to generate the Prevalence of poverty which is a key impact indicator in measuring development programs impact on poverty reduction. In putting the impact at the first level of information need to better address poverty in the 19 th century, Booth has shown that the importance of measuring and assessing indicators at impact level for 10 Charles James Booth (30 March November 1916) was an English social researcher and reformer. He is most famed for his innovative work on documenting working class life in London at the end of the 19th century, work that influenced UK government intervention against poverty in the early 20th century and contributed to the creation of Old Age pensions and free school meals for the poorest children. (Wikipedia: ) 58

67 evidence-based decision-making in the development field is not a newly introduced practice in development management. The second level of information need pointed by Booth was information on Causes of Poverty. In designing poverty reduction strategies, the analysis of the causes of poverty could help development managers better define the outcomes of the poverty alleviation programs. Hence, Booth has shown here the importance of the effects-outcomes level as a significant milestone in building an effective information system for development and poverty alleviation policies. In putting the need of reliable information on Causes of poverty, just under the impact level, again Booth has shown that the LFM levels and theory of change in the results chain framework were wonderfully well classified and applied in the 19 th century in the research on meaningful information for evidence-based decision-making to advance development objectives. The third level highlighted by Booth in his The life of the people of London, published in 1986, where information is needed to assess and analyze measures to alleviate that are taken and implemented to address poverty. This level focuses on the actions identified, planned and the products and services delivered to the beneficiaries. Booth is highlighting here the importance of monitoring the development program outputs over the implementation of the programs to ensure effective analysis of deliveries for evidencebased decision-making. The information gathered at this level will help assess whether the program products and services are delivered with the high quality required in the learning for operational adjustments during program implementation. As Monitoring and Evaluation Systems were unfolding in the development field the last decades, three different main stages are noted and analyzed through the literature review, each one characterized by a certain number of dominant Monitoring and Evaluation practices and approaches based on the trends of development programs management in that given period. Each of the stages was influenced at its beginning by critical trends and learnings capitalized as challenges to advance development programs in the previous cycle. The first most known stage of the Monitoring and Evaluation System history is what we call here the Network Charts Stage in the 1960s and 1970s. This stage was mainly characterized by a focus on the capacity to deliver the development programs products and services timely as a pure unveil of the productivity approach in development management. In the early 1980s, as tangible development results were yet to be recorded in many developing countries, there was a growing need to assess development programs outcomes and impact, instead of focusing on implementation aspects and outputs as products and services. That period marked the Clinical-Trial Methods for Impact Evaluation Stage that dominated the Monitoring and Evaluation Systems during two decades, , including the even early 2000s in many areas of the developing world. Heavy 59

68 investments were made by various development partners in measuring development impacts that might be attributable to their specific actions as a justification of their investments. In the early 2000s, the MDGs were launched in the development field with eight core results as global development goals commitment of the Heads of States and International Organizations led by the UN. That global strategic shift in planning for development, and the need to align with internationally recognized benchmarks and development indicators increased the use in the field of Monitoring and Evaluation Systems of IT web-based information systems that allow real-time access to indicators on achievements of development objectives at countries, regions and international-global levels. That inaugurated what we called here the Web-based information application stage which dominated the Monitoring and Evaluation System practices in the 2000s to current NETWORK-CHARTS STAGE ( S) In the early 60s, in his book Development projects observed, A. O. Hirschman analyzed key factors of uncertainties that characterized the development projects implementation both at the supply and demand sides. In the supply side, he identified key uncertainties related to project administration, finance and the technology for products and services delivery (Hirschman, 1967). Mainly in developing countries, development management focused generally in the 60-70s on the design and implementation aspects of projects, while the World Bank top management was debating on their relevance to advance development and focusing more on the costs-benefits analyses aspects of the projects (Alacevich, 2014; Pamies-Sumner, 2014). To address these uncertainties listed by Hirschman and other development practitioners, many countries and donors developed Monitoring and Evaluation System that address the challenges due to the Hirschman listed uncertainties. This approach launched a new era of the Monitoring and Evaluation System history that put more attention on projects administrative, financial and technology aspects as well as the respect of timelines in the production of the intended outputs and services. In search of the literature for project management standard tools, David Cordingley (1995) highlighted the predominant and quantitative incidence in research papers of network charts and Gantt charts compared to the low occurrence of Logical Framework Approach papers (Cordingley, 1995). As he argued, the first reason related to the fact that many development organizations such as USAID and CIDA, first users of the LFA in the 1960s and 1970s, did not share their experiences frequently. Thus, there was a lack of scientific information on USAID and CIDA experience in Logical Framework Approach use in the 1960s and 1970s. In West Africa, the World Bank development programs designed and implemented in various countries dominated that period, and as we argue here, the focus was mainly on outputs oriented system for learning and decision-making. These efforts, during that time, to advance development in Africa through funding 60

69 mainly from World Bank and the International Monetary Fund, strategically prioritized infrastructures development in transports, electricity, agriculture, education, health, etc. Hence, the engineering and technical approaches dominated the approaches in implementing development infrastructures-oriented projects that period. Henceforth, putting at the center stage of development planning and M&E Systems the outputs-oriented information systems through the use of techniques such as the Project Evaluation and Review Techniques (PERT) networks and GANTT-charts as planning, learning and decision-making tools to address program implementation Uncertainties. This network and Gantt charts stage during the 1960s and 1970s stood in the fact that there was more interest in focusing on project implementation aspects, efficiency and timeliness in delivering products and services (outputs) for projects oriented on infrastructures and other physical structures construction in the developing countries. Monitoring and Evaluation Systems were seen primarily as a tool focusing on project activities. This approach defined monitoring as a continuous assessment both of the functioning of project activities to systematically assess implementation schedules, and the use of project inputs by targeted populations as per their expectations (Edmunds & Marchant, 2008). As shown by S. Pamies-Sumner in Historique des méthodes d évaluation, development priorities were oriented infrastructures development in the 60-70s, and the cost-benefit analysis as well as other qualitative approaches put forth through financial and social sciences, dominated the assessment of development programs success that period (Pamies-Sumner, 2014). In that period, this difference in outlook between network charts practitioners (mainly project s managers and staff) at project implementation level and thinkers, or other decision-makers using Logical Framework Approach was one of the sources of the discussion on Monitoring and Evaluation Systems effectiveness. Would it be focused mainly on project deliveries and ensure timely achievement of the outputs, or should it put more interest in results at upper levels (outcomes and impacts) and analyze their relationships in the context in which projects are appraised and implemented to contribute to broader objectives and changes? Failing to consider the linkages results in the implementation of activities for the sake of tradition without a clear purpose for how the activity results in a change (Renger & Renger, 2007:196) leads undoubtedly to what Renger & Titcomb called Activity trap in development programs (Renger & Renger, 2007:196). Cook et al. clarified that performance measurement improves organizations capability to monitor their effectiveness in achieving development goals. This definition of performance management provided by Cook et al. is used by some practitioners to characterize a project Monitoring and Evaluation System. This kind of Monitoring and Evaluation approach focuses on projects deliverables, outputs or products and services mainly. The collection of data on number of products generated is the main task 61

70 along with the analysis of achievements against targets. The tools for the Monitoring and Evaluation activities are generally managed by the responsible of the implementation of activities and tasks. For example, a development project which is infrastructures oriented developing roads will focus on indicators such as the number of kilometers of roads constructed, the number of dams realized, etc. Information related to the inputs are also collected. Analyses are limited only to the respect of timelines and costs, and respect of products and services quality and functionality, and assessment of the project efficiency. This approach was oriented on the projects internal aspects and the Uncertainties that may challenge the implementation of the planned activities. In the West Africa region, many countries accessed to independence early 1960, with a weak infrastructure system as a legacy of the colonial system. Infrastructure development was the main focus of development programs in the 60-70s to reduce the gap North-South regarding transport, energy, education, agriculture, etc., and promote the improvement of the people s livelihood. Achievements of the targets outlined in development policy documents were the main objective of managers. Hence, the learning and knowledge systems such as M&E systems focused mainly on the products and services aspects of the development programs. Tools related to the network chart system were adopted as main Monitoring and Evaluation approaches to measuring development program achievement towards the desired results at medium to shortterm levels. Development programs reports focused more on output indicators such as the number of kilometers of road constructed, the number of power plants established, the number of schools constructed or the number of health facilities realized, etc. Little attention was put on the broad-level indicators such as the percentage of increases in health access or reduction of mortality due to the program intervention, etc... At the same time, managers focused more on day-to-day implementation topics related to the procurement systems, the number of contracts signed timely, the quantity of products and services delivered within the initial timelines, etc... There was a wide adoption of the Gantt Chart and the PERT network in the field of Monitoring and Evaluation during that period and beyond. The other factor is linked to the easy technical access to those tools and the simplicity of instruments for data collection and analysis to inform and interpret output, products and services indicators IMPACT EVALUATION CLINICAL-TRIAL-METHODS STAGE In the early 1980s, development practitioners and organizations agreed that a lot of investments were made in the previous decades (60-70s) in development programs, but little information was available to show their positive impact on the people s livelihood mainly in Africa where poverty was instead increasing. That quasi-absence of evidence-based information on development programs outcomes and long-term 62

71 sustainable positive changes produced in population welfare and capacity of institutions and organizations including the public sector, led to a deep review and reorientation of the evaluation approaches in early 1980s (Bamberger, 2010; Bamberger & White, 2007; D. B. Cameron, Mishra, & Brown, 2015; Cracknell, 1988) with more focus on Impact Evaluation. Impact Evaluations are designed and implemented in a way that seeks to measure tangible development outcomes from a specific program or a set of programs by using experimental or scientific quasiexperimental approaches as they refer to counterfactual-based 11 methods and solving the problem of selection bias (Baker, 2000; Bamberger, Rao, & Woolcock, 2010; D. B. Cameron et al., 2015; Gauck, 2011). Before that period, definitions of Monitoring and Evaluation System focused on performance information generation and analysis for evidence-based decision-making (Shepherd, 2011). That shift from performance measurement system activities-outputs-oriented to more comprehensive results-based outcomes oriented Monitoring and Evaluation System has not been fully successful in many cases and the greater intensity introduced in Monitoring and Evaluation activities was not able to foster Monitoring and Evaluation Systems effectiveness in the Aid Industry the decades. The lack of teleological effectiveness in much of the development project and policy prescriptions is in large part because of our inability to model the complexity of the socio-economic system we attempt to address. (Nelson, 2014:476). The stage of quantitative impact evaluations is introduced in the field of development management by the increasing need from international organizations to demonstrate effective and specific attributable impacts of their development interventions. Rao et Woolcock explained: in conducting evaluations, quantitative methods are best suited to measuring levels and changes in impacts and to drawing inferences from observed statistical relations between those impacts and other covariates. (Rao & Woolcock, 2003:167). Hence, Rao and Woolcock described this approach as the domination of quantitative approaches in evaluations of development programs (Rao & Woolcock, 2003). According to Bamberger et al., the rise of impact evaluation in the field of development is mainly driven by pressure from international organizations (Bamberger et al., 2010). Despite many pressures from the international development organizations and institutions such the World Bank and International Monetary Fund, measuring the impact of development projects and programs in Africa was a great challenge in the 80-90s as well as it is nowadays. One of the biggest challenges is what 11 The counterfactual is a comparison between what actually happened and what would have happened in the absence of the intervention. Data can be collected on the factual. But we cannot observe what would have happened to those affected by the intervention if the intervention had not happened. (White, 2011:3). 63

72 to select among the numerous impact evaluation approaches ranging from strong quantitative statistical and econometrics approaches to qualitative participatory approaches or mapping of outcomes tools. The debate was on what approach is best suited to generate relevant, rigorous, unbiased, evidence-based information on development programs outcomes for capitalization and learning for decision-making. Some practitioners stressed the difficulty to run quantitative approach in projects of weak developing countries environment; other criticized the weakness of the level of viability and robustness of qualitative methods. Besides, impact evaluation is hard to prepare, manage and implement (McMillan & Willard, 2013; Randall & Coast, 2014; Rist, Boily, & Martin, 2013; Vaessen, 2010). Another challenge to preparing and implementing appropriate impact evaluations mainly in Africa is the weak quality of human resources in the 80-90s to undertake robust quantitative studies. High-quality impact evaluations require strong scientists to design and run the methods. Hence, Monitoring and Evaluation Systems in the Africa needed to be well prepared, trained and assisted technically in handling those sophisticated approaches. As White argued that Impact evaluation is the counterfactual analysis of the impact of an intervention on final welfare outcomes. Carrying out such analysis requires analysis of data collected at the appropriate time and places, often using statistical methods which may strike many as unnecessarily sophisticated. (White, 2011:2). The impact evaluations are also very costly to development projects, especially when sophisticated methods are used. However, Impact Evaluation is the only way to measure scientifically, in an adequate development measurement framework, what are the development programs outcomes and what changes occurred in the livelihood of the beneficiaries. Another difficulty is to measure how impact at sectoral levels, or in specific areas of developing countries are contributing to the broader development objectives of the countries. The challenges to undertaking good evaluations in developing countries including the West Africa region could be addressed by improving the quality of how the evaluations are managed. According to Hawkins, In delving into the details of evaluation management practice, it is important to first clarify the roles and key responsibilities of the various players. (Hawkins, 2012:28). Significant amounts of efforts and financial resources, as well as time, are spent in impact evaluations in developing countries. However, the fact is the approaches are not easy to design and implement. Also, few were learned from the various reports of impact evaluations so far implemented for improved policy and decision-making. Major development objectives that were supposed to be addressed by development projects and programs in the West Africa region are yet to be addressed and remain priority targets in all countries development plan. 64

73 Are impact evaluations effective and being contributing to advance policy and decision-making towards development objectives in the West Africa region? The commitment to prepare and undertake robust impact evaluations is still there, the willing to support from international development organizations and institutions such as the WB as well, the capacity of development programs exists to support the process, and the need and necessity to gather, analyze impact data and inform on programs outcomes is critical and well recognized by development managers. The last two decades of the last century were hence dominated largely by the advent of the need to implement rigorous impact evaluations as a strong component of the Monitoring and Evaluation System effectiveness in developing countries (Ba, 1997; Bamberger, 2010; Bamberger & White, 2007; Carley, 1986; Initiative, Evaluation, Paper, Paper, & White, 2009; Johnson, Williams, Wavell, & Baxter, 2004). This strong orientation of Monitoring and Evaluation System effectiveness in the 80-90s was mainly explained by the output products and services network oriented Monitoring and Evaluation Systems in the 60-70s with critical gaps in development programs results and outcomes information. Many scholars and researchers entered to and pushed to that way in taking the opportunity to create a clinical-trial counterfactual approach in impact evaluation design and implementation. Finally, the international community noticed that while huge amounts of resources and efforts are being spent, little information was available on development impact to inform policy and decision-making systems. Both approaches based on outputs systems and counterfactual designs were criticizing one another without a view to better integrate both (Cracknell, 1988; Gauck, 2011; Hughes & Hutchings, 2011; Mayne & Rist, 2006; Picciotto, 2010; Rist et al., 2013). This is one of the reasons why, in the early 2000s, the international development community and head of states decided to define worldwide development objectives, results, and indicators, to improve and harmonize ways to measure development achievements and impacts, the MDGs (Apthorpe, 1999; Asongu, 2014; Gauthier, 2005; Hulme, 2007; Talesco, 2015) WEB-BASED INFORMATION SYSTEM STAGE The increasing introduction and development of IT systems in development management dominated the early 2000s including in West Africa, and that shift changed a lot the ways Monitoring and Evaluation Systems are seen effective. The impact of IT systems on management practices including the use of mobile IT system is highlighted here: By revolutionizing the spatiotemporal framework of organizations, mobile IS affects key functions of management, such as supervision and control, which evolve towards more open and mobile forms. (Leclercq-Vandelannoitte, Isaac, & Kalika, 2014:13). 65

74 The web-based application advent and development changed positively the way Monitoring and Evaluation System are functioning by making possible in many cases the timely availability of information for policy design and decision-making. Development programs started designing their Web-based IS and hiring international specialized firms to implement them. This new use of web-based systems in the early 2000s is partly explained by the advent of the MDGs and conditionality to align with these global objectives to access to funding for development financing (Haque & Burdescu, 2004; Talesco, 2015). Development program managers need timely information in their learning processes to generate the necessary knowledge for decision-making. Web-based platforms became a necessity progressively in managing development programs while the effectiveness of the tool is yet to be established as some development programs succeed without web-based platforms. What is shown is that the web-based systems accelerate the transmission of data and information and may contribute to the quality of information generated and facilitate the possibility to integrate and harmonize, as well as combine information from various sources. There are some web-based platforms sponsored by development organizations which push their supported project to use such systems. Hence, top-down approaches are usually heavy in the use of web-based platforms. For example, the UN-sponsored platform DevInfo: DevInfo is a UN-sponsored and endorsed data dissemination application currently utilized in more than 100 countries to monitor socioeconomic data. The newest version, DevInfo 7.0, is web-based and contains many features that increase the accessibility of the data, particularly because of its powerful visualization and mapping features. (Dickey & Oswalt, 2014:2). The UN-DevInfo platform is an M&E system that focuses on the MDGs indicators and targets in various countries and helps harmonize and consolidate country data towards the establishment of meaningful information on the MDGs achievements. Other lead private web-based platforms exist that dominate the development field in the West Africa region. Those include the French TOMATE TomMonitoring (former TechPro) which is a platform that allows the M&E System to monitor development deliveries and results through a web-based platform ( Mainly used by World Bank sponsored projects, it is well known in francophone countries. Another leading private firm in the design and implementation of M&E web-based platform is the Canadian ADA Consultant Group ( PROSE system which is an integrated M&E system that tracks development results and achievement in real time. It is used by several development programs in West Africa as well as regional organizations. In Senegal, the 2SI-Solutions Informatiques et Telecom ( designed tailored web-based M&E System to various programs in the region including World Bank and USAID projects as well as other Governments programs. 66

75 In the rise of country development strategies including poverty reduction strategies and emerging plans and development actions, Governments in the West Africa region are engaged in various consultations and exchange visits in Europe, Asia, and the US, in the process of learning in how to build effective web-based platforms to monitor their development strategies. Development programs in the various sectors are directed by Governments and development partners to build web-based M&E Systems to facilitate the exchange of information and improve collaboration in the process of decision-making to advance development objectives. The main challenges of these web-based platforms are the high costs and the weak capacity to update and maintain the systems. The role of such systems is not well understood by many of their users. They are mainly viewed and used as systems for data collection and storage, and not always as systems that facilitate and support M&E Systems in their role to improving effective development management through evidencebased decision-making and measured risk-taking to advance development objectives CONCLUSION ON LITERATURE REVIEW OF MONITORING AND EVALUATION SYSTEM HISTORY This chapter reviewed the history of the M&E System through a literature review to examine the different steps as M&E System were unfolding in the development field. The most important finding is the approach to collecting and analyzing data and information on development achievements and outcomes on the welfare and freedom of people, is an old practice. Only in taking into account the roots of M&E System in the most ancient practices enriched with experiences gained after the WWII to the recent periods, will help design ways to building effective M&E System frameworks. considering the history of M&E will help clarify which approaches and methods are better suited to address the research question and sub-questions. It was clarified that effective Monitoring and Evaluation System is key to improved evidence-based decisionmaking since the 19 th century by C. Booth. The chapter described through a classification the different relevant stages of M&E Systems the last five decades to conclude on the most important aspects to considers in the research approach and methods. The Network and Charts stage has shown that the M&E Systems were established progressively with a focus on the most critical development priorities that characterize the concerned periods. This context specific approach in M&E System history should be taken into account in the process of building a model for its effectiveness. The findings on the different other stages of M&E System deployment will be used to fine tune the research approach and methods in better characterizing the M&E Systems probable frameworks that will be assessed and finalized in the research. 67

76 CHAPTER 5. THE RESEARCH APPROACH This section presents the approach used to respond to the main research question and related sub-questions. As clarified in previous sections, the research originated from practitioner experience in the field of development in West Africa. In designing the M&E system model, the approach is based mainly on a review of the existing literature, the consolidation of the current knowledge, and a practitioner insight from rigorously analyzed experiences in the West Africa region and a survey design to collect and analyze data from existing development programs in West Africa. To advance management theory, argued Edmondson and Mcmanus, a growing number of scholars are engaging in field research, studying real people, real problems, and real organizations. (Edmondson & Mcmanus, 2007:1115). The research focuses mainly on real experiences of the writer in Monitoring and Evaluation System practice in the region and the survey feedbacks from development programs on their current M&E Systems. Knowing from practice is key in this research, the critical gaps in the literature do not allow an approach purely experimentally designed, but will be built on a mix practice-based and experimental approaches to creating knowledge, as highlighted by Nicolini: From a practice-based perspective, knowledge and organizational phenomena appear as intimately related. (Nicolini, 2011:602). Iteration 1: Literature review Iteration 2: Case Studies Iteration 3: Survey Objectives 1- Systematic review of M&E System history ab3:d34nd models 2- Characteristics of M&E System components 3- Operationnalization of M&E System Effectiveness 1- Operationnalization of M&E System dimensions 2- Decription of proposed measurements for all M&E System dimensions and sub-dimensions 1- Feedbacks on the M&E System Framework 2- Selection of the measurement for M&E System dimensions 3- Analyses of linkages between M&E System dimensions and effectiveness Approaches and Methods 1- Selection of appropriate literature on M&E System, Results-Based Namagement, Knowledge and Information Management, Evidence- Based Decision-Making 2- Using the most relevant IS model to charecterize the M&E System Effectiveness framework 1- Selection of representive cases in the West Africa region based on the 1- Sampling development programs in researcher experience West Africa by sector, level, and 2- Interviews of core M&E dimension Managers and Program Managers on 2- Online Survey, using SPHINX the M&E System 3- Organize face-to-face and remote 3- Documentation review on how the feedbacks M&E Systems are designed, operating and assessed Table 3. The general research approaches The research approach is trifold: Literature-Case Studies-Survey (Table 3), and comprises then three iterations or evidence-based building blocks, within each of which comprehensive, rigorous, valid approaches are used to draw findings and conclusions on the research questions, as well as managerial recommendations to advance Development Management practices. In the design of the detailed M&E System effectiveness framework, the two first iterations aim at exploring the existing knowledge and to 68

77 learn from specific well selected cases in finalizing the framework as a comprehensive model ready to be tested in the field, while the third iteration is a sample survey to verify the validity of measurements proposed in the framework and their interrelations. The first iteration is an analysis of the literature to review existing knowledge in the operationalizing of the dimensions of effective M&E system. In analyzing the dimensions of effective M&E system, one approach would be to ask directly to the development practitioners and managers what might be considered as effective M&E system dimensions. To avoid the risk of professional bias in selecting relevant dimensions, it would be risky to directly gather managers views in a context where meaningful scientific knowledge is intermittent in the M&E area. M&E system is a broad participatory approach where many stakeholders are interacting in the process of learning and decision-making. As M&E system effectiveness itself was difficult to define and seems to be an abstract, it was then better, in our view, to look to the existing literature that takes into account the various perspectives in characterizing an effective M&E System worldwide and apply it to the context of West Africa. Data are collected using both structured and semi-structured tools through qualitative and quantitative methods. Data collection tools were developed in line with the research question and sub-questions. Data collected through literature and documentation review approach involved the review of relevant Development Management, Performance Measurement and related fields articles and scientific reports published in scholar journals and development platforms, to generate documented evidence on the effectiveness and robustness of the model selection and testing processes. A structured literature review guide was used to summarize information generated from the examination. For the case studies in Senegal, Mali and the West Africa region (five countries), qualitative approaches are applied to enable probe in-depth and to obtain information for validating observations and findings gathered through document reviews. Semi-structured interviews with the major informants identified from the various development programs in the region were conducted. Selection of the individuals is based on their knowledge and influence related to the specific research questions and components of the measurement framework proposed following the literature review. Also, the selection is made to ensure a diverse and representative range of stakeholders. A quantitative approach was used for the third iteration through a formal survey based on a structured questionnaire. Selected development programs were asked to respond to the survey through a web platform offered by the Business Science Institute partner Sphinx. The selection of development programs for the survey was based primarily on the criteria of coverage and country of the intervention as whether the program is regional, national or sectoral, then the criteria of domain of intervention was added to make sure that the most relevant development sectors aligned with the MDGs/SDGs and international development 69

78 objectives and country frameworks are taken into account. The survey was expected to cover fifty projects/programs to ensure a good sample. Generally, M&E Manager, Program Manager, M&E staff, partner representatives and government counterparts were a target in priority to ensure meaningful feedback on the effectiveness of their M&E Systems as per the proposed measurements in the framework FIRST ITERATION: REVIEW OF THE DIMENSIONS OF EFFECTIVE M&E SYSTEM The first iteration of the research approach revisited the most relevant literature on Development Management and Performance Measurement System, to consolidate and initiate the model for effective M&E-System. While a literature review is never exhaustive, this iteration focuses on the essentials of the literature on effective M&E-System both from journals articles and development institutions platforms and professional productions. This review analyzed the Monitoring and Evaluation System as an Integrated Information System (IS) for Development Management and Performance Measurement System. M&E System is analyzed as an Information System, as to quote Tom Barton again, The M&E system is a form of information system, which is a broad term for information selection, gathering, analysis and use. It can be described as a logic chain of linked ideas starting (and continuing) with information users. (Barton, 1997:9). The six dimensions of the updated D&M Information System Success Model (DeLone & McLean, 2003) - that are: System Quality, Information Quality, Service Quality, Usage, User Satisfaction and Net Benefits - guided the design of the research model for Effective-M&E-System. That helped generate a well-structured model for the next steps of the research. The model was operationalized into pillars and sub-dimensions based on the general approach of D&M IS Success Model that was also operationalized into indicators and measures based on the findings from the literature review in the components of Effective M&E System. The measurement of Information Systems (IS) success or effectiveness is critical to our understanding of the value of efficacy of IS management actions and IS investments. (DeLone & McLean, 2003:238). Relevant articles were selected as well as relevant technical reports, conference procedures for the main M&E learning events the last twenty years in the world of Development Management. Findings from these documents were critically analyzed to support different conclusions in the building of the comprehensive proposed M&E System measurement framework. This rigorous approach to review and analyze existing scientific and professional findings in the M&E System literature is a critical learning process in the building of effective M&E System measurement framework as relating to the D&M Information System Success Model - Intention to Use-Use-User Satisfaction dimensions. During the last decades, many discussions took 70

79 place before and after the Paris Declaration on Aid-Effectiveness in 2005, on whether M&E Systems are effective and how to improve their work for better decision-making in Development Management. The research explored in this first iteration the M&E-System in the development world from a historical view. There are many stories about M&E-System emergence and becoming recognized as a strong system for decision-making in Development Management. Many of those theories are donors-led or development organizations or expert-based views, each donor and group of development partners trying to build a story which is showing its important role in the origin and definition of the M&E-System as a development management tool for decision making and measured risk-taking. The literature review analyzed the three broad phases of M&E-Systems in the post-world War II Development Management approach in mainly the developing countries, from the first stage of Monitoring and Evaluation in Africa prior and during the rise of Logical Framework approach in the 60s. A focus is put on the first stage of M&E-System development that was mainly oriented on what we call in this research the Stage of Network-Chart M&E System focusing more on knowledge and information related to development programs implementation, program operations, timelines and use of technical and financial resources related indicators - in short, an input-output dominant approach. Many developing countries and regional bodies, mainly in African historical development stage in the 50s and 60s during the productivity-led programs in the development field, prioritized the building of infrastructures, other power plants, etc., in the national and regional development frameworks. This strategic intent infrastructuresorientation influenced the M&E-System settings and functioning during that period. From the early 60s to the end of the 70s, those development programs operations-led M&E-Systems dominated the development field to become what we called the Stage of Network-Chart M&E System as shown by the findings from the literature and documentation review. In the same iteration and approach, the review revisited the crucial period around the initial emergence of Logic Models and Development-change framework methods. The findings under this examination helped better understand the pre-berg dominant period of productivity of development programs mainly outputoriented and more focused on financial flows, operations and products and services delivery without a critical attention to the So what? question that requires a rigorous assessment of significant sustainable results of the development actions implemented. We looked then at the second phase of the M&E-System development, which was analyzed in the literature that relates to the rising in the early 80s of an excessive impact assessment approach supremacy and a vigorous rejection of the network chart inputs-outputs-led approach in M&E Systems in the development field. That period coincided with the nascent Washington Consensus and the Berg report that inspired and initiated massive structural adjustments in the developing countries. Following decades of implementation of 71

80 productivity-led programs in the 60s and 70s without strong organizations and institutions mainly in Africa, welfare and livelihood were not recorded at the level of the populations and communities and levels of initial expectations. Instead, poverty, scarcity and weak institutions were the growing perceptible reality in Africa. As shown by the analysis of the review, from the early 80s, attention was more put on development impact that can be measurable and evidenced. Any funding of development actions should have been supported by strong evidence on the impact and intermediate results targeted. That period marked the beginning of what we called the Stage of Clinical-Trial-Methods of impact measurement theories. Each development organization and institution was investing resources in research and impact evaluations mainly using counterfactual experimental or quasi-experimental approaches to demonstrate attribution of any single minimum change observed in the area of intervention, to its development programs or programs implemented through its supports. Also, as we were moving to the initial M&E System measurement framework, we analyzed in the literature the third phase of M&E-System development, that is what we called the Nascent Stage of Web-Based Information Systems that took rise from the MDGs early 2000s and the booming development of Information Technologies in the world. In addition to that, as shown by the review, the new IT-applications through smartphones and androids that initiated easy access web solutions, inaugurated a phase where M&E-Systems are dominated by IT-Systems. Almost all organizations and institutions were working through Information Technology systems, to show in real time their results and learnings worldwide and then putting more attention on their visibility than the actual effects of the development programs actions on the targeted population wellbeing and livelihood. The evidence-based decision-making approach through this third-stage suffered from simple communication tools and IT-misleading instruments in place for optimum knowledge sharing and learning to foster improved risk-taking in managing development programs mainly in Africa. The review helped analyze the growing need for M&E-System to generate more meaningful information for decision-making to advance development in Africa. In this iteration, in its last sub-section, through a deep analysis from what was learned in the literature review, we mapped the different components of effective M&E-System and designed the research model that served the basis of testing and applying the M&E-System effectiveness in the next iterations. M&E System effectiveness is a broad concept of decisionmaking tool used by development institutions and managers for better learning and improved risk-taking in managing development programs. Hence, as stated in previous sections, M&E System effectiveness should be found in the capability of the organizations and institutions for effective program management to advance development. A mapping of the different descriptions from the analysis of effective M&E-System effectiveness with the organizations and institutions improving capabilities has shown that Effective-M&E- 72

81 System core pillars are: 1) Results-Based Management practice, 2) Knowledge and Information Sharing Culture, and 3) Evidence-Based Decision-Making Practice. In this last section of the iteration, the approach revisited the literature in these three dimensions of an effective M&E-System materialization in the hosting organization or institution. How these three dimensions are unfolding progressively in an organization or institution hosting a more and more effective M&E-System is analyzed, and findings added to the description of development programs M&E-System effectiveness. This process of the construct of the dimensions and sub-dimensions of effective M&E System led to an in-depth review of existing knowledge in the three capabilities of organizations hosting such systems to generate the real scientifically valid measurements on 1) Results-Based Management practice, 2) Knowledge and Information Sharing Culture, and 3) Evidence-Based Decision-Making Practice. The analysis of the M&E-System Net-Benefits and how they apply to the Development Management field led to a discussion on improved development policies design and formulation, improved tactical, operational and strategic measures during development program implementation; and finally, to a discussion on the contribution of Effective M&E-System to the capability of organizations and institutions to advance development objectives that would positively change people welfare. These findings grounded a future field of research on The Impact of Effective M&E-System on Development that is not in the scope of this research SECOND ITERATION: CASE STUDIES The second iteration is a qualitative observation of three case studies based on a representative sample of development programs selected from our experience within West Africa and focuses on how M&E-Systems work in the field of development in local, national and regional levels programs. While the first iteration focuses on the dimensions of effective M&E System using the D&M IS Success Model components on Intention to Use-Use-User Satisfaction, this second iteration is to operationalize in the M&E field of the three dimensions: System Quality, Information Quality, and Service Quality. The cases we ve selected are based on our experience in the West Africa region, and the understanding of the critical phases of the M&E systems analyzed, and that understanding would help clarify what are the measurements to define as a proposed operationalization of the three dimensions of M&E System effectiveness to be specified: System Quality, Information Quality, Service Quality. An approach would be to select cases in the West Africa region randomly. Indeed, that might lead the research to examples not relevant necessarily to the topic we want to review. Another approach would be to look at the existing documents on cases or to follow experts and managers recommendations to select the cases. While those 73

82 approaches would increase neutrality on the selection of the cases in this research, they would not be relevant enough to show what we need to capture in this process. We opted, as to refer to the objective of the building of knowledge from our experience, to select cases we know in practice and where we were involved in the process as M&E Expert Team Leader or M&E Expert Team Member. This approach provides many opportunities to access to a various array of data and other relevant information on the processes. Also, having worked with those institutions at the time they were addressing, in their M&E systems, issues that relate to the specific topic we want to analyze, guaranteed the possibility of this research to access to and consult with the relevant individuals for interviews and facilitate feedback over the course of data analysis. The types of data we accessed to are: the notes we ve taken during our assignment to the institutions in various periods ranging from 2009 to 2014, the notes from our partners and other teams members, documents on programs published by the organization, M&E manual and procedures, M&E reports, job description and performance assessment of relevant M&E staff and partners, program evaluation reports, field visits reports, various internal data generated by the M&E systems, reports to board or executive meetings on the M&E information and decision-making processes, development program documents within the organization and donors procedures, partnerships agreements, etc. From all these data, we just selected what is relevant for our research related to the M&E system effectiveness model we are designing. While these are not rigorous standard case studies known in the field of research, they helped explore the reality in the M&E practices of organizations in the West Africa region when they were addressing issues related to their planning, learning, and decision-making processes. In analyzing the measurement that would capture the Information System - System Quality dimension of an effective M&E System, we studied the Office Du Niger Contact Plan M&E System. As explained in the next chapters, we were involved as M&E Expert Team Leader for the design and operationalization of the Office du Niger M&E System, as member of a team of four experts mobilized by the French development firm SOFRECO to support the organization through a funding from the Netherland Cooperation, in restructuring activities and improvement of governance procedures in order to move with the partners to a more effective budget support to develop irrigation systems in Mali and the West Africa region for better food sufficiency and security. The process of designing of the Office du Niger M&E system is an important process of designing M&E System because it involved several Ministries sections, local government in Segou, the various zones of the irrigation system, and all development partners involved in the programs. For fourteen months, we worked with a team of experts and the Office du Niger Direction of Planning and Statistics to design within a participatory approach one on the biggest M&E system of the agriculture and food security sectors in the West Africa region. We accessed and capitalized much information from our 74

83 literature review of the M&E system and history of the Office du Niger to the initial assessment made with various stakeholders. We accessed a lot of interviews summary during that period and summarized discussions on the evaluation reports. Also, we accessed to the M&E Procedures and Manual designed at the end of the mission. We accessed to relevant documentation on Office du Niger Contract Plan and various cooperative agreements signed with the development partners as well as Office du Niger organization and coordination system. There was a relevant documentation we were able to access during the process, in addition to that, we organized remote consultations with the Direction of Planning and Statistics responsible for the M&E System. We used mainly a content analysis of the various reports and interviews guides as well as triangulation to generate the findings on the measurement relevant to build the System Quality dimension of the effective M&E System as per the hypothesis based on the program logic change. The measurements identified to be tested were presented at the end of the case study. We looked to the different sub-dimensions that would make the System at high quality from its design to its operations and maintenance. Based on the findings from the case, we were able to develop a commensurate measurement framework that would help to capture this dimension. The same approach was used in studying the M&E System of the National Agency for Rural Electrification in Senegal (ASER) which is in charge of the rural electrification program of the Government of Senegal. The ASER program is one of the biggest in the country and is mobilizing all departments of the Ministry in charge of Energy and the major development partners working in the country. The ASER M&E System was studied to clarify and define strong measurements for the Information Quality dimension of the effective M&E System. From our experience in the West Africa region, the ASER M&E System was selected based on the work made in the agency to generate information for various stakeholders involved in the rural electrification program which is a national development program. We led a study in ASER in 2012 to help improve how the agency s M&E System was organizing the information managed by various stakeholders involved in the program implementation. That mission led several assessments and organized various workshops on the M&E system information linked to the program s logical framework and needs of the decision makers such as the managers, the government, and financial partners as well as the private sector responsible for the implementation in various regions of the country. The various documents and interviews conducted during the assignment as well as recent interviews with the main technical staff in the agency, help generate relevant information that is scrutinized using content analysis and mapping of key sub-dimension of Information Quality in the context of effective M&E System of the ASER rural electrification programs. This systematic approach guided the case study and provided 75

84 the findings on what would be the measurements and variables for the sub-dimension Information Quality of the M&E System effectiveness proposed model, to be tested in the third iteration. The third case we studied in this iteration relates to the Regional Program Water and Food Security Initiative (IESA) in Africa funded by the Spanish Cooperation (AECID) and implemented by the United Nations Food and Agriculture Organization (FAO). We led the FAO regional M&E system including IESA more than three years and capitalized much information on the M&E System and processes of generating and sharing reports and other M&E information with the program s core stakeholders in the West Africa region. Because the program was regionally implemented in seven countries, creation and distribution of M&E information within all interested parties at national level and local level as well as at regional level at FAO West Africa Office and in relation with other FAO Offices in other regions of the African continent, and with FAO headquarter in Rome, enabled to understand how M&E System is delivering its products and services to satisfy decision- makers needs. Using this case, we analyzed the sub-dimensions of Service Quality of an effective M&E System. Being at the heart of this process was a critical advantage in this research as we have capitalized a lot of information and stakeholders feedback on how the M&E System was delivering its services for improved decision making on water and food security in the different countries covered by the program. Main documents consulted include the program documents, description of logic changes in the various countries, the initial assessment of the M&E system and the reports from the Ouagadougou workshop in 2010 which launched the M&E system services definition and clarification of procedures, roles and responsibilities of various stakeholders involved, the different reports and dashboards produced by the programs M&E Officers in the different countries, M&E trainings and meetings reports, field visits reports in seven countries and various project localities and targeted regions in each country, and various interviews forms used with the relevant M&E Officer who worked in the program from different Ministries of Agriculture in the countries covered by the activities. We used content analysis and other qualitative analyses of all the relevant information to generate findings on what would be the relevant measurements and variables that assess the sub-dimensions of the Service Quality of the FAO IESA M&E System. The measurements and variables generated were discussed and analyzed at different levels of countries and stakeholders to design a rigorous set of assessment criteria for effective M&E System Service Quality that might be generated when tested in the third iteration. In short, this second iteration, through relevant cases selected from our experience in working with M&E Systems in the region, helped design key sub-components, measurements and variables that served as criteria for testing in the third iteration. The design of a model, Maneen (1983), cited by Savall and Zardet in The Qualimetrics Approach Observing the Complex Object (2011), defined qualitative methods as 76

85 a palette of interpretation techniques that attempt to describe, decode, translate, and, in general, to be coherent with the significance of the phenomena. (Savall & Zardet, 2011). This second iteration of the research, as explained here, focused on programs in critical development areas including agriculture, food security, energy, and other social sectors issues that occupy a central place in developing countries development strategies and policies, selected from our experience in the region. These development programs M&E-Systems are analyzed in each case as an integrated information system devoted to improved decision-making. M&E-System design and implementation are closely linked to the country or the region development status specificities or context. In West Africa, poverty and lack of applied best practices of governance might hinder the effectiveness of M&E-Systems. The different actors involved in the M&E-Systems at country and regional levels and the various approaches used by Development Managers also can lead, in a research process, to multiple sources of information on how the systems work. Case Study approach is then an appropriate option for the second iteration of this research based on our practice in scrutinizing how effective M&E-Systems are in the West Africa region. A case study is an inquiry that studies a contemporary phenomenon in its daily context, when the limits separating the phenomenon from its context are unclear, in the course of which multiple sources and proof are utilized, and which take advantage of previous theoretical developments to guide the gathering and processing of data. (Savall & Zardet, 2011:25). Robert K. Yin clarifies, in his fifth edition of Case Study Research- Design and Methods (2014), that, articulating a theory about what is being studied and what is to be learned helps to strengthen a research design when doing case study research You can then examine the quality of your emerging design in relation to four tests commonly used in social science research: (a) constructive validity, (b) internal validity, (c) external validity, and (d) reliability. (Yin, 2014:28). He also points out that multiple-case study designs have increased in frequency in recent years, but they have advantages and disadvantages in comparison to single-case study. For this research, we used a multiple-case study design to cover a good spectrum and representative development programs and M&E- Systems in the West Africa while we were implementing the analysis of different dimensions of effective M&E System dimension as an integrated Information System. 77

86 5.3. THIRD ITERATION: SURVEY - HOW TO ASSESS AND APPLY THE M&E-SYSTEM EFFECTIVENESS MODEL? The third iteration used a questionnaire produced in line with the M&E-System effectiveness measurement framework developed, its dimensions and sub-dimensions, measures and variables proposed. The questionnaire was built around the key components of the M&E-System Effectiveness model we designed and raised in the two first iterations as a powerful result of the combined outcomes from findings in Iteration- 1 and Iteration-2. For the M&E-System effectiveness model Component related to Information Quality, findings from the case studies were used to develop the different levels of measurement for this dimension of the model. Information Quality, for example, generally relates, as shown through the analyses, to the importance of linkages between technical deliverables, products and services, and financial flows in development program management to help better learn on the program efficiency and effectiveness; It also seeks to track and assess information on the capacity of Managers to translate all contributions from beneficiaries into financial data that can be merged with those of the program through the regular funding mechanism. The questionnaire was applied to major existing programs in the West Africa region and assesses how they are applying Monitoring and Evaluation System and how effective are those Monitoring and Evaluation Systems regarding the model designed in this research. The aim was to test the applicability of the model through a sample survey to verify it with existing programs at regional, national and local levels and to learn from the results how the framework functions and what needs to be improved in the final versions. Also, this survey, which was based on a saturation approach in sizing the sample, helped provide different sets of the framework that integrate the specificity of the development programs and the aim of their M&E-Systems within the hosting organizations and institutions. Interviews were organized through face to face or Skype meetings and retrospective analysis throughout the region and also, mainly with online questionnaires through Sphinx Declic. Based on the main findings from the literature review and mapping of Monitoring and Evaluation experiences at different levels in the West Africa countries, core sub-elements in each dimension of the D&M IS Success Model were validated in a first section through the survey questionnaire. The questionnaire was modeled with 5-levels Likert measures for each criterion of the dimensions of M&E- System Effectiveness measurement framework. Dane Bertram, citing Uebersax clarified that Dr. Rensis Likert objective was to create measures for psychological attitudes that can be interpreted as metric scale (Bertram, 2007). 78

87 For each measure of the Likert scale used, it was set a benchmark of the highest value, clearly announced and explained based on M&E-System effectiveness criteria and indicators identified in the previous iterations. The use of this benchmarking approach helped guarantee that the Likert scales are applied in a very consistent way that ensures high-quality information, the reliability of scores selected and minimum subjectivity in the responses. The benchmarks were built during this iteration 3, to ensure clear description of the highest value (5) of the scales. A description was provided based on iteration 1 and 2 analyses and findings. Data were analyzed using statistical software including Sphinx and SPSS as well as Office package (Excel). Data analysis led to the verification of the applicability of the model and how it operates at different levels of organizations and institutions: regional, national and local. The analysis also led to findings that constitute the basis for providing different adaptations of the model depending on the level/depth of the M&E-System effectiveness assessment and the sector, area of the development program. This iteration-3 led to robust discussions on how M&E-System effectiveness will help organizations meet their highest potential and capability to advance development through improved Results-Based Management, Knowledge and Information Sharing, and Evidence-Based Decision Making. In conclusion, the research approach used a mixed-method with three iterations. The first iteration, the literature review, which is a systematic review of the existing knowledge in M&E System as a strategy in managing development actions, focused on the essential publications and professional reports on the topic. It used tools related to documentation review and content analysis. The second iteration, the case studies, used a qualitative approach in collecting analyzing the data from interviews and documentation and various reports of the cases selected. The cases are selected from the researcher experience in the West Africa region, and the relevant linkages of the cases with the dimension of effective M&E System studied. The third iteration is a quantitative survey based on a structured questionnaire that helped gather data online from sampled development programs in the West Africa region. The data were analyzed using different quantitative techniques including multiple correlations analyses, and multiple regression analyses. The approaches and methods for data collection and analysis will be presented in detail in chapter 7 (Research Methods and Techniques). 79

88 CHAPTER 6. RESEARCH METHODS AND TECHNIQUES For each iteration, commensurate methods and techniques were used to collect and analyze the data. These are described in four building blocks: 1. Objective of the iteration, 2. Methods and techniques, 3. Units observed, and 4. Data collection and analysis methods. Four all the three iterations: literature review, case studies, and survey, the objective was explained in Chapter 2 (2.4. The Research Purpose and Questions). This chapter 6 (Research Methods and Techniques) presents the research methods and techniques for data collection and analysis. In its last section, Chapter 5 presents the main aspects of the field work in iteration 2 (6.2. Methods and Techniques for Iteration 2) and 3 (6.3. Methods and Techniques for Iteration 3) METHODS AND TECHNIQUES FOR ITERATION 1: DIMENSIONS OF EFFECTIVE M&E SYSTEM As described in previous chapters, the main objectives of this iteration were to determine: 1. What are the key relevant lessons and knowledge from the M&E System history? 2. What are the main steps and different models used for M&E System preparation and implementation? 3. what would be relevant IS success framework that might guide the building of the M&E system effectiveness framework? The approach used here is mainly based on literature review and content analysis. Online research of articles related to the fields of Development Management and M&E System through the libraries of the Universities of Lyon 3 and the UTS, in addition to the articles shared by the professors during various seminars, were the main sources of information. Some key professional sources including the OECD/DAC, WB, and the UN System resources were also used as a source of documents and technical reports relevant to this research. A first mapping (Table 4) was used to classify the different domains of the documents (articles and reports) collected. The mapping of the relevant articles and reports used the criteria in the table below. The criteria were selected on topics relevant to the research questions and domain ranging from development theories and institutions and management, project management, monitoring and evaluation systems, information and knowledge management, evidence-based decision-making and resultsbased management. Table 4. Mapping of relevant articles and reports Domain Project Management, Aid effectiveness and MDGs Title and Author Date of publication Main findings relevant to the research Summaries Quotes Related literature and research question 80

89 Domain Evidence-based decisionmaking Project Management, Aid effectiveness and MDGs Development Programs Evaluation Information System Theory Institutions and organizations Theory and Development Management Information and Knowledge Management M&E and Performance Management Results-based Management and Public Performance Management research theories and models Title and Author Date of publication Main findings relevant to the research Summaries Quotes Related literature and research question In summary (Table 5), 648 documents, mainly articles, and reports were scrutinized to gather the needed information and findings on the relevant topics to the research. Table 5. Documents consulted in the literature review Domain Number of documents consulted Project Management, Aid effectiveness and MDGs 43 Evidence-based decision-making 42 Development policy design 206 Development Programs Evaluation 46 Information System Theory 13 Institutions and organizations Theory and Development Management 50 Information and Knowledge Management 43 M&E and Performance Management 63 Project Management 9 Results-based Management and Public Performance 73 Management research theories and models 60 Total

90 6.2. METHODS AND TECHNIQUES FOR ITERATION 2: CASES STUDIES The main objective of this iteration 2 is to respond to the second sub-question of this research: What are the key components (and variables) at different levels of the Monitoring and Evaluation System effectiveness model relevant in studying M&E-System role to the effective evidence-based decision making in advancing development in West Africa?. As described in previous sections, the approach used is based on the analysis of selected cases in West Africa that are relevant to the three dimensions of the M&E System as an integrated information system: Information Quality, System Quality, and Service Quality. The cases were selected from our experience in the development field in the region, through a reasoning approach. The main criteria were the coverage and domain of the project and the level of development and/or implementation of the M&E System. The programs should cover main development priority in the region including infrastructure development, agriculture, etc. the regional programs should cover at least three different countries in West Africa. The programs should include the main levels of effective coordination of development actions, namely the regional and national levels. The programs should cover main development partners in the region. From our experience, we have a list of development projects and programs we ve supported the M&E system, that includes: 1. The Program for Development of Handcraft Businesses in the Northern of Senegal (PROMART-SN). The program target group was the handcrafts organizations. Its main objective was to foster business development and increase incomes in the rural areas of the northern of Senegal. It was funded by the EWA (ENTWICKLUNGSWERKSTATT AUSTRIA and the Government of Senegal). 2. The National Program of Rural Infrastructures in Senegal (PNIR-Senegal), which the main objective was to reduce poverty in rural areas of the country through the development of rural infrastructures in a decentralized participatory approach. The program was funded by the Government of Senegal, the World Bank and the International Fund for Agriculture Development (IFAD). 3. The Senegal Producers Organizations and Agricultural Services Program (PSAOP-Senegal) funded by WB, IFAD and the Government of Senegal, which the main objective was to improve agricultural productivity and food security in the country. 4. The USDA Food for Progress Program and USAID Food for Peace Programs Bamtaare Fuuta Tooro, in the northern of Senegal, that aimed at reducing child mortality through improved 82

91 food and nutrition access and maternal health in the northern of Senegal. The programs were funded by USDA and USAID. 5. The Economic Growth Project, funded by USAID which the main objective was to develop agriculture value chains and increase businesses and income of the farmers and household food security in Senegal. 6. The Office du Niger Contract Plan Program in Mali multi-donor project and implemented by the government of Mali through the ON, which objective was to ensure food self-sufficiency in the Mali and West Africa countries. 7. The Pastoral System Support program, in Chad, which main purpose was to develop livestock production, increase farmers income and improve food security in the country. The program was funded by the African Development Bank. 8. The Rural Electrification Program in Senegal funded multi-donor and implemented by the National Agency for Rural Electrification (ASER) on behalf of the Government of Senegal, the Ministry in Charge of Energy. 9. The Water and Food Security Program in West Africa, funded by the Spanish Cooperation (AECID) and implemented by the Food and Agriculture Organization of the UN (FAO), which objective was to ensure food security through the development of agricultural irrigation in 7 countries of the West Africa region. 10. The National Program for Food Security (NPFS), in Nigeria, funded by the Government and implemented by FAO. The main objective was to improve food security in the country. The first step was to select with well-defined criteria (Table 6) among these cases, the most relevant to the research that will help gather relevant information on the dimensions and components of the M&E System measurement framework. The result is described in the table below. Table 6. Criteria for selection of the case studies Case selected (3) Criteria for selection Relevance to this research Dimensions treated with the case Data collection and analysis 1. The Rural Electrification Program in Senegal 1- the program is recent ( ) 2- the Program M&E System focus is relevant as it relates to how it operates in The program is relevant as its M&E System is a good case in the country according to many development stakeholders The main dimension of the M&E System effectiveness measurement treated with this case relates to the Information Documentation review Literature Content analysis Interview with core stakeholders 83

92 Case selected (3) Criteria for selection Relevance to this research Dimensions treated with the case Data collection and analysis managing the information 3- the program focus in national as it covers the country wide In addition, the program M&E System focuses on the approaches in information collection, organization, and management Quality dimension What are the key elements and criteria to consider in assessing the Information Quality of an M&E System? 2. The Office du Niger Contract Plan Program in Mali 1- The program is recent ( The program touches several development priorities in West Africa: water irrigation, agriculture, infrastructures, decentralization, poverty reduction, food security. 3- A critical phase of the project M&E System development was analyzed, the conception phase to scrutinize how the System Quality is gauged and what can be considered as key measurement in this component In , ON entered a critical phase of reviewing its M&E System to align with new standards of the various partners towards a budget support approach and ensure improved transparency, governance, and Results-Based Management The program s M&E System is a great case in studying the main measurements to adopt with this dimension of M&E System effectiveness model: System Quality The main aspects treated are related to the various stages of M&E System design, preparation, maintenance and operations (resources and techniques) For each stage, variables and measurements relevant to the assessment of the effectiveness of the System Quality are progressively selected Documentation review Literature Content analysis Interview with core stakeholders 3. The Water and Food Security Program in West Africa 1- The program is recent ( ) 2- It is a regional program touching seven countries in The program involves several stakeholders at various levels: regional, national and local, it required a good The Service Quality dimension is treated with the case as it analyzes the key products and services Documentation review Literature Content analysis 84

93 Case selected (3) Criteria for selection Relevance to this research Dimensions treated with the case Data collection and analysis the West Africa region 3- It is implemented by a UN agency (FAO) in relation with Ministries in charge of Agriculture and Irrigation Systems 4- Its main objective was to ensure food security in the West Africa region organization of tailored products and services in terms of information and knowledge sharing. The program s M&E System is a good case in analyzing the Service Quality of the M&E System effectiveness framework delivered by the M&E System The case analyzes the M&E System deliverables to different stakeholders, the M&E System responsiveness and flexibility in regard to the various demand of information and knowledge, to guide evidencebased decisionmaking process Interview with core stakeholders Data collection focused on documentation review and interviews with core stakeholders in the three cases. The list of documents consulted and stakeholders interviewed is in Annex 6 (A.1. List of documents reviewed and stakeholders interviewed for the case studies). Systematic analysis of various documents on the cases was made to gather relevant findings on how to measure these dimensions of M&E System effectiveness. Interviews with core stakeholders were done during the different phases of our interaction during our experience with the programs. A presentation of the three programs and relevance of their selection as cases are made in the Annexes: the Office du Niger M&E System (A.3. Introduction to Office du Niger Contract Plan Case 1), the ASER M&E System (A.4. Introduction to the ASER Program and Monitoring and Evaluation System Case Study 2), the FAO-IESA M&E System (A.5. Introduction to the IESA Program and Monitoring and Evaluation System Case Study 3) METHODS AND TECHNIQUES FOR ITERATION 3 The third iteration objective is to respond to the sub-question c- What is the meaningful scientific acceptable framework to measure the effectiveness of Monitoring and Evaluation System, through the operationalization and testing of the initially proposed framework? of the research through a well- 85

94 organized survey that covers important development program in the region. The sub-questions were operationalized into variables and measurement translated into the questionnaire designed for the survey Design of Data Collection Tool for the Survey The main tool for data collection designed is the survey questionnaire which was based on the proposed framework/model and results from the case studies. Hence, the questionnaire is composed of different sections that are commensurate with the M&E System effectiveness framework proposed, namely: M&E System- System Quality M&E System-Information Quality M&E System-Service Quality M&E System Effectiveness- Results-Based Management Practice M&E System Effectiveness- Knowledge and Information Management M&E System Effectiveness- Evidence-Based Decision-Making M&E System Net Benefits- Improved Policy and Program Design M&E System Net Benefits- Improved Operational-Tactical-Strategic Decisions M&E System Net-Benefits- Improved Capability to advance development. For each section, relevant variables were selected based on the results of the literature review in iteration 1 and case studies in iteration 2. For the variables, Likert-Scores were created to measure the level of ranking by the programs and stakeholders interviewed. The Likert Scores (Very Poor, Poor, Average, Good, Very Good) selected to test the validity of each component of the M&E information system model as scales variables are interval measures and linked to a clear rating system to avoid any subjectivity in the answers. They are defined as follows: Very Poor corresponds to 20% or less for percent/ratio variables - 20 or less than 20 cases for flow variables 20 or less in a ranking system from 0 to 100 for stock variables, Poor corresponds to 21% to 40% for percent/ratio variables 21 to 40 cases for flow variables 21 to 40 in a ranking system from 0 to 100 for stock variables, Average corresponds to 41% to 60% for percent/ratio variables 41 to 60 cases for flow variables 41 to 60 in a ranking system from 0 to 100 for stock variables, Good corresponds to 61% to 80% for percent/ratio variables 61 to 80 cases for flow variables 61 to 80 in a ranking system from 0 to 100 for stock variables, Very Good) corresponds to 81% to 100% for percent/ratio variables 81 to 100 cases for flow variables 81 to 100 in a ranking system from 0 to 100 for stock variables. 86

95 For each variable/question, scientific benchmarks were defined to assess the ranking and responses proposed by the respondents. The variables/questions in the questionnaire for each dimension of the M&E System framework are presented in Annexes (A.6. Questionnaire for Iteration 3 Survey). The content of these tables was used to design a user-friendly web-based questionnaire for the survey. The different variables and benchmarks reflect the conclusions and findings from iteration 1 and iteration 2. These set of variables and benchmark are the proposed measurement of the M&E System effectiveness model, to be tested through the survey with relevant representative programs in the West Africa region Sample Design for the Survey The survey universe is the development programs and projects in the 15 ECOWAS countries, namely: Benin, Burkina Faso, Cape Verde, Côte d Ivoire, The Gambia, Ghana, Guinea-Bissau, Guinea-Conakry, Liberia, Mali, Niger, Nigeria, Senegal, and Togo. The development programs targeted the following development sectors: Agriculture, Infrastructures, Energy, Poverty - Social Impact, Gender Empowerment, Governance and Democracy, Food Security, Rural Development, Education Research, Health, Trade, and Investment, or other relevant development sectors. The survey focused on development programs that have reached the midcourse of their implementation or are close less than three years ago, to make sure that critical M&E activities have been already or are supposed to be implemented, and the memory of the M&E system is still alive to allow relevant data collection. Hence, the universe was limited to projects and programs that have reached mid-course or closed less than three years ago. At the project/program level, the respondents were part of the management team and/or M&E team, technical partners or government staff members of the program s steering committee to make sure relevant people who know the program M&E system are interviewed to respond to the questionnaire. The list of program and projects which have responded is in the Annexes (A.2. List of programs/project surveyed in West Africa) Data collection and analysis approaches The questionnaire was designed in the SPHINX-DECLIC platform and sent to approximatively Sixty (60) development programs in West Africa, selected based on the criteria described above. Forty-Six (46) responded to the questionnaire which is a rate of 76,66% of responses. To reach this rate, many approaches were used. The follow-up notices integrated into the Sphinx web-based software helped recall some respondents and get positive feedback. In some countries, such as Senegal and Burkina Faso, where it was 87

96 initially difficult to get the responses online, face-to-face interviewed were organized as well as remote interviews to complete the questionnaires and update the Sphinx database. Globally, the data collection went very well, except in some cases the respondents complained about the length of the questionnaire. The quality of the questionnaire was appreciated, the feedback received will be analyzed in the results section. Data collected were scrutinized for quality check and cleaned when needed in relation with the respondents (the questionnaire contained their Skype and telephone contacts). Data analysis started with a well-organized and clean database. Overall the quality and representatively of the information were very good. Data analysis was done in Sphinx and SPSS partly and Excel. Correlation analyses and correspondence analysis were conducted with Sphinx to analyze the relations between and within components of the M&E System effectiveness model. Once the duplications and correlations were identified, the components were reorganized with variables that are independent and representative of the measurements needed. Then, we run regression analysis to analyze the functional and explanatory relations between the core components of the M&E System effectiveness framework. Finally, a revised model was proposed based on the results of the iteration3. 88

97 CHAPTER 7. THE PROPOSED FRAMEWORK FOR EFFECTIVE M&E-SYSTEM This Chapter presents the results of the literature review in responding to the first research sub-question: How to build a commensurate framework in studying the effectiveness of development program M&E- System as part of Development Management strategy? It looked at the literature of relevant researches, how to translate existing and new knowledge on Information System into a commensurate model that will serve as the primary hypotheses to measure M&E System effectiveness. The Chapter shows the results and analyses how to understand better an effective M&E System and what would be the proper scientifically acceptable framework to track its effectiveness and success. This chapter also presents the analyzes of the existing knowledge in the development field related to effective M&E System and practices in various development programs fields to generate a proposed model for effective M&E System measurement framework. It presents progressively what is effective M&E system and its different dimensions as findings from the review. In between the stage of output-oriented, charts-networks performance measurement approaches, the stage of quantitative impact measurement methods, and the web-based application M&E systems promoted by the international organizations and academic researchers in the aid industry field, in general as conditionality in financing development and guidelines for managers, suffered subsequently the development programs comprehensive effective M&E systems for improved decision-making. The long run dispute between these sides of performance management and effective results-based M&E, hindered clear an appropriate conceptualization, development and set of effective M&E systems in the Aid Industry field mainly in developing countries. The strength of effective M&E is in bridging the gaps between these different stages of the M&E System history (4.5. Conclusion on Literature Review of Monitoring and Evaluation System). From the institutions and organizations designing and managing M&E Systems for improved policy-design and decision-making towards development objectives success, the analysis of the role of M&E system is critical to understand better how any effectiveness of such system would be measured. One critical consideration is that organizations and institutions in charge of development programs need strong systems for a results-based planning and management approaches to better focus appropriately on tangible development objectives. A better results-based approach sustained by an effective M&E system is critical to the organization to ensure a proper process for improved policy design and development programs implementation. As we argue here, a strong results-based, knowledge and information management, and evidence-based decision-making approaches are key to organizations and institutions in charge of development programs, 89

98 and M&E Systems are important feeding elements to planning, implementing and managing development programs, and learning processes. An effective M&E System would then have a positive impact on the organizations practices of results-based management, Knowledge and Information management, and evidence-based decision-making. As effective M&E System is unfolding, the improved practice of development management capabilities will be a core for organizations and institutions to advance development objectives. There is then a clear and direct linkage between the results-based management, Knowledge and Information management, and evidence-based decision-making cultures at levels of institutions and organizations in charge of development actions, and their M&E System effectiveness. An effective M&E System should impact positively on the organizations capability to manage a sound and reliable learning system towards the attainment of the desired development goals. The learning and information management system is highly linked to the effectiveness of the M&E System hosted by the organization in charge of the development programs. An M&E System is then a learning platform for improved decision-making to advance development objectives. In short, as a learning platform, it should then contribute to the organization capability to better manage knowledge and information. Effective M&E System would then be found in the effective management of knowledge and information for a better learning for improved decision-making at organizational capability level. Making better decisions is key to the organization managing development programs in more and more difficult internal and environmental contexts. The learning process would contribute to the capability of the organization to improve its evidence-based decision-making culture. Taking decision based on reliable evidence generated through the effective M&E System is one of the key objectives of the organizations and institutions in charge of designing, implementing and managing development programs to advance development objectives. There would then be a direct link between the evidence-based decision-making process at the organizational level with the M&E System effectiveness. Figure 4. Dimensions of effective M&E System at organizational level 90

99 In short, Results-Based Management Practice, Knowledge and Information Management culture and Evidence-Based Decision-Making Process, would be the organizational capabilities that would reflect the effectiveness of the M&E System set forth by the organizations to design, implement and manage development actions (Figure 4). These three dimensions are analyzed in the next sub-sections to lead to rigorous measurements of effective M&E System EFFECTIVE M&E-SYSTEM AS AN INTEGRATED INFORMATION SYSTEM As explained in previous sections, M&E-Systems are relevant learning logics to development programs results and performance and knowledge base information sharing, as they contribute to the development information systems of organizations and institutions. Monitoring and Evaluation Systems are expected to be the bottom-to-up feeding elements to knowledge base information on development programs results for measured risk-taking and improved decision-making (Acquaah, Zoogah, & Kwesiga, 2013). Monitoring and Evaluation is a performance management tool that promotes accountability through knowledge generation and learning system, and is a powerful development management tool for strategic information generation and uses for improved learning and evidenced-results based management and decision-making (Parker, 2008). As we argue here, M&E-System is not personal base knowledge. Instead, it is a process of organizational collaboration, learning, and decision-making framework and builds on clear institutional approaches to advance measured risk-taking in development management. From a knowledge management view, it is close to the Nonaka and Takeuchi concept, who conceptualize knowledge creation as a series of epistemological conversions between tacit and explicit knowledge: socialization, externalization, combination and internalization, which are said to work along a spiral that moves up and down between four ontological levels : individual, group, organization and inter-organization. (Zhu, 2008: ). M&E is a set of continuous activities on data collection, storage, management, analysis and creation of meaningful, evidence-based information on development results and performance (relevance, efficiency, effectiveness, impact, and sustainability) for decision makers and other stakeholders interacting with or interested in the given development actions. As explained by Keith Mackay, more governments realize the importance of M&E System to improve their performance (Mackay, 2008). The main objective of a Monitoring and Evaluation System is to provide evidence that can feed decision-making process at all levels including development planning process, programs and policy design and implementation. Three words compose the M&E-System concept: Monitoring, Evaluation, and System. As previously highlighted, Monitoring comprises a set of continuous technical activities that relate to data collection, 91

100 management, analysis and presentation, on the development programs results through well-defined indicators, benchmarks and milestones as well as mapping of successful strategies and tactics that ensure the achievements are leading to the desired program outcomes. Evaluation activities relate to the robust and rigorous assessments of programs performance and impact at certain times before, during and after the program life-cycle. A System if referring to as Monitoring and Evaluation being an Information System (IS) since it gathers, creates, manages and shares information within and outside organizations and institutions for management purposes. As we argue here, how development programs M&E-Systems perform, partly determines the quality and strength of the development performance as they are results-based. According to Kusek et al., results-based M&E System is a powerful public management approach (J.Z.Kusek et al., 2005). As highlighted in previous sections, in 1996 Tom Barton, defined Monitoring and Evaluation System as an integrated Information System: The M&E system is a form of information system, which is a broad term for information selection, gathering, analysis, and use. It can be described as a logical chain of linked ideas starting (and continuing) with information users. (Barton, 1997:10). As such, Monitoring and Evaluation should be considered as a form of Information System for analyzing its effectiveness or success. In looking at the Information System literature, the Delone & McLean Information System Success model (Figure 5) is a wonderful model that can also apply in the field of Monitoring and Evaluation of development programs if carefully and rigorously operationalized, as In the D&M IS Success Model, systems quality measures technical success; information quality measures semantic success; and use, user satisfaction, individual impacts, and organizational impacts measure effectiveness success. (DeLone & McLean, 2003). Technical success in the Monitoring and Evaluation System addresses monitoring techniques and evaluations designs, approaches, and methods. Information quality is looking at the evidence generated by the Monitoring and Evaluation System for decision makers, mainly called achievements and findings at different levels of results and performance. The use of the system and its information products then impacts or influences the individual user in the conduct of his or her work, and these individual impacts collectively result in organizational impacts. (DeLone & McLean, 2003). The DML IS Success Model (2003) is used in this research to operationalize the M&E System effectiveness model. The cycle Intention to Use-Use-User Satisfaction is the center stage that materializes the information system effectiveness. An M&E System is effective when, first, the intention to use its products and services by Managers and other development stakeholders is high. When the stakeholders are convinced that the M&E System is built on high-quality information, they might be more committed to using its services for decision-making. While their satisfaction in using the M&E System grows, the intention will increase at a higher level for the utilization of the M&E System. 92

101 Figure 5. DML IS Success Model, 2003 A systematic application of the DML model would not be relevant in the field of development while the dimensions developed in the model are critical for this research. Instead, the dimensions of the DML are operationalized to take into account the institutional aspects of development programs in the learning for better decision-making and policy design process. The M&E System effectiveness measurement framework will then be based on the DML Success model. To operationalize its components, sub-components, and measurements, the approach will be based on development field experiences and literature review as explained in the research approach (5.1. First Iteration: Review of the Dimensions of Effective M&E System) EFFECTIVE M&E-SYSTEM AND RESULTS-BASED MANAGEMENT PRACTICE Managing for results is critical to development objectives and achievement of tangible results for greater welfare and livelihood of the people. When development results are clearly defined, and targets set, strategies for implementation in the decision-making process would need good evidence to make sure the expected outcomes and results are being achieved. Development institutions and organizations are characterized based on their mandate, objectives, visions, missions to achieve development objectives. Hence, they need rigorous evidence to show how they are achieving their mandate. The clear definition of what the development institutions are doing, resources they mobilize through various contributions and partnerships, programs they design towards the achievement of their mandate to advance development for the people s wellbeing, are key results-based management elements, as to refer to what Drucker argued here in management and institutions, that without institutions, there would be no management and vice-versa, management fosters institution s contribution to advance development (Drucker, 1954). 93

102 Without institutions, there would not be Management, and Results-Based Management provides the orientation and managerial frameworks for improved strategic planning, risk management, and performance monitoring according to Meier (2003). The aim is to improve organizations and institutions delivery of tangible development results measured through rigorous evidence on the achievements, as argued Meier: Results-Based Management (RBM) is a management strategy aimed at achieving important changes in the way organizations operate, with improving performance in terms of results as the central orientation. RBM provides the management framework and tools for strategic planning, risk management, performance monitoring and evaluation. (Meier, 2003:6). According to Meier in RBM (Figure 6), the How? question in managing for results addresses the Inputs- Activities and Outputs levels of the development results chain, and concerns first of all the projects delivery and partners who are involved in the process. What do we want? addresses the key short-term to mediumterm development results. And Why? Addresses the long-term changes that occur after program implementation. This logic chain of changes generated by the program is the same as the Logical Framework is designed. The results based approach in then, not a new approach in development management (Meier, 2003; Try & Radnor, 2007; UNDG, 2010). Figure 6. Results Chain, (Meier, 2003:14) The Results-Based Management is seen as a management strategy for achieving development results as per the programs logic chain. The management strategy is designed and used by different stakeholders responsible for the development programs design and implementation. A strategy is seen as a set of approaches and practices that underline the way the development program is defined and carried out. This strategy is described in the logical framework approach as the third logic, the logic of implementation. The logic of implementation starts with understanding what is the program theory of change and the needed 94

103 actions to move from the planned inputs to the desired outputs and leading to the outcomes and positive development impacts, and taking into consideration, while moving from lower to upper levels of results, of the risks and assumptions linked to the context. This general approach in managing development programs by results is well described by the UNDG as a strategy by which organizations ensure effectively their role in achieving development results (UNDG, 2010) For any development program management approach to lead to success, there is need to integrate all the stakeholders responsible for actions and program delivery. Even if development projects stakeholders designate core Managers who are accountable for the implementation, the results-based management approach seeks to use all relevant decision-makers in a participatory way to achieve successfully the intended results. Strengthening best practices in delivering development results through greater cohesion and participation of core stakeholders in the process of decision-making, is a key way to ensure Development Management effectiveness as stressed in a 2008 policy paper by UNICEF Evaluation group. Results-Based Management is at the center stage of effective Development Management towards improved capabilities of organizations to advance greater welfare and livelihood of people through better efficiency and accountability (UNESCO, 2008). For organizations in charge of development programs to ensure effective Development Management through the proper practice of Results-Based Management, there is need to guarantee that recording results, analyzing the transformational logic of development chains, and the process of decision-making in managing for results are automatically grounded in an operative evidence-based practice. Results-Based Management and Evidence-Based Management Practice [which is different with Evidence-Based Decision Making], are closely linked to effective Development Management. Development results should be tangible and visible in the people s welfare and improved market linkages for greater businesses in various industries. Hence, tangible results are to be evidenced by rigorous information generated through effective M&E- System. According to Barens et al., The basic idea of evidence-based practice is that good-quality decisions should be based on a combination of critical thinking and the best available evidence. Although all management practitioners use evidence in their decisions, many pay little attention to the quality of that evidence. The result is bad decisions based on unfounded beliefs, fads and ideas popularized by management gurus. (Barends, Rousseau, & Briner, 2014:2). There are abundant definitions of Results-Based Management in the development literature and various descriptions of what would be the core components of the approach at the level of the organizations in charge of development programs. These kinds of definition are reiterating the literature on the Logical Framework approach and Theory of Change methods. Finally, the definitions proposed to clarify and 95

104 operationalize the concept are limited to a mere description of techniques and tools or various standards that have yet to reach consensus in their utility in the research field and development practitioners level (DAC, 2001; Jorjani, 1998; Meier, 2003; Try & Radnor, 2007; Yun-jie, 2009). To be better positioned in analyzing how Results-Based Management operates at the institutional level, there is need to link the approach to effective M&E System and Evidence-Based approach in managing development. As Morrell and Learmonth put it here, in quoting Berends et al.: Evidence-based practice in management is recently defined as making decisions through the conscientious, explicit and judicious use of the best available evidence from multiple sources by: 1.Asking: translating a practical issue or problem into an answerable question, 2.Acquiring: systematically searching for and retrieving the evidence, 3.Appraising: critically judging the trustworthiness and relevance of the evidence, 4.Aggregating: weighing and pulling together the evidence, 5.Applying: incorporating the evidence into the decision-making process, 6.Assessing: evaluating the outcome of the decision taken. (Morrell & Learmonth, 2015:520). Results-Based Management will be assessed through these criteria defined by Berends et al. In short, the six dimensions proposed by Berends et al. (2014) in assessing Evidence-Based Management practice will be operationalized to measure the Result-based Management dimension s sub-components in the M&E System effectiveness model proposed (Figure 7). Figure 7. Measurements for the Results-Based Management dimension of effective M&E System, The Author,

105 7.3. EFFECTIVE M&E-SYSTEM AND KNOWLEDGE & INFORMATION MANAGEMENT In their role of managing development actions towards greater sustainable welfare of people, the capabilities of organizations in charge of development programs to create and use knowledge and information for improved decision-making and measured risk-taking, are key functions in development performance management. A Monitoring and Evaluation System is a crucial enabler regarding the organization's improved capability to leverage knowledge and information to advance development. M&E Systems create data, information, and knowledge for organization s improved decision-making in development management. Data, information, and knowledge are unique concepts confused in the literature (Choo, 1996; Kock, McQueen, & Baker, 1996; Parker, 2008). Kock et al., in a study published in 1996 clarifies that, data is mainly about features such as figures that present the facts in situ through agreed benchmarks or indicators, and information is the rigorous interpretation made from the data analysis, while knowledge is a set of robust findings and conclusions generated from the information when confronted with the domain of interest and the needed actions toward positive change. Hence, knowledge should lead to actionable designs (Figure 8). Figure 8. Knowledge and information enable effective action, (Kock, McQueen, & Baker, 1996:10) M&E Systems create information in a continuous manner from data collection put forth and defined in the M&E strategy. Data are collected through various monitoring activities on program s inputs, outputs, outcomes and impacts, and during program s performance and impact evaluation focusing on data related to relevance, efficiency, effectiveness, impact, and sustainability. The data collected through the M&E 97

106 Systems are analyzed, and meaningful feedback and information generated through various reports and documentation within and outside the organizations in charge of the program management. Development institutions generally organize learning and sharing platforms by convening various stakeholders interested in the program objectives and share the information created through the M&E System activities. Before, during and after those learning occasions, from various exchanges, brainstorming in a participatory approach, knowledge is generated and used for the forward-looking policy-making towards improved capability to advance development in the programs areas. Choo clarifies the three reasons organizations use knowledge: An organization uses information strategically in three areas: to make sense of change in its environment; to create new knowledge for innovation, and to make decisions about courses of action. (Choo, 1996:329). Development organizations use knowledge to take relevant decisions in advancing development objectives through a continuous improvement of their capability to deliver quality results and produce a positive impact on people s wellbeing. They also use information translated to knowledge to foster innovation and create new models ready to scale-up within or outside the organizations by other various stakeholders. Development organizations, finally, use knowledge to take and implement actions toward the development objectives. M&E Systems strengthen knowledge creation and knowledge management within development organizations. Knowledge generated through effective M&E Systems is dynamic and context specific and meet Nonaka et al. requirements on valid knowledge within organizations: Knowledge is dynamic, since it is created in social interaction amongst individuals and organizations. Knowledge is context specific, as it depends on a particular time and space. Without being put in a context, it is just information, not knowledge. (Nonaka, Toyama, & Konno, 2000:7). The two sides of knowledge, tacit knowledge and explicit knowledge (Evanschitzky, Ahlert, Blaich, & Kenning, 2007; Ioannis, Nikolaos, Dimitrios, & Dimitrios, 2004; Li & Gao, 2003; Nonaka, 1994; Nonaka & Toyama, 2007; Nonaka, Umemoto, & Senoo, 1996; Van De Ven & Johnson, 2006; Zhu, 2008) are impacted positively by an effective M&E System. A participatory M&E System strengthens tacit knowledge as embedded in action, day to day activities, procedures, and other values and beliefs. Experts insights are key in an effective M&E System mainly in evaluation processes when conclusions are built on findings from various information and grounded in experts experiences and other known best practices. An effective M&E System can support experts insights and findings rigorously based on information acquired outside the formal existing internal M&E information system. The explicit knowledge within a development organization is centralized and managed through the existing M&E System. The M&E system organizes data collection and analysis through various levels and platforms that are reviewed and capitalized to generate meaningful feedback and information mobilized to create the tacit knowledge for improved 98

107 decision-making and measured risk-taking. As clarified here by Nonaka et al.: There are two types of knowledge: explicit knowledge, and tacit knowledge. (Nonaka et al., 2000:7). The figure below published by Nonaka et al. in 2000 describes the various steps in creation and transformation of knowledge also key in building the M&E System effectiveness framework (Figure 9). Figure 9. The SECI process, (Nonaka et al., 2000:12) Effective M&E System collects, organizes and analyzes data from various sources directly or indirectly to create meaningful feedback and information that will be shared through various platform inside and outside the development organizations into learning occasions, meetings, events and other processes to generate reliable useful knowledge actionable for improved decision-making and measured risk-taking in managing the development programs. Internal M&E sessions and capacity building activities, through a progressive and dynamic learning process, can lead to the meaningful transfer of tacit knowledge amongst different stakeholders, professional and decision-makers interacting in the management process. This kind of knowledge sharing, transfer inside development organization is what is called Socialization or participatory learning by doing approaches in the progressive internal institutional capacity building. The effective M&E System facilitates the transformation of tacit internal knowledge into explicit external knowledge when it helps guide the rigorous capitalization and formatting of various internal experiences and expertise towards higher caliber information translated into sound knowledge and know-how that can be used by other organizations or in other development proposes to advance development objectives better. This process of translation of tacit internal knowledge into explicit external knowledge called Externalization of knowledge is highly improved when M&E Systems are truly effective. M&E Systems are in charge of collection and analysis of a wide array of data on the development programs indicators at different levels of the results frameworks and theory of change. Data can be collected directly from primary sources during surveys and various consultations over the implementation of the programs. 99

108 Data can be collected from different secondary sources including statistics produced at the national level in various sectors covered by the development programs, households surveys or other data collected through international platforms. Quantitative and qualitative data collected and analyzed through the M&E System and translated into knowledge to build internal capability to understand various aspects related to the development program. This connection between big data and the internal and external capability of the organization in charge of development programs through the effective M&E System is a transformation of global and significant explicit knowledge to internal tailored explicit knowledge used at the organization level of development program management. M&E Systems are essential knowledge creation and management systems that bring to development managers and other decision makers relevant feedbacks on programs achievements and results as learning points through various platforms. Knowledge generated through the M&E systems are important feeding elements to improved decision-making and measured risk-taking. In a study published in 2008, Edmund & Marchant clarified that M&E systems bring light to the analysis of whether development programs work or not: The primary function of M&E is to provide pointers on how to do things better through a better understanding of what works and what does not. M&E is the bringing together of information and learning. (Edmunds & Marchant, 2008). Beyond its role of providing feedback on whether the programs work or not, as we argue here, M&E Systems contribute to the building of organizations capability to advance development through better knowledge and information management. As a learning system for the organization to better use knowledge and improve development strategy, policy-making and program implementation, M&E System contributes to the building of the organization capability. As to quote Choo: Current thinking in management and organization theory recognizes three distinct areas in which the creation and use of information play a strategic role in determining an organization's capacity to grow and adapt. First, organizations search for and evaluate information in order to make important decisions. In theory, this choice is to be made rationally, based upon complete information about the organization's goals, feasible alternatives, probable outcomes of these alternatives, and the values of these outcomes to the organization. (Choo, 1996:329) Knowledge and information are key to solve problems related to development programs preparation, implementation, and evaluation. The M&E System supports the creation and sharing of information and knowledge on development programs and strengthens the organization's capability to enhance improved decision-making and measured risk-taking. Effective M&E System is a very important aspect that helps the development organization to leverage and share knowledge and information as to refer to Augier et al. 100

109 analysis in 2001: When people solve complex unstructured problems, they bring knowledge and experience to the situation and as they interact during the process of problem-solving, they create, use and share knowledge. (Augier, Shariq, & Vendelø, 2001:125). Figure 10. Knowledge Transfer and Comparative Advantage, (Li & Gao, 2003:12) Enhanced organizational performance to advance development through creation, share and use of knowledge in a dynamic learning process is the main objective of an effective M&E System (Figure 10). As evidence-based decision-making and measured risk-taking are complex activities in development management as well as in any market-driven activity, knowledge and information management are crucial as to quote again Choo: Although organizational decision making is a complex, messy process, there is no doubt that it is a vital part of organizational life: all organizational actions are initiated by decisions, and all decisions are commitments to action. (Choo, 1996:330). However, what are the key factors of development organizations capabilities that are influenced by a good learning, knowledge and information management through effective M&E System, is a key question to understand the linkages between effective M&E Systems, Knowledge and Information Management and the dynamic capabilities at organization level to advance development objectives. Anantatmula & Kanungo, in an interesting article published in 2006, explained that the ultimate goal of knowledge management is to transform knowledge learning, sharing and use into competitive advantage via enhanced organizational performance, stressing that: the ultimate goal of knowledge management is to effectively utilize the intellectual capital in the enterprise and to transform such knowledge assets into competitive advantage through enhanced organizational performance. (Anantatmula & Kanungo, 2006:33). In short, Anantatmula & Kanungo identified key capabilities that are improved through effective knowledge management among which the most relevant to development management are: Sharing best practices, Improved productivity, Enhanced quality, Improved employee skills, Improved communication, Enhanced collaboration (Anantatmula & Kanungo, 2006). These factors will be analyzed and 101

110 operationalized as measurements in the M&E System effectiveness framework proposed (Figure 11), under the Knowledge and Information Management sub-dimension. Figure 11. Measurements for the Knowledge and Information Management dimension of effective M&E System, The Author, EFFECTIVE M&E-SYSTEM AND EVIDENCE-BASED DECISION- MAKING Organizations and institutions in charge of managing development programs are continuously in a dynamic learning and decision-making processes over the programs lifetime. A development program is launched when policy-makers start thinking on how to address the development problems underlying the needed changes. Once the idea is brought to the table of decision-makers, the first step is to analyze the feasibility and profitability of the proposed actions. Policy-makers decide on what are the relevant programs to implement towards the global and local development objectives. This first decision in program choice or selection is critical in achieving development objectives. Sound and strong development programs should then be selected, planned, funded and implemented. In the policy-making process debate in development management, many approaches for the collection of data, analyzing of information in selecting better programs, have been widely described in the literature. Many of those approaches are based on political power and economic approaches. In developing countries, the tragedy of poverty and weak infrastructures to support development actions, sometimes oblige the governments and policy-makers to decide on what programs to funding and carrying-out without any reliable evidence in their probable success. In these cases, decisions are based only on the intuition of advisors or wishes of other political lobbies or leaders. Also, usually in developing countries, there are donors or development partners with funding lines that need to be consumed or used in the supported countries even though the programs presented are not relevant, rather, supports are only guided by the need 102

111 to burn the available funding before they return to their origin. In those cases, decision-makers are only guided by the rates of disbursement rather than the reliable evidence that shows greater results. In showing development programs successes and impacts, many development managers use intuition, and limited beneficiaries feedback to gauge development outcomes, instead of carrying out relevant studies that will provide good evidence on development impacts. In short, decision-making in the development field is not always based on evidence that shows the successes towards the indented results. The development programs M&E systems, if they are effective, may support improved evidence-based decision-making. As decisions of development managers may impact on lives of people, the bad decisions might conduct countries and organization to bad choices with a negative impact on people s livelihood. The quality and reliability of decisions taken are then very important for development programs success. As argued here by Barends et al. The basic idea of evidence-based practice is that good-quality decisions should be based on a combination of critical thinking and the best available evidence. Although all management practitioners use evidence in their decisions, many pay little attention to the quality of that evidence. The result is bad decisions based on unfounded beliefs, fads and ideas popularized by management gurus. (Barends et al., 2014:3). M&E Systems are decision support systems for development managers in various organizations, through the collection of data, analyzes, and provision of meaningful information that can generate relevant knowledge for actions to advance development objectives. Various types of data are collected by M&E Systems from a large array of sources relevant to the programs. Data are analyzed by relevant experts mobilized in the M&E activities using context situation with data analysis techniques relevant to each case to generate findings and conclusions that bring actionable knowledge. Hence, M&E Systems contribute to the reinforcement of development organizations capability to improve their decision-making processes. As argued here by Briner et al. Evidence-based management is about making decisions through the conscientious, explicit, and judicious use of four sources of information: practitioner expertise and judgment, evidence from the local context, a critical evaluation of the best available research evidence, and the perspectives of those people who might be affected by the decision. (Briner, Denyer, & Rousseau, 2009:19). During the implementation of development programs, decision-makers need information and knowledge produced timely. Some decisions relate to the program's operations; these are grounded with information on operations and processes of the development program in the area of intervention and generally lead to 103

112 operational adjustments. The operational adjustments from the day to day information and knowledge use and decision-making, lead to changes in the program s short-term action plans in timelines, and changes in the programming of outputs, products and services quality and delivery. The information generated through the M&E System is produced in different documentation including periodic reports, other web-based platforms, for stakeholders and decision-makers to access to the needed information. Internal meetings and various coordination occasions lead to the creation of the needed knowledge and action plans. While the development field is a changing environment and context, political regimes with different objectives are coming to power mainly in Africa with sometimes huge changes in the development agendas; there are needs for information and knowledge on the results of the ongoing programs to make the necessary robust adjustments in policy-making and programs design and implementation. M&E Systems carry out assessments and evaluations to provide to program managers with the needed information to create the relevant knowledge that may lead to successful tactical adjustments and other long-term adjustments at strategic levels. For a policy decision to be evidence-informed for successful development actions, there is need to have effective M&E System that can provide relevant information and learning and knowledge creation occasions over the development program preparation and implementation even in a changing political and contextual environment. As clarified here by Lander et al. Evidence-informed development refers to the practice of making decisions in development policy and practice informed by the best available evidence. (Langer, Stewart, Erasmus, & de Wet, 2015:463). Scholars have shown the positive correlation between quality evidence and high-quality decisions; the same finding is valid in the field of development management. To avoid the pilotage-à-vue, which means implementing development programs without measured steps, there is need to gather and analyze relevant data and produce critical information and useful knowledge that can support measured risks and meaningful evidence-based decisions to advance action plans to success. According to Rausch, The ability to manage, and to lead, can be significantly improved with high-quality decisions in all managerial responsibilities because decisions are the foundation for action. Looking at that from the opposite side, effective actions are based on sound decisions, and sound decisions pay attention to all controllable matters which impact on their outcomes. (Rausch, 2003:979). Developments organizations and institutions need strong decision-making system to ensure that development policy elaboration and implementation are grounded by meaningful evidence, learning and knowledge sharing approaches. M&E System, at this juncture, plays a critical role in improving Evidence- Based Decision-Making in development organizations. In the graph below (Figure 12), Choo has shown in 104

113 the 90s the importance of organization s rational behavior towards a higher quality of performance programs. Figure 12. Organizations as decision-making systems, (Choo, 1996:332) In analyzing the decision-making process effectiveness, Schilling et al. in 2007 (Figure 13), studied the six dimensions of the decision-making process that cover 1. the information processors, 2. the approach to processing information, and 3. The results-oriented dimensions (see table below). Figure 13. Dimensions to assess decision process effectiveness, (Schilling, Oeser, & Schaub, 2007:231) In the world of development practices and evaluation research, an evidence is a finding or conclusion derived from robust data collection and meaningful analysis. The evidence is characterized by its robustness, meaning that different actors or researchers using the same data and analysis techniques will have to the end the same finding or conclusion. Development practices and assessment culture, in designing and 105

114 implementing evaluations, are supposed to culminate to findings and conclusions that are evidence-based for better management decisions. Decision-Making in the development area within the particular context of West African countries is a very uncertain activity as programs and projects are designed and implemented in a critical pathway where the theory of change is generally challenged by critical assumptions and risks mainly related to the political, institutional and socio-economic environment. At this juncture, an effective M&E system that will facilitate evidenced-based knowledge creation and dissemination is critical for development programs success including in West Africa region. The quality of the information generated through the M&E system will ensure the reliability of the knowledge generated for use in the decision-making process. The more the information users are satisfied, the better is the M&E system quality. Strong monitoring and evaluation designs will generate reliable and meaningful evidence for decision-makers. All the information produced by the M&E system is about evidence on inputs use and management, outputs or products and services creation, outcomes measurement, impacts assessment, performance control, etc. In short, in the proposed model for M&E System effectiveness, the six Schilling et al. dimensions of decision process effectiveness are used to measure the factors of the Evidence-Based Decision-Making component of the organization capability influenced by effective M&E System in addition to the two dimensions analyzed first, namely Result-Based Management, and Knowledge and Information Sharing (Figure 14). Figure 14. Measurements for the Evidence-Based Decision-Making dimension of effective M&E System, The Author, M&E-SYSTEM EFFECTIVENESS NET BENEFITS As shown in the previous sections, through effective M&E System, development organizations and institutions may record greater capability in three development management practice domains: 1. Results- 106

115 Based Management, 2. Knowledge and Information Management, and 3. Evidence-Based Decision- Making. These development management capabilities, when strengthened through increased effectiveness of the M&E System hosted, and implemented by the development organizations and institutions, will help improve ways for better preparation, implementation, and achievement of development objectives in a purely market-driven approach to greater welfare and improved livelihood of the people. With improved Results-Based Management, organizations could better meet the client needs through participatory approaches and results oriented policy-making. Policy-Making is a key learning process in Development Management and should involve all relevant stakeholders interested in the development objectives (Demir & Miller, 2010; Sidney, 2010; Treib & Pülzl, 2010). If development actions are not well defined and aligned with the needs of the people and the market requirements, along with the establishment of the necessary institutional environment, there is little chance to achieve development objectives. For policy and development programs to be well defined, well-articulated and implemented accordingly by development organizations and institutions, there is need to gather and use well-informed evidence-based knowledge in the process of defining realistic targets and benchmarks towards the desired outcomes (S. Griggs, 2010; Hollinger & Staatz, 2015; Sadovnik, 2010). The evidence-based knowledge generated through effective M&E System could support better policymaking when the organization starts planning the development actions. According to Mara S. Sidney Central to the policy design perspective is the notion that every public policy contains a design a framework of ideas and instruments to be identified and analyzed. (Sidney, 2010:84). The development policy design and framework need well-informed knowledge and benchmarks to set and clarify the development objectives and results. To define tangible development objectives that meet people s needs, national and global economic development requirements, organizations build on the findings from relevant assessments or capitalization on previous and ongoing programs. M&E Systems, in the field of the development sectors to be addressed by the established policy, may be very useful information and knowledge resource for defining well-articulated development objectives and results. According to John Young, Better utilization of research and evidence in development policy and practice can help save lives, reduce poverty and improve the quality of life Policy-making is a dynamic, complex, chaotic process, especially in developing countries. (Young, 2005:727). Weak development organizations and institutions, and weak development management capabilities may hinder effective policy-making in development countries. Non-effective M&E System increases the weakness of capability to define and implement relevant development policies mainly when development organizations have little control of resources within a changing and dynamic international environment. Hence, effective M&E System might 107

116 contribute to foster development management capabilities in the developing world through improved policy designs. Better Results-Based Management increases the capabilities of development organizations and institutions to define well-aligned policies and prepare well-focused development programs that will contribute greatly to improved people s welfare. The definition of development programs uses relevant benchmarks and milestones towards the desired outcomes. A good Knowledge and Information Management coupled with effective Evidence-Based Decision-Making would ensure better implementation of development programs through improved capability to select and carry out good decisions throughout programs life time, and after. Development organizations need to take timely decisions to adjust program implementation in early phases of their inception and implementation, changing then some program s outputs or schedules to better align the results with the intended outcomes and impacts. When the designed policies and programs meet with the field realities, the institutional and environmental constraints, there is a continuous need to readjust development programs implementation for better quality and timeliness of products and services to be delivered. These continuous adjustments at lower levels of the programs results chains and theory of change, are called operational adjustments and should be grounded in well-defined evidence for the learning and knowledge management over the implementation of the program. Operational and tactical adjustments are good in a participatory approach, mainly when they are based on field program feedbacks and analysis of program s outputs indicators and financial management. These adjustments originating from the learnings from the program s M&E system, the routine monitoring of program s activities, are key to the success of development policy implementation. In Africa, where the context is quite particular, and pollical matters are huge, there is strong need to set effective M&E System that can lead to greater learning towards evidence-based decision-making for improved operational decisions. Effective M&E System, through improved organization's capability in taking relevant operational and tactical decisions grounded with evidence-based learnings, is key for better development policy implementation (Edkins, Geraldi, Morris, 2013; Eggers, 1994; Hirschman, 1967; Schieg, 2009). Over programs implementation, there might be needs for greater adjustments at program outcomes level, when the initial ones are to be changed due to the economic context in the area of implementation. Development organizations and institutions might be called to adjust ongoing programs due to more indepth knowledge of the beneficiaries status and their environment. These adjustments are program intermediate results and outcomes levels are called tactical adjustments and meant to adjust programs direction in terms of expected changes at beneficiaries level. Feedbacks and learnings from the M&E System, through evaluation reports and various reports pertaining to the implementation of program actions in the field, are feeding the evidence needed to ground tactical adjustments as they have great impact on the 108

117 program's expected results at intermediate level of the change logic (Johnson et al., 2004; Samset, Forss, & Cracknell, 1994) CONCLUSION ON THE SELECTED M&E SYSTEM FRAMEWORK In conclusion, the proposed framework in the figure below, as a summary of the development in the sections under this chapter will serve as the basis for operationalization and testing in field work in iterations 2 and 3. The proposed selected framework for effective M&E System (Figure 15) contains five building blocks structured from the results and analysis of the literature. The three first building blocks constitute the dimensions of M&E System as established in the DML Information System Success model: 1. M&E System Quality, 2. M&E Information Quality, and 3. M&E Service Quality. The literature review on M&E System helped operationalize the sub-dimensions of the M&E System dimensions. Figure 15. M&E System Effectiveness Framework, version zero (to be tested) The fourth building block that was characterized in the DML IS Success model, the Intention to Use, the Use, and the User Satisfaction, was structured on the dynamic managerial capabilities of organizations in charge of development actions, that are probably influenced by an effective M&E System: 1. Results-Based Management, 2. Knowledge and Information Management, and 3. Evidence-Based Decision-Making. The linkages between these capabilities and effective M&E System dimensions were clarified with the analysis from the literature review. The fifth building block is composed of the dimensions for the net benefits component of effective M&E System proposed framework. These were also established with the analysis 109

118 of results from the literature review: 1. Improved policy and program design, 2. Improved operational decisions, 3. Improved tactical decisions, 4. Improved Strategic decisions, and 5. Improved capability to advance development. 110

119 CHAPTER 8. RESULTS AND DATA ANALYSIS OF THE CASE STUDIES This chapter presents the results of the analyses of the cases studies that seek to operationalize the dimensions of effective M&E: System Quality, Information Quality, and Service Quality. It analyses the data from the different cases progressively to extract what would be relevant to consider and generalize in the proposed M&E System effectiveness framework. It presents progressively the case of Office du Niger which focuses on System Quality, the case of the Senegalese Rural Electrification Program which focuses on Information quality, and the case of the FAO Water and Food Security Initiative in Africa which focuses on Service Quality THE OFFICE DU NIGER CASE STUDY This activity funded by the lead technical and financial partner of the ON, the Embassy of the Netherland (A.3. Introduction to Office du Niger Contract Plan Case 1) launched an important process of designing the new Monitoring and Evaluation System of the agency. With an international context of the rising commodities and food prices in the middle of the 2000s, and within a national context of excessive growing of quantity of food imported to Mali since 2002, the ON was the cornerstone of the ambitious government strategy and policy of food self-sufficiency and security, poverty alleviation and economic development of the country drawn in the National Economic and Social Development Plan (SNDES). In comparison with the other two cases, in the ON M&E System case study, we looked at the System Quality dimension of the M&E System effectiveness framework to analyze and select the appropriate, relevant measurements to be considered for the proposed framework. The study has shown that key sub-dimensions should be considered in analyzing the System Quality dimension that include: 1) Design Quality, 2) Set-up Quality, 3) Operations Quality, 4) Maintenance Quality, and 5) Resources Quality Design of the ON M&E System We find that the level of participation of different stakeholders to the elaboration of the Monitoring and Evaluation plan procedures is key to the ON Monitoring and Evaluation System success and effectiveness. All stakeholders have to be informed and should have appropriated the program theory of change and results framework; they should also have been involved in the definition of the objectives of the Monitoring and Evaluation System and its activities over the program lifetime. The scope of the Monitoring and Evaluation System should then be a shared perspective within the organization including the scope of the monitoring activities and evaluation studies. The appropriation of the different stakeholders of the Monitoring and 111

120 Evaluation System scope and operations, within a participatory approach, has facilitated according to various stakeholders the definition and acceptance of roles and responsibilities of each direction or office in the field in implementing Monitoring and Evaluation activities without any need to use authorities directives. The Monitoring and Evaluation System of the ON is an internal mechanism and comprises agreed upon management and decision-making procedures easy to implement. In many cases, such the ON Contract Plan, indicators are generally very complex technically, mainly the indicators related to agricultural productivity and water management and efficiency. Working in the past in a silo approach to define key performance indicators for the entire Contract Plan, increased the risk for the DPS Monitoring and Evaluation officers to define non-relevant or not well-defined indicators that the personnel in the field and other departments criticized and found difficult to update as they were not aligned with their practices and managerial knowledge. The monitoring of the Contract Plan indicators was organized by the DPS Monitoring and Evaluation unit in the newly defined Monitoring and Evaluation procedures in different sections that present the results chain as it is linked to the various departments and offices scopes in the field. The indicators needed then to be selected, defined and operationalized clearly to avoid any misunderstanding and misinterpretation by the officers responsible for data collection and analysis. The different activities are measured by a number of indicators in the Contract Plan defined in a participatory approach with all technical and financial departments and offices in the field. The most important aspects of that process were quality, costs, and simplicity. To ensure better definition of data collection tools and data analysis techniques for improved learning, knowledge and information sharing, and decision-making, the ON information management system was organized on key elements: 1) Data collection tools, 2) Actors involved in the management and implementation of the Plan Contract, 3) the IT application that allows the organization of the information and makes data accessible to users in the form of dashboards and then enabling managers and decision makers to access timely information for decision-making. In short, the clarity of the selected indicators in the M&E System is a strong requirement to ensure effectiveness. The process of selection should involve all relevant technical department. Indicators and benchmarks should be well aligned with the results framework. There is also need that all indicators be well defined and meet consensus in the understanding of all relevant program stakeholders without any differences in their interpretations and analysis. The DPS of the ON defined in a participatory approach twenty-one data collection tools distributed in all technical and managerial components to gather needed information in the building of the Contract Plan indicators and data analysis for various reports and presentations. Three data collection tools were analyzed in this study as a sample to understand better how the procedures in data collection worked and what would 112

121 be the requirements. Data collected to monitor and evaluate activities and results under the water consumption section is critical to assess the water use efficiency in the ON zones. Water is central to the ON Contract Plan, and the Malian Government concerns, agriculture development, and food security are widely depending on water management and control in the country. The Contract Plan places a preponderant attention to the management of water which is one of the essential mandates of the ON. In short, the quality and simplicity of data collection tools are key to effective M&E. The M&E data gathering and analysis tools and techniques should be readily available to and applicable by all relevant actors involved in the M&E activities without any risk of professional or personal biases over the program lifetime. In developing action plans and defining roles and responsibilities in implementing activities to make the Monitoring and Evaluation System effective, the internal capacities of the organization was considered. The Monitoring and Evaluation System had to meet stakeholders needs and expectations in the process of decision-making and capitalization of best practices. The Monitoring and Evaluation System was set to deliver high-quality information and to ensure knowledge share and learning system are effective for improved decision-making. As such, there should have been relevant capacity in place to perform the planned activities for data collection and analysis. The DPS of ON, to ensure effective implementation of the Monitoring and Evaluation activities and participation of all stakeholders involved, considers the technical feasibility of the data collection tools and analysis techniques. Quality and simplicity then guided the choice of type of instruments and techniques for data collection. In short, the feasibility of timely data collection and analysis is key to M&E System effectiveness as we look to the ON M&E System set-up. Data gathering and analysis activities should be highly realistic and feasible technically and financially within the approved budget, resources and time frame over the program lifetime. To ensure a better implementation of Monitoring and Evaluation activities and an effective Monitoring and Evaluation System, ON defined three levels of information sharing and learning for better decision-making. For the ON Monitoring and Evaluation System to be effective, the DPS defined lines of communication, roles, and responsibilities of different actors within a participatory approach. As functions and responsibilities of the various departments of the ON were defined in official decisions, there was needs to update and integrate the M&E functions to the decisions for the Departments to act accordingly. The process involved a revision of the scope of work of various personnel involved in the M&E System. The DPS reported that M&E tasks were usually considered as secondary and not integrated directly into the decisions on roles and responsibilities of the various departments except for the DPS. In short, it appears that the clarity of functions and responsibilities of different actors is key to ensure System Quality for M&E System 113

122 effectiveness. All actors involved in the M&E system should have a clear scope of work well defined without any overlapping or missing in the human resources utilization for M&E purposes as identified in the M&E plan and budget. We noted that the ON authorities have taken into consideration factors that might hinder effective M&E System mainly at levels of data collection and analysis including the illiteracy of the farmers when they are involved in data collection, the unclear objectives, and activities to be implemented, lack of updated and recorded data, difficulty in transmitting timely the required information, low level of capacity to run IT systems. To address these issues, the ON designed a capacity building plan that contained activities related to training of core staff, filed visits, and workshops. In short, we find that the capacity building of the core stakeholders involved in the M&E System is key to ensure effective participation and would support them to play their roles and responsibilities more effectively in the implementation of M&E activities. A highquality capacity building plan which is realistic and feasible technically and financially should exist that defines capacity building activities including internal seminars, training in formal or short courses, exchange visits, etc. commensurate with the needs of different actors and objectives of the M&E System to allow them easy access and implementation of the M&E procedures and activities. The program M&E procedures should be readily available and understandable by all actors involved in the M&E system, transparent and well-crafted without any possibility of confusion in the definition of core concepts, indicators, technical and financial procedures in the M&E activities implementation Quality Set-up of the M&E System It was noted that one of the key challenges in ON M&E System was its set-up to make it functional after the various activities of designing the procedures, techniques, M&E Plan, budget, and activities. These challenges included, but were not limited to, the timely recruitment of the needed staff or reorganization of the DPS, the mobilization of resources to implement the activities in the M&E Plan, the implementation of capacity building activities for the M&E actors, testing the M&E tools before their use in the field, design and implementation of baseline surveys and data collection to establish the first set of benchmarks for future measurement of changes. In the DPS, there was already an existing M&E team as well as in the field of the different zones. The shifts in the scope of work of the M&E staff due to the new approaches developed to ensure effective M&E of the Contract Plan made it necessary to review and update the M&E teams scopes. These needs involved a lot of changes on the human resources aspects. Some agents in the field were replaced, and the process of recruitment was directly managed by the Human Resource division of the department in charge of finance and administration in relation with the DPS. Many enumerators also were replaced in the field. The various 114

123 heads of offices in the different zones argued that the process of updating the M&E staff was critical and arduous as it was challenging to find the required experts with the needed skills and background. A committee was set at the DPS that included staff from the DFA to accelerate the process as the assessment made it clear that the timely availability of the M&E staff was key to ensure the M&E System effectiveness. The zones where the process took longer to be completed because of lack of suitable candidates who want to work in those remote areas, faced a lot of difficulties to make their M&E System working the first year. The DPS staff organized a lot of missions to fill the gaps that involved a lot of funding for travel and logistics. In short, we find that the timely recruitment of M&E Staff to make them available early in the process is key to ensure effective M&E. All M&E core staff and other agents and consultants working in the M&E System should be timely hired and recruited as defined in the initial M&E Plan and time frames and should be timely ready to implement M&E activities as per their scopes of work. Various actors in the M&E System at DPS and DFA argued that there were difficulties in mobilizing funding for M&E activities as the partners, and the government were reluctant to any idea of adding funding to make the M&E System functional. Despite the fact that there were lines in the budget for M&E as per the approved Contract Plan, mobilizing resources was challenging. Hence, the DPS and the zones were not able to implement all the planned activities, that impacted greatly on the M&E System functionality the first two years of the Contract Plan implementation. Difficulty in mobilizing resources timely compromised the survey on agriculture production the first year. The DPS used estimates in few areas to forecast the quantities produced in the Contract Plan report. Hence, the quality of the data generated was not accepted by all stakeholders. In short, we find that resources should be available timely for M&E activities to ensure effective M&E System. All resources including technical, financial and logistics should be accessible and mobilized timely to implement the program M&E activities as defined in the approved M&E plan and budget. The M&E Staff in the DPS benefited from various training programs including short courses in a US University to improve their capacity to carry out data analysis and impact evaluations. There were a lot of in-house training and seminars organized by the technical assistants to build the capacity of various actors in the M&E System on techniques of data collection and analysis as well as communication and learning process management. But, some actors in the zones were not able to benefit from the training, because many of them were involved in local activities or could not have the necessary funding to attend the workshops when they occur outside of their areas. In those areas where M&E staff did not participate in the training and capacity building programs fully, the M&E activities were not well-implemented, and the quality of reports was weak compared to other zones which benefitted from the training. Despite a lot of supports 115

124 provided by the DPS team, there was a technical gap that impacted in the M&E reports from those areas. In short, the implementation of capacity building activities in the M&E Plan is crucial to ensure effective M&E. All relevant M&E actors involved in the M&E system should benefit, when needed and planned, from appropriate capacity building activities to improve their abilities and skills in implementing the M&E activities they are in charge. The initial phase of the Contract Plan M&E System required to design and implement a comprehensive baseline to set the information on the status of all selected indicators prior to starting the activities. The baseline data is key to ensure future assessments of outcomes and changes on the people livelihood. Because of the revision of the Contract Plan theory of change and improvement of its results framework, there was new information that needed to be generated not only linked to agriculture productivity and production. In addition, the information included indicators on poverty and food security in the area of the Office du Niger. The baseline also included the design of a quasi-experimental approach to ensure proper assessment of the programs impacts at midterm and post periods of the current Contract Plan. In the last decade, the ON faced various challenges in measuring the changes generated by its programs. Because the ON was not able to inform partners and the government on effective positive changes in the people livelihood produced by its programs, it was then difficult to request more funding to move water management and irrigated agriculture to success despite its significant potential. In short, we find that the establishment of a baseline for the Contract Plan for future assessments was key in the M&E System set-up to make it effective in its capacity to carry out appropriate evaluations. All program indicators for which a baseline value is mandatory for future evaluations and comparisons, reliable and quality baseline information should be collected and analyzed prior to start carrying out the program activities as per the procedures and methods defined in the M&E plan Quality M&E Operations For the ON to ensure effective M&E, there was a need to create the necessary conditions for the M&E System to operate without critical difficulty. The conditions for effective quality operations included, but were not limited to, the existence of quality data exchange platform that ensures real time sharing of information. The ON possesses the SIGON which is the information system dedicated to the management of cultivation areas and water management, but mainly focusing on a system of vouchers to ensure timely payment by the Farmers of their royalty fees in occupying the irrigated farms. One option was to integrate to the SIGON a module on M&E System database, which was objected by the financial unit arguing that the SIGON is mainly for financial operations. The DPS then explored ways to create a tailored M&E web- 116

125 based platform. Because of lack of resources, the platform was not available the first two years of implementation of the Contract Plan. The different units and DPS continued to work with Excel Sheets and MS-Word tables, using their s for data exchange. That made the exchange of information very challenging and not well organized. Hence, compromising the M&E System effectiveness. ON continued to face several problems linked to information exchange, and reports generation as many of the indicators were calculated manually. We find that the existence of a stable data exchange platform readily available to all stakeholders that integrates models to calculate and generate indicators at all levels of the results chain is key to make the M&E System effective. The M&E system should be able to develop or acquire, deploy a proper input-output M&E information system accessible to all core stakeholders and M&E actors for updates purposes and generation of commensurate reports periodically. The DPS of the ON is managing various types of data related to the Contract Plan implementation and various agricultural production statistics for the Ministry of Agriculture as well as food security indicators in the ON zone. One of the key findings from our discussions with the DSP staff was the difficulty in organizing the paper storage or a data filing system to ensure that all sources of verification of the information generated in the M&E System are well secured for data quality assessment purposes. Hence, it was difficult for evaluators to verify whether the data presented originated from relevant and valid sources to ensure quality. A lot of discussions arise during review meetings where development partners generally question the quality of the data produced by the ON. The main argument to show that the data are collected in well-designed formats and approaches was to show an effective effort to ensure metadata information and documents on data sources are available for quality check. As this was not well organized by the M&E teams in Segou and the different zones, there were critical concerns in the quality of the data in the M&E System. In short, we find that the organization of metadata and information sources of verification for data quality audit purpose is necessary to ensure effective M&E operations. there should be for each program indicator a set of well-defined metadata and means of verification that are very well managed and filed within the M&E platform and made readily available for M&E system audit purpose or data quality assessment. The ON operates in a large area composed of seven different zones in the Segou region of Mali. Various offices operate at a decentralized level in each zone to manage the irrigation system and provide the needed supports to the farmers organizations. The DPS highlighted the difficulty sometimes in mobilizing needed resources to cover all the zones periodically and ensure proper functionality of the M&E System and provide at the same time the technical support and coaching to the local M&E staff of the different areas. Discussions with the M&E teams in the different zones have shown that their face-to-face interaction with the DPS 117

126 technical teams is not regular as needed. The new Executives of the ON took decisions to improve the capability of the M&E team in the DPS to regularly visit the zones and provide technical supports and backstopping to the local M&E teams to ensure that the M&E System is fully functional and operates as planned in the M&E procedures Manual. In short, we find that the capacity of the M&E team to visit all program zone of influence for data collection feedback and analysis is key to M&E System operations and success. The M&E team should have the capacity financially and logistically at any time of the program lifecycle to reach out all localities of the program zone of influence to implement M&E activities as defined and approved in the M&E plan and budget. While there was a need to ensure mobility of the M&E team in the DPS at Segou, the new executives of the ON provided funding to ensure adequate functioning of the M&E teams in the various areas covered by the Contract Plan. The M&E teams in the zones Offices play critical roles and responsibility in the M&E System and are functional with the necessary logistics even though there was a need to improve their capacity in handling technical M&E activities. There were activities planned to enhance their skills and ability in data collection and analysis. The DPS found it difficult in some cases to convince the decentralized offices to provide the required resources to their local M&E teams in the implementation of the capacity building plan. That impacted somehow negatively in the M&E System effective operations. We find that the existence and functionality of decentralized M&E sites to cover program zone of influence is key to ensure effective operations of the M&E System in the various areas covered by the program. The M&E system should relate to existing relevant decentralized M&E Units to reach out all programs zone of influence localities and activities; those decentralized units should have the required skills and capacity to implement the program M&E activities locally and to collect and analyze M&E information for local feedback purpose and be able to establish technical relationship with the headquarter in term of knowledge and information sharing. The setting of the ON M&E System used a comprehensive approach that was not a system for only the M&E teams; it also involved the different departments. Roles and responsibilities of the various departments were clarified, and several meetings and internal seminars were organized to facilitate collaboration and participation of all relevant actors to the M&E activities. In the process of communicating that the M&E System in not just a tool for M&E teams, rather it needs the contribution of all actors involved in the program's activities, the DPS made an excellent deal in ensuring better participation of the various actors in the process of data collection, analysis and information generation. The DPS organized several internal seminars with various departments to share the information and learn on achievement on specific components of the Contract Plan. This process greatly enhanced the effective implementation of the M&E 118

127 activities. We find that level of involvement of program components core actors in the implementation of the M&E activities is key to ensure effective M&E operations Maintenance Quality of the M&E System In analyzing the ON M&E System, we found that maintenance occupies a central place in the process of making the system effective. Maintenance of the M&E System includes the capacity building activities of M&E actors, the technical maintenance of the M&E infrastructures and equipment, and the regular review of the M&E System effectiveness. To ensure proper coordination of the M&E work toward its effectiveness, there was a need to organize reviews and assessments of the M&E System periodically in order to identify the gaps and take to necessary corrections and adjustments. There were not tailored M&E system assessments organized in the ON until the advent of the Helen project which started with an in-depth and comprehensive assessment of the system. Quick assessments were carried out during midterm evaluations of previous programs, where generally evaluations sub-questions relating to review the data collection and analysis approaches were embedded in the scope of evaluations. The outcomes of the evaluations reports on the M&E System effectiveness were generally limited, and recommendations were focusing only on operational aspects of the M&E System and tools for data collection. In a view to creating a productive dialogue and participatory adjustments of the M&E System, we find that there should be regular reviews/assessments of the M&E system to ensure its effectiveness through appropriate progressive adaptations, at least one M&E system review/assessment per year should have been performed. The quality of the data exchange system is key to ensure effective information management and sharing for decision-making. While working with Excel sheets and MS-Word tables, and using s for sharing of information, was not easy for the M&E team as in some areas access to the internet was difficult, the DPS was planning to improve the data exchange platform system. We find that the quality of the program M&E data exchange platform was key to the M&E System effectiveness and it should have been accessible at any time to program stakeholders, for example, a web-based system with good internet access everywhere and very limited shortcomings which were the plan for the DPS. Another finding is the importance of the capacity building of the M&E staff and different actors involved in the M&E System. The DPS organized numerous training sessions for the M&E team in the different zones and sent the M&E staff in the DPS overseas to attend international training on RBM and M&E System. This helped improve the M&E System effectiveness, and the quality of tools and techniques for data collection and analysis as well as the way reports were prepared and shared in the decision-making process. We find that the deployment of a strong capacity building plan for the actors involved in the M&E 119

128 System would help improve its effectiveness greatly, 100% of the M&E staff should have benefited from staff development program through training or exchange visits. The DPS was able to leverage several partnerships through the Helen project to support the M&E capacity of the institution. These include various international M&E platforms and professional organizations. These memberships and partnerships improved the documentation on M&E approaches and practices, the attendance to online seminars on M&E and participation to some international workshops on M&E experiences and best practices. That helped the DPS to improve the knowledge on M&E approaches and practices and to fine tune the internal seminars contents in the building of capacity of the M&E teams in the different zones. We find that technical support received from specialized M&E agencies and other organizations at international and national level is key to ensure improved M&E practice and effectiveness. The M&E system should have received at least one technical support per year from professional partners and recognized M&E agencies or organizations or specialized government departments on M&E best practices and guidelines Quality and Sufficient Resources for the M&E System Operations The M&E unit of DPS is led by a highly experienced agricultural economist with a strong background in statistics. He is seconded by a highly skilled socio-economist. Both have good records and experience in M&E Systems in the country and have certificates on M&E approaches with recognized institutions. This team was able to support technically the process of designing the ON M&E System and to provide technical assistance to the M&E teams in the different zones. The institution also needed good quality M&E staff in the field to support the DPS team. Additional recruitments were then made in line with the recommendation made by the Expert of the Helen project. Resources were mobilized for the implementation of the M&E capacity building plan. It was clear, with the amount of funding managed and the volume of programs, and the vast size of the area to cover as well as the intensity of the agricultural value chains activities, that ON needed highly skilled M&E team members. We find that the qualification and level of competency of the program M&E managers are key to the M&E System success. All M&E Managers in the program should have at least all required skills and qualification for M&E expert including strong knowledge of development policies and programs, strong skills in M&E approaches, impact and performance evaluation, monitoring, reporting, statistical background, data collection, and analysis, as well as communication skills. The volume of M&E activities is significant in the ON M&E Plan including the routine data collection in the large area of irrigated land and farmers exploitations, the periodic surveys and assessments, the numerous meetings and learning events to be organized. In implementing the M&E activities properly, the 120

129 DPS used numerous experts, consultants, and enumerators as well as supports staff in the farmers organizations. Developing their scope of works and the terms of reference for the various studies, and hiring these different support staff and individuals as well as development firms require scrutinizing the level of qualification and skills of the mobilized personnel. To implement the various activities through support services, there was a need to leverage good and relevant external experts, consultants, and enumerators. Resources were allocated in the ON M&E Plan and budget, but mobilizing them to carry out timely the needed activities with a required set of skills was a challenging process for the DPS. We find that the qualification and level of competency of other M&E human resources (enumerators, consultants, etc.) are crucial to the M&E System operations success. All M&E agents, surveyors and other consultants who are working in the M&E system should have the required qualification, skills, and knowledge to carry out their duties as defined in their specific scope of work. Finally, the mobilization of resources is a challenging process in a context where many partners are not willing to support the M&E System, instead, prefer to fund investment in irrigation systems and agriculture production. The DPS took the time to mobilize the needed equipment to carry out all the start-up activities in setting the M&E System. The volume of activities in the M&E Plan and the importance of results-based management for the institution which now is moving with the Government and partners from projectized funding approach to a more coherent budget support system predicated the DPS to be best positioned in mobilizing funding for improved Contract Plan management and effective M&E System. We find that quality of equipment used to carry out the M&E activities as well as the existence of appropriate financial resources as per the approved M&E plan to conduct M&E activities is necessary to ensure M&E System effectiveness. All equipment to be used or in use by the M&E system should be at the high quality of standards and well maintained, and functional, as well as all the resources defined and approved in the M&E Plan and budget should be actionable easily by the M&E Managers to implement M&E activities without any delays. In short, the analysis has led to the critical measurements to be considered for the next iteration survey, that are summarized in the table below (Table 7). Table 7. Sub-dimensions and measurements for System Quality dimension of M&E System effectiveness framework Sub-dimensions of System Measurements to be considered Quality dimension 1) Design Quality Existence of a high-quality M&E plan / procedures (before program implementation) Level of participation of different stakeholders in the elaboration of the M&E Plan/Procedures 121

130 Sub-dimensions of System Measurements to be considered Quality dimension Clarity (and specificity) of the selected indicators along the program results chain Quality (and simplicity) of tools and techniques for data collection and analysis Feasibility of timely data collection and analysis Clarity of roles and responsibility of different actors in the M&E system Existence of capacity building plan commensurate with the M&E Plan/Procedures objectives The M&E procedures are readily available to the different actors 2) Set-up Quality Availability and timely recruitment of M&E staff to implement program M&E activities Timely availability of resources to implement M&E activities Capacity building of core actors responsible for the M&E activities implementation Tests of M&E tools and techniques prior to M&E activities implementation Existence of quality baseline information for future program results assessment 3) Operations Quality Existence of M&E data exchange platform accessible to core program stakeholders Organization of metadata for M&E data audits purpose Capacity of the M&E team to visit all program zone of influence for data collection feedback and analysis Existence/ Functionality of decentralized M&E sites to cover program zone of influence Level of involvement of program components core actors in the implementation of the M&E activities 4) Maintenance Quality Frequency of reviews/assessments of the M&E system design and operations Quality of the program M&E data platform % of M&E agents benefitting from staff development program Technical support received from specialized M&E agencies and other organizations 5) Resources Quality Qualification and level of competency of the program M&E managers Qualification and level of competency of other M&E human resources (enumerators, consultants, etc.) Quality of equipment used to carry out the M&E activities Existence of appropriate financial resources as per the approved M&E plan to conduct M&E activities 122

131 8.2. THE SENEGAL RURAL ELECTRIFICATION PROGRAM M&E SYSTEM CASE STUDY CASE 2 This Case Study is introduced in detail in the Annex (A.4. Introduction to the ASER Program and Monitoring and Evaluation System Case Study 2). In studying the Senegal Rural Electrification Agency (ASER) Monitoring and Evaluation System, the research focuses on the dimensions of information quality in an information system model as described by DeLone and McLean, in their revised success model of In the analysis of what the ASER Monitoring and Evaluation System is showing and how it operates, we cross-checked with different knowledge gathered in the literature with the case study findings on the Monitoring and Evaluation System information quality. The analysis of the ASER case through the dimensions of quality information has shown five criteria key to ensure high-quality information in the ASER Monitoring and Evaluation System: 1) Quality inputs information, 2) Quality outputs information, 3) Quality outcomes information, 4) Quality performance information, and 5) Quality risks management information Information on the program inputs According to Barton, Data for financial and physical monitoring are collected on a regular and frequent basis throughout the implementation phase of the project, according to the level of project activities and outputs. (Barton, 1997). He clarifies here that input-output information relates to financial and physical information on the project implementation process. Whereas data on finance is key, qualitative information on other resources used is critical to the analysis of program efficiency. The relation of the Outputs to the Inputs of a project is called the Efficiency of a project, and answers the question How productively did we use our resources? (Grun, 2006). As we argue here based on the different interviews with ASER staff, there must be a strong analytical system linking the financial information to the physical delivery or outputs. ASER spends huge funds in various areas including core costs such as salaries and benefits, utilities, capital equipment, supplies, other direct costs, indirect costs and overhead withdrawal. There is difficulty in attributing those core costs to the ASER programs effective outputs at beneficiaries' levels. To solve that issue, the planners put in place an analytical accountancy system that obliges to align any given resource use from the programs funding to a specific program result or output. To do so any single expenditure is linked analytically to a given results and its underlying activities. That allowed a results-based tracking of the expenses and facilitates the linkage of financial data to physical and technical activities. This inputs-to-output tracking system is a core component 123

132 of the ASER M&E-System information and ensures its quality and what Grun calls Value for money in development programs (Figure 16). Figure 16. R. Grun, Value for Money in development program (2006) In many development programs, such as ASER rural electrification program, beneficiaries are requested to contribute to the conception, implementation, funding and even evaluation of development actions. This facilitates the ownership and prepares the different needed elements for the achievement and impacts of the program to sustainability. Beneficiaries contributions are key to ASER institutionalization. In the regional program, development actions are demand driven. ASER set its objectives and mobilizes resources internally in the country and with support from different partners including the private sector. As the ASER programs are demand driven and implemented by the beneficiary institutions and organizations, the monitoring of inputs and financial flows to specific development actions is more complex. ASER designed a tool to track and inform financial inflows and use (Table 8). The tool presented below tracks resources from the Government, Donors, Private Sector and Beneficiaries as well. Table 8. Tool to track funding mobilization in ASER programs Source of funding Total Expenditu res for past years Budget Current year Total Expenditures Semester 1 Total Expenditures Semester 2 Rate of Expenditures ASER GOVERNMENT DONORS PRIVATE BENEFICIARIES TOTAL Source: ASER M&E Manual,

133 Information on the program outputs To achieve its expected results, ASER should deliver quality and viable outputs or products and services to its clients or beneficiaries. Over its life cycle, during inception and annual work planning sessions, in alignment with its desired results and expected outcomes, ASER plans for specific outputs or products and services set forth and implement needed activities for their achievement. The way those products and services are achieved through the implementation of the program shapes key program performance aspects regarding efficiency, effectiveness, and respect of timelines at different levels as the programs evolve. A framework to measure achievement of program outputs was then critical to ASER. ASER put in place a strong system to monitor achievements on program outputs as that is part of its performance assessment by the GOS (Government of Senegal). Even though resources are utilized and timelines respected for activity implementation, in many cases, ASER information users realized that the planned quantity of services are not achieved as planned, or benefits are not sustainable over implementation and after programs end-line. Timely information of decision makers and clients on program outputs is critical to address those issues, as argued here by Cook et al.: Performance measurement systems help managers understand when their programs are succeeding or failing by signaling potential management problems when the performance indicators do not track in the desired function. (Cook, Vansant, Stewart, & Adrian, 1995). One of the biggest challenges of Monitoring and Evaluation Systems such the one designed by ASER is the coverage of the program results by the outputs indicators. Are the selected outputs indicators covering all the program s intermediate results? Missing a right coverage of program results by the output indicators, as we argue here may generate misleading in managing the program and in taking adjustment based on the measured performance. Therefore, ASER took critical actions to make sure its selected output indicators give a good estimate of program results achievement. Tracking the right data is essential for any Monitoring and Evaluation System to avoid misleading decision-makers and over-collection of data not relevant to the organization s knowledge and information management. Heavy data are usually available in the development programs Monitoring and Evaluation Systems such as ASER programs, little provide relevant information on programs products and services and meet at the same time data quality standards. In its Categories of influence affecting the use of performance data, Thomas J.Cook et al. listed Data quality control (data audits) as one of the core criteria that may affect the use of performance information in decision-making and measured risk-taking in development business (Cook et al., 1995). Quality information is key to the learning process and evidence-based decision-making approach. 125

134 ASER defined quality standards to test the validity of its indicators and reliability of information generated for decision-making. The main criteria for quality information in development program learning and evidence-based decision-making process related to the quality of data and validity of the analysis approach. First of all, data or indicators should match with the measure of the development objective, should be defined clearly, they should be comprehensive and describe perfectly the entire aspects of the program, they should be exact with minimum error and deviation, and last but not least, they should be accessible for interpretation and use (Galdemar, Gilles, & Simon, 2012:12). As discussed in previous sections, development programs at sectoral level originate from and contribute to national and global development frameworks. One of the objectives of national statistics agencies in the development sector is to consolidate contribution of the development program to national indicators and statistics, be that for any priority sector, as it is for the energy sector and ASER programs. On behalf of the Ministry in charge of Energy, ASER sends information on its programs achievement to the National Agency of Statistics (ANSD) and the Direction of Economic and Development Cooperation (DCEF) 12. To fully play its role, the national agencies such as ANSD and DCEF define data format and measurement policies set in the national guidelines for economic and statistical surveys policy 13. The quality of indicators generated by the ASER Monitoring and Evaluation System and their easy use and consolidation with the national level statistics information system, largely determine the validity and relevance of the ASER programs information at national level. That is to ensure that information generated by the ASER at output level (products and services) is not only used internally for program management purpose only, in addition, it is also key to the national statistical agencies for use to inform government and other sectorial ministries on development achievements linked to other social and economic data for larger analyses. In short, the quality of output information generated by development programs such as ASER programs, as shown and analyzed through this section, relates to major criteria that are: 1) the timely availability of the outputs indicators data that show information on targets versus achievements, to program managers for informed, evidence-based decision making over the program lifetime; 2) the coverage of the different program s components and specific results by the program output indicators to make sure all program s results and achievements are informed and taken into account in the decision-making process for 12 DCEF is responsible to the Government to report on development programs achievements, while in each Ministry, there is a planning, coordination, and monitoring unit that is responsible for coordination and supervision of development prgrams as well as Monitoring and Evaluation of the development actions under its Ministry. A National coordination of those units is set at the level of the Prime Minister cabinet. The National Economics and Statistics Agencies are responsible for that national coordinating unit to inform on the country economic and social performance in all sector. 13 In defining their Monitoring and Evaluation Systems procedures, development programs such as ASER programs should align with those national directives and policies in data collection and analysis so as to allow easy and relevant consolidation at the national level. 126

135 adjustments and other policy related planning needs; 3) the capacity of output indicators to meet data quality standards set forth by national statistical policies and guidelines on development programs data; 4) the ability of the project output data to be consolidated easily at national level with other sectoral or national statistics over time for larger analysis of development trends in the country Information on the program outcomes Usually, development programs such as ASER programs are heavily criticized for being unable to inform the public and stakeholders on their expected impacts and outcomes defined in their original theory of change. It is critical to building an approach to inform decision-makers and citizens on the program s final results, and as stated by Bamberger et al. in a research article published in 2010: In recent years there have been increasing demands to measure the effectiveness of international development projects. (Bamberger et al., 2010:21). This recurrent concern of the clients is expressed to the implementers in developing countries where development is still a big challenge, and reliable information on development progresses quasi-inexistent. Therefore, if we look carefully at the ASER results matrix and Monitoring and Evaluation plan as well as various reports, there is a tentative to inform government and donors on the reduction of rural electrification gap linked to increases in household income and increased local economic growth. ASER set systems and approaches to measuring those changes in partnership with Environment and Development Third World (ENDA). The agreement between ASER and ENDA felt short in being implemented according to the initial work plan. The main challenge, while roles and responsibilities were clearly defined in the ASER Monitoring and Evaluation Manual, was to mobilize appropriate resources in implementing this activity. Despite these difficulties, the two organizations were able to carry out a baseline study and to inform on few outcome indicators from the LFM. These efforts we noted were not sufficient enough to ensure timely availability of reliable information on the rural electrification program impact and outcomes. For example, under its Intermediary Result 1 The rural households access to electricity is improved in a sustainable manner, ASER planned and implemented activities through a component called Achievement 1 Develop rural households access to electricity. The main output indicator selected by ASER to measure this is The cumulative number of subscriptions to electricity in rural areas covered by the program to be collected from the private operators reports and documentation. From this output to the Intermediary Results 1 measured by the rate of servicing electricity in rural areas, the rate of electricity coverage in rural areas, the rate of rural electrification, the standard deviation of rural electrification rate intra and interadministrative regions, the % of rural poor households having access to affordable electricity, and the % of rural poor household being serviced with affordable electricity, there at lot of steps to perform before, during 127

136 and after activities implementation under Achievement 1. The same approach applies to all linkages between Intermediary Results X and Achievement X in the ASER detailed LFM. These steps as part of the program logical implementation approach are generally implemented within a participatory approach and toward key stakeholders and beneficiaries involved. The first step was to ensure that the ASER programs deliver quality products and services that meet the clients/beneficiaries needs and demand, so as they use the program outputs appropriately. The second step was to verify the clients/beneficiaries satisfaction after having used the products and services delivered by the ASER programs. Finally, to assess whether the clients/beneficiaries and different stakeholders involved in the actions were adopting the products and services, hence were setting their own mechanisms for replication and scale-up with minimum and progressively without the program support. Only by guaranteeing these critical steps in the implementation processes, would the ASER programs ensure sustainability of achievements towards greater impacts. Therefore, it was critical to ASER, to focus in its Monitoring and Evaluation System in building approaches and implementing studies to assess the use of the program products and services delivered to clients/beneficiaries, their satisfaction on the relevance, utility and quality of such products and services, and their willingness to ensure effective adoption of the products and services along with the mechanisms set forth for their preparation and delivery 14. To measure changes at program objectives level, ASER needed to develop a more comprehensive approach to ensure effective assessment of changes in the living conditions of the rural population improved through effective rural electrification attributable to its program. To measuring quantitative changes due to ASER rural electrification program on the clients/beneficiaries, ASER set systems to monitor the effects (outcomes) of its programs on the clients/beneficiaries, households, micro-entrepreneurs managing production units in rural areas, rural social infrastructures, and their managers, other institutional stakeholders, their responses to the products and services received. Indicators that measure those outcomes focus on changes in clients/beneficiaries behaviors following their use of the program's products and services, at levels of their satisfaction, appropriation, and adoption of the products and services 15. Another approach designed by ASER is the impact evaluation studies that measure short-term impact of the programs that occur over the implementation of the activities with the aim to gather needed information when it comes to measuring short and long-term impacts after the implementation of the program in some regions of the country. These impact evaluation studies during program implementation are carried out as 14 This evaluation activity closely linked to the monitoring system is called Beneficiary Assessment. 15 This kind of program outcomes monitoring is measured during ASER periodic effects and impacts assessments. 128

137 case studies with selected clients/beneficiaries who may be affected by the program's activities in their zone of influence 16. The data for these studies are collected during the specific period of the programs lifetime and can be in quantitative and qualitative form. Even if the sources of information are the direct observation and interviews of direct clients/beneficiaries, data of this kind of outcomes monitoring may, time to time, be originated from secondary data in the routine Monitoring and Evaluation data collection and analysis, or from other implementing partners in the zone of influence. Therefore, systematically checking the quality is required to ensure the reliability of information Information on the program performance In a historical view, France, United-Kingdom, United-States of America, Canada and Australia are the leading countries in performance measurement of public policies. As noted by Galdemar et al., «The notion of performance relates to the achievement of objectives or results, or more broadly to the creation of value.» (Galdemar et al., 2012:9). More specifically, performance information in development programs is viewed as a management approach that guides decision makers in policy and program design, implementation and assessment. Moreover, to verify whether actions set and products and services delivered are leading to the achievement of the intended changes at beneficiaries level. Performance information and measurement criteria may vary depending on the priorities set by decision makers and major risks in the country or development program zone of influence or agencies in charge of the implementation. An agreement and generalization are noted with many of the performance measurement criteria. In development program management literature, performance measurement criteria cover the following agreed areas 17 : relevance, efficiency, effectiveness, impact and sustainability (Akroyd, 1995; J. Cameron, 1993; Cook et al., 1995; Cracknell, 1994; K. Mackay, 2008; Segone, 2010b; Valadez & Bamberger, 1994). These criteria are assessed by ASER M&E System during specific periods over the program's lifetime. Under its mandate, ASER is implementing several projects linked to its general LFM and objectives assigned by the government. The performance assessment studies are organized during annual reviews, mid- 16 This kind of impact monitoring helps ASER to verify if the strategies used and tactics to implement the program's activities are leading to the desired objectives. 17 These are the five development performance evaluation criteria adopted by DAC/OECD, as stated by T. Chianca, The establishment of the DAC criteria can be considered, at the time of its inception, a great step forward in the direction of improving the quality of development evaluations. (Chianca, 2008:42). 129

138 term performance evaluation, and final performance evaluation. The annual reviews focus mainly on programs efficiency and effectiveness criteria while the midterm and final evaluations concentrate on all performance evaluation criteria. ASER uses mix methods for its performance evaluations. For annual reviews 18, the sources of information are the various reports from operators and other implementing partners and periodic Monitoring and Evaluation reports generated by different ASER divisions and departments. Before the organization of the annual reviews, ASER starts sending its technical divisions and planning and M&E units to the field in the different regions of the country where its program activities are implemented, with standardized templates for data collection, compilation, and analysis. The midterm evaluation is planned in conformity with the government and financial and technical partners rules, procedures and timelines. In particular, during midterm evaluation, all five program performance criteria are assessed. The evaluation 19 will determine whether the program theory of change is still valid in relation to new policies defined at country level and technically in relation with processes and approaches generated in the same sector. In addition to the five internationally agreed criteria to assess the ASER program performance, the ASER Monitoring and Evaluation System uses systematic qualitative approaches to assess the program activities Best practices and lessons learned and, uses the Strengths-Weaknesses- Opportunities-Threats (SWOT) analysis approach to evaluate the program different methods and technologies introduced by the different operators implementing the contracts in the zone of influence. These additional systematic qualitative approaches will determine whether the strategies, innovations, and approaches used by the various implementing stakeholders are leading to the program objectives and expected results. Assessing the relevance of ASER program is mandatory to the agency and the Ministry in charge of Energy in the country as specified in the ASER Performance Contract Letter signed by the Government with the ASER s steering committee. The analysis of the relevance concerns the adequacy of the programs defined 18 The annual review, as part of the program management system, determines whether activities implemented are on track, and lead to the desired results and outcomes. Reports generated from the annual review process will feed the annual work-planning sessions. Results from the annual review documents and success stories will also contribute to the program operational adjustments and the elaboration of the next year action plans, approaches, and strategies for program implementation. The annual review is an internal participatory program assessment and is an important learning and decision-making process. Sometimes, external studies can be implemented to support the annual review in the case specific adjustments are needed to maintain the program in the right direction. 19 Is the program still relevant to the country, beneficiaries and different stakeholders? are the results at intermediary levels still sufficient enough to generate the desired program outcomes? These are the main evaluation questions for the criteria on the relevance of program activities and strategies. Typically, the relevance criteria are assessing the extent to which: activity is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance of a program or a project, it is useful to consider the following questions: To what extent are the objectives of the program still valid? Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? Are the activities and outputs of the program consistent with the intended impacts and effects? (Chianca, 2008:43). 130

139 and implemented by the agency and not the agency itself. How the agency translates the Government policy in the sector of rural electrification and how it addresses the needs of the various beneficiaries will then be assessed 20 periodically during the annual review of program portfolio, mid-term evaluations of specific programs and projects implemented and during final program performance evaluation. To assess the efficiency of its programs, ASER collects regularly relevant information on activities implemented in its zone of influence by the different implementers and operators under specific contracts. The regular data collection is ensured through what is called in the ASER Monitoring and Evaluation System, the Implementation Monitoring that relates to the monitoring of ASER programs annual action plans physically, technically and financially, it also relates, the same time, to the organizational aspects of the implementation, the coordination and synergies between partners in the field in respect to their specific roles and responsibilities. The technical and physical monitoring assesses the distribution of the deliverables, products, and services of the various ASER program activities. The financial monitoring focuses on the inputs and costs of activities defined through the analytical plan of ASER and is informed through the financial reporting system. The inputs relate to the financial, technical and human resources and are managed with specific categories called cost centers and through financial reports. The ASER technical and financial monitoring data is analyzed during program performance assessment to measure the efficiency and effectiveness of the program mainly through two criteria or ratios that measure 1) the performance related to the respect of timelines, 2) the performance associated with respect to planned costs or budgets. These ratios on program performance timelines and budget are analyzed globally and by activity in the analytical plan through different technical components. In that process, the monitoring of the organizational system that completes the technical, physical and financial monitoring, focuses on the institutional capacity development aspects, synergies, coordination and the capacity building of all stakeholders involved in the implementation. In many cases during program performance evaluation periods, ASER managers using the Monitoring and Evaluation information, discover that while the implementation respected the original timelines, many activities have not generated the quantity and quality of products and services expected, or some the benefits 20 According to OECD/DAC definition of the Efficiency criteria as cited by Chianca, it measures the outputs qualitative and quantitative in relation to the inputs. It is an economic term which signifies that the funding - uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency of a program or a project, it is useful to consider the following questions: Were activities cost-efficient? Were objectives achieved on time? Was the program or project implemented in the most efficient way compared to alternatives? (Chianca, 2008:43). The last three questions highlighted by Chianca relate to the three major resources used to implement activities, namely: time, finance and human/technical. How these resources are combined and used efficiently together will determine largely the success of program implementation that leads to the necessary outputs, products, and services and later to the expected program outcomes and impact. 131

140 are not sustainable during and after the program implementation. The organizational aspects analyzed during performance assessment relate to the recruitment of staff, the capacity building program, the organizational and institutional relationships, the internal organization of ASER including the Monitoring and Evaluation System to learn on the necessary adjustments to make to the program implementation moving forward for better performance. Through this effectiveness criterion 21, ASER managers assess whether the program activities are on track and lead to the desired products and services and expected outcomes. The evaluators will examine data from the Monitoring and Evaluation System related to the monitoring of the program objectives in addition to the collection of other primary information from different stakeholders involved in the program implementation. That is the monitoring of the ASER programs strategic objectives for a continuous and systematic adjustment to the beneficiaries and population needs for rural electrification. The monitoring of the objectives will allow a broad understanding of the conditions of achievement of ASER objectives in rural electrification based on the current level of success and engagement, quality and status of the activities, products, and services at the time of the assessment. That also helped ASER identify constraints and or positive and adverse effects of the products and services delivered so far. The monitoring of the program effects and impact generates data that serve to assess ultimate results during program performance evaluations periods. As stated above, this monitoring relates to the ASER programs effects, responses of beneficiaries, households connected to the rural electricity power plants or solar systems, the micro entrepreneurs managing the production and processing units in the rural areas accessing to the ASER electricity and other products and services, the social rural infrastructures such as schools, boreholes, health centers, etc., connected to the rural electricity grid thanks to ASER and the local rural institutional partners. That also relates to the measurement of ASER impact in the short term that occur during program implementation. This kind of impact monitoring is different with the impact evaluation studies and is assessed by studying specific cases on selected beneficiaries that may be affected by the program activities. This effects and impact monitoring will help ASER determine whether the program implementation strategy and primary benefits are leading to the desired program outcomes in a sustainable manner. To quote Chianca again, the Impact criterion of the performance evaluation assessment analyses: the positive and negative changes 21 According to OECD/DAC development program performance criteria cited by Chianca, the Effectiveness criteria of performance measurement assesses the extent to which program - activity attains its objectives. In evaluating the effectiveness of a program or a project, it is useful to consider the following questions: To what extent were the objectives achieved or are likely to be achieved? What were the major factors influencing the achievement or non- achievement of the objectives? (Chianca, 2008:43). 132

141 produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. (Chianca, 2008:43). Sustainability 22 and Lessons learned are two additional criteria examined during program performance assessment in development management. The environmental sustainability is assessed by ASER through its Intermediary Result 4, The global environment is preserved and improved measured by the indicators: i) Quantity of CO 2 avoided, ii) % of clean energy in the total electricity production through ASER support, iii) % of beneficiaries serviced with clean energy electricity, iv) Rate of recovery of used batteries, v) Rate of recovery of used LBC (ASER M&E Manual). The preservation of the global environment for future generation is key for any given development program. Sustainable development that goes beyond the environmental sustainability is described by Clayton et al. (1994) as approaches to address the integration in the long term and where possible of development objective at social, economic and environmental levels Information on program risk management ASER has set a system to monitor and manage risks and assumptions of its programs as shown in its full logical framework matrix. As clarified in the LFM, risks, and assumptions that may hinder or challenge the program results chain progression towards higher impacts, are clearly defined in the LFM elaborated in a participatory approach with all relevant stakeholders. The risks and assumptions identified in the ASER LFM relate at its broad level to: the reinvestment of income generated from the program benefits to the people livelihood and welfare, the absence of natural disasters, the adherence of the beneficiaries to the technologies proposed by the program, the acceptance by the population of the commercial transactions and agreements modes suggested by the operators providing electricity, etc. At all results levels, specific risks and assumption are defined and addressed by the program managers. Assessing and managing those risks will ensure smooth running and achievement of program results as stated by the vertical logic in the LFA: If Results at level n are achieved, and Risks and Assumptions at level n+1 are addressed, then Results at level n+1 would be addressed (Table 2). 22 According to DAC/OECD cited by Chianca, Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after program implementation has been ceased. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a program or a project, it is useful to consider the following questions: To what extent did the benefits of a program or project continue after funding ceased? What were the major factors which influenced the achievement or non- achievement of sustainability of the program or project? 133

142 Figure 17. Development program risk assessment tool, the author, 2016 The risk assessment and management is ensured through the Monitoring of the Program Context set in the ASER Monitoring and Evaluation System, the monitoring of the context in which the program operates and how risks and assumptions are affected. It includes the monitoring of the institutional and political aspects that may influence the capacity of the program activities to be implemented appropriately, or the ability of the beneficiaries to respond positively to the program products and services. All risks and assumptions in the LFM are assessed through two dimensions: the probability of occurring (High, medium, low) and the impact on the program results in case it occurs (Important, Average, Negligible). Based on the result of the risk assessment, made within a participatory approach with core stakeholders well experienced on the subject, the risk is then classified within a matrix (Figure 17). When a risk/assumption is assessed on its two dimensions (probability of occurrence and impact on the program results), and is situated in the Red Area of the matrix (its probability of occurring is then High or Medium, and its impact on the program results is Important or Average), then the program managers will define a set of actions to treat the risk or assumption so as it will not challenge the program performance. There are two types of treatment depending on the capacity of the program managers to influence whether the probability or the impact dimension. For example, in the case of the risk/assumption related to the absence of natural disasters (Table 32), as it will be difficult for the ASER managers to influence the probability of occurrence of disasters, that is beyond the capacity of the Agency, and sometimes even the country as a whole, there will be limited solution to address the occurrence or to Mitigate the risk, then ASER managers will define a set of actions to Attenuate the impacts of the risk on the program results for instance through subsidies to the affected population or disaster insurance schemes. In the case of the risk/assumption the adherence of the beneficiaries to the technologies proposed by the program (Table 32), if the assessment shows the risk is in the Red Area, then the ASER Manager will 134

143 define a set of actions to reduce its probability of occurrence or will Mitigate it, for example through sensitization campaign on the new technologies introduced, their importance and benefits generated for the population and entrepreneurs. Mitigation (reducing the probability of occurrence of the risk/assumption) and attenuation (lessen the impact of the risk/assumption on the program results) are the two set of management tools ASER managers use to manage the program risks and assumptions throughout programs lifetime based on information generated by the program context and risks/assumptions continuous assessments through the program Monitoring and Evaluation System. In short, the table below summarizes the results gathered in the dimensions of Quality Information for an effective Monitoring and Evaluation System, such the one designed, developed and implemented by the ASER from the results of the case study. These will be used as variables in iteration 3 survey. Table 9. Sub-dimensions of M&E Information Quality and measurements to be considered Sub-dimensions of M&E Measurements to be considered Information Quality 1) Inputs information quality Timely information on inputs indicators Existence of an analytical accounting system Accuracy of program financial data Quality of conversion of Program beneficiaries counterparts/contributions, Inputs indicators that meet data quality assessment standards 2) Outputs Information Quality Timely information on outputs indicators Outputs Indicators coverage of program implementation results outputs indicators that meet data quality assessment standards Capacity to consolidate program outputs data with national statistics 3) Outcomes Information Timely information on outputs indicators Quality Outcome indicators coverage of program intermediate results Outcome indicators that meet data quality assessment standards Coverage of outputs (products and services) use information Coverage of beneficiaries satisfaction with the program outputs Coverage of beneficiaries adoption of the program outputs Accuracy of information on program intermediate changes 4) Performance Information Quality at beneficiaries level Availability of quality information on the program relevance Availability of quality information on the program efficiency Possibility to make cost benefit and cost effectiveness analysis from the M&E data 135

144 Sub-dimensions of M&E Information Quality 5) Risk Management Information Quality Measurements to be considered Meaningful analysis with Outputs/Outcomes indicators to establish effectiveness Meaningful analysis with Outcomes/Impact indicators to establish program sustainability Capability to translate best practices and lessons learned into meaningful data for performance assessment Risk assessment of all assumptions in the Logical Framework Continuous analysis of program result chain, vertical logic, with risk management data 8.3. THE FAO WATER AND FOOD SECURITY INITIATIVE M&E SYSTEM CASE STUDY This Case Study is introduced in detail in the Annex (A.5. Introduction to the IESA Program and Monitoring and Evaluation System Case Study 3). The Water and Food Security Initiative (IESA) funded by the Spanish Agency of Cooperation for International Development (AECID) is a regional project implemented technically by the Food and Agriculture Organization of the UN (FAO) in five West African countries. Better control and management of water, a scarce resource, is the principal pillar of the program, by introducing a certain number of measures to agricultural productivity and improved food security of the most vulnerable households in the region benefitting from the intervention. The IESA seeks to support West African countries and FAO efforts to achieve food security via a better management of the potential in water and the development of irrigated agriculture. The initiative completes and supports the countries and ECOWAS efforts in the operationalization of regional and national strategies and programs (ECOWAP, NFSP) elaborated in support of FAO. The program covers five countries of ECOWAS: Burkina Faso, Guinea Conakry, Mali, Niger, and Senegal. The program had a regional coordination unit within the regional office for Africa of the FAO in Accra, Ghana. This regional coordination unit was composed of a regional coordinator, irrigation and agricultural specialist, a regional M&E expert and a Communication Specialist. The team worked under the supervision of the FAO West Africa coordinator and other technical offices established in Accra. The team was reporting to a task force composed of various food security and management units in FAO Regional Office for Africa (RAF) Accra and FAO headquarters in Roma, Italia. The task force was led by the Technical Office for Support to Integrated Food Security (TCSF). At the country level, the management unit of the program was embedded in the Ministry in charge of irrigation and agriculture or rural development. The management unit team was composed of a National 136

145 Coordinator and M&E Specialist who are generally, civil servants designated by the Ministry in charge of the project. The decentralized offices of the Ministry and the extension services, in relation to the farmers organizations, were in charge of the management of the program at the local level, project sites. A steering committee composed of the FAO representative in the country, the Ministry Department agents, farmers organizations leaders, extension services, the private sector in irrigation systems, and other relevant stakeholders in the countries, was in charge of the general direction and management of the program at the country level. The M&E System set forth by the program is a regional-national platform of knowledge exchange to advance the program objectives at regional and national levels. It uses M&E guidelines from DAC/OECD, FAO and the different countries M&E strategies, to build a regional M&E approach. The M&E approach used a capacity building strategy of national M&E Specialists to strengthen countries efforts in managing food security programs. The regional M&E Specialist assessed the capability of the FAO Sub-Regional West Africa office and countries M&E systems linked to the program. Based on the findings and conclusions of the assessment, and recommendation from the regional learning event organized in Ouagadougou in October 2010, an action plan was set to implement critical activities between 2011 and The most relevant data collected during that period is used in the analysis of this case study in addition to the recent interviews with the various stakeholders during the process of this research in the field work. The IESA M&E System involved a lot of stakeholders at different regional and national levels with various needs of information in terms of content and format for different types of decision-making processes. For the producer in the Tahoua region of Niger, the most relevant information was the available area of land irrigated and the maximum area he/she could access to, as well as the level of funding he/she can leverage through the microfinance institutions to buy the needed agriculture inputs or pay for manpower, and information related to the extension system that supports technically the producers and their procedures, etc. The national steering committee in Guinea Conakry needed information on funding available and priority areas where water drainage and irrigation can make a difference in improved agricultural productivity and food security, and relevant information on the program impact on people s livelihood. The decision-makers in FAO West Africa office seek information on best practices for scale-up in other countries and relevant statistics on food security at country and regional levels for global analyses and strategies on food and nutrition; they also need information on how funding is used at country level to implement the program activities. The task force seeks information on the program impact and funding mobilization from various partners. This case study was used to operationalize the dimension M&E Service Quality of M&E System Effectiveness framework, and its selected relevant measurements. As this program involved many 137

146 stakeholders and decision-makers at various levels of the West Africa region and countries, the M&E System capability to service all these actors with tailored information in the learning events and decisionmaking processes was an interesting example where the dimension Service Quality would be effectively assessed for learning and generalization for other development programs in the regions. The following subsections deal with the results of the analysis classified by selected areas in the case study. The key lessons learned from the case helped map the criteria of success for Service Quality into: Information Availability, Information Accessibility, System Responsiveness, System Flexibility and Adaptability, System Sustainability Availability of Quality Information One of the key roles of the IESA M&E System was to generate timely the mandatory reports required by various stakeholders in the process of decision-making. These reports ranged from simple notes and tables at local levels of the countries for the local governments and Ministries decentralized services, the graphs and figures presented in the offices of farmers organizations that show periodically records and statistics on productions and incomes generated from sales of agriculture products, the reports to the districts and at national level for the Ministry in charge, the reports for the project management at national, and regional levels; reports and snapshots compiling countries data, and mandatory reports to the donor AECID. Delays in producing such reports and dashboards would be a risk in ensuring effective management of the program and better regional and national coordination. Efforts were made at various levels to ensure the timely production of the reports and the review of various reports was done through remote consultation and monthly calls within the M&E and technical teams of the program at various levels. In short, we find that the capacity of timely generation of program M&E mandatory reports is key to ensure information availability at various levels of management, as well as M&E dashboards; those should be made available timely to program managers and other stakeholders. All (100%) of the program mandatory reports should be high quality and produced within the agreed timeframes in the program documents and M&E Plan and made timely available to decision-makers and other interested stakeholders, as well as all of the program dashboards, should be of high-quality and produced within the agreed timeframes in the program documents and M&E Plan and made timely available. Another way explored by the IESA management to make information timely available to stakeholders, was to consider learnings, knowledge management, and communication tools. The IESA management in Accra hired a Communication and Marketing Expert to improve the way the program communicates its achievements and results through the development of a regional communication and outreach strategy. These kinds of adjustments supported the M&E System towards the effective ability to timely make 138

147 available program achievements and results in formats appropriate for the selected stakeholders at the different levels of the region. IESA carried out, in addition, a global capitalization on best practices in food security projects and water management and irrigation techniques, organized the results into various communication tools to inform policy-makers at different levels of responsibility in the region. In short, we find that the existence of a knowledge sharing system on M&E information and other M&E data communication tools is key to improve development program information availability and M&E System effectiveness. The program Managers should be able to organize at least twice a year learning events on the information generated by their M&E system where discussions and brainstorming are made on their use and dissemination for a better sharing of the program results and achievements for improved policymaking and improved adjustments of ongoing programs. Beyond the program mandatory M&E reports and evaluations and other studies reports, the M&E team should be able to produce more snapshots, briefs, memorandums and other communication tools on the program M&E information for a better sharing of program results and achievements Effective Information Accessibility Even if the information is available to stakeholders in the process of decision-making, that does not mean the information is accessible. Many actors at different levels provide negative feedback in the assessment of the M&E System reports and the way information was presented to them, mainly there was difficulty to understand the figures. Many actors argued that they need someone to explain the content of the M&E reports as they were too technical and interpretations are scientific and difficult to understand. Whereas some tables exist, difficult to interpret in many cases, the needed evidence-based information to take necessary actions was not reported in the documents received. That s why the assessment recommended simplifying the way information is presented to stakeholders given the type of actor and level of competencies. Some actors at a high level do not need too long reports with a lot of details. Also, there was a need to make sure the needed evidence is in the various M&E information for decision-makers to move from System1 to System2 in taking the necessary adjustments (Kahneman & Klein, 2009; Klein & Kahneman, 2010; Tversky & Kahneman, 1986). In short, we find that program managers are having access timely to evidence-based data on program achievements and results they need, as well as other stakeholders having a good level of access and understanding of M&E data and information, are key to the M&E System accessibility and utility for effective evidence-based decision-making. At any time of the program lifecycle, all program managers should have easy access to evidence-based information on the program results and achievement through the M&E system, as well as all other program stakeholders should have easy access to evidence 139

148 information on the program results and achievement produced through the M&E system without any difficulty of interpretation of the data and findings. The final external evaluation was organized to assess the IESA performance, accomplishments and results to feed FAO future programming in water management and food security in Africa, and for a potential IESA phase 2. Implementing the evaluation, the evaluation team requested data from the M&E team to make the necessary analyses. The evaluation report stressed in some cases a lack of data in the M&E System to support performance analyses: «Certains indicateurs importants n ont pas été suivis comme les délais pour les Dossiers d Appel d Offre (DAO) - date de lancement, date de traitement, date d'exécution, date de réception - qui aurait permis d'évaluer la performance de la FAO. De même, des indicateurs pour apprécier la performance des services publics décentralisés comme le nombre de visites / mois des agents à côté du nombre d agents par site. Il y a aussi le taux d'exécution par composante qui aurait dû être réalisé annuellement et présenté aux comités de pilotage pour rendre des comptes aux Gouvernements et aux donateurs.» (FAO, 2013:19) In fact, the M&E staff argued, in responding to the evaluation team, that the information highlighted here including those related to the process of procurements and construction of irrigation systems, are actually collected in the M&E tools, but they were not integrated into the periodical reports to be available to the different stakeholders. There is a lot of information available in the M&E System but not accessible to various stakeholders because of lack of coordination in the elaboration of reports. In short, we find that the accessibility to monitoring data for evaluation purpose, and for the large public, media, and researchers, as well as the technical user friendliness of M&E data for all users, are key conditions for effective M&E Service Quality. At any time of the program life cycle, evaluations or other study teams mobilized should have access to all needed M&E data straight from the inception of their studies to avoid overlaps in data collection or missing data, and to ensure consistency in the production of evidence and findings. The M&E managers should make sure that all partners, relevant public, researchers and media who are interested in the program M&E information have easy access to relevant data and information for their specific purposes for a better sharing, dissemination, and use of the evidence generated through the program M&E system. Finally, all of M&E information users should be able to easily interpret or use the data and findings and recognize the user-friendly of the M&E information. 140

149 M&E System Responsiveness Water and food security are important thematic areas in the FAO learning system, especially West Africa is a complex development field, mainly the Sahel countries, when policy-makers are critically dealing with water control and agricultural productivity. Hence, the IESA M&E System faced a lot of request for specific data and information from various stakeholders inside and outside the organization. In some cases, when the concerned countries have strong M&E team and systems, the responses to the requests are made timely. In many cases, there was difficulty to produce tailored information not included directly in the M&E mandatory reports and dashboards. In many countries, even the government department uses data from a project implemented or funded by international organizations because of lack of reliable information from the official services. Development program should be responsive to specific demand of information, even from the government, as they generally have the necessary resources to do so. In short, we find that, the availability of M&E data upon requests by the program stakeholders, the capacity to carry out timely specific M&E studies requested by the program managers or the governments, the ability to conduct timely specific M&E studies, and the ability of the M&E system to respond to the sectorial Ministry information request on the program achievements and results, are key factors of effective M&E System responsiveness. The M&E team should be at any time highly responsive and reactive when it receives new requests of specific studies or assessments expressed by the program managers and other decision makers within a revised scope and budget approved. The M&E team should be able at any time, on short notice, to prepare tailored, high-quality reports or presentations for meetings and other learning events purposes to share the program results, achievements, and evaluations findings and recommendations. The M&E team should collaborate well with their sectorial Ministry counterparts and be highly responsive in providing Government with the M&E data, information on the program in order to make sure that the Government is aware and has appropriated the program achievements and results M&E System Flexibility and Adaptability FAO reviewed in 2012 some aspects of its organizational results to adapt with its vision and new strategic plan. There was some difficulty in the implementation of the IESA program, to align the program results with the ones of the strategic plan, and develop indicators under the new set of Organizational Results. In the various countries where the program was operating, the national FAO offices also started a process of developing the new country assistance strategies, and the alignment of the country IESA results framework to the FAO national strategies results framework. The program was still well aligned in theory to the new strategies; there were gaps and technical challenges to realign the IESA results as quickly as needed to report on the new results properly. 141

150 In, short, we find that the capability of the M&E system to integrate efficiently new national directives and/or international standards on results-based management, monitoring and evaluation, and the ability of the M&E system to integrate timely new information needs following programs revisions and strategic adjustments, are key to ensure the System flexibility which is one of the key factors of Service Quality of the M&E System. The M&E system should be highly flexible to adjust and integrate new information needs following program tactical or strategic adjustment with limited effect on resources and time frames M&E System Sustainability In some countries covered by the program, the national coordination units experienced changes in the M&E teams composition, or at the local level. It was first challenging to find replacements in some cases. Where replacements were found, the challenge was their capacity to take over properly the M&E System. These challenges create differences across countries in the effectiveness of their M&E System. Countries where stability was noted presented more effective M&E System. This inability of the M&E System to face changes in the staff was a criterion of the weak effectiveness of the M&E System at the national level. We find that the capacity of the M&E system to remain fully functional when major changes occur in the M&E team is a key factor of sustainability of the system. Even if unintended changes occur in the M&E team composition, the M&E system should be resilient and should remain fully functional after those changes. As the program covered numerous zones in the different countries, government decentralized offices and extension services were the local program units managing the activities with the farmers organizations. Hence, the M&E System was managed at the local level by these decentralized offices and the extension agents. Many of them had not the required skills for M&E even though they were very conversant and highly experienced in data collection in rural areas mainly. This was the reason why there were a lot of missions organized by the national M&E teams to regularly visit the field and provide technical support as well as ensure that the M&E information is generated timely. These kinds of routine missions and supports were costly for the project and covered the largest portion of the budget for M&E activities. We noted that, many of the field M&E units were not fully operational and required a lot of supports and supervision from the national level. In short, we find that the capacity of the M&E decentralized units to be functional with minimum supervision and support from the headquarters, as well as the ability of the program managers and other stakeholders to access M&E information with minimum support from the M&E team are key factors of success for sustainable M&E System. The M&E decentralized units should be fully functional with very limited supervision from the headquarters and should be able to deliver high-quality inputs that meet the essentials of data quality standards at any time during the program lifetime. All the program managers should have easy access to the M&E data with very 142

151 limited support from the M&E team and should be able to understand, interpret and use the M&E information with very limited technical support from the M&E team. The IESA M&E System was able to assemble numerous quantity of data and generate useful information on agriculture production with small scale irrigation systems in various countries in the West Africa region. After the program closure, the databases were handed over to the FAO country team and the Ministry department which managed the program. Some of the data were used in the design of the probable new phase of IESA; others were used for related project documentation. But most of the data and information remained unused hence stored without apparent utility. We find that the utility of the M&E data after program closure for capitalization purposes is a key factor to ensure the M&E System sustainability. The M&E data and information should be recognized important and useful even beyond the program schedule and lifetime for post evaluations and capitalization purposes or as baselines for future programming. In short, the analysis of the IESA case has led to the critical measurements to be considered for Service Quality dimension of Effective M&E System framework in the next iteration, that is summarized in the table below (Table 10). Table 10. Sub-dimensions and measurements for Service Quality dimension of M&E System effectiveness framework Sub-dimensions of Service Measurements to be considered Quality dimension 1) Information Availability Capacity for timely generation of program M&E mandatory reports Capacity for timely production of evaluation and other M&E studies reports % of M&E dashboards made available timely to program managers and other stakeholders Existence of a knowledge sharing system on M&E information Existence of M&E data communication tools 2) Information Accessibility % of program managers having access timely to evidence-based data on program achievements and results Level of access of other program stakeholders to M&E data Access to monitoring data for evaluation purpose Access to program M&E data for the large public, media, and researchers Technical user friendliness of M&E data for all users 3) System Responsiveness Availability of M&E data upon requests by the program stakeholders Capacity to carry out timely specific M&E studies requested by the program managers Capacity of the M&E team to produce tailored reports timely for program management and learning events 143

152 Sub-dimensions of Service Quality dimension 4) System Flexibility and Adaptability Measurements to be considered Capacity of the M&E system to respond to the sectorial Ministry information request on the program achievements and results Capability of the M&E system to integrate efficiently new national directives and/or international standards on resultsbased management, monitoring and evaluation Ability of the M&E system to integrate timely new information needs following programs revisions and strategic adjustments 5) System Sustainability Capacity of the M&E system to remain fully functional when major changes occur in the M&E team Capacity of the M&E decentralized units to be functional with minimum supervision and support Capacity of the program managers and other stakeholders to access M&E information with minimum support from the M&E team Utility of the M&E data after program closure for capitalization purposes In short, the analyses of the three case studies have led to the selection of proposed meaningful measurements for each of the three dimensions of the M&E System and their sub-dimensions: Information Quality, System Quality, and Service Quality. The measurements will be used to populate the sections in the questionnaire for the survey in iteration 3 to test the truthfulness of the proposed M&E System Effectiveness model. The next chapter presents the results and analyses form the survey. The three cases were compared, and each of them focused on a specific dimension of effective M&E System as to highlight the specificities; and the findings were compared to conclude on the selection of meaningful measurements to be tested in the iteration 3, the survey using a structured questionnaire (See Annex A.6. Questionnaire for Iteration 3 Survey). 144

153 CHAPTER 9. RESULTS AND DATA ANALYSIS FROM THE SURVEY (ITERATION 3) This chapter presents the analysis of data from the survey in iteration 3. Data from 46 development programs in the region are analyzed to test and finalize the proposed framework for M&E System effectiveness. The first set of analysis used Multiple Correlations to test the significance of relationships between the measurements proposed from the case studies in iteration 2. The analysis of multiple correlations helped simplify the number and quality of the selected measurements to finally set the meaningful independent measurements for each sub-dimension of the three dimensions of M&E in the proposed model: System Quality, Information Quality, and Service Quality. The analysis also looked at the interactions and relationships between the three dimensions using a Multiple Regression analysis with SPHINX. A summary was them made to present the main results on the selected measurements, how they interact, and how System Quality and Information Quality together influence Service Quality. The chapter also presents the results of Multiple Regression analyses carried out to find relationships between the dynamic capabilities in managing development actions of the organizations, namely Results- Based Management, Knowledge and Information Management, and Evidence-Based Management, and the three dimensions of M&E System. Finally, the chapter presents, the results of the Multiple Regression analyses carried out to assess the relationships between the M&E System Effectiveness net benefits, namely Improved Policy and program Design, Improved Operational, Tactical, and Strategic Decisions, and Improved Capability to Advance Development. The last section of the chapter presents, as a conclusion, the outcome which is the relevant final meaningful framework for M&E System Effectiveness, and the different implications DIMENSIONS OF EFFECTIVE M&E SYSTEM M&E System - System-Quality The existence of a high-quality M&E plan/manual before implementation of program activities is relatively correlated (0.56) with the level of participation of the different stakeholders in the process of elaboration of the manual (Table 11). When development programs start the implementation of their activities, the relevant stakeholders are engaged in various processes of preparation and discussions on the choices related to the program actions. Hence the spending time in the course of designing the M&E procedures might be challenged by other priorities related to the implementation of activities and investments. In short, to ensure better involvement in the process of the design of M&E manual/procedures, there is need to design those procedures prior to the implementation of program activities. 145

154 Table 11. Matrix of correlations for variables/measurements of System Quality Design Quality Variables for Design Quality - Correlations 2. How was the level of participation of different stakeholders in the elaboration of the M&E Plan/Procedures? 3. Clarity (and specificity) of the selected indicators along the program results chain 4. Quality (and simplicity) of tools and techniques for data collection and analysis 5. Feasibility of timely data collection and analysis 6. Clarity of roles and responsibility of different actors in the M&E system 7. There is/was a capacity building plan commensurate with the M&E Plan/Procedures objectives 8. The M&E procedures are easily accessible to the different actors 1. The program/project has/had a high-quality M&E plan / procedure (before program implementation) The clarity (and specificity) of the selected indicators along the program results chain are likely to be stronger when the program possesses M&E manual/procedures before the implementation of its activities, as the correlation between the two variables is high (0.63). Defining a clear set of indicators and along programs, results chain needs early analyzing of the different program objectives and logic chains and selection of relevant results and indicators and can be only performed before any implementation of program activity. Hence the correlation between these two variables is high. The existence of M&E manual/procedures before program implementation means the definition of a clear set of indicators along program results chain and vice versa. Feasibility of timely data collection and analysis is critical to development programs as data on program results is the first step in the learning process. Hence managers need quality data for different purposes of analysis in producing meaningful information on program successes and achievements. This measurement of the M&E System design quality has positive relatively high (0.59) correlation with the existence of M&E manual/procedures prior to program implementation (Table 11). That could be explained by the designing of the M&E procedures and plan before program implementation will allow the different actors to take the needed time and information, learn from previous programs M&E System in the programming of data collection and analysis over the program implementation. 146

155 The clarity of roles and responsibility of different actors in the M&E system is also positively linked to the existence of an M&E manual before program activities implementation (0.52). If the roles and responsibilities of different actors are well defined prior to program implementation, there will be room during program start-up and the first year of implementation to make the necessary adjustment for the way forward with clear pictures and mapping of roles and responsibilities of various stakeholders involved in the M&E System. The clarity of the roles and responsibility is a key dimension of the design of the M&E system and is closely linked to the time the M&E procedures are elaborated. When roles and responsibilities are defined just after the program starts implementing its activities, there is little chance they remain clear to the stakeholders already involved in different other activities, and there might be the risk for the actors to consider M&E activities are additional tasks that are not relevant to programs results achievement, hence performing them is not required. Clearly, the existence of a program high-quality M&E plan/procedure prior to implementation of activities is not significantly correlated with the following variables: 1. Quality (and simplicity) of tools and techniques for data collection and analysis, 2. The existence of a capacity building plan commensurate with the M&E Plan/Procedures objectives, 3. The M&E procedures are readily available to the different actors. These independent dimensions of the Design Quality sub-dimension might be a good set of measurements to be considered. Nevertheless, the accessibility of the M&E procedures to the various actors is positively significantly correlated with the quality (and simplicity) of tools and techniques for data collection and analysis (0.52). In short, the Design Quality sub-dimension of the M&E System System Quality dimension of the M&E System effectiveness framework would be better assessed through the following measurements: 1. The existence of program a high-quality M&E plan/procedure prior to implementation of activities, 2. Quality (and simplicity) of tools and techniques for data collection and analysis, 3. Existence of a capacity building plan commensurate with the M&E Plan/Procedures objectives. Globally, for Set-up Quality variables (Table 12), the test of correlation is showing an Alpha of standardized Cronbach of 0.82, which demonstrates all variables are not independent but strongly correlated between them. The timely availability/recruitment of the M&E team to implement M&E activities is strongly correlated with the timely availability of resources (0.68). In the same time, the timely availability of resources for M&E System implementation is highly correlated with the capacity building of core actors involved in the M&E system. Early availability of funding to implement M&E activities is key to the Set-Up quality as it conditions directly other characteristics such as the timely recruitment and availability of the M&E team and the capacity building of core actors involved in the M&E system (Table 12). 147

156 Table 12. Matrix of correlations for variables/measurements of System Quality Set-Up Quality Variables for Set-Up Quality - Correlations 2. Resources were timely available to implement M&E activities 3. Capacity building of core actors responsible for the M&E activities implementation 4. There were tests of M&E tools and techniques prior to M&E activities implementation 5. Existence of quality baseline information for future program results assessment 1. The M&E team was available/recruited timely to implement program M&E activities The cores measurements that are not significantly correlated to the early availability of resources are the existence of quality baseline information for future program results assessment (0.49) and the testing of M&E tools and techniques prior to M&E activities implementation. Hence, these are the main variables to be considered for the set-up quality sub-dimension of the M&E System System Quality dimension. In conclusion, the M&E System-System Quality-Set-Up Quality would be assessed by the following measurements: 1. The timely availability of resources to implement M&E activities, 2. The Tests of M&E tools and techniques prior to M&E activities implementation, and 3. The existence of quality baseline information for future program results assessment. In analyzing the table of correlations for Operations Quality (Table 13), it is clear that the existence of an M&E data platform accessible to the program's stakeholders through generally web-based IS or the organization's Intranet system, is highly positively linked to the capacity of the organization M&E managers and actors to better organize the metadata for audits and assessment purpose. It is also highly positively related to the level of involvement of program components core actors in the M&E activities; a data platform is then a wonderful way of socialization for development programs in implementing their M&E activities. There is a strong link (0.73) between the existence and functionality of decentralized M&E sites to cover the program zone of influence and the capacity of the M&E team to visit all zone of influence for data collection, feedback, and analysis (Table 13). Decentralized M&E sites in the program zone of intervention may greatly help facilitate the work of the M&E team in the headquarters to cover the different zones and perform their jobs. Overall, despite the high statistical link shown in the correlation, these two dimensions can work perfectly as good measurements of the Operations Quality of the M&E System. Table 13. Matrix of correlations for variables/measurements of System Quality Operations Quality 148

157 Variables for Operations Quality - Correlations 2. Organization of metadata for M&E data audits purpose 3. Capacity of the M&E team to visit all program zone of influence for data collection feedback and analysis 4. Existence/ Functionality of decentralized M&E sites to cover program zone of influence 5. Level of involvement of program components core actors in the implementation of the M&E activities 1. Existence of M&E data exchange platform accessible to core program stakeholders The organization of metadata for M&E data audits purpose is also highly linked to the level of involvement of program components core actors in the implementation of the M&E activities (0.78). When the responsible for program activities implementation are involved in the M&E System activity, there is great chance to get the M&E metadata data well organized and effectively used when data quality audits come over the program implementation and beyond. One of these measurements might be a good representative of the other and vice-versa. In short, the key measurements to be selected for the sub-dimension Operations Quality of the dimension System Quality of the M&E System framework are: 1. The existence of M&E data exchange platform accessible to core program stakeholders, 2. The capacity of the M&E team to visit all program zone of intervention for data collection feedback and analysis, 3. Existence/ Functionality of decentralized M&E sites to cover program zone of intervention. Table 14. Matrix of correlations for variables/measurements of System Quality Maintenance Quality Variables for Operations Maintenance Quality- Correlations 1. Frequency of reviews/assessments of the M&E system design and operations Quality of the program M&E data platform Percentage of M&E agents benefitting from staff development program 4. Technical support received from specialized M&E agencies and other organizations In the analysis of this correlation matrix of the measurements of the sub-dimension Maintenance Quality of the System Quality dimension of the effective M&E System framework (Table 14), the frequency of reviews/assessment of the M&E System design and operations is greatly linked to the quality of the program M&E data platform or portal (0.70). All the other dimensions for measurements 149

158 of this sub-dimension are likely independent parameters. Meaning, the frequency to carry out M&E system assessment or review (main), the percentage of M&E agents benefitting from staff development activities over the program implementation (0.27), and technical support received from specialized M&E agencies and other organizations (0.11) are not significantly correlated and seems to be independent while generally agencies and organizations that provide technical supports to M&E systems generally request initial review or assessment prior to executing their support. In short, the sub-dimension Maintenance Quality of the dimension System Quality of the effective M&E System framework would be measured through: 1. The frequency of reviews/assessments of the M&E system design and operations, 2. Percentage of M&E agents benefitting from staff development program, 3. Technical support received from specialized M&E agencies and other organizations. Table 15. Matrix of correlations for variables/measurements of System Quality Resources Quality Variables for Operations Resources Quality - Correlations 2. Qualification and level of competency of other M&E human resources (enumerators, consultants, etc.) 3. Quality of equipment used to carry out the M&E activities 4. Existence of appropriate financial resources as per the approved M&E plan to carry out M&E activities 1. Qualification and level of competency of the program M&E managers In analyzing the relations between variables under the sub-dimension Resources Quality of the dimension System Quality (Table 15), there is a significant positive linkage between the qualification and level of competency of the program M&E managers and the qualification and level of competency of other M&E human resources (enumerators, consultants, etc.) As the first ones are generally responsible for developing the scope of works and hiring the second ones, the best the M&E Managers are, the greater would be the other personnel recruited to perform M&E activities over the program implementation. If the M&E managers are not very good in the required fields, there is little chance they identify, recruit and supervise high caliber personnel to implement the M&E activities, mainly if they have full responsibility for that process. The existence of quality equipment to carry out the M&E activities is positively correlated with both the qualification and level of competency of the program M&E managers (0.63) and the existence of appropriate financial resources as per the approved M&E plan to carry out M&E activities (0.54). If the program has high-quality M&E Managers who have the required competencies and qualifications, and at the same time the appropriate financial resources are available as per the M&E plan and budget, there is an 150

159 excellent chance the M&E equipment needed and described by the M&E Managers for the bidding would be purchased timely for the sake of smooth activities and operations running on the M&E System. In short, the sub-dimension Resources Quality of the System Quality dimension of effective M&E System would be measured through the following criteria: 1. The qualification and level of competency of the program M&E managers, 2. The qualification and level of competency of other M&E human resources (enumerators, consultants, etc.), 3. The existence of appropriate financial resources as per the approved M&E plan to conduct M&E activities. The scores of the three measurements were added to create a variable called System Quality Synthesis, which is a quantitative variable that descriptive statistics are as follows (Table 16). Table 16. Descriptive Statistics for System Quality Synthesis The analysis of differences between programs on the design of their M&E System was made. Based on the table on descriptive statistics, the mean score for M&E System design quality is (64.81%). Overall, the M&E Systems in the West Africa region perform 64.81% of the requirements in the process of designing their M&E System (Table 16). In short, in average they perform maximum two requirements among the three selected criteria. Most of the M&E Systems in the region are not entirely well designed. Table 17. System Quality Synthesis Relation with the Program/Project domain/sector We looked at the differences in the design between the sectors of the development programs. The analysis (Table 17) has shown that the relationship is not significant (p-value = 0.65; Fisher = 0.50). The weakness in designing good M&E System does not have significant relation with the programs sectors, all of the 151

160 sectors have difficulty in designing good M&E Systems even though the means in Infrastructures (49.38), Gender Empowerment (48.08), Food Secutity (48.75), Agriculture (49.26), Poverty and Social Impact, Education-Research (49.43) and Rural Development (48.33) are below the average (49.49) in the region. We looked at the relationship between System Quality and M&E System and the dimension (regional, national) of the development programs (Table 18), the linkage is weakly significant (p-value = 0.13; Fisher = 1.68). The national programs (49.44) have M&E System better designed slightly compared to regional programs (44). Designing M&E Systems for regional programs might be more complex due to the number of institutions, organizations, and stakeholders involved. Many of them may have a different understanding of M&E procedures and techniques, and different capabilities. Hence, designing regional M&E System might be more challenging in West Africa because of weak capacity of regional organizations in this field. Table 18. System Quality Synthesis linkage with the Dimension of the program/project In analyzing the linkage between the quality of the design of M&E System and the duration of the program/projects implemented by the organizations (Table 19), the results have shown that there is a significant linkage (p-value = 0.05; Fisher = 3.11). Development programs with two-year duration have weak quality design level (39.2) compared to three-year, four-year and five-year programs. Two-year programs are generally emergency programs that do not take or have not the necessary time to design effective M&E Systems. Table 19. System Quality Synthesis Linkage with the Program/Project duration? 152

161 In conclusion, the System Quality dimension of the effective M&E System model proposed in line with the results in iteration 1 (literature review), after the analysis of the first case study of the iteration 2, the case of the Office du Niger Contract Plan program, has shown being captured through tangible sub-dimensions including: 1. Design Quality, 2. Set-Up Quality, 3. Operations Quality, 4. Maintenance quality, and 5. Resources Quality. From results of the case study, proper measurements were proposed, for each sub-dimension. The analysis of data from the survey in iteration 3 has shown results presented in this section, through the deep analysis of correlation matrixes between measurements of each subdimension. The main outcome was the effective simplifying of the number of measurements in each sub-dimension with the selection of significant quasi-independent variables. The analysis has shown that quality design of the M&E procedures is key to any success of the M&E system. To ensure a successful design of the M&E system, there is need to start the process very early, prior to the implementation of any program activity. The M&E plan, manual or procedures must be developed before the program s stakeholders engage in the implementation of the various activities, generally during the start-up period or just before, in the inception phase. A well-designed M&E System should present quality tools and techniques for data collection and analysis. Data gathering and analysis are the key initial M&E activities in the learning process, hence, designing well-crafted and straightforward tools for these activities is key to any success in the design of the M&E System. A welldesigned M&E System also should integrate a capacity building plan commensurate with the M&E Plan/Procedures objectives, as M&E is a critical activity in managing the development program, there is need to ensure the capacity building plan exists for the progressive improvement of capacity of the actors in charge or involved in the M&E System. In many cases, organizations design good M&E Systems but fail in making them operational because of difficulty to ensure a good set-up process. One of the key elements of a good set-up of the M&E System is the timely availability of resources to implement M&E activities. The difficulty in mobilizing the needed resources to set the required equipment, personnel for the M&E activities at an early stage of the program might hinder the basic foundations of effective M&E System. Hence the timely availability of resources to ensure the mobilization of those needed elements is key in the program start-up period to ensure that quality conditions for M&E System effectiveness are in place. Tools for data collection and techniques for analysis should be tested and aligned with the level of quality required to inform rigorous knowledge creation process. Many programs M&E Systems fail in collecting quality data on the right aspects of the program's theory of change because of unsuccessful tools not tested at early stages of the programs. Development programs are subject to various assessments and evaluations including performance and impact assessments over their implementation. 153

162 Hence baseline information availability for possible comparisons and rigorous evaluations are key to inform program outcomes for decision-making and future programming. The existence of quality baseline information for future program results assessment is a key aspect to perform during the M&E System set-up generally before implementation of any program activity. One of the key sub-dimension of the M&E System Service Quality is how the system operates (Operations Quality) and ensures the timely delivery of all the required products and services to feed the process of decision-making. For the M&E System to function effectively, as shown by the results and data analysis, there might be a need to set an M&E data exchange platform accessible in real-time to core program stakeholders. This data exchange platform is not necessarily to be a web-based application; it might be any form of exchange platform aligned with the organizations way to organize their information sharing system. It is central to ensure that all stakeholders have access to the needed information to take appropriate decisions and actions at all levels of management and program implementation. Even though a data exchange platform is important, face-to-face work in the M&E System through field visits, local workshops, including with the direct beneficiaries, are critical aspects of program effective M&E System. Decentralized M&E Systems can facilitate data collection as the enumerators and data collectors are closer to the beneficiaries and the places where the development actions are implemented. Development programs implemented nationwide should use the existing local institutions to decentralize their M&E System instead of using newly created costly units that disappear at the end of the programs. Making M&E Systems effective is a continuous work in the organizations in charge of development management. The effectiveness of the M&E System starts before the implementation of the program activities, continues over its implementation and beyond. There are critical steps that need to be performed to ensure that the M&E system is effective for improved decision-making and measured risk-taking. The different actions taken by the organizations in their effort to ensure that the M&E System is and remains effectively prepared and carried out over the program implementation; they are called here M&E System Maintenance Quality sub-dimension. To ensure the M&E system remains effective, there is need to organize its assessment on a regular basis, at least during program performance evaluations and annual reviews. A good measurement of the Maintenance Quality is to verify if the M&E capacity building activities are effectively implemented, one of the main indicator is the percentage of M&E agents benefitting from staff development program. Training, coaching, and transfer of knowledge, socialization of best practices are key elements to ensure a proper maintenance of the M&E System effectiveness. The M&E Managers should also be up-to-date with the international practices and standards in designing and implementing their M&E activities, hence supports from 154

163 international M&E agencies and training are important to ensure the M&E managers are updated on current knowledge in M&E standards. Resources are key to any given development activity including design and implementing an effective M&E System. If there are not sufficient resources as per the M&E plan and budget, there will be little chance to make the M&E System effective. On the critical measurement under this sub-dimension is the qualification and level of competency of the program M&E managers, M&E Systems need good managers that have the necessary skills, competencies, and leadership to manage the system towards its effectiveness. The M&E managers, who supervise the development of the scope and the M&E activities in line with the M&E plan, propose the description, in relation to the specialized units, of the needed equipment and logistics to run and coordinate the M&E activities. Hence, another good measurement of the Resources Quality sub-dimension is the qualification and level of competency of other M&E human resources (enumerators, consultants, etc.). The different actors, consultants, and experts involved in the M&E System activities should be selected based on qualification and competencies as per the defined scope. A third key measurement of the Resources Quality sub-dimension of System Quality is the existence of appropriate financial resources as per the approved M&E plan to carry out M&E activities. Financial resources are key to the M&E System effectiveness; they should be available and mobilized timely for the M&E Managers to conduct the needed activities as per the approved budget and scopes. In short, M&E System Effectiveness System Quality sub-dimension of the Service Quality Measurement were presented and discussed in this conclusion to see what are the rigorously established measurements and criteria that need attention of organizations and institutions in charge of development management in the building of an effective learning system for improved decision-making and measured risk-taking for greater capability to advance development objectives M&E-System Information Quality In analyzing the table of correlations (Table 20), the existence of program analytical accounting system (linking clearly Budget/Expenditures to activities, outputs and other characteristics of deliveries) is key and is linked to the accuracy of the program financial data (0.67). It is also slightly related to the percentage of program inputs indicators that meet/met data quality assessment standards (0.56). Another measurement relevant to this sub-dimension of Inputs Information Quality is the effective approach for conversion of program beneficiaries contributions. The analytic accounting system is critical to link activities and achievements to expenditures and ensure the possibility to track how the development program uses the mobilized resources to implement the agreed action plan and activities. It also relates to the accuracy of the financial data, with which the analysis is showing a significantly high correlation. The existence of 155

164 a clear approach to estimate the contribution of beneficiaries to the implementation of programs activities is one of the good measurements of the Input Information Quality; this variable is showing little correlation with the other variables. Hence it is an independent measurement. Table 20. Matrix of correlations for variables/measurements of Information Quality Input Information Quality Variables for Input Information Quality - Correlations 2. Program analytical accounting system (linking clearly Budget/Expenditures to activities, outputs and other characteristics of deliveries) 3. Accuracy of the program financial data 4. Approach for conversion of program beneficiaries contributions 5. Average percentage of program inputs indicators that meet/met data quality assessment standards 1. Average percentage of program inputs indicators that are timely informed In summary, the Input Information Quality sub-dimension of the Information Quality dimension of the effective M&E System framework will be measured by three key variables: 1. The program analytical accounting system (linking clearly Budget/Expenditures to activities, outputs and other characteristics of deliveries), 2. The approach for conversion of program beneficiaries contributions, 3. The program inputs indicators meet data quality assessment standards. The average percentage of program output indicators that meet data quality assessment standards is showing significant positive linkages with the average percentage of program output indicators that are timely informed (0.66), and the program outputs indicators cover/covered fully all the program results (0.69) (Table 21). Development programs output indicators are key to measure the achievements towards the results and need to meet agreed upon high-quality data standards to ensure that information and knowledge created for decision-making are using the right available data and facts from the program implementation. Development programs set a certain number of tools and techniques to collect and assemble data for various uses that lead to the creation of rigorous and meaningful information on the program implementation and achievements. Hence, having data respecting data quality standards set forth by the M&E Managers in the organizations strategic approach in the process of information creation, is critical to any success in taking decisions that would lead the programs to the desired outcomes. 156

165 As shown in the ASER case study, the program output indicators should be well aligned with the sector results in order to allow easy consolidation with national/sector statistics. To make sure the program has not built its M&E System in isolation, it should be possible to merge the programs output information to the national statistics and inform on how the program is a contribution to the objectives of the national or regional development strategies. How the programs output data are well aligned with the sectoral objectives, statistical alignment to the country and regional data sets is one wonderful measurement of the Output Information Quality. Table 21. Matrix of correlations for variables/measurements of Information Quality Output Information Quality Variables for Output Information Quality - Correlations 2. The program outputs indicators cover/covered fully all the program results 3. Average percentage of program output indicators that meet/met data quality assessment standard 4. The program output indicators are well aligned with the sector results in order to be easily consolidated with national/sector statistics 1. Average percentage of program output indicators that are/were timely informed In short, the sub-dimension Output Information Quality of the Information Quality dimension of effective M&E System would be measured through the key variables as follows: 1. The program outputs indicators coverage of the program results, 2. Program output indicators meet data quality assessment standard; 3. The program output indicators alignment with the sector results for consolidation with national/sector statistics. The analysis has shown (Table 22) that the quality of coverage by the outcomes indicators of beneficiaries use of program products and services is highly linked to the quality of coverage by the outcomes indicators of beneficiaries satisfaction with program products and services (0.73). The processes of intermediary transformations from products and services delivery (0.70), the satisfaction of beneficiaries (0.73), the users of the products and services, the adoption of the process by the beneficiaries (0.81), and the higher intermediate changes (0.72), have shown progressive correlations that confirm the M&E of outcome information alignment with the program's theory of change that leads to the desired impacts. The three selected best measurements in this Outcome Information Quality sub-dimension of the Information Quality dimension of effective M&E System are: 1. Percentage of program outcomes indicators that are/were informed timely, 2. Quality of outcomes indicators coverage of program intermediate results, 157

166 3. The quality of information generated by outcomes indicators on intermediate changes at beneficiaries level. Table 22. Matrix of correlations for variables/measurements of Information Quality Outcome Information Quality Variables for Design Quality - Correlations 2. Quality of outcomes indicators coverage of program intermediate results? 3. Quality of coverage by the outcomes indicators of beneficiaries use of program products and services? 4. Quality of coverage by the outcomes indicators of beneficiaries satisfaction with program products and services? 5. Quality of coverage by the outcomes indicators of beneficiaries adoption of program products, services, and approaches? 6. Quality of information generated by outcomes indicators on intermediate changes at beneficiaries level? 1. Percentage of program outcomes indicators that are/were informed timely The availability of quality information for relevance analysis, the availability of quality information for efficiency measurement and the availability of quality information for effectiveness measurement are highly correlated among them (0.76, 0.86 and 0.90). During performance evaluations, the existence of quality M&E data is key to the success in gauging these three core criteria. The analysis is showing that the quality of information needed to assess relevance, effectiveness, and efficiency of development programs are linked (Table 23). Availability of quality information for effectiveness measurement is also highly correlated with the availability of quality information for meaningful program sustainability analysis (0.71), while quality information for sustainability measurement is showing little correlation with quality information on efficiency and quality information on relevance (respectively 0.58, and 0.68). The capability of the M&E system to translate best practices and lessons learned into meaningful information is not significantly correlated with all the other variables (quality information for effectiveness, quality information for efficiency, quality information for relevance, and availability of quality information for meaningful program sustainability analysis - respectively 0.47, 0.52, 0.45 and 0.59). 158

167 Table 23. Matrix of correlations for variables/measurements of Information Quality Performance Information Quality Variables for Design Quality - Correlations 2. Quality information is available for program efficiency analysis purpose 3. Quality information is available for meaningful program effectiveness analysis purpose 4. Quality information is available for meaningful program sustainability analysis purpose 5. Capability of the M&E system to translate best practices and lessons learned into meaningful information? 1. Quality information is available for program relevance analysis purposes In short, the Performance sub-dimension of the Information Quality dimension of the Effective M&E- System can be measured with three core independent variables: 1) Quality information is available for program efficiency analysis purpose, 2) Quality information is available for meaningful program sustainability analysis purpose, and 3) Capability of the M&E system to translate best practices and lessons learned into meaningful information. Table 24. Variable/measurement of Information Quality Risk Management Information Quality E. Risk Management Information Quality 1. All the assumptions and risks in the program/project logical framework are/were translated into meaningful risks mitigation/attenuation milestones monitored and informed on a regular basis 5/5 means that all risks and assumptions in the program results/logical framework have been properly assessed in terms of probability of occurrence and impact on the program and mitigation/attenuation actions are defined, monitored and reported as appropriate over the program implementation The last sub-dimension of the dimension Information Quality of Effective M&E-System framework is Risk Management Information Quality and is measured by the variable All the assumptions and risks in the program/project logical framework are/were translated into meaningful risks mitigation/attenuation milestones monitored and informed on a regular basis - all risks and assumptions in the program results/logical framework should have been properly assessed in terms of probability of occurrence and 159

168 impact on the program and mitigation/attenuation actions defined, monitored and reported as appropriate over the program implementation (Table 24). In analyzing the relationships between Information Quality and the sector of the development program, the results have shown that the linkage is not significant (Table 25). The Information Quality does not depend on the sector of the development program while Education-Research (44.14), Agriculture (44.94), Rural Development (43.94) are below the average of Information Quality (45.54). These sectors need critical attention when it comes to managing M&E Information Quality as their level is weak compare to the average. Table 25. Information Quality Synthesis Relation with the Program/Project domain/sector What is/are the Pro... Synthesis Informati... Agric ultur e Infras tructu res Energ y Pover ty - Social Impa ct Gend er Empo werm ent Gover nanc e and Demo cracy Rural Food Devel Secur ity opme nt Educ ation Healt - h Rese arch Trade and Inves tmen t Other Total Mean 44,94 46, ,21 45,15 48,2 45,5 43,94 44,14 48,64 46, ,54 Standard 8,21 7,55 4,52 6,31 7,19 2,68 6,39 7,3 8,01 7,89 2,08 7,83 7,08 deviation Frequency Effective responses : 46 p-value: = 0,60 ; Fisher = 0,48 (The relationship is not significant) There is a significant relationship (Table 26) between Information Quality and the duration of the development program. Information Quality is low (36.4) for two-year program and higher (46.19) for threeyear program, and for four/five-year programs (45.89). For emergency programs (two years) great attention should be put on Information Quality as the level is low compared to programs with three or more years period. Table 26. Information Quality Synthesis Linkage with the Program/Project duration? 160

169 The relationship between Information Quality and coverage of the development program is not significant (Table 27). But regional development programs have more challenges related to Information Quality in their M&E System. Table 27. Information Quality Synthesis linkage with the Dimension of the program/project In conclusion, in studying the second dimension of the Effective M&E System framework, we looked at the case of the Senegalese Rural Electrification Program (ASER) showing the different aspects and steps in scrutinizing what are the key measurements that might underline the sub-dimension Information Quality. Five core sub-dimensions were identified covering the large spectrum of information needed for development programs: Inputs Information, Outputs Information, Outcomes Information, Performance Information, and Risk Management Information. Quality information in these five dimensions are key to ensure the effectiveness of this dimension of Effective M&E System framework. The analyses of the correlations between variables for measurement of each sub-dimensions have shown that correlations exist between variables that helped simplify the number of measurements retained for further analysis. For quality information on inputs of development programs, indicators that measure this sub-dimension include the percentage of input indicators that meet agreed upon data quality standards at the organization level. Inputs information relates to the description and tracking of the use of resources, technical and financial, to implement development programs. Organizations need to put in place agreed upon standards for the inputs measurement that should align with national and regional standards for the establishment of relevant benchmarks and possible comparisons of efficiency. The indicators that measure the program inputs should meet the standards for data quality. One finding is the capability of the M&E System to monitor and report rigorously on beneficiaries contribution is a key factor of quality inputs information. Another key measurement of the quality of development programs inputs information is the quality of the analytical accounting system as shown in the ASER case study and supported in this analysis. The analytical accounting system is linking the planning and use of resources to the program results and activities. When it is well designed so as any single use of program resources is linked directly to a given activity and program results, the inputs information might have greater chance to be at high quality. 161

170 The analysis also examined the measurements for the sub-dimension Output Information Quality of the Information Quality dimension of the Effective M&E System framework. The ASER case study concluded that four measurements need to be considered for this sub-dimension. The analysis has shown the most relevant measurements might be summarized to three. The coverage of output indicators of the program s results is a key measurement of Quality Output Information. Outputs indicators measure achievements and program deliveries in products and services to the various beneficiaries. These outputs indicators should cover the program results entirely to avoid missing any development achievement. This also includes the capability not to overestimate or underestimate programs products and services. M&E System Managers should then make sure that the output indicators track perfectly all the program results. Another key measurement is the extent to which outputs indicators meet data quality standards like the inputs indicators. Data quality standard are very important to clarify and share with all stakeholders involved in the M&E System. One key measurement of the quality of output information for a given program is the alignment of the output indicators with the sectoral results and statistics for national consolidation of development programs actions. Development programs are contributing to the achievement of national or regional policy on specific results, so their output indicators must be aligned with the results they are intended to contribute to. This is, as shown by the analysis, a key factor of measurement of the quality of output information. Another sub-dimension examined under the dimension Information Quality is the Outcome Information Quality which shows, in the ASER case study conclusion, that key measurements to be considered are six. The analysis shows this might be summarized to three core measurements based on the correlations examined. One of the key measurements of outcome information quality is the timely availability of the informed outcome indicators. Generally, development programs are reluctant to invest in outcomes assessments because these exercises are generally costly or the approaches used are also known to be much difficult to apply for many M&E Systems. Therefore, outcome information might not be available in all time compared to input or output information that are collected through routine M&E activities. Yet, not all managers can inform timely decision-makers on the program outcomes or impacts. Hence, the timely availability of outcome information is a wonderful measurement of this sub-dimension. The second key measurement for the outcome information quality is the quality of the coverage of the program intermediate results by the outcome indicators. All program outcome indicators should be captured by well-defined benchmarks and indicators. The objective is not to list a long set of indicators to capture all changes possible at intermediary levels but to select key indicators that can be collected and inform on the main aspects of the changes that occur. The objective is to be realistic, systematic and meaningful when it comes to measuring program intermediary results linked to the lower levels of the results chain. A third key measurement confirmed here is the quality of the information generated by outcome indicators on changes 162

171 at beneficiary level. The quality of data collected and information generated on outcomes at beneficiary level is key to measure the quality of the M&E System on outcome information. In analyzing the measurement for the sub-dimension Performance Information Quality of the Information Quality dimension of effective M&E System, six variables identified in the ASER case study were examined. Three were maintained as core measurements to be considered. The analysis has shown that the quality of information generated for the assessment of the three key performance criteria, namely, relevance, efficiency, and effectiveness, are significantly correlated positively. The analysis has shown that the quality of information for the efficiency analysis during performance evaluation is a key measurement of performance information quality and represents effectively the quality of information on relevance and effectiveness. The quality information needed to analyze program efficiency includes products and services delivered, timeframes for the implementation of activities, strategies in procuring, logistics, financial and human resources management towards the expected results. Information on similar programs is also required to measure benchmark and make relevant comparisons. A second measurement to consider is the quality of available information for program sustainability analysis. The capacity of development organizations to measure the sustainability of their programs is very important to ensure that activities are leading to the desired long-term impacts. Information is needed to measure operational, financial and institutional sustainability. The capability of the M&E System to translate best practices and lessons learned into meaningful information is the third measurement to be considered in this sub-dimension. The M&E System should be equipped with tools and approach to capitalize on best practices and lessons learned on all aspects of development program implementation. Only one measurement was selected in the case study to measure the sub-dimension Risk Management Information Quality. The variable is maintained as there is no room to cross-analyze it with other measurements How M&E System Quality Influences M&E Information Quality? This section analyzes the linkage between System Quality which addresses all the steps in designing and making the M&E System functional and the Information Quality which described all the information managed in the M&E System. The quality of the information generated from the M&E System might have a linkage with the Quality of the M&E System design and functioning. To address this question, we used a linear regression approach, Information Quality as a variable to explain, and System Quality as the explanatory variable. The results are presented below. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of variable: M&E Information Quality from the variable: M&E System Quality. 163

172 Coefficient p-value Standardized coefficient Contribution const < M&E System Quality 0.59 < Equation 1. Equation of model M&E Information Quality vs. M&E System Quality M&E Information Quality = * M&E System Quality The model accounts for 51.89% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of fisher: F = P-value of F: p(f) = <0.01. The M&E System Quality explains about 51.89% of the M&E Information Quality variance in this model, and the two sub-dimensions are highly correlated with The regression statistics shows the consistency of the model and how the way the M&E System is designed, functions and operates explains strongly the quality of the information generated. In short, the equation of the model is showing that if the M&E System is not well designed, set-up, well operating, well maintained and appropriate resources mobilized, there is little chance for the M&E Information to be at high quality M&E-System Service Quality The capacity of the M&E System to generate timely program mandatory reports is a key measurement of the sub-dimension Information availability of the Service Quality dimension. Timely availability of information is key to decision-makers. Development programs are set in short periods, generally, one to five years, as data are collected at certain times, information should be available timely to feed decisionmaking process. Development programs design and carry out evaluations and other tailored studies to assess different aspects related to the program management and results chain. The M&E System should be able to implement timely such studies and make the results available for decision-makers. These two aspects are significantly positively correlated (0.77) as shown by the analysis (Table 28). The mandatory reports can be timely available, but might not contain the appropriate information or dashboards. The percentage of required dashboard available timely is then another measurement of information availability. The analysis has shown, these two measurements are also positively significantly correlated. In short, the timely 164

173 availability of M&E mandatory reports is a key measurement of information availability and can represent the two other measurements. Table 28. Matrix of correlations for variables/measurements of Service Quality Information availability 1. Capacity for timely generation of program M&E mandatory reports Capacity for timely production of evaluation and other M&E studies reports % of M&E dashboards made available timely to program managers and other stakeholders 4. Existence of a knowledge sharing system on M&E information Existence of M&E data communication tools Knowledge sharing system is key to ensure that M&E information is available for stakeholders to learn and take relevant decisions on program implementation and results. The existence of a knowledge sharing system in a key measurement of the subdimension Information Availability of the dimension Service Quality of the M&E System framework. The third key measurement is the existence of communication tools for M&E data. In short, the Information Availability subdimension of the Service Quality dimension is measured through the key following variables: 1) Capacity for timely generation of program M&E mandatory report, 2) Existence of a knowledge sharing system on M&E information, 3) Existence of communication tools for M&E data. Table 29. Matrix of correlations for variables/measurements of Service Quality Information accessibility 1. % of program managers having access timely to evidence-based data on program achievements and results Level of access of other program stakeholders to M&E data 3. Access to monitoring data for evaluation purpose

174 1. % of program managers having access timely to evidence-based data on program achievements and results Access to program M&E data for the large public, media, and researchers 5. Technical user friendliness of M&E data for all users There are significant linkages between % of program managers having access timely to evidence-based data on program achievements and results and level of access of other program stakeholders to M&E data (0.83), and with access to monitoring data for evaluation purpose (0.77). In serving program managers with relevant evidence based information, the M&E System ensures that all other relevant stakeholders also have access to information on program achievements and results (Table 29). In this, case, there will also be a chance for the evaluation teams to access M&E data for evaluation purpose. The other correlations are not significant enough. In short, the sub-dimension Information Accessibility of the Service Quality dimension of effective M&E framework is captured through the following measurements: 1) % of program managers having access timely to evidence-based data on program achievements and results, 2) Access to program M&E data for the large public, media, and researchers, 3) Technical user friendliness of M&E data for all users. Table 30. Matrix of correlations for variables/measurements of Service Quality System Responsiveness 1. Availability of M&E data upon requests by the program stakeholders Capacity to carry out timely specific M&E studies requested by the program managers 3. Capacity of the M&E team to produce tailored reports timely for program management and learning events 4. Capacity of the M&E system to respond to the sectorial Ministry information request on the program achievements and results Significant relationships exist between three proposed measurements for the sub-dimension System Responsiveness of the dimension System Quality of the effective M&E System framework. The responsiveness of the M&E System is key to ensure a good learning process for evidence-based decision- 166

175 making and measured risk-taking in managing development. Over the program implementation various stakeholders and program managers might request information that was not initially planned, the effective M&E System should be able to mobilize techniques and resources to satisfy the demand for the new information need. The capacity to carry out timely specific M&E studies requested by the program managers is significantly linked to the capacity of the M&E team to produce tailored reports timely for program management and learning events (0.79). Likely the M&E System responsiveness to carry out new studies that were not initially planned is linked strongly with the capacity to produce any relevant report required by program managers even if the templates and guidelines do not exist in the M&E manual. Same for the capacity of the M&E system to respond to the sectorial Ministry information request on the program achievements and results (0.71) is linked to the second measurement (Table 30). In short, the sub-dimension System Responsiveness of the dimension Service Quality of the effective M&E System framework is measured through the following variables: 1) Timely availability of M&E data upon requests by the program stakeholders, 2) Capacity to carry out timely specific M&E studies requested by the program managers, 3) Capacity of the M&E system to respond to the sectorial Ministry information request on the program achievements and results. Another sub-dimension of the Service Quality dimension of effective M&E System framework analyzed is the System flexibility. The system flexibility is measured through the following variables that are linked positively (0.75). Both variables are maintained to allow comparison: 1) Capability of the M&E system to integrate efficiently new national directives and/or international standards on results-based management, monitoring and evaluation, 2) Ability of the M&E system to integrate timely new information needs following programs revisions and strategic adjustments. Table 31. Matrix of correlations for variables/measurements of Service Quality System Sustainability 1. Capacity of the M&E system to remain fully functional when major changes occur in the M&E team Capacity of the M&E decentralized units to be functional with minimum supervision and support 3. Capacity of the program managers and other stakeholders to access M&E information with minimum support from the M&E team 4. Utility of the M&E data after program closure for capitalization purposes

176 Development programs are generally planned for maximum five years and extended in many cases. The sustainability of the M&E System as shown in the case of the Water and Food Security initiative of FAO and AECID is very important for organizations that manage development within the similar framework and for long term measurement of development impact. The capacity of the M&E System to remain fully functional after major changes in the M&E team is a critical measurement of its sustainability. The five measurements proposed from the case study are not significantly linked as shown in this analysis, they are independent variables (Table 31). Three were selected as the most independent of relevant measurements of the sub-dimension System Sustainability. In short, the sub-dimension System Sustainability of the dimension Service Quality of the M&E System framework will be measured through the following variables: 1) Capacity of the M&E system to remain fully functional when major changes occur in the M&E team, 2) Capacity of the program managers and other stakeholders to access M&E information with minimum support from the M&E team, 3) Utility of the M&E data after program closure for capitalization purposes. Conclusion on Service Quality. The Dimension Service Quality of the M&E System effectiveness framework has five sub-dimensions, as shown by the case study on the FAO regional development program Water and Food Security Initiative in Africa funded by AECID. The five sub-dimensions are: 1- Information Availability, 2- Information Accessibility, 3- System Responsiveness, 4- System Flexibility, 5- System Sustainability. This section analyses the different relationships between the selected measurements of these five sub-dimensions. The analysis has shown, for the sub-dimension Information Availability, the core measurements include the capacity of the M&E System to generate timely M&E mandatory reports on the development programs. The timely availability of such information is key to decision-makers in the organization leaning processes to advance the development actions to success. These reports include the periodic documents produced for the Government and various partners involved in the programs; they also include tailored reports requested by the management and other key stakeholders. A second important measurement for this sub-dimension is the existence of knowledge and information sharing system that serves the platform for various stakeholders to access and learn on the M&E information generated. Without a vibrant and effective knowledge and information sharing system, there is little chance for the M&E information to reach to relevant stakeholders and decision-makers who need to use them as evidence-based information to take the necessary actions to advance the development objectives. A third fundamental measurement of this sub-dimension is the existence of communication tools to share the M&E Information generated. M&E information should be simplified and presented in a user-friendly way that increases the access for different stakeholders to the needed information. M&E System should then be equipped with various communication tools to present summaries from the database and different studies carried out in the program implementation. 168

177 To ensure a better accessibility of program M&E information by various stakeholders, there is a need for the program managers to access timely to all needed evidence-based information on the program achievements and results. Delivering timely the needed evidence-based information for management purposes and learning process is one of the key roles of an M&E System to be effective. Another key measurement of the sub-dimension Information accessibility is the access to M&E information for the large public, media, and researchers. The M&E information should be disclosed to the large public and population to ensure transparency and equity on development program achievements and results. This democratization of the M&E information is a key factor to ensure participation and evidence-based decision-making in managing development actions. For the M&E System to be effective through a greater Service Quality, it must be responsive to all relevant requests on information and knowledge related to the program achievements and results. The responsiveness, as shown through this analysis, could be measured by the timely availability of M&E information upon requests from the main stakeholders. Development actions generally are demand driven, and many stakeholders involved in the process are concerned about the achievements and results, that is not necessarily in the mandatory reports. Then, the M&E System should be prepared to receive and respond timely to various information requests from the key actors. In many cases, the Government, program managers and core development partners involved in the actions might request specific studies to enrich the decision-making process with additional relevant information. The M&E System should then be better prepared to carry out those specific studies on the program results to feed the policy-making or adjustment process. The third key measurement of this sub-dimension of the system responsiveness ensuring Service Quality is the capability of the M&E System to respond to the sectoral ministry requests the program achievements and results, mainly at outputs and outcomes levels. The analysis of the case study on the Water and Food Security Initiative of the FAO has shown that System Flexibility is a key sub-dimension of the Service Quality dimension of Effective M&E System framework. In analyzing the relationships between the key proposed measurements of the System flexibility, two variables are maintained. The first core measurement of the system responsiveness is M&E System capability to integrate efficiently new relevant national and international directives or standards on RBM and M&E procedures. The second measurement is the M&E System ability to integrate timely new information need as they are resulting from program review and adjustments Relation Between Dimensions of Effective M&E System This section of the analysis examines the relationships between the three dimensions of effective M&E system: Information Quality, System Quality and Service Quality. For M&E System to be effective, it 169

178 should be built under these three dimensions of the DML Information System success model. These dimensions were operationalized through the three case studies and measurements were proposed. The analysis of the data from the survey in the third iteration has shown the relevant measurements and variables to be considered under each of these dimensions and their respective sub-dimensions. Organizations managing development actions set forth M&E system as performance measurement system to track development results and verify if activities are leading to the desired outcomes. The design and setting of the M&E System is a crucial step that determines the System Quality. The System Quality gauged through the existence of critical guidelines on M&E procedures prior to program implementation, the testing of the M&E tools before their use, the selection and definition of relevant indicators in the program theory of change and results framework. When the M&E System is well defined, set and operates accordingly to the existing resources, including human, technical and financial, there is a chance it generates relevant and quality information. The information generated, if at high quality, might be serviced to the various stakeholders in a way that increases effective learning processes toward better decision-making. The analysis here examines the relationships between the three dimensions and the Service Quality as an independent variable and System Quality and Information Quality as dependent variable or explanatory factors. The results are presented below. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of variable: M&E Service Quality from the variables: M&E System Quality, M&E Information Quality Equation of model: M&E Service Quality = * M&E System Quality * M&E Information Quality Coefficient p-value Standardized coefficient Contribution const M&E System Quality 0.40 < M&E Information Quality 0.53 < Quality of estimate: The model accounts for 74.70% of the variance of the variable to be explained. Coefficient of the multiple correlation: R =

179 P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. The model composed of System Quality and Information Quality explains 74.70% of the variance of Service Quality. The coefficient of multiple correlations has shown a good model. Figure 18. Linkages between the dimensions of M&E System In short, quality system and information contribute the effective M&E Service. In the M&E effectiveness framework, the three dimensions are linked, and Quality System and Quality Information contribute to the building of Quality Service of the M&E System (Figure 18) M&E SYSTEM EFFECTIVENESS COMPONENTS M&E-System Effectiveness Results Based Management In this section, the analysis focuses on the probable relationships between the three dimensions of M&E System (System Quality, Service Quality, and Information Quality) and one of the three organizational dynamic capabilities, namely the Results-Based Management practice. In previous sections, it was shown that the three dynamic development management capabilities at organization and institution level that are influenced positively by an effective M&E System: Results-Based Management Practice, Knowledge and Information Management, and Evidence-Based Decision-Making. The analysis bellow explores the 171

180 relationships between RBM practice (variable to explain) and the three dimensions of the M&E System effectiveness framework (explanatory variables). Number of complete observations: 46, out of a total of 46 observations. Estimate of value of variable: RBM Practice from the variables: M&E System Quality, M&E Information Quality, M&E Service Quality. Influential variables: 1 variables contribute to the explanation (with the threshold of 5%): M&E Information Quality. Equation of model: RBM Practice = * M&E Information Quality Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const M&E Information Quality 0.37 < Quality of estimate: The model accounts for 36.40% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. Among the three dimensions of the M&E System, only Information Quality has a significant influence on the Results-Based Management practice of the organization in charge of development actions management. M&E Information Quality ensures quality data and information on the core aspects of the development program theory of change and results framework, namely Inputs Information Quality, Outputs Information Quality, Outcomes Information Quality, Performance Information Quality, and Risk Management Information Quality. Data and quality information of those aspects of development program analyzed and presented by M&E System will feed the different steps of Results-Based Management at the organization level (Figure 19). 172

181 Figure 19. Relationships between RBM Practice and dimensions of M&E System M&E-System Effectiveness Knowledge and Information Management Knowledge and Information Management is a key capability to development management for the organization in charge of development actions. As shown in previous sections, KIM is a fundamental capability for organization and institution to advance development toward success. Effective M&E System is an influential critical factor to KIM. This section analyzes how the three dimensions of effective M&E System influence the KIM. The KIM is the dependent variable to explain, and the three dimensions of Effective M&E System are the explanatory variables. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of the variable: KIM Practice from the variables: M&E System Quality, M&E Information Quality, M&E Service Quality. Influential variables: 2 variables contribute to the explanation (with the threshold of 5%): M&E System Quality, M&E Service Quality. Equation of model: KIM Practice = * M&E System Quality * M&E Service Quality Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const

182 Coefficient p-value Standardized coefficient Contribution M&E System Quality 0.17 < M&E Service Quality 0.30 < Quality of estimate: The model accounts for 78.26% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. M&E System Quality and M&E Service Quality have a significant influence on the Knowledge and Information Management (Figure 20). How organizations and institutions manage knowledge and information depends greatly on the M&E System Quality and the M&E Service Quality. The quality of the estimate is high as the model explains 78.26% of the variance of the variable to explain and the coefficient of multiple correlations is very high, R=0.88. Figure 20. Relationships between RBM Practice, KIM and dimensions of M&E System M&E-System Effectiveness - Evidence-Based Decision Making Evidence-Based Decision-Making (EBDM) is another key development management practice that increases the capacity of organizations and institutions to advance development objectives. This section analyzes how it is influenced by the key components of the M&E System effectiveness framework. EBDM is the 174

183 dependent variable to explain, and Information Quality, System Quality and Service Quality is the explanatory variables. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of the variable: EBDM Practice from the variables: M&E System Quality, M&E Information Quality, M&E Service Quality. Influential variables: 1 variables contribute to the explanation (with the threshold of 5%): M&E Service Quality. Equation of model: EBDM Practice = * M&E Service Quality Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const M&E Service Quality 0.57 < Quality of estimate: The model accounts for 66.51% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. Figure 21. Relationships between RBM Practice, KIM, EBDM and dimensions of M&E System 175

184 M&E Service Quality is, among the M&E System dimensions, the only influential factor for Evidence- Based Decision-Making practice. The other dimensions of effective M&E System, namely Information Quality and System Quality, as shown in the previous section, influence directly M&E Service Quality. Hence, they influence indirectly EBDM (Figure 21) Relationship between Development Management Capabilities This section examines if KIM which is only influencing improved capability to advance development is influencing RBM and EBDM. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of the variable: RBM Practice from the variables: KIM Practice. Influential variables: 1 variables contribute to the explanation (with the threshold of 5%): KIM Practice. Equation of model: RBM Practice = * KIM Practice Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const KIM Practice 0.60 < The quality of estimate: The model accounts for 32.98% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. The analysis has shown that Knowledge and Information Management influences the Results-Based Management practice in organizations and institution in charge of development actions. While RBM is key to ensure proper planning and implementation of development programs as well as their evaluations and adjustment towards to expected outcomes, KIM is necessary to guarantee that RBM is effectively using Quality M&E Service built on effective M&E System Quality. 176

185 Number of complete observations: 46, out of a total of 46 observations. Estimate of value of variable: EBDM Practice from the variables: KIM Practice. Influential variables: 1 variables contribute to the explanation (with the threshold of 5%): KIM Practice. Equation of model: Influence of explanatory variables: EBDM Practice = * KIM Practice Coefficient p-value Standardized coefficient Contribution const KIM Practice 0.98 < The quality of estimate: The model accounts for 54.33% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. As shown in this analysis and in the graph above, KIM is linked to the Evidence-Based Management practice of organizations in charge of development actions. EBDM is key to ensure improved operational, tactical and strategic decisions are well performed to lead the development actions to success. The results have shown that KIM is a key contributor to effective EBDM EFFECTIVE M&E SYSTEM -NET BENEFITS This section analyzes the relationships between M&E System net benefits and the dynamic capabilities of organizations in managing development actions. Organizations need a certain number of capabilities to advance development actions to success, those include, as seen in previous sections, Results-Based Management, Knowledge and Information Management, and Evidence-Based Decision-Making. Those capabilities when made operational through effective M&E System, will lead to the M&E System net benefits: Improved Policy and Program Design, Improved Operational Decisions, Improved Tactical Decisions, Improved Strategic Decisions, and Improved Capability to Advance Development. The analysis here explores these relationships in detail. 177

186 Improved Program and Policy Design In describing the net benefits of the M&E System effectiveness framework, the Improved Policy and Program Design, called here NETBENEF01, was the first benefit analyzed in the M&E System effectiveness description. This sub-section analyzes how the dynamic capabilities of organizations in managing development are influencing this benefit. RBM, KIM, and EBDM are the explanatory variables, and NETBENEF01 the dependent variable to explain. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of the variable: NETBENEF01 from the variables: RBM Practice, KIM Practice, EBDM Practice. Influential variables: 1 variables contribute to the explanation (with the threshold of 5%): RBM Practice. Equation of model: NETBENEF01 = * RBM Practice Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const RBM Practice 0.12 < Quality of estimate: The model accounts for 29.24% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. In short, Improved Policy and Program Design is a function of Results-Based Management (RBM) practice at the organization level. RBM is a wonderful development management approach that combines planning, monitoring and evaluation methods and techniques. It is highly influenced by the Information Quality Dimension of Effective M&E System. In turn, it influences strongly the organization capability to design quality policy and program for future programming. Effective M&E System builds on Quality Information 178

187 that increases the capability of development organizations and institutions RBM practice, that will then lead to improved program and policy design in future planning processes Improved Operational Decisions In the analysis of the needed capabilities to manage development actions properly, and their recurrent outcome on decisions made to pilot or adjust development programs, the second benefit gauged was the Improved Operational Decisions, called here NETBENEF02. As shown in previous sections, operational decisions for programs piloting and short-term adjustments, touch the inputs-outputs levels of the program theory of change or results framework. During annual reviews, mid-term evaluations or in even day-to-day program management, organizations take decisions on operations related to inputs-outputs to adjust progressively development actions towards success that lead to the desired outcomes. The analysis bellow scrutinizes the relationships between the three managerial capabilities, namely RBM, KIM and EBDM, influenced by the dimensions of effective M&E System, and the Improved Operational Decisions benefit. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of variable: NETBENEF02 from the variables: RBM Practice, KIM Practice, EBDM Practice Influential variables 1 variables contribute to the explanation (with the threshold of 5%): EBDM Practice. Equation of model NETBENEF02 = * EBDM Practice Influence of explanatory variables Coefficient p-value Standardized coefficient Contribution const EBDM Practice 0.13 < Quality of estimate The model accounts for 53.78% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <

188 Coefficient of Fisher: F = P-value of F: p(f) = <0.01. The analysis has shown that among the three dynamic capabilities of organizations to manage development actions, only Evidence-Based Decision-Making has a significant direct influence on Improved Operational Decisions. This benefit or outcome of effective M&E System is greatly influenced by the capability of the organization to build decisions on rigorous evidence. Evidence-Based Decision-Making is a key influential factor to improved operational decisions in managing development as operational decisions touch adjustments at input and outputs levels of the development programs results framework. In short, to ensure a successful pathway towards improved operational decisions in managing development actions through effective M&E System, there is need to create rigorous and effective M&E Service dimension of the M&E System, which is built on quality M&E System (design, functioning and maintenance) and quality M&E Information on program results and indicators. In turn, the quality M&E Service, which will service all program stakeholders with the appropriate knowledge needed in the process of decision-making, will influence positively the Evidence-Based Decision-Making capability of organizations in charge of the development actions. Then, the effective EBDM will finally guarantee that operational adjustments and decisions are well crafted and meaningful to advance development actions to success Improved Tactical Decisions Tactical decisions concern adjustment of program intermediary results and outcomes, and are generally made after mid-term evaluations. Policy decisions at the higher level of the country or region can also lead to tactical adaptation of development programs. The analysis here gauges the relationship between Improved Tactical Decisions and the three development management practices influenced positively by effective M&E System, namely RBM, KIM, EBDM. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of variable: NETBENEF03 from the variables: RBM Practice, KIM Practice, EBDM Practice Influential variables: 2 variables contribute to the explanation (with the threshold of 5%): RBM Practice, EBDM Practice. Equation of model: NETBENEF03 = * RBM Practice * EBDM Practice 180

189 Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const RBM Practice 0.11 < EBDM Practice Quality of estimate: The model accounts for 54.90% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. The analysis has shown that Improved Tactical Decisions, decisions touching the program-specific results, intermediary results, made during mid-course of program implementation or orientation from the core stakeholders to better align the program scope with national or regional priorities, will occur only through the two dynamic capabilities of the organizations in charge of development actions, namely Results- Based Management and Evidence-Based Decision-Making. Combining these two capabilities will lead to better decisions at a tactical level ensuring a better planning and achievement of results at the intermediate level of the theory of change, leading to the desired outcomes Improved Strategic Decisions Strategic decisions touch the development program objectives at higher levels. The only strategic shift could result in those decisions. Generally, they result in program closure or significant modification of the expected results. Strategic decisions occur when the program objectives are to be realigned for better impact, or are no more relevant to address the desired changes. The strategic decision can be taken during midcourse reviews and adjustments. The analysis here is scrutinizing the relationships between strategic decision and the three development management practices necessary to advance development objectives and influenced by effective M&E System, the RBM, KIM, and EBDM. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of the variable: NETBENEF04 from the variables: RBM Practice, KIM Practice, EBDM Practice. 181

190 Influential variables: 2 variables contribute to the explanation (with the threshold of 5%): RBM Practice, EBDM Practice. Equation of model: NETBENEF04 = * RBM Practice * EBDM Practice Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const RBM Practice EBDM Practice 0.08 < Quality of estimate: The model accounts for 50.16% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <0.01. The analysis has shown that like Improved Tactical Decisions, Improved Strategic Decisions is influenced positively by two of the three capabilities, RBM and EBDM. Both tactical and strategic decisions need strong RBM and EBDM practices. These two capabilities are critical for program managers and different stakeholders to take the right decision at tactical and strategic levels Improved Capability to Advance Development Objectives The capability of organizations and institutions to advance development objectives is gauged through the level of the impact made by the development actions they initiate and implement. The greater the impact is, compared to the expected outcomes, the greater is their capability to advance development. This analysis is exploring the linkages between the three capabilities needed to manage effectively development actions, namely RBM, KIM and EBDM, and the Improved Capability to Advance Development Objectives. Number of complete observations: 46, out of a total of 46 observations. Estimate of value of variable: NETBENEF05 from the variables: RBM Practice, KIM Practice, EBDM Practice 182

191 Influential variables: 2 variables contribute to the explanation (with the threshold of 5%): RBM Practice, KIM Practice. Equation of model: NETBENEF05 = * RBM Practice * KIM Practice Influence of explanatory variables: Coefficient p-value Standardized coefficient Contribution const RBM Practice 0.09 < KIM Practice 0.08 < Quality of estimate: The model accounts for 53.75% of the variance of the variable to be explained. Coefficient of the multiple correlation: R = P-value of R: p(r) = <0.01. Coefficient of Fisher: F = P-value of F: p(f) = <

192 Figure 22. Effective M&E System Framework, the status from the results of analyses with quantified coefficients (Standardized) Knowledge and Information Management (KIM) and Results-Based Management (RBM) are key to ensure greater capability of organizations and institutions in charge of development actions capacity to advance development objectives towards greater welfare and freedom of people. Improved Capability to Advance Development Objectives is the ultimate goal of development management, as all practices should lead to the significant capability to achieve development outcomes. Hence, KIM and RBM are essential management practices that affect directly the capability of organizations and institutions to advance development objectives. In short, the three capabilities that are necessary for managing development actions that are, as shown in previous sections, Results-Based Management (RBM), Knowledge and Information Management (KIM), and Evidence-Based Decision-Making (EBDM), are influenced by the dimension of M&E System (Figure 22). In turn, their effective practice will lead to improved program and policy design, improved decisions for operational, tactical, and strategic aspects of the development programs, and improved capability of organizations and institutions to advance development objectives. The analysis has shown that RBM is influenced directly by the M&E Information Quality dimension of M&E System, and will in turn influence improved policy and program design, improved tactical and strategic decisions, and improved capability to advance development. EBDM is influenced by M&E System Quality of M&E System. In turn, it influences directly improved operational, tactical and strategic decisions. Knowledge and Information Management is influenced both by M&E System Quality and M&E Service Quality of M&E System. In turn, it influences improved capability to advance development. 184

193 CHAPTER 10. KEY FINDINGS OF THE RESEARCH The main objective of this research was to contribute to the building of knowledge base on the M&E System effectiveness in the field of development management, through the clarification of a rigorous framework to measure the effectiveness of M&E System. M&E System is defined here as a strategy to improve decisionmaking in managing development program for greater capability to advance development objective, through data collection and analysis on development targets and results, the creation of meaningful information and knowledge actionable by development managers in the process of evidence-based decision-making and measured risk-taking. The M&E System was analyzed not only as a technique or a top-down tool to serve the needs of sponsored results frameworks. instead, M&E was analyzed as a robust management strategy that influences the dynamic capabilities of organizations and institutions in managing development actions toward the desired outcomes of improved welfare and freedom of people. The main question of this research was: How to measure the effectiveness of development programs Monitoring and Evaluation Systems for improved evidence-based decision-making and measured risk-taking to advance development - The context of West Africa? In addressing the main objective, the approach used different sub-questions and methods to respond to the research question CONCLUSION ON ANALYSES IN THE THREE ITERATIONS The iteration three used a sample survey of development programs in the West Africa region to analyze linkages between the different components of the proposed M&E System Effectiveness Framework detailed in the Iteration 2. The total sampled development programs is forty-six (six) among which there are national and regional development programs in 10 countries among the 15 ECOWAS countries; there are development programs in all the priority development sectors in the sample, including agriculture, infrastructure, energy, environment, democracy and governance, poverty reduction and social protection, education, health, etc.; the targeted development programs have at least reach mid-course or are closed less than three years ago; M&E Managers, Program Managers, Technical and Financial Partners, other program components managers, responded to the questionnaire built using the Likert Scales around tailored benchmarks from iteration 2 results. The data was collected and cleaned through Sphinx Declic as well as the analysis; only a few analyses required the importation of the database to SPSS. The analysis comprises multiple correlation analyses to gauge the linkages among the key proposed measurements of the three dimensions of M&E System, modeled from the DML IS success model, namely M&E Information Quality, M&E System Quality, and M&E Service Quality. The approach also used multiple regression analyses to assess the linkages between 185

194 the three dimensions of M&E System with the Three dynamic capabilities in managing development programs, namely Results-Based Management (RBM), Knowledge and Information Management (KIM), and Evidence-Based Decision-Making (EBDM). In the same vein, multiple regression analyses were used to test the linkages between the Development Management capabilities and the five net benefits of Effective M&E System, namely Improved Policy and Program Design, Improved Operational Decisions, Improved Tactical Decisions, Improved Strategic Decisions, and Improved Capability to Advance Development Objectives. The analyses led to the revised M&E System Effectiveness Framework below, showing the tangible quantified relationships between the dimensions of M&E System, the Development Management capabilities and the M&E System net benefits. The major outcome is the revised quantified model below. The quantities represent the Standardized Coefficients from the multiple regression analyses. Figure 23. The M&E System Effectiveness Framework with Quantified relationships DESIGNING THE PROPOSED M&E SYSTEM EFFECTIVENESS FRAMEWORK TO TEST The first sub-question was: How to build a commensurate framework in studying the effectiveness of development program M&E-System as part of Development Management strategy? We recalled the history of M&E System to find the gaps on why M&E System does not perform at the level managers need 186

195 information and knowledge to make evidence-based decisions to advance development objectives. We noted that M&E System belongs to the old history of managing development and discovered that, while not naming it specifically, Charles Booth, a statistician from London who worked on analyzing the impact of the support service program for the poor people living in London in the 19 th century, stressed the importance of data collection and analysis and creation of meaningful information on characteristics of poverty to effectively deal with poverty. We concluded that M&E is not originated after WWII development projects, but is an old development management practice. In analyzing the history of M&E System after WWII, we noted the different phases that relate to the stage and concerns of development programs in specific periods. The Gant-Network and Charts stage dominated the field of M&E in the 60-70s as development programs that period focused mainly on infrastructure building in developing countries where operational matters including budget, timelines, and delivery were the focus. The Clinical Trial impact assessment approach dominated the 80-90s in the M&E field after the advent of the Berg report, the Washington Consensus, the structural adjustments in developing countries mainly in Africa Sub-Sahara. Assessing development results and focusing on attributable changes were the main objectives of M&E Systems. The Web-Based Platforms stage dominated the M&E System in the 2000s after the advent of the MDGs and harmonized targets and benchmarks to track development achievements and results. In analyzing the literature of M&E System and Information System, we noted that in the 90s, Tom Barton qualified the M&E System as an Information System and described key factors that need to be considered in designing M&E Systems. We then looked at the Information System literature, what would be the best IS model that could be used as a basic model for M&E System Framework. We find that the revised DML IS success model (2003) would be an excellent basis for the M&E System effectiveness framework. We also find the model designed mainly for IT IS should not be applied automatically for the M&E System effectiveness framework and needs to be scrutinized through tangible case studies. We then utilize the literature review of M&E System the last five decades to note the main components of M&E that would be relevant to build the proposed framework. We find that three aspects, described in the DLM IS model as a cycle, should be considered in the M&E System Effectiveness framework. First, the three sub-components of M&E System, namely M&E Information Quality, M&E System Quality, and M&E Service Quality. Second, the M&E Effectiveness sub-dimensions, which are the dynamic capabilities that institutions and organization in charge of development actions need to ensure that development management is leading to improved welfare and freedom of people, namely Results-Based Management, Knowledge and Information Management, and Evidence-Based Decision-Making. Third, the M&E System net benefits that reflect the outcomes to which an effective M&E would lead organizations 187

196 ability to manage development actions and reach development goals, improved policy, and program design, improved operational, tactical and strategic decision, and improved capability to advance development DRAWING KEY PROPOSED MEASUREMENTS FOR THE M&E SYSTEM EFFECTIVENESS FRAMEWORK The second sub-question of the research is: What are the key components (and variables) at different levels of the Monitoring and Evaluation System effectiveness model relevant in studying M&E- System role to the effective evidence-based decision-making in advancing development in West Africa? We analyzed different cases in the West Africa Region, relevant to the proposed dimensions of M&E System; we noted effective development management capabilities and M&E System net benefits selected in the proposed framework. We then draw various findings in responding to the second subquestion that deals with the operationalization of the different measurements to be considered. To analyze the key measurements to be considered under the M&E System Quality dimension, we studied the case of the Office du Niger Contract Plan We find that the key subdimensions to consider under M&E System Quality are Design Quality, Setup Quality, Operations Quality, Maintenance Quality, and Resources Quality, that could be assessed through: Design Quality can be measured through, first, the existence of program high quality M&E plan prior to implement activities, which means that prior to the implementation of any of the program activities, a robust, clear and well-crafted M&E Manual/Plan is designed in a participatory way and includes the definition of the program theory of change, the objectives of the M&E system, the description and definition of the scope of all monitoring and evaluations activities over the program lifetime, the roles and responsibility of different stakeholders, the M&E work plan and budget, the deliverables and timelines; second, the quality (and simplicity) of tools and techniques for data collection and analysis, that the M&E data gathering and analysis instruments and techniques are readily available to and applicable by all actors involved in the M&E activities without any risk of professional or personal biases over the program lifetime; third, the existence of a capacity building plan commensurate with the M&E Plan/Procedures objectives, meaning that a high-quality capacity building plan which is realistic and feasible technically and financially exists that defines capacity building activities including internal seminars, training in formal or short courses, exchange visits, etc. commensurate with the needs of different actors and objectives of the M&E System. Set-up Quality can be measured through, first, the timely availability of resources to implement M&E activities, meaning that all resources including technical, financial and logistics are available and mobilized to implement the program M&E activities as defined in the approved M&E plan and budget; 188

197 second, the Tests of M&E tools and techniques prior to M&E activities implementation, that all M&E tools were tested prior to their application to ensure their comprehensiveness, quality and relevance in order to avoid any pitfall during the M&E activities implementation; third, the existence of quality baseline information for future program assessment, meaning that all program indicators for which a baseline value is mandatory for future evaluations and comparisons, reliable and quality baseline information is collected and analyzed prior to starting carrying out the program activities as per the procedures and methods defined in the M&E plan. Operations Quality can be measured through, first, the existence of M&E data exchange platform accessible to core program stakeholders, that the M&E system has developed and deployed an easily accessible and relevant input-output M&E information system accessible to all core stakeholders and M&E actors for updates purposes and generation of commensurate reports periodically; second, the capacity of the M&E team to visit all program zone of intervention for data collection, feedback and analysis, that the M&E team has the capacity financially and logistically at any time of the program lifecycle to reach out all localities of the program zone of influence to implement M&E activities as defined and approved in the M&E plan and budget; third, the existence and Functionality of decentralized M&E sites to cover program zone of intervention, that the M&E system can relate to existing relevant decentralized M&E Units to reach out all projects zone of influence localities and activities, and those decentralized units should have the required skills and capacity to implement the program M&E activities locally and to collect and analyze M&E information for local feedback purpose and be able to establish technical relationship with the headquarter in term of knowledge and information sharing. Maintenance Quality can be measured through, first, the frequency of reviews/assessments of the M&E system design and operations, that at least one M&E system review/assessment is performed per year; second, the percentage of M&E agents benefitting from staff development program, that 100% of the M&E staff have benefitted from staff development program through training or exchange visits, etc.; third, the technical support received from specialized M&E agencies and other organizations, that the M&E system has received at least one technical support per year from technical partners, recognized M&E agencies or organizations and/or specialized government departments. Resources Quality can be assessed through, first, the qualification and level of competency of the program M&E managers, that all M&E Managers in the program have at least all required skills and qualification of M&E expert including strong knowledge of development policies and programs, strong skills in M&E, impact and performance evaluation, monitoring, reporting, statistical background, data collection and analysis, communication skills; second, the qualification and level of competency of other M&E human resources (enumerators, consultants, etc.), that all M&E agents, enumerators and 189

198 other consultants who are working or worked in the M&E system have fully the required qualification, skills and knowledge to carry out their duties as defined in their specific scope of work; third, the existence of appropriate financial resources as per the approved M&E plan to conduct M&E activities, that all the resources defined and adopted in the M&E Plan and budget are actionable easily by the M&E Managers to implement M&E activities without any delay. Analyzing the key measurement of M&E Information Quality, we studied the case of the Senegalese Rural Electrification Program. We find that the main sub-dimensions to be considered under M&E Information Quality are Inputs Information Quality, Outputs Information Quality, Outcomes Information Quality, Performance Information Quality, and Risk Management Information Quality. Following the data analysis and multiple correlations analyses, we find that: Inputs Information Quality can be measured through, first, the program analytical accounting system (linking clearly Budget/Expenditures to activities, outputs and other characteristics of deliveries), that any expenditure in the program should be linked to one single specific activity that also should be linked to one single result and its indicator to allow estimation of costs of the results achieved so far and allow merging technical implementation to financial expenditures; second, the approach for conversion of program beneficiaries contributions, that any type of contribution of the beneficiaries to the program implementation is appropriately converted into financial information and reported accordingly - beneficiary contribution could be financial or other types of efforts such as paying their own participation to program activities, providing manpower in infrastructure construction, or giving raw materials or food or goods or initial savings to access credits, etc.; third, the program inputs indicators meet data quality assessment standards, that all selected input indicators reported over periods of time meet data quality standards agreed upon and set forth by the stakeholders or the direction which is in charge of data quality standards in the given sector where the program is operating. Output Information Quality can be measured through, first, the program outputs indicators coverage of the program results, that all the program results are captured by a commensurate set of output indicators without any possibility of overlapping in measurement or any possibility of missing, and the set of output indicators for each result should give a clear measurement system of the achievements and should be easy to use for performance assessment; second, the program output indicators meet data quality assessment standard, that all selected output indicators reported over periods of time meet data quality standards agreed upon and set forth by the stakeholders or the direction which is in charge of data quality standards in the given sector where the program is operating; third, the program output indicators alignment with the sector results for consolidation with national/sector statistics, that all program output indicators are aligned with the sector ones and are easy to consolidate with the sector 190

199 statistics allowing the program to show its contribution to the achievements in the national or regional results framework. Outcome Information Quality can be measured through, first, the percentage of program outcomes indicators that are/were informed timely, meaning that 100% of the outcomes indicators selected for reports are up to date at any time of the program life cycle; second, the quality of outcomes indicators coverage of program intermediate results, that all the program intermediate results are captured by a commensurate set of outcome indicators without any possibility of overlapping in measurement and any possibility of missing, the set of outcome indicators for each intermediate result should give a clear measurement system of the outcome or intermediate result and should be easy to use combined with a lower level corresponding outputs indicators for performance/effectiveness analyses; third, the quality of information generated by outcomes indicators on intermediate changes at beneficiaries level, that the information generated through the outcome indicators is highly robust and reliable for use and generalization in the program zone of influence, targeted communities, groups and direct and indirect beneficiary with minimum statistical limitation in the approach of data collection and analysis. Performance Information Quality can be measured through, first, that quality information is available for program efficiency analysis purpose, that high quality financial and technical data are available for robust analyses of the program efficiency at all level, the costs of all program activities and outputs should be available integrating human, technical and financial resources used as well as up to date outputs indicators dashboards with possibility of comparison of planned vs. actual to establish the program implementation performance indexes; second, that quality information is available for meaningful program sustainability analysis purpose, that high-quality information and data are available on the program impacts and outcomes indicators dashboards and other survey reports to allow sustainability analysis through the building of causal relationships between impacts and outcomes as per the program theory of change and results framework and establishment of the program operational, technical, social and financial sustainability; third, the capability of the M&E system to translate best practices and lessons learned into meaningful information, that the M&E system has set clear strategy and methods to capture and capitalize the program best practices and lessons learned from the implementation and to create relevant tools and occasions to share those findings allowing program improvement and knowledge sharing with various actors interested in the program. Risk Management Information Quality can be measured through the fact that all the assumptions and risks in the program/project logical framework are/were translated into meaningful risks mitigation/attenuation milestones monitored and informed on a regular basis, meaning that all risks and assumptions in the program results/logical framework have been adequately assessed in terms of 191

200 probability of occurrence and impact on the program and that mitigation/attenuation actions are defined, monitored and reported as appropriate over the program implementation. With a multiple regression analysis, we find that the way the M&E System is designed, functions and operates explains strongly the quality of the information generated. If the M&E System is not well designed, set-up, well operating, well maintained and appropriate resources mobilized, there is little chance for the M&E Information to be at high quality. To analyze key measurements for the M&E Service Quality dimension of the M&E System, we studied the FAO Water and Food Security Initiative in Africa, which covered five countries and involved several stakeholders in various organizations. We find that M&E Service Quality has the following key subdimensions: Information Availability, Information Accessibility, System Responsiveness, and System Sustainability. From the results of multiple correlation analysis, we find that: Information Availability can be assessed through, first, the capacity for timely generation of program M&E mandatory reports, that all (100%) of the program mandatory reports were high quality and produced within the agreed timeframes in the program documents and M&E Plan and made timely available to decision-makers and other stakeholders; second, the existence of a knowledge sharing system on M&E information, that the program Managers organize at least twice a year learning events on the information generated by the M&E system, where discussions and brainstorming are made on their use and dissemination for a better sharing of the program results and achievements; third, the existence of communication tools for M&E data, that beyond the program mandatory M&E reports and evaluations and other studies reports, the M&E team produce more frequently snapshots, briefs, memorandums and other communication tools on the program M&E information for a better sharing of program results and achievements. Information Accessibility can be measured through, first, the percentage of program managers having access timely to evidence-based data on program achievements and results, that at any time of the program lifecycle, all program managers (100%) had easy access to evidence information on the program results and achievement produced through the M&E system without any difficulty of interpretation of the data and findings; second, the access to program M&E data for the large public, media, and researchers, the M&E managers ensure that all (100%) partners, relevant public, researchers and media who are interested in the program M&E information have easy access to relevant data and information for their specific purposes for a better sharing, dissemination, and use of the evidence generated through the program M&E system; third, the technical user friendliness of M&E data for all users, all (100%) of M&E information users have no technical problem in their interpretation or use of the data and findings and recognize the M&E information is user-friendly. 192

201 System Responsiveness can be measured through, first, the timely availability of M&E data upon requests by the program stakeholders, all M&E data or information requested by any of the program stakeholders is made available within short notice and timely by the M&E managers or are easily accessible directly through the M&E data portal; second, the capacity to carry out timely specific M&E studies requested by the program managers, the M&E team is at any time highly responsive and reactive when it receives new requests of specific studies or assessments expressed by the program managers and other decision makers within a revised scope and budget approved; third, the capacity of the M&E system to respond to the sectorial Ministry information request on the program achievements and results, the M&E team is collaborating very well with their sectorial Ministry counterparts and is highly responsive in providing Government with the M&E data, information or other relevant stories on the program in order to make sure that the Government is aware and has appropriated the program achievements and results. System flexibility can be measured through, first, the capability of the M&E system to integrate efficiently new national directives and/or international standards on results-based management, monitoring and evaluation, the M&E system is highly flexible and adaptive in integrating new directives or orientations from the Government or International recognized organizations in terms of guidelines, procedures or indicators to adapt with limited impact on the M&E plan and budget; second, the ability of the M&E system to integrate timely new information needs following programs revisions and strategic adjustments, the M&E system is highly able to adjust and integrate new information needs following program tactical or strategic adjustment with limited effect on resources and timeframes. System Sustainability can be measured through, first, the capacity of the M&E system to remain fully functional when major changes occur in the M&E team, even if unintended changes occur in the M&E team composition, the M&E system is highly resilient and remains fully functional after those changes; second, the capacity of the program managers and other stakeholders to access M&E information with minimum support from the M&E team, all the program managers (100%) have easy access to the M&E data with very limited support from the M&E team and are able to understand, interpret and use the M&E information with very limited technical support from the M&E team; third, the utility of the M&E data after program closure for capitalization purposes, the M&E data and information are recognized very important and useful even beyond the program schedule and lifetime for post evaluations and capitalization purposes or as baseline for future programming. We find that M&E Information Quality and M&E System Quality influence the M&E Service Quality. The three dimensions of M&E System have strong relationships and for the M&E System to service the stakeholders and decision-makers with relevant information and evidence in the process of managing 193

202 development actions adequately, there must be a high-quality M&E information and high-quality M&E setup, operations and resource mobilization as well as proper maintenance and capacity building of M&E agents. We looked at the needed capabilities in managing development actions. First, the Results-Based Management approach. We then note that RBM is closely linked to evidence-based management as it ensures that the development program is well planned against realistic, relevant and tangible development targets designed in a well-crafted theory of change, clear results frameworks, that will ground an effective management and decision-making processes in piloting and adjusting the required activities towards the desired outcomes. We find that RBM could be measured through a set of pillars proposed in the model designed by Barends et al (Barends et al., 2014). These measurements are tracking the effectiveness of RBM through evidence-based management by assessing whether development managers in the organizations and institutions are: Asking, translating a practical issue or problem into a measurable question; Acquiring, systematically searching for and retrieving the evidence; Appraising, critically judging the trustworthiness and relevance of the evidence; Aggregating, weighing and pulling together the evidence; Applying, incorporating the evidence in the decision-making process; Assessing, evaluating the outcome of the decision taken. In analyzing the key measurements for Knowledge and Information Management, we note that in their role of managing development actions towards the greater sustainable welfare of people, the capabilities of organizations in charge of development programs to create and use information and knowledge for improved decision-making and measured risk-taking, are key functions in development performance management. We note that the key measurements proposed by Anantatmula & Kanungo (Anantatmula & Kanungo, 2006) for effective knowledge management, are good assessment dimensions to be considered in the proposed framework: Sharing best practices, Improved productivity, Enhanced quality, Improved employee skills, Improved communication, Enhanced collaboration. For the Evidence-Based Decision-Making capability, we analyzed the existing knowledge, and note that M&E Systems are decision support systems for development managers in various organizations, through the collection of data, analyzes, and provision of meaningful information that can generate relevant knowledge for actions to advance development objectives. Scholars have shown the positive correlation between quality evidence and high-quality decisions; the same finding is valid in the field of development management. To avoid the pilotage-à-vue, which means implementing development programs without measured steps, there is need to gather and analyze relevant data and produce critical information and useful knowledge that can support measured risks and meaningful evidence-based decisions to advance action plans to success. The measurements of effective Evidence-Based Decision-Making proposed by Shilling et 194

203 al. (Schilling et al., 2007) were used to set the key pillars of EBDM under the M&E System Effectiveness framework: general participation, top-down versus bottom-down, the quantity of stakeholder information, transparency and comprehensibility, ration-based versus intuitive-based, creativity, and strategic insights. In analyzing the main objective of M&E System, we note that M&E System might influence the dynamic capabilities of organizations and institutions in managing development actions, the RBM, KIM, and EBDM. The hypothesis was that organizations with enhanced capabilities to manage development actions would record net benefits. The net benefits we analyzed are ranked at three levels as outcomes of development management with effective M&E System, and we noted that these ranks are: the process of planning and the inception of development action in policy making and program design, the management of ongoing development actions at operational tactical and strategic levels, and the capability to advance development actions to their objectives of greater welfare and freedom of people. We then find that the net benefits of improved development management capabilities through effective M&E System are: improved policy and program design, improved operational decisions, improved tactical decisions, improved strategic decisions, and improved capability to advance development. In responding finally to the first sub-question (How to build a commensurate framework in studying the effectiveness of development program M&E-System as part of Development Management strategy?), we proposed the M&E System Effectiveness Framework, version zero (Figure 15. M&E System Effectiveness Framework, version zero (to be tested)) that we tested through responses to the following sub-question THE MEANINGFUL ACCEPTABLE FRAMEWORK FOR MONITORING AND EVALUATION SYSTEM EFFECTIVENESS The third sub-question of the research is: What is the meaningful scientific acceptable framework to measure the effectiveness of Monitoring and Evaluation System, through the operationalization and testing of the first proposed framework? We used a quantitative approach in analyzing the data from forty-six sampled development programs in West Africa, gathered through SPHINX with a structured questionnaire. We analyzed the relationships between the dimensions of M&E System (M&E Information Quality, M&E System Quality, M&E Service Quality), the M&E System effectiveness influences key dynamic capabilities needed in effective development management (Results-Based Management, Knowledge and Information Management, and Evidence-Based Decision-Making). 195

204 Figure 24. M&E System Effectiveness Framework We find the significant linkages between M&E System Quality and the two other dimensions of M&E, Information Quality and Service Quality. Well-designed M&E System through the elaboration of a clear manual of procedures that includes a clear definition of roles and responsibilities of the different actors, realistic M&E work plan and budgets, the involvement of decentralized M&E units in the development program areas in the M&E System, the availability of actionable technical and financial resources, and the way the M&E System is maintained including capacity building of core actors involved in its implementation, is key element to the success in generating meaningful robust information, and being well positioned in servicing the decision-makers with relevant evidence in the process of development actions preparation, implementation and capitalization. Together, M&E System Quality and M&E Information Quality influence positively M&E Service Quality that includes information availability, information accessibility, system flexibility and responsiveness, and system sustainability and resilience. Results-Based Management practice of organizations and institutions in charge of managing development actions, as a key needed capability to ensure that development activities are leading to the desired outcomes, is a key dynamic capability which effectiveness is influenced directly by M&E Information Quality, and indirectly, through M&E Information Quality by M&E System Quality. Quality information on inputs (inputs indicators to meet data quality standards, good approaches to convert and integrate beneficiaries contribution, existence of an analytical codification in the financial system linking results to funding use), quality outputs information (program outputs indicators cover fully all program results, meet data quality standards, are aligned with the sector results and statistics), quality outcomes information (quality of outcomes indicator coverage of program intermediary results, timely update of outcome indicators, quality 196

205 of information on changes at beneficiaries level), quality performance information (availability of quality data for program efficiency analysis, program sustainability analysis, capability to translate best practices and lessons learned into meaningful information), and quality risk-management information (all assumptions and risks in the program logical/results framework are translated into meaningful mitigation/attenuation milestones monitors in regular basis), are necessary to ensure effective Results-Based Management practice at level of organizations and institutions in charge of development actions. All these influential sub-dimensions of Quality Information also need Quality System to effectively influence the RBM practice (Figure 19). The pathway to make RBM effective is to ensure first Quality M&E System is set and operates well, then second, Quality M&E Information is available. We find that effective Knowledge and Information Management, including learning, as a key needed capability for organizations and institutions in managing development actions, is influenced directly by M&E System Quality and M&E Service Quality, and by M&E Information Quality indirectly through M&E Service Quality. The existence of quality M&E information is necessary but not sufficient to ensure effective KIM. Effective KIM needs quality and strong capability of the M&E to service quality information through the needed formats and methods. Hence, M&E System responsiveness, information availability and accessibility, M&E System flexibility and resilience and well as sustainability, are key sub-dimensions of M&E System Service Quality that are necessary to ensure effective Knowledge and Information Management, and learning process. The way M&E System is designed, set, operates, is maintained, its resources mobilized and used, have significant influence in the Knowledge and Information Management effectiveness. It is clear that the availability of high-quality M&E Information does not directly influence KIM, there is need to translate the information into right formats, reports, and snapshots in various forms through the M&E Service Quality to ensure information availability be useful for effective KIM. Also, the high-quality M&E System (design, operations, resources, and maintenance) is necessary to ensure the success of this pathway towards effective KIM. We find that the third capability in managing development actions which effectiveness is necessary for organizations and institutions to lead programs and policy to success, Evidence-Based Decision-Making, is influenced directly by the M&E Service Quality and indirectly by the M&E Information Quality and M&E System Quality through effective M&E Service Quality. Decision-makers need an effective M&E System service high-quality information to ensure that decisions taken are evidence-based. Information Quality and Service Quality influence indirectly EBDM through Service Quality. Ultimately, organizations need to ensure a strong Service Quality of the M&E System including information availability, accessibility, system responsiveness, flexibility, and sustainability, to set the needed basis for effective evidence-based decision making for measured risk-taking to advance development actions to success. 197

206 The analysis of the results in various multiple regressions has shown the pathway for organizations in charge of development activities to ensure Improved Policy and Program Design for future planning. Organizations in charge of development actions are in a continuous process of planning new policy or programs to adapt their development strategies to a dynamic international and local context. Many of development actions are five-year lifetime or less projects or programs. Hence, the cycle of policy-making determines the projects periodicity. In their M&E System, organizations learn from ongoing and past development actions to design new policy and programs. Therefore, there is a chance that effective M&E System contributes to the improved policy and program design. We find that the pathway for improved policy and program design starts with quality M&E System and quality M&E Information. The design, operations, resources and maintenance of the M&E System, when well performed by organizations, influence positively the quality of M&E information (quality inputs, outputs, outcomes, performance and risks management information), which, in turn, influences the effective practice of Results-Based Management in the organizations, which will contribute significantly to improved policy and program design for future planning. If the M&E System is well designed, operates without major challenges, mobilizes its resources to perform M&E activities without difficulty, and is well maintained, then, information collected and analyzed on the development program achievements and results would be of high-quality, and there will be great chance to ensure effective RBM in the organization, which is the basis to ensure improved policy and program design for future planning. Effective Knowledge and Information management, built on M&E System Quality and M&E Service Quality, reinforces the RBM practice and contributes then to Improved Policy and Program Design. Figure 25. Pathway for Improved Policy and Program Design in the M&E System Effectiveness Framework 198

207 The pathway for Improved Operational Decision-Making takes a different perspective to build on effective Evidence-Based Decision-Making. M&E Information Quality and M&E Service Quality influence positively M&E Service Quality, which in turn influences EBDM. Development organizations when implementing development actions take operational decisions that focus mainly on input management and outputs delivery. Operational decisions touch the operations of projects and programs, that ground the actions implemented towards the development goals. Hence, operational decisions are critical to ensuring development programs success. We find that operational decisions are influenced directly by the EBDM practice within the organization. The pathway for improved operational decisions starts with Service Quality, which is influenced positively by Information Quality and System Quality. Service Quality influences directly the EBDM practice which in turn generates directly improved operational decisions. The pathway is completed by the positive influences on EBDM from effective KIM. System Quality and Service Quality facilitate effective KIM through improved learning processes, which contributes greatly to effective EBDM. All these dimensions of effective M&E System should function effectively to ensure improved operational decisions. Figure 26. Pathway for Improved Operational Decision-Making in the M&E System Effectiveness Framework While improved operational decisions need only effective EBDM pathway, Improved Tactical and Strategic Decisions necessitate both effective EBDM and effective RBM pathways. Tactical decisions are decisions that will affect the programs intermediary results, and strategic decisions affect the higher levels of programs, specific objectives, objectives and goals. Hence, EBDM is necessary but not sufficient. There is need to ensure effective RBM practice together with effective EBDM. Effective KIM contributes to RBM and EBDM to generate improved tactical and strategic decisions. 199

208 Figure 27. Pathway for Improved Tactical and Strategic Decision-Making in the M&E System Effectiveness Framework Improved Capability to Advance Development of organizations and institutions in charge of development actions is the highest objective of effective M&E System. Effective M&E System has a long term positive impact on the capability of organizations and institutions to advance development objectives. We find that two dynamic capabilities are necessary to ensure the improved capability to advance development objectives, Results-Based Management and Knowledge and Information Management. This is the only effective M&E System net benefit influenced directly by KIM practice. Learning processes is a key success factor for an organization to be able to move development actions to success. Figure 28. Pathway for Improved Capability to Advance Development in the M&E System Effectiveness Framework 200