A monitoring and evaluation framework for the Local Development and Solidary Economy Promotion Project

Transcription

1 A monitoring and evaluation framework for the Local Development and Solidary Economy Promotion Project José Silvério Lage Martins Filho 1 Virgínia Maria Dalfior Fava 1 Trajano Augustus Tavares Quinhões 1 This paper discusses a new monitoring and evaluation framework for the Local Development and Solidary Economy Promotion Project (Projeto de Promoção do Desenvolvimento Local e Economia Solidária PPDLES), which is implemented through the Ministry of Labor and Employment (MTE) in association with the Ministry of Environment (MMA) and the Ministry of Social Development and Fight Against Hunger (MDS). The project is based on the work of solidarity based economy agents (ADS) who foster solidary economic enterprises, and is oriented by the principles of sustainable development, social participation, environmental awareness and economic growth. The ADS action focuses on the Bolsa Família Program beneficiaries, which receive financial transfers and are included in specific public policies. In this sense, this paper proposes a method to monitor the ADS work, in order to evaluate the PPDLES effectiveness and the participation of PBF beneficiary families. This proposal is based on the construction of a database with information collected by the ADS, through enterprises identification and activity reports, which will be analyzed by the three Ministries involved. The monitoring strategy is anchored in three principles: economic, social and environmental sustainability, which will develop three different indexes. These indexes will support the construction of a broader index of sustainable development that will contribute for the project management, planning of field trips, and to identify key factors of success and/or difficulties. The four indexes will also contribute to direct decisions on certain policies in order to improve the work of the ADS and will support the evaluation of the PPDLES effectiveness. This framework is innovative as a result of its subdivision into three indexes, which will be able to monitor compelling public policy aspects: economic, social and environmental sustainability. In addition, it is a flexible tool which can be regionally applied and for other kinds of social projects. Keywords: Monitoring and Evaluation, Local Development, Solidarity Based Economy, Intergovernmental policies and Bolsa Família Program. 1. Introduction A monitoring and evaluation system is necessary to get credible answers to questions that address the accountability concerns of stakeholders, give information on the progress toward expected targets and goals, and provide substantial evidence as the basis for any mid-course corrections. It is an important mechanism of feedback that is essentially a managerial tool that governments can use to measure and evaluate outcomes and outputs, feeding this information back into the ongoing processes of governing and decision making (Kusek & Rist, 2004). Hass and Springer (1998, apud Silva, 2002) cite some reasons to justify public policies monitoring and evaluation, as: to decrease the uncertainty in governmental decisions; to increase the rationale and the consistency of decisions in regards to policies and programs; to create understanding and new perspectives about public problems and to get feedback from the people, which are responsible for the implementation of the results; and to strengthen public debate, putting the evidence into the decision process tree. 1 Ministério do Desenvolvimento Social e Combate à Fome Correspondence corcening this article should be sent to jose.lage@mds.gov.br. 1

2 Therefore, this system allows for the control of public actions efficiency, focusing public investments on certain groups in which the benefits will have greater impact or on groups of higher economic and social vulnerability. It can contribute to drive actions, to avoid mistakes, to improve the program design and to disclose successful experiences. The Bolsa Família Program PBF (Family Grant) is a conditional cash transfer program, which is the most important social policy implemented in Brazil to combat poverty and to assure protection to families in economic and social vulnerability. The PBF has three dimensions of action, as follows: I. the immediate relief of poverty, through direct cash transfers for the poorest families; II. the interruption of the poverty intergenerational cycle, through the fulfillment of education and health commitments by the families; and III. the increase of families autonomy, through complementary programs and actions. The third dimension of action is based on the fact that poverty is a complex and multidimensional phenomenon and it aims to promote families sustainability through an economic, social and environmental emancipation process. The complementary programs are organized and regular actions directed to PBF beneficiary families, through partnerships between the Ministry of Social Development and Fight Against Hunger (MDS) and other departments of the central government, other levels of government and NGOs (non-governmental organizations). They constitute a poverty confrontation strategy, which requires coordination of governmental actions, creation of intergovernmental partnerships and focalization on local problems. These actions are supposed to have an impact on more than the immediate relief of poverty, bringing permanent social inclusion. There are programs and actions in four areas: capacity building, job and income, housing conditions and citizenship. The Local Development and Solidary Economy Promotion Project (Projeto de Promoção do Desenvolvimento Local e Economia Solidária PPDLES) is one of the PBF complementary programs and is implemented through the Ministry of Labor and Employment (MTE) in association with the Ministry of Environment (MMA) and the MDS. The PPDLES action focuses on PBF beneficiaries. PPDLES was initially planned to be executed by the MTE. It focused solely on the economic aspect of local development. After the creation of the Single Registry for Social Programs (CadUnico) by the Federal Government, in 2001, a new intergovernmental strategy was established, in order to increase intergovernmental coordination and the synergies between the different social policies, focusing the governmental action on the most vulnerable Brazilian families, through the CadUnico and its database. This strategy stimulated partnerships between different government agencies, using the CadUnico as a planning and conception tool for social policies. A Working Group (GT) was created as a result of a partnership between MTE, MDS and MMA, in order to implement PPDLES and to extend its objectives, including social and environmental aspects. PPDLES is based on the concepts of Solidarity Based Economy and of Sustainable Development, and aims to include poor and vulnerable families into the labor market. Their targets are to promote access to job and income and to stimulate solidarity and sustainable development for socially and economically excluded communities. 2

3 The Solidarity Based Economy is a strategy to promote citizenship, job and income to socially marginalized people. Technological and economic development can promote human and social development, through the creation of solidarity enterprises. The economic, political and cultural results are shared with all participants. In this sense, Solidarity Based Economy is a strategy for a new Sustainable Development framework (I Conferência Nacional de Economia Solidária, 2006). The idea of Sustainable Development was initially founded on environment preservation and ecological concern. However, this is insufficient to promote sustainability. It is necessary to supply modern biotechnology to teach people and offer additional policies of knowledge, credit and market access. These needs are the three pillars of Sustainable Development: economic viability, criterions of social relevance and environmental prudence (Sachs, 2000). PPDLES actions are implemented by solidarity based economy agents (Agentes de Desenvolvimento Solidário ADS), whose most important attributions are: to identify local needs (as infra-structure) to support the local production of goods and services; to orient governmental actions in their territories; to incentive the reinsertion of workers of bankrupted companies into the labor market; to support solidarity enterprises in the benefited communities; to stimulate new sustainable, collective, cooperatives and associated enterprises; to incentive sustainable enterprises networks; to encourage the population to participate in the projects events; to monitor the solidarity enterprises; and to stimulate the articulation of PPDLES with other related policies. Nearly 500 agents were trained in the last 15 months and hired by the executing organism, Fundação Universidade de Brasília (Fubra), convened with MTE. They work at poor neighborhoods in more than 800 cities in all Brazilian s regions. PPDLES is a pilot project. It has been implemented for less than fifteen months and there are some innovations in its design and its monitoring and evaluation. It is expected to be expanded in the next years, according to its results. The monitoring and evaluation of the effectiveness of PPDLES implies in the monitoring and evaluation of ADS actions. The main ADS actions and their results are supposed to be monitored through the approach described in the second section. The third section presents the development of the monitoring and evaluation framework. At last, the final section presents the lessons learned with this system to PPDLES, to the complementary programs and to PBF. 2. The Monitoring and Evaluation Strategy This section presents the characteristics of the strategy that will be used to monitor and evaluate the PPDLES. The concepts of monitoring and evaluation are distinct, but complementary. According to OECD (2002, apud Kusek & Rist, 2004): Monitoring is a continuous function that uses the systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress in the use of allocated funds. 3

4 Evaluation is the systematic and objective assessment of an ongoing or completed project, program or policy, including its design, implementation and results. It should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision making process of both recipients and donors. Indicators produced by the monitoring allow an evaluation process, which results in directions of interventions to improve the project, program or policy. It is not an attribution of the monitoring system to judge the outcomes of the monitored program. This is rather a competency of the evaluation system (Silva, 2005). Therefore, evaluation is a complement of monitoring, in that when a monitoring system demonstrates that the efforts are going out of track, then good evaluative information can help direct interventions and can clarify realities and trends noted within the monitoring system (Kusek & Rist, 2004). The proposed evaluation strategy combines quantitative and qualitative methodologies, in order to assess the program in different dimensions. On the quantitative evaluation side, an interpretation of the indicators produced at the monitoring strategy is made. This interpretation gives information about the project, program or policy course, related to its objectives. As a consequence, it is possible to identify whether the objectives were reached and how far they are from the parameters initially defined. This quantitative evaluation directs the search for detailed information about why some indicators are distinguished from the others, positive and negative ones. The explanation of these results is obtained by a qualitative evaluation. In this paper, the focus will be on the monitoring and evaluation of PPDLES results, ex-post, through a rationalist approach, which implies the definition of satisfactory indicators to verify the achievement of expected goals, results and impacts. They allow the analysis and supervision of policies and social changes. These indicators are based on well-known parameters and defined by people in charge of the monitoring (Dagnino, 2002). These parameters are the action s previously stated objectives that should be reached by ADS and that are described on their strategic lines of direction. These ideas, however, can be generalized for other programs and policies, considering their specificities. This strategy has a pragmatic perspective in order to solve problems and to improve the program. The advantages of this evaluation strategy are its useful product results and faster information for the managers (Vaitsman, Rodrigues & Paes-Sousa, 2006). The results obtained by this monitoring and evaluation strategy reflect changes in the local reality, influenced by the action (or the set of actions) of a social actor, in cooperation with other partners (Garcia, 2001). In the specific case of PPDLES, these social actors are the ADS, and the results mean their performances. However, the appropriate indicators of this monitoring system search for correlations between the changes and the ADS work. It s difficult to identify cause-effect relationships because of the public policy complexity (Garcia, 2001) and the multidimensional poverty characteristics. 3. The Monitoring and Evaluation Framework Applied to PPDLES The partnership between MTE, MDS and MMA to implement PPDLES implied the necessity to monitor and evaluate the program results in all three stakeholders aspects of interest, based on the adopted strategy. 4

5 This framework is subdivided into four steps: 1) dataflow mapping; 2) development of tools; 3) construction of indicators; and 4) quantitative and qualitative evaluations. The dataflow mapping and the development of tools are important to get precise and organized information about ADS action. The construction of indicators and the quantitative and qualitative evaluations are necessary to use this information in order to compare the ADS work results with established parameters to the project and to drive governmental interventions. 3.1 Dataflow mapping In order to discuss the dataflow mapping among the agents of the program, it is first necessary to understand the value of the information, which can be represented when that information is essential for the development of a process or when it changes the direction of a decision (Bernardes, 1996 apud Nascimento, 1999). For the present proposal, information is fundamental to structure a monitoring and evaluation process and to direct the governmental action towards a better management of PPDLES and of the instrumental for the ADS s action, whose difficulties, options and unsolved situations we intend to identify. Mapping the flux is a critical phase, because from that on we can develop tools for monitoring and evaluation process and, finally, for decision making process. Mapping is the basis for searching and structuring the data that are part of the strategy indexes. Based on those, quantitative and qualitative analyses are made, which will guide the administrative decisions and the implementation to PPDLES. There were three reports that were fulfilled by the agents and sent to their respective state coordinators. The coordinators validated the information of the reports and sent them to the Fundação Universidade de Brasília (FUB), which is the organ responsible for the execution of PPDLES. Those reports were analyzed and condensed by FUB. The Working Group (GT) created two new tools (questionnaires) to be submitted to the ADS and decided to maintain the previous data flux for the present monitoring strategy. However, FUB will systematize the data and sent them to the Ministries. GT is the responsible for data analysis, for ADS monitoring and evaluation and for the effectiveness of PPDLES. The flow of these information is established in the administrative structure of the project (Figure 1). Figure 1: PPDLES monitoring dataflow mapping 5

6 3.2 Tools development The lack of a systematization of reports and of data collection made difficult the data manipulation and the identification of ADS work results. GT made an option, then, for the creation of two new tools that allow the systematic, organized and rapid data collection of ADS: the poll of Agents and Entrepreneurships. The poll of Agents searches for the actions performed by each ADS, aiming at articulating networks of productive chain, involving local organisms and institutes in joint actions, participating in forums or local councils of themes related to local and sustainable development, among others. The poll of Entrepreneurships will be applied to each one of the entrepreneurships monitored by the ADS in order to search for information related to the constitution of those with their economic, social, political and environmental aspects. Those tools have direct questions that are followed by orientations to be answered adequately. The polls will be applied to each agent periodically, every six months. That process implies the standardization of the data search and the data organization in a structured basis that uses a standard scheme to store data and a unique logic for data analysis. Thus, it is possible to briefly visualize ADS work and its group, in accordance with the desired focus, allowing a regional analysis, or by the type of activity performed by ADS. 3.3 Indexes construction The answers of the polls will be used to feed the system database on results indicators. GT has defined with great precision the main attributions of ADS and has congregated them into three groups of indicators, according to the impact of such action in one of the three categories of sustainability of the program. The concepts of those ADS attributions were defined in accordance with the Ministries in simple expressions, that clearly indicate the object to be followed. These indicators were distributed in three sets, economic, environmental and social indicators, according with what these indicators proposed to measure, as follows: Economic Sustainability (SE): The economic sustainability is based on a balanced economic development directed to the solidarity based economy -, on the ability of continuously modernizing the production tools, among others (Sachs, 2000). The agent s work has the objective of encouraging businesses that are economically and productively viable. Relations with broader markets and tools adequate to the necessity of each reality are aspects that must be considered when promoting and encouraging an entrepreneurship. For that group of indexes, since economic development falls upon solidarity questions, the points subjected to monitoring include actions and articulations referring to ADS work, the economic and financial viability of the projects in which they participate and the income distribution among the associates. Environmental Sustainability (SA): Environmental sustainability s vision is to develop without harming the environment. Sustainable use of renewable resources and waste treatment are issues considerate in this group and its aim is to show that ADS are concerned about environmental issues. To do so, indexes are direct with measurable questions, for example, if renewable sources resources are used, if there is waste emission to the 6

7 environment, if the entrepreneurship allows deforestation. In these cases, if there is waste reuse for recycling processes or productive chains and if there is reforestation are answers that account positively for the indicator. Social Sustainability (SS): Alternatives created to minimize social problems meet new bases in social development interdisciplinarity. These actions are supported through health access, home, education and employment, i.e., minimal conditions that grant dignity to individuals as human beings. This group of indicators seeks actions that tend to minimize exclusion conditions, whether they are directly or indirectly related to ADS action. Due to the importance of social concepts involved in the agents work, monitoring-related information are at the responsibility of several actors working at the focused location, which are PBF managers, social control instances, Social Assistance Reference Centers, among others. On the complementary programs vision, it is aimed the insertion of PBF s beneficiaries in ADS actions. Each one of these set of indicators will be transformed into an economic, a social or an environmental number index of sustainability ISE, ISA and ISS, through a statistical procedure that attributes them weighs to measure their relative importance. The indexes will vary in an interval between 0 and 1. Numbers near 0 will indicate bad results and numbers near 1 will indicate great results. The economic, environmental and social indexes will support the construction of a fourth and broader index of sustainable development IDS, through the sum of the three indexes (Figure 2). This procedure will result in four indexes and will permit analyses of the impact of each ADS action in the project sustainability. Figure 2: ADS indexes synthesis Moreover, the economic, environmental and social indexes of each ADS will be summed up separately, in order to obtain monitoring global indexes of PPDLES for these three aspects global ISE, global ISA and global ISS. And the synthesis of these three indexes will support the construction of the global sustainable development index of PPDLES global IDS (Figure 3). The variation of these four indexes will show the project effectiveness and evolution. 7

8 Figure 3: PPDLES global IDS synthesis 3.4 Quantitative and Qualitative Evaluation The indexes produced by the monitoring strategy permits a quantitative evaluation that directs a subsequent qualitative evaluation. The monitoring and evaluation strategy also gathers the necessary information to verify the program effectiveness and to plan the program management and possible interventions. The evaluation of each one of the three sets of indicators must enclose the actual situation and the historical evolution and must allow the verification of future possibilities in order to analyze the sustainability of the local development strategy (Reis & Silveira, 2000). The broader index of sustainable development will contribute to the quantitative evaluation of the functioning of all PPDLES. Low numbers indicate that the general objectives of the program were not reached. High numbers indicate that the program is meeting the expected targets. The estimation of economic, social and environmental indexes will allow evaluations of these aspects individually. The index associated with each ADS will show the effectiveness of his or her work to promote the local sustainable development of his or her region, through a quantitative evaluation tool. Low numbers indicate fails and difficulties in reaching the expected targets, or the non-fulfillment of the work obligations. High numbers indicate great experiences and success in the objectives. The estimation of economic, social and environmental indexes will be used to get information about ADS activities in these aspects, separately. This quantitative information contributes to the management of any public policy or program. These results only give a general idea about PPDLES and about ADS work. The indexes are insufficient to direct interventions to improve the program. It is necessary to know why the monitoring strategy disclosed low or high numbers for these indexes. It is necessary to use these indexes to support a qualitative evaluation that will clarify the reasons of difficulties and successes. The search of qualitative information demands more work. Therefore, the quantitative evaluation contributes to direct the efforts to only some essential points that can be improved (with lower indexes) or to show the ways to reach PPDLES objectives (with higher indexes). 8

9 The search of qualitative information to identify difficulties and successes of PPDLES requires direct contact with ADS, through or telephone, or field trips. Each Ministry will be responsible for the qualitative evaluation of aspects of its own interest. For example, the Ministry of Social Development and Fight against Hunger (MDS) will search for reasons that would be influencing social index. Thus, this information would permit the management of this aspect of PPDLES and the planning of MDS intervention strategies. 4. Lessons Learned This monitoring and evaluation strategy will be useful to verify the potential of ADS actions to promote sustainable projects of local development in their territories and to identify key factors of successes or difficulties. In addition, the M&E framework will also be used in a conceptual way, having an impact on other policies and programs, through an increase in the level of knowledge (Calmon, 1999, apud Silva, 2005). One advantage of this strategy is the subdivision of the global sustainability index into three specific indexes: economic, social and environmental aspects of PPDLES. A low index indicates that there is a problem, which can be solved through interventions that are pointed out by its indicators. Moreover, this framework attends the monitoring needs of the three partner government departments by using only one monitoring and evaluation system and permits specific interventions for each department. A third advantage of the monitoring and evaluation framework is the possibility to identify common problems of different regions and ADS and to propose supra regional actions that can address this kind of problem. It can be done by analyzing the sustainability indexes of ADS work in a specific region. Those three advantages allow the allocation of government funds where they are really necessary. The proposed monitoring and evaluation framework is flexible and can not only be implemented for PPDLES evaluation, but also be extended and replicated to other complementary programs and actions. Some of PBF complementary programs intend to finance (as micro credit programs) or develop projects (as Biodiesel program) in order to promote income access. The presented monitoring and evaluation framework can be used to verify these projects sustainability, through indexes of sustainable development, which permits to predict whether these projects will be successful. Apart from the contributions to the present PBF complementary programs, the monitoring and evaluation framework helps to plan new complementary programs and actions to PBF beneficiary families, increasing their probability of achieving social and economic emancipation. As an example, the Biodiesel program encourages small farmers to plant crops for the production of biodiesel (vegetable fuel). The difficulties of insertion of PBF families in this program refer to the lack of technology and knowledge on adequate ways to plant these seeds. Therefore, it is necessary to offer credit and training programs for these families, before their insertion in the Biodiesel program. Thus, the monitoring and evaluation framework presented in this paper is an important tool that can help to promote the sustainability of PBF programs, and subsequently contribute to combat social exclusion and poverty in Brazil. This framework is also important in the context of intergovernmental policies, because it can be useful to other Ministries and it may be replicated to other federal policies with MDS support. 9

10 5. References Dagnino, R. (2002). Metodologia de análise de políticas públicas. São Paulo: GABI: UniCamp. Franks, T. R. (1996). Managing Sustainable Development: Definitions, Paradigms, and Dimensions. Sustainable Development, vol. 4, Garcia, R. C. (2001). Subsídios para organizar avaliações da ação governamental. Planejamento de políticas públicas, Brasília, n.º 23, pp Kusek, J. Z. & Rist, R. C. (2004). Ten steps to a results-based monitoring and evaluation system. Washington DC: The World Bank. I Conferência Nacional de Economia Solidária (2006). Anais. Brasília, 26 a 29 de junho de Nascimento, V. M. (1999). Método para mapeamento do fluxo de informações do processo de suprimento na indústria da construção civil: estudo de caso múltiplo em empresas do subsetor edificações. Dissertação de Mestrado. Florianópolis: UFCS. Sachs, I. (2000) Caminhos para o desenvolvimento sustentável. Rio de Janeiro: Garamond. Silva, R. T. (2002). Eficiência e eficácia da ação governamental: Uma análise comparativa de sistemas de avaliação. Brasília: Ipea. Silva, M. F. S. (2005). Modelo de referência para análise e desenvolvimento de sistemas de monitoramento de intervenções de governos municipais no campo habitacional. Tese de doutorado. Florianópolis: UFSC. Singer, P. (2002). Introdução à Economia Solidária. São Paulo: Editora Fundação Perseu Abramo. Reis, L. B. & Silveira, S. (2000) Energia elétrica para o desenvolvimento sustentável: Introdução a uma visão multidisciplinar. São Paulo: USP, EDUSP. 10