Executive Summary March 2017

Transcription

1 Executive Summary March 2017 Impact on grant performance and management behavior following introduction of Principal Recipient Management Dashboards for grants from the Global Fund to Fight AIDS, Tuberculosis and Malaria: An evaluation of the 2014 pilot Authors Christine Onyango Saba Waseem Contributors Catherine Severo Clare Gibson Maria Trujilllo Lisbeth Loughran Neann Mathai Anaise Kanimba This report is made possible by the support of the American People through the U.S. Agency for International Development and the U.S. President s Emergency Plan for AIDS Relief. GMS is funded by the U.S. President s Emergency Plan for AIDS Relief and the U.S. Agency for International Development under contract No. AID-OAA-C The contents of this report are the sole responsibility of Grant Management Solutions, and do not necessarily reflect the views of the U.S. Agency for International Development, the U.S. government, or the Global Fund to Fight AIDS, Tuberculosis and Malaria.

2 Introduction In 2014, Grant Management Solutions (GMS) and the Secretariat of the Global Fund to Fight AIDS, Tuberculosis and Malaria (Global Fund) piloted the introduction of a grant management dashboard tool for principal recipients (PRs) of grants from the Global Fund with seven PRs in six countries (Cote d Ivoire, the Dominican Republic, the Lao People s Democratic Republic (Lao PDR), Senegal, South Africa, and Uganda). Eighteen months later, GMS carried out a follow-up evaluation to determine the extent to which the seven PRs had institutionalized the PR dashboard in the grant-management process and to identify the performance outcomes achieved as a result of using the tool for data-driven management improvement. This executive summary presents the findings of that evaluation and discusses what might be expected from further rollout. (The full report is available on request from GMS). Assessment objectives This assessment of dashboard institutionalization and management improvement among the seven PRs in the pilot had the following objectives: 1. To determine the changes in management systems, practices and program performance that had occurred as a result of PRs using the PR Management Dashboard. 2. To identify the key factors that foster or inhibit dashboard implementation and consistent use. 3. To determine whether management changes have led to performance improvement. The PR Management Dashboard: What is it? Designed specifically for PRs, the PR Management Dashboard is a visualization of grant-performance data on a single computer screen, with subpages of SR-level programmatic, financial, and managerial information over consecutive reporting periods. The dashboard is produced using two applications: Microsoft Excel and SAP Crystal Dashboard Design The generic PR Data Master and dashboard visualization can be adapted to any grant, any PR, and any disease (HIV/AIDS, malaria, or tuberculosis), disease combination, or HSS grant. The tool is currently available in English, French, Portuguese, and Spanish. The dashboard visualization displays four categories of indicators: finance, program, procurement and supply management (PSM) 1 (focused on the management, availability and quality of key health products), and general management (focused on availability of key staff, reporting and compliance with commitments to the Global Fund). The visualization uses four colors (red, yellow, green, and purple) to indicate levels of performance based on Global Fund thresholds or thresholds jointly set by the PRs and CCM. The dashboard may be printed, archived, circulated as an attachment, or displayed on a website or a computer screen. The PR Management Dashboard was created through collaboration between GMS, the Global Fund and German information technology company SAP SE, with input from PRs in five countries. 2 To produce the dashboard a license, is needed, which 1. In this executive summary, as in the PR Management Dashboard, the term procurement and supply management or PSM is used in place of the newer phrase health products management, which now supersedes PSM. 2. Prior to the pilot, PRs in five countries (India, Nepal, Myanmar, the Democratic Republic of the Congo, and Honduras) validated the indicators to be used in the PR Management Dashboard. 2 An evaluation of the 2014 pilot of Principal Recipient Management Dashboards: Executive Summary

3 the Global Fund makes available thanks to a partnership with SAP; no license is needed to view the dashboard on a computer. The PR Management Dashboard: Why use it? Contextualized within a management-change process, the PR Management Dashboard promotes the Global Fund principles of transparency, accountability and partnership. Through the display of PR and SR data, the dashboard shows the contribution of the PR and each of its SRs to their shared results, thereby demonstrating the partnership among them all and enabling the PR and SRs to hold themselves accountable not only to each other but also to the CCM. When a PR engages its SRs in collective analysis of the data displayed in the PR Management Dashboard, the PR is modeling improved management and thus aligning its behavior with the Global Fund s management standards for PR and SR implementers. Complete adoption or institutionalization of the dashboard process is demonstrated by quarterly or semiannual production of the dashboard with progressive improvement of data quality, completeness, and timeliness. The expected outcomes from using dashboards include the following: analysis and use of the dashboard visualizations during participative review, analysis and investigation first among PRs and SRs and then with the CCM; management and programmatic decision making to reduce risks; accelerated implementation and funds absorption; and increase in grant performance (reflected in grant performance ratings). This change to a more transparent, data driven, and accountable management process requires varying degrees of organizational change among groups of implementers. The following graphic shows the expected immediate management improvements, intermediate implementation improvements, and subsequent changes in grant performance rating and health outcomes that might ensue from improved management using dashboards. Data collection and methodology At the design stage, GMS created data collection instruments that would be deployed over the twoyear period between the pilot and the evaluation. Baseline data included information on the IT readiness of the PR, and assessment of the PR s management, monitoring and evaluation (M&E) and PSM arrangements as well as its readiness for management change. Updates were gathered at each visit the GMS team made to pilot PRs during the two-year period and at 6 and 12 months after the last such visit. An evaluation visit was conducted by one of the two principal authors between November 2015 and February Data were collected through interviews and focus groups of 110 country-level informants, and from telephone interviews with Global Fund fund portfolio managers, or FPMs. These data were supplemented by secondary quantitative data on programmatic activities, grant performance ratings and disbursement rates. All data were triangulated to minimize bias, strengthen validation, and enrich the findings. Scores were calculated for the eight elements of complete adoption, and for the five elements of improved grant management and implementation results: each PR was scored accordingly. 3 An evaluation of the 2014 pilot of Principal Recipient Management Dashboards: Executive Summary

4 Key findings and illustrative cases seven PRs had fully institutionalized the dashboard, sharing it regularly with subrecipients (SRs) in a transparent review process promoting inter-sr dialogue and positive emulation. Three other PRs had integrated the dashboard for use in internal data-driven management, with limited or no sharing with SRs, while the remaining PR had rejected the dashboard after two highly assisted periods of production. The first three PRs demonstrated immediate improvements in management and implementation problem solving: the first two improved their grant scores to A1; the third PR s grant ended after three dashboards with no improvement. PRs that did not modify their management practices saw declines in grant ratings. The graphic Composite Adoption Scores of PRs shows the relative rankings of the seven PRs based on their adoption scores. At 18 months after the end of introductory technical support, three of the The graphic Improvements in Grant Management, Implementation and Ratings through the Use of the PR Management Dashboard summarizes the adoption scores and the outcome scores calculated for each PR. 4 An evaluation of the 2014 pilot of Principal Recipient Management Dashboards: Executive Summary

5 Use of the dashboard seems to have enabled all PR users to reduce some implementation risks, but led to improved grant ratings only for those PRs that most aggressively pursued problem identification and rapid resolution of obstacles to grant implementation. Two examples show the potential for improved performance resulting from data-driven management problem solving. First, in Uganda in September 2014, The AIDS Support Organisation (TASO), the civil society PR for the malaria grant, learned from its dashboard data that there was an overstock of one ACT drug. TASO contacted the government s national medical stores and proposed redistributing the overstocked product through the national supply chain mechanism to other parts of the country that needed this product. By the end of the grant (December 2014), the overstock was eliminated without product wastage. Second, Alliance Cote d Ivoire, PR for the HIV/AIDS grant, saw persistent low performance, with a number of indicators (including Number of persons tested who know their HIV status ) attaining only 35% of target after six months. The PR tested solutions to address hypotheses about the problem and used the dashboard to observe impact. The PR determined that the barriers were 1) inadequate collaboration by district health authorities and 2) failure of SRs to adequately report on health products used during voluntary counseling and testing (VCT). The PR requested the national AIDS program communicate with district health authorities to reassure them that the community health workers (CHWs) had been adequately trained, and required SRs to use a form for justification of the last VCTrelated health products used. Through its advocacy with the national AIDS program, the PR was successful in getting district-level health authorities to integrate CHW testing needs into its quantification of VCT supplies so that sufficient quantities could be issued to CHWs. Four reporting periods later, overall performance of this indicator had improved from 35% to 93%. Discussion of results From the dense body of evidence from the seven PRs and their stakeholders about the first months since introduction of the PR dashboard, four patterns emerge that suggest how organizational and attitudinal factors affect the way in which PRs adopt data-driven management change and the extent to which modified management affects grant performance. Three institutional characteristics appear to influence dashboard use as well. Organizational and attitudinal factors Multiple observations were grouped into the following four factors: 1) Senior management s willingness to engage in and encourage management improvement. Cited and observed as a significant determinant of adoption across all types of PRs, level of commitment is an indicator of demand for the tool by its main intended users and is a prerequisite for its successful implementation. Where a champion existed at the senior management level (e.g., in Cote d Ivoire, Senegal (Plan International), South Africa), PRs were able to use the tool to achieve improved performance outcomes. 2) Midlevel staff capacity and engagement with dashboards, including IT capacity. The presence of M&E and IT capacity and enthusiasm among PR operational staff to regularly produce and use the dashboard emerged as a significant factor in its sustainability. Staff at the most successful adopters were able to perceive quickly the 5 An evaluation of the 2014 pilot of Principal Recipient Management Dashboards: Executive Summary

6 benefits of the tool for reporting to senior management and monitoring SR performance. This engagement was backed by high levels of capacity to collect and verify data, and maintain the tool. The staff were able to resolve technical issues without additional technical support. 3) Communications, transparency, accountability, grant review processes between PR and SRs. Where mechanisms and activities for review of performance and management problem solving between PR and SRs existed, dashboard adoption improved effectiveness and enabled positive emulation between SRs. Initial fears of negative comparisons between SRs were allayed as the usefulness of individual data became clear, and focused strengthening was provided. All five PRs that implemented dashboards fully or even to a limited degree were able to improve communication with SRs over time. 4) Data quality, completeness, timeliness. Review of dashboards leads to intense scrutiny of indicators: most PRs discovered data issues during dashboard implementation. Data-quality issues featured prominently as an obstacle to full dashboard implementation for most publicsector PRs participating in the pilot. These ranged from problems with obtaining data from national M&E systems, SRs or implementing partners to ongoing problems with timeliness or quality of data submitted. Governmental PRs started with lower levels of data quality, with little direct control over regions or districts in implementation and reporting. They faced difficulty setting up systems to regularly obtain PSM data from their govern-mental agencies responsible for procurement and supply of health products. Despite these challenges, most governmental PRs were able to make some degree of improvement in data quality of indicators, as did the civil society PRs. Comparing baseline and final scores on these factors in spider diagrams showed that factors 1 and 2 were predictive of the extent of dashboard institutionalization: strong seniormanagement engagement and midlevel management capacity were predictive of more extensive pursuit of management improvement. Factors 3 and 4 were responsive to the management improvement process with dashboards. Institutional characteristics Seeking to understand the dynamics between the predictive and responsive factors, GMS identified three pairs of institutional characteristics that cluster the pilot PRs into behavior groups. Civil society PRs or public-sector PRs. The three civil society PRs are apex organizations, with no higher organizational authority above them in the country. All include in their missions organizational strengthening of local civil society associations and community organizations such as their SRs. The role of PR both identifies them as leaders in the civil society community and provides a revenue stream for a portion of their staff. Timely, good quality performance and reporting assure their revenue streams and allow them to fulfill their institutional missions: two powerful performance incentives. The public-sector PRs all are embedded in or responsible to higher levels of government. Two have broader national strategic or programmatic disease-related responsibilities for HIV or for TB: acting as PR is a secondary role for them. Organizational development of implementers is not part of their mandate. While Global Fund support is often a significant part of financing for their programs, the core funding of these PRs comes from the national budget, not the Global Fund. 6 An evaluation of the 2014 pilot of Principal Recipient Management Dashboards: Executive Summary

7 The two groups of PRs had very different initial management styles and structured relationships with their SRs. The NGO PRs all had formal subagreements with their SR organizations, which are separate legal entities. All three had regular meetings and performance discussions with SRs, and practiced some form of transparency with them. For these PRs, the PR dashboard made the SRs performance more visible than it had been and enabled them to facilitate sharing and comparison of SR information without requiring a fundamental shift in the way they managed their SRs. None of the three public-sector PRs had regular group meetings with their SRs: one had an elaborate individual SR supervision process; another planned quarterly feedback sessions with SRs but sometimes had to prioritize other activities and cancel the sessions; the third had no work plan or budget for such encounters or other national meeting opportunities on which to build. For these public-sector PRs the dashboard created greater accessibility to grant performance data for internal management purposes. Dashboard data were used fairly consistently in internal discussions, especially by M&E staff. However, the structural barriers to SR participative performance analysis seem to have been too great to scale up dashboard use; breaking down these barriers would have contradicted other more permanent lines of communication and authority between governmental units. PRs with or without project management units (PMUs). In the public sector, PMUs are almost all situated either in ministry of finance PRs or MOH PRs, have some amount of dedicated staff charged with financial management, reporting coordination, communication with the Global Fund country team and the CCM, and coordination with or oversight of pharmaceutical procurement. Technical oversight and coordination, procurement and sometimes M&E and technical reporting, are usually in the purview of the national disease programs. The theoretical advantages of a PMU include greater visibility of Global Fund activities, greater involvement of high-level decision makers in supervision and leadership, and efficiency of staffing for common grant management functions. In the INGO sector, PMUs are usually set up when the INGO is a large organization with a broad range of other activities and/or when the INGO is responsible for more than one grant: the advantages of efficiency and focus are intended to be the same. Three of the PRs in the pilot had PMUs (two public and 1 NGO). All three had little or intermittent contact with their higher level managers, who, while stating their support for dashboard introduction, were not often attentive to reviewing dashboard data, nor to actively leading, approving, or supporting management changes based on problem investigation. Since PMUs usually lack line authority over implementers, this inaccessibility to senior leaders often left PMU staffers powerless to address situations that they had identified as needing attention after dashboard analysis. CCMs already using the old CCM grant dashboard or not using any dashboard. In three countries, the CCM already used the older CCM dashboard (from 2009) as an oversight tool. They were initially reluctant to allow the PR to replace it with the new PR dashboard and required the PR to produce both. CCMs preferred their current system since they could compare performance between grants, and 7 An evaluation of the 2014 pilot of Principal Recipient Management Dashboards: Executive Summary

8 since their secretariats managed those dashboards. This dampening effect was acknowledged by both PRs and CCM staff in these three countries. However, these three CCMs did rapidly come to embrace and even promote the new dashboard once an initial PR adopter demonstrated the dashboard s utility during special meetings. Thereafter, CCM engagement led to adoption of the new dashboard by all PRs in two of the three cases. Conclusions on evidence-based expectations from continued rollout of the PR management dashboard This evaluation found that PRs varied in the degree to which they adopted the dashboards, with some using them for more limited purposes than originally postulated. Different degrees of adoption were influenced in part by varying initial motivation for using the dashboard as well as the managerial and attitudinal factors described earlier in this executive summary. Partial adoption, such as for internal management only, while sufficient for those PR s immediate needs, led to more attenuated results. A number of structural, managerial and attitudinal factors influence dashboard implementation and consistent use for datadriven management. In particular, the extent to which senior management agrees to and actively supports management change and data-driven decision making, the capacity and engagement of PR staff for dashboard use and the type of PR (civil-society or public-sector), seem determinant to the way in which the dashboard-management process will be institutionalized. Two other factors respond most clearly to data-driven management change: 1) the extent of collaborative, analysisdriven dialogue between and among the management-level staff of PRs and SRs and 2) the quality of data and timeliness of reporting. Prior use by the CCM of the CCM grant dashboard that was introduced in 2009, on the contrary, may slow institutionalization of the PR Management Dashboard. Introduction of the new CCM Summary (designed to replace the 2009 CCM grant dashboard) through the wholeof-country approach helps to overcome the barrier of CCM resistance to PR dashboards. These results suggest that introduction of the PR Management Dashboard through the wholeof-country approach for all PRs and the CCM, rather than with individual PRs, has more potential for risk management and performance impact on grants financed by the Global Fund. Grant Management Solutions 4301 N Fairfax Drive, Suite 400 Arlington VA 22203, USA For more information, contact : info@gmsproject.org 8 An evaluation of the 2014 pilot of Principal Recipient Management Dashboards: Executive Summary