Evidence to Policy initiative (E2Pi) Estimating Benchmarks of Success in the AMFm Phase 1

Transcription

1 Evidence to Policy initiative (E2Pi) Estimating Benchmarks of Success in the AMFm Phase 1 Presentation to IHME, March Gavin Yamey MD MPH, lead, San Francisco hub, E2Pi

2 My My background Background Clinical medicine, journalism, public health 2009 Kaiser Family Foundation Mini Fellow in Global Health Reporting (Sudan, Kenya, Uganda) Telegraph (UK) 5 October 2009 The Net Gains of Keeping Mosquitoes at Bay

3 GH policy collaboration» GHG at UCSF» SEEK Development, Berlin Evidence synthesis & policy analysis to inform discussion & decisions on critical policy and strategic issues Two focus areas» GH financing partnerships: Global Fund, GAVI, UNITAID» European donors Core funding from Gates Foundation + contracts

4

5 Overview AMFm objectives Terms of reference Evidence synthesis & key informant interviews 3 approaches to estimating success benchmarks Pragmatic mixed methods approach Weighted mean approach Monte Carlo multivariate sensitivity analysis Estimates of success benchmarks Balancing success measures

6 Why Was the AMFm Launched? WHO recommends ACTs for uncomplicated P. falciparum malaria BUT <15% children (range 3 25%) with fever in SSA receive ACTs [World Malaria Report 2009] Key reason: about 50 75% patients with suspected malaria seek R x in private sector» ACT course typically costs $6 8» times as much as older drugs (e.g. CQ, SP)

7 AMFm: Global ACT Price Subsidy Scheme Donor funded ACT price subsidy managed by the Global Fund Donors subsidize the cost of ACTs paid by 1 st line buyers» buyers who purchase ACTs directly from the manufacturer, or procurement agents buying on their behalf» private/public/ngo Proponents argue: subsidy will be passed along supply chain to the consumer Subsidy is combined with supportive interventions

8 donor co payment AMFm Technical Design, Nov 2007

9 Objectives of AMFm 1. Reduce price of QAACTs compared to price of other antimalarials [AMs] (e.g. CQ, SP, AMT) 2. Increase availability of QAACTs in public/private outlets 3. Increase market share of QAACTs and crowd out other AMs 4. Increase use of QAACTs, including by vulnerable groups subsidy + supportive interventions ACT price falls ACT availability increases ACT access and use increase malaria burden falls

10 AMFm Pilot (Phase 1) Pilot launched in August 2010 in 8 countries» Cambodia, Ghana, Kenya, Madagascar, Niger, Nigeria, Tanzania, Uganda Independent evaluation (Macro & LSHTM): before and after design, measures 3 out of 4 objectives (not use) Data collection must be complete by Nov 2011 (by then, only 3 countries will have had subsidized drugs for at least 1 yr) Global Fund Board will decide at its 2 nd meeting in 2012 whether to continue, expand, suspend, or terminate the program

11 Overview AMFm objectives Terms of reference Evidence synthesis & key informant interviews 3 approaches to estimating success benchmarks Pragmatic mixed methods approach Weighted mean approach Monte Carlo multivariate sensitivity analysis Estimates of success benchmarks Balancing success measures

12 E2Pi s Terms of Reference E2Pi was commissioned to: Provide evidence that will help the AHC to reach decisions on how to judge success of AMFm Phase 1 Estimate what might be realistically expected at 1 year and 2 years for the 4 AMFm objectives Present ways of balancing and judging performance across the 4 objectives, including: Balanced Scorecard GF type approach Weighting of indicators

13 Process and Timeline 15 days to conduct initial literature review/key informant interviews June 2010: draft presented to AHC Additional interviews and literature review Peer review: revised version sent to 9 external peer reviewers Addressed reviewer comments Oct 2010: revision presented to AHC Dec 2010: final paper to AHC & Global Fund Board

14 Overview AMFm objectives Terms of reference Evidence synthesis & key informant interviews 3 approaches to estimating success benchmarks Pragmatic mixed methods approach Weighted mean approach Monte Carlo multivariate sensitivity analysis Estimates of success benchmarks Balancing success measures

15 Literature Review Four major sources of evidence 1. Sub national ACT subsidy pilots and national ACT subsidy programs 2. Country level data on other national ACT scale up programs e.g. funded by GFATM 3. Country level data on commodity social marketing programs e.g. zinc, condoms 4. Global level data on oral rehydration therapy scale up

16 33 Key Informant Interviews Malaria experts: ACTwatch, CHAI, Global Fund, HAI, MMV Social marketing organizations: PSI, MSI Drug companies: GSK, Novartis, Cipla, Ipca Local drug importer/distributor: Surgipharm (Uganda) Academia: GWU, Johns Hopkins, LSHTM, Karolinska Institute, MIT Zaragoza

17 ACT Subsidy Pilots: Design and Scale DESIGN SCALE Kenya Cluster RCT 3 districts, 18 clusters Tanzania Quasi-randomized trial 2 intervention districts 1 control district Uganda Non-randomized, controlled trial 4 intervention districts 1 control district Angola Uncontrolled trial 2 municipalities (95 pharmacies) Only 1 trial is published: Sabot OJ et al (2009) Piloting the Global Subsidy: The Impact of Subsidized Artemisinin Based Combination Therapies Distributed through Private Drug Shops in Rural Tanzania. PLoS ONE 4(9): e6857

18 ACT Subsidy Pilots: Outcomes AVAILABILITY Proportion of private outlets stocking ACTs rose from 0% to 69 81% at 1 yr (3 pilots); control data only for TZ pilot (fell from 1% to 0%) MARKET SHARE PRICE USE Increased from 0 1% to 38 51% at 1 yr, lower among poorest SEQ (3 pilots); control data only for TZ pilot (rose from 0% to 6%) Similar to price of suboptimal AMs at 1 yr (4 pilots) in intervention districts; no control data in 3 controlled trials Uganda (non randomized): Increase from baseline of 16 percentage points in intervention group but outcomes were better in control group Kenya (RCT): Increase from baseline of 40.2 percentage points in the intervention arm and 14.6 percentage points in the control arm at 1 yr

19 Why Not Extrapolate Directly From ACT Pilots? Pilots are small scale, focusing on sub national level Additional distribution mechanisms (Angola, Uganda) Tight monitoring of price violations (Angola) Intensive donor funded supportive interventions Design weaknesses: small samples, lack of randomization, one was uncontrolled, one trial introduced a new intervention Results represent trial conditions not real world conditions Great care should be taken in trying to learn lessons for the AMFm Phase 1 from the small pilots KI INTERVIEW

20 Data from 6 National ACT Subsidy Programs Country Lead organization Launch yr Age group Outlet Coverage Cameroon Govt All age groups Public and private health facilities Countrywide Senegal Govt All age groups Pharmacies Countrywide Cambodia PSI 2002 All age groups Pharmacies, drug shops 17 of 20 malaria endemic provinces DRC PSI 2006 Children under 5 Pharmacies Limited to some districts Madagascar PSI 2003 Children under 5 Pharmacies, private providers, community agents Countrywide Rwanda PSI 2007 Children under 5 Pharmacies Countrywide

21 National ACT Subsidy Programs: Availability Only 1 program has baseline data: Rwanda: 80 90% at 18 months after launch (baseline 10%) Cambodia: Child ACT 6%, Adult ACT 22% at 1 yr Cameroon: Low availability (key informant interviews) Senegal: 29% infant ACTs, 43% child ACTs, 11% adult ACTs at 1 yr Other AMs still very widely available in all countries except Rwanda (e.g. Senegal: SP availability % in private outlets)

22 National ACT Subsidy Programs: Market Share MARKET SHARE Very few data available; no baseline data Cambodia: ACT accounted for only 28% of all AM sales in private outlets at 6 yrs; mono therapies still accounted for 50% of all sales in commercial private sector E2Pi analysis of PSI sales volume data:» It took PSI at least 3 yrs to reach substantial sales volumes for ACT (confirmed by PSI interviews)

23 Trends in PSI Sales of Subsidized ACT Feb 2003 July 2003 Red arrows show when subsidized ACT sales began It took at least 3 yrs to substantially increase sales volumes Sept 2006 Jan 06

24 National ACT Subsidy Programs: Use No baseline data Low levels of ACT use in 3 countries» DRC: 1% at about 1 yr» Senegal: 4% at 2 3 yrs» Madagascar: 2.4% at about 5 yrs

25 National ACT Subsidy Programs: Price Very few available data (specific data for only 2 countries) Senegal: mean consumer price for adult ACT at private outlets was $1.34 at 1 yr, similar to RRP, lower than SP price ($2.00)» Private outlets purchased subsidized ACTs at $0.99 i.e. subsidy largely passed on to consumers Cambodia: mean consumer price for adult ACTs was $1.07 at 4 yrs, 535% of CQ price ($0.20)» Private outlets purchased subsidized ACTs at $0.42» Similar mark up for RDTs (sold to retailers at $0.10/test; consumers pay mean price of $0.37/test)

26 Summary of Results from Pilots/Programs Pilots found a rapid rise in ACT availability in private outlets, as did one national program Subsidies were associated with reduced consumer prices (i.e. subsidies were largely passed along the supply chain to the consumer) ACT market share increased rapidly in pilots, crowding out other AMs, but not in national programs Pilots found conflicting evidence on ACT use (one twas positive, one was negative) and national programs found very low levels of use at 1 5 yrs

27 Evidence from National ACT Scale Up Programs Data sources ACTwatch outlet and household surveys Surveys conducted in the context of the Global Fund 5 Year Evaluation Other national level surveys (DHS; MICS; MIS) Peer reviewed studies in public health journals Results ACT availability, market share, and use were poor across surveys and countries at 1 2 yrs (and beyond), with a few exceptions (see next 2 slides) Results suggest great caution in expecting dramatic changes within 1 2 yrs at national scale in AMFm

28 Analysis of ACTwatch Outlet & Household Surveys Two possible baselines for estimating success of ACT scale up:» Yr that ACT was adopted as 1st line treatment» Yr of national ACT roll out Yr that ACT was adopted as 1st line is problematic: large time lag between change in national drug policy and actual ACT roll out* Yr of national ACT roll out is more appropriate to help learn lessons for AMFm We determined yr of national roll out from reports by ACTwatch, Global Fund, PMI, Ministries of Health *Source: Amin, A. et al. (2007) The challenges of changing national malaria drug policy to artemisinin based combinations in Kenya. Malaria Journal 6:72.

29 Results of ACTwatch Surveys, 7 countries (1 st line ACT, national ACT roll out as baseline) AVAILABILITY Ranged from % (1 6 yrs) MARKET SHARE Ranged from % (1 6 yrs) USE Ranged from 2.4% to 19.3% (1 3 yrs) PRICE Large price differences between ACTs and other AMs» ACT prices up to 40 x higher than mono therapies

30 Global Fund 5 Year Evaluation The Evaluation.indicated very low levels of use of ACT thus far 5% or lower for all countries except Tanzania (20% in 2007) and Zambia (8 15% in ). This finding is the most perplexing, showing the least improvement in coverage of the four primary malaria interventions.

31 Commodity Social Marketing Programs Literature review on two social marketing (SM) models NGO model (primarily used in low income countries) Focuses on population groups that cannot afford to pay commercially viable prices Donor subsidy to keep prices low Additional supply chains set up by the NGO Manufacturer s model (used in middle income countries) Aims to be self sustaining without donor support Uses a commercial company s existing distribution channels Source: Meekers D, Rahaim S (2005) The importance of socio economic context for social marketing models for improving reproductive health: Evidence from 555 years of program experience. BMC Public Health 5:10

32 Evidence on Outcomes in the NGO model PRICE AVAILABILITY Price increases are associated with dramatic fall in use Water purification: % at 1 yr Condoms: 25 39% at 4 6 yrs MARKET SHARE USE Condoms/oral contraceptives: 10 15% at 3 yrs SUZY Project, Bangladesh, a very similar model to AMFm:» national subsidized commodity SM program with SIs» treatment of a major childhood illness» care seeking from private providers» aims to crowd out ineffective drugs (anti diarrheals, antibiotics)

33 Zinc Usage Rates at 1 2 yrs % Children Baseline months months receiving Zn City slum 4% 16% 19% City non slum 15% 26% 25% Municipal 7% 18% 17% Rural 4% 11% 12% Usage leveled off from 1yr to 2 yrs Usage was higher in higher quintile wealth assets 8 15% increase from baseline Source: Larson CP et al (2009) Impact Monitoring of the National Scale Up of Zinc Treatment for Childhood Diarrhea in Bangladesh: Repeat Ecologic Surveys. PLoS Med 6(11): e

34 Manufacturer s Model of Commodity SM Few studies on the manufacturer s model of commodity SM that can help guide expectations for AMFm Phase 1 Only 1 study examined impact of model on metrics relevant to AMFm» Study on socially marketed oral contraceptive pill (OCP) in Morocco» Increase in market share of 3% from baseline at 2 y and 12% at 10 y Manufacturer s model tried without success in Africa (e.g. OCP in Nigeria) Source: Agha, S. et al (2005) When Donor Support Ends: The Fate of Social Marketing Products and the Markets They Help Create. Bethesda, Maryland: Abt Associates Inc.

35 Global Data on ORT Scale Up Global usage rate increased from 35% to 41% between Annual increase of 0.39% Source: Forsberg BC et al (2007) Diarrhoea case management in low and middle income countries an unfinished agenda. Bull World Health Organ 85:42 48

36 Results of Key Informant Interviews: Industry Expectations Introducing new drugs into developing country and emerging economy markets» Northern drug manufacturers:»market share of 10% at 1 yr and 20% at 2 yrs» Drug manufacturers in India:»Market share of 4 5% at 1 yr and 10% at 2 yrs»higher expectations for ACT under AMFm: market share of 10% at 1 yr and 25% at 2 yrs

37 Dominant Messages from Other Key Informants Time scale of 1 2 yrs is short, especially to see changes in use Rural areas likely to show less uptake Uptake will depend on a range of factors, including:» quality of supportive interventions and distribution systems, socioeconomic factors, regulatory frameworks, structure of private supply chains, malaria treatment seeking behavior, urban rural population ratio Small countries with supportive government are likely to achieve best results (e.g. see Rwanda s national ACT subsidy program)

38 Overview AMFm objectives Terms of reference Evidence synthesis & key informant interviews 3 approaches to estimating success benchmarks Pragmatic mixed methods approach Weighted mean approach Monte Carlo multivariate sensitivity analysis Estimates of success benchmarks Balancing success measures

39 Mixed Methods Approach to Estimating Benchmarks Data collection Literature review Collected unpublished data 33 key informants Data appraisal Summarized range of outcomes To what extent do study conditions resemble AMFm? Initial benchmarks Estimated initial set of benchmarks Discussed these with selected key informants Input from AHC Received input at two AHC Meetings (June, Oct 2010) AHC suggestions informed our report Final estimates

40 Success Benchmarks (Mixed Methods Approach) Yr 1 Yr 2 Availability increase of 20 percentage points from baseline (QA-ACTs) increase of 40 percentage points from baseline (QA-ACTs) Market Share increase in ACT market share of percentage points from baseline fall in market share of artemisinin monor x (AMT) from baseline increase in ACT market share of percentage points from baseline fall in market share of AMT from baseline Use Price increase of 5-10 percentage points from baseline ACT price < 300% price of dominant non-qaact (usually CQ, SP) ACT price <price of AMT (useful but not sufficient to determine success) increase of percentage points from baseline ACT price < 150% price of dominant non-qaact (usually CQ, SP) ACT price <price of AMT (useful but not sufficient to determine success)

41 The Challenge of Estimating the Price Benchmark AMFm Technical Design: AMFm.will be measured against its ability to reduce consumer prices of a treatment course of an effective coformulated AM from the current level of USD 6 10 to a far lower level of USD (which is competitive with current retail prices of CQ and SP) for the majority of patients. Thus our estimates are for a price relative to dominant non QAACT, usually CQ or SP (we also note that co paid ACTs should cost less than AMT a useful but not sufficient benchmark) But: there are few empirical data to guide expectations for how quickly ACT prices will fall in the AMFm pilot countries Price change was the indicator with the weakest empirical basis for setting expectations at 1 and 2 yrs

42 The Challenge of Estimating the Price Benchmark (cont d) ACT prices fell rapidly in AMFm pilots, more slowly in 2 national ACT subsidy programs (1 program: subsidy was largely passed on to consumers) Key informants: prices of co paid ACTs are likely to be high initially, then fall with time, especially with strong SIs (e.g. Daily Nation reports high initial prices in Kenya) Our benchmarks: <300% price of dominant AM (usually CQ, SP) at 1 yr, <150% price of dominant AM at 2 yrs (reaching parity beyond 2 yrs) e.g. Nigeria: SP makes up 52% of all AMs sold/distributed and costs $0.54 (adult course). ACTs cost $6 8. Our success benchmarks: ACT prices have fallen to under $1.62 by yr 1 and under $0.81 by yr 2

43 Two Alternative Approaches to Estimating Success Benchmarks (introduced after peer review process) Extrapolated outcomes to 1 and 2 yrs Weighted each study: Extent to which it resembled AMFm Methodological rigor Simple weighted mean approach Aggregate weighting model approach using Monte Carlo simulations (Stephane Verguet, IHME)

44 Step 1: Extrapolation of Available Data For both approaches, available data points were extrapolated to estimate the results at 1 and 2 years Extrapolation was necessary as these two quantitative approaches require a minimum number of data points yet few studies measured outcomes at 1 and 2 yrs (e.g. Uganda pilot: usage was only measured at 1 yr) Extrapolation assumed a linear scale up model

45 Step 2: Weighting System Assigned each study a weight of 1 4 points 1 point if study was national (rather than local) 1 point if study examined a treatment (rather than preventive intervention) 1 point if there was a price subsidy in place 1 point for rigor (defined as randomization in a trial or random selection in a survey) Weighting involved a judgment call so we also estimated a standard deviation (SD) for each weight based on our level of confidence in our weighting (SD captured the uncertainty)

46 Example of Data Inputs: Studies on Usage Study Result 1y Result 2y Result 3y Result 4y Result 5y Wt (mean µ) Wt (SD σ) Pilot: Uganda 16% 32% Pilot: Kenya 40.2% 80.4% Pilot: Benin 4.5% 9.0% ACTWatch: Madagascar 1.2% 2.4% ACTWatch:Uganda 3.52% 7.04% 10.56% 14.08% 17.6% ACTWatch:Zambia 4.83% 9.65% 14.48% 19.3% Survey (Simba 2010): 18.8% 37.6% Tanzania Zinc: SUZY trial: Bangladesh 10.3% 10.8% Vitamin A: Survey (Zagré 2002): 33.3% 66.6% Burkina Faso ORT: Survey of 40 countries (Forsberg 2007) 0.39% 0.78% Shaded boxes show extrapolations. Data show changes from baseline

47 Step 3: Estimation of Success Thresholds Weighted mean approach We summed the weighted values and calculated a weighted mean for the 4 AMFm indicators Monte Carlo Multivariate Sensitivity Analysis Analysis by Stephane Verguet (IHME) Assumed a normal distribution for the weights (mean = weight, SD = uncertainty) 95% CI is +2 SD 1000 Monte Carlo simulations to estimate the aggregate uncertainty from the model weight inputs Mean = wt SD = uncertainty

48 Overview AMFm objectives Terms of reference Evidence synthesis & key informant interviews 3 approaches to estimating success benchmarks Pragmatic mixed methods approach Weighted mean approach Monte Carlo multivariate sensitivity analysis Estimates of success benchmarks Balancing success measures

49 Success Benchmarks Derived From 3 Approaches Indicator Year 1 Year 2 Availability (%) Price (US$) Market share (%) Use (%) Weighted mean Indicator Year 1 (95% C.I.) Year 2 (95% C.I.) Availability (%) 22.3 ( ) 36.5 ( ) Price (US$) 3.00 ( ) 3.79 ( ) Market share (%) 11.8 ( ) 21.9 ( ) Use (%) 11.8 ( ) 22.1 ( ) Modeling approach Indicator Year 1 Year 2 Availability (%) Price (US$) <300% dominant non QAACT <150% dominant non QAACT Market share (%) Use (%) Mixed methods approach

50 Overview AMFm objectives Terms of reference Evidence synthesis & key informant interviews 3 approaches to estimating success benchmarks Pragmatic mixed methods approach Weighted mean approach Monte Carlo multivariate sensitivity analysis Estimates of success benchmarks Balancing success measures

51 Approaches to Balancing Metrics of Success Balanced Scorecard Global Fund approach to rating grant performance Weighting systems

52 AVAILABILITY benchmark: 20 percentage point increase from baseline (QAACTs) COUNTRY RESULT: USE benchmark: 5-10 percentage point increase from baseline (QAACTs) COUNTRY RESULT: CONTEXTUAL FACTORS: Date that grant agreement was signed Date that co paid drugs arrived in country PRICE benchmarks: QAACT price<300% of price of dominant non-qaact and price of co-paid QAACT<price of AMT COUNTRY RESULT: AMFm Phase 1 ONE YEAR MKT SHARE benchmarks: percentage point increase from baseline (QAACT) and decrease in mkt share of AMT COUNTRY RESULT: Date when SIs started, type of SIs Other contextual factors (e.g. political instability, urban:rural population ratio, policy on regulation of outlets) Baseline indicators (availability, use, price, market share)

53 AVAILABILITY benchmark: 40 percentage point increase from baseline (QAACTs) COUNTRY RESULT: USE benchmark: percentage point increase from baseline (QAACTs) COUNTRY RESULT: CONTEXTUAL FACTORS: Date that grant agreement was signed Date that co paid drugs arrived in country PRICE benchmarks: QAACT price<150% of price of dominant non-qaact and price of co-paid QAACT<price of AMT COUNTRY RESULT: AMFm Phase 1 TWO YEARS MKT SHARE benchmarks: percentage point increase from baseline (QAACT) and decrease in mkt share of AMT COUNTRY RESULT: Date when SIs started, type of SIs Other contextual factors (e.g. political instability, urban:rural population ratio, policy on regulation of outlets) Baseline indicators (availability, use, price, market share)

54 The Project Team Marco Schäferhoff Policy Analyst, E2Pi s Berlin team Lead author Dominic Montagu Lead, Health Systems Initiative Project consultant

55 Thank You Gavin Yamey Lead, Evidence to Policy initiative (E2Pi) Global Health Group University of California San Francisco