Assessing the Performance of Routine Immunization Systems in Bihar, India with a Congruence Model: Findings and Reflections W. Scott Gordon PATH March 2, 2011
Overview of Presentation Introduction to PATH Context of research changing approaches to immunization in India and Bihar Project objectives Research design Findings Discussion and reflections on model and approach Conclusions Page 2
PATH s mission Improving the health of people around the world by: Advancing technologies Strengthening systems Encouraging healthy behaviors Page 3
PATH s Global Presence Page 4
PATH s Areas of Work Vaccines and immunization Maternal and child health and nutrition Reproductive health Emerging and epidemic diseases Health technologies Page 5
Bihar: A Challenge Study Context: Bihar India State in North India, bordering Nepal Population 93 million (9% of total population) Poor infrastructure and annual floods add to service delivery challenge Poor roads and power supply Frequent polio SIAs 15 days / month Gujarat Rajasthan Punjab Chandigarh Uttarakhand Haryana Delhi Dadra Daman & Nagar & Diu Haveli Maharashtra Lakshadw eep Jammu & Kashmir Himachal Pradesh Goa Karnataka Uttar Pradesh Madhya Pradesh Andhra Pradesh Puducherry Tamil Nadu Kerala Chhattisgarh Orissa Sikkim Arunachal Pradesh Assam Nagaland Bihar Meghalaya Manipur Jharkhand West Bengal Tripura Mizoram Andaman & Nicobar Islands
Fully Immunized Children in India DLHS-2 Data not available 14 to 40 40 to 55 55 to 75 75-85 85-93.4 India JA M MU & K A S HMIR JA M MU & K A S HMIR RAJA S THA N GU JAR AT HIM AC HA L P RA DE SH PUN JAB HAR YANA UTTA RA NCHA L UTTAR PRADESH MA DHY A PR AD ES H HIM AC HA L P RA DE SH PUN JAB Bihar: 24.4 % Bihar: 41.4 % UTTA RA NCHA L BIH AR SIK K IM JH AR KH AN D W E ST BE NGAL ARU NA CHA L PR. AS S AM NAGA L AND ME GHA LA YA MA NIP UR MIZOR AM RAJA S THA N GU JAR AT HAR YANA UTTAR PRADESH MA DHY A PR AD ES H BIH AR SIK K IM JH AR KH AN D W E ST BE NGAL ARU NA CHA L PR. AS S AM NAGA L AND ME GHA LA YA MA NIP UR MIZOR AM D&N HAVELI CHHA TTIS GARH OR ISSA D&N HAVELI CHHA TTIS GARH OR ISSA MAHARASH TRA MAHARASH TRA AND HRA PRADESH AND HRA PRADESH GOA KA RNA TA K A GOA KA RNA TA K A POND ICH ER RY A& N ISL AN DS POND ICH ER RY A& N ISL AN DS TAM IL NA DU LA KS HA DW EE P KE RA LA TAM IL NA DU LA KS HA DW EE P KE RA LA India: 48% India: 54% DLHS-2 (2002-04) DLHS-3 (2007-08) Source: http://www.mohfw.nic.in/dlhs/dlhs08_release_1.htm
Muskan (In English Smile) Enhanced political commitment through change in State administration Target children 0-23 months: ~ 4.7 million Augmentation of immunization efforts started in 2005 through special immunization drives Expansion of financial resources and policy flexibility through National Rural Health Mission Formalized as Muskan in Oct-2007 Involvement of village level Mahila Mandal * Inter-Sectoral Coordination Muskan Strategy Identification & Tracking of beneficiaries Performance based incentives for service providers Review of Microplan Page 8
Routine Immunization System in Bihar Sub-district level microplanning Expansion of workforce through contracted nurses Expanded immunization sessions (Wed and Sat)- Village Health Day Incorporation of supportive supervision Courier based vaccine delivery system Contracted cold chain management Expanded focus on recording and reporting (RIMS, HIMS) Performance based incentives for nurses and community health workers Page 9
Project Objectives To capture and depict the strengths and weaknesses of the immunization system in Bihar State To show the alignment among different system components (formal and informal structures, resources, processes and knowledge and skills) and their impact on the system performance To provide the basis for recommendations to improve the immunization system and its performance Page 10
Congruence & PRISM Models
Alignment Model for Bihar Formal Organizational Determinants Inputs Resources Environment History Individual and Behavioral Determinants Alignment Technical Determinants Performance Informal Organizational Determinants Page 12
Key Activity and Performance Domains Assessed Management and decision making Planning (including the development and use of microplans) Service provision - measured across three performance domains Coverage of services or activities Quality of services Safety of services (both injection safety and safe disposal) Reporting and record keeping Support and supervision (including training) Logistics and cold chain Community mobilization Page 13
Study Tools and Methods Tools for Immunization Systems Performance Assessment (TISPA) Performance and Process Assessment Tool (PPAT) ASHA AWW ANM LHV/MO MOIC & DIO Observation Availability of Resource Assessment Tool (ARAT) Health Facilities Immunization Management Assessment Tool (IMAT) Health Facilities Participant Surveys Organizational and Behavioral Assessment Tool (OBAT) All Health Workers Page 14 Record Review
Study Area and Participants CHAMPARANWEST KAIMUR CHAMPARANEAST SITAMARHI SHEOHAR GOPALGANJ MADHUBANI SUPAUL KISHANGANJ ARARIA SIWAN MUZAFFARPUR DARBHANGA SARAN MADHEPURA SAMASTIPUR SAHARSA PURNIA VAISHALI BUXAR BHOJPUR PATNA BEGUSARAI KHAGARIA KATIHAR BHAGALPUR NALANDA MUNGER ARWAL LAKHISARAI ROHTAS JEHANABAD SHEIKHPURA JAMUI BANKA AURANGAABAD NAWADA GAYA Districts and Blocks Sampled for Study District Block Kishanganj Kishanganj Thakurganj Kochadhaman Muzaffarpur Aruai Musahiri Saraiya Patna Barh Bikram Maner Gaya Barachatti Mohanpur Imamganj Health Personnel Interviewed in the Study Administrative level DIO CMO/MO LHV or Male Supervisor ANM AWW ASHA District 4 Block PHC 12 40 8 PHC 8 8 Sub-center 16 48 AWC 16 48 48 Sub-Totals 4 20 40 48 48 96 Total Number Participants 256
Study Conduct and Analysis Field surveys, November 2009- February 2010 Mixed gender survey teams conducting observation and participant surveys in Hindi Triangulation approach to data collection and analysis Review of formal policies, guidelines, and broad budget allocations based on key activities and performance domains Observation and review of activities and records by field staff Assessment of expressed priorities, values, self-efficacy, and capacity Study conduct coordinated with external ethnographic study and findings jointly reviewed and validated Page 16
Findings There were consistent and relatively uniform expressions of organizational priorities and values across all of the activity domains. The stated priorities and values did not translate into the actions of the staff - a substantial know do gap. The know-do gap also translated into inconsistent performance across most of the activity domains and on many of the technical activities There were tensions between the program s stated goal of decentralization and the manner through which accountability was maintained and planning implemented in the immunization system. All districts showed varying limitations in resources yet no these limitations did not show a consistent link to the performance of technical activities. The formal incentives within the system did not appear to be significantly associated with the performance of many of the technical activities Page 17
Coverage Quality Safety Community Engagement Findings Values and Priorities Consistent Expressions of Values and Priorities Coverage 91% feel committed to improving the immunization status of the target population Quality 85% set doable and achievable targets for performance Safety 97% without safe disposal of immunization waste, diseases can occur Community engagement 83% use immunization data for community mobilization and education Priority Activities of Health Workers 100% 90% 80% 70% 60% ANM 50% LHV 40% MOIC 30% 20% 10% 0%
Findings Management & Decision Making Inputs Provision of untied funds to all levels (70% to block and below) History of highlystratified managerial culture Individual High perceived ability to solve problems based on data Formal Formal policies to decentralize decision making Centralized direction of microplans Informal Varying levels of decision making in some districts Perceived political interference in decisions Some use of evidence- Technical Few directives on policies and priorities provided to block and PHC levels Limited documentation of decisions at block levels Limited availability of data at PHC levels Performance Limited awareness of program performance at block level based decision making Tensions or areas of misalignment between system components
Findings Management & Decision Making Coverage 57% Coverage 27% CHAMPARANWEST Coverage 44% GOPALGANJ SIWAN CHAMPARANEAST SITAMARHI SHEOHAR MADHUBANI SUPAUL MUZAFFARPUR DARBHANGA SARAN MADHEPURA SAMASTIPUR SAHARSA VAISHALI ARARIA PURNIA KISHANGANJ BUXAR BHOJPUR PATNA BEGUSARAI KHAGARIA KATIHAR KAIMUR ROHTAS AURANGAABAD ARWAL JEHANABAD GAYA NALANDA MUNGER BHAGALPUR LAKHISARAI SHEIKHPURA JAMUI BANKA NAWADA Coverage 32% Challenges to Decentralization and Improved Coverage District Fully Vaccinated Children (DLHS 3) Staff have Low Decision Making Ability Transparency is important Odds Ratio CI OR CI Kishanganj 27% 1 (basis of comparison) Muzaffarpur 57% 4.2 1.2-14.7 5.4 1.2-24.5 Patna 44% 4.8 1.4-17.4 5.4 1.2-24.5 Gaya 32% 1.7 0.5-5.9 3.8 1.0-15.3
Findings Planning Formal Block-level microplans serve as core planning activity Inputs District funds allocated for microplan formulation History of immunization as centrally-planned and directed program Individual Inconsistent staff participation in development and use of microplans Informal. Reported high priority and positive value for data collection and use Informal pressure against local adaptation Technical Unidirectional flow of data Performance Strong correlation between microplan engagement and performance of other duties Tensions or areas of misalignment between system components
Findings Planning Staff Engagement in Microplanning Processes Input area Create or update household lists (According to ASHA/AWW) Percent indicating that they engage in specified activities ASHA & ANM LHV/MO AWW 3% 17% 3% Identify pregnant women 23% FN 50% 5% Identify newborn children Identify/note uncooperative households Track missing women/children Follow up beneficiaries for full immunization Develop immunization roster/calendar Provide inputs into microplan Page 22 19% FN 46% 10% Not asked processes and frequency measured 13% 2.5% 17% 2.5% 19% 7.5% N/A 38% 58% 48% Microplanning Staff Engagement ANMs engaged in planning immunization rosters had 3 times higher likelihood of conducting number of scheduled sessions and 4 times higher likelihood of receiving supervision. Highly significant association between supervisor engagement in microplanning activities and supervisory coverage rates
Findings Service Provision Formal Universal coverage policy goal Policies for expanded outreach services Inputs Financial incentives to ASHAs & ANMs based on coverage Increased budgets for outreach services, safety supplies and equipment Individual Weak link between incentives and individual behavior Informal Safety dominant priority of staff Coverage least prioritized Technical Limited availability and use of safety supplies Performance Notable gaps in safety procedures Significant variation in levels of session coverage Tensions or areas of misalignment between system components
Findings Service Provision Priorities & Incentives Immunization Areas Considered a Priority Position Immunization Activity or Performance Area Coverage Quality Safety Beneficiary s Awareness ANM 21% 31% 67% 65% LHV 25% 42% 65% 48% MOIC 0% 6% 6% 29% 52% of ASHAs responded that they received cash rewards for meeting their targets 54% of ANMs reported receiving cash rewards. No significant correlation between the receipt of rewards and District Odds of citing maintaining/achieving coverage as a priority for immunization services Proportion of scheduled sessions held Completion of due lists for future sessions, Marginally significant relationship between ANMs citing cash rewards and the recording of vaccine administration (OR 2.7 p=.09) Page 24
Findings Record Keeping & Reporting Formal High attention to data collection and reporting Inputs State data center developed Training funds available and utilized Individual Mixed relationship between self efficacy and tested abilities Training had positive effect on reporting accuracy Informal High agreement with value of data collection Technical Consistent availability of personnel but inconsistent availability of equipment High demands on staff for reporting Performance Inconsistent and inaccurate reporting at session sites and PHCs and reporting Tensions or areas of misalignment between system components
Findings Reporting & Record Keeping Reported use of Immunization Data Cited Use of Data ASHA AWW Clinical Get appreciated and 11% 2% received incentive Information on how many 13% 47% vaccinated according to targets Easy to call beneficiary for 6% 3% Immunization Benefit of Immunization 7% 3% and preventing Ensure no child is left out 17% 17% and get information on drop outs Work improved in 3% 12% immunization program To know about 13% 1% beneficiaries Knowledge of next session 7% 6% and immunization due Other 2% 7% Don't Know 21% 4% Inconsistent & inaccurate reporting 54% of the ANMs correctly recorded vaccines administered during session 39% of ANMs noted dropouts for follow up via due lists. 18% of block PHCs had 50% or fewer health sub-centers submitting reports 76% of block PHCs had less than 65% health sub-centers submitting reports Average difference between PHC registers and computerized reports -30%
Findings Community Mobilization Formal ASHAs and VHSCs established for mobilization activities Inputs Incentives paid for mothers group meetings and coverage Individual AWWs showed greater engagement than ASHAs Coordination between CHWs inconsistent Technical Blurred responsibilities between ASHAs and AWWs Performance All frontline workers inconsistent in mobilization activities Informal Identified as key activities and factor for low coverage Tensions or areas of misalignment between system components Page 27
Findings Community Mobilization Activities and responses by Community Health Workers Session-based activities ASHA & AWWs with household list available/present ASHA/AWW brought household contact list based on due list Percentage of mobilizers undertaking activities 31% 33% Shared due list with ANMs 40% Checked how many women from contact list attended session ASHA/AWW requested attending women to remind their neighbors to go for vaccination especially those on list 25% 38% Question Household list prepared according to the policy guideline Does the households list mark households having pregnant women Does the households list mark households having newly born (<32 days) Does the household list mark which households have children under two year olds Did you receive a due list from ANM after last immunization session ASHA Percentage reporting yes AWW 24% 55% 6% 27% 4% 25% 2% 20% 13% 35% Page 28
Discussion Prevalent know-do gap Prevalence of consistent values and priorities expressed both through explicit responses to questions and participant concerns Gap between values and performance highlights difficulty of linking incentives and structures with culture Decision making, planning and management Engagement with microplanning process has significant potential to improve engagement Microplanning process, through reliance on centrally directed schedules and plans, conflicts with goal of decentralization Organizational norms exerting strong pressure and limiting use of available resources
Discussion Community engagement Sporadic and limited engagement with educating and engaging community Misalignment of incentives and community engagement activities Disincentive for coordination between community health workers (AWWs and ASHAs) Incentives not well aligned with community engagement duties Limited engagement of clinical staff with community Data collection and reporting Significant investment of resources yet conducted as perfunctory process Lack of integration of forms, processes, and use of data Page 30
Assessing the Model and the Approach Conceptual challenge of measuring/evaluating alignment across multiple domains Challenge of capturing the complexity of immunization system Multiple causal pathways between systems misalignment and systems performance Consistent and pervasive response bias - dominance of formal organizational values and culture Challenge of capturing limited documentation o f formal organizational structures, policies and resources Page 31
Conclusions Bihar continues to make significant progress in improving the performance of its routine immunization program Changing the performance of systems as complex as immunization requires the consideration of numerous components, including Resources within the system Policy and formal structures within the system The behavior of the system s participants Informal culture within the systems Technical needs, capacity, and interventions There is a role for refined tools to examine the coherence and functioning of systems Page 32
Acknowledgements The Bill & Melinda Gates Foundation The State Health Society Bihar PATH Study Team: Dr. W. Scott Gordon Dr. Pritu Dhalaria Dr. Lysander Menezes Dr. Anwer Aqil Dr. Dai Hozumi Dr. Shalini Khare PATH s Bhopal and New Delhi staff PATH s field investigators The health workers in Kishanganj, Muzaffarpur, Patna, and Gaya The communities that allowed us to observe their services
W. Scott Gordon sgordon@path.org Page 34