Mnitring and Evaluatin Prf. Jay Arnsn
Mnitring Mnitring Gal: assess prgress f prject/prgram at regular intervals; find ut what s ging n (budget, persnnel, deadlines, prgress, areas needing mre wrk, etc.) Internal prcess cnducted by peple cnducting interventin Iterative Allws fr mid-prject crrectins and mdificatins and gives early indicatin f success r failure Imprves day-t-day decisin making
Evaluatin Evaluatin Identifying and reflecting upn the effects f what has been dne, and judging the value f it. Assesses relevance, effectiveness, efficiency, impact, sustainability, lessns learned, etc. Can be dne either by peple carrying ut prject (internal evaluatin) r by external (presumably independent) evaluatrs Increased recgnitin that participatry evaluatins tend t be the mst accurate and useful Depends n deciding upn desired impacts befre prject gets ging Often required by funders fr additinal mney r future pprtunities t wrk with them (qutatin frm Taylr 2001, p. 1)
Planning Planning Planning Bth mnitring and evaluatin required advanced planning and need t be written int prjects FROM THE VERY START! Wh (interventin grups and cntrl grups) What (will be mnitred/evaluated: inputs, utputs, gals, metrics fr change, etc.) Hw (methds and cllectin f data) When (baseline data, varius pints in midst f prject, at the end)
Data cllectin I What is yur level f analysis? individuals, families, cmmunities, villages, cities, kisks, business centers, etc. gender, age, ethnic grups, lcatin sampling techniques (randm sampling vs. purpsive sampling) Sample size and statistical significance
Data cllectin 2 Need t make sure that peple in yur cntrl grup and interventin grup are equivalent Peple wh participate in technlgy prjects are ften self-selecting, s yu can t just use peple wh d NOT participate as the cntrl grup. Randmizatin is the gld standard, but nt likely t happen in the kinds f prjects yu are undertaking See MIT s Pverty Actin Lab (www.pvertyactinlab.rg) fr mre n this subject
Fur Types f Evaluatins Simple Pre-Pst (Type I) Pre-Pst with Cntrl Grups (Type II) Allws assessment f extent t which change bserved can be attributed t prject Pre-Pst with Treatment f Determinants and Knwn Cnfunding factrs (Type III) Pre-Pst with Cntrl Grups and Treatment f Determinants and Knwn Cnfunding factrs (Type IV)
Methds Structured questinnaires Direct bservatin by researchers Clsed questin surveys Ecnmic impact assessments (audits) Semi-structured individual interviews Fcus grups Participatry methds such as public frums, invitatins t the cmmunity t submit cmments, etc.
Things t keep in mind Data frm varius mnitring perids must be cmparable yu have t cllect it the exact same way each time Gd mnitring and evaluatin plans incrprate quantitative and qualitative methds Yu shuld plan t share results with the cmmunity
Resurces M&E can require significant human and material resurces If yu re undertaking an evaluatin f a prject that affects several villages, yu may need t lcate and train 20+ data cllectrs, several data entry specialists, mre than ne persn wh can analyze the data prperly, plus smene t manage these peple and interact with ther members f the prject team Mst f these peple will require sme degree f training Frm a material standpint, yu may need t design and validate a survey, arrange fr transprtatin fr data cllectrs (which in the absence f gd public transprtatin systems can be difficult), gain access t ffice space, cmputers and basic supplies The lcal rganizatins that yu re wrking with may be able t prvide yu with many f these resurces
Cnceptual Framewrk Hw t start yur M&E prcess (befre yu begin implementatin): 1. Write descriptin f planned steps (in the rder that yu will carry them ut) 2. Nte assumptins that yu are making (cause/effect; cst/benefit; wh will benefit; what will happen; etc.) 3. Identify indicatrs f the prgress yu expect, s that yu can begin cnsidering hw t mnitr and evaluate them
Cnceptual Framewrk ULTIMATE GOAL (BENEFITS) IMPACTS OUTPUTS WORK/ACTIVITIES INPUTS Ecnmic and Scial Develpment and Pverty Alleviatin (everybdy benefits, especially the prest f the pr) Imprved Cmmunicatin, Access t Infrmatin and Markets Functin Cell Netwrk Installatin, Training, Capacity Building Cell phnes, Infrastructure, Villagers, Cmpanies, Teachers, NGO $$, etc.
Assumptins and Metrics Fur Questins: 1. What are sme assumptins in my very crude framewrk? 2. What needs t be measured in rder t d mnitring and evaluatin? 3. When d I need t start measuring? 4. What will my cntrl grup lk like (if I include ne)?
A Few Traps in M&E On-the-grund realities (muddy rads, scial stratificatin, lcal skepticism, etc.) Incmplete mdels Inadequate BASELINE data cllected Failure t cllect adequate data n ptential cnfunders and effect mdifiers (externalities that might affect the utcme f yur prject either psitively r negatively) Bias frm internal M&E N plan t incrprate mnitring results int prject as it is taking shape N lcal use f data gathered during M&E
The Plitics f M&E Like just abut everything else, M&E is inherently plitical Wh gets t define the indicatrs, methds, and sampling techniques (can be mitigated t a certain extent by making M&E participatry) M&E is ften dne in the cntext f btaining funding r reprting back t funders Nbdy wants t be part f a failed prject, s there may be a tendency t glss ver negative aspects M&E des nt autmatically aid cmmunity members just like technlgy, it needs t be designed with peples needs and desires in mind
Suggestins Lk arund fr prjects similar t the ne yu are ding and examine M&E plans There may be standardized instruments available fr use Ask funders fr infrmatin abut what kinds f M&E they require, as well as examples f gd evaluatins they have received Fr mre resurces, see: Wrld Bank s: Mnitring and Evaluatin: Sme Tls, Methds, and Appraches Nte: web address is very lng, s use a search engine t find it
An exercise Get int yur simulatin grups and: 1. Create a cnceptual framewrk fr yur prject 2. Nte assumptins made in framewrk 3. Cme up with mst relevant indicatrs 4. Chse a type f evaluatin frm the fur just presented, and defend yur chice
Reminders Dn t frget t analyze distributin f benefits, especially amng pr, wmen, and ther marginalized grups Dn t frget t evaluate sustainability and exit strategies these areas are in great need f imprvement in technlgy-based develpment