Monitoring, Evaluation and Adaptive Management Following INFFER Assessment (INFFER step 7)

Transcription

1 Mnitring, Evaluatin and Adaptive Management Fllwing INFFER Assessment (INFFER step 7) David Pannell, Geff Park, April Curatl, Anna Rberts, Stephanie Spry, Sally Marsh Intrductin The Investment Framewrk Fr Envirnmental Resurces (INFFER) is a tl fr planning and priritising public investments in natural resurces and the envirnment. It fcuses n achieving utcmes cst effectively. This dcument relates t step 7 f the INFFER prcess (Table 1). It cvers the prcess f mnitring and evaluatin f the implemented prjects, with an emphasis n adaptive management. Table 1. Steps in the INFFER prcess Descriptin f Step Relevant Dcument 1. Develp a list f significant natural assets in the Significant Asset Identificatin Guide relevant regin(s) 2. Apply an initial filter t the asset list, using a simplified set f criteria Filtering Significant Assets Prir t Detailed Assessment 3. Define prjects and cnduct detailed assessments f them Prject Assessment Frm, and Prject Assessment Frm Instructin Manual 4. Select pririty prjects Selectin f Pririty Prjects 5. Develp investment plans r funding prpsals Develpment f investment plans r funding prpsals 6. Implement funded prjects Implementatin f funded prjects 7. Mnitr, evaluate and adaptively manage prjects Mnitring, Evaluatin and Adaptive Management fllwing INFFER Assessment (this dcument) The thinking behind this step Mnitring and evaluatin f envirnmental and natural resurce prjects ften tends t be cnducted primarily fr reasns f reprting and accuntability. While the INFFER prcess

2 Mnitring, Evaluatin and Adaptive Management recgnises the imprtance f reprting, it als emphasises the imprtance f mnitring and evaluatin t learn frm the prject and imprve its delivery f utcmes ver time i.e. adaptive management. Reprting A number f elements f the Prject Assessment Frm prvide infrmatin that assists with reprting n prject prgress and utcmes: 1.1 Asset identificatin: makes clear the natural asset that is the fcus f the prject. This asset shuld be the fcus f mnitring fr utcmes. 2.1 Prject gal(s): identifies the utcmes that are intended t be achieved by the prject. Reprt n prgress twards the gal(s). This gal is lng term. Shrter term intermediate utcmes are prvided at Wrks and actins: specifies the wrks and n-grund actins that the prject aims t have implemented. Reprt n prgress twards implementatin f these wrks and actins. 2.6 Psitive and negative spin-ffs frm the prject: reprt n psitive and negative spin-ffs that actually ccur. 3.3 Private adptin f wrks and actins: reprt n actual levels f adptin relative t the predicted levels. Sme thught will be required n hw adptin will be measured. 3.5 Apprvals: reprt n apprvals that have and have nt yet been granted. 4.1 t 4.3 Delivery mechanisms: reprt n the extent t which delivery mechanisms have been implemented. 4.4 Sci-plitical risks: reprt n any impacts n the prject f sci-plitical risks. 4.5 Csts: reprt actual budget expenditure cmpared t planned expenditure. 5.4 Intermediate utcmes: reprt prgress twards the intermediate utcmes specified. Mst imprtantly, if the INFFER guidelines have been fllwed crrectly, the prject gal(s) specified in Q2.1 will be realistic, achievable gals that will be attained as a result f the prject. The delivery mechanisms specified in Q4.1-Q4.3 will lead t the wrks and ngrund actins specified in Q2.2, which will lead t the gal(s) in Q2.1. Thus the prject gals prvide a strng basis fr meaningful reprting. Reprting can be fcused n utcmes, nt just activities and expenditure. Adaptive management Adaptive management is a structured, iterative prcess f decisin making in the face f uncertainty, with the aim f reducing that uncertainty ver time thrugh mnitring the system that is being managed. It shuld be mre than just trial and errr it shuld invlve systematic effrts t learn and imprve. Versin 18, 18 July

3 Mnitring, Evaluatin and Adaptive Management Adaptive management is abut infrmatin: cllecting infrmatin, analysing its significance and implicatins, and respnding t it. Frm the perspective f adaptive management, the key purpse f mnitring a prject is t learn frm the infrmatin cllected s that this prject and subsequent prjects can be imprved. This learning shuld be purpseful and directed twards imprving the prject. INFFER fcuses n the achievement f utcmes we want t see natural assets in better cnditin than they wuld therwise have been. Implementing the prject s delivery mechanisms is nt sufficient they shuld deliver utcmes. INFFER helps managers t mnitr a prject in a way that pays attentin t the achievement f utcmes. In the curse f cmpleting the INFFER Prject Assessment Frm, users are required t prvide infrmatin abut the wrks and actins required t achieve the prject s gal(s), and the likelihd that these wrks and actins will be adpted. Ideally, the infrmatin used is based n empirical evidence and/r cmputer mdels. A useful way t think abut mnitring fr adaptive management is that its rle is t update the evidence and mdels that were used t cnduct the riginal prject assessment. We recmmend that, as the prject prceeds, every year r secnd year, the rganisatin shuld update the Prject Assessment Frm fr this asset. Much f the infrmatin wuld be unchanged, but sme may need t be updated. Surces f infrmatin t update the Prject Assessment Frm culd include: Observatins frm the prject itself. After sme experience with the peratin f the prject, the riginal infrmatin abut the impacts f wrks r the adptin f wrks may lk t ptimistic, r t pessimistic; The results f a feasibility assessment cnducted as the first phase f this prject. Particularly fr large prjects, we recmmend that there shuld be an initial phase f mre detailed feasibility assessment befre prject implementatin cmmences. New research evidence r cmputer mdelling. This culd include research r mdelling cnducted as part f the prject r independent f it. When this further infrmatin is cnsidered, prject managers may wish t respnd in any f a variety f ways, including: Expanding resurcing f the prject; Changing the delivery mechanisms used; Changing the wrks and actins being targeted by delivery mechanisms; Changing the peple being targeted by delivery mechanisms; Investing in further research r mdelling t fill a knwledge gap whse imprtance has becme mre apparent; Withdrawing frm the prject r scaling it dwn. The Cnservatin Measures Partnership (CMP) has develped a set f standards and guidelines fr management f cnservatin prjects 1. The elements in these Standards are sequenced in a similar but slightly different way t INFFER. We suggest that it is mst lgical 1 Cnservatin Measures Partnership (2007). Open Standards fr the Practice f Cnservatin, Versin 18, 18 July

4 Mnitring, Evaluatin and Adaptive Management t refer t the Standards nce yu have identified which assets will be prtected in the prject (i.e. nce yu have reached INFFER step 5 2 ). In the appendix we reprduce their advice abut the adaptive management elements f their Standards. Appendix: Edited extract frm Cnservatin Measures Partnership Open Standards fr the Practice f Cnservatin 4. Analyze, use, adapt This step invlves managing yur data as it cmes in and regularly analyzing it t cnvert it int useful infrmatin. In particular, yu need t analyze yur prject s results and cre assumptins as well as peratinal and financial data and then adapt yur wrk plans as necessary. The amunt f time needed t cmplete this step is ften underestimated by prject managers, leaving them with lts f data that they have cllected but have nt analyzed r used. By making this a deliberate step, yu shuld find it easier t bserve and understand changes, slve prblems, and make imprvements t yur prject. 4A. Prepare yur data fr analysis Analysis is a prcess f transfrming raw data int useful infrmatin. Analysis shuld nt happen at nly ne pint in time during the life f the prject. T cntinuusly understand what is ging n in yur prject and t be able t change things in a timely fashin it is essential t capture and analyze yur mnitring data as part f rutine prject wrk. T d this, yu need t have sund data management systems in place. Yu need t have methds and systems established fr recrding, string, and prcessing data. This includes prcesses fr systematically checking, cleaning, and cding raw data as sn as yu get them and fr string and backing-up yur data. This wrk shuld be dne fr bth prgrammatic data as well as peratinal and financial data. Output fr this standard practice includes: Develpment and regular use f systems fr recrding, string, prcessing and backing up prject data. 4B. Analyze results One f the mst imprtant aspects f adaptive management is that it allws yu t systematically assess whether yu are n track t achieve yur stated gals and bjectives. Yur mnitring data shuld prvide yu with the infrmatin needed t see whether yu have achieved yur expected intermediate results and whether yu are n track t achieve lng-term success. In additin, adaptive management als allws yu t determine why certain activities have succeeded r failed. Yur mnitring data prvide yu with the pprtunity t see whether the cre assumptins yu laid ut in the planning steps [in INFFER step 3] hld true in reality. By testing these cre assumptins, yu are in a better psitin t adapt and change yur prject activities accrdingly. 2 If yu have used INFFER, steps 1 and 2 f the CMP Standard are already dne. Versin 18, 18 July

5 Mnitring, Evaluatin and Adaptive Management Fr learning and effective cmmunicatin, it is imprtant that the right peple be invlved in the analyses and/r made aware f the results f the analyses. As a general rule, analyses shuld invlve members f the prject team. Hwever, input frm utside experts r thse with ther perspectives is valuable during the analyses f yur mnitring results. T check if yu are n track r why smething may have succeeded r failed, yu shuld undertake the fllwing tasks: Cnsider yur results in the cntext f yur cnceptual mdel and results chains [i.e. are the bserved results cnsist with the results that were predicted at this stage f the prcess, based n yur understanding f cause and effect?]; Review yur assumptins and assess if yur assumptins are crrect, if yu are n track t meet yur gals and bjectives and if yur strategies are having the desired impact; Assess the utility f yur indicatrs; and Determine if yur gals and bjectives were set at an apprpriate level and if the timeline fr achieving them was apprpriate. Depending n the type f data that yu have and yur infrmatin needs, these analyses can range frm frmal statistical studies t simple qualitative assessments. It is als imprtant t cnsider whether the peratinal prcesses supprting yur prject are functining prperly. Yu may have a prject that uses the perfect strategies t address the threats and pprtunities affecting yur cnservatin targets, but maybe yur team is nt perating efficiently r it des nt have the administrative r financial supprt it needs t d its jb well. Sme questins yu might want t explre during yur analysis include: T what extent d yu have sufficient resurces (e.g., financial, human, administrative, plitical) t carry ut yur prject? T what extent d yu have the physical infrastructure and equipment (e.g., ffice space, vehicles, cmputers) yu need t d yur jb? T what extent des yur prject team perate smthly r are there areas where yu culd imprve hw the team functins? (e.g., cmmunicatins, delegatin f respnsibilities) Outputs fr this standard practice include: Analyses f prject results and assumptins. Analyses f peratinal and financial data. Dcumentatin f discussins and decisins. 4C. Adapt yur strategic plan Cllecting and analyzing data as part f rutine mnitring activities allw yu t determine hw effective yur interventins are and what yu need t d t adjust yur prject t reach yur gals and bjectives mre efficiently. As the final part f this step, yu need t use what yu have learned during the analyses and discussins t mdify and ptimize yur activities. This is the essence f adaptive management. All the planning that yu did earlier was nt meant t be a ne-time event, never t be revisited r used again. Instead, in rder t learn ver time and t cntinue t imprve the Versin 18, 18 July

6 Mnitring, Evaluatin and Adaptive Management effectiveness f yur prject, yu must revisit and adjust yur prject parameters and cre assumptins, actin plan, mnitring plan, peratinal plan, wrk plan and budget. Therefre, yu may need t update all sectins f yur strategic plan t reflect what yu have learned. As yu make changes, yu shuld als dcument the ratinale behind them s that thers will understand what yu learned and why yu made these changes. When updating yur strategic plan, yu shuld als incrprate findings frm analyses dne utside f yur prject team. Fr example, if yur prject has undergne a frmal evaluatin r audit (see Step 5C [f the Standard]), yu shuld examine the findings and see hw yu can use them t adapt and imprve yur prject and yur strategic plan. Output fr this standard practice includes: Revised prject dcuments (including actin plan, mnitring plan, peratinal plan, wrk plan, and budget). 5. Capture and share learning The final step in the management cycle invlves sharing lessns and frmal prducts with key internal and external audiences. It als invlves giving and receiving feedback, cnducting evaluatins and audits, and prmting a learning culture. In this step, it is imprtant t fster learning nt nly within the prject but als at an institutinal level and, mre generally, within the cnservatin cmmunity. With this in mind, these standards include practices that yur rganizatin shuld adpt at an institutinal level t help fster learning and sharing. 5A. Dcument what yu learn As yu g thrugh the prcess described in these standards, yu shuld make sure yu harvest and dcument the lessns that yur prject team is learning n a regular basis. Lessns can take the frm f frmal statistical results r anecdtal stries and can fcus n smething as large as yur cre prject assumptins r smething as specific as a new and imprved way f tracking prject expenses. One f the keys t harvesting lessns is t keep track f learning questins that emerge as yu g thrugh the prject management prcess and then try t answer these questins when data becme available t d s. Anther key is t prvide time and incentives t d this wrk. Harvesting lessns requires a balance between art and science and will require patience and making time in any wrk plan fr these tasks. Make sure that yu dcument r recrd thse lessns s that they are available in the future t yur team and yur rganizatin. Outputs fr this standard practice include: Dcumentatin f key results and lessns. 5B. Share what yu learn If yu capture what yu have learned in written r recrded dcuments, yu will be able t remember frm year t year what yu have dne, what yu fund wrked and what didn t, and what yu plan t d in the future. This will help yur current prject team ver the lng term and will ensure that new prject staff will have a recrd f what yu did and what yu learned. Prductin f frmal dcuments will als help yu cmmunicate yur findings t practitiners arund the wrld. Dcumenting and sharing what yu have learned will help Versin 18, 18 July

7 Mnitring, Evaluatin and Adaptive Management practitiners wrking under similar cnditins, dealing with similar threats, and using similar tls t benefit frm yur successes and avid any pitfalls r prblems yu may have encuntered during the implementatin f yur prject. In rder t create dcuments that a variety f audiences will understand, internalize and use, yu must understand hw they typically receive messages, and what they wuld be interested in learning. Althugh we present cmmunicatins as the final step in the cycle, yu really need t be preparing fr cmmunicating yur results and ther relevant prject infrmatin much earlier. T effectively reach yur audiences, yu need a clear cmmunicatins and disseminatin strategy. Yu need t decide which lessns yu wish t cmmunicate t these pririty audiences, determine the best frmat t reach each key audience, and then develp and distribute yur cmmunicatins prducts. Fr example, yu may use infrmal cmmunicatins means ( , phne calls) t share lessns with yur internal audiences (the prject staff, partners and ther stakehlders). Yu shuld make sure t prvide: Clear management recmmendatins t all the right peple based n yur analysis; Necessary details t help interpret results; Alternatives and cntingencies based n the results; and Regular reprts t all team members. Fr cmmunicating and sharing lessns with yur external audiences (dnrs, ther practitiners, brader public), yu will prbably use mre frmal cmmunicatins means (reprts, presentatins, vides). Cmmunicatins prducts can encmpass many different frms ranging frm frmal academic papers t stries and vides. It is imprtant t evaluate each prduct t see if it effectively cmmunicated yur messages and t learn what yu might d t imprve similar effrts in the future. Finally, yu shuld als lk t thers in the cnservatin cmmunity as surces f infrmatin and learning fr yur prject. Sme f the best surces f lessns are the experiences f thers. Outputs fr this standard practice include: Identificatin f key audiences. Develpment f a cmmunicatins strategy. Regular reprts r ther types f cmmunicatin t prject team members and key stakehlders. Develpment and distributin f apprpriate cmmunicatin prducts. Use f ther peple s cmmunicatin prducts. 5C. Create a learning envirnment The last standard f practice in the cycle invlves creating a perfrmance and learning culture within yur prject team, acrss yur rganizatin and partners, and amng cnservatin practitiners arund the wrld. A perfrmance and learning culture at these levels is imprtant t ensure that all parties learn and benefit frm yur team s experience. Althugh this is listed as the last step, it really is smething yu need t cultivate right frm the start. T effectively apply these standards, yu need t wrk in a prject envirnment that prmtes learning and adaptatin ver time. This means that yu, yur team, and yur Versin 18, 18 July

8 Mnitring, Evaluatin and Adaptive Management rganizatin shuld be cnstantly reflecting, seeking feedback, and prviding feedback. That feedback culd be frmal r infrmal and might cme internally frm yur team members r ther staff members in yur rganizatin. Alternatively, it might cme frm external mechanisms, such as evaluatins, which assess a prject against its wn stated gals and bjectives, and audits, which assess a prject against an external set f prcess standards, such as the nes utlined in this dcument. In creating a learning envirnment, it is imprtant t be pen t utside pinins that can give yu fresh and insightful perspectives. Creating a learning envirnment is nt easy. It requires leaders and dnrs wh understand the need t reallcate scarce resurces frm immediate actin t the lng-term wrk f adaptive management. It ften requires enabling practitiners t take sme chances and questin the cnventinal wisdm related t specific cnservatin tls and strategies. It requires prviding prject teams with the institutinal security that innvatin and questining assumptins are valued in their rganizatins. And it requires a cmmitment t share bth successes and failures with ther practitiners arund the wrld t create true cmmunities f practice. Outputs fr this standard practice include: Regular feedback shared frmally r infrmally. Evaluatins and/r audits at apprpriate times during the prject cycle. Demnstrated cmmitment frm leaders t learning and innvatin. A safe envirnment fr encuraging experimentatin and questining the status qu. A cmmitment t share success and failures with practitiners arund the wrld. Versin 18, 18 July