Producer Demonstration Sites (PDS) Monitoring, Evaluation and Reporting (MER) Guidelines

Transcription

1 Prducer Demnstratin Sites (PDS) Mnitring, Evaluatin and Reprting (MER) Guidelines I. What is MER? Mnitring, Evaluatin and Reprting (MER) has been clearly summarised in the fllwing way 1 : The prcess f Mnitring, Evaluatin and Reprting (MER) is a key part f prject management, accuntability and reprting n the impact f the research, develpment, extensin and adptin and, f curse, practice change. MLA has develped a Mnitring and Evaluatin Framewrk t guide the cllectin and reprting f data frm PDS prjects s as t infrm and assist MLA in assessing the value f its investments, t mre effectively reprt utcmes and benefits back t its stakehlders, and t cntinually imprve the management f the PDS prjects. This brief guide is directed twards prject leaders wh are verseeing/managing MLA funded PDS prjects. It has resulted frm extensive trialling f the cncepts in MLA and elsewhere and will cntinue t be mdified based n new needs and experience gained thrugh its implementatin. II. Why is MER imprtant? As nted abve, unless yu have a very clear MER plan t guide the cllectin and reprting f data frm PDS prjects, and that plan is successfully implemented, it will be very difficult t determine whether the prject has been beneficial; whether peple directly and indirectly invlved have btained value frm it; whether further benefits may accrue t the industry ver time; and generally whether it has been a gd investment by MLA n behalf f its levy payers (and thus t be able t reprt utcmes and benefits back t its stakehlders). III. Key factrs f MER The MER framewrk develped by QualDATA fr MLA is based n sme simple yet imprtant factrs/questins which seek t btain the necessary infrmatin t allw the prject t be easily, yet cmprehensively evaluated. Evaluatin requires an understanding f prcesses used, prducer engagement that ccurred, practice change bserved / measured and impact. These key factrs are: a. What did we d? Simply describe all the inputs t and utputs frm the prject e.g. Prject prcesses plans and steering cmmittee ntes $ invested and frm where they came Number f participants direct (cre participants - invlved in demnstratin sites) and indirect (bserver - part f a brader grup r attending field days etc) 1 Jeff Cutts and Grdn Stne f QualDATA 1

2 Trial /demnstratin data btained t demnstrate what we did Prducts and infrmatin dcuments prduced and cmmunicated b. Hw well did we d it? Measure whether anyne has changed their knwledge and awareness abut the issue r their skills t influence it: Surveys f participants (thse wh are directly invlved (cre participants) and thse that are indirectly invlved (bservers)) befre and after the prject/event t assess changes t Knwledge, Attitudes, Skills and perceived value in relatin t the slutin that is being demnstrated fr prducer cnsideratin and pssible adptin. c. Has it changed what peple d (have they adpted different practices)? By participating in the prject (r bserving it) have peple changed what they are ding? Have peple made specific changes (adpted new practices / technlgies) as a result f the prject? If changes were made, what was the adptin scale (i.e. whle farm/business, partial)? Survey f cre participants t benchmark the targeted practices and perfrmance metrics befre and after the demnstratin. The purpse f the survey is t enable quantitative demnstratin f practice change and imprved perfrmance utcmes. Will peple be mre likely t change practices in the future (intentins r aspiratins)? d. Is anyne better ff? Are there any key lessns/learnings fr ther prjects? Have peple actually benefitted frm the prject and by hw much? What are the csts and benefits frm making these changes fr individuals? Are mre peple likely t benefit in the future (cre and bserver participants)? What have we learnt that we expected? What have we learnt that we didn t expect? Are there any lessns fr thers/prjects? f. Is the industry better ff? Hw might the brader industry benefit frm the prject? Wh else might the practice change apply t (e.g. wuld thers in the regin be likely t adpt it?) Has this been cmmunicated? 2

3 IV. MER step by step T develp a mnitring, evaluatin and reprting (MER) plan, the fllwing six simple stages are prpsed: a. What is yur bjective r Key Result Area (KRA) T start, yu will need t define what it is yu plan t achieve within the life f yur prject. Describe clearly what the Objective r Key Result Area (KRA) f the prject is seeking t achieve. Think abut the bjective frm the perspective f changing a practice in yur enterprise. Such bjectives shuld be described in SMART terms S Specific M Measurable A Achievable R Relevant T Time-based An example may be: By (prject end), the prject will have demnstrated (utputs) new packages and calculatr tls with (demgraphics) 20 prducers t better link (practices) pasture dry matter measurements t decisins n ptimum stcking rate t imprve (utcmes) the prductin f (species) beef per ha (indicative 15% increase) applicable t 50% f the beef industry in state/area. 3

4 b. What business driver(s) (utcme) are yu fcussing n in the prject? Identify (refer t the applicatin frm) precisely what business driver(s) yur prject will seek t address. Sme examples (key MLA metrics) may include: Prductivity (select at least ne metric) Prfitability (select at least ne metric) Perfrmance Metrics Prductin efficiency (kg LWT / ha) r (kg LWT / ha /100mm rainfall) Animal prductin efficiency (kg LWT /DSE r AE r LSU) r (kg LWT / DSE / 100mm rainfall) Pasture prductivity (kg DM/ ha) r (kg DM / ha / 100mm rainfall) Stcking rate (DSE r AE r LSU / ha) Reprductive efficiency (kid, lamb r calf survival % r marking % r weaning %) Labur efficiency (DSE r LSU r AE / FTE)) Mrtality rate (%) Other, please list.. Whle farm Return n capital (%) indicatrs Operating prfit ($) Enterprise Indicatrs Cst f Prductin ($/ kg LWT) Grss Margin ($ / ha) Grss Margin ($ / DSE r LSU r AE) Other, please list.. Envirnmental Grund cver (%) Whle farm bidiversity Tnnes / ha reductin in sil lss Other, please list c. What metrics will yu capture t demnstrate success? Identify what metrics (measurements) may be available frm yur prject. These will need t be measured befre yur prject cmmences (e.g. baseline) and at the end s as t assess change. Examples include: Number f prducers invlved in demnstratin sites (mandatry) Number f prducers bserving demnstratin sites (mandatry) Number f head f livestck invlved (mandatry) Area (ha) invlved (mandatry) Prject steering cmmittee decisins and ntes Csts f inputs fr prject (inputs, labur) (mandatry) Outcmes frm demnstratin sites (e.g. reprductin rate, weaning rate) (mandatry) Benefits frm utcmes (e.g. $ value f increased weight gain) (mandatry) Knwledge/attitudes/skills f cre and bserver participants befre and after prject (mandatry) Prducer practice (relevant t the tpic/prject) befre and after prject (mandatry) 4

5 Measure f ecnmic and prductivity perfrmance metrics befre and after the prject (mandatry) Field days held examples f engagement Media events/utputs Frecasted ptential impacts well after the prject (e.g. 12 mnths after cmpletin f the prject) d. Hw will yu capture/measure these metrics? Identify what measurement systems r appraches yu will emply t capture the infrmatin required fr the MER e.g. Recrds f inputs Steering cmmittee ntes regarding decisins Narratives frm prducers directly invlved in the prject and specific case studies n the value r impact frm their invlvement in the PDS Surveys pre and pst the prject fr cre participants and bservers (mandatry) Simple benefit cst analyses Media mnitring Surveys sme cnsiderable time after the prject (including secndary impact examinatin (e.g. prducers wh have changed practices the fllwing year r the amunt f a particular prduct used 2 years after the trial). Such surveys are the respnsibility f MLA, nt the prject. e. Cmpilatin and reprting Ensure yu keep a gd recrd f all the infrmatin yu are capturing and include it in Milestne reprts at every pprtunity. The MLA reprting templates will ensure cnsistency f data reprting. f. Keep it simple! The art t a gd MER is keeping everything as simple as pssible. This will ensure yu capture the infrmatin yu abslutely need at minimum cst and d s in a frm that can allw clear evaluatin f the prject and reprting f its utcmes t stakehlders. 5

6 V. An example MER plan The fllwing is a simple wrked example f a generalised MER fr a PDS prject. It is prvided fr guidance purpses nly. KRA: By June 2017, in 10 extensively managed cattle enterprises in nrthern Australia, demnstrate and quantify the value f a single clstridial vaccinatin f either 5in1 r 7in1 vaccine t reduced mrtality in yung cattle Key business driver (metric being examined): Mrtality rate as measured between marking and weaning Evaluatin level [1] Generic Perfrmance Measures Prject Perfrmance Measures (Please fill in and delete example) Inputs What did Number f cre prducers invlved 10 n-farm demnstratins sites we d? in demnstratin sites & their representing 10,000 head f cattle Describe the planned demgraphics 50 bservers cvering 40,000 head f cattle and expected inputs Number f prducers bserving Funds: $25k p.a. frm MLA used fr invlved in yur demnstratin sites & their prfessinal fees, travel and field days prject, including demgraphics Funds: $50kp.a. in kind cntributed t funds, resurces, Number f head f livestck vaccines and prfessinal time develpment & invlved Prject manager appinted prjects structures Area (ha) invlved Steering cmmittee appinted and meeting Prject steering cmmittee twice a year decisins and ntes Investments ($ s) frm MLA and ther parties (cash and in-kind cntributins) and what was purchased prfessinal time, prject inputs Evaluatin Methds (Please fill in and delete example) Gd recrds f all prject plans and activities Prject steering cmmittee ntes [1] Nte: The headings in clumn 1 are als listed in the PDS Final Reprt template. 6

7 Outputs - What did we d? Describe the utputs planned/expected frm yur prject, including engagement activities & prducts frm demnstratin sites Outputs frm demnstratin sites (new knwledge & data) (e.g. reprductin rate, weaning rate, mrtality rate, gender, management methds, cst f vaccine, extra labur and cst f prductin) Field days held, demgraphics cllected, and M&E cnducted Media events/utputs New knwledge & data frm the 10 demnstratin sites Annual Field day targeting 50 prducers representing 40,000 head f cattle New infrmatin package develped n the value f vaccinatin Extensin and cmmunicatin activities e.g. 3 field days held 5 media releases Data frm demnstratin sites in milestne reprts Cmpilatin f media activities Cpies f infrmatin package develped Changes in knwledge, attitudes and skills - Hw well did we d it? Describe the changes in KASA that yu are planning t achieve. Change in knwledge/attitudes/skills f cre and bserver participants befre and after prject/activity Experience f prducers invlved in the PDS extent t which they fund the prject/ activity useful r f value. What was mst helpful in supprting capacity change? X% f cre prducers have greater knwledge f the value f vaccinatins and ther animal management practices Y% f cre prducers have increased their skills and cnfidence in animal husbandry practices Y% f bserver prducers have greater knwledge f the value f vaccinatins and ther animal management practices Key findings Narratives and Case Studies frm peple invlved in the PDS 2 Pre prject surveys (baseline) and pst prject survey Pst event survey/feedback sheets (e.g. field day) that assess changes Practice changes Has it changed what peple d? Describe the practice changes that yu are expecting t achieve by the end f yur prject Prducer (cre & bserver) practice (relevant t the tpic/prject) befre and after prject The extent f practice change adptin (# f cattle) and where Influence the prject had n practice change achieved 10 cre participating prducers representing 10,000 head, adpt single sht clstridial vaccinatin 50 additinal prducers (ttalling 40,000 head) intend t adpt single sht clstridial vaccinatin as a result f interacting thrugh the PDS via field days Baseline surveys (practice change and impact) as abve 2 Narratives and case studies will be captured using a simple prfrma/template (see PDS Narratives and Case Studies Guidelines). They will gather data and experiences frm the peple directly r clsely invlved in the PDS. This infrmatin will help t further define the value r impact frm their invlvement in the PDS. 7

8 Benefits Is anyne better ff? Describe the benefits that yu are expecting t achieve as a result f the prject Benefits frm utcmes (e.g. $ value f decreased mrtality rate cmpared t baseline) Csts t achieve utcmes (e.g. increased inputs, labur) Benefit Cst and Sensitivity analyses at the business level What are the unintended/unexpected benefits r cnsequences? Prject learnings, barriers / enablers t adptin 10 cre participating prducers representing 10,000 head, adpt single sht clstridial vaccinatin resulting in a reduced weaner mrtality f 3% Enterprise prductivity imprves by x% Imprved understanding f what the main barriers and enablers t adptin f these techniques may be Data frm demnstratin sites Benefit Cst Analysis (BCA) at enterprise level Lnger term surveys and data capture n impacts (e.g. reseller infrmatin) General bservatins / utcmes Is the industry better ff? Ptential impacts (practice change & prductivity) at the end f the prject and well after the prject has cncluded (e.g. 2 years later) fr the brader target audience BCA f brader industry impact (prductivity, prfitability, envirnmental & scial) Single sht vaccinatin adpted by x prducers by 20xx Single sht vaccinatin is relevant t x% f industry, and if adpted by the target audience has the ptential t deliver industry benefits f $xm p.a. and reduce industry mrtality rates by 3% This prject will assist MLA in reducing the cst f endemic disease and imprve animal welfare Surveys f key persnnel at the cmpletin f prject, and in ne r tw years time Extraplatin f BCA results t relevant part f the industry 8

9 VI. Chrnlgy f PDS data cllectin This flw chart prvides a clear time-based illustratin f key timings fr data-cllectin during a prject. Prject agreement signed Pre-dem Survey Pst-dem Survey Fllw-up Survey PDS Demnstratin Prject Recrds Trial Data, Narratives and Case Studies Field days Media cverage Pre-field day survey Pst-field day survey Ntes: Pre-demnstratin surveys wuld be t establish baseline measures f knwledge, skill, attitudes and practices in relatin t the issue in questin e.g. D prducers already vaccinate and if s why and if nt why nt?; What is the biggest impediment t changing practices? Pst-demnstratin surveys will assess if the trial has made any difference t knwledge, skill, attitudes and practices in relatin t the issue in questin. Similar t pre- and pst- field day surveys Fllw-up surveys wuld ccur at least 6 mnths (mst likely 12 t 24 mnths) after cmpletin f the PDS and wuld help determine the changes made n-farms as a result f participating t the PDS and their benefits. The surveys may be frmal r frm 3rd parties fr example sales f vaccines. 9