Assessing Advocacy? Yes it is Possible!

Assessing Advocacy? Yes it is Possible! Rachel Callanan JD, MNM Reid Zimmerman PhD, CFRE October 2011 Callanan / Zimmerman 1

Warm-Up October 2011 Callanan / Zimmerman 2

Is it Really Possible? True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information. Winston Churchill Why go through all this trouble? Are some of you willing to share your answer to why your organization conducts or is contemplating advocacy evaluation? October 2011 Callanan / Zimmerman 3

For what purpose/s do you evaluate your public policy advocacy work? (check all that apply) Evaluators Answer Options Response Percent Response Count To measure staff performance. 50.0% 11 Reporting outcomes to funders. 77.3% 17 Reporting outcomes to constituency or stakeholders. 77.3% 17 Internal monitoring of progress towards goals. 86.4% 19 Reporting progress towards goals to funders. 63.6% 14 Reporting progress towards goals to constituency or stakeholders. 59.1% 13 To learn from the information and adjust strategy/tactics. 90.9% 20 Use what you ve learned to help other organizations improve their practices. 31.8% 7 Other 0.0% 0 October 2011 Callanan / Zimmerman 4

For what purpose/s would you evaluate your public policy advocacy work? (Check all that apply) NON-Evaluators Answer Options Response Percent Response Count To measure staff performance. 50.0% 6 Reporting outcomes to funders. 41.7% 5 Reporting outcomes to constituency or stakeholders. 58.3% 7 Internal monitoring of progress towards goals. 66.7% 8 Reporting progress towards goals to funders. 41.7% 5 Reporting progress towards goals to constituency or stakeholders. 50.0% 6 To learn from the information and adjust strategy/tactics. 91.7% 11 Use what you ve learned to help other organizations improve their practices. 41.7% 5 Other 0.0% 0 October 2011 Callanan / Zimmerman 5

Causal Diagram of Beckhard s Change Formula Pushing Force (Non-directional) Dissatisfaction with the Status Quo (Inertia) a.k.a. Cost of Change Resistance to Change Pulling Force (Directional) a.k.a. Desirability of the end state Compelling Vision Believability First Steps Beckhard, R. & Harris, R.T. (1987). Organizational transitions: Managing complex change (2 nd ed.). Addison-Wesley: Reading, PA Beckhard, R. & Pritchard, W. (1992) Changing the essence: The art of creating and leading fundamental change in organizations. Jossey-Bass: San Fancisco

Traditional Approaches to Evaluation Retrospective: looking back after project complete Summative: goal of the evaluation is to make a judgment about program s worth, effectiveness or impact Focus on outcomes & impact, rather than improvement or learning Scientific method has dominated the field of evaluation October 2011 Callanan / Zimmerman 7

Discussion question What are some of the unique challenges for advocacy evaluation? October 2011 Callanan / Zimmerman 8

Unique Challenges for Advocacy Evaluation 1) Complexity and iterative nature of advocacy and policy change (Guthrie et al., 2005) 2) Role of external forces (Guthrie et al., 2005) 3) The time frame (Guthrie et al., 2005 4) Shifting strategies (Guthrie et al., 2005) 5) Little experience or capacity for evaluation (Guthrie et al., 2005 6) Differing expectations or value for evaluation between funders and advocates (Guthrie et al., 2005) 7) Defensive outcomes (Reisman et al., 2007; Egbert and Hoechstetter, 2009) 8) Differing needs, philosophies, and requirements among foundations (Reisman et al., 2007). October 2011 Callanan / Zimmerman 9

Logic Model Overview INPUTS ACTIVITIES OUTPUTS OUTCOMES October 2011 Callanan / Zimmerman 10

A few words about logic models May be simple or complex Helps your organization fit your campaign strategy, tactics and evaluation into a whole picture your theory of change. October 2011 Callanan / Zimmerman 11

More Comprehensive Logic Model Inputs Activities Outputs Participation Outcomes Short Medium Long Assumptions External Factors University of Wisconsin-Extension, Program Development and Evaluation October 2011 Callanan / Zimmerman 12

Advocacy and Policy Change Composite Logic Model Inputs Activities/ Tactics Interim Outcomes Policy Goals Impacts Capacity Building Prep/ Planning Policy & Politics Communications & Outreach Advocacy Capacity Policy What the advocacy strategy is trying to achieve in policy arena Big changes & benefits sought as a result of policy change or goal Contextual Factors Audiences 2009 President and Fellows of Harvard College. Adapted and reproduced with permission from Harvard Family Research Project (http://www.hfrp.org)

Agreement Far from Close to Simple Plan, Control Certainty Far from October 2011 Callanan / Zimmerman 15

Agreement Far from Simple Plan, Control Technically Complicated Close to Certainty Far from October 2011 Callanan / Zimmerman 16

Agreement Far from Socially Complicated Simple Plan, Control Technically Complicated Close to Certainty Far from October 2011 Callanan / Zimmerman 17

Agreement Far from Socially Complicated COMPLEXITY Simple Plan, Control Technically Complicated Close to Certainty Far from October 2011 Callanan / Zimmerman 18

Agreement Far from Socially Complicated COMPLEXITY Simple Plan, Control Technically Complicated Close to Certainty Far from October 2011 Callanan / Zimmerman 19

Theoretical Grounding has Addressed Some Challenges 1) Both capacity and advocacy strategies need to be measured (Alliance for Justice, 2004) 2) Prospective evaluation is important-- measuring progress towards goals is important to capture and measure (Guthrie, et al., 2005, 2006) October 2011 Callanan / Zimmerman 20

Patton s theory of Developmental Evaluation is applicable to advocacy evaluation 1) Developmental Evaluation accepts that progress toward a goal may be the only measure of success in some contexts. 2) Focus on internal accountability rather than external. 3) Distinguished from formative evaluation because not aspiring to summative evaluation. (Michael Q Patton, 1994, 2006, 2011)

Seven Principles for Policy Change Evaluation 1. Expand the perception of policy work beyond sate and federal legislative arenas to include administrative and regulatory advocacy, as well as within the judicial branch, and local and regional advocacy. 2. Build an evaluation framework around a theory about how a group s activities are expected to lead to its long-term outcomes (theory of change). 3. Focus monitoring and impact assessment for most grantees and initiatives on the steps that lay the groundwork and contribute to the policy change being sought rather than the final outcome of passing the legislation. October 2011 Callanan / Zimmerman 22

Seven Principles Continued 4. Include outcomes that involve building grantee capacity to become more effective advocates including interim progress measures and progress that establishes their capacity to impact future policy efforts. 5. Focus on the foundation s and the grantee s contribution, not attribution because it can be almost impossible to determine absolute attribution given the range of factors that may have influenced an outcome. 6. Emphasize organizational learning as the overarching goal of evaluation for both the grantee and the foundation with a focus on strategies that support learning rather than judgment. 7. Build grantee capacity to conduct self-evaluation including resources, training, and technical assistance (Guthrie et al., 2005). October 2011 Callanan / Zimmerman 23

Methods for Evaluating Advocacy Scientific proof not required for program evaluation in this context However evaluation principles still apply and many methods are the same Methods chosen will depend on answers to key questions October 2011 Callanan / Zimmerman 24

Methods for Evaluating Advocacy Common methods: stakeholder surveys or interviews to gather feedback case studies detailing and analyzing a campaign focus groups with stakeholders/advocates media tracking of incidence of subject coverage in media media content or framing analysis--more in depth than tracking participant observation--evaluator attends meetings policy tracking--progress in the policy process public polling (Coffman and Reed, 2009). October 2011 Callanan / Zimmerman 25

Methods for Evaluating Advocacy Unique methods: bellwether methodology: structured interviews of influential people to gauge where an issue is in the broader landscape policymaker ratings: assess the level of support for and influence on an issue intense period debriefs: debrief through focus groups or individual interviews to evaluate campaign system mapping: visual mapping of system done to better understand dynamics and changes in system (Coffman and Reed, 2009). Grassroots actions October 2011 Callanan / Zimmerman 26

Case Studies Exercise October 2011 Callanan / Zimmerman 27

Resources: Advocacy Progress Planner: http://planning.continuousprogress.org/ A User s Guide to Advocacy Evaluation Planning: http://www.hfrp.org/evaluation/publications-resources/auser-s-guide-to-advocacy-evaluation-planning Unique Methods in Advocacy Evaluation: http://www.calendow.org/uploadedfiles/evaluation/coff man%20reed%20unique%20methods%20(paper).pdf October 2011 Callanan / Zimmerman 28

Citations Alliance for Justice (2004). Investing in Change: A Funder s Guide to Supporting Advocacy. Washington, DC: Alliance for Justice. Coffman, J. (2009). A user s guide to advocacy evaluation planning. Cambridge, MA: Harvard Family Research Project. Retrieved from http://www.hfrp.org/evaluation/publications-resources/a-user-s-guide-to-advocacy-evaluationplanning. Coffman J. and Reed, E. (2009). Unique methods of advocacy evaluation. Advocacy Evaluation Advances: A National Convening on Advocacy and Policy Change Evaluation, Los Angeles, CA, January 2009. Los Angeles, CA: The California Endowment. Retrieved from http://www.calendow.org/uploadedfiles/evaluation/coffman%20reed%20unique%20methods%20(paper).pdf Egbert, M. and Hoechstetter, S. (2006). Mission possible: evaluating advocacy grants. Foundation News and Commentary 47(1), 38-43. Retrieved from http://www.foundationnews.org/cme/article.cfm?id=3545. Guthrie, K., Louie, J., and Foster, C.C. (2005). The Challenge of Assessing Policy and Advocacy Activities: Strategies for a Prospective Evaluation Approach. Los Angeles, CA: The California Endowment. Retrieved from http://blueprintrd.com/text/challenge_assess.pdf. Guthrie, K., Louie, J., and Foster, C.C. (2006). The Challenge of Assessing Policy and Advocacy Activities: Part II-- Moving from Theory to Practice. Los Angeles, CA: The California Endowment. Retrieved from http://blueprintrd.com/text/06_10_challengeofassessing.pdf. Patton, M.Q. (1994). Developmental evaluation. Evaluation Practice 15(4), 311-320. Patton, M.Q. (2006). Evaluation for the way we work. Nonprofit Quarterly 12(2), 28-33. Patton, M.Q. (2011). Developmental Evaluation. New York: The Guilford Press. Reisman, J., Gienapp, A., and Stachowiak, S. (2007). A Guide to Measuring Advocacy and Policy. Baltimore, MD: Organizational Research Services. Retrieved from http://www.aecf.org/upload/publicationfiles/da3622h5000.pdf. Teles, S. and Schmitt, M. (Summer 2011). The Elusive Craft of Evaluating Advocacy. Stanford Social Innovation Review. Retrieved Sept. 30 from http://www.ssireview.org/articles/entry/the_elusive_craft_of_evaluating_advocacy/. University of Wisconsin Extension, Logic Model. Retrieved Sept. 30, 2011, from University of Wisconsin-Extension- Cooperative Extension, Program Development and Evaluation Unit Web site: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Your Presenters: Rachel Callanan, JD, MNM Regional Vice President of Advocacy for MN & WI American Heart Association 4407 W. 77 th Street Edina, MN 55435, 612-803-1008 Rachel.callanan@heart.org Reid A. Zimmerman, PhD, CFRE RAZimmerman Consulting 4729 550th Street, Pine City, MN 55063 O: 320-358-3583 C:651-295-7892 reidazimmerman@gmail.com October 2011 Callanan / Zimmerman 30