SF BRACE Evaluation Framework and Plan

Transcription

1 SF BRACE Evaluation Framework and Plan Table of Contents 1. STAKEHOLDERS a. Involved in the Program Operations b. Served or Affected by the Program c. Primary Users of the Evaluation & Evaluation Interests 2. PROGRAM DESCRIPTION a. Key Definitions b. Overview of BRACE Steps c. SF BRACE Goals d. SF BRACE Outcomes 3. EVALUATION DESIGN a. Purpose of the Evaluation b. SF BRACE Process and Outcome Evaluation Questions for Steps 1-4 c. SF BRACE Short & Long Term Outcome Evaluation Questions 4. CREDIBLE EVIDENCE a. BRACE Step 1 b. BRACE Step 2 c. BRACE Step 3 d. BRACE Step 4 e. BRACE Step 5 5. JUSTIFYING CONCLUSIONS 6. LESSONS LEARNED 7. REFERENCES 8. APPENDICES a. Appendix 1-Scoping Matrix b. Appendix 2-Community Resilience Indicator System Evaluation Plan c. Appendix 3- Short Term Outcome Evaluation Plan

2 1. STAKEHOLDERS Involved in Program Operations o SFDPH, Public Health Preparedness and Emergency Response Branch, EMS SF CARD and The City Administrator of the City and County of San Francisco. o Neighborhood Empowerment Network & Resilient Bayview o California Department of Public Health (CDPH), Served or Affected by the Program o Direct Program Audience: San Francisco service providers, and residents, part of the Neighborhood Empowerment Network and it s Empowered Communities Program, San Francisco City Agencies o Indirect Program Audience: San Francisco Vulnerable Residents and visitors Primary Users of the Evaluation: Who needs the evaluation results? How much time and effort was devoted to stakeholder engagement? o Centers for Disease Control o Program on Health Equity & Sustainability & SF Dept. Of Public Health, both to identify areas of improvement programmatically and to give a baseline idea of community situation or status o Planning Groups: City Agencies planning against climate change: DOE, DOP, DEM, Emergency Preparedness Councils City Admin: Resilient Diamond Heights, Resilient Bayview Stakeholder Group SFDPH Environmental Health Program on Health Equity and Sustainability Emergency Preparedness and Response Branch SF Planning Depts. SF City Admin Office Dept. of Environment Dept. of Planning Dept. of Emergency Mgmt. Emergency Preparedness Councils Community Based Planning Neighborhood Empowerment Network Resilient Bayview Resilient Diamond Heights SF CARD Centers for Disease Control California Department of Public Health 2. PROGRAM DESCRIPTION Evaluation Interests Information for cross-sharing and collaboration How much awareness was increased for community populations The scope for forecasting health impacts What best practices can be learned from this project? What worked/ didn t work How effectively were stakeholders engaged Action-ability of tools develops Incorporation of vulnerable populations incorporated into city adaptation planning What worked/ didn t work Did activities help formulate climate change strategies Evaluation Planning Approaches, Questions, Metrics

3 Key Definitions: Resilience: The ability of groups or communities to cope with external stresses and disturbances as a result of social, political, and environmental change. Capacity Building: Overview of BRACE Steps: BRACE Step 1 1. A: Climate and Health Profile 1. B: Vulnerability Assessment 1. C: Community Resiliency Indicator System BRACE Step 2 2. A: Projected Burden of Disease BRACE Step 3 3. A: Assessing Public Health Interventions 3. B: Creating Public Health Emergency Response Plans 3. C: Developing material and presenting internally to health department BRACE Step 4 4. A: Creation of Neighborhood Level Adaptation Plans 4. B: Implementation of Plans BRACE Step 5 5. A: Evaluation SF BRACE Goals: 1. Promote community resilience through education, empowerment and engagement to reduce vulnerability to climate change. 2. Increase both local level capacity and internal department capacity to utilize climate health science. 3. Incorporate stakeholder engagement in the development of climate change mitigation and adaptation actions. 4. Implement adaptation efforts which achieve health co-benefits and improve health disparities. 5. Serve as a model for local health departments

4 BRACE Outcomes: The outcomes of San Francisco s Climate and Health Program are to work with community based organizations and community members in disadvantaged areas to identify and describe current capabilities to respond and adapt to climate change. With an understanding of the social, environmental and economic conditions that enable these capabilities or create barriers, we will develop a model of community capacity and establish a set of indicators to describe community resilience. With a better understanding of community capacity, we will deliver public health interventions and adaptation plans specific to San Francisco communities that will mitigate increases in climate-related health impacts, help eliminate existing health disparities and develop local adaptive capacity. Short Term Outcome-Increase both local level capacity and internal department capacity to understand climate science and health impacts from Climate Change Intermediate Outcome-Increase public understanding and awareness of the public health impacts from Climate Change Long-Term Outcome-Improve health disparities and achieve health co-benefits through implementation of adaptation plans. Promote community resilience through education, empowerment, and engagement to reduce vulnerability to Climate Change

5 3. Evaluation Design Types of Evaluation Methods: The evaluation methods that will be used will be combination of formative, process, summative and outcome approaches. Formative evaluation uses information gathered through assessments to guide program improvement, process evaluation determines whether the program was delivered as intended, summative evaluation informs whether goals and objectives were met, while outcome evaluation focuses on the organizational or social condition the program is expected to change (Rossi et. al 2004). Since the program timeline spans multiple years with identified goals to be achieved at the end of quarters and years, some findings from both formative and process evaluation strategies could be used to further develop or make modifications to future program activities and to the activities of partnering programs like the Neighborhood Empowerment Network neighborhood deployment initiatives in San Francisco s Bayview and Diamond Heights neighborhoods. Because of the innovative approach that BRACE takes to capacity building and increasing awareness of climate change impacts, documentation and record keeping will be of importance and support more of the formal evaluation goals of the project. Purpose for Evaluation: Were short term process outcomes and project goals met Were long term program goals met To create an evaluation that produces results to be used by diverse stakeholder, including community and city stakeholders that can continue to facilitate adaptation planning To document program processes to share best practices with interested stakeholders To share quantitative and qualitative data gathered from community outreach efforts

6 SF BRACE Process and Outcome Evaluation Questions for Steps 1-4 STEP 1: Forecast Climate Impacts & Objectives Identify and Outreach to Stakeholders 1. A: Climate and Health Profile To develop a Climate and Health Profile report and a community resiliency indicator system that qualitatively and quantitatively describes the climate related risk factors and health outcomes of concern for San Francisco over key future timeframes. How will this be disseminated? 1.B: Conduct a Vulnerability Assessment by neighborhood To develop vulnerability assessments that predict climate impacts of geographic neighborhoods that are especially vulnerable, to use the finest resolution possible to be used for neighborhood planning, and to include the impact of influential factors to vulnerability like ethnicity, linguistic isolation and low education, and to finally to use this tool to plan for protecting human health of vulnerable populations. 1.C: Community Resiliency Indicator System (Appendix 2) The goal of the project is to identify indicators that could collectively inform or determine the level of resilience to climate change stressors that a geographic area has in order to advance interventions that increase the geographic area s adaptive capacity. Process Evaluation Questions To what extent were partnerships developed to advance global health aspects of climate change? Does the climate and health profile sufficiently address the primary health impacts faced by climate change in San Francisco? Are the links between health impacts and a changing climate clearly explained in a way that is understandable to the public? Method: Qualitative: feed-back from internal city agency partners and external community partners engaged in climate change dialogue. How often was information shared with stakeholders? How many dissemination strategies were adopted? Method: Quantitative: Number of strategies used. Are the indicators appropriate to measure a community s resilience to climate change and other emergency events? To what extent do the identified indicators collectively denote a larger picture of resilience to climate change stressors? Are the indicators actionable? How will this indicator system be implemented and maintained over Outcome Evaluation Questions To what extent were health impacts and health based evidence integrated into San Francisco climate change planning? How well does the Climate and Health Profile guide future BRACE activities like the vulnerability assessment 1.D. Did the CHPR influence revisions or updates to existing emergency plans like DPH s Heat Annex? Method: Qualitative: Internal program self-reflection. Data collected from key partners in PHEPR & SFDEM. How well did the vulnerability assessments aid in identifying census block groups that are particularly vulnerable? To what extend did the vulnerability assessments assist efforts that aim to support socially and culturally isolate communities? Did the VA influence revisions or updates to existing emergency plans like DPH s Heat Annex? To what extent did participant s awareness of resources that can be used for climate change adaptation and mitigation increase? To what extent did participant knowledge of vulnerabilities to climate change for San Francisco increase? To what extent did the developmental and planning process of the RIP contribute to interagency cross pollination of information?

7 time in order to measure a community s changing resilience? To what extent does the indicator system contribute to San Francisco s city planning and specifically adaptive planning against climate change in support of the city s most vulnerable residents? Using information and maps to build into other project and reports- CDC s HIA for Sheltering in Place Post Earthquake. STEP 2: Project Disease Burden Process Evaluation Questions Outcome Evaluation Questions 2.A: Projected Burden of Disease Implement an epidemiological and statistical process to determine and project the trajectory, magnitude, and additional health burden of five or more health impacts and/or key risk factors associated with future climate change in San Francisco. How well did the Climate and Health Profile facilitate the development of the projections of disease burden? What challenges or benefits were encountered within the developmental relationship between the Climate and Health Profile? Are state-wide models and projections useful for climate change modeling and climate action planning at a localized level? How operational was the final toolkit? STEP 3: Assessing Public Health Process Evaluation Questions Outcome Evaluation Questions 3.A: Assess Suitability of Public Health Interventions & Develop additional Annex analyze existing risk assessments to inform climate related response efforts, align the Climate Grant activities with other ongoing community health assessment initiatives, and avoid duplication of efforts. To produce a final report which summarizes key information contained in recent risk assessments conducted for SFDPH that focus on hazards and vulnerable populations. 3. B: Creating Public Health Emergency Response Plans Develop, disseminate and begin implementing a multi year, climateadaptation plan for the health sector that includes interventions and adaptations with performance measures that are focused on achieving a measurable health impact in response to climate health impact scenarios, including extreme cold and What is the extent of outreach through the NEN, who receives heat warning advisories, and are their inferences that can be made about the audiences that they contact? How well were grant activities aligned with other health assessment initiatives? How influential were the Community Health Profile and Vulnerability Assessment to constructing the Overview section of the plans? To what extent was the final report influential to revisions or updates to plans? To what extent did the risk assessment and prioritization inform PHEPR activities during future grant years? How well were plans adopted by partnering stakeholders? What steps were taken to ensure information was shared intra- and inter- departmental communication and notification protocols? How were resources coordinated in mock exercises? To what extent were public

8 flooding. 3. C: Developing material and presenting internally to health department Objective needs to be articulated and refined STEP 4: Climate and Health Adaptation Planning 4. A: Creation of Neighborhood Level Adaptation Plans To develop neighborhood level action plans that advances investment in both mitigation and preparedness strategies that will reduce the impact of climate change on resident s health. To engage and educate stakeholders on health impacts of climate change and adaptation plans. How many presentations were made? How many different health branches attended? Process Evaluation Questions What is the scope of participants, and how does this support the advancement of FEMA s Whole Approach? Attendance Sheets & Participation % What can be determined by participation drop off or participants who no longer attend community meetings? Interviews/ Focus Group To what extent has there been the creation of a community wide vision of resilience? Mtg Notes, Attendance, & Exercise Protocol Do the preparedness plans adequately address some of the most potential public health threats from climate change in San Francisco? Interview SF CARD- How did you develop the list of threats? _ through the working groups, research Do the preparedness plans have a well-defined and strategic implementation plan? SF CARD information and warning systems developed, and were outreach goals met? Did the presentation to local health department staff increase internal understanding and awareness on the potential health impacts of climate change? Were any action steps developed or planned to increase capacity to respond to climate changed in response to presentations? Outcome Evaluation Questions Did participant s knowledge increase about types of hazards climate change can generate? Surveys & Focus Groups, How well were community stakeholders utilized to inform outreach strategies and the production of a community perspective report about how to protect vulnerable populations?

9 4.B. Implementation of Neighborhood Adaptation Plans Craft and implement hyper local resilience action plans that will benefit and prepare at the individual, organizational, and community level to respond to challenges in a more coordinated fashion across sectors How successful were we in disseminating and implementing the climate change adaptation plan in San Francisco communities? What is success? How many people attended community meetings where the plan was discussed? Were strategies of activation and notification adopted and were their clear protocols for outreaching to community based organizations? How many people saw social media or news posts regarding the climate change adaptation plan? Constant Contact %? What short term or long term action steps were planned, if any, to increase capacity to respond to climate change based on this presentation? Did the implementation of adaptation plans increase community resilience over time? To what extent were local project plans reflective of the needs of vulnerable populations? SF BRACE Short & Long Term Outcome Evaluation Questions STEP 5: Evaluation Process Evaluation Questions Outcome Evaluation Questions 5.A Short Term Outcome: Increase Both Local Level Capacity to Understand Climate Science and Health Impacts from Climate Change (Appendix 3) Learning Awareness Knowledge Attitudes How many presentations were given? What was the diversity of participating organizations and residents? What was the scope of populations represented by participants? Method: Quantitative NEN Participation Database, participation trends, quantitative representation of participation To what extent has levels of trust between program participants and coordinators changed? Did community events increase awareness among the participants about the potential health impacts of climate change in their community? To what extent did participants share information with their own agency, family and friends? Method: Quantitative & Qualitative transcribed notes Post-meeting follow-up surveys (paper or electronic) Focus groups and surveys # of increase learning opportunities Change in awareness and knowledge of health impacts of climate change Change in relationships with city agencies to increase trust and social capital between participants

10 5.B Short Term Outcome: Increase Internal Department Capacity to Understand Climate Science and Health Impacts from Climate Change Action Decision Making Policy Planning 5.C Intermediate Outcomes: Increase Public Understanding and Awareness of the Public Health Impacts from Climate Change (Appendix 3) Learning Awareness Knowledge Social Action Long Term Outcome: Improve health disparities and achieve health co-benefits through implementation of adaptation plans Social Conditions Evaluation of Overall Project Processes and Impacts How many presentations were given? What was the diversity of the departments presented to? Method: Quantitative Number of presentations # of attendees, # of depts.. How many outreach events occurred or were held? How many different public health impacts were presented? How often were heat waves, or climate change related impacts brought up in community meetings? Method: Quantitative and Qualitative Meeting notes, direct observation, informal correspondences Were BRACE products used to develop emergency plans across San Francisco? To what extent were health disparities and health co-benefits integrated into larger city adaptation planning? Integration into DOE s Climate Action Strategy Were efforts to reduce vulnerabilities identified in vulnerability assessments integrated into adaptation plans? (i.e. increase knowledge of socially isolated through home visits regularly made by DPH) To what extent were protocols to effectively communicate to the public successful? How well has DPH continued to share knowledge with CDC and other stakeholders about the implementation of program activities, To what extent has knowledge increased about climate change and related health impacts? To what extent has health impacts of Climate Change been incorporated into city adaptation planning? (DOE) Method: Quantitative & Qualitative Pre-Post Surveys- reflecting information from presentations (12/4/2014) Assessment of SF Dept. of Environment s Climate Action Strategy s degree of adopting DPH s recommendations Were any new ideas on how to problem solve vulnerabilities to climate change discussed by participants? Identification of definitions, symptoms, vulnerability factors, and at risk populations to different climate change emergency scenarios. Method: Quantitative and Qualitative Surveys, meeting notes, direct observation, informal correspondences Following the activation of a Health Department Annex during a Climate Change related hazard, what was the change in reported health disparities at clinics and hospitals? After the activation of a Neighborhood Support Center, were there any instances where an at-risk individual within the neighborhood needed medical assistance or were a NSC member intervened to mitigate an environmental exposure?

11 compiling findings, promising best practices and lessons learned to serve as a model for local health departments? What were the significant challenges or barriers faced during this project? Methods: Presentation to DPH and city employees. Participation in BRACE West calls. Evaluation of the Evaluation Was SFDPH successful in integrating project partners into the evaluation process? Do selected evaluation questions at project levels and corresponding data sources help to answer the larger programmatic outcome evaluation question? 4. Indicators & Credible Evidence What evidence will be used to indicate how the program has performed? BRACE Step 1 1. A: Climate and Health Profile Existing literature Existing Environmental Health regulatory data Metadata Access Tool for Climate and Health (MATCH) 1. B: Vulnerability Assessment 1. C: Community Resiliency Indicator System Participation, characteristics of participants, frequency and consistency of participation recorded through sign in sheets Before and after survey of presentation participants with questions that will answer o Reaction-feelings towards the program, acceptance of activities o Learning- increased knowledge of resources and tools, participant s programs and projects o Actions- increasing use of SFIP website, development of other working groups o System Change- Integration of Climate Change adaptation and resilience planning into city policies, practices and departments Qualitative information gathered from cross sector working groups in the form of transcribed notes The Sustainable Communities Index ( a comprehensive set of performance indicators for livable and sustainable cities, could be utilized to track neighborhood resiliency progress over time. BRACE Step 2 2. A: Projected Burden of Disease Modeled disease burden data from CDPH

12 BRACE Step 3 3. A: Assessing Public Health Interventions NEN s Constant Contacts lists to determine scope of sector representation and audience representation for districts most active in- Bayview, Diamond Heights, Miraloma Park 3. B: Creating Public Health Emergency Response Plans # of Internal city agency Emergency Response Committee Meeting Notes, and supporting documents to review communication impacts 3. C: Developing material and presenting internally to health department BRACE Step 4 4. A: Creation of Neighborhood Level Adaptation Plans Neighborhood Empowerment Network Participation Database, participation trends, quantitative representation of participation Survey questions/ After Actions interview or focus group Qualitative information gathered in the form of transcribed notes Surveys, post meeting follow up surveys Development of exercises that facilitate the creation of a community wide vision of resilience Neighborhood Action Plan developed by the cross sector working group that records the community s priorities on how they would like to see city and community resources aligned in order to ameliorate risk & and enhance the attainment of developmentally appropriate outcomes Community Resilience Indicator System 4. B: Implementation of Neighborhood Adaptation Plans What was the estimated total reach of communication efforts? What follow-up actions are being taken by the community after being presented with the adaptation plans? STEP 5: Evaluation 5. A: Short Term Outcome: Increase Both Local Level Capacity and Internal Department Capacity to Understand Climate Science and Health Impacts from Climate Change Internal health department presentations on the potential impacts of climate change on public health Neighborhood Empowerment Network Participation Database, participation trends, quantitative representation of participation Qualitative information gathered in the form of transcribed notes of any conversations generated in response to the presentation or questions that arose Post-meeting follow-up surveys (paper or electronic) Focus groups and surveys # of increase learning opportunities Change in awareness and knowledge of health impacts of climate change Change in relationships with city agencies to increase trust and social capital between participants 5. B: Intermediate Outcomes: Increase Public Understanding and Awareness of the Public Health Impacts from Climate Change

13 Community meeting minutes and notes # of community listening sessions # of districts presented in # of times climate change arises as a topic of conversation Degree of knowledge shared between participants and their organizations Development of new projects or initiatives that help to mitigate negative impacts of climate change Surveys Interviews- conducting a 2-3 key interviews over the phone may be less disruptive than conducting surveys. 5. C: Long Term Outcome: Improve health disparities and achieve health co-benefits through implementation of adaptation plans Protocols and steps taken to communicate with local service providers integrated into plans Effectiveness of outreach methods Meetings with key city agencies involved with response efforts around climate change related health impacts # of 5. D: Evaluation of Overall Project Processes and Impacts 5. E: Evaluation of the Evaluation 5. Justifying Conclusions What conclusions regarding program performance are justified by comparing the available evidence to the selected standards? Is there any room for error in crediting different outcomes to the program? 6. Lessons Learned How will the lessons learned from the inquiry be used to improve public health effectiveness? How can we build into program planning, and evaluation planning strategies to ensure that these goals are achieved? What stakeholders will be using the results of the evaluation? o Neighborhood Empowerment Network o PHEPR o Future PHES inquiries The dissemination of research, plans and information on climate health by presenting at the following: o The State of California s Climate Action Team Public Health Workgroup o Bay Area Regional Energy and Climate Resilience Public Health Sector Discussion o San Francisco's City agency meeting about climate adaptation in San Francisco o The 139th Annual American Public Health Conference in San Francisco o The 2nd CDC/NOAA Climate Science Symposium in Atlanta. o The Annual NAACHO Conference in Los Angeles.

14 Using the results from our evaluation efforts, SFDPH hopes to continually engage with its stakeholders and communities in a meaningful way to develop strong and resilient communities. We hope that our project will be used as a model for other municipalities in projecting health impacts from climate change at the local level and using that information to build strong and resilient communities at the neighborhood level.

15 References Bours, D., McGinn, C., and Pringle, P. (2014). Selecting indicators for climate change adaptation programming. SEA Change CoP, Phnom Penh and UKCIP, Oxford. Leiserowitz, A., Smith, N. & Marlon, J.R. (2010) Americans Knowledge of Climate Change. Yale University. New Haven, CT: Yale Project on Climate Change Communication. Rossi, P., Lipsey, M., Freeman., (2004) Evaluation: a systematic approach (7 th ed.). Thousand Oaks: Sage U.S. Department of Health and Human Services Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, Innovation Network, (2013) Evaluation plan workbook. Washington, DC: Innovation Network, INC, 2014.