Experimentation for better policy-making. CES Breakfast session, January 18, 2011

Transcription

1 Experimentation for better policy-making CES Breakfast session, January 18, 2011

2 Governance in the 21 st Century Governments serve in an increasingly unpredictable context; they need to improve their ability to anticipate emerging trends, risks, and opportunities. Experimentation can contribute greatly to the anticipative and adaptive capacities of governments and societies. Experimentation and pilot testing on a small scale can accelerate learning, encourage innovation, and help shape a better policy response. (The New Synthesis Project, 2010) 2

3 Experimentation as an applied research method The systematic use of experiments, i.e. demonstration projects based on rigorous evaluation methods, can contribute greatly to the anticipative and adaptive capacities of governments and societies. 3

4 How so? Putting new ideas to test: Identifying barriers during program or policy implementation Getting a sense of take-up rates by targeted population Measuring intended and unintended outcomes Measuring impacts of a new program the difference it really makes Providing cost effectiveness or cost benefit analysis 4 Spreading innovation Promoting collaboration For greater accountability Killing bad ideas

5 Demonstrations vs Experiments 5

6 What s rigorous evaluation in this context? The capacity to measure the difference a new approach or program makes. To measure the true impact of a new program, we need to know what would have happened if the program had not been introduced...we need a counterfactual: to account for natural maturation processes, e.g. children skills are constantly improving to account for factors external to the program, e.g. state of the economy varies to account for regression to the mean phenomenon, e.g lone mothers on welfare eventually go back to work 6

7 Displaced workers re-integrating labour market as a result of being offered an earnings supplement (!" '!" &!" %!" $!",-../ #!"!"! # $ % & ' ( ) * + #! ## #$ #% #& #'!"#$%&'(")&*+#,")&-../0#)1#$&

8 Displaced workers being offered supplement versus those not receiving the offer (!" '!" &!" %!" $!",-../ / #!"!"! # $ % & ' ( ) * + #! ## #$ #% #& #'!"#$%&'(")&*+#,")&-../0#)1#$&

9 What s rigorous evaluation in this context? The capacity to measure the difference a new approach or program makes. To measure the true impact of a new program, we need to know what would have happened if the program had not been introduced...we need a counterfactual: Random assignment of participants to a treatment group and a control group provides the best method to create a counterfactual Not the only method available Not always possible to do RA 9

10 Properties of social experiments 10 The defining characteristic of a social experiment is random assignment of some pool of individuals to two or more groups that are subject to different policy regimes. With a large enough sample, random assignment insures that the two groups of individuals are identical, on average. This is true for all observable and unobservable characteristics -- such as motivation, self-confidence, determination and all other personal attributes that can explain why an intervention will succeed with one individual but not another. Unlike nonexperimental methods, properly implement social experiments are guaranteed to provide internally valid impact estimates no selection bias. Nonexperimental methods may be equally reliable in any given application; we simply cannot know a priori that they are reliable, as we can with experimental methods you can only match on measured characteristics.

11 How Random Assignment Works Note: Both program and control group members continue to have access to government programs and services available to members of their community. 11

12 Experimentation is more than Impact Analysis Experiments need to rely on a several evaluation methods: 12 Are we delivering what we think we are delivering? Monitoring and accountability, operational evaluation Can we deliver this better or more efficiently? Formative evaluation, process evaluation, implementation analysis What difference does it make? Is it worth doing? Impact evaluation, cost-benefit analysis Why does it work? Qualitative methods: focus groups and key informant interviews

13 Government use of randomized trials/social experiments in social policy 13 Widespread use in the US over last 40 years 225 experiments completed from 1962 to 2003 Mostly from Departments of Labor, Education, and Health and Human Services Growing interest in continental Europe in connection to discussions on social innovation (European Commission) and improving education methods (Switzerland) Major surge in popularity in the international development field in last 5 years or so 232 experiments completed in 39 countries by J-Pal and affiliates since 2003 and 141 ongoing projects In Canada, use of demonstration and experimental projects remains limited only half a dozen large scale projects completed

14 Examples of Canadian experiments

15 Social Research and Demonstration Corporation A non-profit, non-partisan social policy research organization operating at arms length from government Was created in 1991 at the request of Employment and Immigration Canada to implement and evaluate the Self-Sufficiency Project Since then, SRDC has had responsibility for the implementation and evaluation of most large social experiments conducted in Canada Our mission is: to help policy-makers and practitioners identify and implement policies and programs that improve the well-being of all Canadians, with a special concern for the effects on the disadvantaged; to raise the standards of evidence used in assessing government policies and programs. SRDC now includes some 30 permanent researchers working from three offices in Ottawa, Vancouver and Toronto. SRDC receives no core funding from governments or private funders 15

16 The Self-Sufficiency Project Testing an innovative making work pay strategy that used temporary earnings supplements to help long-term welfare recipients achieve self-sufficiency through employment. A 10-year study involving more than 9,000 single parents in New Brunswick and British Columbia. Results have found employment impacts that are among the largest ever seen in a welfare-to-work program; little net cost to government, and an effective approach to raising the incomes of poor families and reducing poverty. Financial incentives were even more effective when combined with employment services. Project also showed small positive effects on the school performance of young children. SSP contributed to inform several provincial earnings supplementation programs for welfare recipients and the implementation of the federal WITB. 16

17 SSP impact on full-time employment by Months From Random Assignment '# &" &# %" %# $" $# " #!" *+,-+./01+,23 4,56+,701+,23 8/3.96!$%!(!% ' ) $' $) %' %) &' &) '' ')!"#$%&'()"*'+,#-"*'.&&/0#*1#$ 17

18 The Earnings Supplementation Project 18 A test of an earnings supplement to encourage EI recipients to go back to work quickly. Intervention: workers who would become re-employed in a job that pays less than the job lost were offered 75% of earnings losses for up to two years. Operated in nine communities in seven provinces and involving 11,500 participants. Tested with two groups displaced workers or repeat EI users. Results show no impacts with repeat EI users; small temporary employment impacts with displaced workers. ESP produced evidence that this type of policy for displaced workers needed to be approached with caution.

19 Displaced Workers Monthly Percentage of Sample Members Who Were Employed Full Time )#$ (#$ '#$ &#$ %#$ "#$ #$!"#$ -.// / / :2/;<4 # " % & ' ( ) * +, "# "" "% "& "' "(!"#$%&'(")&*+#,")&-../0#)1#$&

20 The Community Employment Innovation Project 20 EI and welfare recipients gave up their benefits in exchange for up to three years work in community-based jobs. Participants were paid a community wage. Community organizations were responsible for developing job opportunities that met local needs. Project was testing the capacity of the social economy to provide work opportunities in a high unemployment region (Cape Breton Island). Results showed that: CEIP was an attractive option for many EI and IA recipients program better targeted at IA beneficiaries benefits accruing to communities were substantial From a cost-benefit perspective, CEIP is a suitable policy tool only if one had dual objectives of providing support for both unemployed workers and vulnerable communities.

21 CEIP impacts on EI sample A 53 percentage point impact at peak $## +% +# *% *# )% )# (% (# %% %# '% '# &% &# "% "# $% $# % #!%!$#!$%!"#,-./ :150;8< $ & % ) + $$ $& $% $) $+ "$ "& "% ") "+ &$ && &% &) &+ '$ '& '% ') '+ %$ %& No significant impacts a year after end of CEIP eligibility!"#$%&'()"*'+,#-"*'.&&/0#*1#$ 21

22 IA Sample: Permanent reductions in IA receipt three years after CEIP comes to an end '## &# +,-.,/012, ,-812, /:7 "# $# %# A stable 42 percentage point reduction in IA receipt during program #!%#!$#!"# ' '' %' (' $' )' "' *'!"#$%&'()"*'+,#-"*'.&&/0#*1#$ A sustained 12 percentage point reduction in IA receipt 22

23 learn$ave 23 A unique Canadian test of asset-based policy for low incomes. Participants could accumulate savings in Individual Development Accounts which were matched at a rate of 3 for 1 (maximum savings of $6,000 over three years). Permissible uses of matched savings: Education (post-secondary or skills development) Small enterprise development 4,800 participants across 10 sites in Canada. Final results showed that while many low-income Canadians interested in education do eventually enrol in education courses and programs, a program like learn$ave would increase the level of participation in PSE education programs by over 20%. However, cost-effectiveness analysis suggests that this may not be the most cost effective way to promote education among lowincome populations.

24 Impacts on Education Enrolment Results Covering 54 Months

25 CareerMotion A small-scale experiment designed to test whether the provision of labour market information (LMI) can improve the level of confidence and competency among post-secondary education (PSE) graduates with regards to their career and employment decision-making. Goal is to address the problem of under-employed PSE graduates at the time that demand for qualified workers is increasing. Some interim results show positive impact of Web-based LMI intervention. 25

26

27 Literacy and Essential Skills in the Workplace Object is to measure the impacts of Literacy and Essential Skills (LES) training on workers and workplaces. Project will involve 1,000 workers whose literacy scores fall short of job requirements. Half of these workers will receive LES training and the other half will serve as a control group. Tourism was selected as target sector. Some 80 to 100 establishments to participate. Project is building partnerships across sectors. 27

28 Project Organization 28

29 Projects to promote access to PSE Future to Discover (Millennium Scholarship Foundation project) testing with 5,400 high-school students in New Brunswick and Manitoba an early-intervention post-secondary information program and an advanced guarantee of student financial assistance, called Learning Accounts. AVID evaluation (Millennium Scholarship Foundation project) testing a college-preparatory program known as Advancement Via Individual Determination, for students who are academically in the middle. Some 1,522 students participating, with half receiving the program, in 18 schools in BC. Life After High School (HRSDC Learning Branch) 29 Some 5,000 students from 25 low PSE transition high schools in BC are being provided information about the benefits of post-secondary education and step-by-step practical assistance about how to apply for postsecondary education and for student aid.

30 Random assignment for Future do Discover Project Participants in 51 schools New Brunswick participants, Learning Accounts eligible New Brunswick participants, Learning Accounts ineligible, and Manitoba participants Random Assignment Random Assignment Comparison Group Learning Accounts only Explore Your Horizons plus Learning Accounts Explore Your Horizons only Explore Your Horizons only Comparison group

31 Considerations

32 When to Use Random Assignment Need evidence about impacts to inform policy and administrative decisions Alternative methods unlikely to yield acceptable level of certainty Able to meet ethical and legal standards Able to match research goals and operational and political realities Expected value of experiment relative to cost needs to be high: Is the issue important from a policy perspective? Are the potential net social benefits significant? Can it lead to policy change eventually?

33 Design/Operational Challenges Duration Sample sizes Recruitment; sampling methods Nature of informed consent External validity Potential contamination Treatment differential/treatment intensity Attrition; survey responses from control group GE effects or community effects

34 Influence on policy ʻʼGovernments don't often use research directly, but research helps people reconsider issues, it helps them think differently, it helps them reconceptualize what the problem is and how prevalent it is, it helps them discard some old assumptions, it punctures old myths." Whiting Professor Carol Weiss, Harvard University 34

35 How to increase use of experiments? Value evidence-based policies Dedicated funds needed Top bottom or bottom-up? Top-bottom, if interested in scalability Bottom-up, if emphasis is on finding out innovative practices Government role should be to support projects, but insist on evaluation to learn what works Public sector knowledge brokers are essential Role of public sector schools and public management schools in training brokers Duration is a shared responsibility of government officials (more forward looking) and researchers (improved methods) 35

36 Conclusion Innovations are needed to face emerging problems (e.g. obesity, managing diversity, low saving) and difficult challenges (e.g. working poor, school drop-outs, youth crime, social exclusion) Experiments should be conducted with a view to learn what works. Whether governments are main actors or enablers, there is a need to document, monitor and evaluate. 36

37 Contact Information Social Research and Demonstration Corporation Société de recherche sociale appliquée 37