What constitutes rigorous evidence for policy design, implementation and evaluation?

Transcription

1 What constitutes rigorous evidence for policy design, implementation and evaluation? Javier M. Ekboir ILAC coordinator Institutional Learning and Change Initiative of the CGIAR 1

2 Overview of the presentation What is generally understood as a rigorous explanation? Understanding causality The nature of data What is then an explanation? Quantitative and qualitative methods in complex processes Institutional Learning and Change Initiative of the CGIAR 2

3 Decision makers have called for evidence-based policies The claim is based on the assumption that it is possible to collect objective data and find universal lessons (e.g., best practices or best fit) It is an extension of what is perceived as the scientific method Is this objectivity possible? Institutional Learning and Change Initiative of the CGIAR 3

4 The accepted principles of a scientific approach (until 1970s) Four principles were accepted as givens Classical physics was the model for science Science was the best method for knowing the world Scientific findings were approximations to an objective truth that could be discovered Mathematical logic and language were the models for clear reasoning and explicit statements It was a model of deterministic and simple causalities Institutional Learning and Change Initiative of the CGIAR 4

5 How did scientists accept something as true? Falsification principle: Theories could not be proven true; but they could be proven false by finding one instance where they did not apply, especially by hypotheses testing (1960) This approach was seldom used in practice but the idea of science as objective, precise knowledge became ingrained in western societies Since the 1980s, new sciences (e.g., sociology, biology and complexity theories) provided alternative models of science Institutional Learning and Change Initiative of the CGIAR 5

6 How do scientists accept something as true? Today there is no consensus about what is a valid explanation, confirmation, theory testing or even science It is recognized that there is no one science, therefore, no single model or method to accept scientific information is universally valid Even more, each sub-science has its own methods Institutional Learning and Change Initiative of the CGIAR 6

7 Can scientists explain without data? Major scientific theories were developed without empirical support Institutional Learning and Change Initiative of the CGIAR 7

8 Aesthetics Simplicity How do scientists select a theory? Philosophical beliefs Consensus A theory can be accepted as rigorous without empirical support No theoretical debate was ever solved with better data (Miller 2000) Institutional Learning and Change Initiative of the CGIAR 8

9 There is no universally accepted We have an intuition definition of what is a cause Causes can prevent something from happening Some factors need to be present for something to happen Others need to be absent Institutional Learning and Change Initiative of the CGIAR 9

10 Causality is not objective Causality chains are very long We decide (based on our knowledge, beliefs and goals) which causes are important This explains why different disciplines explain the same phenomenon differently Rigorous determination of causality is not defined by a particular method, but by clearly stating the assumptions and logic used Institutional Learning and Change Initiative of the CGIAR 10

11 The attribution problem Back to the short circuit, who is responsible for the fire? Each of these causes is necessary for the fire to happen (or not happen) But they cannot each be responsible for 100% of the outcome In a complex process, outcomes cannot be attributed to individual causes, but to sets of interacting causes Econometrics cannot replicate experimentation (LaLonde 1986); therefore it cannot attribute effects Institutional Learning and Change Initiative of the CGIAR 11

12 The attribution problem (2) Even in experimental settings, causality and outcomes are defined by the experiment What does the RCT evaluation of Progresa mean? Institutional Learning and Change Initiative of the CGIAR 12

13 Argentina s amazing discovery What is a counterfactual? Every data set (experimental or not) is open to an infinite number of hypotheses (underdetermination thesis) Every random process has more than one possible outcome and it is impossible to know which will occur before the event Every outcome of a random process prevents other outcomes from occurring Remember: causality is defined by the researcher or evaluator! For each cause, there are a large (possible, infinite) number of counterfactuals, even in an experimental setting Institutional Learning and Change Initiative of the CGIAR 13

14 There are no objective data Let me show you the picture of a bird Or is it a rabbit? Data are collected and have a meaning only within a theoretical framework ( theory-ladenness of data) Data are important because they provide information to build and confirm theories Institutional Learning and Change Initiative of the CGIAR 14

15 Hypotheses cannot be tested This was shown in 1906! (T, H, A 1, A 2,, A n ) T: Education influences income H: the influence is positive A 1 : measured after high school or college? A 2 : 2 or 5 years after completion?, A n : functional form The more sophisticated a model, the more and stronger auxiliary assumptions it uses Institutional Learning and Change Initiative of the CGIAR 15

16 Use of mathematical and statistical tools does not mean that the analysis is rigorous Institutional Learning and Change Initiative of the CGIAR 16

17 Then, what do researchers do? They use theories and data to build explanations But there is no universal consensus on what a valid explanation is The acceptance or rejection of a narrative depends on the consensus among the majority of scientists and other stakeholders about what is a good explanation This does not mean that all knowledge is relative Just that knowledge is determined by the interaction of social conventions, theories and data Institutional Learning and Change Initiative of the CGIAR 17

18 Two types of research approaches Quantitative (Variable oriented) Qualitative (Case oriented) They differ on how they use theory and data to build narratives Mixed methods are increasingly used Institutional Learning and Change Initiative of the CGIAR 18

19 Quantitative research Seeks broad patterns by studying a small number of dependent variables across a large number of cases Height vs Weight Weight (lbs) Height (ins) Institutional Learning and Change Initiative of the CGIAR 19

20 Assumptions on which quantitative analysis is based Populations are homogeneous with well defined distribution functions Populations are defined prior to the collection of data and analysis Explanations are heavily variable-oriented, and the individuality of each case is not relevant Causation is predetermined and stable, often additive and linear, making the approach insensitive to causal complexity Institutional Learning and Change Initiative of the CGIAR 20

21 How quantitative analysis works Using a-priori knowledge and theories, a concise representation of the phenomenon is built With this representation, large data sets of a few variables are collected The data are then analyzed with statistical tools to find correlations, which are seen as confirmation of the theories Institutional Learning and Change Initiative of the CGIAR 21

22 Qualitative analysis Emphasizes in-depth analysis of cases to show how the different aspects mutually interact to form the whole case A small number of different cases may be compared Institutional Learning and Change Initiative of the CGIAR 22

23 Assumptions on which qualitative analysis is based Populations are sets of heterogeneous cases Populations are often redefined as the research advances Cases are configurations of many aspects that should be understood at the level of the specific instance Causation is contextual, plural, nonlinear, non additive and changing Institutional Learning and Change Initiative of the CGIAR 23

24 How qualitative analysis works Using prior information and theories, important information that must be collected is identified As the information is collected, different theories and causal links are checked against collected information on a large number of variables and interactions (making sense) Based on these results, new theories are developed and additional information is collected until a satisfactory explanation has been constructed (or until the resources are exhausted) Institutional Learning and Change Initiative of the CGIAR 24

25 In short Quantitative analysis: one theory is used to statically compare many observations of a few variables Qualitative analysis: in-depth observation of many variables in a few cases are used to dynamically compare many theories Institutional Learning and Change Initiative of the CGIAR 25

26 Why researchers trust their methods and distrust others For the qualitative researcher, confidence comes from depth For the quantitative researcher, it comes from breadth In general, qualitative methods do not meet the standards for valid statistical inference Often, neither quantitative methods do Institutional Learning and Change Initiative of the CGIAR 26

27 Validity of non-experimental analyses In depth case studies provide the basis for constructing generalizations that hold, at least, for the cases analyzed; often these generalizations have wider relevance Many areas of science where experimentation is not possible (e.g., astronomy, geology, history and many social sciences) depend on the analysis of a small number of uncontrolled cases Institutional Learning and Change Initiative of the CGIAR 27

28 Which approach is more appropriate? Depends on the goals and the problem under study Quantitative approaches are useful for the study of relatively stable, simple relationships that hold for large numbers of cases Qualitative approaches are appropriate for the analysis of complex relationships that change over time or space In complex processes, quantitative methods are less effective (less rigorous) because they limit exploration of possible explanations and bet that the posited explanation is the closest to the truth Institutional Learning and Change Initiative of the CGIAR 28

29 In short For accountability, simple quantitative methods are probably better than sophisticated models and qualitative analyses For learning, qualitative methods supported by simple quantitative indicators are definitively better Institutional Learning and Change Initiative of the CGIAR 29

30 Thanks Institutional Learning and Change Initiative of the CGIAR 30