Quantification of Defects via Expert Extrapolations Lies, Damn Lies, and Statistics? Lonnie Haughton, CDT, LEED AP Richard Avelar & Associates (Oakland, CA) Quantification of Construction Defects FEWA-Westcon November 2008 1 Within the limits of the commission from the client, a defects expert s most common duties are to: carry out an objective forensic survey of a building envelope s construction and its performance; impartially consider and evaluate the results reported by other experts; & serve as an independent educator for stakeholders in the case. Quantification of Construction Defects FEWA-Westcon November 2008 2 1
To these ends, a construction expert should not be an advocate for any position contrary to his or her honest professional evaluation of the issues. A common analytical error by some reviewers of forensic evaluations is the assumption that the only valid survey protocol is random sampling. Quantification of Construction Defects FEWA-Westcon November 2008 3 In a sense, such misjudgments are a testament to the strength of the general public s blindly abiding faith in the inerrant power of statistical formulas; however, this error also often is evidence of a fundamental lack of understanding di of the actual process of assembling the components of the building envelope. Quantification of Construction Defects FEWA-Westcon November 2008 4 2
For example, consider the installation of nail-flanged windows by a small crew during a 8-hour work day and assume that on previous days this team already has installed numerous other windows at the project. It is safe to predict that on this day the manner and quality of the various window installations by this crew will be generally consistent. Quantification of Construction Defects FEWA-Westcon November 2008 5 Further, barring revised instructions from their superintendent or other inspectors, it is reasonable to assume that the team s standards and output on this day will be generally consistent with their standards and output of the previous day. In other words, a generally consistent day-to-day level of workmanship (whether good, middling or deficient) is found at most projects. Quantification of Construction Defects FEWA-Westcon November 2008 6 3
Quantification of Construction Defects FEWA-Westcon November 2008 7 Difficulties for the random sampler also occur at multi-story buildings due to the horizontal nature of the work carried out on scaffolding. Crews who work on scaffolding to flash and install rows of windows will proceed in a generally consistent assembly line manner, using more or less the same means, methods and materials at each opening. Quantification of Construction Defects FEWA-Westcon November 2008 8 4
Quantification of Construction Defects FEWA-Westcon November 2008 9 Yes, during the entire course of a project s construction there will be inconsistencies and deviations (for better or worse) in the flow of the intertwined streams of work. Quantification of Construction Defects FEWA-Westcon November 2008 10 5
However, years later after the workers have been dispersed and their managers memories have faded and the availability of project records is limited any attempt to evaluate via statistical sampling the patterns (or the blips in the patterns) of their outputs almost certainly will be inaccurate and incomplete. Quantification of Construction Defects FEWA-Westcon November 2008 11 FEWA-Westcon November 2008 Quantification of Construction Defects 12 12 6
Quantification of Construction Defects FEWA-Westcon November 2008 13 13 FEWA-Westcon November 2008 Quantification of Construction Defects 14 14 7
Also, we investigators must consider the generally vertical nature of leakage paths of travel. If we find water damage at a window head, it would be foolish not to chase the leakage up the wall; however, a fundamental tenet of random sampling is that the findings from any particular sample cannot be used in any manner to shape the course of the continued sampling. Quantification of Construction Defects FEWA-Westcon November 2008 15 Still, during the course of defects litigation some advocates continue to use the statistics mantle of authority to disguise invalid arguments in opposition of an expert s findings. Similarly, it is not uncommon for some experts to make wildly non- credible claims and extrapolations that grossly exceed the capabilities of their sampling design. Quantification of Construction Defects FEWA-Westcon November 2008 16 8
Unfortunately, such excesses tend to raise concerns that the reports of all construction defects experts simply are representative of the three types of gross misrepresentations: Lies, damn lies, and statistics. Quantification of Construction Defects FEWA-Westcon November 2008 17 The rules of evidence require an expert s sampling methodology and testimony to be based upon scientifically valid principles. Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 US, 579 (1993) and Kumho Tire Company v. Patrick Carmichael, 526 US, 137 (1999) However, any sampling methodology that is not feasible, practical or cost effective cannot be considered representative of the scientific method Quantification of Construction Defects FEWA-Westcon November 2008 18 9
To address these concerns, it is instructive to compare the tenets of random statistical sampling with the methodology prescribed in ASTM E 2128 (Standard Guide for Evaluating Water Leakage of Building Walls). Section 11 of this excellent standard includes, in part, the following protocol for carrying out a qualitative survey of the building envelope: Quantification of Construction Defects FEWA-Westcon November 2008 19 ASTM E 2128 investigative protocol: 1. An evaluation is conducted in response to a problem situation and a non-performing wall, and may involve several techniques and procedures specifically adapted and applied in a systematic manner to diagnose a specific problem. Quantification of Construction Defects FEWA-Westcon November 2008 20 10
ASTM E 2128 investigative protocol: 2. The information systematically accumulated in a leakage evaluation is analyzed as it is acquired. The new information may motivate a change in approach or focus for subsequent steps in the evaluation process. Quantification of Construction Defects FEWA-Westcon November 2008 21 ASTM E 2128 investigative protocol: 3. The building envelope evaluator is expected to establish a cause and effect relationship between wall characteristics and observed leakage. This requires an appropriate selection of activities and a logical analysis and interpretation of the acquired information. Quantification of Construction Defects FEWA-Westcon November 2008 22 11
ASTM E 2128 investigative protocol: 4. If they are to be considered legitimate and substantiated, the conclusions and findings from an evaluation must be rationally based on the activities and procedures undertaken and the information acquired. Quantification of Construction Defects FEWA-Westcon November 2008 23 ASTM E 2128 investigative protocol: 5. The record should be sufficiently complete so that any interested party can duplicate the evaluation program and acquire similar information. Quantification of Construction Defects FEWA-Westcon November 2008 24 12
Clearly, the purposeful inquiry prescribed by the authors of ASTM E 2128 is inconsistent with random statistical sampling. However, the survey protocol of ASTM E 2128 is fully consistent with qualitative sampling designs that have been validated within the social sciences. Quantification of Construction Defects FEWA-Westcon November 2008 25 What would be considered bias in statistical sampling, and therefore a weakness, becomes intended focus in qualitative sampling, and therefore a strength. The logic and power of purposeful sampling lie in selecting information- rich samples for a step-by-step evaluation of issues of importance to the purpose of the inquiry. Quantification of Construction Defects FEWA-Westcon November 2008 26 13
While the goal of quantitative sampling is to evaluate levels of statistical significance the methodology of qualitative inquiry within the social sciences is to produce findings that have substantive t significance, ce, which refers to the strength & importance of a meaningful relationship. Quantification of Construction Defects FEWA-Westcon November 2008 27 In determining substantive significance, both the analyst and the subsequent reviewers must address these kinds of questions: a. How solid, coherent, and consistent is the qualitative evidence in support of the expert s findings? Quantification of Construction Defects FEWA-Westcon November 2008 28 14
In determining substantive significance, both the analyst and the subsequent reviewers must address these kinds of questions: b. To what extent and in what ways do the findings further a deep understanding of the observed conditions? Quantification of Construction Defects FEWA-Westcon November 2008 29 In determining substantive significance, both the analyst and the subsequent reviewers must address these kinds of questions: c. How well do the researcher s findings correlate and define causal relationships in a manner that maximizes understanding of the various processes and phenomena of interest? Quantification of Construction Defects FEWA-Westcon November 2008 30 15
In determining substantive significance, both the analyst and the subsequent reviewers must address these kinds of questions: d. To what extent are the analyst s findings consistent with knowledge derived from other sources? Quantification of Construction Defects FEWA-Westcon November 2008 31 While statistical analysis follows formulas and rules, qualitative analysis is founded on the knowledge and conceptual capabilities of the experienced analyst. An absolute cornerstone of this process is the researcher s credibility no substantive rigor can be given to findings by professionals who have demonstrated a willingness to promote distorted or false data. Quantification of Construction Defects FEWA-Westcon November 2008 32 16
In summary, skilled qualitative analysis of the building envelope can constitute a form of scientific rigor that meets the legal standards for rules of evidence. In contrast, random probability analysis, in and of itself, typically is not a practical or legitimate t method for understanding the sources and mechanisms of water leakage. Quantification of Construction Defects FEWA-Westcon November 2008 33 Even so, it is important to recognize that qualitative and quantitative sampling methodologies constitute alternative, but not mutually exclusive, strategies for investigative surveys. The pragmatic investigator will implement a range of qualitative and quantitative sampling measures that best evidence credibility when reviewed by the target audience. Quantification of Construction Defects FEWA-Westcon November 2008 34 17
The skilled investigator also practices a situational responsiveness that recognizes that differing methods and techniques are appropriate for different circumstances. For example, the appropriate method for evaluating the distribution of window installation deficiencies may differ greatly from the best protocol for quantifying the distribution of manufacturing defects. Quantification of Construction Defects FEWA-Westcon November 2008 35 And, some defects simply inform us of the overall level of quality control Quantification during of Construction the original Defects FEWA-Westcon construction November 2008 36 36 18
Finally, we should recognize two key phases of expert quantification: Generalization of the limited results of the qualitative survey to the entire project, followed by: Determination of the later removals needed to identify the locations and extents of these hidden problems at the remainder of the project. Quantification of Construction Defects FEWA-Westcon November 2008 37 It is this perhaps costly distinction (between initial theoretical extrapolations of hidden damages to the non-surveyed portions of the building and then the subsequent determinations of the appropriately fair scope of work necessary to pinpoint these damaged areas at a later time) that t is the most likely l trigger of heated charges of biased advocacy. Quantification of Construction Defects FEWA-Westcon November 2008 38 19
Often, there will be no mutually agreeable answers to such fairness disputes regarding repair costing, constructability and durability; however, these challenges generally are separate and distinct from any evaluation of the overall competence, integrity and substantive significance of the qualitative survey process that was carried out in accordance with the principles of ASTM E 2128. Quantification of Construction Defects FEWA-Westcon November 2008 39 Key References: M.Q. Patton, Qualitative Research & Evaluation Methods, Sage Publications 1980 (revised 2002) M.Q. Patton, How to Use Qualitative Methods in Evaluation, Sage Publications 1987 J.W. Creswell, Research Design Qualitative & Quantitative Approaches, Sage Publications 1994 Y.S. Lincoln and E.G. Guba, Naturalistic Inquiry, Sage Publications 1985 V. Barnett, Sample Surveys & Methods, Oxford University Press 2002 D. Campbell, foreword to R.K. Yin, Case study research: Design and methods, Sage Publications i 1989 (revised 2003) ASTM E 2128, Standard Guide for Evaluating Water Leakage of Building Walls, ASTM International 2001 END OF PRESENTATION 40 Quantification of Construction Defects FEWA-Westcon November 2008 40 20