Reliability and checklists John Kunz CIFE, Stanford University
Reflections - Tuesday
+/ Tuesday
Pre-plan examples Sketch Build checklist Do Monitor and challenge ICE pre-plan Problems Outcomes Participants Agenda Resources Member of Member of postsession space, Meeting Problems Outcome intent Intended Participant Pre-session pre-plan Role in ICE for session Desired outcomes Agenda items met? Participants discipline assignments team session wrapup technologies, to focus on (Yes/Partial/No) (yes/no) team models, tools (yes/no) Specify architectural spaces first floor Size building systems Identify building construction components for first floor Spaces to model are specified Systems in first floor are sized and specified Components (first floor) that take > 1 hour to install are listed and added to spec of components to model Mary Architect Owner share project goals and objectives Joe MEP Engineer Define and share team charter Hamid PM PM assure availability of BIM authoring and review tools for team members Yes Discipline expert Yes Review project goals and objectives No Discipline expert No Specify spaces, systems, components to model Yes Facilitator Yes assign modeling tasks to individuals Partial Yes No Smart Boards Excel Meeting space with tables, chairs for team of 10
Product sketches How to clarify their meaning for project
Smartboard example 6
Submission-2 Comments and questions
Reflections - Thursday 8
Agenda Thursday Reflections, review of Tuesday Checklists for reliability Submission-1 Examples ICE session: project definition w/organization analysis and optimization Initial student presentations SimVision charrette to analyze and optimize a predefined baseline organization
Reliability Overview Session Objectives [Lecture/ discussion; Understand and experience: Reliability issues in AEC globally Invisibility of reliability and risk in projects Checklists to improve reliability Risk assessment to identify reliability 10
Reliability Data from Chile post-earthquake ~500K/~5M homes damaged or destroyed: <2σ ~4 joint failures /~100 in (collapsed) buildings: 2σ 4/~10,000 post-1985 buildings collapsed in major damage area: >3σ Conclusions: Chile and developed economies We can design and construct structures well For structures, Statistical performance OK to good Society needs higher reliability from AEC Reliability of (designconstruction-operation) management is the issue 11
Invisibility of reliability and risk in projects Project success factors: Cost Schedule Quality Safety Reliability - Indicates that projects predict, measure and manage against these factors 12
Risk assessment Do formal risk assessment periodically during project Small number of experts Consider factors that affect life-cycle risk/success Market and demographic changes Practice (methods) changes Natural and societal challenges Design/construction risks Create mitigation activities to address biggest risks Flexibility (buffer) in design Checklists in design/construction processes 13
Reliability issues Spectrum of task types: Simple: many people can quickly learn to do the work, e.g., shovel gravel Specialized: large amounts of training need to learn to do the (virtuoso) work, e.g., Master-builder (pilot/surgeon); high strength welding Complex: High uncertainty and knowledge required exceeds that of any individual, e.g., construction (flight/surgical) team; design structural system Reliability management a requirement in situations with complexity: w/complexity, checklists required for success 14
Causes of low-reliability Knowledge missing: we do not know how to do something we are trying to do The fact that virtually all buildings have actual energy >> predicted suggests that we lack some important knowledge Incompetence: we do not do something we know how to do The fact that most buildings in Chile withstood big earthquakes suggests that we know how to design and build for earthquake performance 15
Checklists address the competence problem Good Checklists foster teamwork and discipline. They Define Tasks ( stupid things) to do Collaboration (Information to exchange) among multiple responsible stakeholders Allow specialists freedom to manage unpredicted conditions Need to balance Freedom (judgment) vs. discipline (procedure) Craft flexibility vs. protocol structure Specialty action vs. group collaboration Enable practice to monitor and challenge 16
Reliability in other industries Airline operations, surgery: Lots of performance data, e.g., on-time records; evidence based medicine Checklists, e.g., Flight checklist for entire aircraft crew; Pause time for checks in surgery Carefully crafted by industry, company, local groups Apply to very small steps in processes that have statistical reliability problems; not used for many steps Designed to provide structure and professional flexibility Enable extremely high process reliability when used broadly 17
Culture of effective Checklist use Power: for complex, non-routine problems, push to source of greatest actionability: the periphery Away from center Senior management: Sets goals and objectives Measures progress Facilitates (and assures) communication Team members: Have shared responsibility to act, to check: monitor and challenge Act professionally: selfless, skilled, trustworthy, disciplined 18
Culture of effective Checklist use Initiate activation phenomenon Share names (if not already known) Share concerns and observations at start of each major activity, e.g., the day more engagement in checklist use Of marginal help, at best Published books of standards Financial incentive for performance 19
Effective Checklists Precise: Clear, relatively short (~5-9 major steps), uncluttered Include Do Confirm steps Often part of a family of checklists, e.g., checklist for each day, for different circumstances Have strengths: Help experts remember important steps in complex procedures Clarify priorities and sequences Prompt teamwork Enable rapid dissemination of new knowledge Do work often dramatically Have limits: Cannot do skilled work Cannot force conformance Require training Require careful development 20
AEC Checklists that can help reliability Examples: Gantt charts, deliverable schedules, punch lists, w/ Tasks (process steps) Responsibilities: who does task; who helps Options, e.g., resources, start/end times, next steps (successors) Commitments: Formal: RFIs, formal meetings, punch lists Informal: emails, informal meetings VDC checklists: process to specify functional intent, design content and behaviors of project models and projects 21
AEC Reliability data Large amounts of data are needed to identify process steps that affect life or company health and safety Some industries have accident cause data, e.g., OSHA, NTSB Some performance data exist in AEC, e.g., RS Means cost OSHA Total Construction safety Incidence Rates per 100 Full-time Workers: 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 8.8 8.6 8.3 7.9 7.1 6.8 6.4 6.3 5.9 5.4 Process reliability data largely invisible, even within companies Process reliability must be exceptionally high (> 6σ) to assure acceptable lifecycle product reliability 22
Management Quality Survey: 20 questions in three broad areas [Bloom] Management practices Hospital Operations Management Performance and target management Talent and People management Example dimensions evaluated Performance dialogue and review Interconnection of targets Performance tracking Target balance Consequence measurement Dimension Scoring criteria 1.1) Measures tracked do not indicate directly if overall hospital objectives are being met. Tracking is ad hoc 2.3) Most key performance indicators are tracked formally. Tracking is overseen by senior staff 3.5) Performance is continuously tracked and communicated, formally and informally, to all staff using a range of visual management tools 24
India project Intervention aimed to improve 38 core textile management practices in 6 areas for example: 25 25
Intervention aimed to improve 38 core textile management practices in 6 areas for example: Frequent monitoring Management Context India project 26 26
Share of 38 management practices adopted Share of key textile management practices adopted Experiment gave treatment plants free consulting for 5 months - adoption of these 38 management practices rose India project.2.3.4.5.6 Treatment plants Intervention Treated Control plants Control Excluded plants (not treatment or control) -10-8 -6-4 -2 0 2 4 6 8 10 1227 Months after treatment
Quality defects index (higher score=lower quality) 20 40 60 80 0 Performance improved e.g., quality defects down 50% Start of Diagnostic Start of Implementation End of Implementation India project 100 120 140 Control plants 97.5 th percentile Average ( symbol) Treatment plants 2.5 th percentile 97.5 th percentile Average (+ symbol) 2.5 th percentile -20-10 0 10 20 30 40 weeks Weeks since after diagnostic the start of phase the diagnostic 28
Step in detailed schedule Example Checklist Checks to make before start of step Monitor and Challenge 29
Step in detailed schedule Example Checklist Checks to make before start of step Monitor and Challenge 30
Example Checklist Step in detailed schedule 31
Impacts of Safe Childbirth Checklist Single centre testing, Gokak, India Preliminary findings 3 months of preintervention data Data on ~400 childbirths collected (admission to discharge) 3 weeks of checklist implementation data Indicator (selection) Washing hands and wearing gloves when doing vaginal examination or delivery Checking baby s temperature after birth Routinely administering Oxytocin within 1 minute after birth Baseline Post- Intervention <10% >90% <5% >90% <5% >90% 32
Checklist templates and VDC We can use checklists to do Project definition (v) and define the: 1. Functional objectives what you want for your project; 2. Scope or form what you will or did do; 3. Behaviors predictions and measurements of what you did We can use these (actionable) checklist templates to: Specify work we do, e.g., BIM (create checklist) Verify work (e.g., BIM content) wrt specification (w/checklist) monitor and challenge Store results of analysis (in checklists) Evaluate how well the designed/built scope responds to objectives for the Product, Organization and Process, i.e., manageable elements of the project (using a checklist) 33
Questions to consider in Project Definition actionability for team, for us What is the problem? what actions can a manager take? Manager can adjust functions, scope and details of product, organization and process Project definition defines them For whom is it a problem? Project definition describes the organization and identifies the stakeholders in the decision(s) that may/will face the problem Who is the decision maker(s) Project definition identifies stakeholders who can act Why do the modeling and analysis? Project definition identifies impact of performance on assessed goodness objectives, which allows team to identify impacts of choices So what? Adjust modeling and analysis level of detail to fit the impact of the decision on form or scope, schedule, quality and risk Inspired by Science and Decisions: Advancing Risk Assessment, National academies press, 2009. 34
Product breakdown structure: Design scope or form of Product Content of Commitments BIM content template A LOD Building Components Spaces Systems Note: functional intent of product element is implicit Foundation Aboveground steel B LOD Wall systems Skin Patient rooms x 400 Windows Public areas Services e.g., CT Specification of BIM content iroom work space BIM elements HVAC Legend BIM Content Specification Level of detail (LOD) - AIA Type of product Product sub-type Level of detail LOD) - comments Components Foundation 100 (Conceptual) Minimum dimension > 4" Components Above-ground steel 100 (Conceptual) Minimum dimension > 4" Components walls - Exterior, basic 200 (Approximate geometry) Minimum dimension > 4" Components walls - Interior - Partition 100 (Conceptual) Minimum dimension > 4" Components Roof 300 (Precise geometry) Architectural, not construction detail Components Windows 300 (Precise geometry) Architectural, not construction detail Components Doors 200 (Approximate geometry) Spaces Rooms 100 (Conceptual) at least 2 rooms; size of each 150-250 ft2 Due date of next BIM version BIM content conforms to spec [Yes/No] Roof 35
Work breakdown structure: Design scope or form of Work Content of Commitments production plan template A LOD Tasks Design Build Commission Operate Note: functional intent of task element is implicit Design: building elements [Architect] Task Priority Short description 1 2 Responsible team Commitments - example Coordination Budget (FTEhours) dependent team(s) Approval team Done ontime Due date A (Contract requirement) Add door in BIM PM 20 Architect PM 1/1/2010 No Comments/ model image(s) B LOD Assess: Behaviors [Owner] Design: Building systems [Architect] Design: Building systems [HVAC/MEP designers] Legend Type of process task Task sub-type 36
Metrics Implementation template Metrics Name Comment Target value Tolerance: +- Intent How to use in management Source of data Type [P, O] Stakeholder s who saw data last week Collection frequency Prediction Objective Weight Evaluation Predicted/ measured value (how you are doing) Assesed value M Quality: POE satisfaction wrt program (%) 100 5 Guide commissioning, next job Client assessment O Owner only Turnover time + 6-24 months 40 86 1 E Cost conformance to plan (item actual - predicted/predicted) 100 5 Plan next job Client assessment O PM only Turnover time 25 98 3 T Schedule conformance to plan (%) 100 10 Plan next job Client assessment O Alll on team Turnover time 35? NA R Predicted Cost conformance to plan (item actual - predicted/predicted) 100 5 Attention management Periodic project progress report P Subteam only Weekly 15 99 3 I Production schedule conformance to plan (%) 90 10 Attention management Periodic project progress report P Alll on team Weekly 10 75 2 C Assessed Quality conformance to plan (% of items with rating >=4 on scale 1:5) 100 10 Attention management Periodic project progress report P Subteam only Weekly 15 95 3 S Stakeholder participation that is timely and meaningful (%) 90 10 Adjust plan by stakeholder review and assessments Periodic project stakeholder survey P Alll on team Weekly 10 90 3 37 P: Process; O: Outcome (C) 2013
How to use VDC templates for project definition and as checklists Generally, in <= 20 minutes each, Sketch Review Implement in template(s) Review: add checks For identified areas of greatest risk