Integration Evaluation Framework for Integrated Design Teams of Green Buildings: Development and Validation

Transcription

1 Integration Evaluation Framework for Integrated Design Teams of Green Buildings: Development and Validation Rahman Azari 1 and Yong-Woo Kim 2 1 Assistant Professor, College of Architecture, Construction and Planning, University of Texas at San Antonio, 501 W. Cesar E. Chavez Blvd., San Antonio, TX 78207, USA; PH (210) ; Corresponding Author rahman.azari@utsa.edu 2 Associate Professor, Department of Construction Management, University of Washington, 120 Architecture Hall, Box , Seattle, WA 98195, USA; PH (206) ; yongkim@uw.edu ABSTRACT Integrated Design (ID) process encourages integration of team members in design phase of green building projects through intense collaborative processes and free exchange of information. While integration in general and ID in particular has been well theorized by construction management research community, there exists no systematic mechanism in the field to help owners, architects and managers of green project teams assess the level of integration in their projects ID team environment in a practical manner. The key objective of the present article is therefore to use a qualitative-quantitative methodology to propose and validate an integration evaluation framework for green project teams, and to statistically test the association between integration level and project success. The framework can be used by green project teams for comparison, benchmarking or educational purposes as well as for integration evaluation and improvement in ID team environments. This research also provides empirical evidence to anecdotes suggesting positive link between team integration and project success in green projects. Keywords: Evaluation, integration, integrated Design, green buildings, validation, CIPP INTRODUCTION Due to complexity of green project requirements and diversity of disciplines involved in the design process of these projects, the owners and project team managers try to apply Integrated Design (ID) process to ensure delivery of targeted outcomes. In the ID process, an integrated team of key stakeholders (owner, architect, contractors, suppliers, users, etc.) is developed early on in the design process and the members intensely collaborate in ecocharrettes, that is, inclusive brainstorming sessions, to define and agree upon project goals pertaining to sustainable design and construction and to collectively decide about some important aspects of the project design. Similar to Integrated Project Delivery (IPD) and Concurrent Engineering (CE), the focus of the ID teams is to create integration. But integration in the ID teams is different in scope and expected outcome. Indeed integration in IPD is about integration of information, leadership, agreements and processes (CMAA 2010), and in CE is about integration of processes and people, while integration in the ID context is oriented toward integration of people (i.e., disciplines), information, and building systems in order to meet project s green targets. Previous studies in the field (Korkmaz 2007, Yudelson 2008, Kibert 2008, 7group and Reed 2009) have well theorized the issue of integration in green construction projects and have even anecdotally linked it to project success. However, only a few (Korkmaz 2007) offer tangible metrics for the assessment of integration in the ID team environment, without

2 organizing the metrics in a quantifiable evaluation framework. As a result, project teams and managers in green construction projects often lack a systematic mechanism to self-evaluate and quantify integration, a gap that is also highlighted by researchers (Xue et al 2010) in the context of construction field in general. The green construction community also needs to be provided with empirical evidence to anecdotal suggestions about the link between integration and project success. A systematic and quantifiable integration evaluation framework can help teams to not only understand the integration and collaboration level that exists between key participants in a green project but also identify the integration-related areas that need to be improved. This improvement of integration would in turn have direct positive impacts on project success, as the empirical results of this research would suggest. Ideally, the statistical association between integration and project success can be used to predict success through integration score. The research questions of interest in this research include: 1. How to measure the degree of integration in the ID team interactions in the design process of green building projects? 2. Are green building projects with higher degrees of integration in the ID team environment more successful? To address the research questions, the authors conducted a sequential qualitativequantitative research to develop, implement and validate an evaluation framework to assess the ID team integration in green building projects. More specifically, the research objectives were framed as the following: a. Develop the evaluation framework Identify the evaluation factors indicators that could be used to measure the integration level within the ID team environment; Organize the factors/indicators in a systematic evaluation model; Propose a measurement format to quantify the evaluation b. Implement and validate the proposed evaluation framework; and c. Test the hypothesis that: Higher integration in the ID team environment results in more successful project outcomes. LITERATURE REVIEW The concept of integration is widely used in construction industry. Nam and Tatum (1992) use integration as an antonym to disintegration which, according to them, is the outcome of incongruent goals and consequent divergent behaviors in construction projects. They also state that there is a close relationship between integration and cooperative project environments. Similarly, Baiden and Price (2011) define integration as where different disciplines or organizations with different goals, needs and cultures merge into a single cohesive and mutually supporting unit with collaborative alignment of processes and cultures. The need for integration in construction industry primarily stems from the fact that fragmented procurement to delivery of construction projects leads to a lack of efficiency and effectiveness in delivery of projects with respect to time, cost, etc. (Love et al 1998, Baiden et al 2006). Literature links high wastes, increased project costs and time, and adversarial relationships in the industry to separated project organizations (CMAA 2010). Mitropoulos and Tatum (2000) believe that the fragmentation in project organization is a result of two major factors: the complexity of construction projects and the high degrees of specialization involved in design and construction of projects. Numerous studies highlight a need to more collaborative

3 and integrated approaches in order to address this challenge as well as to decrease the uncertainties associated with construction projects and improve the predictability of results (Love et al 1998, CMAA 2010, Mitropoulos and Tatum 2000, Egan 1998, Egan 2002, Fairclough 2002). Mitropoulos and Tatum (2000) maintain that integration is critical in building design for two reasons: to avoid problems that could happen after design phase and to achieve an optimized design solution. Integration, according to them, becomes more needed in project projects with higher levels of uncertainty, complexity, and speed in construction (Mitropoulos and Tatum 2000). Indeed, in complex construction projects, there would be a need to create an environment of free exchange of information from the early stages of project development that necessitates integration of team members. This integration can also tackle the problem of adversarial relationships and lack of trust within project teams that is believed to be the main criticism in current practices in delivery of construction projects (Ruparathna and Hewage 2015). Collaboration is a primary element in integration concept and is defined as a creative process undertaken by two or more interested individuals, sharing their collective skills, expertise, understanding and knowledge (information) in an atmosphere of openness, honesty, trust and mutual respect, to jointly deliver the best solution that meets their common goal (Wilkinson 2005). Collaboration in construction occurs across organizational boundaries between owners, design team, contractors, etc. (Smyth and Pryke 2008). Table 1 displays some of the parameters that affect the quality of collaboration in construction projects, along with the supporting literature. Factor Literature Table 1. Factors affecting collaboration Accountability Chan et al 2004, Gabriel 1991, Yudelson 2008 Commitment Chan et al 2004, Yeung et al 2007, Yudelson 2008 Communication Diallo & Thuillier 2005, Hauck et al 2004, Yeung et al 2007, Yudelson 2008 Compatibility Kumaraswamy et al 2005a, 2005b Timely Involvement Chan et al 2004, Yeomans et al 2006, Yudelson 2008 Joint Operations Chan et al 2004, Hauck et al 2004, Yeomans et al 2006, Yudelson 2008 Mutual Respect Forbes & Ahmed 2010 Trust Ballard 2006, Chan et al 2004, Cheung et al 2003, Diallo & Thuillier 2005, Rahman & Kumaraswamy 2004, Xu et al 2005, Yudelson 2008 While collaboration and free exchange of information are widely mentioned in discussions around integration, integration thinking, or systems-thinking, which refers to considering the relationships among the constituent subsystems of a system in order to create an optimized performance seems to be less explicitly examined. In systems-thinking, the decisionmakers of a project realize that a decision about one subsystem would impact the performance of other subsystems; therefore, any decision-making on the subsystem of interest must be made by involving all disciplines, as representatives of subsystems, in the decision-making process. Recent trends in construction industry have tried to improve integration from multiple angles. Building Information Modeling (BIM) has shown great potential to enhance the integration of AEC team members by providing an environment of free exchange of information. The increased collaboration that is achieved through BIM can lead to increased efficiency, time

4 and cost-savings, profitability, improved relationships, and lower rates of errors (Azhar 2011, Porwal and Hewage 2013). The industry has even developed BIM-based applications that can be used to optimize project cost, time, and environmental impacts during design phase (Inyim et al 2015). Integrated Project Delivery (IPD), as another development in the industry, tries to create integration in four aspects of agreements, leadership, information and processes so that the maximum value is delivered to owner (CMAA 2010). Integrated Design and Delivery Solutions (IDDS), established by the International Council for Research and Innovation in Building and Construction (CIB), is the framework for an integrated and coordinated merger of people, process and technology issues in order to enact a radical and sustained transformation of the construction industries (Owen et al 2010). This framework, which is founded on four components of collaborative process, enhanced skills, integrated information and automation systems, and knowledge management, tries to take the industry from the current levels of BIM adoption to more holistic and sophisticated integration solutions proposed by IDDS (Owen et al 2010). While the mentioned integration approaches in the industry are broad in concept, the ID process, another integration approach, is limited in scope and copes only with design process of green building projects. US Department of Energy (DOE 2009) defines the ID process as the process in which multiple disciplines and seemingly unrelated aspects of design are integrated in a manner that permits synergic benefits to be realized. The extensive literature on Integrated Design (ID) process agrees that the process recognizes the interdependency of building systems, the complexity of green building design, and the need for early collaboration of all disciplines in order to make optimum decision for the whole building (IEA 2003, Yudelson 2008, 7group and Reed 2009, Kibert 2008). The level of complexity that exists in green projects usually requires more collaboration and integration of project parties. Robichaud and Anantatmula (2011) highlight coordination and communication across a team of multidisciplinary members as the most significant challenge to delivery of cost-effective green buildings. Indeed, to design an optimally-functioning green project, the team has to design all building systems as a unified whole, by considering the impacts of each system on other systems performance (Yudelson 2008, 7group & Reed 2009, Kibert 2008). Achieving the integration of systems requires integration of the representatives of all design disciplines in a team to freely share their information and collectively design the building. Therefore, integration, in the context of the ID process of green buildings, is about integration of people (i.e., disciplines), information and building systems in a holistic life-cycleoriented way that ensures delivery of value and sustainable outcomes to owner. It implies using a collaborative team of architects, engineers, building occupants, contractors, etc. to collectively consider various aspects of sustainability (water efficiency, energy efficiency, site, indoor environmental quality, etc.) from the earliest stages of building design and to realize the synergies and tradeoffs between these aspects over the life-cycle of a project. The outcome would be an optimized building that provides a healthy productive environment for the occupants, is sustainable to environment, and generates value to owner. Researchers and practitioners in the industry usually have a similar approach to the ID process, even though their focus might be slightly different. Kowk and Grondzik (2007) highlight several steps toward integrated design. According to them, these steps include establishing commitment, team formation and setting of goals, information gathering, conceptual and schematic design, testing, design development, construction, and assessment and

5 verification. Yudelson (2008) lists the elements of integrated design as commitment to integrated design, setting goals and criteria for the team, commitment to zero cost increase, front-loading the design process with environmental charrettes, allocating time to feedback and revisions before committing to final design selection, and engagement of all team members in all decisions. Robichaud and Anantatmula (2011) define a green project delivery process that emphasizes early definition of goals and priorities, integration of project team, design with the entire team, use of rewards in contracts and provision of training. Synthesizing previous definitions and elements, integration in this research is defined as timely collaboration of relevant project stakeholders with the goal of encouraging systemsthinking to deliver optimized value to the client and the ID process is the way through which this definition takes place in the settings of green building projects. RESEARCH METHODS Similar to social science research, construction management research applies a variety of research methods and paradigms, ranging from qualitative (i.e., inductive approach; interpretivisim paradigm) to quantitative (i.e., deductive approach; positivism and realism paradigms). Wing et al (1998) characterize construction management research as having a practical nature that requires generalizability of results and testing hypotheses (hence, positivism paradigm and quantitative methods should be applied). Yet, they believe that qualitative methods (interpretivism paradigm) can be applied in addressing certain problems in construction research, such as understanding human behaviors (Wing et al, 1998). Due to the nature of the problem of interest in this research which required a combination of in-depth exploration of a phenomenon (i.e., integration in the ID team interactions) and development of generalizable outcome and results, the authors chose to use a sequential qualitative-quantitative research methodology to achieve the research objectives. This methodology has been illustrated in Figure 1. In the qualitative step of research, case-study examination and interviews with industry and academic experts were conducted to identify the key factors to be assessed for evaluation of the ID team integration and to provide a checklist of evaluation indicators to be used for operationalization of the factors. Three case studies, representing various levels of integration in the ID process, were selected based on their reputation and media coverage and were explored in detail. Table 2 shows the characteristics of the case-studies. Extensive interviews were conducted with the stakeholders of these projects to understand how the ID process in these projects functioned, what parties were involved in the design process and at what point during the project development they got involved, what factors facilitated success or inhibited it, and so forth. In addition to extensive interviews with case-study project stakeholders (a total of 12 interviews), 15 additional industry professionals with extensive experience in the area of integrated design that represented major ID disciplines were interviewed to achieve a further insight into what major team-related factors impact the design process and its success, and to identify obstacles, challenges, and facilitators of integration. By interviewing the professionals, the researcher also aimed at isolating some operationalized definitions, or evaluation indicators, for evaluation factors. The interviews were then coded and analyzed using Dedoose qualitative research tool. The content data analysis helped identify the evaluation factors and indicators that are important in assessing the ID team integration. Through the content analysis the authors

6 listed the themes related to integration assessment that emerged in responses by all the interviewees. This list of themes, i.e., integration evaluation factors, became the backbone of the evaluation framework. During interview sessions, once a theme (collaboration, for instance) emerged in the discussion, the authors asked the interviewee how they defined it and what the best and the worst forms of that theme in an ID team environment would be. This helped us write operational definitions, i.e., evaluation indicators, for the identified evaluation factors. The outcome of content analysis included a checklist of integration evaluation factors and their corresponding indicators. The integration evaluation factors in this checklist were included based on the unanimous opinion of the interviewees. In case of collaboration as an evaluation factor, because of the breadth of concept, the authors used the literature to break the concept down into 8 sub-factors. Finally, the evaluation indicators were determined based on the interview data as well as literature suggestions. Figure 1. Methodological steps in evaluation framework development and validation Table 2. Information on green project case-studies Case ID Green Rating Floor Area (sf) Construction Year Building Type Project Delivery Case 1 Living Building 52, Office CM@R Case 2 LEED Gold 196, Educational CM@R Case 3 LEED Silver 77, Research Lab CM@R To ensure that the evaluation indicators cover the content domains i.e., all important aspects of the ID team integration -, a content validity test was conducted by extending the checklist of the indicators to a group of experts (representing both industry and academia) and asking them to check: a) whether the checklist covers all aspects of the ID team integration and b) if the indicators present in the checklist can properly assess those aspects. The checklist was

7 then revised based on the comments received. More specifically, the experts suggested the addition of two indicators to the list; one to capture commitment and another to capture systems-thinking. They also suggested changes in the wording of some indicator statements. In the final step of framework development efforts, the evaluation factors and indictors were organized into an evaluation model. The authors chose to adopt Context, Input, Process and Product (CIPP) (Stufflebeam 2003) as the evaluation model following a review of various models used in different disciplines. CIPP is one of the most popular models in education and business contexts and is regarded by the literature among the best with respect to feasibility, utility, and accuracy (Zhang et al 2011). The framework, proposed by Stufflebeam (1983), evaluates the evaluation subject by assessing its context, inputs, process and products. Following the organization of the factors and indicators within the categories of the evaluation model, a measurement format was designed to help quantify the evaluation results. Once the framework to assess the integration was developed, the authors had to make sure it was valid and fitted the purposed it had been designed for. The objective of the second phase of research, the quantitative phase, was to implement the proposed evaluation framework and validate it. The literature highlights various measures for validation. Some of the widely mentioned measures include content validity, construct validity, internal validity, external validity, and reliability (Babbie 2010, Creswell 2012, Hinkin 1995, Yin 2002). Content validity was met in qualitative phase through the expert feedback. In quantitative phase, the authors converted the proposed evaluation framework into a questionnaire survey and distributed it among the ID teams of buildings rated by Leadership in Energy and Environmental Design (LEED) green rating system and asked them to self-assess the integration level of their ID team environment. Statistical analysis was then used to validate the proposed evaluation framework with respect to construct validity, internal validity, external validity and reliability. To determine the needed sample size for questionnaire-based study, the authors used a rule of thumb guideline suggested by several studies (Green 1991, Vanvoorhis and Morgan 2007) based on which the needed sample size for multiple regression should be at least 50+m, where m is the number of independent variables in a regression model. In the present research, the maximum number of potential independent variables was determined to be 7; hence, a sample size of at least 57 cases was determined to be needed. As mentioned, the questionnaire survey was designed by incorporating the proposed evaluation framework developed in the qualitative phase of research. The statement of the questions reflected the evaluation indicators, and the response format was based on a 5-point Likert-scale, representing various levels of agreement with each evaluation indicator statement. In addition, some questions were added to collect demographic information about the owner type, project size, cost, etc. By filling out this survey, LEED project participants served as the self-evaluator of their own projects. The survey was pilot-tested and then distributed through several online social networking platforms for professionals, such as LinkedIn. The researchers also recruited their professional and personal contacts to reach other potential LEED project participants. 79 responses were collected over a two-month period. This sample size was considered to be sufficient, given the fact that the needed sample size for the research was 57. The collected responses entailed a representative sample of LEED-rated projects. Figure 2 illustrates a profile of the collected sample with respect to project parties involved in evaluation, floor area, owner type, project delivery systems, LEED versions, and LEED certification level.

8 Figure 2. A profile of the questionnaire respondents and represented projects Using Stata IE 10 as the statistical analysis software, the collected data were recoded, examined for missing values (and treated, if needed) and organized. Also, three indices were constructed, based on the formulas 1, 2, and 3 (which will be presented later in the measurement format of the proposed framework), in order to consolidate the scores assigned by the evaluator to the evaluation indicators under different categories. Challenge Index (CI) was constructed to represent the evaluation indicators in the context category. Integration Assessment Index (IAI) was built to represent the input and process categories; and Performance Index (PI) consolidated the evaluation indicator scores associated with the product category of the proposed framework. Table 3 shows the list of variables, their assigned codes and type of data. Once the database was completed, the authors started analyzing the data, first for reliability assessment. Reliability is a validation measure that concerns the consistency of measurement (Weiner IB and Greene 2011) and the repeatability of results (Babbie 2010). The most widely accepted statistical measure for reliability of evaluation indicators is internal consistency method using Cronbach s Alpha (Hinkin 1995) which measures the degree to which evaluation indicators complement each other in their measurement of the subject of evaluation (Yin 2002): the higher Cronbach s Alpha, the higher the reliability. In order to avoid the misleading effects of potential inflation in Cranbach s Alpha which can happen due to numerous indicators in the framework (Streiner 2003), the authors used Cronbach s Alpha in this research along with Item-Test, Item-Rest, and Inter-Item correlations. The authors also used 0.60 as the minimum acceptable level of Cronbach s Alpha (Hair et al 1998).

9 Table 3. Variables along with their associated evaluation category/factor and assigned code Code Evaluation category Variable Type PAR Party represented by the respondent Nominal Categorical LOC Location Nominal Categorical YEAR Construction year Continuous OWN Owner type Nominal Categorical SIZE Project size (sf) Continuous COST Project cost (dollar) Continuous LDVR LEED version (1.0, 2.0, etc.) Nominal Categorical LDTCR Total LEED credits achieved Continuous LDECR Energy & Atmosphere credits achieved Continuous CMP Context Complexity and Uncertainty level Ordinal Categorical PTY Context Priority level Ordinal Categorical PDS Context Project delivery system Ordinal Categorical CONT1 Context Integration level of contract Ordinal Categorical CONT2 Context Sustainability inclusion in contract Ordinal Categorical CI Context Challenge Index Continuous TEAM Input Team capability level Ordinal Categorical TOOL Input Implementation of tools and technology Ordinal Categorical COL1 Process Collaboration > Accountability Ordinal Categorical COL2 Process Collaboration > Commitment Ordinal Categorical COL3 Process Collaboration > Communication 1 Ordinal Categorical COL4 Process Collaboration > Communication 2 Ordinal Categorical COL5 Process Collaboration > Compatibility Ordinal Categorical COL6 Process Collaboration > Involvement Ordinal Categorical COL7 Process Collaboration > Joint operations 1 Ordinal Categorical COL8 Process Collaboration > Joint operations 2 Ordinal Categorical COL9 Process Collaboration > Mutual Respect Ordinal Categorical COL10 Process Collaboration > Trust 1 Ordinal Categorical COL11 Process Collaboration > Trust 2 Ordinal Categorical LEAD Process Leadership Ordinal Categorical SYS1 Process System-thinking 1 Ordinal Categorical SYS2 Process System-thinking 2 Ordinal Categorical IAI Input & Process Integration Assessment Index Continuous LDCR Product LEED certification level Ordinal Categorical SUSC Product Cost success level Ordinal Categorical SUST Product Schedule success level Ordinal Categorical SUSI Product Innovation level Ordinal Categorical SUSS Product Safety success level Ordinal Categorical PI Product Performance Index Continuous To test for construct validity, internal validity and external validity, a multiple regression analysis technique was applied. Regression analysis can depict construct validity by reflecting the statistical association between variables in the model, which then can be checked for agreement with the literature suggestions (DeVellis 2003). Also, because regression analysis can provide the opportunity to study research variables under controlled conditions, it can help assess internal validity, which is about ruling out plausible rival hypotheses (Rosenthal and Rosnow 1991). The authors used project cost and challenges as control variables in this research. In addition, external validity, which is about generalizability of the research findings to other samples (Rosenthal and Rosnow 1991), could be tested by showing representativeness (Babbie 2010) of the sampled data.

10 Through the multiple regression analysis, the authors also tested the hypothesis that: higher integration in the ID team environment results in more successful project outcomes. The regression model to test this hypothesis initially included four variables, as shown in Figure 3: a. Independent variable: integration level (measured through Integration Assessment Index (IAI), as defined by the evaluation framework, and representing evaluation indicators present in input and process categories of evaluation framework); b. Dependent variable: project success (measured through Performance Index (PI), as defined by the evaluation framework, and representing evaluation indicators present in product category of evaluation framework); and c. Control variable: challenges level (measured through Challenge Index (CI), as defined by the evaluation framework, and representing evaluation indicators present in context category of evaluation framework); and d. Control variable: project cost Figure 3. Variables in initial multiple regression model and direction of their effects Figure 4. Multiple regression analysis methodology The authors initially included the challenge level and cost as control variables in the model because integration level is not the only factor that can impact the success of green building projects. Indeed, success or failure of a project can be a result of the level of challenges (complexity, uncertainty, priorities, level of scope definition, effect of project delivery systems and contractual support) that it faces or the monetary resources that it is taking advantage of.

11 Therefore, the addition of the project cost and the challenge level as control variables in the regression model would help isolate the effect of integration on project outcomes. In other words, by holding constant the challenge level and project cost, it becomes possible to measure how integration alone affects project outcomes. Figure 3 shows the variables in the multiple regression model and potential direction of their effects on project success. Multiple regression analysis of the model shown in Figure 3 entailed a methodology that is illustrated in Figure 4. Based on this methodology, the data were visually examined for outliers. Following the removal of outliers from the data, a preliminary bivariate regression analysis was conducted with the presence of dependent variable and either of other variables. The purpose of this analysis was to examine if there was a relationship between variables in the first place. Then, a multiple regression analysis was run with all variables included. To ensure the accuracy of results, multiple regression results should be diagnosed, as several problems might be present in analysis with the potential to add bias to and skew the results. These problems include outlier data, non-normality, heteroscedasticity, multicollinearity, and non-linearity. The results of diagnosis analysis revealed the presence of outliers and nonlinearity. To detect and remove outliers, DfFit and studentized residuals were used as the measures of detection. DfFit focuses on the change in parameter estimates resulting from exclusion of an observation (Miles and Shevlin 2001). Studentized residuals examine regression residuals that have been divided by their standard deviations (Wooldridge 2009). Five observations were detected commonly by the two measures and were excluded from the analysis. The regression model was re-run using the data that were now free from the effect of outliers. Non-linearity was also present in regression analysis. This occurs when the relationship between independent and dependent variables is not linear. This is a major concern in Ordinary Least Square (OLS) regression models. Non-linearity was checked by visually inspecting the residual scatterplots, which revealed concerning deviation of the cost variable from linearity. To treat the problem, this variable was transformed through a natural logarithmic function. The transformed variable was labeled LGCOST (natural logarithmic function of COST). Re-analysis of the model using the transformed cost variable failed to show any statistical significance of its effect on project success. Therefore, project cost was dropped off the model and the regression analysis was re-run using the new model (Figure 5). RESULTS The authors present the results of this research in two parts. First, the proposed CIPP-based integration evaluation framework, which is illustrated in Figure 6 and presented as a checklist in the Appendix, is described and then validation results are presented.

12 Figure 6. Proposed CIPP-based integration evaluation model for the ID teams of green projects

13 CIPP-based Integration Evaluation Framework for the ID Teams The outcome of qualitative phase of research was the CIPP-based integration evaluation framework for the ID teams of green building projects. As Figure 6 shows, this framework consists of four major components: a) the CIPP evaluation model and its four categories of context, input, process and product, b) a list of evaluation factors organized under CIPP evaluation categories, c) evaluation indicators corresponding to the evaluation factors, and 4) a measurement format. The framework is explained in the following sections. Evaluation Model and its Categories: As mentioned before, the CIPP evaluation model was adopted in this research to ensure comprehensiveness of assessment. The CIPP model is widely used for evaluation purposes in business and education contexts. Based on the CIPP model, an evaluation effort should be conducted through a comprehensive framework under the four categories of context, input, process, and product (Stufflebeam 1983, 2003). Context evaluation focuses on the needs, challenges, and opportunities within a defined environment that affect the performance of the process being evaluated. Input evaluation assesses the resources available and proposed strategies. Process evaluation focuses on the activities and factors critical to successful completion, and product evaluation determines whether the intended outcomes were achieved (Stufflebeam 1983). Figure 6 displays the evaluation factors for the ID team integration, separated by the CIPP evaluation categories. Evaluation Factors (20 factors): The second component of the proposed evaluation framework represents the evaluation factors; i.e. macro-level areas to be evaluated under each and all four categories of the CIPP model, in order to assess the integration level of the ID teams in green building projects. 20 factors were identified through qualitative research. Due to its breadth, the authors broke collaboration, as an evaluation factor, into 8 sub-factors (such as accountability, commitment, communication, etc.) to help specify it, as shown in Figure 6. Evaluation Indicators (65 items): The evaluation factors identified in the qualitative phase of research were broad concepts difficult to evaluate. To facilitate integration evaluation and provide tangible and measurable criteria for evaluation of the factors, they were operationalized (i.e. specified) into 65 evaluation indicators. These evaluation indicators were identified based on the interviews with industry experts as well as suggestions by previous studies in the field. The resultant final list of evaluation indicators included 65 items. Figure 6 shows the number of evaluation indicators per each CIPP category. The complete list of evaluation items can be found in the appendix section of this article. To show an example of evaluation indicators specified for an evaluation factor, Figure 7 shows the indicators that were designed to capture the presence of systems-thinking. Figure 7. Four (4) evaluation indicators were specified to capture systems-thinking

14 Measurement Format: The final component of the proposed evaluation framework is a measurement format for quantifying the results of evaluation. Quantification of the evaluation provides the opportunity to compare integration levels across projects (and their associated ID teams) and link those to project outcomes. In designing the measurement format, first a response format for evaluation indicators was provided. The response format would show various degrees of change in each evaluation indicator and allow the evaluator to choose the one among them that best reflects the subject of evaluation. Likert-scale is one of the most common response formats used in various studies (DeVellis 2003); therefore the authors chose it as the response format for the evaluation framework of this research. The points in Likert-scale were designed to represent various degrees of agreement with the declarative statement of the evaluation indicator. A 5-point Likert-scale provides the highest reliability and is the most widely used type of Likert-scale in the scale development literature (Hinkin 1995). Accordingly, a 5-point scale was used in this research with five degrees of strongly disagree, disagree, neutral, agree, and strongly agree. The next issue in designing a measurement format is scoring and indexing. Since validation and hypothesis-testing was a future step in this research, the evaluator s scores on the evaluation indicators needed to be consolidated into indices that could quantify the performance of teams under each category of the CIPP model. To this end, one important issue was the weighting of evaluation indicators. The researchers had to decide whether the evaluation indicators had equal or different weights with regard to their importance in measuring the subject of evaluation. While there is no firm rule as to weighting, the literature in the field recommends equal weighting of evaluation indicators unless there are compelling proved reasons to use differential weighting (Babbie 2010). Moreover, equal weighting can be considered objective because the subjectivity of the researcher with this method remains limited to the influence of the subject matter experts on selection of evaluation indicators (Maggino and Ruviglioni 2009). For the purpose of the evaluation framework developed in this research, equal weighting was used for two reasons: 1. There is no literature and theoretical framework in the field supporting the differential significance of the evaluation factors (such as trust, mutual respect, collaboration, etc.) in achieving integration; 2. Equal weighting improves the objectivity, simplicity, and robustness of the developed evaluation framework. With equal weighting, and assigning the scores of 1, 2, 3, 4, and 5 to strongly disagree, disagree, neutral, agree and strongly agree, respectively, the following equations could be applied to construct the indices of interest. Three indices were designed: a) Challenge Index (CI) representing challenges arising from the context of a project; 2) Integration Assessment Index (IAI) representing input and process categories which would show the level of integration maturity; and 3) Performance Index (PI) representing the product category which could be used as a measure of project success. The following equations were used for building these indices: Formula 1: CI = S % Formula 2: IAI = (S ( + S *+ ) Formula 3: PI = S *. Where, - CI, IAI, and PI refer to Challenge Index, Integration Assessment Index and Performance Index, respectively;

15 - S c, S i, S ps, and S pt refer to the scores assigned to evaluation indicators in the context, input, process, and product categories, respectively. The minimum value for each index can be determined by assigning a score of 1 (representing strongly disagree ) to all evaluation indicators. Likewise, using a score of 5 (representing strongly agree ) for all indicators results in the maximum values for CI, IAI, and PI. The performance of a given project under these indices would vary within the range between minimum and maximum values. To rate projects based on their performance under these indices, the range between minimum and maximum for each index was translated into multiple intervals, as shown in Table 4. Table 4. CI, IAI and PI indices and their weight ranges Challenge Index (CI) Score Integration Assessment Index (IAI) Score Performance Index (PI) Score Extremely Challenging Extremely Integrated Extremely Successful Moderately Challenging Moderately Integrated Moderately Successful Somewhat Challenging Somewhat Integrated Somewhat Successful Mildly challenging Mildly Integrated Mildly Successful Not challenging Fragmented Unsuccessful 9-15 Validation Results As mentioned before in the methods section, the proposed evaluation framework was implemented through a questionnaire survey of LEED-rated projects. The authors then analyzed the collected data for validity check (reliability, construct validity, internal validity and external validity) and tested the hypothesis proposed by the research. Reliability Assessment: Tables 5, 6, and 7 display the results of reliability assessment for the evaluation indictors present in the proposed evaluation framework. As Table 5 shows, the Cronbach s Alpha for the context category of evaluation framework is , which is above the minimum acceptable level of 0.6 (Hair et al 1998). The last column in the table also reports the Alpha for each evaluation indictor, which is essentially the alpha level that the category would have if that indicator were removed. Since the alpha levels for all indicators are lower than their category s alpha level, their presence in the category contributes to the concept they represent. Moreover, examination of Inter-Item correlation and Item-Rest correlation shows acceptable levels of correlations. Table 5. Reliability assessment results for Context indicators using internal consistency method. Item Obs. Sign Item-Test Correlation Item-Rest Correlation Average Inter-Item Covariance Alpha CMP PTY CONT CONT Category

16 As shown before, the evaluation indictors in the input and process categories of the proposed framework were consolidated to form a single index, Integration Assessment Index (IAI). Therefore, reliability assessment for the items in these two categories is performed in a single analysis, as they relate to one underlying construct. Table 6 shows the results of reliability assessment for these items. The review of Table 6 reveals that the Cronbach s Alpha (0.9586) is well above the minimum acceptable level. The high Cronbach s Alpha either indicates the internal consistency, and therefore, reliability of the indicators, or it may imply their redundancy; however, redundancy cannot be the cause in this case because the correlation analysis of the items does not show high correlation of the indicators. Moreover, the Item-Rest correlation is within the acceptable levels. Table 6. Reliability assessment results for Input and Process indicators using internal consistency method. Item Obs. Sign Item-Test Correlation Item-Rest Correlation Average Inter-Item Covariance Alpha TOOL TEAM COL COL COL COL COL COL COL COL COL COL COL LEAD SYS SYS Category Table 7. Reliability assessment results for Product indicators using internal consistency method. Item Obs. Sign Item-Test Correlation Item-Rest Correlation Average Inter-Item Covariance Alpha SUSC SUST SUSI LDCR SUSS Category

17 Finally, the evaluation indictors in the product category of the framework were checked for reliability. The results, as shown in Table 7, indicate that the evaluation indictors represent acceptable levels of internal consistency as judged by the Cronbach s Alpha and Item-Rest correlations. There is only one indicator in this list whose removal can improve the alpha coefficient: SUSS, i.e., the level of a project s success in achieving safety. However, because its Inter-Item correlation was within normal limits and its presence was meaningful based on the theory and literature in the field, this indicator was still kept in the model. Validity assessment: As mentioned before, construct and internal validity were checked using a regression analysis of the main hypothesis involving the key constructs/variables of the proposed framework. The hypothesis of this research was defined to be: higher integration in the ID team environment results in more successful project outcomes. Table 8. Multiple Regression Analysis Results Variable Slope Standard 95% Confidence Interval T P>t Coefficients Error Lower Upper IAI CI Constant Number of Observations = 70 F(2,67) = R-squared = Root MSE= Prob > F = Adjusted R-squared = Figure 5. Revised multiple regression model Integration Assessment Index (IAI), Challenge Index (CI) and Performance Index (PI) were used as measures of integration, challenges, and project success, respectively. The results of multiple regression analysis of the model shown in Figure 5 are displayed in Table 8. As the results indicate, the effects of integration level (IAI) and challenges (CI) on project success (PI) are statistically significant, at the 99% confidence level (p<0.01). This shows that the primary hypothesis of this research holds true. The effect of integration on project success is positive. The slope coefficient of for IAI indicates that one scale level increase of IAI would result in a increase in PI scale level, holding the level of challenges constant. In other words, one Likert-level increase, for instance from agree to strongly agree, in one of the evaluation indictors comprising the Integration Assessment Index (IAI) is expected to result in a increase in Performance Index scale level. It is important here, however, to highlight some of the psychometric limitations associated with the use of Likert-scale that need to be considered in any interpretation of the results. First of all, in a Liker-scale, it is generally assumed that the distances between successive categories are equivalent (e.g., the difference between strongly disagree and disagree is the same as the difference between disagree and neutral ). While this may not

18 be always the case, the equidistance assumption contributes to easier analysis of results. Another limitation has to do with the lack of a point of reference that often occurs in Likert-scale statements which leads to potential response inconsistency from individual to individual, even when data on the same phenomenon or object is collected. Also, there is much debate whether the Likert-scale should be treated as interval or ordinal data. When measuring the same variable, the Likert-scales are usually treated as interval. Examining the effect of the Challenge Index on Performance Index reveals that the effect is statistically significant and negative ( ; p=0.008). This indicates that increasing challenges in a green building project reduces the project s success. R-squared, the coefficient of determination, for this analysis is , which indicates that this model can explain about 92% of the changes in Performance Index. The results of the multiple linear regression analysis revealed that integration positively affects success in green building projects, when controlling for the level of challenges. This adheres with what the literature in the field suggests. Indeed, various studies in the field have theorized (7group and Reed 2009, Kibert 2008, Yudelson 2008), or qualitatively and empirically shown (Korkmaz 2007, Korkmaz et al 2013), that integration level in design process positively impacts project outcomes. In an in-depth qualitative study of 12 case-studies, Korkmaz, Swarup & Riley (2013) measured team integration level using parameters such as involvement, design charrettes, communication, compatibility, presence of LEED Accredited Professional (AP) in team, prior experience, use of energy modeling, and LEED education to contractors, and examined the relationship between project delivery systems, level of integration, and achievement of the outcomes. They proposed that higher levels of integration result in more successful outcomes with respect to sustainability, cost, schedule, etc. The regression analysis results also indicate a negative relation between the level of challenges in a green building project, as represented by the Challenge Index, and project success. Considering the fact that the Challenge Index is made of a consolidation of several factors including project uncertainty, the lack of contractual support for integration, and so forth, the results could highlight the importance of reducing the challenges level by improving contractual integration of project parties and encouraging free exchange of information. To summarize the results, the authors tested the research hypothesis that higher integration in the ID team environment results in more successful project outcomes. Also, the results helped validate the proposed integration evaluation framework based on five measures of content validity, construct validity, internal validity, reliability and external validity. The content validity was already met during development of the evaluation framework when a group of experts were asked to peer-review the framework and offer their comments and critiques. The framework was then revised to address the comments. It was ensured that the revised framework covers all factors affecting the ID process and that the evaluation indictors are adequate to measure those factors. To meet the construct and internal validity, two key strategies were applied: a. All possible factors and variables with the potential of affecting the integration of project participants in the ID process of green buildings were investigated through literature review and qualitative research, and were included in the evaluation framework. In particular, the addition of contextual factors (such as complexity, sustainability, etc.) to the evaluation framework was intended to improve internal validity by capturing aspects of change in project outcomes that are not related to the integration of project participants.