A PERFORMANCE EVALUATION MODEL FOR BUILDING INFORMATION MODELING (BIM) ORGANIZATIONS Thanit Aphiworakunphat Graduate student, Department of Civil Engineering, Faculty of Engineering, Chulalongkorn University, Phayathai Road, Pathumwan, Bangkok 10330 Thailand Veerasak Likhitruangsilp Ph.D., Associate Professor, Department of Civil Engineering, Faculty of Engineering, Chulalongkorn University, Phayathai Road, Pathumwan, Bangkok 10330 Thailand Abstract Building Information Modeling (BIM) can benefit various aspects of the construction industry, including people, processes, and technology, which eventually minimize the total operational cost. The benefits of BIM can be categorized into the tangible benefit and the intangible benefit. In the organizations that adopt BIM (called BIM organizations), it is often challenging to evaluate their enhanced performance. A main reason is that there is no systematic framework to assess the effects of using BIM on organizations. In this paper, we propose a model for evaluating the performance of BIM organizations. The paper focuses on identifying Critical Performance Indicators (CPIs) by interviewing a selected group of BIM users in the AEC organizations in Thailand. Based on the balanced scorecard (BSC) concept, we list the CPIs that reflect four main business perspectives, namely, the financial perspective, the customer perspective, the business process perspective, and the learning and growth perspective. The model is created based on the results from questionnaire surveys and in-depth interviews on relevant topics. The Key Performance Indicators (KPIs) are then concluded and used for evaluating the performance of BIM organizations. The results are then verified through in-depth interviews with a group of experts to finalize the CPIs of BIM organizations. Keywords Building Information Modeling, Critical Performance Indicator, Key Performance Indicator, Performance Evaluation I. INTRODUCTION In present, Building Information Modeling (BIM) is an important technology in the construction industry. In the pre-construction stage, BIM is useful for architects and engineers for design and constructors for planning and estimating [1]. It can generate 3D models that are easy to communicate with customers, entail minimal mistakes, and shorten design time. Per these advantages, BIM users can earn profits for their organization [2]. Even though the construction industry is increasingly using BIM, the implementation is still limited [3]. A main reason is that most AEC organizations cannot evaluate their performance once BIM is adopted [4]. Thus, it is necessary to develop a tool for evaluating the 75
performance of BIM organizations. In this paper, the model for evaluating the performance of BIM organizations is developed using Critical performance indicators (CPIs), which are identified by interviewing the BIM users in AEC organizations. Based on the balanced scorecard (BSC) concept [5] the CPIs can reflect four main business perspectives, namely, the financial perspective, the customer perspective, the business process perspective, and the learning & growth perspective. II. RESEARCH METHODOLOGY The research methodology consists of five major steps, as shown in Fig. 1. A. Review existing BIM evaluation guidelines We first review and analyze BIM performance evaluation guidelines. The results include how to evaluate the use of BIM [6], measurable returns on investment of BIM [7], how to measure a business' success [8], performance evaluation models, how to determine the reliability of indicators, and a collection of KPIs related to the performance evaluation model for BIM organizations. B. Summarize essential contents of the performance evaluation model for BIM This step is to analyze and identify the contents of the performance evaluation model for BIM according to the reviewed BIM performance evaluating guidelines. Such content encompasses important data that are required to develop a model for evaluating the performance of BIM organizations such as BIM goals, BIM uses, and BIM KPIs. C. Specify important KPIs of the evaluation model for BIM This first step is to design a questionnaire for collecting data. We specify important indicators from the reviewed BIM performance evaluation guidelines to determine BIM KPIs, which are also collected from interviews consisting of four steps: 1) Specify BIM goals of the organization. The organization must provide the expected purposes before using BIM. 2) Determine BIM uses. In this step is to determine appropriate BIM uses by considering the BIM goal. 3) Define the KPIs. The definitions of the KPIs which are related to BIM goals and BIM uses are summarized from relevant literature. The meaning of each KPI may be defined or calculated depending on each goal. 4) Classify the KPIs. The results are classified into two types: quantitative (e.g., percentage) and qualitative, involving the evaluation of time, cost, and quality. D. Analyze the data from interviews In this step, we specify the criteria of each BIM KPI, classify the KPIs into the four perspectives (as shown in Fig. 2), and analyze the KPIs by ordering and measuring their scores, as discussed above. The score measurement entails two types: quantitative and qualitative. 1) Quantitative, by using a 4-point scale: 4 (most effective), 3 (strongly effective), 2 (effective), 1 (less effective), and 0 (ineffective) 2) Qualitative, by using pass/reasonable and not pass/ unreasonable In this paper, we identify the minimum score of each KPI and each perspective from interviews by averaging the score. E. Verify the evaluation model The results are verified by the selected BIM organizations. Herein, we perform the reliability analysis [9] of the KPIs and select the CPIs by measuring a Cronbach s alpha coefficient. 76
The values on the 5-point Likert scale are used to evaluate the respondents ratings of indicators. The respondents are asked to rate the importance of these factors from 1 to 5: (1 - very unimportant, 2 - not important, 3 - neutral, 4 - important, and 5 - very important) Fig. 1 Research methodology Fig. 2 KPI classification III. RESULTS AND DISCUSSION A. BIM KPIs By reviewing documents, journals, and related theories which are necessary, we gather theories of evaluating the use of BIM and compile important factors for BIM implementation. Two main fundamental concepts for this research are as follows. 1) Important business theories are the appraisal of organization performance, the analysis of the cost and feasibility of an organization, the organizational assessment using critical success factors [8], balanced scorecard (BSC) [5]. All of these are used to design a questionnaire to gather the information about BIM goals of an organization. Each organization provides the expected purposes before using BIM and identifies BIM uses, which must relate to the purpose of BIM goals in the organization for analyzing the BIM KPIs. - BIM goals. The success of a BIM organization can be measured by first defining the goals for that organization. - BIM uses. This can be called a BIM service and function. It is a unique task in an organization where BIM is utilized to support the planning, design, construction, and operational processes. BIM uses must relate to the purpose of BIM goals and help achieve the goals, as shown in Fig. 3. - BIM KPIs. To obtain the appropriate KPIs for assessing the success of an organization, the organization's strategies must be specified. A set of KPIs may vary according to BIM goals and BIM uses, which are determined in the previous steps. Thus, a set of KPIs should be capable of determining whether or not the goals have been achieved and of measuring the effects derived when specific BIM uses are implemented. 77
shows the ranking of the results of BIM KPIs (qualitative). TABLE II Ranking BIM KPIs (quantitative) Fig. 3 Relationship of BIM goals and BIM uses 2) The reliability analysis is applied to a selfassessment questionnaire by using the correlation coefficients. The average result will be called a Cronbach s alpha coefficient. Table I shows the KPIs related to the performance evaluation of using BIM in an organization, which are collected from the literature review. TABLE I BIM KPIs and description TABLE II (Continue) Ranking BIM KPIs (quantitative) B. Ranking the BIM KPIs The results from the interviews with the BIM organizations are different from the BIM KPIs shown in Table I. According to the BIM organizations, important indicators that influence the BIM performance are identified. Table II shows the ranking of the BIM KPIs (quantitative) and Table III TABLE III Ranking BIM KPIs (qualitative) 78
C. VERIFY THE RESULTS The above results are verified by the selected BIM organizations. They are tested by the reliability measurement using a Cronbach s alpha coefficient to screen the CPIs. The alpha levels are examined for each of the four perspectives: the financial perspective, the customer perspective, the business process perspective, and the learning & growth perspective. For each of the KPIs, the accepted value of Cronbach s alpha coefficient must be 0.70 or more. IV. CONCLUSIONS This paper investigates the important indicators that influence a model for evaluating the performance of BIM organizations. The three main factors are BIM goals, BIM uses and BIM KPIs to define the BIM CPIs, which is selected by ranking the results of BIM KPIs and identify the minimum score of each CPI and each perspective. These results can be used to assess the advantages and disadvantages of using BIM in organizations. Alternatively, any organization can use these results for benchmarking with other organizations to eventually maximize the benefits from the BIM execution in the organizations. V. REFERENCES 1. Lee, G., R. Sacks, and C.M. Eastman, Specifying parametric building object behavior (BOB) for a building information modeling system. Automation in Construction, 2006. 15(6): p. 758-776. 2. Son, H., et al. The adoption of building information modeling in the design organization: An empirical study of architects in Korean design firms. in ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction. 2014. Vilnius Gediminas Technical University, Department of Construction Economics & Property. 3. Grilo, A. and R. Jardim-Goncalves, Value proposition on interoperability of BIM and collaborative working environments. Automation in Construction, 2010. 19(5): p. 522-530. 4. Won, J. and G. Lee. Identifying the consideration factors for successful BIM projects. in Proceedings of the International Conference on Computing in Civil and Building Engineering, Nottingham. 2010. 5. Norton, R.S.K.a.D.P., Using the Balanced Scorecard as a Strategic Management System. 1990. 6. Editorial.Team, A., How Do You Know if BIM is Worth The Investment For Your Firm? 2016. 7. Love, P.E.D., et al., From justification to evaluation: Building information modeling for asset owners. Automation in Construction, 2013. 35: p. 208-216. 79
8. Alias, Z., et al., Determining Critical Success Factors of Project Management Practice: A Conceptual Framework. Procedia - Social and Behavioral Sciences, 2014. 153: p. 61-69. 9. Field, A., Discovering statistics using SPSS:(and sex and drugs and rock'n'roll). 2009. 80