Exploring the Dimensions of Organizational Performance: A Construct Validity Study

Transcription

1 Feature Topic: Construct Measurement in Strategic Management Exploring the Dimensions of Organizational Performance: A Construct Validity Study Organizational Research Methods 16(1) ª The Author(s) 2013 Reprints and permission: sagepub.com/journalspermissions.nav DOI: / orm.sagepub.com P. Maik Hamann 1, Frank Schiemann 1,2, Lucia Bellora 1, and Thomas W. Guenther 1 Abstract Organizational performance is a fundamental construct in strategic management. Recently, researchers proposed a framework for organizational performance that includes three dimensions: accounting returns, growth, and stock market performance. We test the construct validity of indicators of these dimensions by examining reliability, convergent validity, discriminant validity, and nomological validity. We conduct a confirmatory factor analysis with 19 analytically derived indicators on a sample of 37,262 firm-years for 4,868 listed U.S. organizations from 1990 to Our results provide evidence of four, rather than three, organizational performance dimensions. Stock market performance and growth are confirmed as separate dimensions, whereas accounting returns must be decomposed into profitability and liquidity dimensions. Robustness analyses indicate stability of our inferences for three dissimilar industries and for a period of 21 years but reveal that organizational performance dimensions underlie dynamics during years in which environmental instability is high. Our study provides an initial contribution to the clarification of the important organizational performance construct by defining four dimensions and validating indicators for each dimension. Thus, we provide essential groundwork for the measurement of organizational performance in future empirical studies. Keywords factor analysis, quantitative research, reliability and validity, measurement models, organizational performance Organizational performance (OP) is fundamental to strategic management research. Research in this field builds on the assumption that strategy influences OP (Lubatkin & Shrieves, 1986). Furthermore, 1 Faculty of Business Management and Economics, Technische Universität Dresden, Dresden, Germany 2 School of Business, Economics, and Social Science, University of Hamburg, Hamburg, Germany Supplementary material for this article is available on the journal s website at Corresponding Author: P. Maik Hamann, Technische Universität Dresden (TU Dresden), Faculty of Business Management and Economics, Chair of Business Management especially Management Accounting and Control, D Dresden, Germany. Lehrstuhl.controlling@mailbox.tu-dresden.de

2 68 Organizational Research Methods 16(1) OP is the most common concept addressed in empirical studies in this field; for example, 28% of 439 empirical articles reviewed by March and Sutton (1997) and 29% of 722 articles reviewed by Richard, Devinney, Yip, and Johnson (2009) include OP in their research design. The OP construct refers to the phenomenon in which some organizations are more successful than others. A construct is a conceptual term that researchers define to describe a real phenomenon and is unobservable by nature (Edwards & Bagozzi, 2000). Consequently, OP is subject to the problem of unobservables in strategic management research (Godfrey & Hill, 1995, p. 519). This problem is best described in reference to the predictive validity framework (PVF). The PVF includes two levels: the conceptual level and the operational level (Bisbe, Batista-Foguet, & Chenhall, 2007). At the conceptual level, theories explain relationships between constructs through propositions. Subsequently, these propositions are empirically tested at the operational level, at which researchers apply indicators to measure a construct. Indicators are observed scores or quantified records (Edwards & Bagozzi, 2000). The link between the two levels (i.e., between constructs and their indicators) is crucial to advances in theoretical relationships between constructs. Only if this link is rigorously established can empirical findings at the operational level be used to test theoretical propositions involving unobservables at the conceptual level. This link is established by examining construct validity. Construct validity reflects the correspondence between a construct and a measure taken as evidence of the construct (Edwards, 2003, p. 329). Construct validity encompasses four criteria: reliability, convergent validity, discriminant validity, and nomological validity (Schwab, 2005). Paradoxically, in the past, a majority of strategic management researchers regarded construct validity and the measurement of constructs as low-priority topics (Boyd, Gove, & Hitt, 2005). Consequently, unobservables (e.g., OP) have often been measured by single indicators whose construct validity has rarely been assessed. From the PVF, it follows that related theoretical inferences from such studies are seriously undermined (Combs, Crook, & Shook, 2005; Starbuck, 2004; Venkatraman & Grant, 1986). Because of its importance for strategic management research, a growing number of studies examine the measurement of OP. These studies are shown in Table 1 and encompass two groups: (a) factor analyses of the dimensionality of OP (Devinney, Yip, & Johnson, 2010; Fryxell & Barton, 1990; Rowe & Morrow, 1999; Venkatraman & Ramanujam, 1987) and (b) reviews of the OP measurement practices used in strategic management research (Murphy, Trailer, & Hill, 1996; Richard et al., 2009; Tosi, Werner, Katz, & Gomez-Mejia, 2000). The first group of studies provides evidence of the multidimensionality of OP. However, these studies disagree on the number of OP dimensions and do not systematically examine the construct validity of indicators that measure these dimensions. Reviews of OP measurement practice provide evidence that empirical studies in strategic management research employ a plethora of different and unrelated indicators (Murphy et al., 1996); for example, Richard et al. (2009) reviewed 213 studies and identified 207 different OP indicators. In this review, 49% of the studies measure OP with a single indicator despite the multidimensional nature of OP, and 52% of the studies employ only cross-sectional data sets. However, none of the aforementioned studies develop a framework of the dimensions of OP at the conceptual level or examine the construct validity of OP indicators based on such a framework. Combs et al. (2005) directly address the first gap in the literature and develop a framework of the OP dimensions based on a synthesis of prior studies that focus on OP dimensions and a review of OP measurement practices. They divide OP into three dimensions: accounting returns, stock market performance, and growth. Subsequently, they test the OP framework by conducting a confirmatory factor analysis (CFA) based on a correlation matrix of five OP indicators derived from a metaanalysis. Despite the significant contribution made by Combs et al., their study has three limitations. First, Combs et al. do not offer clear definitions of the OP dimensions. Specification of the conceptual domain and clear definitions of constructs are prerequisites for construct validity (Schwab, 2005). Second, the CFA with three factors and five OP indicators does not satisfy the twoindicator rule of model identification (Kline, 2011). Consequently, Combs et al. offer only

3 Table 1. Previous Studies That Examine the Dimensions of Organizational Performance. Study Number of Dimensions Number of Indicators Number of Studies/ sample size Method Dimensions of Organizational Performance Operational Performance Reviews and meta-analytic studies Combs, Crook, and Shook 3 5 Not reported (2005) a (238 studies) Tosi, Werner, Katz, 8 30 Not reported and Gomez-Mejia (2000) a (137 studies) Richard, Devinney, Yip, and Johnson (2009) Murphy, Trailer, Narrative review and meta-analytical CFA Meta-analytical EFA Absolute financial performance Accounting returns Growth Stock market Operational performance Return on equity short term Return on equity long term Change in financial performance Stock performance Market return 3 n/a 213 Narrative review Financial performance Shareholder return and Hill (1996) b Liquidity 4 n/a 52 Narrative review Efficiency Size Profit Internal performance indicators Product market performance Studies using primary or secondary data Devinney, Yip, and Johnson (2010) Rowe and Morrow (1999) (2,398 firm-years) Murphy et al. (1996) b 9 (4) 19 (8) 995 (586) 4 10 Not reported EFA Accounting measure Sales measures (sales growth) Cash flow/profitability dimension Market value CFA Financial (accounting) Stock market Subjective reputation rating PCA (CFA) Fryxell and Barton (1990) CFA Venkatraman and Ramanujam (1987) Liquidity Sales measures Profitability (sales growth) Sales efficiency Profit growth Income efficiency Absolute income Employee efficiency Accounting-based measures MTMM and CFA Profitability Sales growth Profit growth Market-based measures Size Note: We allocate the dimensions of organizational performance to the framework of Combs et al. (2005), which is shown in boldface. This framework also separates operational performance and organizational performance. CFA ¼ confirmatory factor analysis; EFA ¼ exploratory factor analysis; PCA ¼ principal components analysis; MTMM ¼ multitrait-multimethod matrix. a Combs et al. (2005) and Tosi et al. (2000) only report the overall number of primary studies that they use in their reviews. This number is provided in parentheses. b Murphy et al. (1996) conduct a narrative review and an empirical analysis that is based on the results of their review. Consequently, we include this study in both lists. Furthermore, the results of their exploratory PCA and their CFA are different insofar as the CFA encompasses only a subset of the indicators that are employed in the PCA. We present details pertaining to their CFA in parentheses. 69

4 70 Organizational Research Methods 16(1) preliminary empirical evidence for their proposed OP framework. Third, Combs et al. do not test the construct validity of OP indicators, which represent their conceptual framework at the operational level of the PVF. We address the limitations of Combs et al. (2005). First, we discuss the semantic relationships of OP to related performance constructs and clearly define the OP dimensions. Second, we empirically test Combs et al. s proposed OP framework. Third, we examine the construct validity of indicators of the OP dimensions by addressing reliability, convergent validity, discriminant validity, and nomological validity. Our study closes the remaining gap in the literature because it proposes a measurement scheme for OP at the operational level and systematically examines its construct validity. To achieve this aim, we conduct a CFA based on the following: (a) an analytically derived set of 19 OP indicators; (b) secondary, objective OP data; and (c) a large sample of listed U.S. organizations. Our sample consists of 37,262 firm-years for 4,868 listed U.S. organizations from three dissimilar industries (industrial, consumer services, and technology) over a 21-year period beginning in We also analyze the robustness of our results for each of the 21 annual data sets and for each of the three industries. Our empirical results demonstrate that growth and stock market performance depict distinct dimensions of OP. In contrast to Combs et al. (2005), the accounting returns dimension must be decomposed into the two dimensions of liquidity and profitability. For each of the four OP dimensions, we identify a set of indicators that provides a reliable and construct-valid measurement scheme of the related OP dimension. Our robustness analyses provide strong evidence of stable inferences of the construct validity of the four-dimensional OP measurement scheme across both time and industries. However, during periods of high environmental instability (e.g., after the burst of the dotcom bubble in 2002 and during the financial crisis beginning in 2008), the fourdimensional measurement model s fit to the data is weakened. Our study is among the few in strategic management that directly address the link between the conceptual level and the operational level of an important construct in the field. We concur with Venkatraman (2008) in that it is this type of attention to the details of construct operationalization that is needed in strategy research (p. 791) and that it will thus increase the rigor of future research. We contribute to the literature on OP in three ways. First, we contribute to the clarity of the OP construct in terms of definitions, semantic relationships, contextual conditions, and coherence (Suddaby, 2010). 1 Second, we propose a construct-valid measurement scheme of OP and its four dimensions. Researchers are encouraged to apply the measurement scheme in future empirical studies. Third, we highlight the importance of measuring OP using longitudinal data. Cross-sectional OP data may be prone to weak construct validity during years of high environmental instability. Thus, researchers employing cross-sectional OP data should carefully evaluate the validity of their measurement. Dimensions of Organizational Performance The conceptual domain of OP can be specified only by relating this construct to the broader construct of organizational effectiveness. Organizational effectiveness is defined as the degree to which organizations are attaining all the purposes they are supposed to (Strasser, Eveland, Cummins, Deniston, & Romani, 1981, p. 323). Organizations obtain different effectiveness assessments based on diverse constituencies. Therefore, organizational effectiveness encompasses OP and other performance concepts (i.e., corporate environmental or social performance), which are relevant for practice and research. In the strategic management literature, researchers concentrate on operational performance and OP (Venkatraman & Ramanujam, 1986). Operational performance refers to the fulfillment of operational goals within different value chain activities that may lead to subsequent OP (Combs et al.,

5 Hamann et al ). Common performance indicators, such as growth in market share, product quality, patent filings, or marketing effectiveness, measure distinct dimensions of operational performance. In contrast, OP is defined as the economic outcomes resulting from the interplay among an organization s attributes, actions, and environment (Combs et al., 2005, p. 261). The definition of OP corresponds to measurement practices in strategic management research because a majority of researchers assess OP based on economic indicators (Murphy et al., 1996; Richard et al., 2009). Thus, OP is synonymous with the concepts of financial performance or corporate economic performance (Fryxell & Barton, 1990). OP is relevant to both research and practice because in the legal system (i.e., bankruptcy law or commercial law) and in economic theory, OP (i.e., economic outcomes) constitutes the final aim of economic activities. Combs et al. (2005) propose a consistent OP framework with three dimensions: accounting returns, stock market performance, and growth. Accounting returns are defined as the historical performance of organizations that is assessed through the use of financial accounting data as published in annual reports (Fryxell & Barton, 1990). As shown in Table 1, Combs et al. (2005) argue for a single accounting returns dimension, whereas other studies identify several dimensions that are derived from accounting returns indicators. However, we expect at least two separate dimensions to be reflected by accounting returns indicators. First, a liquidity dimension, which is defined as a firm s ability to meet its financial obligations based on cash flows generated from its current operations, is expected (Weygandt, Kimmel, & Kieso, 2010). Second, a profitability dimension, defined as an organization s efficiency in utilizing production factors to generate earnings, is expected. Accounting research highlights the difference between earnings (e.g., net profit) and cash flows that is traced to revenue and expense accruals (e.g., Dechow, 1994). Accruals mitigate timing and matching problems associated with the allocation of cash flows to single periods but are subject to distortions caused by discretionary accounting choices (e.g., a depreciation method or the useful life of assets). Additionally, Rappaport (1993) stresses the divergence between the accounting-based return on investment and the cash flow rate of return. Stock market performance reflects the perceptions of investors regarding organizations future performance (Fryxell & Barton, 1990). This dimension is measured using capital market indicators, such as total shareholder return (TSR). However, capital market indicators are also influenced by the momentum and volatility of capital markets, the economy, and psychological effects (Richard et al., 2009). Stock market performance reflects future OP, in contrast with accounting returns, which entail a historical perspective. As shown in Table 1, previous studies provide consistent evidence regarding stock market performance as a distinct OP dimension. Organizational growth is defined as a change in an organization s size over time. Organizational growth is a dynamic construct that is commonly evaluated based on three concepts of size: sales, employees, and assets (Weinzimmer, Nystrom, & Freeman, 1998). As shown in Table 1, previous studies that investigate the OP dimensions focus on sales growth and disregard employment and asset growth. Previous examinations of the dimensionality of OP are subject to three limitations. First, the number of indicators used is small. For example, Fryxell and Barton (1990) use four indicators, and Venkatraman and Ramanujam (1987) employ three indicators. However, a small number of indicators may not capture the entire conceptual domain of a construct. Second, indicators are often not chosen analytically. For example, Murphy et al. (1996) chose 19 OP indicators based on their frequent usage by researchers. These indicators include absolute returns (e.g., net income), return ratios (e.g., return on assets), size (e.g., number of employees), and ratios of balance sheet items (e.g., debt to equity). Given the conceptual domain of OP, the adequacy of some of these indicators is questionable; for example, size and static balance sheet items differ conceptually from OP (Combs et al., 2005; Tosi et al., 2000). If indicators are chosen inadequately, spurious factors may emerge or true

6 72 Organizational Research Methods 16(1) factors may be obscured in factor analyses (Fabrigar, Wegener, MacCallum, & Strahan, 1999). Third, cash flow return indicators are absent in the majority of previous studies. Devinney et al. (2010) and Rowe and Morrow (1999), who both include a single cash flow return indicator in their factor analysis (cash flow return on sales and cash flow return on equity, respectively), are exceptions. This limitation is important because we expect the single accounting returns dimension proposed by Combs et al. (2005) to divide into two dimensions (i.e., liquidity and profitability) when the convergence of cash flow returns and profitability indicators is examined systematically. Research Design Assessment of Construct Validity During the process of construct validation, four criteria are evaluated: reliability, convergent validity, discriminant validity, and nomological validity (Schwab, 2005). We employ CFA to examine construct validity. First, the theory-testing approach of CFA is appropriate for the evaluation of the two competing models, the three-op dimension model and the four-op dimension model, that emerged from our discussion of previous research. Second, this approach enables an examination of the overall fit of a measurement model to a data set. Third, CFA permits researchers to test the significance of factor loadings. Fourth, CFA supplies indices that provide insights into reliability, convergent validity, and discriminant validity (Bagozzi, Yi, & Phillips, 1991; O Leary-Kelly & Vokurka, 1998). Table 2 presents the methods and indices that are applied to assess the criteria of construct validity (see also Bagozzi & Yi, 1988). Prior to the assessment of the construct validity criteria in a CFA, the overall fit of the measurement model to the data must be established (Anderson & Gerbing, 1988). The assessment of the overall measurement model fit to the data is based on the chi-square statistic, the Comparative Fit Index (CFI), the root mean square error of approximation (RMSEA), the standardized root mean square residual (SRMR), and the Akaike s Information Criterion (AIC). The methodological literature criticizes the use of definite cutoff criteria for these goodness-of-fit indices. Goodness-of-fit indices are sensitive to the misspecification of a model and to sample size, model types, and data non-normality. Consequently, definite cutoff criteria may yield a high Type I error (i.e., rejecting acceptable misspecified models) if they are too conservative (Marsh, Hau, & Wen, 2004). We account for this cutoff criteria ambiguity by differentiating between cutoff criteria for acceptable and good fits of the measurement model to the data and by reporting more than one goodness-of-fit index, as recommended by Hu and Bentler (1999). 2 We compare the competing models of OP based on their overall measurement model fit to the data. Hereafter, we employ the best fitting model to examine the four criteria of construct validity. Reliability is defined as the ratio of systematic variance to total variance (i.e., the degree to which an indicator is free of random error). Reliability is a necessary prerequisite for validity (Schwab, 2005). Convergent validity is defined as the extent to which multiple indicators represent a common construct. A number of indicators of the same construct should exhibit high levels of covariance to be considered valid measures of the construct in question (Bagozzi et al., 1991). In contrast, discriminant validity is defined as the degree of divergence among indicators that are designed to measure different constructs (Edwards, 2003). The methods that we apply to assess these criteria of construct validity are presented in Table 2. Nomological validity is based on evidence pertaining to the relationships between measures of the construct under investigation and measures of other constructs. This evidence should be consistent with relevant theory or with the results of previous empirical studies (Schwab, 2005). Consequently, we test the relationships between the dimensions of OP and the determinants and consequences of OP. Capon, Farley, and Hoenig (1990) conducted a meta-analysis of the

7 Table 2. Statistics and Methods That Are Applied to Assess Construct Validity. Steps in Assessing Construct Validity Assessment Criteria Explanation and Thresholds for Acceptability Overall fit of the measurement model to the data Chi-square statistic of the likelihood ratio test H0 hypothesis of the likelihood ratio test is the exact fit of a specified model to a population (MacCallum, Browne, & Sugawara, 1996, p. 132). Acceptance of H0: p value >.05. Comparative Fit Index (CFI) The CFI describes the relative improvement in the fit of the model in comparison with the fit of the Root mean square error of approximation (RMSEA) Standardized root mean square residual (SRMR) Akaike s information criterion (AIC) independence model. Thus, this index overcomes sample size effects (Bentler, 1990, pp ). Acceptable fit: CFI >.90; good fit: CFI >.95. The RMSEA measures the discrepancy between the covariance matrix estimated from the model and the observed matrix. This criterion adjusts for the model degrees of freedom (MacCallum et al., 1996, p. 134). Acceptable fit: RMSEA <.08; good fit: RMSEA <.05. The SRMR measures the mean overall difference between observed and predicted correlations (Hu & Bentler, 1999, p. 1). Acceptable fit: SRMR <.08; good fit: SRMR <.05. AIC is a predictive fit index that measures model fit based on the model s capacity to be replicated in future samples. This criterion considers the model degrees of freedom and allows for a comparison of different non-nested measurement models; lower values indicate a better fit (Akaike, 1974, p. 716). Reliability Item reliability Item reliability is examined using the R 2 value that is associated with each indicator to factor equation. This criterion measures the strength of the linear relationship between an indicator and its latent factor (Bagozzi & Baumgartner, 1994, p. 402). Acceptable item reliability: R 2 >.4. Construct reliability Construct reliability represents the proportion of systematic variance in a set of indicators (Bagozzi & Baumgartner, 1994, p. 403; Edwards, 2003, pp ). Acceptable construct reliability >.6. Average variance extracted Average variance extracted measures the amount of variance in a set of indicators that is accounted for by the latent factor in the model (Fornell & Larcker, 1981, pp ). Acceptable average variance extracted >.5. Convergent validity Standardized factor loadings Factor loadings with the theoretically predicted sign, an estimate above.5 (acceptable convergence) or above.7 (good convergence), and statistical significance constitute evidence of convergence (Carlson & Herdman, 2010, p. 1). (continued) 73

8 Table 2. (continued) Steps in Assessing Construct Validity Assessment Criteria Explanation and Thresholds for Acceptability Discriminant validity Fornell-Larcker criterion The average variance extracted for a factor is compared with all squared correlations of this factor with other factors in the overall measurement model. If the average variance extracted is greater than the squared correlations in all cases, this result is a strong indicator of discriminant validity (Fornell & Larcker, 1981, p. 46). Chi-square difference test The model fits of two measurement models are compared for each possible combination of pairs of factors. In the first model, the correlation between the two factors is constrained to 1.0, whereas this correlation parameter is freely estimated in the second model. Finally, a chi-square difference test between the chi-square values of these two models is performed. A statistically significant difference indicates adequate discriminant validity (Anderson & Gerbing, 1988, p. 416). Nomological validity a Antecedents and consequences We model research and development intensity ( ), capital investment intensity ( ), market concentration ( ), and market share (þ) as antecedents and survival (þ) as a consequence of organizational performance in a structural equation model. a The expected signs are given in parentheses. Capon, Farley, & Hoenig (1990); Lee (2009); and Bercovitz and Mitchell (2007, p. 72) provide empirical evidence for our analysis of nomological validity. 74

9 Hamann et al. 75 determinants of financial performance. Their findings are based on 320 primary studies and provide the most robust results pertaining to the determinants of OP. Four of the constructs that Capon et al. examine are applicable in our analysis to establish nomological validity: research and development (R&D) intensity (measured as R&D expenses divided by sales), capital investment intensity (measured as capital investment divided by sales), market concentration (measured as the Herfindahl index), and market share (measured as an organization s sales divided by the total sales in its industry). Lee (2009) conducted a study on the determinants of firm performance (measured as the return on assets) in a sample of 7,158 listed U.S. organizations and replicated the findings of Capon et al. regarding these four constructs. Furthermore, OP has been found to strongly influence organizations long-term business survival (Bercovitz & Mitchell, 2007). Thus, we include survival as a consequent construct in our analysis to establish nomological validity. 3 Identification of Indicators We apply a systematic approach to derive a set of OP indicators, which are presented in Table 3. First, we identify a number of OP indicators that have been used in prior empirical studies, as reviewed by Combs et al. (2005) and Richard et al. (2009). These indicators are typically ratios that measure accounting returns. Accounting returns ratios consist of a numerator that expresses the accounting outcome and a denominator that indicates an organization s size. Second, we insert these previously applied indicators in a matrix by presenting the accounting outcomes in rows and the size measures in columns. Third, we add indicators to complete the matrix by systematically considering all possible combinations of accounting outcomes and size measures. Previous factor analyses of OP indicators do not systematically cover all of these combinations. In this matrix, we derive growth indicators from all size measures and indicators of accounting outcome change. In addition, we employ all stock market performance indicators, as identified by Combs et al. (2005). Our set of indicators includes six hybrid indicators (see footnote a in Table 3). Hybrid indicators address overlapping areas of two or more OP dimensions. As shown in Table 3, our set of indicators has three important characteristics. First, the models that are derived from our set of indicators satisfy the two-indicator rule of model identification for CFA (Kline, 2011). Second, we directly examine the consequences of the inclusion of hybrid indicators. Hybrid indicators may represent a parsimonious manner of measuring two OP dimensions with a single indicator, but they lead to the interpretational confounding of these dimensions (Burt, 1976). Consequently, Combs et al. (2005) recommend the avoidance of hybrid indicators but do not test this avoidance empirically. For each of the two competing OP frameworks (i.e., the three-op dimension model and the four-op dimension model), we define two submodels. The first submodel includes all 19 OP indicators, whereas the second submodel excludes the six hybrid indicators. Thus, we perform CFAs for four competing models. We expect a lower fit to the data for the models with hybrid indicators because hybrid indicators are subject to confounded measurement (Burt, 1976). Third, we are confident that our set of indicators is content-valid regarding the dimensions of OP. Content validity is a prerequisite for construct validity and refers to the extent to which an indicator represents the conceptual domain of a construct (Edwards, 2003). Content validity is established by (a) defining the construct of interest and (b) selecting indicators that are theoretically and logically connected to the construct. We developed definitions of the OP dimensions in the previous section, and we discuss the relations between these OP dimensions and our set of indicators in the following section. Accounting returns are measured using eight indicators that are related to different aspects of this dimension. Employees, assets, and the market value of equity constitute conceptually different production factors. Sales provide a basis on which to examine economic product market success. Cash

10 Table 3. Set of Indicators That Are Used for Construct Validation. OP Dimensions Accounting Outcomes Size Measures Employees Sales Assets Market Value Change in Accounting Outcomes Accounting returns Cash flow Cash flow return per employee Net income Return per employee Growth Employment growth Cash flow return on sales Return on sales Sales growth Cash flow return on assets Return on assets Assets growth Cash flow return on Cash flow market value a growth a Return on Income market value a growth a Market value growth b Stock market performance Total shareholder return, Sharpe ratio, Jensen s alpha, Treynor index, Tobin s Q a, and market-to-book ratio a Note: Indicators are presented in boldface if they have been used in empirical studies that measure organizational performance (OP), as reviewed by Combs, Crook, and Shook (2005) and Richard, Devinney, Yip, and Johnson (2009). The remaining indicators have been developed to complete the matrix analytically. Indicators are italicized if they have been employed in previous factor analyses of OP indicators (see Table 1). a We exclude hybrid indicators from two of the four competing models because these indicators may conflate two of the OP dimensions. Thus, we are able to directly investigate the effect of hybrid indicators in the proposed models. b We exclude market value growth from our analysis because total shareholder return (TSR) is defined as the sum of change in share price plus any dividends paid out in relation to the previous year s share price. Consequently, TSR and market value growth are highly correlated (r ¼.964; p <.001). 76

11 Hamann et al. 77 flow represents liquidity generated from operating activities, and net profit represents the profitability of these activities. 4 Consequently, we are able to examine whether the four cash flow return indicators and the four net income return indicators belong to a single accounting returns dimension or to two distinct dimensions: a liquidity dimension and a profitability dimension. Absolute accounting return measures are excluded from the CFA because they reflect size. Consequently, absolute accounting return measures loaded together with absolute size measures on a single factor in an exploratory factor analysis conducted by Tosi et al. (2000). The two indicators that are based on the market value of equity also address the stock market performance dimension and are thus hybrid indicators. The growth dimension is measured by five indicators. 5 Three of these indicators denote major size concepts, namely, sales, employment, and assets (Weinzimmer et al., 1998), and two indicators represent changes in accounting outcomes (cash flow growth and net income growth). The last two indicators are hybrids because they address the growth dimension and either the accounting return dimension or the liquidity and profitability dimensions. Stock market performance is measured by six indicators that are divided into two groups. The first group of four indicators represents change in the market perceptions of an organization s value during a specific time period. Three of these indicators (Jensen s alpha, the Sharpe ratio, and the Treynor index) relate the stock market return of a share to its risk (Combs et al., 2005). TSR represents a shareholder s gain over a given time period. TSR is the change in share price plus dividends relative to the previous year s share price. The second group of two indicators represents ratios of market value to book value. Tobin s Q is calculated as the ratio of the market value of an organization s equity plus the book value of its liabilities to the book value of its total assets. Market-to-book ratio represents the market value of equity in relation to the book value of equity (Richard et al., 2009). Rappaport (1993) criticized the market-to-book ratio, stating that it is an unreliable proxy for stock market performance because it can be influenced by distortions of book value caused by discretionary accounting choices. Because these two indicators capture the accounting perspective and the stock market dimension, we classify them as hybrid indicators in the sense of Combs et al. (2005). Sample Description Our sample consists of 37,262 firm-years for 4,868 U.S. organizations from three industries over a period of 21 years beginning in We selected organizations from three dissimilar industries based on the Industry Classification Benchmark: industrial, consumer services, and technology. These three industries offer a sufficient sample size for both our primary and robustness analyses. Furthermore, these three industries differ in their customer structure, business models, and production technology. We analyzed all U.S. organizations that belong to the three industries and are listed on the capital market within the specified time period. Following the recommendations of Dess and Robinson (1984), we used secondary, objective data. We obtained capital market data from the Thomson Reuters Datastream database and accounting data from the Thomson Reuters Worldscope database. In our sample, all firm-years with missing values are deleted. For each indicator, we winsorized the topmost and bottommost percentiles (by year and industry). We standardized all indicators by year and industry to a mean of zero and a standard deviation of one. Because of the longitudinal nature of our sample, the observations are not independent. The univariate skewness and kurtosis indicate non-normality for the standardized indicators in our sample. 6 We employ Mplus (version 6.1) for our analyses because this software allows us to conduct single-level CFAs that are robust with regard to the non-normality and the non-independency of observations. To do so, we cluster firm-years by firm and employ the Mplus COMPLEX analysis and the MLR estimator. 7

12 78 Organizational Research Methods 16(1) Results Comparison of the Competing CFA Models As shown in Table 4, we conducted CFAs for four competing models: (a) a three-factor model with hybrid indicators (3FM-A), (b) a three-factor model without hybrid indicators (3FM-B), (c) a fourfactor model with hybrid indicators (4FM-A), and (d) a four-factor model without hybrid indicators (4FM-B). Thus, we are able to compare the models for three OP dimensions and those for four OP dimensions, as well as the models for the inclusion of hybrid indicators and those for the exclusion of hybrid indicators. First, the comparison of the three-factor models (3FM-A and 3FM-B) and the four-factor models (4FM-A and 4FM-B) provides evidence of the superiority of the four-factor structure. The chisquare differences between nested models with hybrid indicators (Dw 2 ¼ 16,443.36; p <.001) and nested models without hybrid indicators (Dw 2 ¼ 13,368.36; p <.001) are both significant. Directly comparing the AICs for the pairs of nested models (3FM-A vs. 4FM-A and 3FM-B vs. 4FM-B) indicates an improved fit of the measurement model to the data for the four-factor models. In addition, neither the 3FM-A nor the 3FM-B demonstrate acceptable fit to the data, based on CFI, RMSEA, and SRMR. Second, the comparison of models with hybrid indicators (3FM-A and 4FM-A) and models without hybrid indicators (3FM-B and 4FM-B) provides strong support for Combs et al. s (2005) recommendation to avoid hybrid indicators when measuring OP. AIC is lowest for the two models without hybrid indicators (3FM-B and 4FM-B). On the one hand, including hybrid indicators in measurement models increases the number of non-zero covariances in the sample covariance matrix. On the other hand, in the model-implied covariance matrix, only some of these covariances (but not all) are freed to be estimated by modeling cross-loadings of hybrid indicators toward different factors (Burt, 1976). This issue increases the level of misspecification in such models, which is made evident by the goodness-of-fit indices. In addition, most factor loadings of the six hybrid indicators in models 3FM-A and 4FM-A are below.5. This result indicates low convergence of hybrid indicators with indicators directly measuring OP dimensions. Tobin s Q (l 3FM-A ¼ l 4FM-A ¼.361) and marketto-book ratio (l 3FM-A ¼ l 4FM-A ¼.249) also exhibit low convergence with other stock market performance measures. This last finding underscores Rappaport s (1993) criticism of OP indicators that inappropriately couple an accounting criterion with a stock market criterion. Third, all models display significant chi-square statistics. Thus, all models exhibit some degree of misspecification. This misspecification arises partially from fixing the majority of parameters (e.g., cross-loadings of indicators to factors other than their primary factor) in all models to a value of zero to develop a parsimonious measurement model of OP. However, correlations between the OP indicators, the sample covariance matrix of these indicators, and the modification indices indicate that small cross-loadings (i.e., l <.2) exist for all OP indicators. Thus, we examine whether the goodness-of-fit indices imply that the level of misspecification in the model is acceptable. In this respect, only 4FM-B exhibits a good fit to the data. All of the fit statistics meet the cutoff criteria for good fit (CFI ¼.950, RMSEA ¼.042, SRMR ¼.043). In addition, among the four models, the AIC is lowest for this model (AIC ¼ 1,012,209). These findings suggest the superiority of 4FM-B in comparison with the other models in terms of both fit and parsimony. Thus, we employ 4FM-B to examine reliability, convergent validity, discriminant validity, and nomological validity. Construct Validity of Model 4FM-B As shown in Table 5, all factors in 4FM-B demonstrate evidence of reliability with regard to all three criteria. The item reliabilities of all indicators are above.4, whereas the sales growth indicator has the lowest value for item reliability (.440). The values for construct reliability and average variance

13 Hamann et al. 79 Table 4. Comparison of Single-Level Confirmatory Factor Analysis Models. Indicators 3FM-A 3FM-B 4FM-A 4FM-B Standardized factor loadings Accounting (ACC) Cash flow return per employee.536***.552*** Cash flow return on sales.611***.639*** Cash flow return on assets.652***.610*** Cash flow return on market value.349*** Return per employee.845***.882*** Return on sales.805***.845*** Return on assets.919***.865*** Return on market value.624*** Cash flow growth.113*** Income growth.297*** Liquidity (LIQ) Cash flow return per employee.766***.805*** Cash flow return on sales.756***.832*** Cash flow return on assets.899***.832*** Cash flow return on market value.637*** Cash flow growth.248*** Profitability (PRO) Return per employee.844***.889*** Return on sales.771***.829*** Return on assets.919***.865*** Return on market value.637*** Income growth.300*** Growth (GRO) Employment growth.683***.688***.683***.687*** Sales growth.663***.662***.664***.663*** Assets growth.801***.799***.801***.799*** Cash flow growth.034***.032*** Income growth.063***.066*** Stock market performance (SMA) Total shareholder return.980***.979***.980***.980*** Sharpe ratio.850***.851***.849***.850*** Jensen s alpha.833***.834***.833***.833*** Treynor ratio.881***.882***.881***.882*** Tobin s Q.361***.361*** Market-to-book ratio.249***.249*** Cash flow return on market value.015*.047*** Return on market value.107***.102*** Correlations between factors ACC with GRO.238***.217*** ACC with SMA.177***.162*** GRO with SMA.227***.219***.227***.219*** LIQ with PRO.722***.750*** LIQ with GRO.093***.099*** LIQ with SMA.149***.129*** PRO with GRO.236***.216*** PRO with SMA.182***.163*** Fit statistics Chi-square (df) 35,040*** (141) 17,047*** (59) 18,597*** (139) 3,678*** (56) CFI (>.95) RMSEA (<.05) SRMR (<.05) AIC 1,656,857 1,056,041 1,601,775 1,012,209 Note: n ¼ 37,262 firm-years. The four models test the factor structure three factors (3FM) versus four factors (4FM) and the inclusion of hybrid indicators (A) and the exclusion of these indicators (B). Residuals of indicators that share the same denominator (e.g., sales or assets) are allowed to correlate in all four models for arithmetic reasons. Hybrid indicators are presented in italics. Fit statistics that indicate good model fit to the data are presented in boldface. Cutoff criteria of good model fit to the data are presented in parentheses for each goodness-of-fit index. CFI ¼ Comparative Fit Index; RMSEA ¼ root mean square error of approximation; SRMR ¼ standardized root mean square residual; AIC ¼ Akaike s Information Criterion. *p <.05. ***p <.001.

14 80 Organizational Research Methods 16(1) Table 5. Evaluation of Construct Validity Based on the 4FM-B. Factor Variable Liquidity Profitability Growth Stock Market Performance Item reliability (>.40) a Cash flow return per employee.648*** Cash flow return on sales.692*** Cash flow return on assets.692*** Return per employee.791*** Return on sales.687*** Return on assets.748*** Employment growth.472*** Sales growth.440*** Assets growth.638*** Total shareholder return.960*** Sharpe ratio.723*** Jensen s alpha.695*** Treynor ratio.777*** Reliability of constructs Construct reliability (>.60) a Average variance extracted (>.50) a Discriminant validity: Fornell-Larcker criterion b Liquidity.677 Profitability Growth Stock market performance Discriminant validity: Chi-square difference test c Liquidity 3,241.46*** (3) 8,706.33*** (3) 9,386.71*** (3) Profitability 6,953.09*** (3) 11,473.98*** (3) Growth 14,101.30*** (3) Nomological validity: Antecedent constructs d Research and development intensity ( ).227***.271***.043***.012* Capital investment intensity ( ).097***.112***.051***.006 Market concentration ( ).022**.018*.034***.012** Market share (þ).053***.054***.014***.013*** Nomological validity: Consequent constructs Survival (þ).022**.040*** *** Note: n ¼ 37,272 firm-years. a The thresholds for item reliability, construct reliability, and average variance extracted are given in parentheses. b The Fornell-Larcker criterion of discriminant validity is satisfied if the average variance extracted for a factor is greater than its squared correlations with all other factors. The average variance extracted is presented on the diagonal. c The chi-square difference test is performed between two two-factor models. In the first model, the correlation between the two factors is constrained to 1.0. In the second model, this correlation is freely estimated. A significant chi-square difference indicates discriminant validity. Differences in degrees of freedom are given in parentheses. d Nomological validity is tested in the 4FM-B. In this model, all measures of the antecedent constructs are regressed on each single performance dimension, and each single performance dimension is regressed on survival. Comparative Fit Index (.934), root mean square error of approximation (.036), and standardized root mean square residual (.042) indicate the fit of this model. The expected signs are given in parentheses. The regression coefficients are presented in boldface if they are statistically significant and display the expected sign. *p <.05. **p <.01. ***p <.001.

15 Hamann et al. 81 extracted are above the thresholds for all factors, with the lowest values,.761 and.517, respectively, determined for the growth dimension. As Table 4 shows, 4FM-B provides evidence of convergent validity for all factors. First, the factor loadings of all indicators exhibit acceptable convergence (i.e., l >.5). Second, if we consider the stronger criteria of good convergence (i.e., l >.7), all indicators of the liquidity, profitability, and stock market performance factors are above this threshold. The convergence of the three growth indicators is slightly weaker. The factor-loading estimate of the employment growth indicator (l ¼.687) and the sales growth indicator (l ¼.663) are both statistically significant but slightly below the threshold for good convergence. As shown in Table 5, all factors in model 4FM-B demonstrate discriminant validity. The Fornell- Larcker criterion holds for all factors. The chi-square difference tests, which compare fixed and freely estimated two-factor models for all pairs of factors, support this conclusion. As Table 4 shows, the liquidity and profitability factors exhibit the highest correlation (r ¼.750) among the four factors. All other correlation coefficients are considerably lower (i.e., r <.25). These results indicate four dimensions of the OP construct. As shown in Table 5, our analyses provide evidence of nomological validity with regard to the four OP dimensions. Ten out of 16 regression coefficients of the antecedents of OP are statistically significant and display the expected signs, according to Capon et al. s (1990) meta-analysis of 320 primary studies. The majority of OP indicators that are included in their meta-analysis belong to the profitability and liquidity dimensions. Accordingly, all regression coefficients between the four determinants and these two dimensions are statistically significant and in the expected directions. Three regression coefficients of survival for the OP dimensions are significantly different from zero and show the expected (i.e., positive) signs (liquidity, profitability, and stock market performance). However, growth appears to be unrelated to the survival of companies in our sample. Robustness Analyses Table 6 presents the results of our robustness analyses. Our inferences regarding the construct validity of 4FM-B are stable across industries and time periods. We repeat our primary analyses of the construct validity for each industry and each year separately. The overall model fit is acceptable for all three fit indices in all three industries and in 18 out of 21 years. The model fit is lowest for years with high environmental instability, as indicated by the volatility and the annual return of the S&P 500 index (e.g., in 2002, after the burst of the dotcom bubble, and in 2008 and 2009, during the financial crisis). In particular, sales growth and employment growth fail to demonstrate item reliability and convergence for years with high environmental instability. With regard to the growth dimension, only the asset growth indicator demonstrates acceptable values for reliability and convergence for all years. The average variance extracted for the growth dimension is below the threshold for almost every year after 2000 except 2005, indicating a weak construct reliability for the last 11 years. The discriminant validity of the dimensions of OP is evident in all industries. However, for five of the years, the correlation coefficient between the profitability and liquidity dimensions is high (i.e., r >.8). Consequently, during these years, the two dimensions do not discriminate as strongly as implied by the primary analysis. Overall, our results generally remain unchanged across industries and time periods. Discussion and Conclusions The results of this study reveal the existence of four independent OP dimensions: liquidity, profitability, growth, and stock market performance. The evidence of the construct validity of the fourdimensional OP measurement scheme is strong and consistent across different time periods and