An Investigation into the Use of Nonfinancial Performance Indicators. by Financial Analysts

Transcription

1 An Investigation into the Use of Nonfinancial Performance Indicators by Financial Analysts Paul J. Coram Senior Lecturer The University of Melbourne Theodore J. Mock Arthur Andersen & Co. Alumni Professor of Accounting The University of Southern California and the University of Maastricht Gary S. Monroe Professor The Australian National University 5 July 2006 Acknowledgments: We would like to thank George Klersey and David Woodliff, participants at the 2004 Accounting and Finance Association of Australia and New Zealand Conference and the 2005 Australian National Centre for Audit and Assurance Research Annual Audit Research Forum, and workshop participants at The University of Melbourne and Deakin University for insightful comments on earlier versions of this paper. Finally, we thank the financial analysts who participated in this study. Data availability: Data and the instrument used in this study are available on request.

2 An Investigation into the Use of Nonfinancial Performance Indicators by Financial Analysts ABSTRACT: This study examines whether nonfinancial performance indicators impact financial analysts judgments and decision making when assessing company performance and the associated effect of assurance. It also assesses whether the financial information is positive or negative trending results in an asymmetric response by analysts. We address these research questions through a verbal protocol study that examines the information processing strategies and types of information utilized by analysts in valuing a company. Results suggest that there was significant use of nonfinancial performance indicators in performing the company valuation and that financial statements, and in particular the income statement, were the main source of information utilized. Also, there was some evidence of use of the assurance reports, however, there was no significant association found between provision of assurance reports and utilization of nonfinancial performance indicators. Results also showed an asymmetric response to the positive and negative financial information, which resulted in differential utilization of the financial information and nonfinancial performance indicators. Financial information was more utilized when it was negative and nonfinancial performance indicators were more utilized when the financial information was positive. Keywords: nonfinancial performance indicators; assurance; valuation; financial analysts. 2

3 1. INTRODUCTION This study uses concurrent verbal protocols to assess whether nonfinancial performance indicators impact users judgments and decision making when assessing company performance and the associated effect of assurance. In addition, the study examines factors that may affect the level of utilization of this information. In particular, the study examines whether the signal from the financial information (positive or negative) affects the utilization of the nonfinancial performance indicators and also whether provision of an assurance report affects the level of use and reliance on the nonfinancial performance indicators. More generally, the study assesses the decision making processes of analysts in valuing a company and the type of information they utilize in this process. This study represents the first time the verbal protocol research methodology (and NVivo ) has been applied to examine how analysts process nonfinancial performance indicators in making valuation judgments and decisions. The importance of this issue is underpinned by the fact that the American Accounting Association Financial Accounting Standards Committee (AAA FASC) (2002) performed a review of the research literature relating to disclosure of nonfinancial performance measures. They concluded that although the evidence suggested there was value in nonfinancial disclosures, the practice of this disclosure was varied, unstructured and uncommon. Therefore, they suggested that further research into this issue was warranted. The AAA FASC acknowledged that regression studies have found an association between certain types of nonfinancial disclosures and stock prices but cautioned that these results showed association and not necessarily usage. By using verbal protocol analysis, this study provides richer insights 3

4 into information use in judgment and decision making than has been provided by archival studies. Additionally, it is the first research to examine the impact of assurance services on voluntary disclosure of nonfinancial performance indicators, which also was raised by the AAA FASC as an important issue. Furthermore, this study considers variations in analyst behavior that might be expected from the psychology literature. This literature suggests that negative information is more diagnostic and elicits more cognitive analysis compared to positive information, which may suggest that individuals place greater reliability on negative information. The corollary of this assertion is that positive information is less diagnostic and receives less cognitive analysis. Therefore, it is anticipated that there would be less perceived reliability and greater uncertainty about the disclosure of positive information. This rationale suggests greater uncertainty in positive trending financial information disclosure will change individuals decision making processes and increase the utilization of nonfinancial performance indicators as an alternative information source in performing a company valuation. Such a finding would suggest context specific reliance on nonfinancial performance indicators. The study required participants to perform a valuation of a company. The company was a medium sized private company that was intending to list on the stock exchange. The data were collected by obtaining concurrent verbal protocols whilst the participants performed the valuation. Our study found that there was significant utilization of nonfinancial performance indicators by analysts when they were performing the company valuation. However, the utilization of this information and decision making processes were affected by whether 4

5 the financial information was positive or negative trending. Specifically, when the financial information was positive trending compared to negative trending, there was more utilization of nonfinancial performance indicators. Further, it was found when the financial information was positive trending there were less algebraic calculations performed, and more reliance on information from memory in performing the valuation task compared to when the financial information was negative trending. The assurance reports on the nonfinancial performance indicators were read by all the analysts who were provided with these reports, and there was evidence of further processing of this information. For each of the four cases where assurance was provided on the nonfinancial performance indicators, the use of nonfinancial performance indicators was greater than the same case when assurance was not provided, although these differences were not statistically significant. Also, there was considerable utilization of the audit reports, with seven of the eight analysts utilizing this information in their decision making process. In examining the usage of financial information, the results show that the financial statements were the main source of information utilized in particular the income statement, followed by the balance sheet and then the cash flow statement. Finally, in actually making their decisions, all of the analysts used the capitalization of future maintainable profits valuation method. Finally, the decisions of the participants were examined. It was found that the participants in the positive trending financial information group assessed the company s stock price significantly higher than those in the negative trending financial information group. However, confidence in their valuation estimates was significantly lower than 5

6 those in the negative trending financial information group. This also suggests greater uncertainty in the financial information when it was positive trending. II. THEORY AND RESEARCH QUESTIONS Information Used in Performing a Valuation There have been a number of studies that have looked at the decision making processes of financial analysts in evaluating company performance (Biggs 1984; Bouwman et al. 1987; Anderson 1988; Hunton and McEwen 1997), which is the type of task undertaken in the present study. In tracing the information search behavior of 11 financial analysts, Biggs (1984) found that the subjects used two different types of strategies, a predictive strategy and a historical strategy, and that there was a high degree of similarity in the subjects behavior. An interesting point noted was that variations in background information such as economic conditions could affect a subject s cognitive representation of the task, or what Newell and Simon (1972) called a problem space. This feature will be evaluated in the present study as it will look at how the financial performance manipulation affects the subject s cognitive representation of the task. Bouwman et al. (1987) examined the decision making processes of professional financial analysts screening prospective investments. The protocol analysis produced a descriptive model of the financial screening process, including identification of decision making processes, decision rules and the types of knowledge required to perform the task. This study also observed that most experts used a directed search strategy rather than a sequential search strategy. Anderson (1988) also used protocol analysis to compare 6

7 the information search and evaluation behavior of professional and non-professional analysts and found professional analysts were more directive in their search strategies. One particular finding of Anderson (1988) relates to the present study because it reflects utilization of financial and nonfinancial information by financial analysts, as follows: one factor of interest was that most of the professionals felt that good financial data were ultimately necessary, but not sufficient, indicators of financial viability. The most important factors, most felt, were the product itself and the customer base. (Anderson 1988, 441) Hunton and McEwen (1997) took a different approach and used eye movement protocols rather than concurrent verbal protocols to capture the cognitive search strategy of 60 professional financial analysts. Nevertheless, as with the previously mentioned studies, their main finding was that more accurate analysts employed a directive information search strategy, whereas less accurate analysts employed a sequential search strategy. Klersey and Mock (1989) performed a review study of the use of verbal protocol analysis in auditing research. They suggest that the major purpose of verbal protocol analysis is to examine the connection between a decision maker s decision processes, the use of memory and the actual choice, decision or action taken. Protocol analysis is unique because it provides information about the processes used by subjects in making their judgments. Klersey and Mock (1989) noted three main methodological criticisms of concurrent verbal protocols. The first was that verbalisation may change the subject s performance and, therefore, their cognitive process. This possibility is supported by research by Boritz (1986). The second was that the data may be incomplete because a portion of the information used by the participant might not be verbalised, particularly if 7

8 the participant is an expert. The third criticism was that the subject may report a parallel process, which differs from the actual thought process. However, Ericsson and Simon (1993) argue that instructions to think aloud can be designed not to alter the cognitive processes and that concurrent reports provide a nearly complete record of information sequence for a task. Klersey and Mock (1989) conclude that despite these arguments about the overall validity of verbal protocol studies, this methodology is a useful tool which provides important insights into auditor judgments. Given that protocol analysis is a useful research method to gain an understanding about how decisions are made, we use this approach to address six research questions as are discussed below. A number of studies have used verbal protocol analysis to evaluate the information used by financial analysts in performing company valuations (Biggs 1984; Bouwman et al. 1987; Anderson 1988; Bouwman et al. 1995). All of these studies found the most important information used in evaluating a company was the income statement. However, there have not been any studies in the last ten years addressing this question using verbal protocol analysis. 1 Given the changing business environment over the last ten years, the sort of information analysts currently use in valuing companies should be of interest to financial statement providers, users and standard setters. The above discussion leads to the following research question: RQ1: What information do analysts utilize in performing a company valuation? 8

9 The next research question addresses the issue of whether nonfinancial performance indicators are useful in performing a company valuation. A few studies using archival data have addressed this question. These studies found nonfinancial measures to be of value (Amir and Lev 1996), and leading indicators of financial performance (Ittner and Larcker 1998; Banker et al. 2000). The present study, by using concurrent verbal protocol analysis, addresses a concern raised by the AAA FASC (2002) about empirical archival research on the impact of nonfinancial disclosure. The AAA FASC referred to the regression studies that have found an association between certain types of nonfinancial disclosures and stock prices and cautioned that these results showed whether the nonfinancial measure is associated with stock prices, not whether investor actually uses the measure (AAA FASC 2002, 355). By using verbal protocol analysis this study provides insights into the use of information in judgments and decision making that were not captured by archival studies. Although not explicitly examining the use of nonfinancial information, Anderson (1988) in a verbal protocol study found that financial analysts showed significant interest in non-gaap data of a nonfinancial type, such as product line, competitors and customers. Bouwman et al. (1995) did examine non-gaap disclosures (mainly qualitative disclosures rather than nonfinancial performance indicators) and found that analysts would like more qualitative information when evaluating companies. The next research question addresses the extent of use of nonfinancial performance indicators by analysts. RQ2: To what extent do analysts utilize nonfinancial performance indicators in performing a company valuation? 9

10 Research Question 3 addresses the issue of whether the signal (positive versus negative trending) of the financial information affects analysts use of nonfinancial performance indicators. This question is based on prospect theory (Kahneman and Tversky 1979), which suggests that individuals make assessments that differ from the assessments that using expected values would suggest. Furthermore, these assessments vary depending on a reference point such as an individual s uncertainty or their wealth. This deviation from expected utility theory comes from the observation that decisions often are made as if the decision maker had a value function that differs for gains versus losses. Individuals assign value to gains and losses rather than final assets. This value function is predicted to be concave for gains and convex for losses but it is also steeper for losses than gains. In summary, this theory suggests that investors are more sensitive to losses than gains (Kahneman and Tversky 1979). Further, this theory is not just applicable when comparing losses to gains. Consistent with prospect theory is the assertion that negative aspects of an object, event or choice are weighted more heavily than positive aspects in judgments (Kahneman and Tversky 1984; Peeters and Czapinski 1990). This literature states that individuals react more to losses or negative information, than to gains or positive information, suggesting greater reliance on negative information and greater uncertainty in positive information. This is supported from the study by Anderson (1988), which found professional subjects more heavily weighted characteristics perceived as negative compared to those perceived as positive. He suggested that this indicates greater reliance on the negative characteristics, which 10

11 implies less reliance and more uncertainty about positive characteristics. When there is greater uncertainty in the financial information it is expected that analysts would look to alternative information sources to assess future performance. A further reason for analysts looking for alternative information to support positive valuation assessments could be associated with the evidence that suggests that analysts optimistically bias their forecasts (e.g., Lin and McNichols 1998; Michaely and Womack 1999; Dechow et al. 2000). This indicates that more justification is usually needed for their recommendations, thereby further suggesting that they would look for alternative information sources to support positive financial information. In summary, if nonfinancial performance indicators are leading indicators of financial performance as suggested by empirical studies (Amir and Lev 1996; Ittner and Larcker 1998; Banker et al. 2000) then it seems reasonable to expect that there would be greater use of this information under these conditions. The research question is therefore: RQ3: In performing a company valuation, do analysts under conditions of positive trending financial information have greater utilization of nonfinancial performance indicators and less use of financial statements than analysts under conditions of negative trending financial information? The final research question of this set asks whether financial analysts use assurance reports on the nonfinancial performance indicators. Assurance theory suggests that assurance reports are valuable in decision making because they enhance the reliability of the information provided (Mautz and Sharaf 1961). Agency theory and the information hypothesis explains why a demand for audit services would be expected in unregulated environments (Jenson and Meckling 1976; Watts and Zimmerman 1983; Wallace 1987). 11

12 There is significant evidence that shows demand for audits in regulated and unregulated environments to support these theories (Chow 1982; Watts and Zimmerman 1983; Abdelkhalik 1993) and further support is provided by studies that have shown that the market places a stock price premium on independently audited information (Dopuch et al. 1986; Willenborg 1999). Although these studies have all looked at financial statement audits, there are good arguments to suggest that this would also be observed in relation to assurance of other information as provision of assurance services is generally considered a natural extension of the traditional financial statement audit role (Elliott 1998). Although it took a within-firm perspective, the study by Libby et al. (2004), provides evidence on the value of assurance on nonfinancial disclosure. They found that assurance increased managers usage of unique balanced scorecard (BSC) measures, thereby reducing the common measures bias observed from prior studies (e.g., Lipe and Salterio 2000; Banker et al. 2004). The following research question is therefore proposed. RQ4: To what extent do analysts utilize the assurance report on nonfinancial performance indicators in performing a company valuation? The next section discusses the final two research questions that relate to how analysts process information in performing a valuation. Decision Making Processes in Performing a Valuation As discussed in developing Research Question 3, prospect theory states that individuals react more to losses (Kahneman and Tversky 1979) or negative information (Kahneman and Tversky 1984; Peeters and Czapinski 1990) than to gains or positive 12

13 information. However, other studies in psychology have also evaluated the different cognitive processes observed when individuals respond to negative compared to positive information. Skowronski and Carlston (1989) suggested that negative information is more likely to be diagnostic and proposed a category diagnosticity approach to explain people s negativity bias. Taylor (1991, 67) suggested that negative events elicit more physiological, affective, cognitive, and behavioral activity and prompt more cognitive analysis than neutral or positive events. Studies have also found that negative information affected states led people to narrow and focus their attention (Broadbent 1971; Eysenck 1976) on features that elicited the negative state (Wegner and Vallacher 1986). These findings have been utilized in the marketing literature (Anderson and Salisbury 2003) to explain why market level expectations are more sensitive to decreases in perceived quality compared to increases in perceived quality. In trying to draw a summary of expectations from the above research, it seems clear that if negative trending financial information is provided, it would be expected to be viewed as more salient (Anderson and Salisbury 2003) and weighted more heavily than positive information (Taylor 1991), thereby resulting in more cognitive processing than if positive trending financial information is provided. Consistent with this, in the context of company valuations, Anderson (1988) suggested there is greater reliance on negative information and, therefore, less reliance and greater uncertainty in positive information. From the above discussion, the following is expected to be observed from examination of the information acquisition and processing operators reported in this study. These operators are defined in Table 2. For negative trending financial information: 13

14 Less information retrieval from memory (IRM operators), because the focus is more on the information presented. Less generation of assumptions (GA operators), because the focus is more on the information presented and there is less perceived ambiguity. Less generation of queries (GQ operators), because the focus is more on the information presented and there is less uncertainty. More algebraic calculations (AC operators), because the focus is more on the financial information presented which is conducive to ACs. When positive financial information is provided, it is expected that the reverse of all of the above will be true. Consistent with the above discussion, the following research question is proposed: RQ5: In performing a company valuation, do analysts under conditions of negative trending financial information have greater utilization of information acquisition and processing operators that reflect more certainty, and draw more from financial information contained in the case than under conditions of positive trending financial information? A number of prior studies have also examined the type of search strategy utilized by analysts. Bouwman et al. (1987) found that most experts used a directed rather than a sequential search strategy. Anderson (1988) found that professionals used a directed search strategy in valuing a company compared to non-professionals who used a sequential strategy. Hunton and McEwen (1997) found that more accurate analysts employed a directive information search strategy. Given the detailed information obtained from the verbal protocols and the interest in prior research in examining this 14

15 issue, a research question relating to the type of processing strategies employed by financial analysts is investigated. RQ6: What types of information search strategies do analysts use in performing company valuations? III. METHOD Verbal Protocol Research Methodology The authoritative text by Ericsson and Simon (1993) titled Protocol Analysis was used to determine the appropriate procedures in performing this verbal protocol study. In particular it should be noted that protocol analysis is thinking aloud, not describing or explaining thoughts (Ericsson and Simon 1993). This is an important distinction because if the participant is required to describe or explain their thoughts, it may cause them to consider other information not normally required to complete the tasks and may change their thought sequences. The value of protocol analysis is due to its ability to gain insight into the decision making processes, which is derived from the articulated thought process, not by an explanation of it. To achieve the goal of capturing a decision-makers thought processes, Ericsson and Simon (1993) suggest that when using concurrent verbal protocols, interactions with the subject be minimized and that the participant should be provided with a practice problem. For this study, once the study started, the only interaction with the participants was by the following instruction and only if the participant was silent for more than seconds: What are you thinking about now? 15

16 To satisfy Ericsson and Simon s second suggestion the following practice problem was given: Try to multiply these two numbers in your head and think aloud as you get the answer: What is the result of multiplying 24 x 36? A numeric calculation is suggested in Ericsson and Simon (1993) as a good method to familiarise participants with the required task. Following the practice exercise, the decision task protocols were started and tape recorded by the researcher. Participants and Task Eight financial analysts from six stock brokerage firms located in Sydney and Perth Australia participated in this study. 2 Their participation was solicited by writing a letter to the managing director of a number of stock brokerage firms. This letter was then followed by a phone call requesting volunteers. All participants were given an information sheet about the study and then they signed a consent form to acknowledge they were willing to participate. The participants details are outlined in Table 1. The average length of experience as a financial analyst was 9.8 years. 3 All of the participants with lower levels of financial analyst experience had previously worked in a business related field, e.g., accounting firms, therefore all of the participants had the appropriate knowledge and experience to complete a task of this type. The case study itself also provided a vetting process on whether the participants could complete the task, as evidenced by one potential participant who was replaced after stating that he did not have the necessary skills to perform the valuation. 16

17 [INSERT TABLE 1 ABOUT HERE] The participants task while thinking aloud was to perform a valuation of a medium sized private retail company intending to list on the Australian Stock Exchange. To gain more insights into decision making processes and the research questions, trends of financial statements and nonfinancial performance indicators, and assurance on nonfinancial performance indicators were varied in the experimental materials. Financial performance was provided as either upward or downward trending over three years. The nonfinancial performance indicators were in the form of a BSC that was showing performance either better or worse than target and a BSC assurance report was either provided or not provided on this information. As a consequence, there were eight versions of the case, with each participant receiving a different version. The development of the case was based on a real company with significant amendments. The information in the case comprised: company information; nonfinancial performance indicators; assurance report (where applicable); and financial statements. The development of this material is discussed as follows. Case Study Development Before developing the case study materials, six interviews were conducted with senior people involved in company valuations. They included two partners and two directors in corporate finance from two accounting firms and a director and senior analyst from two stock brokerage firms. The interviews were to elicit information about the type of information they thought was important in performing company valuations, and more specifically, what nonfinancial information was of interest to them. There were three 17

18 main nonfinancial disclosures of significance elicited from these interviews: quality of management and corporate governance; strategy; and nonfinancial performance indicators. The latter two were incorporated into the case study by use of a BSC, and other information about company valuations from these interviews was also considerred in developing the case study. After development of the case study, it was reviewed by a number of senior academics and amendments made based on their suggestions. Company Information The company background information was not manipulated and provided general information about the company, such as: the type of products that it sells; how long it has been operating; its number of stores; where its products are sourced; and its general strategy in the market. Industry information was also provided, such as: level of competition; total sales; and the different types of retail formats. This information was elicited from reviewing the information provided about the real company on which this case study was based, and several other comparable companies in the industry. In addition to this information, the average and range of price earnings (PE) ratios 4 for the industry were provided. They were calculated by reviewing the PE ratio for the real company and the PE ratios for other companies in the industry. Nonfinancial Performance Indicators The nonfinancial performance indicators were disclosed in the form of a BSC outlining performance measures that were related to the strategy employed by the company. These did not come from the real company or from any of the comparable 18

19 companies because it was not provided in any of their annual reports. 5 It was instead based on BSC information that had been used for similar types of companies in the prior literature (Kaplan and Norton 1996; Lipe and Salterio 2000). In this section of the case study, the BSC was introduced and briefly explained, followed by a discussion of the company s strategy. The strategy was worded to ensure that it linked clearly to the specific nonfinancial performance indicators provided. This linkage is important because a key feature of the BSC approach to performance measurement is in ensuring that the measures are clearly linked to the firm s strategic objectives. The BSC was chosen as the structure to represent nonfinancial information in the present study because it is widely used as an internal management tool and because it relates strategy to nonfinancial performance. The BSC was based on the case studies of RadWear and Workwear developed by Lipe and Salterio (2000). Their cases were developed following Kaplan and Norton s (1996) Kenyon Stores example. In the Lipe and Salterio (2000) study, they were evaluating differences in perceptions between common and unique measures in the BSC framework. In the present study, most of the scorecard measures used in the case study were developed from the common measures used in the Lipe and Salterio (2000) case study. The assurance report provided in the case study was a high level of assurance based on the guidance in AUS 108 Assurance Engagements. This means that the auditor forms a conclusion that the subject matter conforms in all material respects with identified suitable criteria (AUS ). 6 In this scenario the suitable criteria was whether the nonfinancial information has been properly collected, summarised and reported. All of the above information (company information, nonfinancial performance 19

20 indicators, and assurance report) was listed in the order it would be expected to appear in an annual report. Therefore, the next information provided was the financial statements. Financial Statements The three basic financial statements comprising the income statement, balance sheet and cash flow statement were provided with two years comparative information. These statements were manipulated as to whether the information was positive or negative trending. The financial statements were based on a real company, however, they were amended for the purposes of this study. The primary change was that they were reduced in size from the real company by a factor of ten. There were also a few other amendments to individual figures so that the financial statements did not have any information that might act as a red flag to analysts in valuing the company. This amended financial information for one year was then entered onto an Excel spreadsheet and became the middle year s figures in the case study (out of the three years of financial statement information provided). A financial year where the information was improving was created by increasing the operating profit after income tax by about 20 percent and then adjusting all of the other figures on the income statement to be consistent with an increase in profit by that amount. To create a financial year where the information was deteriorating, the operating profit after income tax was reduced by about 20 percent and then all of the figures on the income statement were adjusted to be consistent with such a change. To create the downward trending financial information, the year with the best financials was shown as the first year and the year with the worst financials was shown as the current (third) year. To create the upward trending financial information, this was 20

21 reversed. The Excel spreadsheet was designed so that these changes were automatically reflected in all of the figures on the balance sheet and cash flow statement. The figures in the balance sheet were held as constant as possible, except for changes that were necessary because of the movements in the income statement over the three years. To add to the realism of the case study, the financial information was accompanied by an unqualified audit report that was not manipulated in the study. This was provided immediately following the financial statements. Coding of the Protocols The taped protocols were professionally transcribed. After transcription, the written documents were examined in conjunction with listening to the tapes and amended if necessary. This was to ensure that the protocols had been correctly transcribed and also to assist in identifying pauses that indicated different protocol phrases. When this was completed the transcribed protocols were further reviewed and parsed into discrete protocol phrases for coding. There were two different types of coding applied to the protocols: activity codes and information codes. Activity codes were to examine the decision making processes of the analysts and the information codes were to examine what information they utilized in performing the task. The activity codes were double coded, however, the information codes were not because they were quite objective categories. The coding scheme for the activity codes is based on the type used by Biggs and Mock (1983). The Biggs and Mock (1983) coding scheme is based on Newell and Simon s (1972) theory of human processing, which indicates that a participants behavior will be composed of a 21

22 sequence of operations (processes) applied to states of knowledge (information) (Biggs and Mock 1983, 238). This coding scheme was slightly amended to reflect the different nature and research objectives of this task. The final activity code definitions for this study are shown in Table 2. [INSERT TABLE 2 ABOUT HERE] The information codes were assigned based on the verbalizations that were obtained from each participant s examination of the case study. Table 3 shows the categories of information codes used for this study that were developed based on the information provided in the case study. [INSERT TABLE 3 ABOUT HERE] The first part of the coding process involved the two coders discussing the activity coding scheme and the definitions of the operators to assist in developing an agreed understanding of the definitions. This was followed by each coder independently coding the first 77 phrases of the protocol for Participant B. Once this was done the codes were compared and the amount of non-chance agreement was measured using the kappa coefficient. Based on the 12 activity code categories the kappa coefficient was calculated as This indicates a high level of reliability compared to prior studies. 7 The disagreements were discussed and reconciled between the two coders. This process assisted in clarifying the shared understanding of the definitions of the operators for the coding of the remaining phrases in the protocols. As part of this process it was also noted that there were some disagreements about the parsing of the protocols into phrases. These disagreements were few, however, based on discussions related to these disagreements the remainder of the protocol for Participant B as well as all of the other protocols were 22

23 reparsed. Once the protocols had been reparsed, the remainder of Participant B and the rest of the protocols were independently coded by each of the coders. When this was complete, kappa coefficients were then calculated on the remainder of the protocol for Participant B as well as the other protocols. Table 4 shows the kappa coefficient across the 12 activity codes for each of the eight protocols. [INSERT TABLE 4 ABOUT HERE] These figures indicated a high degree of consistency between the two coders, with an average overall kappa coefficient of The lowest level of agreement was for participant G. This was probably because Participant G had a very low proportion of read phrases, which were the most objective phrases to code, thereby meaning that a greater proportion of this protocol related to phrases that were more subjective to code. For all of the protocols, differences in coding of the phrases were discussed and reconciled between the two coders to obtain the final agreed coding for all of the phrases. The final agreed reparsed protocols with all phrases coded were then entered into a NVivo software package for analysis. 8 After all of the data were included in NVivo, the information codes were assigned to each of the protocol phrases. The number of characters, words and phrases for each of the participants protocols in the study are detailed below in Table 5. [INSERT TABLE 5 ABOUT HERE] As can be seen in Table 5, there is a reasonable degree of variation in the length of the eight protocols. The next section discusses how the results inform the research questions. 23

24 IV. RESULTS Table 6 shows the overall information item codes that were attended to by the analysts and addresses Research Question 1, which evaluates what information analysts use in performing a company valuation. [INSERT TABLE 6 ABOUT HERE] The financial statements including the audit report were the most utilized, accounting for 45.5 percent of all phrases of the protocols. The nonfinancial information including the BSC assurance report was the next most utilized (29.6 percent), followed by the company information (18.8 percent). The industry average PE ratio was reported separately and accounted for 6.1 percent of all phrases of the protocols. The earnings per share (EPS) figure which is part of financial statements accounted for 3.3 percent of all phrases of the protocols. All of the analysts used the EPS and the PE ratio to perform their valuation by the capitalization of future maintainable profits method, which explains the high usage of these figures. Within the financial statements, the income statement was the most attended to with 18.2 percent of all phrases of the protocols (including EPS). This is consistent with all previous protocol studies that have examined this issue (Biggs 1984; Bouwman et al. 1987; Anderson 1988; Bouwman et al. 1995) and provides current evidence on the importance of accrual accounting numbers. The next most important was the balance sheet with 11.4 percent (including share capital); followed by the cash flow statement with 5.9 percent of the phrases. The calculation of ratios from the financial statements accounted for 6.4 percent of the phrases. Tables 7 and 8 follow to address Research Questions 2 to 4. 24

25 [INSERT TABLE 7 ABOUT HERE] [INSERT TABLE 8 ABOUT HERE] Research Question 2 examines whether analysts utilize nonfinancial performance indicators in performing a company valuation. The utilization of the nonfinancial performance indicators across the eight protocols varied from 18.4 percent to 40.6 percent as can be seen on Tables 7 and 8, with an overall average of 27.8 percent (excluding the assurance report), which indicates a reasonable level of attention to this information. In particular, review of the protocols revealed that three of the participants made specific comments indicating they found the presentation of nonfinancial information useful. 9 Some of these phrases of protocol are presented below. Participant B 63: Nonfinancial performance indicators. 64: Clearly instructive to see how they are doing in relation to their targets. Participant C After reviewing general information on BSC: 35: obviously they have looked at themselves from a number of different angles, 36: which is encouraging but viewing themselves for a public listing no doubt. After going through BSC indicators: 77: well I understand those things 78: pretty good to have that information. Participant D 69: Comment about a balanced scorecard approach is useful 70: because that gives some sort of sense of what is driving the assessment of the company internally and what they think are important 71: and linking the strategy to performance is also quite important. 72: In many industries or companies its very hard to get a sense of the method in the madness used in running the companies 73: but companies that are able to show some sort of tool set to be able to do that show some sort of a robustness in their reporting systems, management systems, etc. 25

26 74: so that is quite valuable. After going through BSC indicators: 97: so that is actually very good information. 98: Its not something that you often see. 99: Such companies that are willing to put things like nonfinancial performance indicators, quantify those, discuss the differences between target and actual its quite useful stuff 100: and not many companies do it 101: so 102: when you do see companies provide that sort of stuff it does give you a better sense and better feel 103: and probably a higher level of confidence in your assessment of the company. There is evidence from the frequencies in Table 7 and Table 8 to suggest that the nonfinancial performance indicators were of interest to the analysts in terms of the information utilized in their decision making processes in valuing the company. In terms of what parts of the nonfinancial information were used, the largest number of phrases related to the introduction of the detailed nonfinancial performance indicators stating that the company was using the BSC approach and that it was linked to the company s strategy. The relative utilization of the specific nonfinancial performance indicators in the case was reasonably consistent across the categories of customer-related, internal business processes and learning and growth. 10 The nonfinancial performance indicators were also manipulated in the case as being positive or negative, i.e., they were better or worse than target. Mann-Whitney U tests provided no evidence that participants altered their final share value based on the different manipulations of the nonfinancial performance indicators or that they perceived them to be significantly different in their analyses. There are a couple of possible reasons why no significant effect for this manipulation was observed. First, it is possible that the disclosure of nonfinancial performance 26

27 indicators says something about the way the company is run and that is the message that makes a difference to analysts rather than whether they are meeting targets or not, which would be consistent with the high number of phrases of protocol related to the introduction of the nonfinancial information, under the categories of balanced scorecard and strategy. Second, there is evidence from perusal of the protocols that analysts assessment of the nonfinancial performance indicators was by intrinsically assessing the actual figures rather than comparing them to the targets provided. Research Question 3 considers whether analysts under conditions of positive trending financial information have greater utilization of nonfinancial performance indicators and less use of financial statements than analysts under conditions of negative trending financial information. Mann-Whitney U tests were performed to compare the usage of the nonfinancial performance indicators and financial statements between the groups who received the positive and negative trending financial information. Due to the large variation in the number of phrases across the protocols, the Mann-Whitney U test was calculated using percentages of total phrases rather than the absolute number of phrases relating to the information item codes. In addressing Research Question 3, the percentage of nonfinancial performance indicators used by participants in the positive trending financial information group was 33.0 percent compared to 21.9 percent in the negative trending financial information group. The results of the Mann-Whitney U calculation showed the mean ranks were 5.5 compared to 3.5. This difference between the two groups was not quite significant (Z= , p=0.124, one-tailed). However, all three of the analysts who made comments indicating that they thought the nonfinancial information was particularly useful had been 27

28 provided with positive financial information. The Mann-Whitney U test and these comments provide some evidence that analysts give greater attention to nonfinancial performance indicators when financial information is positive trending. The percentage of financial information used by participants in the negative trending financial information group was 47.7 percent compared to 36.9 percent in the positive trending financial information group. The Mann-Whitney U calculation showed the mean ranks were 6.0 compared to 3.0. This difference between the two groups was significant (Z=-1.732, p=0.042, one-tailed). This suggests that when the financial information is negative trending it receives greater attention by analysts than when it is positive trending. These findings provide further evidence on the salience of negative information. It also shows that positive financial information receives less attention, which could be due to more uncertainty of this type of disclosure. However, when positive financial information is provided there is some evidence of greater utilization of nonfinancial performance indicators, suggesting that participants see this as useful corroborating information under these circumstances. Research Question 4 is concerned with the usage of the assurance report on the nonfinancial performance indicators. Examination of the protocols where the participants received an assurance report indicated that all of the analysts (four) who received an assurance report on the nonfinancial performance indicators read the report. In addition, there is evidence of further processing of the assurance report by two of the four, as detailed in the phrases of protocol reported below: Participant B 91: which appears to be clean, independent audit report 28

29 92: OK. Participant F 102: Conclusion based on our review, we conclude that in all material respects the nonfinancial indicators were properly collected, summarised and reported. 103: OK 104: that is interesting. Further examination of this issue showed the use of nonfinancial performance indicators was greater than when assurance was not provided, although this was not statistically significant (Z=-0.577, p=0.564, one-tailed). However, for each of the four cases where assurance was provided the use of nonfinancial performance indicators was greater than the same case when assurance was not provided. Even though it was not directly related to the research question, another question is whether analysts adjusted their valuations or confidence in their valuations based on the assurance information. Valuations or confidence judgments of analysts when provided with assurance was not found to be significantly different from those without this treatment. However, in three of the four cases where assurance was provided the valuations and confidence judgments were numerically greater than the same case when assurance was not provided. Finally, the analysis of phrases of protocol related to the audit report on the financial statements showed it was of significant interest to analysts. It was attended to by all of the participants in performing their valuations, except for Participant B. Some of the phrases of protocol are presented below. Participant C 139: Auditor s report is fine. 140: Just have to read the auditor s report to make sure there are no tricks in here. 141: One has always got to look at the auditor s report, 142: can pass it over so easily 29

30 143: OK. Participant D 128: There is an independent audit report 129: which is useful. 130: It seems to pretty much be in order. Participant E 28: Company has had an audit of the financial information shown on pages six to eight 29: that is encouraging. 30: The company is required to be audited as a large propriety company under the definition in the Corporations Act 31: should be useful information. Participant F 52: Its had an audit its a large proprietary company 53: I will have a read of that. 54: The audit stuff would be important. Participant G 10: I suppose just given that it s had a full financial audit, 11: you can take the financial numbers pretty much as given, you would hope, 12: which increases your confidence of just relying on those. Participant H 229: Looking at the audit report 230: making sure that there is nothing tricky in there. 231: So they have got a clean audit report. 232: Big 4 auditor - 233: I guess that is a positive. Research Question 5 relates to whether negative trending financial information elicits differing information acquisition and processing, specifically whether this condition results in greater utilization of information acquisition and processing operators that reflect more certainty and draw more from financial information contained in the case, than under conditions of positive trending financial information. 30