Re-engineering IT Internal Controls: Applying Capability Maturity Models to the Evaluation of IT Controls

Transcription

1 Re-engineering IT Internal Controls: Applying Capability Maturity Models to the Evaluation of IT Controls Roger S Debreceny University of Hawai`i at Manoa rogersd@hawaii.edu Abstract Financial and management accounting relies not only on traditional computerized accounting information systems but also on many application systems that feed data to the entries that make up the financial accounting systems. The importance of IT has been recognized by auditing standards setters. In response, a variety of organizations have developed control frameworks for the IT lifecycle. COBIT, published by the IT Governance Institute (ITGI), is a well understood and widely used control framework. An important element of the various elements and tools that make up the COBIT framework is the Capability Maturity Model (CMM) that is included in the COBIT Management Guidelines. The six-level CMM is drawn from the software engineering research community. A given level of the CMM allows managers and others to determine the capacity of the entity to manage its risks and strategic and operational outcomes, for that particular process. Whether a given capability maturity level correlates to a particular level of internal control, under auditing standards, has not been determined. This research is an exploratory attempt to determine the capability maturity of organizations; address metrication issues in measuring capability maturity and correlate capability maturity with the state of internal control over financial reporting. 1. Introduction The importance of designing, building, and assessing the quality of internal controls on the lifecycle of information technology (IT) investment has been heightened since the passing of the Sarbanes-Oxley Act. A well established framework for the establishment of controls over IT is COBIT, promulgated by the IT Governance Institute. This framework encompasses the whole gambit of IT investment from strategic visioning, to acquisition, development and deployment of IT as well as monitoring and feedback mechanisms. A key element of COBIT is a capability maturity model (CMM) for each element of the framework. The CMM concept is drawn from the software engineering discipline, in particular the Software Engineering Institute (SEI). The CMM concept in turn drew intellectual inspiration from the quality movement. As the term suggests, the CMM measures an entities capability to successfully manage the controls that make up each element in the COBIT framework. Arguably, the CMM level for a given element predicts potential control failures that relate to the particular element. Research on CMM in the COBIT framework is a null set. Indeed formal research on CMM, as distinct from design rationale and implementation advice from the SEI, is surprisingly slight. This paper sets out a research agenda for the study of CMM both within the context of COBIT framework as well as the in the broader context of the internal control on financial reporting. The paper introduces a new methodology for assessing an entity s capability maturity on each of the elements in the COBIT framework. It describes the highly preliminary testing of this assessment process on a moderately large health services provider. The testing shows that there is considerable variation in the quality of Capability Maturity Models between the various framework elements. The paper makes some initial judgments on the acuity of the CMM assessment process. It sets out a number of conclusions for the research agenda based on this testing. The remainder of this paper proceeds as follows: The literature review in Section II discusses the nature of internal control and discusses the /06/$20.00 (C) 2006 IEEE 1

2 relationship between internal controls over financial reporting and controls exercised over the information technology lifecycle. It describes the nature of control frameworks over IT, paying particular attention to the COBIT framework. The Capability Maturity Model is introduced and analyzed. Finally, in this section, a suite of research questions are laid out. In the third section, methods for testing the Capability Mature of organizations are described. In the fourth section, the method by which the testing is implemented is set out. The next section discusses the results of testing the CMM in a large health services provider. The final section sets out some conclusions, describes the limitations to the study and discusses how the research agenda set out in the second section might be furthered. 2. Literature Review The extent of corporate frauds and failures of the early 00 s and the subsequent passage of the Sarbanes-Oxley Act 2002 have brought the nature and effectiveness of internal controls in complex entities increasingly into focus. The Act introduced requirements for SEC registrants to assess their internal controls (ICs) and particularly their internal controls over financial reporting (ICFR). Registrants must report both the initial state of their ICFR control deficiencies and material weaknesses. These managerial assessments are now coupled to an integrated audit that reports separately on the effectiveness of ICs as well as opine on the financial statements. Whilst the rationale for the changes wrought in the Act were not directly related to failures in IT, there have been significant implications for building, documenting and assessing IT controls, at least as they pertain to financial reporting. In modern organizations IT is pervasive and central to the process of adding value. Computerized management information systems are fundamental enabling technologies in the preparation of accounting information. It is difficult to conceive of any reporting entity of any size that does not employ a wide range of such systems that originate, transform or analyze information that eventually result in some disclosure in the financial statements. IT is central to the information gathering, communication and monitoring roles of the IC process, as defined in COSO. Increasingly, however, IT is fundamental to control activities -- The policies and procedures that help ensure management directives are carried out envisaged in COSO including authorizations, verifications, reconciliations, reviews of operating performance, security of assets and segregation of duties [1] are now increasingly facilitated by IT. The pivotal role played by IT in IFCR has been recognized by auditing standards promulgated by the Auditing Standards Board and, more recently, the Public Company Accounting Oversight Board (PCAOB). The Board s Auditing Standard #2 An Audit of Internal Control Over Financial Reporting Performed in Conjunction with An Audit of Financial Statements notes, in paragraph 50 that IT general controls might have a pervasive effect on the achievement of many overall objectives of the [COSO-based] control criteria [2]. The Board also discusses the manner by which auditors should address the role that information technology plays in particular scenarios as they devise tests of aspects IFCR. The objective of maintaining control over IT systems is, of course, more than just achieving IC strictly over financial reporting. Achieving strategic and operational alignment between IT and the entity is central to the objectives of the IT function [6, 7]. The concept of IT Governance has been developed over the last decade, in response to the growing awareness of the need to align business and IT goals in addition to maintaining the core operational outcomes of the IT function. The IT Governance Institute defines IT governance as being an integral part of enterprise governance that consists of the leadership and organisational structures and processes that ensure that the organisation s IT sustains and extends the organisation s strategies and objectives [8]. IT governance is a subset of the entity s overall corporate governance IT governance stresses the alignment of IT with overall enterprise goals, benefit maximization through IT and the productive use of IT resources and management of risks that arise within and from IT investments [8] IT Governance The literature on IT governance is in its infancy. A strand of the literature views IT governance through a strategic alignment lens. From this latter perspective, high IT alignment with business needs and objectives result in superior business performance [9-12]. An empirical literature on the benefits to the firm of IT alignment has only recently begun to move from supposition and casual empiricism. There are several case studies and survey studies (e.g. [13]) that address the financial benefits to the firm of IT alignment. In addition 2

3 there a handful of studies that investigate the reaction of capital markets to (e.g. [14-16]). A relatively new literature extends the concept of alignment to the nature and extent of IT governance and interrelationship of IT governance with wider firm-level governance processes [7, 12, 17]. This research strand is at a relatively early stage of theory development. There are few empirical studies that correlate the state of IT governance with firm characteristics. An even newer literature evaluates the impact of IT governance mechanisms on the financial performance of the firm. [18] go so far as to argue that advanced IT governance processes at the firm level leads to improved financial performance. They set up a simplified assessment process that allows entities to monitor their IT governance at a high level of abstraction. A second approach to IT governance is from an audit and control perspective [19, 20]. This literature derives from processes that accompany the binding of IT controls into the wider control environment. As De Haes and Van Grembergen note, IT governance is more than just IT management [20]. IT governance encompasses IT operational and strategic management, the response of IT to business objectives and the configuration of specific IT governance mechanisms to wider firm-level governance. De Haes and Van Grembergen identify the elements of IT governance as having structural (e.g. IT steering committee(s)), process (e.g. Strategic Information Systems Planning, Service Level Agreements, IT alignment/governance maturity models etc.) and relational elements (e.g. mechanisms to encourage active participation and collaboration between principal stakeholders and cross-functional business/it training and rotation) [20]. The focus of this study is on the process aspects of IT governance, in particular the measuring the extent of IT governance and operational maturity levels and relating that level of maturity to the overall state of IC and governance development. The de facto standard for the construction and maintenance of IC at the entity-wide level is COSO. There a variety of best-practice frameworks for IT including ITIL (IT Infrastructure Library) at the service management layer; various ISO standards including ISO 17799/BS7799 in the area of security risk management and COBIT, of which more shortly, and the CICA s ITCG (Information Technology: Control Guidelines) across the complete IT investment lifecycle COBIT The most prominent of these frameworks, at least from an IC perspective, is COBIT (Control OBjectives for Information and related Technologies). This is a framework for the development of controls on IT from the IT Governance Institute, an arm of the Information Systems Audit and Control Association (ISACA) [22, 23]. COBIT defines four primary domains of control that are relevant throughout the lifecycle of information systems from planning, development, acquisition to deployment along with a set of controls on the monitoring and feedback processes. The four domains within COBIT are Planning and Organise (PO), Acquire and Implement Automated Solutions (AI) Deliver and Support (DS) and Monitor and Evaluate (ME). Within each control domain there are a series of control objectives that define the elements of control over a given business process that a well managed entity would be likely to employ. The control processes are best seen as a comprehensive set of best practices for establishing management and control over the complexity and change that categorize the modern IT function. Further, there is a posited structural relationship within the COBIT, with framework elements being pre-conditions for other elements. Outputs from each control objectives over business processes are predicated inputs to other business processes. The COBIT framework includes the control objectives as well as a variety of supporting and ancillary protocols, including the Audit Guidelines and Management Guidelines. The Management Guidelines are designed to provide a toolset for managers of the IS function to develop the controls throughout their organization and across the range of activities which the function undertakes. The Guidelines include sets of Critical Success Factors (CSFs), Key Performance Indicators (KPIs), Key Goal Indicators (KGIs) and, perhaps most importantly, a statement of Capability Maturity Model levels, to which I will return to in a later subsection COBIT and Sarbanes-Oxley COBIT has become particularly important for domestic organizations since the passage of the Sarbanes-Oxley Act in Under this act, the management of SEC registrants is required to base their evaluation of their ICFR on a suitable, recognized control framework (s. 240). The Public 3

4 Company Accounting Oversight Board identifies the Treadway Commission s Internal Control Integrated Framework (COSO) as an appropriate framework for this purpose [2]. This framework provides general guidance on the shape of IC with its identification of the overall control environment, the assessment of risk, the policies and procedures that make up the control activities adopted by the entity and the communication and monitoring or feedback mechanisms that accompany the other informal and formal aspects of the IC processes. The COSO framework is expressed at a high level of abstraction and whilst it includes some delightfully dated general guidance on IT controls, it provides guidance at a strategic and operational level that can be applied to any organization and to all components of modern organizations. It was not designed to provide an IC framework for the management of the IT lifecycle. The PCAOB s Auditing Standard #2 provides some limited additional guidance on IT controls including a requirement that when assessing assertions, the auditor should determine the source of likely potential misstatements in each significant account and that as part of that process they should assess the nature and complexity of the systems, including the use of information technology by which the company processes and controls information supporting the assertion (para 69). A further requirement in the Standard is that auditors should assess the extent of information technology involvement in each period-end financial reporting process element (para 77). When analyzing the relationship between COBIT and the requirements of the Sarbanes-Oxley Act, the ITGI [24], notes that most SEC registrants will require additional guidance on IT controls over and above that provided by COSO. The ITGI asserts that whilst there is some level of overlap between COSO and COBIT, at for example the level of developing the control environment, there is a high level of compatibility between the two frameworks. ITGI shows that there are only seven of the 34 control objectives in the COBIT framework do not bear directly on the COSO components that underpin SEC registrants assessment of the ICFR. The ITGI provides guidance on the steps that those responsible for the controls over IT of an SEC registrant should take in assuring that the registrant has achieved an appropriate level of IC. The stages are (1) planning and scoping the engagement; (2) conduct a risk assessment; (3) evaluate the control design over IT; (4) evaluate the operational effectiveness of IT controls and finally (5) identify and remediate deficiencies. Of direct relevance to this paper is the third stage when control designs are evaluated. The ITGI notes that: In this phase, an IT organization must step back and evaluate the ability of its control program to reduce IT risk to an acceptable level. More specifically, it requires that control attributes, including preventive, detective, automated and manual, be considered when designing an approach to effectively address risks. For example, if a change management risk is identified that would result in unauthorized programs being migrated into the production environment; a properly designed control would prevent this from occurring. In this example, a detective control that identifies unauthorized programs in production after the fact may not be appropriate. [24] 2.4. Capability Maturity Models The core tool that the ITGI suggest be employed to assess the overall IT control environment is the capability maturity model (CMM). The CMM describes the stages of maturity that the entity may go through in developing a thorough level of control, quality and productivity in the management over the particular business process defined in the control objective. The concept of the CMM in CobiT draws directly from the Software Engineering Institute s Capability Maturity Model (CMM). Whilst the scope of CobiT encompasses the complete lifecycle of IT investment, the SEI s CMM is firmly rooted in improving the quality of software development. The CMM provides a generic method for establishing the levels of maturity to process improvement. The five levels of maturity are: 0 Non-existent; 1 initial; 2 repeatable; 3 defined; 4 managed and 5 optimizing. The levels of maturity are designed to indicate the ability of the entity to develop software or complete other business processes as defined in the CMM s process domains. The CMM framework provides a staged approach to building quality into the stages of product development, deployment and maintenance [28]. An important historical influence in the development of the CMM is the concept of quality conformance and embedding quality into all the stages of software development and deployment that was strongly influenced by the quality assurance movement in manufacturing associated particularly with Deming and Juran [29-32]. The CMM and CMMI builds directly upon the work of Humphrey [33, 34]. 4

5 An important distinguishing characteristic of the CMM is its systematic engineered approach to the development of control processes, in a manner that should enable the systematic prediction of control failures from the level of maturity. The ITGI notes, in its discussion on the control objectives and SOX that [f]or the purposes of establishing internal control, some organizations may be willing to accept IT controls that fall somewhere short of stage 3. However, given the Sarbanes-Oxley Act s requirements for independent attestation of controls by external audit, controls will more than likely require the attributes and characteristics of stage 3 or higher for key control activities [24] Research Questions Whilst the capability maturity, somewhat adapted, have been incorporated directly into the COBIT Management Guidelines, the research literature on capability maturity models and COBIT is essentially a null set. I must draw on research on the use of CMM and other process improvement frameworks in the software development and deployment domains for guidance and theory development. Surprisingly, given the importance of software development for a wide range of entities in the economy, the empirical research literature on capability maturity models is limited and fragmented. [36] undertook three studies of success factors in CMM adoption. They found that most software development organizations were at low levels of migration to upper levels of the maturity model. In a related study, [37] conducted a survey of software developers. They found that software productivity was gained only at higher levels of the capability maturity model. These studies raise interesting questions as to whether there is some minimum level of maturity before controls over corporate information systems are effective, in the terms of the Auditing Standard #2, as discussed in the previous sub-section. We know, however, very little about the distribution of levels of capability maturity between the different business processes within organizations. I know nothing of the differences in capability between organizations that have made different choices in their organization of different aspects of their IS function: e.g. insourcing versus outsourcing or rapid application development versus traditional development teams. The existence of a path model between the various control objectives that make up COBIT has been predicted but there is no evidence to support such a path model. Do entities indeed recognize that there are path dependencies between the various control objectives? We have much to learn about the relationship between the levels of maturity and the state of ICs in organization in general and over financial reporting in particular. Is there a minimum level of maturity that entities must achieve to ensure control is developed and maintained, as predicted by the ITGI? Does a given level of capability maturity indeed predict control weaknesses? How do the assessment of overall controls in the IT function relate to the decidedly non-engineering approach of the walkthrough suggested as the primary assessment technique suggested in Auditing Standard #2. From the above discussion flow the following research questions: RQ1: Is the level of maturity of IT controls distributed evenly across the various domains and process as identified in COBIT? RQ2: Is the maturity model framework as set out in COBIT useful for the assessment of pervasive/general IT ICs? RQ3: What are the influences of IT management design choices, such as insourcing versus outsourcing, on the levels of capability maturity? 3. Measurement Whilst the concept of a given level of capability maturity is intuitively appealing, developing measurement models are difficult. It is not the case that an organization can readily place itself on the scale. The point differences of the capability maturity model set out in the previous section are somewhat too broad and expressed in to high a level of abstraction to be capable of being operationalized. Some recommend using the control objectives for each of the business processes in a self-assessment of the entity s capability maturity model. Such a process is unlikely to generate a sufficiently granular and accurate measure of the capability for that particular business process as there is insufficient guidance on what is entailed in assessing the readiness of the entity at each level of the maturity model. Further, whilst the general descriptions set out by the ITGI do provide guidance on distinguishing the stages in the CMM, there must be a set of attributes that can more reliably provide metrication of an entity s given maturity. To that end, the ITGI has prepared a set of attributes for each of the levels of each of the control 5

6 objectives for the 34 business processes that make up COBIT. These statements are designed to be used as the foundation for a quantitative assessment of the capability maturity for a given organization. A panel of experts developed these questions. The questions are designed to be applied to the business process owner(s) in the entity. For each statement the respondent is asked Do you agree? Possible responses are across a four element Likert scale, made up of Not at all, (i.e. do not agree at all that this statement applies to my organization); A little (agree a little that this statement applies to my organization); To some degree and Completely. In order to tease out the attributes of the maturity model, the panel of experts employed a variety of types of questions. The first grouping can be termed a stage of maturity class of question. For example, at Maturity Stage Level 1 for the Control Objective PO3 Determine Technological Direction there is the statement Management recognises the need for technology infrastructure planning, but has not formalised either a process or plan. Recall from the previous section that of Stage Level 1 is Initial/Ad Hoc Process and that a generic description of attributes of this level are There is some evidence the organization recognizes that controls and related procedures are important and need to be addressed. However, controls and related policies and procedures are not in place and documented. The statement indicates that management is aware of the need but has not converted that awareness into action. At Maturity Stage Level 3 for PO3, there is the statement Management is aware of the importance of the technology infrastructure plan. This statement implicitly assumes that management has moved from the stage at which awareness exists to a stage where a technological plan is in existence and it is recognized that such a plan is important as a fundamental control in the establishment of an appropriate technological direction. It is highly desirable that there be clear distinction between each level of maturity. It would not show an appropriate degree of metrication of if the responses to statements were equally distributed across the levels. Strong coalescing around a point on the capability maturity level is the most appropriate response. 4. Method CobiT is designed for moderately to significantly large IT installations. Organizations should have sufficient complexity to make choices on issues such technological direction, in-sourcing versus outsourcing, risk management and in-house bespoke development versus purchase of packaged software. Each site must provide access to appropriate business process owners. In this first iteration of the research, a single site is chosen for investigation. The entity, X Inc., is a health services provider. It has approximately 1600 employees, of which 220 are in the IT function. A traditional user of mainframes, in recent years it has moved many existing applications to the Unix and Windows environments. Much of the new development being undertaken by X Inc. is on these latter platforms. Most of the application software is developed in-house, supported by a considerable amount of contracted programming expertise. A limited array of ancillary hardware and software support services are out-sourced. IT is fundamental to all aspects of X Inc. s value-adding business processes. The subjects are process owners for each of the 34 processes within the four domains of COBIT. Data is collected in a mixture of quantitative and qualitative approaches. As discussed in an earlier subsection, the COBIT Management Guidelines include a set of statements that allow entities to determine at what level of the capability maturity model they stand for each of the business processes. These statements are applied in face-to-face interviews of process owners. Each process owner is provided with a statement and asked to assess whether they agreed or disagreed with the statement, using a five point Likert scale. The responses by the process owners allow the quantitative assessment of intra- and inter-process maturity level. In addition, process owners are asked to answer a variety of questions in a depth interview. Each of the statements is read and the respondent asked to assess whether they agreed or did not agree with the statement. Problems with interpretation of the statement or in determining the appropriate response can be resolved immediately. The interviewer is able to follow up on selected responses with additional, unscripted questions. Although time consuming, this approach to data collection is a vital aspect of the study. The complexity of this material and the difficulty in deciding appropriate responses would render mailed questionnaires essentially unusable. Further, ensuring that the questionnaire found their way into the hands of the business process owner would be difficult if not impossible. The co-operation of senior IT management at X Inc. was sought in providing access to process owners. A total of 50 instruments were applied in 14 6

7 interviews with IT managers in X Inc. The responsibility for two control objectives were spread over three managers (DS2 Manage Third Party Services and DS6 Identify and Allocate Costs). The Table 1: Maturity level by Process Min possible =1-Max possible = 5. background of the study was explained and the nature of capability maturity models outlined. Process ML PO1 Define a Strategic IT Plan 4.19 PO2 Define the Information 3.65 Architecture PO3 Determine the Technological 4.19 Direction PO4 Define the IT Organization & 4.02 Relationships PO5 Manage the IT Investment 4.59 PO6 Communicate Management 4.45 Aims & Direction PO7 Manage Human Resources 4.44 PO8 Ensure Compliance with 3.56 External Requirements PO9 Assess Risks 3.09 PO10 Manage Projects 3.64 PO11 Manage Quality 2.72 AI1 Identify Solutions 3.82 AI2 Acquire & Maintain 3.56 Application Software AI3 Acquire & Maintain 3.01 Technology Architecture AI4 Develop & Maintain IT 1.99 Procedures AI5 Install & Accredit Systems 3.12 AI6 Manage Changes 3.18 DS1 Define Service Levels 4.95 The statements were provided to the respondents in random order to minimize bias. Questions and concerns of the respondents were answered interactively. A number of follow up questions were applied during the interview. 5. Results The level of capability maturity was calculated according to standard CMM measurement techniques. DS2 Manage Third-Party Services 3.81 DS3 Manage Performance & 3.97 Capacity DS4 Ensure Continuous Service 4.24 DS5 Ensure Systems Security 3.53 DS6 Identify & Attribute Costs 3.29 DS7 Educate & Train Users 3.96 DS8 Assist & Advise IT 2.70 Customers DS9 Manage the Configuration 3.30 DS10 Manage Problems & 3.68 Incidents DS11 Manage Data 4.36 DS12 Manage Facilities 4.19 DS13 Manage Operations 4.03 ME1 Monitor the Processes 2.65 ME2 Assess Internal Control 2.61 Adequacy ME3 Obtain Independent 3.37 Assurance ME4 Provide for Independent 3.25 Audit Total 3.63 Recall from the previous section that there were four possible responses. These responses and the weight applied to each response were Not at all (0.0); A little, (0.33)); To some degree (0.66) and Completely (1.0). Each of the weighted responses were applied to the appropriate level from 1.0 (Initial/Ad-hoc) to 5.0 (Optimised). Table 2 shows the maturity model level for the control objectives for each of the business processes. The overall capability maturity was 3.63, a surprisingly high number. 7

8 Table 2: Means, Standard Deviation and Co-efficient of Variation by Process Process Description Mean StdDev Coeff. Var. PO1 Define a Strategic IT Plan PO2 Define the Information Architecture PO3 Determine the Technological Direction PO4 Define the IT Organization and Relationships PO5 Manage the IT Investment PO6 Communicate Management Aims and Direction PO7 Manage Human Resources PO8 Ensure Compliance with External Requirements PO9 Assess Risks PO10 Manage Projects PO11 Manage Quality AI1 Identify Solutions AI3 Acquire and Maintain Technology Architecture AI4 Develop and Maintain IT Procedures AI5 Install and Accredit Systems AI6 Manage Changes DS1 Define Service Levels DS2 Manage Third-Party Services DS3 Manage Performance and Capacity DS4 Ensure Continuous Service DS5 Ensure Systems Security DS6 Identify and Attribute Costs DS7 Educate and Train Users DS8 Assist and Advise IT Customers DS9 Manage the Configuration DS10 Manage Problems and Incidents DS11 Manage Data DS12 Manage Facilities DS13 Manage Operations ME1 Monitor the Processes ME2 Assess Internal Control Adequacy ME3 Obtain Independent Assurance ME4 Provide for Independent Audit Total Table 2 shows the mean, standard deviation and coefficient of variation of responses to each statement. It can be seen that the coefficient of variation for the complete population was This is a range from 0.22 for DS1 Define Service Levels to PO9 Assess Risks (0.61). I then analyzed by capability maturity level. Table 3 shows the mean, standard deviation and co-efficient of variation by level. While of moderate concern, these results do show that a reasonable quality of metrication in the instruments that have been developed to assess the level of capability maturity. At the opposite end of the spectrum, there were a small number of processes that had relatively low levels of capability maturity. These included AI4 Develop and Maintain IT Procedures (1.99) and much improved, ME1 Monitor the Processes (2.65). Whether these results truly represent the true level of capability maturity or poor metrication is unclear. 8

9 Table 3: Means, Standard Deviation and Co-efficient of Variation-By Level Level Mean Std Dev Coeff Var N Total Conclusion Information technology plays an increasingly fundamental role in the business processes of effectively all entities. For many modern entities, the loss of IT functionality would drive the entity rapidly towards extinction. Financial and management accounting relies not only on traditional computerized accounting information systems but also on many application systems that feed data to the entries that make up the financial accounting systems. The importance of IT has been recognized by auditing standards setters over many years, but is perhaps most explicitly referenced by the PCAOB s An Audit of Internal Control Over Financial Reporting Performed in Conjunction with An Audit of Financial Statements, which calls for auditors to assess both general and application specific IT controls [2]. The importance of the strategic and operational IT risks that entities face has been recognized by a number of official standards setters such as the International Standards Organization (ISO) (e.g. ISO 15799) but also by organizations such as the IT Governance Institute (ITGI) with its COBIT control framework. The ITGI has suggested that more than 30 of the 34 control objectives over business processes that make up the COBIT framework are directly relevant to ensuring the establishment and maintenance of IC over financial reporting, as described in the Sarbanes-Oxley Act (SOX) and in the PCAOB s Auditing Standard #2 [24]. An important element of the various elements and tools that make up the COBIT framework is the Capability Maturity Model (CMM). A given level of the CMM allows managers and others to determine the capacity of the entity to manage its risks and strategic and operational outcomes, for that particular process. Whilst the CMM has been rooted in software development, the techniques are argued to be sufficiently generic to be used over a variety of engineered processes. Whether a given level of capability maturity correlates to a given level of IC, under standards such as that mandated in Auditing Standard #2, has not been determined. There is no research that allow us to understand this relationship or, for that matter, any aspect of the capability of entities across the complete lifecycle of IT investment. This paper is a first and exploratory attempt to determine the capability maturity of organizations; address metrication issues in measuring capability maturity and correlate capability maturity with the state of IC over financial reporting. A number of statements for each control objective have been developed to allow self-assessment of capability maturity. These are applied to business process owners in a large IT function. This paper is subject to a number of limitations. First, there is only data for one organization. Before a complete set of conclusions can be draw a significantly larger number of organizations must be polled. There are interesting questions that arise on attributes including the level of in-sourcing versus out-sourcing for both hardware and software; the level of internal development versus the level of purchase of packaged software and the importance of IT to the value-adding processes of the entity. Second, the research is dependent on the quality of the instruments for each of the control objectives for the business processes. Finally, I have not attempted to measure the quality of the ICFR. 7. References [1] COSO, "Internal Control - Integrated Framework," Committee of Sponsoring Organizations of the Treadway Commission, New York [2] PCAOB, "Auditing Standard No. 2 An Audit of Internal Control Over Financial Reporting Performed in Conjunction with An Audit of Financial Statements," Public Company Accounting Oversight Board, Washington, DC , March [3] T. Hoffman, "The Sarb-Ox Shift," in Computerworld, 2005, pp. 35. [4] PricewaterhouseCoopers, "The Use of Spreadsheets and Considerations for Section 404 of the Sarbanes- Oxley Act," PricewaterhouseCoopers LLP, Newark, Del [5] R. Panko, "What We Know About Spreadsheet Errors," Journal of End User Computing, vol ,

10 [6] J. N. Luftman, Competing in the Information Age: Strategic Alignment in Practice. Oxford: Oxford University Press, [7] J. N. Luftman, Competing in the Information Age: Align in the Sand. Oxford: Oxford University Press, [8] ITGI, "Board Briefing on IT Governance," IT Governance Institute - Information Systems Audit and Control Foundation, Rolling Meadows, IL [9] B. H. Reich, "Factors that influence the social dimension of alignment between business and information technology objectives," MIS Quarterly, vol. 24, pp , [10] Y. E. Chan, S. L. Huff, D. W. Barclay, and D. G. Copeland, "Business strategic orientation, information systems strategic orientation, and strategic alignment," Information Systems Research: ISR: A Journal of the Institute of Management Sciences, vol. 8, pp , [11] J. Henderson and N. Venkatraman, "Aligning Business and IT Strategies," in Competing in the Information Age: Strategic Alignment in Practice, J. N. Luftman, Ed. Oxford: Oxford University Press, [12] J. N. Luftman and T. Brier, "Assessing and Sustaining Business-IT Alignment Maturity," California Management Review, vol. 42, pp , [13] A. Barua, P. Konana, A. B. Whinston, and F. Yin, "An Empirical Investigation of Net-Enabled Business Value," MIS Quarterly, vol. Vol. 28, [14] B. Dehning and V. J. Richardson, "Returns On Investments In Information Technology: A Research Synthesis," Journal of Information Systems, vol. 16, pp. 7-31, [15] B. Dehning, Vernon J. Richardson, A. Urbaczewski, and J. D. Wells, "Reexamining the Value Relevance of E-Commerce Initiatives," Journal of Management Information Systems, vol. 21, pp , [16] B. Dehning, V. J. Richardson, and R. W. Zmud, "The Value Relevance of Announcements of Transformational Information Technology Investments," MIS Quarterly, vol. 27, pp , [17] J. N. Luftman, Managing the IT Resource. New York, NY: Prentice Hall, [18] P. Weill and J. W. Ross, "A Matrixed Approach to Designing IT Governance," Sloan Management Review, vol. 46, pp , [19] W. Van Grembergen, Strategies for Information Technology Governance. Hershey, PA: Idea Group Publishing, [20] S. De Haes and W. Van Grembergen, "IT Governance Structures, Processes and Relational Mechanisms: Achieving IT/Business Alignment in a Major Belgian Financial Group," presented at 38th Hawaii International Conference on System Sciences, Hawai'i, [21] R. Peterson, "Information strategies and tactics for information technology governance," in Strategies for Information Technology Governance, W. Van Grembergen, Ed. Hershey, PA: Idea Group Publishing, [22] ITGI, "COBIT Framework," Information Systems Audit and Control Foundation, Rolling Meadows, IL [23] J. W. Lainhart, "COBIT: A Methodology for Managing and Controlling Information and Information Technologies and Risks and Vulnerabilities," Journal of Information Systems, vol. 14, pp , [24] ITGI, IT Control Objectives for Sarbanes-Oxley. Rolling Meadows, IL: IT Governance Institute, [25] D. E. Harter, M. S. Krishnan, and S. A. Slaughter, "Effects of Process Maturity on Quality, Cycle Time and Effort in Software Product Development," Management Science, vol. 46, pp , [26] S. P. Dawson, "Continuous improvement in action: Applying quality principles to software," Information Systems Management, vol. 11, pp , [27] P. J. Denning, "What Is Software Quality?" Communications of the ACM, vol. 35, pp , [28] M. B. Chrissis, M. Konrad, and S. Shrum, CMMI: Guidelines for Process Integration and Product Improvement. Boston, MA: Addison-Wesley, [29] J. M. Juran and F. M. Gryna, Quality Planning and Analysis. McGraw-Hill: New York, NY, [30] J. M. Juran and F. M. Gryna, Juran's Quality Handbook, vol. 5th. New York: McGraw-Hill, [31] W. E. Deming, Quality, Productivity and Competitive Position. Cambridge, MA: MIT Center for Advanced Engineering, [32] W. E. Deming, Out of the Crisis. Boston, Mass: Massachetts Institute of Technology, [33] W. S. Humphrey, Managing technical people: innovation, teamwork, and the software process. Reading, Mass.: Addison-Wesley, [34] W. S. Humphrey, Managing the Software Process. Boston, MA: Addison-Wesley Professional, [35] M. Niazi, D. Wilson, and D. Zowghi, "A maturity model for the implementation of software process improvement: an empirical study," Journal of Systems and Software, vol. 74, pp , [36] J. D. Herbsleb, D. Zubrow, D. Goldenson, W. Hayes, and M. Paulk, "Software Quality and the Software Capability Maturity Model," Communications of the ACM, vol. 40, pp , [37] J. J. Jiang, G. Klein, H.-G. Hwang, J. Huang, and S.- Y. Hung, "An exploration of the relationship between software development process maturity and project performance," Information and Management, vol. 41, pp ,