Benchmarking of COBIT 5 PAM Assessments Performed in Brazilian Public Sector Banking Organizations

Transcription

1 DISCUSS THIS ARTICLE Benchmarking of COBIT 5 PAM Assessments Performed in Brazilian Public Sector Banking Organizations By Joao Souza Neto, Ph.D., CGEIT, CRISC, PMP, Geraldo Loureiro, CRISC and Diana Santos, PMP COBIT Focus 24 August 2015 This article presents the process capability assessments of the governance domain of COBIT 5 for 3 Brazilian public sector banking organizations. The goal was to put the new assessment ruler (Process Assessment Model [PAM], based on the ISO/IEC approach) into practice and verify how the organizations would perform while employing criteria that are different from the ones used in the COBIT 4.1 Capability Maturity Model (CMM) approach. Methods In the Brazilian public sector, the Court of Audit implements a biannual survey on IT governance and management. In order to have an impression of the governance capabilities of some organizations in the survey, this research began by using the new assessment ruler on the processes of the COBIT 5 Evaluate, Direct and Monitor (EDM) domain. The capability assessments for level 1 of the governance domain for all 3 organizations were assessed. PAM defines 3 assessment classes, distinct in their purpose, assessor requirements and evidential requirements. A class 3 assessment was chosen because it generates capability assessment results that may indicate critical opportunities for improvement or identify key issues that would support a later class 1 or 2 assessment for the processes in question. This class of assessment considers only 1 instance of each process, evidences and execution by an independent certified assessor. 1 Prior to the assessment, each organization received a nondisclosure agreement signed by the assessors and also answered a prequestionnaire consisting of: The unit to be assessed The sponsor of the assessment The local assessment coordinator Managers who will participate in the assessment Assessors Scope of the assessment (capability level 1 of the EDM processes) Target rating levels Current governance problems related to the EDM processes Class of the PAM assessment (class 3) COBIT 4.1 and COBIT 5 knowledge of the team Awareness of what is involved in a PAM assessment 1 P a g e

2 Some information provided by the organizations in the prequestionnaires is presented in figure 1. Figure 1 Information Included in the Prequestionnaire Organizations The Unit to be Assessed The Role of the Sponsor The Roles of the Managers Current Governance Problems A IT governance planning, building and support (GPBS) IT director Executive manager of GPBS, executive manager of IT infrastructure, IT manager, IT director advisor Communication with the stakeholders is irregular and occurs mainly on demand. IT investments management does not address the whole life cycle, jeopardizing cost management efforts. B IT governance department IT governance general manager IT executive manager, information systems manager, IT consultant, senior IT executive assistant IT risk is not adequately appropriated by the operational risk manager, which increases the organization s risk exposure. Asset management has a limited scope, hindering a global total cost of ownership (TCO) analysis. C IT department Head of the IT department Deputy head of the IT department, head of IT infrastructure division, deputy head of IT infrastructure division, IT coordinator, IT IT value delivery is not managed properly, so it is always difficult to clearly show all the benefits that IT brings to the business. The organization s 2 P a g e

3 analyst risk appetite is not taken into account in the design of ITenabled business solutions. Regarding the property of the rating records, it was agreed that the assessment records would be owned by the assessors and the organizations and, in case of publication of the results of the assessment, neither the name of the organization nor information that may enable its identification would be disclosed. The first part of the assessment was a meeting with the stakeholders where the sponsor explained the objective and the importance of the assessment. The assessors and the local assessment coordinator were presented, COBIT Process Assessment Model (PAM): Using COBIT 5 was explained, the EDM processes goals, practices, activities and outcomes were detailed, and the prequestionnaire was discussed. In this meeting, the stakeholders were asked to choose the target level for their actual processes that mapped to the EDM processes. Regarding target levels, 2 organizations wanted to target governance processes as fully achieving PA1.1 (fully performing processes), whereas the third organization favored targeting processes that performed largely as described in COBIT 5. These levels are defined in PAM according to a standard rating scale defined in the ISO/IEC standard. These ratings consist of: N Not achieved. There is little or no evidence of achievement of the defined attribute in the assessed process. P Partially achieved. There is some evidence of an approach to, and some achievement of, the defined attribute in the assessed process. Some aspects of achievement of the attribute may be unpredictable. L Largely achieved. There is evidence of a systematic approach to, and significant achievement of, the defined attribute in the assessed process. Some weaknesses related to this attribute may exist in the assessed process. F Fully achieved. There is evidence of a complete and systematic approach to, and full achievement of, the defined attribute in the assessed process. No significant weaknesses related to this attribute exist in the assessed process. Before starting the assessment, the rules for attribute classification were presented. For the PA1.1 assessment of EDM processes, there are exactly 3 outcomes to be verified in each process. Here is an extract of the rules proposed: If the outcomes are F, F and L, the process is considered L. If the outcomes are F, F and P, the process is considered P. If the outcomes are F, F and N, the process is considered P. If the outcomes are F, L and L, the process is considered L. If the outcomes are F, L and P, the process is considered P. If the outcomes are F, L and N, the process is considered P. If the outcomes are F, L and L, the process is considered L. In the second part, the capacity of each EDM process was evaluated at level 1 (PA1.1). The assessment consisted of discussing the purpose and the outcomes of each process, and the statements made had to be confirmed with evidence. In all organizations, evidence was presented electronically, on a large-screen monitor that displayed the contents of the intranet of the organizations. 3 P a g e

4 Four days after the assessment, all organizations received a detailed report with a list of the evidences provided for each outcome, the results obtained and suggestions on how to improve the capacity of each process with a gap. Figure 2 presents some suggestions for organization B. Figure 2 Suggestions for Gap Reduction Outcome Evidence Rating Gap Actions to Reduce the Gap EDM02.01: The enterprise is securing optimal value from its portfolio of approved IT-enabled initiatives, services and assets. None N F-N Make evident how IT adds value to the organization and how the value added is optimized. EDM02.02: Optimal value is derived from IT investment through effective value management practices in the enterprise. None N F-N Implement IT value management practices in the organization. One valuable lesson learned is that it is clearer for organizations to start by assessing the outcomes, not the purpose of the process. By doing so, the assessment of the purpose becomes straightforward. Results Though the sponsor and participants were briefed before the assessments about the differences in the COBIT 5 and COBIT 4.1 approaches, the results were disappointing for the organizations because they are used to very high maturity levels in the COBIT 4.1 model. Figures 3, 4 and 5 show the combined attribute classification, the corresponding capacity level, and the gap to reach classification F or L for each organization. The gap was classified as absent when the attribute had already reached the target level; small when the attribute was P for a target level L, or L for a target level F; and significant when the attribute was N for a target level F or L, or P for a target level F. Figure 3 Assessment for COBIT 5 Governance Processes of Organization A Governance Domain Attribute Classification Capability Level Gap for Classification F EDM01 Ensure governance framework setting F 1 Absent EDM02 Ensure benefits delivery F 1 Absent EDM03 Ensure risk optimization F 1 Absent EDM04 Ensure resource optimization P 0 Significant EDM05 Ensure stakeholder transparency P 0 Significant Figure 4 Assessment for COBIT 5 Governance Processes of Organization B Governance Domain Attribute Classification Capability Level Gap for Classification F 4 P a g e

5 EDM01 Ensure governance framework setting F 1 Absent EDM02 Ensure benefits delivery N 0 Significant EDM03 Ensure risk optimization N 0 Significant EDM04 Ensure resource optimization N 0 Significant EDM05 Ensure stakeholder transparency P 0 Significant Figure 5 Assessment for COBIT 5 Governance Processes of Organization C Governance Domain Attribute Classification Capability Level Gap for Classification L EDM01 Ensure governance framework setting F 1 Absent EDM02 Ensure benefits delivery N 0 Significant EDM03 Ensure risk optimization P 0 Small EDM04 Ensure resource optimization L 1 Absent EDM05 Ensure stakeholder transparency P 0 Small Discussion The assessments highlighted that the capability level of process EDM05 Ensure stakeholder transparency is 0 in all 3 organizations, as seen in figure 6. It was verified that if information was not demanded, or pulled, it would not be easily accessible. Figure 6 Benchmarking of the Organizations for Level 1 of EDM Processes Organization A Organization B Organization C EDM01 Ensure governance framework setting EDM02 Ensure benefits delivery EDM03 Ensure risk optimization EDM04 Ensure resource optimization EDM05 Ensure stakeholder transparency P a g e

6 In some organizations, there were doubts about what could be considered evidence of value management and resource optimization. Conclusion The main benefit perceived in the COBIT 5 process capability assessment was the detailed analysis of the execution of the process being evaluated, assessing whether the process achieves its goals and produces the required outcomes. Another important benefit was the introduction of the COBIT 5 PAM for these organizations, which had never tried an ISO/IEC type of assessment for enterprise IT governance. This research is an ongoing activity of the ISACA Brasilia (Brazil) Chapter. Joao Souza Neto, Ph.D., CGEIT, CRISC, PMP Has more than 8 years of experience in IT governance, applying COBIT within Brazil Post. He is also responsible for the IT governance research area in the Universidade Catolica de Brasilia (Brazil). He is founder and educational director of the ISACA Brasilia Chapter. Geraldo Loureiro, CRISC Has more than 8 years of experience in IT auditing, applying COBIT, and more than 8 years of experience as a chief information officer in public organizations in Brazil. He is founder and president of the ISACA Brasilia Chapter. Diana Santos, PMP Is chief IT advisor at a Brazilian federal government agency related to the judiciary. She has collaborated on IT governance and the implementation of COBIT processes. She is marketing and communication director of the ISACA Brasilia Chapter. Endnotes 1 ISACA, Assessor Guide: Using COBIT 5, USA, P a g e