Internal Audit Report

Transcription

1 Internal Audit Report IT Service Level Contract Management/Billing TxDOT Office of Internal Audit

2 Objective To assess TxDOT s process of review and approval for vendor service level achievement as a method of measuring vendor performance. Opinion Based on the audit scope area reviewed, control mechanisms are effective and sustainable and substantially address risk factors and exposures considered significant relative to impacting reporting reliability and operational execution. The organization's system of internal controls provides reasonable assurance that key goals and objectives will be achieved despite improvement opportunities identified. Improvement opportunities identified include minor enhancements that would improve achievement of (control/business) objectives but are not currently resulting in negative impacts to the organization. Overall Engagement Assessment Satisfactory Control Environment Information Technology Division (ITD) Vendor Management has the overall responsibility to manage the contractual obligation ( the Agreement ) between TxDOT and its outsourced IT service provider including the review and approval of monthly invoices and corresponding service levels. IT Vendor Management has established procedures and process maps for the monthly review of the invoices. The procedures cover the review of service level agreement and project deliverables, as well as, the issue-dispute process. The plan to manage the agreement includes procedures to verify the service level metrics (scorecard) reported by the IT service provider each month. The verification of the metrics scorecard compares the reported service level metrics, on a sample basis, to supporting documentation (i.e., service tickets) and reports any discrepancies to ITD leadership and the IT service provider to facilitate adjustments to the invoice as necessary. ITD Vendor Management also develops strategies to drive improvements in vendor performance including a defined customer relations improvement plan and additions or modifications to the established service levels based on trends in performance. Summary Results Finding Scope Area Evidence None identified Service Level Achievement No findings identified August 15,

3 Audit Scope The audit focused on the Service Level Agreement (SLA) review process, verification of reported service level metrics, and reconciliation of invoices to final payments. The audit covered the IT service provider invoices for January 1, 2014 through April 30, Testing was also conducted on the data used to support the April 2014 invoice from the IT service provider, including a sample of metrics for Critical Service Level (CSL) and Service Level Objective (SLO), as well as, documents used to support payment adjustments and the first quarter bonuses and credits. The audit was performed by Milan Hawkins, Jennifer Stanush, and Cynthia Scheick (Engagement Lead). The audit was conducted during the period from June 9, 2014 to July 30, Methodology The methodology used to complete the objectives of this audit included: Reviewing the Master Services Agreement with the IT service provider specifically relating to measuring and reporting of service levels Reviewing applicable state laws and regulations including department policies and procedures related to Procurement and Contract Payments, as well as, Contract Management Interviewing TxDOT personnel in IT Vendor Management Evaluating documentation used to support the monthly invoice and service levels reported by the IT service provider Evaluating IT Vendor Management Standard Operating Procedures and process maps governing the management of the IT service provider agreement and services These procedures were applied as necessary to perform the audit fieldwork, including testing a sample of the service level metrics for both CSLs and SLOs reported by the IT service provider. Background This report is prepared for the Texas Transportation Commission, TxDOT Administration, and Management. The report presents the results of the IT Service Level Contract Management/Billing Audit which was conducted as part of the Fiscal Year 2014 Audit Plan. In 2013, TxDOT outsourced some areas of Information Technology (IT) in an effort to provide technology advancements, new and improved services, and operational efficiencies. TxDOT entered into a Master Services Agreement, effective May 31, 2013, with an outside IT service provider for provision of IT services. The Agreement includes payments for: Transition Services (migration of services from TxDOT to the IT Service Provider) On-going Delivery Services (Application Maintenance, Service Desk, IT Security, Network, and Communications) Transformation Services (consolidation activities, implementation of new technologies, and process changes) ITD Vendor Management is responsible for the management of the Agreement, including the review of reported service level metrics and processing of the monthly invoices. Beginning January 2014, the IT service provider can earn quarterly bonuses and/or assigned credits August 15,

4 based on defined critical service levels. Each month, ITD Vendor management reviews the SLA metrics scorecard prepared by the IT service provider in support of the billed invoice. ITD Vendor Management s review of the scorecard validates if reported service levels are supported by individual service tickets. On a monthly basis, ITD Vendor Management verifies whether service level performance warrants the assessment of a credit or bonus and records this assessment on the SLA financial model. On a quarterly basis, ITD Vendor Management conducts a true-up procedure of the quarter s monthly scorecards to determine the net financial impact of service level performance and applies the net figure to the monthly invoice. We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards and in conformance with the International Standards for the Professional Practice of Internal Auditing. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. Recommendations to mitigate risks identified were provided to management during the engagement to assist in the formulation of the management action plans included in this report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Office of Internal Audit transitioned to Committee of Sponsoring Organizations of the Treadway Commission (COSO) Internal Control Integrated Framework version 2013 in December A defined set of control objectives was utilized to focus on reporting and operational goals for the identified scope areas. Our audit opinion is an assessment of the health of the overall control environment based on (1) the effectiveness of the enterprise risk management activities throughout the audit period and (2) the degree to which the defined control objectives were being met. Our audit opinion is not a guarantee against reporting misstatement and reliability or operational sub-optimization, particularly in areas not included in the scope of this audit. Best Practices The following planning control activities were noted during the audit: The Agreement includes a Governance Model with key principles that state service management processes shall be implemented, performance shall be measured and monitored, and clear accountability shall be established IT Vendor Management established procedures and process maps for key control areas including; invoice processing, SLA review, deliverable (transformation) review, and the issue-dispute process Implementation of a Customer Relations Improvement Plan with weekly updates by the IT Service Provider An effective process map focuses on who does what, in what order, how long it takes and shows when and where critical decisions are made. Our review of the SLA review process map showed it documents the key levels of communication, responsibilities for review, levels of approval, and demonstrates how each of the roles intersect. This results in a better understanding of the responsibilities and contributions each staff and/or section has in the process. August 15,

5 Observations and Recommendations Audit Observation (a): Service Level Agreement Review Methodology ITD Vendor Management Service Level Agreement (SLA) review process focuses on validating Critical Service Levels (CSL) which ties directly to bonuses or credits for the IT service provider, and Service Level Objectives (SLO) which do not tie to a bonus or credit. The review process, performed by two IT TxDOT staff, includes testing a sample of tickets tied to these two SLA metrics. Although the review focuses on a larger sample of CSL versus SLO metrics, additional sampling coverage should be considered for both areas. Testing for the month of April 2014 for CSL and SLO areas were 43% and 0.3%, respectively. Effect/Potential Impact SLA metrics with a higher potential to contain errors (i.e. priority level assignment) may go undetected and could result in payment of bonuses or a non-issuance of a credit. Audit Recommendation ITD Vendor Management should formalize a risk based sampling methodology to include: sampling from all SLO metrics that could impact the achievement of a CSL (i.e. changes in the priority level or incorrect ticket assignment documenting rationale for metrics selected or not selected for review The necessary review of tickets is a manual and time consuming process; however, efficiencies and an increase in coverage can be gained by: splitting up the ticket sample between the two reviewers dedicating additional resources to assist with the review or reprioritizing workloads creating standard templates to document the number of tickets sampled, exceptions identified, and trends noted Consideration should also be given to enhancement of the SLA Review procedures to include the documentation necessary to support the results of the review. August 15,

6 Summary Results Based on Enterprise Risk Management Framework Rating Assessment Grid Exemplary Satisfactory Needs Improvement Unsatisfactory Closing Comments The results of this audit were discussed with the Chief Information Office and ITD Vendor Management Director on August 7, We appreciate the assistance and cooperation received from the ITD Vendor Management employees contacted during this audit. August 15,