Statement of Work. Enterprise Resource Planning System (BSR) Evaluation of Procurement Process Phase II. Submitted To:

Transcription

1 Statement of Work Enterprise Resource Planning System (BSR) Evaluation of Procurement Process Phase II Submitted To: Manatee County School District Board, Audit Committee, and Management Prepared by: Shinn & Company LLC April 5, 2016

2 Statement of Work Manatee County School District Evaluation of Procurement Process Phase II TABLE OF CONTENTS Page Executive Summary... 1 Scope... 3 Methodology... 4 Solicitation Process... 4 Task Order 1 Review Process... 6 Cost Evaluation Process... 7 Table 1 Final Cost Estimate... 7 Table 2 Top 14 Florida School District ERP Survey... 8 Conclusion... 9 Recommendation for the Board: Internal Audit Next Steps Appendix... 13

3 Phase II Phase I Executive Summary The School District of Manatee County requested Internal Audit to provide an implementation review of the Enterprise Resource Planning (ERP) project in order to appraise the effectiveness, efficiency, and transparency of the procurement method, selection process, and ERP implementation as a whole. In order to meet this objective, Internal Audit has reviewed the Procurement Process Phase II, which included a Request for Quote (RFQ) publication, research, review and evaluation of vendor proposals, review of the final Task Order 1, and a recommendation as presented to the Board. In order for a large scale project of this nature to be successful, the District needs: Effective leadership and executive sponsorship, Experienced, committed project director, Engaged and effective project team, In depth research and understanding of the product and market, Transparent, thorough, and legally compliant procurement process, Recognizable and clearly defined business requirements, Strong support and focus on change management and business process reengineering entity wide, Qualified, competent, and experienced implementation partner, o Adequate assignment of implementation resources both money and people, o Tight, yet flexible implementation goals, and o Long term, effective, and redundant training/retraining program for staff. Through our review of the Request for Quote (RFQ), subsequent questions and answers from vendors, vendor submitted proposals, business requirements, RFQ scoring matrix and results, approved Task Order 1, along with observation and discussions with management and project team members, we have found the school district has selected the most qualified implementation partner from the pool of four proposals received which will afford the school district the best opportunity to implement this project efficiently and effectively on the superintendent s timeline. The procurement method followed by management and the project team and the resulting Task Order 1 is consistent with best practices and adheres to State Board of Education Rules 6A-1.012(4), of the Florida Administrative Code. The school district selected vendor must have been a pre-qualified vendor on the State of Florida contract for Information Technology (IT) Consulting Services ( ) in the applicable project area. The school district requires an experienced, Florida K-12, implementation partner that has: previously successful legacy system upgrade or PeopleSoft 9.2 implementations in a Florida school district of like size and nature; dedicated staff with extensive PeopleSoft 9.2 functional and technical lead business experience; strong project and change management plans administered by dedicated resource teams; robust and extensive documentation and training programs for the user community; 1

4 knowledgeable and previously proven data cleansing methods; and will recommend a secure, real time, cost effective cloud based hosting environment. At this time, the Board, management, and personnel continue to be on point and on target for launching a successful pre-implementation environment with appropriate objectives to meet the school district needs in a timely and efficient manner. The board, superintendent and project staff should be commended for expending the appropriate time and effort to thoroughly review and select a qualified, competent, and experienced implementation partner. An implementation partner can make or break a project of this magnitude. Experience and references and proven implementation track records should far outweigh the cost factors and scoring in selecting the best implementation partner. While funds are not unlimited, poor selections for implementation partners because of reduced rates/costs can actually increase the cost of the project or stall it completely. Based on the information provided by the project team and vendor selection process reviewed, Internal Audit finds that the process followed and the recommendation being made to the Board, selection of Ciber, Inc., as the implementation partner for the school district s Oracle PeopleSoft ERP system from the three state pre-qualified vendor proposals, followed all required procurement rules, meets the school district s requirements, and is in the school district s best interest. Next Steps Based on the review of the Task Order 1 (SOW), the following phases are to be evaluated and reported on by Internal Audit (IA): 1. Review of Design Documentation; 2. Security Review; 3. Re-performance of Critical Calculations and Key Reports Review; 4. User Acceptance Testing; and 5. Observation of onsite training and Training Manual Review 2

5 Scope The Manatee County School District management is pursuing implementation of a single, districtwide Tier- 1 ERP Software Package to enable greater financial transparency and compliance with applicable laws and regulations and reduce overall costs and risks by eliminating the need to maintain multiple systems thus improving efficiency, transparency, financial information, and reducing audit findings. The ERP Software Package will empower district leadership with critical and timely insights into the District s operations, helping them to improve management of district initiatives and current resources. The Board and Audit Committee goal for this engagement require Internal Audit to evaluate this process and meet the following objectives: 1. Perform an evaluation of the procurement method and process used to select and purchase new ERP System software. This would include, the review of all documentation to support the process, including but not limited to: a. Attend vendor presentations; b. Meeting minutes; c. Attendance sheets; d. Surveys; e. List of District Employees with voting rights; f. The documented method of the evaluation and weighting of criteria; g. The initial results from GAP Analysis and Site Visits; and h. The final recommendation to go before the Board for Approval. 2. Perform an additional step to evaluate the procurement method and process used to select and contract with an implementation partner to assist with deployment of the ERP system software, PeopleSoft, selected in Objective 1 above. This would include, the review of all documentation to support the process including, but not limited to: a. Review RFQ process managed by Procurement through the school district procurement portal; b. Meeting minutes; c. Attendance sheets; d. The documented method of the evaluation and weighting of criteria; e. Task Order 1 for selected vendor; and f. The final recommendation to go before the Board for Approval. Internal Audit s engagement was conducted under the consulting standards established by the American Institute of Certified Public Accountants. 3

6 Methodology The purpose of this report is to provide Internal Audit s evaluation of the procurement and review methodology used to select and contract for the new ERP software and implementation partner. In order to gain sufficient understanding of the procurement and evaluation process as a whole, Internal Audit performed various steps to determine the scope of review. Internal audit s evaluation included best practice and statutory research, observations and the review of all supporting documentation, including but not limited to: Review of Florida Statutes related to procurement for school districts; Review of existing State of Florida contract for IT consulting services for PeopleSoft implementations; Research on industry best practices for ERP implementation; Review and evaluation of the school district surveys; Review of the RFQ document, vendor questions and answers, and vendor responses; Review of team meeting minutes evaluating and scoring the vendor responses; Review of cost analysis performed including comparison to previously completed implementation costs for Florida K-12 schools; Examination of attendance sheets; Evaluation of the methodology utilized to review vendor response; and Evaluation of the final recommendation, which was presented to the Board for approval to contract with software and implementation vendors. Solicitation Process Internal Audit first reviewed the published Request for Quote (RFQ No CC) which stated the following Solicitation Objective (Section 1.2) and Scope of Work (Section 4): The District is issuing this RFQ to collect quotes on the State of Florida IT Consulting contract for the purpose of integrating PeopleSoft ERP solution and related services. This request does not preclude, the use of an existing competitive solicitation as provided by Subsection 6A (8) F.A.C (5) that allows school boards to purchase from existing bids/responses of other public entities upon agreement by the awarded vendors(s), nor direct negotiation with vendors as provided by Subsection 6A-1.012(8) F.A.C (14), nor no action on behalf of the District.) 4 Manatee County Public Schools is seeking quotes from qualified vendors to provide project management, implementation and change management services for implementation of PeopleSoft 9.2, including but not limited to the following modules: HCM, FSCM (including Hyperion), and ALM. In addition, the RFQ requested a separate cost breakout for hosting fees and ongoing maintenance costs for the ERP system. 4

7 Internal Audit reviewed the three qualified vendor proposals to the RFQ from the following vendors: Ciber, Inc., Meridian Partners, and Highstreet IT Solutions. Specifically Internal Audit: 1. Verified the vendor was an awarded vendor as required on the State of Florida IT consulting contract by reviewing each vendors inclusion on the approved contract list. _contracts_and_agreements/state_term_contracts/information_technology_it_consulting_services /contractors/h_l One additional vendor was eliminated because the vendor was not on the State of Florida contract as required; 2. Verified vendor submitted a complete proposal by the solicitation due date; 3. Reviewed the RFQ scoring matrix and minutes of the scoring meeting provided by the procurement office to ensure that the final scoring matrix reflected the discussion and point scoring as recorded and seemed fair and unbiased. Internal Audit validated that no member of the scoring committee had a conflict of interest with the proposing vendors; 4. Reviewed each vendor proposal following the RFQ scoring matrix evaluation criteria used by the school district project team. Specifically, Internal Audit independently scored the proposals using the scoring matrix evaluation criteria. Internal Audit then compared those scores to the scores awarded by the project team to determine if the scores appeared fair and reasonable based on the proposals as submitted. Internal Audit s review focused specifically on the vendor responses for: a. staff resumes proposed; b. change management method/delivery and communication plan; c. business requirements evaluation and prototype design methods proposed; d. conversion and interface training delivery method quantity and staffing as well as materials recommended; e. testing protocol and timing; f. ability to meet proposed implementation aggressive timelines; g. post implementation resources; h. adequacy of the cost estimate; and i. references provided demonstrate ability of vendor to produce expected results in the methods and timeframes proposed without significant cost overruns or change orders. Internal Audit determined after review of the proposals and the procedures followed by the procurement office, the school district followed all the appropriate procedures in awarding the solicitation to Ciber, Inc. 5

8 Task Order 1 Review Process Once the vendor selection was determined and awarded to Ciber, Inc., the school district and the vendor worked together to develop the detailed, specific Task Order 1. This document becomes the implementation bible by which the project is bound. Internal Audit reviewed the Task Order 1 for the following attributes specifically: 1. Applicable statutory contract requirements for Jessica Lunsford, FOIA, Indemnification, Immunity, Appropriation, etc. as well as attorney review and approval; 2. Overall project plan and approach; 3. Detailed description of goods/services to be provided; 4. Project organization and governance; 5. Period of performance; 6. Deliverable method and requirements; 7. Specific software and equipment purchases/platform required; 8. Specific tasks/work products to be accomplished; 9. Non ERP systems integration needs; 10. Data conversion needs identified; 11. Outsourced computer operations expectations; 12. Change management and communication plan; 13. Training and documentation plan; 14. Testing and acceptance plan; 15. FTE s by project area and detailed agreed upon costs; 16. Change order process and approval; 17. Deliverable and approval process; 18. Performance measurement and reporting process; 19. Vendor and school district roles and responsibilities; 20. Post implementation and warranty period; 21. Security requirements; and 22. Risk Factors and mitigation. Internal Audit found the Task Order 1 to follow standard business practices and be thorough in the level of detail and subject matter included. The Task Order 1 did outline various project risk factors identified by Ciber, Inc. (See Appendix Table 1- SOW Table 11) some of which were more thoroughly reviewed with the project management team and the superintendent during various conversations. These risks are relatively normal in this type of project and will be mitigated to the extent possible by the vendor and the project team. These risks should be included in regular reporting to management and the project steering committee. 6

9 Cost Evaluation Process Internal Audit reviewed the cost evaluation process to determine if the recommendation to management and the board for the approval of the Ciber, Inc. contract seems reasonable and fair and in line with standard ERP system implementation costs. In order to perform this evaluation, Internal Audit obtained all the cost evaluation documents from the project team. These documents included a comparison of the original cost estimate presented to the Board at the January 26, 2016 board meeting to the final proposal submitted by Ciber, Inc. and included in the Task Order 1 dated February 26, See Table 1 below. Table 1 Final Cost Estimate ERP Cost Estimate Considerations 26-Jan 12-Feb Detail Detail Difference Travel Costs $ 685, $ 794, $109, FTE Hours Change Management Cost $ 357, $ 347, ($10,210.00) FTE Hours 2, , Discovery Cost $ 1,086, $ 1,055, ($31,040.00) FTE Hours 6, , Data Cleanse and Conversion Cost $ 341, $ 181, ($160,395.00) FTE Hours 4, ,500.0 (1,950.0) Integration Cost $ 126, $ 181, $55, FTE Hours 4, ,500.0 (1,950.0) Other Development $ 170, $ 170, $0.00 FTE Hours 2, , Functional Implementation Suppo $ 2,768, $ 2,949, $181, FTE Hours 19, , ,431.0 Hyperion Implementation Cost $ 510, $ 510, $0.00 FTE Hours 3, , Training Costs $ 142, $ 550, $408, FTE Hours , Post Production Support Costs $ 271, $ 271, $0.00 Total FTE Ciber Inc.* (6.0) Total Hours Ciber Inc. 42, ,799.0 (5,980.0) Oracle Software Costs $ 1,845, $ 2,814, $ 969, Licensing $ 675, $ 656, $ (18,135.68) Support Fees $ 900, $ 1,925, $ 1,025, Hyperion License $ 270, $ 232, $ (37,742.00) Detail 1/26 Detail 2/12 Difference Total Implementation Cost $ 6,461,295 $ 7,013,211 $551, Internal Hosting Costs $ 200,000 $ - ($200,000.00) Software Cost - Initial Purchase $ 969,000 $ 1,009,659 $40, Software Cost - Years 2 thru 5 $ 876,000 $ 1,804,978 $928, Total Software Costs $ 1,845,000 $ 2,814,637 $969, TOTAL COST $ 8,506,295 $ 9,827,848 $1,321,

10 Internal Audit reviewed the discrepancies in cost estimates along with the thorough review and explanation of the variances received from the project director and project team. The team went to great lengths to ensure that all the business requirements would be met while at the same time paid specific attention to areas where the school district resources could be utilized to reduce out of pocket costs further. All the additions and reductions to the cost estimate seem reasonable in comparison to the work effort, product, and timeline goals. In order to effectively evaluate whether this was a reasonable cost estimate for a system of this size, Internal Audit reviewed the previously obtained school district surveys. In these surveys, the school district requested ERP system information from all the school districts in the state. While the information is not audited financial information, Internal Audit believes it to be reasonable and from a valid source to use as a basis for comparison only. See Table 2 below for a comparison of the Top 14 ERP Implementation Costs, school districts with similar software implementations highlighted. Table 2 Top 14 Florida School District ERP Survey List of School Districts by is by Student Population FY 2011/2012 Rank School District Name Number of Students ERP Solution Implementation Date Purchase Price/ License Fees Implementation Fees Annual Support/ Maintenance Fees Upgrade Fees Customization Fees Agency Own Source Code? 1 Miami-Dade 350,239 SAP 2005 $85,000,000 Unknown Unknown Unknown Unknown Unknown 2 Broward 258,478 SAP 2007 $45,000,000 Included $960,000/yr. Included Additional No 3 Hillsborough 197,041 Infor's Lawson 2003 $2,000,000 $5,000,000 $650,700/Yr. Included In-House Yes 4 Orange 180,000 SAP 1998 $10,800,000 Included $649,299/Yr. Included Additional No 5 Palm Beach 176,901 PeopleSoft (Oracle) 2006 $4,107,253 $20,618,000 $534,190/Yr. Included Additional Yes 6 Duval 125,429 SAP 1999 $10,542,000 Unknown $428,000/Yr. Included Additional Yes 7 Pinellas 103,776 TERMS 1994 Unknown Unknown Unknown Unknown Unknown Unknown 8 Polk 96,070 SAP 2000 Unknown Unknown Unknown Unknown Unknown No 9 Lee 83,895 PeopleSoft (Oracle) Brevard 71,792 TERMS 2002/ Pasco 66,659 Tyler Technologies/ Munis 12 Seminole 64,344 PeopleSoft (Oracle) 13 Volusia 61,524 SmartStream (Infor) 14 Osceola 54,783 TERMS 1998 ENTERPRISE RESOURCE PLANNING (ERP) SURVEY 2015 update $32,465,432/ 3 Yrs. Original 2002 ERP: $1,100,000; Re-Fresh 2013: $5,200,000 Included $412,000/Yr. Included Additional No Included 2012 $2,141,870 $6,630, : $150,000 to 2017: $560,000/Yr. $5,189,470/5 Yrs. Included Included Yes Included Additional In Escrow 1998 $5,000,000 Included $400,000/Yr. Included Additional No 2002 Unknown Unknown $181,622/Yr. Included Additional Yes $1,300,000 (Estimate) Included Cancelled In-House In-House Yes 8

11 As noted in the table above and compared to the final cost of the Task Order 1 approved, the school district is implementing this system for a significant cost savings over Lee County School District by implementing a clone version of their system as recommended in the Ciber, Inc. proposal. The cost is at a reasonable proportional rate to the school districts of a more significant size taking into consideration the year of the implementations and adding that Manatee County School District is implementing a significantly more robust and fully integrated system with as many modules replacing old systems as possible. While the selected vendor is not the lowest cost submitted, the proposal provided by Ciber, Inc. scored significantly higher than the others. Quality experience far outweighs the cost alone. While funds are not unlimited, poor selections for implementation partners because of reduced rates/costs can actually increase the cost of the project or stall it completely. Lastly, due to the magnitude of this system and the level of effort to implement, the school district should assume that the useful life of the system is 10+ years. Internal Audit reviewed the estimated full cost of ownership for the next 10 years to determine the reasonableness of those costs and to ensure there were no unexpected costs. These costs do not contemplate upgrades or additional enhancements the school district could make to the system. It also does not include normal ongoing training or support costs that would be necessary to do configuration and other updates as requested by the system users. The assumption is that these resources will be handled by the current IT staff with some ongoing PeopleSoft training. While these costs can be minimized, they should be considered as part of the ongoing maintenance of the system. ERP Budgetary Cost Estimate Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10 Total Total Total Total Total Year 1-5 Total Total Total Total Total Total Year 6-10 Total 10 Year Total Implementation Costs $ 2,004,194 $ 5,009,017 $ - $ - $ - $ 7,013,211 $ - $ - $ - $ - $ - $ - $ 7,013,211 Software Purchase Cost $ 656,864 $ - $ - $ - $ - $ 656,864 $ - $ - $ - $ - $ - $ - $ 656,864 Software Maintenance & Support Costs $ 306,343 $ 387,026 $ 398,637 $ 410,596 $ 422,914 $ 1,925,514 $ 444,059 $ 466,262 $ 489,575 $ 514,054 $ 539,757 $ 2,723,215 $ 4,648,729 Hyperion Maintenance & Support Cost $ 46,452 $ 46,452 $ 46,452 $ 46,452 $ 46,452 $ 232,258 $ 46,452 $ 46,452 $ 46,452 $ 46,452 $ 46,452 $ 232,258 $ 464,516 On Site Hosting Costs $ - $ - $ - $ - $ - $ - $ - $ - $ - $ - $ - $ - $ 200,000 Direct Implementation Cost Total $ 3,013,852 $ 5,442,495 $ 445,088 $ 457,047 $ 469,365 $ 9,827,848 $ 490,511 $ 512,714 $ 536,027 $ 560,506 $ 586,208 $ 2,955,473 $ 12,783,321 Conclusion Internal Audit has thoroughly reviewed all the documentation provided by the project team related to the procurement and selection process for the PeopleSoft implementation partner. Internal Audit found that procurement processes and best practices were followed in order to select the most qualified vendor. The vendor has the Florida school district K-12 experience, breadth of knowledge, professional work references, proven implementation track record, and partnership with experienced training resources to assist the school district in an effective, efficient, and timely ERP implementation. It is Internal Audit s opinion that the school district has selected the appropriate vendor in the best interest of the school district. 9

12 Recommendation for the Board: Internal Audit recommends the Board continues to support the District superintendent and project team to enable the ERP system to be successful by: o o o o o Monitoring key risks and providing resources where appropriate to mitigate those risks as identified by the project team and the vendor listed in Appendix Table 1; All projects have inevitable risks and the risks identified are not unusual for a project of this magnitude and timeframe. The key to successful projects is the ability to monitor those risks and make informed decisions and choices to mitigate them as much as possible. Keeping abreast of the project through bi-monthly status updates; The project team has a fabulous status report they used every week to gauge status of the project. A similar bi-monthly report would allow the board and audit committee to monitor and gauge the status of the project in order to stay informed. Because this is an aggressive schedule and tight budget, the project team and the vendor will need to ensure project status is monitored very closely and will need swift responses to resource needs at various times throughout the project. Adequately assigning operations and implementation resources both money and people; The greatest opportunity for failure in the design and implementation phase and beyond is the inadequate allocation of staff resources for the project team and the steady state operations group when the project is fully implemented. It is normal and realistic to assume there will be change orders and some budget increases necessary for a project of this magnitude. Staying in touch with the status and monitoring progress will keep the board from having any surprises when applicable change orders and budget increases are requested. Long term, effective, and redundant training/retraining program for staff; Implementing a system of this magnitude is a monumental task especially considering the number of employees and users of the system. Again while funds are not unlimited, ensure that an appropriate amount of resources staff and money are allocated to the ERP project to ensure that once the system is live, the district can reap the full rewards with quality trained staff using the system appropriately. The benefit far outweighs the cost. Commending all district employees who participated in Phase II procurement process for a very thorough and timely review and recommendation. These exceptional project members were engaged and put forth a great effort to plan, review, and evaluate the vendor proposals and Task Order 1 to ensure the district would be have the best opportunity for a successful project launch. 10

13 Internal Audit Next Steps - ERP System Pre-Implementation Review 1. Review design documentation for the purpose of identifying potential information system and/or process control gaps. This will entail: a. Participating with MCSD staff in design documentation sessions; b. Access to MCSD ERP team share pointe site to enable review of all system design documentation including review and sign off processes. Specifically reviewing system workflow diagrams, business process designs, functional designs, FIT/GAP analysis, and the final designs. 2. Review security matrix or topology strategy and data flow documentation to proactively look for potential vulnerabilities related to internal controls, segregation of duties (including the District s planning for the establishment of roles/responsibility access controls). Specifically reviewing and testing the security levels and rights to ensure the district is following the concept of least privilege rights and ensuring that all security levels have adequate segregation of duties established. 3. Review and validate system mapping and re-perform critical calculations and key reports. Specifically review MCSD conversion strategy for completeness, timeliness and accuracy. Performing baseline testing to ensure all reporting is accurate, complete, valid, and consistent between business analytics tools and modules. 4. Monitor User Acceptance Testing. This step will entail reviewing the user acceptance testing plan and determining if test scenarios cover all major functions and control workflows in main modules. This will entail re-performing some tests as well as reviewing tests performed by MCSD staff to ensure that test results are accurately documented and changes if necessary are made to the system and then validate that retesting is performed and reversion testing is complete as each new module is brought on line. 5. Monitor live system training, review training materials in an effort to assess their efficacy. Specifically review the training methodology to ensure that staff is adequately trained in applicable modules. Ensure that training information is documented and available for future reference. 6. Review system documentation to ensure that design documents are adequate and reflect final implemented system. 7. Monitor change orders to ensure that changes are reflective of additional resource needs and were not already covered in the approved Task Order1. 8. Provide monthly report to the Board, audit committee, and management by the first day of the month effective May 1, 2016 in the following format: a. Summary including current status: this will include an overview of the work performed as well as current status of project in relationship to the overall project timeline. This will be reflective of a status as green, yellow, and red. i. Green, good progress has been made, no current issues and project is on schedule, and risks are currently low. ii. Yellow, adequate progress has been made; current issues have been identified along with schedule issues as appropriate. Any risks are moderate, but definitely need to be monitored. iii. Red, inadequate progress has been made, there is a specific control deficiency or issue present, the project is completely off schedule. 11

14 b. Procedures performed last month: this will include an overview of the specific work performed by internal audit during the past month. It will be a summary by preimplementation review area. c. Procedures to be performed in the upcoming month: this will include an overview of the specific work planned by internal audit during the past month. It will be a summary by preimplementation review area. d. Risks/Control Deficiencies Identified: this will include an ongoing status of control deficiencies or risks identified by internal audit, integration partner or school district and the appropriate status and remediation plan for the control deficiency or risk. Below is a draft timeline for Internal Audit tasks above and hours assigned for the Task Order 1. Internal Audit timelines are dependent upon the District s progress and will adjust as needed according to updates from the District Project Team Liaison. INTERNAL AUDIT TENTATIVE PLAN W/ESTIMATED HOURS* No Task Estimated Hours 1 Review Design Documentation Security Review 75 3 Reperformance 90 4 User Acceptance Testing Review 80 5 Training Review 50 Remaining hours on project 440 * Assuming documentation is provided timely and there are no major concerns Reporting will include memorandums after each task is completed and CFO Monthly Reports 12

15 Appendix 13

16 14

17 15

18 16