TxDOT Internal Audit Texas Permit Routing Optimization System (1404-2) Department-wide Report

Transcription

1 Introduction Texas Permit Routing Optimization System (1404-2) Department-wide Report This report has been prepared for the Transportation Commission, TxDOT Administration and management. The report presents the results of the audit of Motor Carrier Division s (MCD) Texas Permit Routing Optimization System (TxPROS). The audit was added to the Fiscal Year 2009 Audit Plan at the request of the Assistant Executive Director for Support Operations and approved by the Executive Director. The objective of the audit is to determine whether the TxPROS project is progressing as planned and the vendor is delivering a quality product. Scope Audit team members included Kathy Baca and Donna Roberts. The audit work was conducted during the period of January 22, through April 9, All work was performed in accordance with the International Standards for the Professional Practice of Internal Auditing of The Institute of Internal Auditors. Audit work consisted of interviewing department personnel and reviewing project documents sufficient to analyze the risks and controls for managing the project. Specifically, this included: reviewing the TxPROS Project Management Plans, contract documents, task schedules, deliverable schedules, deliverable approval documents, sample deliverables and invoices, product testing plans and testing criteria, testing results, status reports, board meeting notes, team meeting notes, s, and the vendor s unsolicited proposal for additional IT development and hosting options. interviewing personnel from the Motor Carrier Division such as the TxPROS Project Management Team members (PMT), systems analysts, map coordinators, permit officers, the contractor for the Central Permit System (CPS), and district permit coordinators; personnel from the Technology Services Division (TSD) such as database administrators, security specialists, infrastructure specialists, network services specialists, system analysts and various section directors; contract specialists in the General Services Division, attorneys in the Office of General Counsel, and an Accounting Specialist in the Finance Division. Background The purpose of the TxPROS project is to develop an automated mapping and routing solution for the transport of oversize overweight (OS/OW) loads on state maintained roadways. The envisioned solution would integrate a software component into MCD s existing CPS and provide customers a web-based, full self-service system that is compatible with TxDOT s base GIS and bridge data and will include parameters such as structure height, lane width, load ratings, oneway attributes, access roads, turn restrictions, and at-grade railroad crossings. The system would allow carriers to self-issue permits within pre-defined parameters, thus reducing the time required to issue most standard OS/OW permits. Complex and superload permits would be serviced by MCD permit officers using the system. Report of 11 July 15, 2009

2 The vendor, ProMiles Software Development Corporation (PSDC), was selected by a team of TxDOT subject matter experts and a purchase order was executed in August The project development process is divided into six phases. Phase 1, Planning and Design solidify the project approach, identify all resources, roles, responsibilities, validate detailed requirements and brainstorm specifics of the application design. Phase 2, Pilot Area Development/Prototype Testing develop the pilot area integrated map, data conversion, and application prototypes to ensure proof-of-concept prior to statewide development. Phase 3, Statewide Application expand the capabilities of the map and application modules, developed for the pilot area in phase 2, to the entire state. Phase 4, Performance/Acceptance Testing ensure the system s ability to meet the business needs of TxDOT and its customers. Phase 5, Training, Marketing, and Documentation conduct training and marketing, knowledge transfer, and develop all user documentation. Phase 6, Implementation and Acceptance finalize and execute those activities required to deploy the TxPROS solution to a statewide production environment and formally close the project. The project is currently in Phase 3. Phases 1 and 2 were completed in May and December 2008, respectively. Opinion The TxPROS project has experienced a few delays as well as significant data problems. At the close of audit fieldwork, April 15, 2009, the project was approximately three months behind schedule (see updated status in Closing Comments, last page of the report). It has successfully progressed through prototype testing but verification of product quality is premature at this point. The audit identified some best practices as well as recommendations for improvement, both of which are discussed in the following sections. Best Practices The MCD Project Management Team (PMT) should be commended for their efforts in keeping the project on target despite some tough obstacles. Some of the best practices exhibited by the PMT include: A detailed set of Project Management Plans developed and maintained in accordance with the Department of Information Resources Project Delivery Framework. These plans were updated at the end of each phase. An extensive repository of project documents developed by the PMT, maintained on MCD s intranet site allowing anyone within TxDOT access to project information at any Report of 11 July 15, 2009

3 time. (The same information is uploaded on the vendor s website allowing anyone within PSDC access to project information at any time.) Kickoff meetings were held at the beginning of each phase to validate goals and expectations. Conversely, close out sessions were held at the end of each phase to discuss lessons learned. Close stakeholder involvement from districts and other divisions, most importantly, the Technology Services Division. Outreach to industry representatives including input in initial design/concept. Timely attention to deliverables; weekly meetings with the vendor and other relevant stakeholders. Summary of Findings No. 1: Some of the data needed to create an optimized route for OS/OW permits is not available from TxDOT sources. No. 2: In the initial phases of the project, the vendor deviated from the contract in terms of staffing, the development standards, and the development techniques used. No. 3: The integration of TxPROS and the Motor Carrier Division s Central Permitting System (CPS) has been delayed. Detailed Findings and Recommendations No. 1: Some of the data needed to create an optimized route for OS/OW permits is not electronically available from TxDOT sources. Examples include accurate data for vertical clearances, lane width, number of lanes, shoulders, curbs, and grade data. Also, data for a number of frontage roads, turnarounds, crossovers, and connector roads are missing as well as distress data for ramps and connectors. The functionality requirements for the GIS mapping capabilities established in the Request for Proposal (RFP) under Attachment B, item 1.1, state that the system must provide a complete GIS description of the Texas state-maintained highway system including full directional details for all exit ramps, frontage roads, and other access roads as well as other attributes required for OS/OW routing such as reference/mile markers, structure height (detour ability, lane width, load ratings, turn restrictions, at-grade railroad crossings, etc.), utilizing TxDOT and other data sources. As the TxPROS project advanced to Phase 2, pilot area testing, it became apparent that accurate, up-to-date data from the Texas Reference Marker System (TRM), Geographic Information System (GIS) and the Bridge Inventory, Inspection and Appraisal System (BRINSAP) was not available. For some of the data, such as attributes for ramps, connectors, or turnarounds, there is no federal reporting or department requirement to actually maintain the data, thus no one has been assigned the responsibility. For other data, such as grade data or bridge height, the Report of 11 July 15, 2009

4 responsibility has been assigned but the database(s) have not been well maintained over the years. It should be noted that the Bridge Division (BRG) is currently involved in an initiative to develop a new system for obtaining up-to-date bridge location/height data using the most recent advances in laser technology. Meetings between the TxPROS team and the team responsible for storing the bridge location/height data via the new BRG bridge data collection project have begun to ensure effective coordination between the two projects. Completion of the new bridge database is scheduled for the August 2009 time frame. Another complicating factor in conflating the data for use by the TxPROS database is that the TxDOT data is gathered using disparate linear measuring systems or methods for physically locating data. For instance, distance from origin is used to locate some data vs. longitude/latitude for other data. Effect: The lack of usable TxDOT data delayed the development of TxPROS in phase 2 as MCD and the vendor struggled to determine the extent of the problems and/or develop ways to overcome them. On August 21, 2008, MCD informed the project Board of the data problems. The decision was made to continue developing TxPROS despite the data issues. Although TxPROS will not produce an optimized route, it will still produce an automated route. For some information, such as bridge location and height, it was determined that the best source for the data was the actual hard copy maps currently used by the permit officers. Using those maps, the vendor has manually entered the bridge data into the TxPROS database. No changes to the contract terms, contract deliverables, or functional requirements were made as a result of the data problems. The vendor was paid full price for the deliverables in phase 2. According to the MCD Project Manager, the vendor built the system with the ability to house and apply the missing data when/if it becomes available in an automated format from TxDOT. Consequently, no significant system modifications will be necessary when the data becomes available. Recommendation: MCD should coordinate with members of the Administration and relevant offices of primary responsibility to determine the cost/benefit of maintaining the data needed for the system to produce an optimized route. The Administration should then assign responsibility and resources as needed. MCD needs to ensure the maintenance agreement with the vendor, still to be negotiated, addresses the fact that the vendor has been paid full price for the deliverables to date and additional payments or significant modifications should not be necessary for housing or applying the missing data if/when it becomes available. Management Response and Action Plan: MCD management concurs with the assertion that some of the data needed to create an optimized route for OS/OW permits is not electronically available from TxDOT sources. MCD management and the TxPROS project management team (PMT) identified data availability and migration activities as a risk during the early planning stages of the project. This risk identification has been communicated to the TxPROS project board Report of 11 July 15, 2009

5 and TxDOT Administration on multiple occasions and was also documented in the project s Risk Management Plan; Much data to migrate, several types of data and unclear descriptions of best source. The largest risk area is data for maps and their associated limitations (i.e. vertical clearances by lanes). MCD informed board members about the data issues during the August 2008 Project Board meeting. The board subsequently directed the PMT to continue developing TxPROS despite the data issues, as the risks of continuing the project with missing data outweighed the inherent risks associated with the current system of manually routing oversize/overweight loads on Texas roadways using paper maps. The issues associated with the lack of usable data were also mitigated by approval from TxDOT s Administration for the Transportation Planning and Programming Division (TPP) to hire additional staff to support TPP s data collection efforts required to support TxPROS data needs. (This approval for TPP to hire additional staff was given during the TxDOT hiring chill that was in effect at the time for most positions within the department.) Further, MCD management views BRG s current bridge location/height data collection effort as a positive step towards being able to incorporate bridge data into the TxPROS system. MCD management will request a meeting with TxDOT Administration and the relevant offices of primary responsibility (OPR) to discuss the cost/benefit of maintaining the data needed for the system to produce an optimized route in the September/October 2009 time frame. Additionally, the TxPROS map data is currently being reviewed by the districts to ensure accuracy and, once verified, this data will be provided to the OPRs for their use in updating existing TxDOT databases. Renewed support from the administration and OPR divisions to secure and maintain the necessary data would be welcome. MCD management agrees that negotiations with the vendor to establish the maintenance agreement for the system need to take into consideration the fact that the data fields for the missing data have already been defined and incorporated into the database. The capabilities to apply the data have been developed despite the current missing/inaccurate status. When the missing data is corrected and made available, the updates will be incorporated during normal quarterly updates. No. 2: In the initial phases of the project, the vendor deviated from the contract in terms of staffing, the development standards used, and development techniques used. Staffing: The vendor s proposal, as presented in March 2007 to the TxDOT vendor evaluation team, included staffing the TxPROS project with a technical writer, a GIS data manager, and a documentation manager, among other key positions. The vendor was to provide the names and resumes of these individuals for PMT approval once the project had been initiated. That did not happen. The GIS data manager and technical writer positions were not filled during phase 1, (Planning and Design), nor phase 2, (Pilot Testing), when these positions would have played a key role. Instead, the vendor s Project Manager filled the role of GIS data manager and technical Report of 11 July 15, 2009

6 writer in addition to his project management and programming responsibilities. The document manager position was filled by the company s administrative assistant who appears to have limited experience in this capacity and does not function in this role as part of her regular job duties. The vendor did not communicate their staffing decisions to the MCD Project Management Team until the end of phase 1. Development Standards: The project s technical standards are contained in Attachment F of the RFP, which states that the database is to be developed in Oracle 10g or higher. These standards had been clearly communicated to the vendor in writing and emphasized again verbally in multiple meetings. There is no doubt the vendor understood and agreed to this requirement; however, they built the database in Microsoft SQL. Apparently the vendor did not have the appropriate licensing for Oracle nor experience in the Oracle environment. The vendor made the decision, without the knowledge or approval of the PMT, to build the initial database in SQL and convert it to Oracle once the license and skill sets were acquired. Development Techniques: The vendor s original response to the RFP, pg. 99 states that the project will be developed using a waterfall approach. A waterfall approach benefits from a detail set of planning and design documents in the initial phase. The TxPROS deliverable schedule was agreed upon based on this approach, which required the detail documents in phase 1. However, the deliverables submitted by the vendor in phase 1 did not contain the level of detail expected. The documents took longer to review and were revised multiple times. The PMT ultimately accepted the deliverables as high level with the understanding they would be updated in later phases. The vendor and PMT subsequently agreed to use a spiral development approach. In a spiral approach, the design is an iterative process the design is completed/produced as the product is built. This method can be more expedient but also more risky if significant problems, unforeseen due to a lack of detail planning/design, arise later in the project. No changes to the contract terms or deliverables were made as a result of the change in approach. Effect: These deviations impacted the project s progress. The quality of the deliverables in phases 1 and 2 suffered due to the lack of dedicated qualified staff. The PMT continued to emphasize the need for a dedicated technical writer and the vendor finally hired one, whom the PMT approved of, in February The database was eventually converted from SQL to Oracle in December 2008 interrupting progression of the project by a month. MCD still does not have a detail design document, which could be risky if the vendor left the project mid-stream or experienced a change in personnel. Recommendation: MCD needs to ensure the vendor is complying with all the requirements of the contract, including the functional and technical requirements. As of May of 2009, the PMT had not yet verified compliance with any of the remaining technical requirements established in the RFP. Since the vendor has already deviated from these standards once, the PMT should be addressing these issues with each phase of development. If re-work is required and causes the project s deadline to shift, MCD should consider assessing liquidated damages as outlined on page 1 of the Best and Final Offer. Report of 11 July 15, 2009

7 MCD should insist that an updated design document be provided as soon as possible, and be updated with each phase. We realize that any IT development project will incur changes to the design as it progresses, and that there may still be many design changes made before the product is accepted. It is still in the department s best interest to ensure that the system documentation is well maintained throughout development. Management Response and Action Plan: MCD management agrees that the vendor s initial staffing of the project was not in strict accordance with their RFO response, and that the vendor must comply with all requirements of the contract. While unfortunate, the vendor s decision to develop the initial database in Microsoft SQL was an attempt to keep the project moving while the vendor obtained the appropriate licenses and development skill sets for an Oracle development environment. This situation has been rectified and the initial SQL database was converted to Oracle at the vendor s expense. The PMT will review all contract deliverables to evaluate and verify the vendor s compliance with the technical requirements established in the RFP. This review will be completed in the November/December 2009 time frame. As progress continues on subsequent phases of the project the PMT will continue to evaluate and validate compliance with the requirements of the RFP. MCD management and the TxPROS Project Management Team are confident that the Spiral/RAD/prototyping development approach is the appropriate system development model for TxPROS. According to Computer World the waterfall model works very well when we are automating the activities of clerks and accountants. It doesn't work nearly as well, if at all, when building systems for knowledge workers -- people at help desks, experts trying to solve problems, or executives trying to lead their company into the Fortune 100." 1 According to the same article, a Spiral/RAD/prototyping development approach helps demonstrate a proof of concept early in the cycle, 2 and calls for considerable feed back and iterative consultation. 3 As the TxPROS project matured, it became obvious to the PMT that a Spiral/RAD/prototyping development model was more appropriate for successfully developing and implementing an automated routing system on the scale of TxPROS, which is a different and more sophisticated system than those previously implemented by other states. The TxPROS Project Sponsor, Project Manager, and other MCD staff members have a history of successfully developing and implementing projects using Spiral/RAD/prototyping development models going back at least to the late 1990s. These projects include MCD s Motor Carrier Credentialing System (MCCS), Single State Registration System (SSRS), Complaint Management System (CMS), and later enhancements to rescue the original Central Permitting System (CPS). The deep backgrounds and skill sets of MCD staff in managing projects using this development approach helps to mitigate risks inherent in the Spiral/RAD/prototyping development model. 1 Kay, Russell. 14 May Quick Study: System Development Life Cycle. Computer World. 26 July 2009 < 2 Ibid. 3 Ibid. Report of 11 July 15, 2009

8 MCD management does not agree with the need to immediately develop a comprehensive design document for the complete system design given the development approach being employed. An updated design document will be completed when the project design is base-lined in conjunction with Milestone 10. From that point forward, all changes to the design will be addressed via an official change management process. To mitigate potential project risks between now and the development of the updated comprehensive design document, project design decisions are faithfully documented and tracked via project working documentation (i.e., development notes, meeting minutes, etc.). As a side note, during the Phase I closeout/lessons learned session, the PMT identified a potential impediment to the success of future TxDOT projects in current TxDOT policy. This policy requires contractually specifying due dates/milestone dates for deliverablesbased contracts prior to conducting Joint Application Design (JAD) sessions and fully identifying system requirements. While this policy may be appropriate for some types of projects, the PMT believes that it is premature to baseline project deadlines prior to completing JAD sessions which fully identify project requirements. No. 3: The integration of TxPROS and the Motor Carrier Division s Central Permitting System (CPS) has been delayed. According to the contract deliverable schedule, testing of the integration of TxPROS and CPS was to occur in phase 2. Phase 2 was completed in December 2008, however the two systems were not actually integrated at that time. The reason the systems have not yet been integrated are twofold: First, CPS requires modifications before it can interface with TxPROS, but the extent of those modifications were not known at the time. Second, there are still decisions to be made on the environment for hosting TxPROS, which could affect the ability of the two systems databases to coexist. The TxPROS contract includes the option to negotiate a vendor hosted solution. MCD favors a vendor hosted solution based on concerns with the Team for Texas (TFT) environment. In July 2008, ProMiles submitted an unsolicited proposal to develop a new CPS, at no cost to the state, and host both the TxPROS and new CPS under a software as service agreement. The vendor claims they can develop a new, modernized, fully integrated CPS within two to four months. The development of the system, and the hosting of both CPS and TxPROS, would be paid for with a user fee, charged to the carriers. Hosting of the two systems outside of the DIR/Team for Texas environment requires obtaining approval from DIR, which was requested and received on December 4, Effect: While the vendor s proposal for a new CPS solves some of MCD s hosting concerns, there are many issues with the proposal that need to be addressed before moving forward. These include, but are not limited to: Defining an appropriate service level agreement (SLA) to control product quality, availability, maintenance, and future enhancements; Report of 11 July 15, 2009

9 Assessing the viability of the vendor s proposed service fee since the department is bound by administrative rule on the amount to charge for a permit; Timing of deposits for the transfer of collected funds to the department, Controlling future cost increases; and Defining ownership rights for the data as well as the system, and determining conditions for transferring the system(s) back to TxDOT if the vendor does not meet the SLAs. MCD had begun efforts to move forward with the CPS modernization proposal by obtaining the DIR waiver and assigning responsibility for developing the new system s business rules. However, there is doubt that the issues identified above could be resolved, and a contract amendment negotiated, without causing serious delay to the TxPROS project. Furthermore, there is doubt that a new system could be developed and implemented in the timeframes quoted in the proposal a concern that is exacerbated by the fact that the vendor did not sufficiently staff the TxPROS project. If the development of CPS takes longer than expected, and the vendor is not recovering any costs in the meantime, the whole project could be in jeopardy. Some of the information presented in the vendor s unsolicited proposal is misleading. For example, the vendor claims that the existing CPS is written in old, brittle code and does not communicate w/ the Motor Carrier Credentialing System (MCCS), the Complaint Management System (CMS), and other TxDOT systems (reference pages 1 and 3 of the proposal, respectively). However, CPS was updated with new code and over 80 enhancements in the fall of It communicates to CMS and MCCS on a limited basis. The vendor s proposal touts significant cost avoidance opportunities, on pg 9, with the elimination of the substantial efforts and risks involved in updating the existing CPS system to interface with TxPROS. However, there had been no cost/benefit analyses made; the cost for modifying CPS had not been determined. It should be noted that the vendor was paid full price for the deliverables in phase 2, which included testing the ability of the existing CPS and TxPROS to hand off information. While the two systems had not yet been integrated, the vendor built a simulation sufficient to conclude that the two systems could effectively communicate. Recommendation: MCD should enforce the original terms of the TxPROS contract requiring the vendor to integrate TxPROS with the existing CPS, using the least amount of modifications to CPS as possible. Once TxPROS is fully functional, and MCD can verify that the vendor has delivered a quality product, then additional options for hosting the systems can be addressed. This may require extending the current contract for a period of time necessary to sufficiently research those options. Management Response and Action Plan: MCD management agrees that the terms of the TxPROS contract should be enforced. Interfacing CPS and TxPROS has proven to be more problematic than originally envisioned, and extensive changes to the existing CPS by MCD technical support staff Report of 11 July 15, 2009

10 and contract personnel will be required to support the interface requirement. Initial estimates place the cost at up to $75,000 for a contract developer and a 5-month development timeframe. Additionally, utilizing these resources for CPS diverts resources from other high-priority projects, such as those needed to split apart and transition a portion of existing MCD systems to the newly-created Department of Motor Vehicles (DMV). CPS was essentially a rescued product after the original vendor s system did not meet many project functionality requirements. MCD staff worked with a contracted project developer to fix the bugs that were hampering basic functionality, and since 2000 MCD has implemented numerous areas of expanded functionality. While CPS currently performs better than it did in 2000, the numerous bandages and workarounds to the core system are problematic and convoluted; new core functionality issues are discovered with resulting emergency bug fixes occurring frequently; and the ability to enhance CPS to meet future needs (such as new legislative requirements, permit types, etc.) without breaking existing functionality is unknown. Unfortunately, the full limitations of CPS were not known at the time the TxPROS project was originally conceived, and only became more apparent as the project matured. The PMT began exploring the options outlined in the vendor s unsolicited proposal to replace CPS based on direction provided by the Project Board in August Part of the reasoning behind the board s decision was potential opportunities to: obtain a fully functioning CPS, capable of expansion to meet future needs, without the potential risks of further workarounds and unknown, lurking bugs in the existing CPS; fully integrate rather than interface with TxPROS; and the opportunity to have a strictly enforceable service level agreement managed by TxDOT. The alternative is for TxDOT to sink additional time and money into a decade-old program with questionable future functionality in order to interface with TxPROS, with unknown service levels under the DIR/TFT contract, and then to eventually replace CPS after TxPROS is complete. The TxPROS vendor s unsolicited CPS modernization proposal poses some challenges, as identified previously to the Project Board and in this Audit Report. Despite the early issues faced and later addressed by the vendor and the PMT, MCD staff and the Project Board believed the proposal should be thoroughly analyzed due to the vendor s longterm experience providing routing systems for nationwide carriers, and based on their success developing and implementing a fully automated permitting system within a short time-frame for the Brownsville Navigation District. To effectively support the CPS interface requirement, MCD staff members have thoroughly documented CPS business rules, have developed high level business requirements needed for the CPS/TxPROS interface, and are currently developing detailed business rule changes required for the CPS/TxPROS interface. The Report of 11 July 15, 2009

11 opportunities/risks and costs/benefits associated with interfacing the existing CPS with TxPROS, versus procuring a new CPS that is fully integrated with TxPROS, will be thoroughly vetted by the PMT in collaboration with the appropriate divisions (TSD, GSD, OGC), the Project Sponsor, the Project Board, the TxPROS Executive Sponsor/TxDOT Administration, TxDOT s Information Resource Council (IRC), and DIR s Quality Assurance Team, as appropriate. Other Delays in Project Progress It should be noted that a couple of interruptions to the project progress were no fault of the vendor. Hurricane Ike hit the gulf coast in September 2008, causing damage to the vendor s facilities. The vendor responded quickly and was back on the project within 11 calendar days, a period well within the two week timeframe proposed in their business continuity plan. Also, security issues with the TxDOT firewall occurred during prototype testing resulting in a two week delay. Closing Comments The results of the audit were discussed in an exit conference with the MCD Division Director and PMT on April 9, 2009, and with the Assistant Executive Director for Support Operations in a separate exit conference on April 15, We appreciate the cooperation received from all staff contacted during the audit. Project Status Update as of July 15, 2009: A change to the contract deliverables and payment schedule was executed June 30, 2009, to allow for additional detailed testing of the prototype using statewide data, allowing extensive Permit Specialist interaction before fine-tuning the system for production status. The change did not add money to the project but did extend the contract date by an additional 3 months. The change should also allow time for the modification of CPS to integrate with TxPROS. As a result, the project is 6 months behind the original schedule. Report of 11 July 15, 2009