PRIME PROVIDER EVALUATION

Transcription

1 PRIME PROVIDER EVALUATION Thuy Luong, P.E. November 3, 2017

2 Table of Contents Purpose of Prime Provider Evaluations Scoring System Converting ESA score to Past Performance Score ESA Score for Providers with No Past Performance Scores Upcoming Changes to Prime Provider Evaluation Frequently Asked Questions 2

3 Purpose of Prime Provider Evaluations Why do provider evaluations? Have to required by rule (43 TAC 9.41 (d)) Want to used as a multipurpose tool How are they used? Management tool - use it to communicate our level of satisfaction Selection tool - past scores are used to compute the ESA, which is used during selection 3

4 Evaluations - All About Communication TxDOT communicates: At the beginning to identify what is expected At logical points to identify and correct problems At the end - the final evaluation TxDOT communicates on two levels: 1 - Is the job getting done? 2 - How well is it being done? 4

5 Evaluations - A Management Tool The Prime Provider Evaluation Form: Provides performance feedback Improves contract management Improves contract product It contains a list of criteria and outcome standards Evaluates the consultant PM s skills and abilities as a project manager Does not evaluate consultant PM s skills and abilities as a professional engineer, surveyor, or architect 5

6 Evaluations - A Selection Tool TxDOT PM enters the evaluation into CCIS Evaluation Score Average (ESA) is computed in CCIS The ESA includes a component for: the consultant project manager the firm The ESA is used during the selection process at either the SOQ evaluation stage, or the proposal/interview evaluation stage Typically, past performance is weighted at 5-10% of the SOQ or interview score or proposal/interview scores 6

7 Scoring System Project manager starts with 100 points Firm starts with 50 points For satisfactory performance on a criterion, the base score is given For less than satisfactory performance on a criterion, points are deducted (e.g. -3, -6) 7

8 Scoring System (continued) The PM/firm evaluations are divided into three groups: The second group consists of the evaluations entered into the CCIS at any time from/including five years prior to the current day to ten years prior to the current day. The first group consists of the evaluations entered into the CCIS at any time from/including the current day to five years prior to the current day. The third group consists of the evaluations entered into the CCIS at any time from/including ten years prior to the current day and earlier. 8

9 Scoring System (continued) The evaluation scores for Group 1 are averaged. (Note: This number will not exceed ) The evaluation scores for Group 2 are averaged. This number is then be averaged with The evaluation scores for Group 3 are ignored entirely. TxDOT is not interested in evaluations 10 years old and older. 9

10 Scoring System (continued) The PM score is calculated as follows. If there is an average evaluation score available for Group 1, that number is divided by to yield a PM score equal to or less than Thus, an average score of would yield a PM score of If there is no average evaluation score available for Group 1, the effective average evaluation score for Group 2 is used. As above, this number is divided by to yield the PM score. 10

11 Scoring System (continued) The firm score is calculated as follows. If there is an average evaluation score available for Group 1, that number is divided by to yield a firm score equal to or less than Thus, an average score of would yield a firm score of If there is no average evaluation score available for Group 1, the effective average evaluation score for Group 2 is used. As above, this number is divided by

12 ESA Procedure For a given PM and prime provider firm, PEPS obtains the ESA in CCIS The ESA for each provider is entered into the scoring spreadsheet during the selection process Documentation - An evaluation report, listing all of the evaluations used to compute the ESA is maintained in selection file 12

13 CCIS Software Application 13

14 PDF lists evaluations used to compute Firm and PM scores PEPS files the PDF to Selection File 14

15 Converting ESA score to Past Performance Score ESA Score Past Performance Score = < ESA < < ESA < < ESA < < ESA 1 15

16 Example of Scoring Spreadsheet Past performance is included as a criterion at the SOQ stage, Interview, or Proposal/Interview stage 16

17 Example of Scoring Spreadsheet (continued) Example of Past performance used at the Proposal/Interview stage: 17

18 ESA Score for Providers with No Past Performance Scores For Project Managers (PM) and Firms with no previous TxDOT experience or no evaluations on files, TxDOT uses an average score. This average is calculated using data from the past 5 years. Summary of the data points used: Number of Data Score ESA Points Used PM 5 YR Average Firm 5 YR Average Total (perfect score 150)

19 How is the 5-year Average Score Used? Evaluations available ESA Evaluations available ESA Evaluations available ESA Evaluations available ESA PM 8* 8* X 7.84 X 7.84 Firm 2* X * X * P.P. Score * Assuming perfect score for Past TxDOT Experience 19

20 Upcoming Changes to Prime Provider Evaluation New system will be in PS-CAMS TxDOT PM completes evaluation in PS-CAMS TxDOT PM s supervisor reviews & approves evaluation Prime provider PM reviews, comments, and dates evaluation using PS-CAMS Consultant Portal New system will be ready for use in Spring 2018 Scores for each criteria will be from 1 to 5 20

21 Upcoming Changes to Prime Provider Evaluation (continued) PM Evaluation Criteria 1.a Quality of Deliverables: Accuracy 1.b Quality of Deliverables: Completeness 2 Timeliness of submittals 3.a Contract administration - Budget 3.b Contract administration - Schedule 3.c Contract administration - Invoices 4 Responsiveness and availability of the PM 5 Resolution of issues 6 Communication and coordination 7 Management of subproviders 8 Adequate use and prompt payment of HUB/DBE firms 9 PM performance Firm Evaluation Criteria 1 Firm Responsiveness 2 Firm Invoicing 3 Firm Resource Management 21

22 Frequently Asked Questions What should I do if the TxDOT PM does not evaluate me? Prime provider evaluation is required: o Per TAC (43 TAC 9.41 (d)) o Per TxDOT Chief Engineer s Memo dated April 20, 2017: Each TxDOT project manager shall complete an end of project evaluation on each consultant contract that they were managing currently. Each TxDOT project manager shall also complete an interim evaluation anytime that a consultant is not performing to expected standards for the Department. o PEPS Division assists in this effort by providing reminder notices to District and Division project managers when contracts are due for an annual evaluations and when contracts are due for end of project evaluation. 22

23 Frequently Asked Questions (continued) Can the TxDOT PM share the evaluation results with the provider PM? That way it could be a lessons learned or opportunity to do better. Yes. Once the TxDOT PM has completed the provider evaluation, he/she must share results with the provider PM and give the provider PM an opportunity to review and comment on the results. 23

24 Frequently Asked Questions (continued) If I don t agree with the TxDOT PM on the results, can I not sign the evaluation? If you don t agree with the results, you should discuss with the TxDOT PM and if the TxDOT PM doesn t change the scores, you don t have to sign the evaluation. 24

25 Frequently Asked Questions (continued) If I don t sign the evaluation, will the scores be included in the TxDOT database? Yes. The evaluation will be input in the database and the scores will be included in the ESA calculation. 25

26 Frequently Asked Questions (continued) As a provider PM, if I disagree with the TxDOT PM s evaluation, how do I protest/dispute the scores? First, you should discuss with the TxDOT PM to get the issue resolved. If it doesn t get resolved, you may request a meeting with the TxDOT PM s supervisor to discuss. 26

27 Frequently Asked Questions (continued) Why doesn t TxDOT use the extra points (i.e. +3, +6) when calculating the ESA? Extra points are not included because the purpose of the ESA is to identify less-thansatisfactory performance. 27

28 Frequently Asked Questions (continued) When TxDOT makes a lot of changes to a project, what must consultants do to get above base scores on meeting deadlines and staying within budget? The consultant should put forth extra effort to submit quality deliverables and still meet project deadlines. 28

29 Frequently Asked Questions (continued) Shouldn t TxDOT use a small percentage of the prime provider evaluation score in the selection process since firms and provider PMs change? Past performance is weighted at 5-10% during the selection process. The PM has his/her own evaluation scores and the firm has its own scores. When the PM goes to a different firm, the PM s evaluation score goes with him/her. 29

30 Questions and Discussion 30

31 Thuy Luong TxDOT PEPS Center of Excellence (713)