TRADE SUPPORT NETWORK RELEASE 2 / USER ACCEPTANCE TEST (UAT)

Transcription

1 TRADE SUPPORT NETWORK RELEASE 2 / USER ACCEPTANCE TEST (UAT) Presenters: Panel: John Hill, CBP Jim Stevens, ecp Kelly Ausherman, Boyds Collection Terri Ellis, Hewlett Packard Bill Inch, CBP Mike Boshes, ecp Michael Hall, ecp Marc Newkirk, ecp Date: September 16, 2003 Customs and Border Protection Modernization New World, New Tools

2 Agenda / Overview Pilot / UAT Overview John Hill, CBP Pilot / UAT Activities Jim Stevens, ecp Training Overall User Satisfaction UAT Test Case User Satisfaction Trade Feedback Kelly Ausherman, Boyds Collection Terri Ellis, Hewlett Packard Account Expansion Bill Inch, CBP Goals 1

3 RELEASE 2 Pilot/UAT Overview John Hill, CBP Customs and Border Protection Modernization New World, New Tools 2

4 Pilot / UAT Overview Release 2 Pilot period is a 90-day test of the system prior to formal acceptance by the government Period of time allows for proof of system readiness through successful deployment, operations support, and user training and support; Included a future workload simulation with 1380 concurrent users. UAT is an 8-week system test following training that utilizes a journal of test case scenarios designed to calculate users ability to use ACE Portal and to capture their satisfaction and comments 66 Trade Account Users representing 41 Trade Accounts attended training UAT Journals containing 25 Test Case Scenarios reflecting the various capabilities of current Release of ACE Portal distributed to each participant at end of 3 Trade Training sessions; an Overall User Satisfaction Survey was provided 70% of completed UAT Journals received from participants 3

5 Pilot/UAT Activities Jim Stevens, ecp Customs and Border Protection Modernization New World, New Tools 4

6 Training Three training sessions for ACE Trade Account Owners: July 08-10, Accounts Overall Course Effectiveness 3.43 July 15-17, Accounts Overall Course Effectiveness 3.56 July 22-24, Accounts Overall Course Effectiveness 3.77 Role-based training scenarios focused on CBP Account Managers, CBP Client Representatives, and Trade Account Owners. Training scenarios for each position with CBP Subject Matter Experts CBP Client Representatives and Account Managers present during the third day of the Trade Account Owner training to provide for interactive set-up of the account information Training was well received overall, earning an aggregate 3.59 on a 5 point scale, Good to Very Good, for overall course effectiveness 5

7 User Acceptance Test Journal UAT Test Journal consists of Test Cases Developed in conjunction with the Training Scenarios Based on input and feedback from CBP Subject Matter Experts Tests the various capabilities in Release 2 of the ACE Secure Data Portal Account Management, Significant Activity Log, Reports, etc. Tests ACE in real world environment using customer s data. Tests Case questions provide insight into the User s satisfaction with the various components of ACE Ease of Use, Ease of Learning, Number of Steps, Amount of Time, Performance, Compared to Current Method, etc. User Survey Questionnaire provide overall satisfaction 6

8 UAT Test Case Example Test Case Manage Importer of Record (IR) Information View IR Details Edit IR Details Before you start the test case, please answer the following questions: Is this a task you perform (or should perform) as part of your job? How frequently do you perform this task? How important is this task? Were you able to complete this test case? How satisfied are you with the Online Help, User Guide, Help Desk Support, Training for this task? Based on your experience doing this task: A typical user would be able to successfully use the ACE Portal for this task after? How satisfied are you with the ACE Portal for this task with the: ease of learning to use; ease of using; usefulness; number of steps; amount of time; application performance Compared to the way you do this task now, the ACE Portal is? Overall, how satisfied are you with the ACE Portal for this task? 7

9 Overall User Satisfaction User Satisfaction ACE Trade UAT User Satisfaction Before UAT After UAT 4.15 User Satisfaction Survey provided to UAT Participants at the conclusion of training (prior to start of UAT) User Satisfaction Survey was also part of the UAT Test Journal (at the conclusion of the Test Cases) Results are positive 1 Trade 7/10 Trade 7/17 Trade 7/24 Trade UAT as of 8/25 Trade User Sessions Scale for Responses: 5 Very Satisfied 4 Satisfied 3 Neutral 2 Dissatisfied 1 Very Dissatisfied 8

10 UAT Test Case User Satisfaction 47 of 65 UAT Journals Returned From Trade Participants 72% 25 total test cases developed in collaboration with trade representatives Results from 12 Test Cases scored between 4 and 4.5 Results from 13 Test Cases scored between 3.5 and 4 Test Case User Satisfaction: Composite Mean of all Test Cases = 4.00 High = 4.43; Low = of the 41 Trade companies represented provided feedback via the UAT Test Journal or the ACE Portal Feedback 93% Based on results from UAT Test Journals and Portal Feedback, enhancements to the ACE Secure Data Portal will occur in future releases 9

11 UAT Evaluation Kelly Ausherman, Boyds Collection Customs and Border Protection Modernization New World, New Tools 10

12 UAT Evaluation Overall Evaluation Ease of Use Easy to Learn Training Preparedness Functionality Likes & Dislikes Limited Training for New Users Limited Number of Roles Location of the PDF icon in HTS Location of Error Messages Confirmation of Deactivation of User Technical Capability Stability 11

13 Value and Benefits of ACE Terri Ellis, Hewlett Packard Customs and Border Protection Modernization New World, New Tools 12

14 Value & Benefits of ACE Visibility of Data Individual IR or Aggregate Entry Info Compliance Info Bond Info Immediate access to info Fewer FOIA Requests Reference Materials HTS Rulings Significant Activity Log Action Plan 13

15 Account Expansion Goals Bill Inch, CBP Customs and Border Protection Modernization New World, New Tools 14

16 Account Expansion Goals Reach 1,100 total ACE accounts by Release 4 Phase I: Release new importer accounts Phase II: Release 3 Brokers for initial 41 importer accounts Phase III: Release 3 and new trade accounts each month Release 3 Release 4 Importers and brokers managed by national and port account managers Additional importers, brokers and carrier accounts Selected land border carrier accounts Release 2 Release 3 Release /month Eval. 1,100 Eval. Add Importers...Add Brokers. Add Carriers for Initial 41 Accounts Phase I Phase II Phase III 15

17 Visit the Modernization & ACE site on or write us at: 16