TAFE SA Starting from scratch

Transcription

1 TAFE SA Starting from scratch An architecture and lessons learnt Hamish van den Ijssel Melissa Hazelwood APSUG - September 2010

2 Overview TAFE SA project overview (5 minutes) Architecture (20 minutes) Lessons Learnt (10 minutes) Questions (10 minutes)

3 TAFE SA 85,000 students 25,000 FTE 53 campuses, APY Lands, overseas partners VET and tertiary students Relationships - TAFE SA, DFEEST and South Australian Government, Federal Government

4 ICT Overview Decentralised -> Centralised Outsourced components No SungardHE exposure Very limited Oracle and Unix exposure Existing application suite circa mid 1990 s

5 Project Overview December 2009 Test hardware delivered February 2010 First logon to Banner test environment January Disaster Recovery Testing February and March End to End testing April CORE release - Registration INB, SSB, ODS Cognos, evisions, eprint Interfaces and reporting June 2011 Admissions Interfaces and reporting August 2011 March 2012 CORE + Workflow, BDMS, Luminis Reporting and interfaces IAM July 2012 onwards CORE++ Post implementation projects

6 Architecture Primary Site State government data centre Secondary Site Tea Tree Gully Campus data centre Primary Site Secondary Site

7 Database Servers HP BL860 s Oracle 11GR2 RAC HP-UX HP EVA8000 and EVA 8100 Primary Site Primary Secondary Site Site EVA8000 EVA8000

8 HP BL460 s VMware vsphere cluster RHEL 5.0, Windows 2003 Oracle Application Server 11G Application Servers Primary Site Primary Secondary Site Site EVA8000 EVA8000

9 Virtual Hosts External facing SSB + Luminis* Internal facing INB, eprint, BDMS, Cognos, WF, evisions, Grid Control Primary Site Primary Secondary Site Site External Internal External DR Internal DR & Dev/Test SSB1 SSB2 SSB3 INB1 INB2 BDMS SSB1 SSB2 INB1 INB2 BDMS SSB4 SSB5 SSB6 GRID Control eprint / evisions SSB3 SSB4 Cognos GRID Control eprint / evisions Cognos WF WF evisions EVA8000 EVA8000

10 Database Utilities ADDM Automatic Diagnostic Monitor AWR Automatic Workload Repository Oracle Grid Control

11 Utilities Backup IBM Tivoli SAN replication software VMware Site Recovery Manager Oracle Load Testing Suite

12 Goals Load Testing Validate system architecture Identify bottlenecks Performance tuning Network, Servers and software Issues 53 campuses; Performance at remote campuses Complexity of the environment

13 Load Testing Oracle Application Testing Suite Integration with Oracle Grid Control Oracle RDBMS Oracle IAS MIB files Servers Operating Systems Database Servers Application Servers Routers F5 s PC s Firewalls

14 Load Testing Phase 1 June 2010 Development hardware Validate system architecture Phase 2 October 2010 Identify issues with regional networks Phase 3- November 2010 Primary site hardware with 80% configured Performing tuning Phase 4 February 2011 Primary site hardware with 100% configured Performance tuning Phase 5 August 2011 Secondary site hardware with Banner 8.X Performance tuning

15 Load Testing Example 1600 virtual users 3 separate test scripts Apply for admission Check registration Course Search All test scripts utilise variables via datafiles, i.e. Username, password, course

16 All transactions versus time Number of transactions over time

17 Errors over time Errors Excessive time for transaction to complete Insufficient time between each step of the test script Web server busy; unable to serve page Excessive time for post / send Database busy / slow Network traffic / issues OATS Test step expected and actual result different

18 Performance over time Total time for transaction to process per millisecond over the life of the test

19 Active users over time Agents session counters

20 Statistics Name Min Max Avg Active Virtual Users Virtual Users with Errors Transactions Per Second Pages Per Second Hits Per Second Kilobytes Per Second Totals Transactions 9021 Transactions with Errors 4256 Pages Hits Kilobytes Name Min Max Avg ApplyForAdminLTST_SSB CheckRegoLTST_SSB CourseSearchLTST_SSB

21 Network

22 Network devices F5 Fibre IPS Database SAN Servers Data guard SAN replication Application Servers Virtual images Hosts Site Recovery Manager Disaster Recovery

23 Interfaces CORE 15 Applications 8 Localisations / 7 Custom 5 External / 10 Internal Primarily Process Rules Engine CORE+ 3 Applications 3 (replacement) SPML interfaces 2 likely Process Rule Engine

24

25 Training 59 days / 152 staffing days of training Sungard Banner General and Security ( 3 days 4 staff) CTOOL (2 days 3 staff) Intro to Banner Administration ( 3 days 2 staff) Oracle DBA Administration (3 days 2 staff) Oracle, PL/SQL, and Database Objects (3 days 4 staff) Student Module Technical (3 days 4 staff) Advance DBA Administration (3 days 2 staff) Oracle backup and recovery (3 days 2 staff) Student Data Mapping for Data Migration (GP-Synchronisation (3 days 1 staff) Oracle Application Server (3 days 4 staff) Super Table (2 days 2 staff) Process Rules Engine (2 days 3 staff)

26 Training Sungard continued Value Based Security (2 days 2 staff) Web Tailor (2 days 3 staff) Luminis I & II (4 days 2 staff) RAC / ASM (3 days 2 staff) ODS (3 days 3 staff) OATS (1 day 3 staff) Luminis I & II (4 days 2 staff) Reporting evisions (1/2 days 2 staff) eprint (1/2 days 2 staff) COGNOS (3 days 3 staff) Regulatory Reporting (3 days 2 staff) Future Grails Handover

27 Training Information overload Project schedule training versus workload Documentation Room bookings Functional and training

28 Lessons What went wrong and what went well Licensing Decentralised and outsource services Security Policies Upgrades Policies and procedures Key Relationships Human Resources Load Testing Vendor Delivery

29 Licensing Include with project tender Know vendors end of sales quarter Holistically evaluate pricing models and support options TCO

30 Decentralised and outsourced Decentralised business and ICT 4 ICT units 53 campuses Outsourced ICT System Administration Networking Architecture decisions

31 Resourcing Recruitment Public service merit selection Project staff Sourced from the business Sourced from industry Sungard consultants Contractors Project as a separate entity from ICT

32 Security Policies South Australian Government ISMF Out of date policies Incompatible with business practices Outsourced security advice

33 Upgrades Desktops Windows / Web browsers Servers Banner Oracle Functional testing

34 Policies and procedures Policies and procedures When, where and how? Documentation When, where, how Release / patch / change management

35 Original SAR to GO-LIVE System Architecture design in October 2009 Technology incompatibilities New technologies New releases of Sungard products Changing WOG policies You don t know what you don t know System administrator design to holistic design

36 Relationships Functional Training ICT Sungard Vendors

37 Technical Team staff (7.0 FTE) Internal (2.0) Two analyst programmers External (3.0) One Analyst Programmer One DBA One DBA / Analyst Programmer Sungard - long term (2.0) Senior DBA Senior Technical Consultant Sungard short term Short term fly-in Non project ICT staff Infrastructure Vendors System Administration Networking Public service merit selection Project as a separate entity from ICT Staffing

38 Load Testing Objective and informative data Easy to use product Excellent support from the vendor Excellent integration with Sungard products Extensive interest from all areas of ICT

39 .. Questions