Real-World Data Management. Improving the tracking and monitoring of survey data for improved analytic outcomes

Transcription

1 Real-World Data Management Improving the tracking and monitoring of survey data for improved analytic outcomes

2 Session Agenda Introductions and Overview U.S. Census Bureau StEPS & StEPS II programs o Background & History o Goals o Challenges Key Takeaways & Lessons Learned Questions & Answers 2

3 What is StEPS? StEPS is the Standard Economic Processing System StEPS is a SAS SCL-based (SAS/AF) application environment (pre-soa, object-oriented) Supports over twenty different surveys Supports seven economic indicators Contains Standard data set structures Integrated modules include: - Data Collection processes - Post-collection processes Editing, Imputation Data review and correction - Administrative features 3

4 Strategic Objectives of StEPS Share processing enhancements across the Econ Directorate. Eliminate redundant programs and code across legacy systems. Reduce resources for the development and maintenance of economic survey processing systems. Ultimately, establish a survey processing infrastructure to introduce new surveys. Continue to reduce the bureaucracy and inefficiencies for updating systems 6

5 StEPS Objectives Achieved One application (database, interface, business logic) vs. many Most surveys now use StEPS in conjunction with surveyspecific programs Over 500 staff in the StEPS Community (users and developers), working toward common goal Currently spans five of the seven Econ divisions New requirements generated for expanded functionality 7

6 Why Redesign StEPS? Reduce System Risk Nearly 20 year old technology, not substantially updated in several years, reduced tech support. Nearly 600 changes tracked in the present system since CCB inception in Present architecture has become difficult to maintain. Improved technical agility Leverage newer SAS solutions that utilize Java for client and web applications in a Service Oriented Architecture (SOA) Improve the user experience Increase functionality, performance and reliability 8

7 StEPS: Current State Client Tier Terminal Emulation Server Tier SAS Server (SAS 9.2) Interface (SAS/AF) Batch processing Analytics Reporting Database Transaction processing Business logic Compute processing (edits, imputation, estimation,..) 9

8 StEPS II: Desired State Scalable Architecture Service Oriented Architecture (SOA) facilitation via stored process/workspace servers Real-time and batch analytics Integrated SAS analytics Server-based Model Separation of Analytics Environment by Role Standard Server Configurations 10

9 Monitoring Data for Control & Accuracy Cleansing data on entry Drop-down menus Duplicate checking Monitoring quality in real-time Business rules Set data quality thresholds Monitor all data Action corrections 12

10 StEPS II: IT Value to the Census Bureau Remove application-to-application complexities Continue to increased data type flexibility J2EE object-oriented programming in conjunction with serviceoriented architecture development Facilitates future integration of other technologies via the SAS Mid-Tier (e.g. modernize database, interface, etc.) Positioned to leverage the Census Enterprise Service Bus Facilitates the use of other enterprise services for improved security (e.g. authentication/authorization) 13

11 StEPS II: Strategic Value to the Census Bureau Creates survey maintenance efficiencies: o Various organizational units are speaking the same language o Reduces risk of maintaining systems that require heroic efforts to maintain over time Increases user ease of system navigation Standardizes processing across surveys, enabling consistent analytics Expands functionality that can be proselytized Expands the number of surveys (using the system) to conduct improved survey processing. 15

12 Lessons learned 1. Leverage ability to manage and integrate data for a wide audience Web Services, Service Oriented Architecture (SOA); write once use across many user groups! Enhanced user collaboration & team-based development, from both the Business and IT sides 2. Custom code requires constant maintenance, not extensible 3. Performance matters: Faster loads, enhanced loading techniques are required 4. Data integration, across many systems, is critically linked to the ability to monitor and control data, group-wide vs. in pockets; beyond ETL only 5. Impact Analysis allows for scenario testing, pre-production; saves time and resources 16

13 In Summary A strategic SAS environment in StEPS II will: 1. Support the current and future SAS community at the U.S. Census Bureau 2. Deliver supporting technologies for: Sustaining and expanding the Reference Architecture to meet member and internal customer requirements Integrating processes from Data and Analysis to Delivery Lower Total Cost of Ownership (TCO) User-based, easy-to-use interfaces Lowering risk by providing the ability to monitor and act upon data issues 17

14