Customer Snapshot Building a DWH with HP Oracle s DB-Machine

Transcription

1 Customer Snapshot Building a DWH with HP Oracle s DB-Machine Christian Maar, CIO Allegro Group Rafal Kudlinski, Head DWH Division Würzburg, 3rd February 2009 EMEA DWH Terabyte Club Customer Meeting

2 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 2

3 Allegro is in the middle of implementing the new 36 month IT strategy. The DWH program is one of the most important initiatives and part of our long term goals to guarantee further rapid business growth. Management Summary Allegro is the most successful ecommerce trading system in Poland and the biggest non ebay auction platform worldwide As a consequence of Allegro s rapid growth over the years the amount of data is continually increasing Allegros data handling is not capable to support fast/real time controlling and business decision making Our plan is not less than to built one of the most efficient and scalable state of the art DWH infrastructures in Poland or even in Central East Europe The DWH program will be done in a phased approach The first phase of the program was dedicated to a complex software and hardware PoC with the result that the full Oracle stack suits best for Allegro s requirements Allegro has to implement a multi layer architecture to avoid complications with our PX environment At the end of 2009 Allegro will have a sophisticated real time DWH ready to use 3

4 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 4

5 Allegro is the most successful ecommerce trading system in Poland and the biggest non ebay auction platform worldwide. The Allegro Group operates 29 websites in 10 countries across Eastern Europe. Figures and Facts about Allegro MORE THAN 12 MILLION USERS Over 6000 new users every day. Over 2 million new users every year. 800 EMPLOYEES 200 employees in the IT department: 120 in the development division and 80 in other IT divisions (IS, DWH, R&D etc.). MORE THAN 500 MILLION PAGE VIEWS The number of page views has doubled within the last 3 years. MORE THAN 11 MILLION AUCTIONS The number of auctions increased by 8 millions over the past 3 years. 5

6 Allegro is using a five tier Architecture to run its Applications. The availability is approximately 99.9x% and the average delivery time for the Index page is around 1 second (approx. 182 billion Page Impressions per year). Figures and Facts about Allegro Current Infrastructure BND Poland (active) 476 servers FFM Germany (semi active) 305 servers Frontend Layer 130 blade servers Blade servers: Blade servers: Cache Layer 80 blade servers IBM LS20, IBM LS21 IBM HS21, HP 460c, HP 865c IBM LS20, IBM LS21 IBM HS21, HP 460c, HP 865c Application Layer 226 blade servers 8GB 32GB RAM 1-2 Quad Core CPU 303 servers 8GB 32GB RAM 1-2 Quad Core CPU Database Layer 48 servers 2 x IBM p x blade servers 1 x DB Machine 2 x IBM p590 Storage Layer 200 TB IBM DS TB IBM DS TB 3PAR S400 54TB Exadata 75TB 125 TB IBM DS TB IBM DS TB 3PAR S400 54TB 6

7 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 7

8 The optimization is impossible, if you don t know what has to be improved. The problem that Allegro has been facing at the moment is a lack of centralized controlling. The data handling is not capable to support fast/real time controlling and business decision making. We Don t Know What We Don t Know General Management IT Marketing Customer Care allegro.pl otomoto.pl otodom.pl ceneo.pl istore.pl... A very variegated panorama of data and platforms. Little capacity for uniform reporting and advanced data analysis. 8

9 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 9

10 A typical company starts their controlling with XLS based reports generated on demand and sent by or published via the web. Becoming bigger a company needs a centralized controlling system and realizes that without a single point of truth detailed and advanced controlling is nearly impossible. Controlling in a Startup Company 10

11 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know We They Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 11

12 As a consequence of Allegro s rapid growth over the years the amount of data is continually increasing. It has already reached a state where it becomes more and more complex to use the available data in an efficient way. Things Becoming More Complex 50 Available Data Terabytes of Data Knowledge Gap Execution Gap Analytic Capability Execution Capability Time

13 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 13

14 IT at Allegro has not reached its full potential, which in our experience involves exploiting innovation to drive constant improvement in the operations of a business and to give a real advantage over competition with new products and capabilities. Moving from Day2day Business to a More Strategic Thinking IT as and service supplier IT focuses primarily on information issues, stays out of business priorities Aligned with the business IT collaborates with executives to meet business goals Competitive differentiation IT looks beyond immediate business needs, considers technology trends and competitive landscape to play proactive role in shaping business strategy Multiyear 1-3 years and should try to move here IT strategy planning horizon Single Year 1-12 month We as a company are here Focus of IT strategy 14

15 A closer look at the Top 10 Strategic Technologies mentioned at the Gartner CIO conference in Cannes in the last quarter reveals that Business Intelligence is still one of the most important areas of interest for top decision makers. Moving from Day2Day Business to a More Strategic Thinking Which technologies will overcome important barriers and limitations during the next three years? Which technologies will move from narrow niches to more widespread adoption during the 2009 through 2011 time frame? How should companies change their plans and approaches based on these new opportunities? Strategic Technologies Affect Run, Grow and Transform the Business Initiatives. Top 10 Strategic Technology Areas for Virtualization 2. Cloud Computing 3. Servers: Beyond Blades 4. Web-Oriented Architectures 5. Enterprise Mashups 6. Specialized Systems 7. Social Software and Social Networking 8. Unified Communications 9. Business Intelligence 10. Green IT Main Focus of IT Strategy 15

16 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 16

17 An optimal IT Strategy must comprehend the business context, applications, technology, organizational capability and governance. DWH is the most important part of the Backoffice Application Portfolio. Process of a IT Masterplan Where the DWH Program Fits in The dynamics of the Internet industry Business Context The realities associated with Allegro s business and their implication on its IT function DWH B up to 20% of the IT budget Governance & Funding The decision-making process and its impact on IT s ability to meet its objectives G IT Strategy A Application Portfolio The state of Allegro s business applications and their ability to support today s and tomorrow s business Organisational Capability The readiness of Allegro s human assets to perform consistently with the needs of the organization O T Technology Infrastructure The degree of simplicity, uniformity and sufficiency that Allegro s technology infrastructure represents 17

18 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 18

19 Our plan is not less than to built one of the most efficient and scalable state of the art DWH infrastructures in Poland. Appliances like the new Oracle Database Machines will deliver a sophisticated infrastructure to enable Real Time data calculation and mass data handling. BI as Part of the IT Masterplan DWH as a Central Point 19

20 The speed of changes in the BI solution market is incredibly high. Our requirements of having the best BI environment on the one side and avoiding too much vendors on the other side led us to the decision to select the Oracle BI solution. BI as Part of the IT Masterplan HD and SW Selection PROJECT/PROGRAM LAUNCH AND MANAGEMENT APPLICATION ARCHITECTURE DESIGN DWH PRODUCT SELECTION BI PRODUCT SELECTION Program Definition Program Planning & Management Business Requirements Definition Gather & document technical requirements Create architecture model Determine phased implementation approach Create the architecture plan Develop configuration recommendations Develop evaluation matrix Research candidate products (IBM, Oracle, Teradata, Netezza) Develop product short list (Oracle, Netezza) Evaluate product options Determine product recommendation Develop evaluation matrix Research candidate products (Cognos, Business Objects, Microstrategy, Oracle) Develop product short list (Microstrategy, Oracle) Evaluate product options Determine product recommendation DWH / BI Technical Architecture 20

21 After a few tests the desicion to select Oracle was obvious. The Oracle DWH machine bet the Netezza appliance in all important areas like data bandwidth and load rate. BI as Part of the IT Masterplan Oracle vs. Netezza Oracle DB Machine Netezza 7.5 GB/s Data Bandwidth Load Rate 500GB/hr 112 Disks 112 CPU Cores 1 Gb/s Ethernet Disk Failure Aborts Queries & Txns Proprietary Hardware Master Node that can be bottleneck + 14 GB/s Data Bandwidth + Load Rate 1500 GB/hr Disks CPU Cores + 20 Gb/s Infiniband + Disk Failure is Transparent + Open Standard Hardware + No Master Node 21

22 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know What We Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Roadmap of Allegro 10 Next Steps and Outlook 11 Q&A 22

23 The DWH program has a very high visibility / awareness within the Allegro Group. Stakeholders are located in nearly every Business division within our company. To avoid re-inventing the wheel we have boarded external companies to support certain project / phases. BI Roadmap of Allegro DWH Program Organisation Front Office Business Sponsor/Business Driver DW/BI Director/Program Manager Coaches Regular Line-Up Special Teams Project Manager Business Project Lead Business Analyst Data Steward/QA Analyst Data Architect/Data Modeler DBA Metadata Manager ETL Architect ETL Developer BI Architect/App Developer/Portal Developer Technical Architect/Tech Support Specialist Security Specialist Tester Data Mining/Stats Specialist Educator 10 internal team members 10 external team members from Oracle and ISE 23

24 The DWH will be build in a phased approach. In the first step we will integrate the auction information, including Customer Care, and migrate/build the necessary data aggregations to enable analysis and standard KPI reporting. Mining and simulation capabilities will follow in a second step. BI Roadmap of Allegro The first phase handles the data transfer from the auction platform. DWH Architecture Data Source Driven Local BI Subprojects User Driven Design DWH Model Load Data Users Items Transactions Billing Aggregate Data Historization KPI Aggregation Reporting and Analysis Standard Reports Ad-hoc Reports Mining Simulation The next phases will connect the various non-auction services to the DWH. 24

25 From an IT architectural point of view, the data from the auction systems and the classifieds will be merged when transferred to the DWH. A set of data marts will be extracted for dedicated requests like operational / financial reporting and for advanced analysis / simulations. BI Roadmap of Allegro Legacy Layer Auction Classified Financial Data Customer Care Data Monitoring Data IT Data Data Warehouse Layer Oracle 11g Auction DWH Model Classified Auction DWH Financial Model Classified Data DWH Model Shops DWH Model Data Mart Reporting / Analysis / Mining / Simulation Layer OBIEE Management Dashboard Simulation Engine Data Mining Tools OBIEE Answers OBIEE Delivers 25

26 During the first phase we will concentrate on the most important reports / KPIs which are mainly connected to our major platform allegro.pl. Integrating click-stream-information will be a huge challenge because of 500 millions items which have to be crunched each day the entire year. BI Roadmap of Allegro KPIs for the First Phase Regs Users Listed auc0ons Listed items Closed auc0ons Closed items Sold auc0ons Sold items Open auc0ons Open items GAV 5K PLN Bids Turnover Refunds Count of new user registra0ons Count of all ac0ve user in Allegro service Auc0ons listed at current day. Items listed at current day. Auc0ons closed at current day Count of items offered on auc0ons closed at current day. Auc0ons sold at current day Count of items sold on auc0ons closed at current day. Auc0ons s0ll open at current day Count of offered items The total sum of prices for all sold items, in case of which the auc0on total final bid price* is equal to or less than gav_limit_5k Count of bids by auc0ons Turnover from items sold on auc0ons closed at current day. Refunds from items sold on auc0ons closed at current day. 26

27 Allegro will implement a multi layer architecture: to avoid complications with our PX environment; to work cost effective because of using low cost hardware in the middle layer; to deliver real time data in the DWH production layer. BI Roadmap of Allegro Load ETL Oracle Data Guard Data Cleansing 2 * IBM P590 Machine (each 8 processors, 16 cores ) DS8300 Array (56TB user space) HP Oracle Database Machine (64 Intel processor cores) HP Oracle Exadata Storage Server (75 TB of raw storage) (14 GB/sec data bandwidth) 27

28 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know We They Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 28

29 Thing big act small and before you start search for the right staff! Next Steps and Outlook DWH implementation ELT implementation BI implementation Web mining data integration Search data integration Operational systems as users DWH/BI team Requirements collection and evaluation BI tool evaluation and selection process DWH infrastructure setup 29

30 Agenda 1 Management Summary 2 Figures and Facts about Allegro 3 We Don t Know We They Don t Know 4 Controlling in a Startup Company 5 Things Becoming More Complex 6 Moving from Day2day Business to a More Strategic Thinking 7 Process of a IT Masterplan 8 BI as Part of the IT Masterplan 9 BI Road Map of Allegro 10 Next Steps and Outlook 11 Q&A 30

31 Q&A 31