Treinamento em Análise Quantitativa & Planejamento de Capacidade. Virgilio A. F. Almeida

Transcription

1 Treinamento em Análise Quantitativa & Planejamento de Capacidade Virgilio A. F. Almeida DATAPREV Rio de Janeiro 17 Dezembro de 2009 Módulo: estudo de caso 2 Departamento de Ciência da Computação Universidade Federal de Minas Gerais - UFMG

2 E business services 2

3 Banking services in Brazil 3

4 Banking services in Brazil 4

5 Banking services in Brazil 5

6 Banking services in Brazil 6

7 Banking services in Brazil 7

8 Banking Transactions in Brazil 8

9 Banking Transactions in Brazil 9

10 Performance Engineering Methodology Chapter 4 Performance By Design 10

11 Typical PE Questions Can the insurance claim li system meet its performance requirements of sub second response time when a natural disaster occurs (e.g., a hurricane). Is the infrastructure of a government agency scalable and can it cope with the computing demands of the new required online security mechanisms? Is the reservation system for cruise lines able to respond to anticipated peak of customer inquiries after a TV ad campaign? 11

12 PE Larger Questions How can one plan, design, develop, deploy, and operate IT services that meet ever increasing demands for performance, availability, reliability, and security? Is a given IT system properly designed and sized for a given load condition? D. A. Menascé. All Rights Reserved.

13 PE Activities Understand dthe key factors that t affect a system s performance. Measure the system and understand its workload. Develop and validate a workload model that captures the key characteristics of the actual workload. Develop and validate an analytic model that accurately predicts the system s performance. Use the models to predict and optimize the system s performance. 13

14 Modeling Process 14

15 Motivating Example: a Call Center 15

16 Goals: Call Center Foster better relationships with customers, creating customer loyalty and ensuring quality service. Improve efficiency and service performance. Identify and explore new sales opportunities. Main Functions: Order status inquiry Shipment tracking Problem resolution status inquiry Requirements: sub second response time and 24x7 operation. 16

17 At the Requirements Analysis Phase Workload definition: Call center s view: Arrival rate of phone calls IT system s view: Functions received from the representatives. DB server view: SQL requests from the application server. LAN view: packet size distribution and interpacket arrival time. 17

18 At the System Design Phase What should the system throughput be to meet sub second response times? 200 customer service representatives and 80% are working during the peak hour. Average think time of 30 sec. 18

19 Call Center Model Using the Interactive Response Time Law: R = N / X 0 Z 1 sec X N Z + R = = 5.16 functions/sec 19

20 At the System Development Phase What should be the capacity of the DB server so that the performance goals are met? Each submitted functions requires 2.2 SQL calls on average. From the Forced Flow Law: X DB = V X DB = tps 20

21 At the Operation Phase Assume DB server is a problem. Response times exceed sub second goal. Measurements during peak hour: queries/hour Each query needs 50 msec of CPU, performs 4 I/Os on disk 1 and 2 I/Os on disk 2. Each I/O takes 8 msec on average. X 0 = / 3600 = 16 queries/sec Service demands: Dcpu = 0.05 sec; Ddisk1 = 4 x = sec; Ddisk2 = 2 x = sec. 21

22 At the Operation Phase (cont d) Utilization computations (Service Demand Law): Ucpu = Dcpu x X0 = 0.05 x 16 = 80% Udisk1 = Ddisk1 x X0 = x 16 = 51.2% Udisk2 = Ddisk2 x X0 = x 16 = 25.6% Response Time (Open QN Model) R R R R ' CPU D = 1 U cpu D = 1 U cpu 0.05 = = 0.25sec ' disk1 disk1 = disk 1 ' disk 2 = D disk 2 1 U disk 2 ' ' ' 0 = Rcpu + Rdisk1 + Rdisk 2 = = 0.066sec = = 0.022sec sec 22

23 At the Evolution Phase Develop Web based interface. Security requirements mandate that new applications be developed for Web access (authentication, auditing, DB access control mechanisms). Local Web Arrival 16 1 Rate (tps) Service demands (sec) CPU Disk Disk

24 Model for Evolution Scenario Open Multiclass Queuing Networks This wokbook comes with the books "Performance by Design," "Capacity Planning for Web Services" and "Scaling for E-Business" by D. A. Menascé and V. A. F. Almeida, Prentice Hall, 2004, 2002 and No. Queues: 3 No. of Classes: 2 Classes Arrival Rates: Service Demand Matrix Classes Queues Type (LI/D/MPn) Local Web CPU LI Disk 1 LI Disk 2 LI

25 Results for Evolution Scenario Open Multiclass Queuing Networks - Residence Times This wokbook comes with the books "Performance by Design," "Capacity Planning for Web Services" and "Scaling for E-Business" by D. A. Menascé and V. A. F. Almeida, Prentice Hall, 2004, 2002 and Classes Queues Local Web CPU Disk Disk Response Time

26 Performance Engineering Methodology 26

27 Total Cost of Ownership (TCO) Hardware costs (purchase and/or leasing expenses) Software costs Communication costs Management costs Support costs Facilities costs (include power) Downtime costs 27

28 A Simple Way to Estimate the Cost of Downtime Dave Patterson EECS Department University of California, Berkeley

29 Motivation Our perspective: Dependability and Cost of Ownership are the upcoming challenges Past challenges: Performance and Cost of Purchase Ideal: compare (purchase cost + outage cost) But companies claim most customers won t pay much more for more dependable products 1. How do you measure product availability? 2. How much money would greater availability save? Researchers arestarting to benchmark dependability: commonplace in 2 to 4 years? Predict hours of downtimeper year per product If customers cannot easily estimate downtime costs, who will pay more for dependability? outage cost = downtime hours X cost/hour of downtime

30 2000 Downtime Costs (per Hour) Brokerage operations $6,450,000 Credit card authorization $2,600,000 Ebay(1 outage 22 hours) $225,000 Amazon.com $180,000 Package shipping i services $150, Home shopping channel $113,000 Catalog sales center $90, Airline reservation center $89,000 Cellular service activation $41,000 On line network fees $25,000 ATM service fees $14,000 Sources: InternetWeek 4/3/ Fibre Channel: A Comprehensive Introduction, R. Kembel 2000, p.8....based on a survey done by Contingency Planning Research."

31 One Approach 1. Estimate on line income lost during outages (Online Income / quarter divided by hours / quarter) X hours of downtime Lost off line sales?employee productivity? 2. Interview employees after an outage, ask how many were idled by the outage, and calculate their salaries and benefits How many employees would answer (honestly)? (Big Brother data collection?) How many companies would spend the money to collect this information? But want CIO system administrator to easily estimate But want CIO, system administrator to easily estimate costs to evaluate future purchases

32 A Simple Estimate Estimated Average Cost of hour of downtime = Employee Costs per Hour * Fraction Employees Affected by Outage + Average Revenue per Hour * Fraction Revenue Affected by Outage Employee Costs per Hour: total salaries and benefits of employees per week divided by the average number of working hours Average Revenue per Hour: total revenue per week divided by average number of open hours Fraction Employees Affected by Outage and Fraction Revenue Affected by Outage are just educated guesses or plausible ranges Since evaluating purchases, estimates OK

33 Caveats Ignores cost of repair, such as cost of operator overtime or bringing in consultants Ignores daily, seasonal variations in revenue Indirect costs of outages can be as important as these more immediate costs Company morale can suffer, reducing productivity for periods that far exceed the outage Frequent outages can lead to a loss of confidence in the IT team and its skills (IT blamed for everything) Can eventually lead to individual departments hiring their own IT people, which lead to higher direct costs Hence estimate tends to be conservative

34 3 Examples Institution Revenue Hours Employee Hours per Week per Week EECS Dept. 10 hrs x 5 days U.C. Berkeley Amazon 24 hrs x 7 days 10 x 5 (Online) Sun Microsystems 24 x 5 10 x 5 (Offline, but Global)

35 Example pe1: EECS Dept. at UC U.C.B. State t funds: 68 staff tff@ $100k / week + 90 $200k/week (including benefits) External (federal) funds per week: School year: 670 (students, $450k Summer: 635 (students, faculty, $675k ~ $44,000,000 / year in Salaries & Benefits or ~ $850,000 / 50 hours / week => ~ $17,000 per hour If outage affects 80% employees, ~ $14,000 / hour Guess at 2002 outages: ~ 50 hours => ~ $680, per year

36 Example 2: Amazon 2001 Revenue $3.1B/year, with 7744 employees Revenue per hour (24x7): ~ $350,000 If outage affects 90% revenue: ~ $320, Public quarterly reports do not include salaries and benefits directly Assume avg. annual salary is $85,000 UC staff ~$70,000; 000; Amazon 20% higher? $656M / year, or $12.5M / week for all 50 hours / week => ~ $250,000 per hour If outage affects 80% employees: $200,000 Total: ~ $520,000 / hour Note: Employee cost/hour ~ revenue

37 Example 3: Sun Microsystems 2001 Revenue $12.5B/year, with 43,314 employees Revenue per hour (24x5): ~ $2,000k If outage affected 10% revenue: ~ $200, Assume avg. annual salary is $100,000 More engineers than Amazon, so 20% higherh $4331M/year, or $83M / week for all 50 hours / week => ~ $1,660,000 per hour If outage affects 50% employees: ~$830,000 Total: ~ $1,030,000 / hour Note: Employee costs 80% of outage cost

38 Purchase Example Comparing 2 RAID disk arrays, same capacity; Brand X Purchase Price: $200, Brand Y Purchase Price: $250,000 Dependability benchmarks suggest over 3 year product lifetime 10 hours of downtime for Brand X vs. 1 hour of downtime for Brand Y Using EECS downtime costs of $14k per hour: Brand X: $200,000+10x$14,000 = $340,000 Brand Y: $250, x$14,000 = $264,000 Helps organization justify selecting more dependable but more expensive product

39 Observations Data was easy to collect 3 s inside EECS Quarterly Reports for Amazon and Sun Quantifies cost difference of planned vs. unplanned downtime, which is not captured by availability ( %) Employee productivity costs, traditionally ignored in such estimates, are significant Even for Internet companies like Amazon Dominate traditional organizations like Sun Outages at universities and government organizations can be expensive, even without a computer related revenue stream Include employee productivity in costs!

40 Is greater accuracy needed? Downtime can have subtle, difficult to measure effects on sales and productivity Are sales simply re ordered after downtime, Or do customers switch to more dependable company? Do employees just do other work during downtime, Or does downtime result in lost work, psychological impact, sothat it takes longer than the downtime to recover? Hence spending much more for accuracy may not be worthwhile hil Also, metric is less likely to see widespread use if it s much more difficult to generate

41 Conclusion Employee productivity costs are significant Goal:easy easy to calculate estimate to lays groundwork for buying dependable products Unclear what htprecision ii is needed ddor possible, so make easy for CIOs, SysAdmins to calculate Estimated Average Cost of 1 hour of downtime = Employee Costs per Hour * Fraction Employees Affected + Average Revenue per Hour * Fraction Revenue Affected Web page pg for downtime calculation for you:

42 TCO Example Basic costofsystemy of = $300, Basic cost of System Z = $350,000 Throughput of System Y = 220 tps Throughput h of System Z = 230 tps System Y expected downtime = 38 hrs/3 yrs System Z expected downtime = 21 hrs/3 yrs Call center charges $5 per call from customers. Avg. call rate = 1,000 calls/hr Avg. cost per hr of downtime = $5,000 Total cost = Basic Cost + Downtime Cost Cost of System Y = $300, * 1,000 * 5 = $490,000 Cost of System Z = $350, * 1,000 * 5 = $455,000 System Z is preferable over System Y. 42