Performance tuning using SQL new features. By Riyaj Shamsudeen

Transcription

1 Performance tuning using SQL new features By Riyaj Shamsudeen

2 Who am I? 15 years using Oracle products Over 14 years as Oracle DBA Certified DBA versions 7.0,7.3,8,8i &9i Specializes in performance tuning, Internals and E-business suite Currently consulting for Cingular through Avion systems Adjunct faculty Northlake College OakTable member rshams4 at yahoo.com Riyaj Shamsudeen 2

3 Agenda Model clause Analytic functions Lag and lead First_value and last_value Dense_rank Subquery factoring Questions Riyaj Shamsudeen 3

4 Model

5 Model Emulates spreadsheet like functionality Very powerful, very flexible Code is very compact Riyaj Shamsudeen 5

6 Model Table used: item_data, Randomly generated data CREATE TABLE ITEM_DATA ( ITEM VARCHAR2(10 BYTE), LOCATION NUMBER, WEEK NUMBER, SALES_QTY NUMBER, RCPT_QTY NUMBER ) ITEM LOCATION WEEK SALES_QTY RCPT_QTY Shirt Shirt Shirt Shirt Shirt Shirt Shirt Shirt Shirt Riyaj Shamsudeen 6

7 Problem #1: Tune inventory report Accesses previous rows Formula: Inventory = last_week_inventory + this week receipts this week sales Riyaj Shamsudeen 7

8 Solution :Original SQL Total logical reads: 3007 select item, location, week, (select sum(rcpt_qty)-sum(sales_qty) from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week <= out1.week ) inventory, sales_qty, rcpt_qty from item_data out1 order by item, location,week / ITEM LOCATION WEEK INVENTORY SALES_QTY RCPT_QTY Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Riyaj Shamsudeen 8

9 Explain plan call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT AGGREGATE (cr=3000 pr=0 pw=0 time=97619 us) 6300 TABLE ACCESS FULL ITEM_DATA (cr=3000 pr=0 pw=0 time=51167 us) 600 SORT ORDER BY (cr=3005 pr=0 pw=0 time= us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=32 us) Riyaj Shamsudeen 9

10 Model Excel like A B C D E week > location sales_qty rcpt_qty inventory sales_qty rcpt_qty In Excel, formula would be B5+C4-C3 in Shirts sheet. Riyaj Shamsudeen 10

11 Model select item, location, week, inventory, sales_qty, rcpt_qty from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty) rules ( inventory [location, week]= nvl(inventory[cv(location), cv(week)-1],0) - sales_qty [cv(location), cv(week) ] +rcpt_qty [cv(location), cv(week) ] ) order by item, location, week / Script: model_example2.sql Riyaj Shamsudeen 11

12 Model inventory [location, week]= nvl(inventory[cv(location), cv(week)-1],0) - sales_qty [cv(location), cv(week) ] - +rcpt_qty [cv(location), cv(week) ] + model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty) ITEM LOCATION WEEK INVENTORY SALES_QTY RCPT_QTY Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Shirts Shirts rules ( inventory [location, week]= nvl(inventory[cv(location), cv(week)-1],0) - sales_qty [cv(location), cv(week) ] +rcpt_qty [cv(location), cv(week) ] ) Cv(location) refers to current value of location from left side of rule Riyaj Shamsudeen 12

13 Formula inventory [1,3]= nvl(inventory[1, 2],0) - sales_qty [1,3 ] +rcpt_qty [1, 3) ] Location Week Rcpt Sales Inventory (623-55)= ( )= ( )= ( )= ( )=433 inventory [location, week]= nvl(inventory[cv(location), cv(week)-1],0) - sales_qty [cv(location), cv(week) ] +rcpt_qty [cv(location), cv(week) ] Riyaj Shamsudeen 13

14 Model Accessing non-existent rows Returns null SQL Output: nvl( inventory[cv(location), cv(week)-1],0) ITEM LOCATION WEEK INVENTORY SALES_QTY RCPT_QTY Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Notice the value is reset, at the onset of different location. Riyaj Shamsudeen 14

15 Explain plan 6 logical reads, Compared to 3007 logical reads. call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT ORDER BY (cr=5 pr=0 pw=0 time=26842 us) 600 SQL MODEL ORDERED (cr=5 pr=0 pw=0 time=17662 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=39 us) Table accessed just once Riyaj Shamsudeen 15

16 More performance issues Suppose, if the SQL need to calculate running total of sales_qty also.. select item, location, week, (select sum(rcpt_qty)-sum(sales_qty) from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week <= out1.week ) inventory, (select sum(sales_qty) from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week <= out1.week ) sales_so_far, sales_qty, rcpt_qty from item_data out1 order by item, location,week This section added Riyaj Shamsudeen 16

17 Explain plan Logical reads jumped from 3007 to 6008 After adding running_sales_qty metrics call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT AGGREGATE (cr=3000 pr=0 pw=0 time=96618 us) 6300 TABLE ACCESS FULL ITEM_DATA (cr=3000 pr=0 pw=0 time=50626 us) 600 SORT AGGREGATE (cr=3000 pr=0 pw=0 time= us) 6300 TABLE ACCESS FULL ITEM_DATA (cr=3000 pr=0 pw=0 time=53670 us) 600 SORT ORDER BY (cr=6005 pr=0 pw=0 time= us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=32 us) Riyaj Shamsudeen 17

18 Model We can easily add another metrics, without any major performance impact, with model. select item, location, week, inventory, sales_qty, rcpt_qty, sales_so_for, rcpt_so_for from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 sales_so_for ) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ], sales_so_for [location, week] = sales_qty [cv(location), cv(week)] + nvl(sales_so_for [cv(location), cv(week)-1],0), ) order by item, location,week Script: model_example3.sql Riyaj Shamsudeen 18

19 Model - Rules sales_so_for [location, week] = sales_qty [cv(location), cv(week)] + nvl(sales_so_for [cv(location), cv(week)-1],0), sales_so_for [ 10, 1] = sales_qty [ 10, 1 ] + nvl ( sales_so_for [ 10, 0], 0 ) Riyaj Shamsudeen 19

20 Explain plan No increase in logical reads, even after Adding a new metric call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT ORDER BY (cr=5 pr=0 pw=0 time=37941 us) 600 SQL MODEL ORDERED (cr=5 pr=0 pw=0 time=25158 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=40 us) Riyaj Shamsudeen 20

21 Model Added more metrics, still, no increase in Logical reads! select item, location, week, inventory, sales_qty, rcpt_qty, sales_so_for, rcpt_so_for from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 sales_so_for, 0 rcpt_so_for ) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ], sales_so_for [location, week] = sales_qty [cv(location), cv(week)] + nvl(sales_so_for [cv(location), cv(week)-1],0), rcpt_so_for [location, week] = rcpt_qty [cv(location), cv(week)] + nvl(rcpt_so_for [cv(location), cv(week)-1],0) ) order by item, location,week Script: model_example3.sql Riyaj Shamsudeen 21

22 Model Problem #2: Tune moving sales average SQL Formula Moving_sales_avg = Average of ( current week sales and last two week sales ) Riyaj Shamsudeen 22

23 Old SQL select item, location, week, ( sales_qty + nvl((select sales_qty from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week = out1.week-1 ), sales_qty) + nvl((select sales_qty from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week = out1.week-2 ),sales_qty) )/3 avg_running_sales, sales_qty, rcpt_qty from item_data out1 order by item, location,week / Current week sales_qty Last week sales_qty Week before last week sales_qty Riyaj Shamsudeen 23

24 Explain plan Total logical reads: 2321 call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT ORDER BY (cr=2321 pr=2 pw=0 time=26089 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=645 us) Riyaj Shamsudeen 24

25 Model Calculating the moving sales avg using this week and past two weeks.. select item, location, week, sales_qty, rcpt_qty, moving_sales_avg from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 moving_sales_avg, sales_qty, rcpt_qty) rules ( moving_sales_avg [location, week] = (sales_qty [cv(location), cv(week)]+ nvl( sales_qty [cv(location, cv(week)-1 ], sales_qty [cv(location), cv(week)] ) + nvl( sales_qty [cv(location), cv(week)-2 ], sales_qty [cv(location), cv(week)] ) ) /3 ) order by item, location,week current week sales_qty Last week sales_qty Week before Last week Script : model_example3.sql Riyaj Shamsudeen 25

26 Model explain plan Just 5 logical reads in Fetch step call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SQL MODEL ORDERED (cr=5 pr=0 pw=0 time=14086 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=43 us) Riyaj Shamsudeen 26

27 Model & Aggregation Aggregation function also possible. Max value of inventory for a store calculated here. Second rule is referring first rule. select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ], max_inventory [location, week] = max(inventory) [cv(location), ANY ] ) order by item, location,week ANY keyword substitutes for all possible Values in that partition. ITEM LOCATION WEEK INVENTORY SALES_SO_FAR SALES_QTY RCPT_QTY MAX_INVENTORY Pants Pants Pants Script: model_features.sql Riyaj Shamsudeen 27

28 Model & Aggregation More granular boundary conditions can be specified too. Maximum inventory within a 10 weeks boundary. select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ], max_inventory [location, week] = max(inventory) [cv(location), week between cv(week)-5 and cv(week)+5 ] ) order by item, location,week / Riyaj Shamsudeen 28

29 Model & Aggregation Data from prior SQL for a 10 weeks boundary ITEM LOCATION WEEK INVENTORY SALES_SO_FAR SALES_QTY RCPT_QTY MAX_INVENTORY Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Riyaj Shamsudeen 29

30 Uses analytic functions under the cover Model & Aggregation Column projection for aggregation rule: max_inventory [location, week] = max(inventory) [cv(location), week between cv(week)-5 and cv(week)+5 ] Model uses analytic functions under the cover: Column Projection Information (identified by operation id): 1 - "ITEM"[10], "LOCATION"[22], "WEEK"[22], 0[22], "RCPT_QTY"[22], "SALES_QTY"[22], 0[22] 2 - "ITEM"[10], "LOCATION"[22], "WEEK"[22], 0[22], 0[22], "SALES_QTY"[22], "RCPT_QTY"[22] 3 - "ITEM"[VARCHAR2,10], "LOCATION"[NUMBER,22], "WEEK"[NUMBER,22], "SALES_QTY"[NUMBER,22], "RCPT_QTY"[NUMBER,22] 4 - "LOCATION"[22], "WEEK"[22], 0[22], MAX(0) OVER ( PARTITION BY "LOCATION ORDER BY "WEEK" RANGE BETWEEN 5 PRECEDING AND 5 FOLLOWING )[22] Riyaj Shamsudeen 30

31 Model Predicate push explain plan for select * from ( select.from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week)-1 ] + rcpt_qty [cv(location), cv(week)-1 ] ) order by item, location, week ) where item='pants' and location=100 and week =10; select * from table(dbms_xplan.display(null,'','typical')); Script: model_pred_push.sql In this SQL, predicates are for all three columns. But, Model rule accesses prior week. Pushing week predicate earlier in the step, will eliminate other weeks, causing logic issues Id Operation Name Rows Bytes SELECT STATEMENT * 1 VIEW SORT ORDER BY SQL MODEL ORDERED * 4 TABLE ACCESS FULL ITEM_DATA PLAN_TABLE_OUTPUT filter("week"=10) 4 - filter("item"='pants' AND "LOCATION"=100) Week predicate is not pushed here Riyaj Shamsudeen 31

32 Model Predicate push explain plan for select * from ( select.from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week) ],0) - sales_qty [cv(location), cv(week) ] + rcpt_qty [cv(location), cv(week) ] ) order by item, location, week ) where item='pants' and location=100 and week =10; select * from table(dbms_xplan.display(null,'','typical')); In this SQL, it is safe to push all three predicates, without affecting functionality Id Operation Name Rows Bytes SELECT STATEMENT * 1 VIEW SORT ORDER BY SQL MODEL ORDERED * 4 TABLE ACCESS FULL ITEM_DATA PLAN_TABLE_OUTPUT filter("week"=10) 4 - filter("item"='pants' AND "LOCATION"=100 AND "WEEK"=10) All predicates are pushed here Riyaj Shamsudeen 32

33 Model Predicate push explain plan for select * from ( select item, location, week, inventory, sales_qty, rcpt_qty from item_data model return updated rows partition by (item, location) dimension by ( week) measures ( 0 inventory, sales_qty, rcpt_qty) rules ( inventory [ week] = nvl(inventory [cv(week) ],0) - sales_qty [cv(location)-1 ] + + rcpt_qty [cv(week) ] ) order by item, location, week ) where item='pants' and location=100 and week =10 / This predicate push works incorrectly in Throws ORA-7445 in , Id Operation Name Rows SELECT STATEMENT 7 * 1 VIEW 7 2 SORT ORDER BY 7 3 SQL MODEL ORDERED 7 * 4 TABLE ACCESS FULL ITEM_DATA PLAN_TABLE_OUTPUT Predicate Information (identified by operation id): filter("week"=10) 4 - filter("item"='pants' AND "LOCATION"=100) select * from table(dbms_xplan.display(null,'','typical')); Location predicate is pushed even though Rule accesses prior row, since location is a partitioning column Riyaj Shamsudeen 33

34 Model Predicates & transitivity explain plan for select m.* from ( select item, location, week, inventory, sales_qty, rcpt_qty from item_data model return updated rows partition by (item,location) dimension by ( week) measures ( 0 inventory, sales_qty, rcpt_qty) rules ( inventory [ week] = nvl(inventory [cv(week) ],0) - sales_qty [cv(week) ] + + rcpt_qty [cv(week) ] ) order by item, location, week ) m, item_data i1 Where m.item = i1.item and m.week = i1.week and m.location = i1.location and i1.item='pants' and i1.location=100 and i1.week =10; Predicates are on i1 alias and these predicates are applied to model subquery also transitively Id Operation Name Rows SELECT STATEMENT 1 1 MERGE JOIN CARTESIAN 1 * 2 TABLE ACCESS FULL ITEM_DATA 1 3 BUFFER SORT 1 4 VIEW 1 5 SORT ORDER BY 1 6 BUFFER SORT 1 7 SQL MODEL ORDERED 1 * 8 TABLE ACCESS FULL ITEM_DATA Predicate Information (identified by operation id): filter("i1"."item"='pants' AND "I1"."LOCATION"=100 AND "I1"."WEEK"=10) 8 - filter("item"='pants' AND "LOCATION"=100 AND "WEEK"=10) Riyaj Shamsudeen 34

35 Rule evaluation order explain plan for select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + rcpt_qty [cv(location), cv(week) ], max_inventory [location, week] = max(inventory) [cv(location), week between cv(week)-5 and cv(week)+5 ] ) order by item, location,week / Script : model_order.sql Rules are evaluated in the order Specified, by default. Note ORDERED keyword in the plan Id Operation Name SELECT STATEMENT 1 SORT ORDER BY 2 SQL MODEL ORDERED 3 TABLE ACCESS FULL ITEM_DATA 4 WINDOW (IN SQL MODEL) SORT Riyaj Shamsudeen 35

36 Rule evaluation order explain plan for select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules sequential order ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + rcpt_qty [cv(location), cv(week) ], max_inventory [location, week] = max(inventory) [cv(location), week between cv(week)-5 and cv(week)+5 ] ) order by item, location,week / Sequential order can be explicitly specified Id Operation Name SELECT STATEMENT 1 SORT ORDER BY 2 SQL MODEL ORDERED 3 TABLE ACCESS FULL ITEM_DATA 4 WINDOW (IN SQL MODEL) SORT Riyaj Shamsudeen 36

37 Rule evaluation order explain plan for select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules automatic order ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + rcpt_qty [cv(location), cv(week) ], max_inventory [location, week] = max(inventory) [cv(location), week between cv(week)-5 and cv(week)+5 ] ) order by item, location,week / If rules have many inter dependencies, use automatic order. Oracle will rearrange rule order evaluation optimally Id Operation Name SELECT STATEMENT 1 SORT ORDER BY 2 SQL MODEL CYCLIC 3 TABLE ACCESS FULL ITEM_DATA 4 WINDOW (IN SQL MODEL) SORT Riyaj Shamsudeen 37

38 Model & For loop For loop can be specified in the rule, only range of weeks between 3 & 9 printed in this example. select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules ( inventory [location, for week from 3 to 9 increment 1 ] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ] ) order by item, location,week / Riyaj Shamsudeen 38

39 Model & iterate Model can iterate rule in a loop until maximum number of iteration reached or until condition reached. Create table iter (n1 number); select fib from iter model return updated rows dimension by (n1) measures (0 fib, n1) ignore nav rules iterate (50) ( fib [-2]=0, fib[-1]=1, fib [iteration_number ]= fib [iteration_number-1 ] + fib [iteration_number-2] ) order by fib / Riyaj Shamsudeen 39

40 Model & in phrase For loop can be specified in the rule, only range of weeks between 3 & 9 printed in this example. select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules ( inventory [location in (10,20), for week from 3 to 9 increment 1 ] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ] ) order by item, location,week / Riyaj Shamsudeen 40

41 Model & positional notation Positional value specified. Rows with location=10. select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules ( inventory [10, for week from 3 to 9 increment 1 ] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ] ) order by item, location,week / Riyaj Shamsudeen 41

42 select item, location, week, inventory, sales_qty, rcpt_qty from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty) ignore nav rules ( inventory [10, for week from 3 to 9 increment 1 ] = inventory [cv(location), cv(week)-1 ] ) order by item, location,week / Model & Ignore Nav Not Available Values [ non-existent rows or null values] considered as 0 instead of nulls in arithmetic calculation. - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ] Riyaj Shamsudeen 42

43 Model Works fine with Cursor_sharing set to force too select item, location, week, inventory, sales_qty, rcpt_qty from item_data where location=:"sys_b_0" model return updated rows partition by (item) dimension by (location, week) measures ( :"SYS_B_1" inventory, sales_qty, rcpt_qty) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-:"sys_b_2" ],:"SYS_B_3") - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ] ) order by item, location,week call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Riyaj Shamsudeen 43

44 Gotcha Of course, if rules are inconsistent, errors thrown: explain plan for select item, location, week, inventory, sales_qty, rcpt_qty, max_inventory from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty, 0 max_inventory) rules automatic ORDER ( max_inventory [location, week] = max(inventory) [cv(location), week between cv(week)-5 and cv(week)+5 ], inventory [location, week] = cv(week) ] + max_inventory [location, week] ) order by item, location,week / ERROR at line 11: ORA-32622: illegal multi-cell reference Two rules are cyclically referring to each other. So, illegal multi-cell reference is thrown. rcpt_qty [cv(location), cv(week) ] - sales_qty [cv(location), Riyaj Shamsudeen 44

45 Gotcha In and , model errors out: select item, location, week, inventory, sales_qty, rcpt_qty from item_data model return updated rows partition by (item) dimension by (location, week) measures ( 0 inventory, sales_qty, rcpt_qty) rules ( inventory [location, week] = nvl(inventory [cv(location), cv(week)-1 ],0) - sales_qty [cv(location), cv(week) ] + + rcpt_qty [cv(location), cv(week) ] ) order by item, location,week / select item, location, week, inventory, sales_qty, rcpt_qty * ERROR at line 1: ORA-32637: Self cyclic rule in sequential order MODEL Just specify rules automatic order To circumvent this bug. Riyaj Shamsudeen 45

46 Gotcha View merging is not considered in SQL with model clause ( at least until ) Few lines from trace ; CVM: Considering view merge in query block SEL$1 (#0) CVM: Checking validity of merging SEL$2 (#0) CVM: Considering view merge in query block SEL$2 (#0) CVM: CVM bypassed: MODEL in this view Riyaj Shamsudeen 46

47 Analytic functions

48 Analytic functions Powerful feature provides ability to access previous and following rows SQL can be written concisely using these functions Works great for data warehouse and reporting queries Riyaj Shamsudeen 48

49 Running sales total Problem: To calculate running sales total for an item and location (store) by week. select item, location, week, (select sum(sales_qty) from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week <= out1.week ) running_sales_total, sales_qty, rcpt_qty from item_data out1 order by item, location,week Script: anfn_example1.sql Riyaj Shamsudeen 49

50 Running sales total ITEM LOCATION WEEK SALES_QTY RCPT_QTY RUNNING_SALES_TOTAL Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Riyaj Shamsudeen 50

51 Performance old SQL all count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT AGGREGATE (cr=1227 pr=2 pw=0 time=21528 us) 6300 TABLE ACCESS BY INDEX ROWID ITEM_DATA (cr=1227 pr=2 pw=0 time=19522 us) 6300 INDEX RANGE SCAN ITEM_I1 (cr=611 pr=2 pw=0 time=11008 us)(object id 44482) 600 SORT ORDER BY (cr=1232 pr=2 pw=0 time=24812 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=33 us) Riyaj Shamsudeen 51

52 Running sales total New way select item, location, week, sales_qty, rcpt_qty, sum (sales_qty) over (partition by item, location order by week rows between unbounded preceding and current row ) running_sales_total from item_data Script: anfn_example1.sql Riyaj Shamsudeen 52

53 Running sales total New way call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation WINDOW SORT (cr=5 pr=0 pw=0 time=2606 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=33 us) Riyaj Shamsudeen 53

54 Running sales total ITEM LOCATION WEEK SALES_QTY RCPT_QTY RUNNING_SALES_TOTAL Pants Pants Pants Pants Pants Pants Pants Pants sum 4629 Pants Pants Pants Pants Pants sum 1490 Riyaj Shamsudeen 54

55 Running sales total ITEM LOCATION WEEK SALES_QTY RCPT_QTY Pants Pants Pants Pants sum (sales_qty) over ( Pants Pants Pants Pants Pants Pants Pants Pants Pants partition by item, location order by week rows between unbounded preceding and current row ) Sorts data by item,location and week ITEM LOCATION WEEK SALES_QTY RCPT_QTY RUNNING_SALES_TOTAL Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Calculates sum of prior rows in this partition and current row Riyaj Shamsudeen 55

56 Moving sales average Problem: To calculate moving sales average for an item and location (store). Moving_sales_avg = Average of ( sales_qty for this week and past two weeks). select item, location, week, sales_qty, rcpt_qty, (select avg(sales_qty) from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week <= out1.week and in1.week >= out1.week-2 ) moving_sales_avg from item_data out1 order by item, location,week Riyaj Shamsudeen 56

57 Moving sales average call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT AGGREGATE (cr=1209 pr=2 pw=0 time=17194 us) 1710 FILTER (cr=1209 pr=2 pw=0 time=12288 us) 1710 TABLE ACCESS BY INDEX ROWID ITEM_DATA (cr=1209 pr=2 pw=0 time=10003 us) 1710 INDEX RANGE SCAN ITEM_I1 (cr=605 pr=2 pw=0 time=5878 us)(object id 46238) 600 SORT ORDER BY (cr=1214 pr=2 pw=0 time=20611 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=35 us) Riyaj Shamsudeen 57

58 Moving sales average New way select item, location, week, sales_qty, rcpt_qty, avg (sales_qty) over ( partition by item, location order by week rows between 2 preceding and current row ) running_sales_total from item_data / Specifying a window of current Row and past two rows Script: anfn_example2.sql Riyaj Shamsudeen 58

59 Moving sales average New way TEM LOCATION WEEK SALES_QTY RCPT_QTY RUNNING_SALES_TOTAL Ties Ties Ties Ties Ties Avg Ties Ties Avg 460 Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Ties Riyaj Shamsudeen 59

60 Explain plan 5 logical reads compared to 1215 logical reads call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation WINDOW SORT (cr=5 pr=0 pw=0 time=5088 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=38 us) Riyaj Shamsudeen 60

61 Lead & Lag functions Lead function provides ability to access later rows from that data partition Lag function provides ability to access prior rows from that data partition You can even specify the offset and default values. Riyaj Shamsudeen 61

62 Lag Problem: Tune query for sales_change_percent Sales change percent formula= (this_week_sales - last_week_sales) / *100 last_week_sales ) Riyaj Shamsudeen 62

63 Lag old way select item, location, week, sales_qty, rcpt_qty, ((sales_qty - last_week_sales_qty )/last_week_sales_qty ) *100 sales_change_percent from ( select item, location, week, sales_qty, rcpt_qty, (select sales_qty from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week = out1.week-1 ) last_week_sales_qty from item_data out1 order by item, location,week ) Script : anfn_lag.sql Riyaj Shamsudeen 63

64 Lag old way call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation VIEW (cr=1178 pr=2 pw=0 time=13670 us) 600 SORT ORDER BY (cr=1178 pr=2 pw=0 time=13659 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=632 us) Riyaj Shamsudeen 64

65 Lag - tuned select item, location, week, sales_qty, rcpt_qty, ((sales_qty - last_week_sales_qty )/last_week_sales_qty ) *100 sales_change_percent from ( select item, location, week, sales_qty, rcpt_qty, lag(sales_qty,1,sales_qty) over (partition by item, location order by week) last_week_sales_qty from item_data out1 order by item, location,week ) Lag is to access prior row, in that partition. First argument : column_name Second argument : offset Third argument : default value if out of partition Partitions data, sorts rows by week in that data partitions Riyaj Shamsudeen 65

66 ITEM LOCATION WEEK SALES_QTY RCPT_QTY Sales change percent Pants Pants Pants Pants lag(sales_qty,1,sales_qty) over ( partition by item, location order by week ) Sales change percent (( )/239 )*100=213.3 Pants Pants Pants Pants Pants Pants Pants Pants Pants Sorts data by item,location and week ITEM LOCATION WEEK SALES_QTY RCPT_QTY SALES_CHANGE_PERCENT Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Lag accesses prior row to calculate sales_change_percent Riyaj Shamsudeen 66

67 Lag sales_change_percent (( )/239 )*100=213.3 ITEM LOCATION WEEK SALES_QTY RCPT_QTY SALES_CHANGE_PERCENT Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Riyaj Shamsudeen 67

68 Lag 6 buffer gets compared to 1180 buffer gets. call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation VIEW (cr=5 pr=0 pw=0 time=4489 us) 600 WINDOW SORT (cr=5 pr=0 pw=0 time=4486 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=37 us) Riyaj Shamsudeen 68

69 Lag and lead Problem: Tune SQL that prints past two weeks and next two weeks sales_qty in the same line. Preceding Preceding This Succeeding Succeeding ITEM LOCATION WEEK week 2 week 1 week week 1 week Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Pants Riyaj Shamsudeen 69

70 Lag and lead -old select item, location, week, rcpt_qty, (select sales_qty from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week = out1.week-2 ) prev_week_2_sales_qty, (select sales_qty from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week = out1.week-1 ) prev_week_1_sales_qty, sales_qty, (select sales_qty from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week = out1.week+1 ) succ_week_1_sales_qty, (select sales_qty from item_data in1 where in1.item=out1.item and in1.location=out1.location and in1.week = out1.week+2 ) succ_week_2_sales_qty from item_data out1 order by item, location,week Script : anfn_lead_n_lag.sql Riyaj Shamsudeen 70

71 Lag and lead call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation TABLE ACCESS BY INDEX ROWID ITEM_DATA (cr=1143 pr=0 pw=0 time=8574 us) 540 INDEX RANGE SCAN ITEM_I1 (cr=603 pr=0 pw=0 time=5512 us)(object id 46444) 570 TABLE ACCESS BY INDEX ROWID ITEM_DATA (cr=1173 pr=0 pw=0 time=8713 us) 570 INDEX RANGE SCAN ITEM_I1 (cr=603 pr=0 pw=0 time=5503 us)(object id 46444) 570 TABLE ACCESS BY INDEX ROWID ITEM_DATA (cr=1173 pr=0 pw=0 time=8836 us) 570 INDEX RANGE SCAN ITEM_I1 (cr=603 pr=0 pw=0 time=5614 us)(object id 46444) 570 TABLE ACCESS BY INDEX ROWID ITEM_DATA (cr=1173 pr=2 pw=0 time=10029 us) 570 INDEX RANGE SCAN ITEM_I1 (cr=603 pr=2 pw=0 time=6720 us)(object id 46444) 600 SORT ORDER BY (cr=4667 pr=2 pw=0 time=43306 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=40 us) Riyaj Shamsudeen 71

72 Lag and lead - tuned select item, location, week, rcpt_qty, lag (sales_qty, 2) over ( partition by item, location order by week asc ) prev_week_2_sales_qty, lag (sales_qty, 1) over ( partition by item, location order by week asc ) prev_week_1_sales_qty, sales_qty, lead (sales_qty, 1) over ( partition by item, location order by week asc ) succ_week_1_sales_qty, lead (sales_qty, 2) over ( partition by item, location order by week asc ) succ_week_2_sales_qty from item_data Prior weeks Following weeks Riyaj Shamsudeen 72

73 Lag and lead - tuned 6 logical reads, compared to 4667 logical reads call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation WINDOW SORT (cr=5 pr=0 pw=0 time=8597 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=35 us) Riyaj Shamsudeen 73

74 First_value & last_value These functions provides first or last values in an ordered set of rows Very useful in finding the top or bottom values and also other columns in that same row. Riyaj Shamsudeen 74

75 First_value & last_value Problem: Tune SQL that finds worst and best seller for an item by week OLD way: Finds maximum sales qty and location select mi.item, mi.week, ma.max_location top_seller, ma.max_sales_qty top_sales_qty, mi.min_location worst_seller, mi.min_sales_qty worst_sales_qty from (select item, week, sales_qty max_sales_qty, location max_location from item_data where (item, week, sales_qty) in ( select item, week, max(sales_qty) from item_data group by item, week ) ) ma, (select item, week, sales_qty min_sales_qty, location min_location from item_data where (item, week, sales_qty ) in ( select item, week, min(sales_qty) from item_data group by item, week ) ) mi where mi.item = ma.item and mi.week = ma.week order by item, week Finds minimum sales qty and location Riyaj Shamsudeen 75

76 First_value & last_value call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT ORDER BY (cr=20 pr=0 pw=0 time=15415 us) 61 HASH JOIN RIGHT SEMI (cr=20 pr=0 pw=0 time=14170 us) 60 VIEW (cr=5 pr=0 pw=0 time=1178 us) 60 SORT GROUP BY (cr=5 pr=0 pw=0 time=1174 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=39 us) 610 HASH JOIN (cr=15 pr=0 pw=0 time=9695 us) 61 HASH JOIN RIGHT SEMI (cr=10 pr=0 pw=0 time=5926 us) 60 VIEW (cr=5 pr=0 pw=0 time=1306 us) 60 SORT GROUP BY (cr=5 pr=0 pw=0 time=1186 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=35 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=31 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=27 us) Riyaj Shamsudeen 76

77 First_value & last_value (tuned) select distinct item, week, first_value (location) over (partition by item, week order by sales_qty desc rows between unbounded preceding and unbounded following ) top_seller, first_value (sales_qty) over (partition by item, week order by sales_qty desc rows between unbounded preceding and unbounded following ) top_sales_qty, last_value (location) over (partition by item, week order by sales_qty desc rows between unbounded preceding and unbounded following ) worst_seller, last_value (sales_qty) over (partition by item, week order by sales_qty desc rows between unbounded preceding and unbounded following ) worst_sales_qty from item_data order by item, week Script : anfn_first_value.sql Riyaj Shamsudeen 77

78 First_value & last_value (tuned) first_value (location) over ( partition by item, week order by sales_qty desc rows between unbounded preceding and unbounded following ) First_value and last_value Retrieves column values from First and last row, in that ordered set ITEM WEEK LOCATION SALES_QTY Shirt Shirt Shirt Shirt Shirt Shirt Shirt Shirt Sort by item, week, sales_qty desc ITEM LOCATION WEEK SALES_QTY Shirt Shirt Shirt Shirt Shirt Shirt first_value/last_value ITEM WEEK TOP_SELLER TOP_SALES_QTY WORST_SELLER WORST_SALES_QTY Shirt Shirt Shirt Shirt Riyaj Shamsudeen 78

79 First_value & last_value (tuned) 6 logical reads, compared to 20 logical reads. Item_data accessed just once, compared to four times. call count cpu elapsed disk query current rows Parse Execute Fetch total Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 30 Rows Row Source Operation SORT UNIQUE (cr=5 pr=0 pw=0 time=4478 us) 600 WINDOW SORT (cr=5 pr=0 pw=0 time=3580 us) 600 TABLE ACCESS FULL ITEM_DATA (cr=5 pr=0 pw=0 time=39 us) Riyaj Shamsudeen 79

80 Dense_rank Tune SQL to print top two selling locations and receipt_qty select item, week, location, sales_qty, sales_rnk, rcpt_rank from ( select item, week, location, sales_qty, (select count(*) from item_data i1 where i1.item = iout.item and i1.week = iout.week and Finding sales_rnk i1.sales_qty >= iout.sales_qty ) sales_rnk, (select count(*) from item_data i1 where i1.item = iout.item and i1.week = iout.week and i1.rcpt_qty >= iout.rcpt_qty ) rcpt_rank from item_data iout ) where sales_rnk <3 order by item, week, sales_qty desc Script : anfn_dense_rank.sql Riyaj Shamsudeen 80