Price index calculation and weighting with web scraped data

Transcription

1 Price index calculation and weighting with web scraped data ESTP course on Automated collection of online prices: sources, tools and methodological aspects Antonio Chessa, Statistics Netherlands THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2 Outline Part 1: Definition of (homogeneous) products Part 2: Choice of index formula and other aspects Part 3: Case study on web scraped data Potential pitfalls when using web scraped (WS) data Comparison of price indices for WS and SC data Comparison of price indices for different methods Summary of findings 2

3 Main problems in index calculation Definition of homogeneous products: GTINs (barcodes) constitute the most detailed product level Is the barcode level acceptable? Choice of index method: What index formula? How to choose the product weights? Direct or chained method? Additional choice aspects 3

4 Part 1: Definition of homogeneous products 4

5 GTIN level: Example of relaunch Shampoo: Elvive, multivitamine 2 in 1, 250 ML, normal hair 5,00 Price Number of items sold 4, , , , , , , , , ,00 0 EAN: EAN: EAN: EAN: Source: Scanner data of a Dutch drugstore chain 5

6 Impact of relaunch on price index Corrected for relaunches Not corrected for relaunches Week 4, 2011 = 100 Source: Chessa A.G. (2013). See references at the end. 6

7 GTIN level: Price indices for clothing 7

8 GTIN level: Price indices for pastries GTINs are grouped according to a set of common characteristics using 3 attributes: (1) type of pastry, (2) size, (3) flavour 8

9 A closer inspection of the results Price indices for girls clothing show very rapid decline The index for pastries at GTIN level has the same trend as the index for GTIN groups The GTIN level thus seems to be suitable for pastries, but not for girls clothing Why do we see such diverging results at GTIN level? 9

10 Assortment dynamics Is there a link with the stability of assortments? Measures of assortment dynamics (bilateral): Flow: % of items that are sold in both months Inflow: % of items that are only sold in the second month Outflow: % of items that are only sold in the first month We keep month 1 fixed (base month) and investigate what happens to the flow rate throughout a year 10

11 Assortment dynamics: (1) Girls dresses 11

12 Assortment dynamics: (2) Pastries 12

13 Some observations The number of matched items ( flow ) decreases very fast within 1 year for girls dresses Items enter at high prices and then drop rapidly When new items are not linked to similar old items, higher prices of new items will not be picked up as price increases The assortment of pastries is more stable 13

14 How could we proceed? GTINs as products works well with stable assortments, but not (always) with dynamic assortments This choice has the advantage of limited monthly maintenance But the risk of relaunches requires a generic and future-proof approach Metadata is needed in order to link GTINs and to pick up hidden price increases 14

15 Metadata for linking or grouping GTINs Retailers product codes: Retailers may have GTIN-links as part of their stock keeping The links are usually 1-to-1 relations. Is this satisfactory? Grouping by item characteristics: Items are combined when they share the same characteristics Which item attributes should we select? What selection methods could be thought of? Let us focus on the second approach 15

16 How we defined products in the past Descriptions were set up for a sample of products Example: Shampoo of Andrélon, 250ml, normal hair A representative item was chosen that satisfies the product characteristics and its price was recorded For large electronic data sets we need a more efficient approach for selecting relevant characteristics 16

17 What does statistical literature offer? Analysis of variance (ANOVA) Adjusted maximum likelihood methods: Likelihood ratio tests Information criteria (e.g. Akaike - AIC, Bayesian - BIC) Idea behind these methods: Detailed product segmentation favours goodness of fit (gof) Each product (GTIN group) requires a unit value estimate Tighter products less degrees of freedom (dof) Find balance between gof and dof 17

18 Brief review of methods The idea of balancing gof and dof is useful Limitations of methods: Simplistic model assumptions (independence of prices over time) Fitting procedures do not take full account of assortment dynamics 18

19 Ideas for a new measure Building blocks: R-squared (explained variance) Flow rate over time How it functions: More (less) detailed products Higher (lower) R-squared More (less) detailed products Lower (higher) flow rate Which level of detail gives the best balance between the two? Work is in initial research phase 19

20 New measure illustrated 1 R-squared 1 Flow rate (product matches) Adding attributes Adding attributes 0 0 One product GTINs as products One product? 20

21 Part 2: Choice of index method 21

22 Index methods in brief Bilateral vs Multilateral methods: Bilateral methods use data from 2 months Multilateral methods use data from more than 2 months So, the length of the time window has to be decided upon when using multilateral methods Practical choice: 13 months (agrees with CPI choices) Weighting of products: Equal weights Different weights: (1) varying over months, (2) constant in time 22

23 Choices concerning index compilation (Price) reference month: Fixed base month Monthly shifting Types of time window: Monthly expanding (in combination with a fixed base month) Fixed length, rolling window Calculation method for current month: Direct index w.r.t. fixed base month Month on month chaining, window splicing 23

24 Examples of index methods 24

25 Some remarks on methods Transitivity/chain drift: Bilateral methods are not transitive for dynamic assortments Price imputation may resolve this for some methods (e.g. Jevons) Imputations are not needed with multilateral methods Multilateral methods are free of chain drift when direct indices are calculated with fixed base window updating methods GEKS and CCDI are sensitive to downward biases when products leave under clearance prices (Chessa et al., 2017) 25

26 Covered in Part 3 (case study) Weighted vs unweighted (equal weighting) methods Bilateral vs Multilateral methods (QU) 26

27 Part 3: Comparison of price indices of clothing for WS and SC data of a Dutch webshop 27

28 Potential pitfalls when using WS data When WS-SC price ratios exhibit trends over time product price changes diverge between WS and SC data Representativity of product weights: No or weak relation between proxies and transaction data Different trends over time in SC and WS weights Persistence of items on website when these are no longer sold risk of downward bias under clearance prices 28

29 Case study: Scope and choices Scope: Clothing and footwear sold by Dutch webshop Coicops: Menswear and Ladies wear 8 product categories within both coicops Product definition within categories: By brand and type (most detailed item group level) Example: A bermuda of Miss Etam Index calculation: Product categories: QU-method (version: GK = Geary-Khamis) Coicop level: Laspeyres type with annual weights 29

30 Input for index methods Web scraped data: Prices as offered on website But what about the quantities? We need proxies, since we cannot scrape expenditures from websites Option 1: Number of web scraped prices, without deduplication Option 2: Number of web scraped prices, after deduplication Weight proxies for categories: Price x Quantity 30

31 QU-method: General form The method extends the unit value concept to nonhomogeneous items Quantities q i,t for item i in month t are transformed into common units : Quality adjusted Unit value in month t: Price index: 31

32 QU-method: Variants Geary-Khamis (GK): a-lehr: As GK, but without price index (deflator) IDB-method: = expenditure share of item i in month t 32

33 QU-indices: Menswear categories 33

34 QU-indices: Ladies wear categories 34

35 Price indices for coicops 35

36 WS: Replacing quantities by 0 or 1 36

37 WS: Monthly chained bilateral indices 37

38 WS: Direct bilateral indices 38

39 Summary of results QU-method (WS vs SC): Price indices are comparable The structurally higher WS prices do not affect index comparisons The comparisons at category and coicop level indicate that the weight proxies based on WS data work very well Deduplication of web scraped prices should be discouraged Bilateral indices: The monthly chained indices collapse Direct indices perform much better 39

40 Points of attention The results are valid for 1 retailer (a web only shop) Why are correlations between SC and WS quantities high? We cannot simply extrapolate the results to other retailers (e.g., retailer with webshop + physical outlets) Approach/strategy to scraping matters 40

41 Some suggestions for future research When having both WS and SC data: More comparative studies Combine both data sources, enrich SC data with WS metadata Invest more in statistical analyses: Can we identify typical patterns in SC data? E.g., how prices and quantities relate/behave over time? Do WS data exhibit such patterns? Findings are useful for countries that do not have SC data 41

42 References (1) Auer, L. von (2014). The generalized unit value index family. Review of Income and Wealth, 60, Chessa A.G. (2013). Comparing scanner data and survey data for measuring price change of drugstore articles. Paper presented at the Workshop on Scanner Data for HICP, September 2013, Lisbon. Chessa, A.G. (2016). A new methodology for processing scanner data in the Dutch CPI. Review on National Accounts and Macroeconomic Indicators, issue 1/2016, Chessa, A.G., Verburg, J., and Willenborg, L. (2017). A comparison of price index methods for scanner data. Paper presented at the 15 th Meeting of the Ottawa Group on Price Indices, Eltville am Rhein, Germany, May Paper available at: /2017_05_10_ottawa_group_07_1_paper.html? blob=publicationfile de Haan, J., Willenborg, L., and Chessa, A.G. (2016). An overview of price index methods for scanner data. Paper presented at the UNECE-ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, 2-4 May

43 References (2) ILO/IMF/OECD/UNECE//The World Bank (2004). Consumer Price Index Manual: Theory and Practice. ILO Publications, Geneva. Krsinich, F. (2014). The FEWS Index: Fixed Effects with a Window Splice Nonrevisable quality-adjusted price indexes with no characteristic information. Paper presented at the UNECE-ILO Meeting of the group of experts on consumer price indices, Geneva, Switzerland, May

44 Thank you! Questions? 44