Importance of data Jay Daley, ICANN Helsinki 2016
Why is.nz presenting? Small registry - Big data pioneer 22 server Hadoop cluster installed in 2012 Full packet capture of DNS servers Data science oriented research team 18% of total staff World class registrar data portal Main business priority for new development Actively building data products Recent success with broadbandmap.nz Open data portal for.nz data 2
Case study https://idp.nz 3
Why data matters Evidence based policy Organisational/community development Cleaner and safer DNS Business - more, new, better Public trust 4
Evidence based policy 5
Current use of data in policy Already some excellent examples RSTEP IANA SLE approximations by Marc Blanchet But data is not public and so No reproducibility Agenda set by those who pay the analysts Very low throughput of research Some people have their own data Gives them real power in the debate e.g. Root server operators and WPAD debate data is the new oil Open Data means Open Debate 6
Case study hoarding? 1000000 Number of registrants by portfolio size Number of registrants 100000 10000 1000 100 10 1 1 2 3 4 5 6 7 8 9 10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 101-200 201-300 301-400 401-500 501-600 601-700 801-900 901-1000 1001-2000 2001-3000 Portfolio size 7
Organisational development 8
Modern business principle Openness and transparency Applied to Diversity, remuneration, expenses, etc Significant organisational benefits Reinforces best behaviour Create culture of community/customer audit 9
Case study Travel funding ICANN funds travel for some attendees Until recently this data was hard to use Only published in PDFs Some meetings missing Names spelled inconsistently No summaries or multi-meeting analysis At Dublin meeting spent several hours Extracting data from PDFs Tidying up names Creating reports Publishing data 10
Example PDF 11
Example output Average travel funds received by Number of meetings attended $14,000 $12,000 $10,000 $8,000 $6,000 $4,000 $2,000 $- 0 2 4 6 8 10 12 14 16 18 20 12
Example publication 13
Case study Diversity Report from AFNIC on ICANN Diversity Based on public data but not easy! 14
Example output 15
Cleaner and safer DNS 16
Very active area Highly developed use of data Multiple research teams, cooperative forums, NFP services and commercial providers Multiple data resources, extensive data sharing Tools: Entrada, Turing, hadoop-pcap, zonemaster Data collection source evidence Passive monitoring (e.g. DNSDB, PassiveTotal) Hand produced by threat researchers Strong sharing culture via data feeds 40+ feeds available (both NFP and commercial) Track domains, IPs, credentials, URLs, etc 17
Case study threat sharing Data shared with cooperative forum then shared with registrars 18
Business more, new, better 19
Data driven business Market intelligence Targeting marketing New products, same customers New products, new customers 20
Market intelligence - basic Registrants prefer shorter names right? 50000 Number of domain names by number of characters 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 21
Market intelligence - advanced Domain name categorisation by industry S Other Services R Arts and Recreation Services Q Health Care and Social Assistance P Education and Training O Public Administration and Safety N Administrative and Support Services M Professional, Scientific and Technical Services L Rental, Hiring and Real Estate Services K Finance and insurance J Information Media and Telecommunications I Transport and storage H Accommodation, Food Services G Retail trade F Wholesale trade E Construction D Electricity, gas and water supply C Manufacturing B Mining A Agriculture, forestry, fishing and hunting 0% 5% 10% 15% 20% 25% 30% 35% Registry Registrar 22
Case study - registrar portal 23
Targeted marketing Domains most likely to renew for 10 years Those in top 20% by observed traffic Domains in danger of cancelling No MX record TLD cross-sell opportunities Simple data matching Industry verticals Machine learning classifier using web site text Expiring domains Valued by algorithm 24
Case study expiring domains 25
New products, same customers Domain name popularity 26
New products, new customers SaaS product market sizing Specific DNS record indicators for each product Counted by regular zone scans Data sold for competitor analysis 27
Public trust Add Presentation Name June 26, 2016 28
Open data and public trust Sunlight is the best disinfectant Open data can prove: Effectiveness of markets and regulators Social benefit Where problems remain 29
Case study Web Index 30
Case Study - CCT Competition, Consumer Trust and Consumer Choice (CCT) Metrics Reporting So near but yet so far 31
Case study - ISOC 204 datasets only one from ICANN (NRO)! 32
Taking this to the next level 33
Toes in the water ICANN F17 Strategic Plan includes: Deploy automated systems to collect data and compute ratio of registered domain names to active IP addresses. Deploy automated systems to collect data and compute ratio of registered domain names to Internet users regionally and globally. Publish analyses of data collected, implications of changes in data over time, and other topics relevant to the use of unique identifiers and evolution of identifier technologies Document growth in ratios in developing regions 34
A goal to consider Data enabled community & industry Data openly shared by all market participants Data is easy to find and use and can be trusted Strong community use of data Authoritative publications ICANN Industry Report Domain Names Contribution to Society Index 35
Steps to make it happen Commit employ a Chief Data Officer Begin cultural change Broaden the principle of openness to include data Set the vision of the benefits Engage community social license Community expectations of openness vs privacy Put legal framework in place Adjust contracts, policies and processes to support open data Determine privacy protection rules 36
Thanks Any questions? Contact: jay@nzrs.net.nz www.nzrs.net.nz