A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic

Transcription

1 A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic A collaborative effort with many thanks to: Dr. Augustine Fou Anti-Fraud Forensic Scientist Marketing Science Consulting Group Andy Hibbler VP of Brand Advertising Whitepages Rami Essaid CEO and Co-Founder Distil Networks

2 Table of Contents EXECUTIVE SUMMARY 3 BACKGROUND AND TERMINOLOGY 3 THE STATE OF DIGITAL AD FRAUD 4 Landscape and Trends 4 Key Issues for Publishers and Advertisers 4 Publisher and Advertiser Perspectives on Bot Traffic 5 Types of Impression and Click Fraud 7 Impression fraud 7 Click fraud 7 Retargeting 8 Diverting ad dollars 8 Covering their tracks 9 There s a whole world of bad bots out there 9 BEST PRACTICES AND ANTI-FRAUD PLATFORMS 9 THE REVERSE PROXY APPROACH TO PREVENTING DIGITAL AD FRAUD 11 Overview of Pixel with Ad Approach 11 Overview of Distil Networks Reverse Proxy Approach 12 Key Differences Between Single Pixel with Ad Approach and Distil Networks Reverse Proxy 12 CASE STUDY: WHITEPAGES PREMIUM AD PROGRAM 13 CONCLUSION 14 COLLABORATORS 15 About Dr. Augustine Fou 15 About Whitepages 15 About Distil Networks 15

3 EXECUTIVE SUMMARY There s no doubt that the digital advertising marketplace both supply-side and demand-side is under sustained attack from increasingly sophisticated automated robot programs (bots) designed to divert, steal, and otherwise defraud to the tune of billions of dollars every year. Measuring and mitigating non-human traffic is more important than ever. In a survey conducted for this report, 37% of buyers are ready to pay a premium of 11% or more for certified non-human traffic, yet 25% of publishers have no way to measure non-human traffic. This paper explores the current landscape as it applies to digital publishers and advertisers, examines currently available approaches to combatting the problem, and highlights one publisher s approach, Whitepages, to validating premium ad inventory. The paper will be of greatest value to premium ad publishers and their clients, but will serve as a useful overview for anyone concerned with the validation of ad-related traffic. BACKGROUND AND TERMINOLOGY In any broad-based initiative, it s important to establish a common language, so the following is a quick primer, developed by the Internet Advertising Bureau (IAB), on the different types of fraudulent traffic discussed in this paper; a comprehensive list with definitions is available on the Trustworthy Accountability Group website (TAG). 1 Illegitimate and Non-Human Traffic Sources Hijacked device any device (browser, phone, app, or other system) modified to call html or make ad requests outside of the user s control and without the user s consent. Crawler masquerading as a legitimate user a browser, server or app that makes page load calls automatically without declaring their true (robot) identity. Data-center traffic traffic originating from servers in ISPs, hosting providers, and other third-party data centers, where the ad is not rendered in a user s device because there is no real human user. 1 The Trustworthy Accountability Group (TAG), a cross-industry initiative established by the American Association of Advertising Agencies (4As), Association of National Advertisers (ANA), and the Interactive Advertising Bureau (IAB), is taking center stage in the fight against digital advertising fraud. TAG s efforts are supported by the Media Rating Council s viewability standards, adopted and aligned with TAG s Inventory Quality Guidelines, and are focused on eliminating fraudulent digital advertising traffic; combating malware; fighting ad-supported Internet piracy to promote brand integrity; and promoting brand safety through greater transparency. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 3

4 Non-Traditional Sources of Traffic on Ads AdWare traffic a device where a user is present and additional html or ad calls are made by the AdWare independently of the content being requested by the user. Proxy traffic traffic that is routed through an intermediary proxy device or network where the ad is rendered in a user s device where there is a real human user. Non-browser User-Agent header a device that declares a User-Agent header not normally associated with human activity. Browser pre-rendering a device that makes html or ad calls prior to the rendering of the resulting assets or web-page to an end user. THE STATE OF DIGITAL AD FRAUD Landscape and Trends According to IAB s recent Internet Advertising Revenue Report, Internet ad revenues in the United States reached $27.5 billion in the first half of At the same time, Dr. Augustine Fou, anti-fraud forensic scientist, and the founder of Marketing Science Consulting Group, posits that as ad spend continues its healthy trajectory towards an estimated $56 billion by the end of 2015, fraud is tracking closely behind. The important data point to note here is the dramatic ballooning of the high/low estimate for fraud since 2010, which reflects the rapid growth of programmatic ad buying and selling; more automation equals more opportunity for digital fraud. However, the gaping difference between the high and low estimates underscores the difficulty in accurately measuring that fraud. Fraud goes up as digital ad spend continues upward Source: Marketing Science Consulting Group Key Issues for Publishers and Advertisers Of that $56 billion in circulation in the digital advertising world, Dr. Fou estimates that for every $3 spent on digital advertising, $1 is being siphoned out of the advertising ecosystem into the pockets of the bad guys. As a result, publishers ad revenues are depleted by one third and advertisers have a lower return on investment, thanks to the fraudulent activities discussed earlier. Fraud siphons 1/3 of dollars out of ad ecosystem Source: Marketing Science Consulting Group WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 4

5 This siphoning-off impacts publishers and advertisers in different ways: Advertisers spend ad dollars in the expectation that those dollars will buy human eyeballs, and attempt to increase that reach and frequency by buying access to more humans. But those ads are reaching fewer humans every day and ads shown to bots will never generate business for the advertiser. Publishers, meanwhile, are attempting to increase their revenues by increasing the number of ads served or the CPM they charge. But when bots come along instead of humans, ad serving resources are drained, and traffic and ad revenue are diverted to the bad guys sites. That means not only loss of ad revenue but damaged reputations. The end result is a breach of the implicit trust relationship between advertisers and publishers. Publisher and Advertiser Perspectives on Bot Traffic As part of a joint webinar with the IAB, Distil Networks conducted a survey in order to gain a better understanding of publisher and advertiser concerns about non-human traffic (bots) and digital ad fraud. The survey was conducted over and online in July and August, 2015 and included 138 respondents, each of which answered two or more questions. 40 to 48% of respondents weren t sure how bot traffic affects their online campaigns and web properties, yet almost no one denies the issue. There also appears to be a clear disconnect between buyers and suppliers perceptions: 37% of buyers think bot traffic has more than an 11% bad affect on their campaigns compared to only 14% of suppliers thinking that bots have more than an 11% bad effect. Perhaps there s a need for suppliers to provide more data-based reassurance to their clients. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 5

6 Respondents were also asked what monetary value they would place on certified human traffic. Two trends are apparent here: There is a clear willingness for buyers to pay some level of premium for guaranteed human traffic, with 37% of buyers willing to pay at least an 11% premium. Close to 40% of both buyers and suppliers were unwilling or unable to express an opinion, which supports the need to be able to measure non-human traffic. In an effort to drill further into current threat perceptions, the survey then asked respondents to identify which types of bot traffic problems were important to their business from the five most common types: Content theft where web scrapers are simply taking down content from sites. Lead fraud and fake registrations form spam and bot-driven form filling. Login attacks large numbers of failed username and password entries. Click & impression fraud bots loading and clicking on links. Skewed analytics difficulty of separating human traffic from bot traffic in order to optimize for human traffic. Here, responses from buyers and sellers show similar levels of threat-specific concerns with click and impression fraud as the clear winner across the board. About half of all respondents felt that skewed analytics was an important problem. Nearly a third of all respondents felt that bot-driven lead fraud and fake registrations was an important problem. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 6

7 Finally, we asked respondents which categories of non-human traffic they are able to measure today. The extent of the problem is clear in those 50% numbers on both sides of the chart if you can t measure a threat, it s impossible to mitigate it. It s also more than a little disturbing to note that 25% of the publishers in the audience were unsure as to what nonhuman traffic they could measure. Types of Impression and Click Fraud Click and impression fraud are the number one concern for most buyers and sellers for good reason these two ad types represent 86% of the digital ad spend. Click and impression fraud both involve the bad guys putting up fake websites, but the process is quite different otherwise. Impressions and clicks remain the biggest targets Impression fraud Impression fraudsters put up fake websites and load a huge number of ads on to them - display ads, rich media, video ads, anything will do (including mobile display ads, which are also bought on a CPM basis). Bots are then used to repeatedly load the pages to generate fake ad impressions, disguising the origins of those ads to avoid detection. Click fraud Click fraud is more sophisticated in that it takes two steps. Fake sites are loaded up with search ads so they re part of search ad networks. Then the fraudsters have bots type in the keywords (usually very expensive ones) to cause the ads to load. When those ads are served up by the network, bots click on the ads to generate the CPC revenue for that site. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 7

8 Retargeting The bad guys own and control millions of these sites, using bots to earn revenue from CPM and CPC. A new and more sophisticated approach known as "retargeting fraud" sends bots to legitimate publishers' websites for the purpose of collecting a cookie. If the bot is targeting, for example, an outdoor site like REI.com, it will visit a number of sites related to outdoor activities. This builds up a cookie profile about that user that appears attractive to advertisers targeting outdoor enthusiasts. When the cookie collector bot visits a site that's controlled by the bad guys, the advertisers, through retargeting, try to show an ad to that user, and pay a premium CPM to do so. Diverting ad dollars Click and impression fraud also mess with analytics. When bots go to premium websites to generate pageviews without clicks, the CTR for the entire site is lowered. The result is that optimizers divert their ad dollars to sites with higher CTRs, operated by the bad guys. The same thing happens with viewability: when the bad guys stack dozens of ads behind each other so that they are all viewable, those sites attract optimizer dollars. Then there s pixel stuffing, which enables many ads to be hidden on a single page. Another growth industry for the bad guys, thanks to programmatic ad buying. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 8

9 Covering their tracks You might think it would be easy to trace these bad guys by looking at the source of those ad impressions and blacklisting those domains (more on this later), but the bad guys are of course ahead of the game employing a variety of techniques to hide their fraudulent activity. Domain/IP spoofing - makes it appear the impression came from a legitimate site. Impression laundering - masks the original URL by layering it through other URLs. Stacked redirects - replace the actual referrer URL with a faked referrer. Passing faked variables - tricks analytics platforms into believing that the impression came from a legitimate site. There s a whole world of bad bots out there Ad fraud is lucrative: when the bad guys send their bots to websites, they're able to load pages, therefore causing ads to load, so they're generating fake ad impressions by the billions, and selling them through the ad exchanges and ad networks. Some are now so sophisticated that they can fake human attributes like mouse movements and pausing, making them extremely difficult to identify as bots. According to Distil Networks 2015 Bad Bot Landscape Report, 41% of bad bots now mimic human behavior. BEST PRACTICES AND ANTI-FRAUD PLATFORMS Advertisers need to continue to be more aggressive in their buying: Focus less on raw numbers and more on observable actions. Consider incentivizing your agency based on actions taken by users, avoiding the temptation of buying lower cost (and quality) inventory. Move beyond viewability. As we ve seen, viewability can be faked by bots and other non-human traffic. And, even if we resolved that part, we d still be left with impressions being loaded by bots. Allocate budget to quality, not quantity. When you allocate money away from long-tail sites in ad exchanges, where fraud is more prevalent, you free that money up for buying on mainstream, premium sites. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 9

10 Demand full transparency. If the suppliers you re buying from can t show you where each ad impression was served, don t shop there. Put your own checks and balances in place and be prepared to challenge your suppliers. On the publisher's side, if you can measure the level and nature of bot activity that s impacting your inventory, you ll go a long way towards engendering trust as well as be able to charge a premium for your ad inventory. As shown in the table below, publishers have a number of tools and techniques at their disposal to support their efforts. Comparison Matrix of Anti-Fraud Solutions Approach Example Vendor Description Mitigation On-page White Ops JavaScript code installed on website (like analytics). Detects parameters about the user and computing environment In-ad exchange Integral Ad Science Real-time decisioning in the bid-stream using rules and blacklists (e.g., if a site is on a blacklist, don t serve the ad) Challenge based Are You a Human Using captchas and other challenges which ask humans to verify themselves as humans Reverse proxy Distil Networks Inspects each page request to determine if it s a bot or not, then passes along human traffic. Bot traffic is blocked before the ad call and pixel fires Blacklists domains that send traffic mostly to bots Eject users and reject sites in real-time based on blacklists Identify and whitelist humans for use in real-time bidding decisioning Blocks bad bot traffic, fingerprints each client, builds real-time whitelists and blacklists Anti-fraud vendors typically take one of four approaches. The on-page approach takes the form of JavaScript installed on the website, similar to Google Analytics, which characterizes visitors that come to the site. Other approaches work in the ad exchange so that, when a bid request comes in, the site on which the ad is about to be served is analyzed, along with characteristics about the user or cookie. If fraudulent activity is detected, the ad isn t served. Then there is the challenge-based approach, such as CAPTCHAs, where the user is asked to prove they are human by solving the challenge. And finally, the reverse proxy approach is a way to detect, measure and block fraudulent traffic before it hits the website. This last approach is discussed in more detail below as a preamble to how Whitepages used the reverse proxy approach in creating their premium ad inventory program. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 10

11 THE REVERSE PROXY APPROACH TO PREVENTING DIGITAL AD FRAUD Many anti-fraud platforms in the market today use a single pixel embedded in an ad in order to identify and track unsolicited non-human traffic. The heritage of the single pixel approach comes from ad serving technology and it s a powerful way to track and measure digital ad fraud. Vendors in this space include White Ops, Integral Ad Science, Forensiq and Double Verify. The heritage of the reverse proxy approach comes from IT security and is a powerful way to stop nonhuman traffic before it interacts with a website s origin servers. That is to say, a reverse proxy approach seeks to answer the question is this a real human attempting to access my website? A reverse proxy enables the incorporation of other techniques, such as turing tests (e.g., CAPTCHAs), while maintaining control and visibility into headers. Below is an analysis of each approach using Distil Networks as the exemplar reverse proxy vendor. 2 Single Pixel with Ad Technology: pixel for tracking Single JavaScript injection Predictable JavaScript every time Probabilistic Reactive, post-ad call User agent + IP = fingerprint IP/network intelligence monitoring Reverse Proxy Technology: sits inline, inspecting each http request Multiple injections Randomized, Spoof Proof Technical tests on browser Proactive, pre-ad call Technical tests on browser Technical tests on browser Overview of Pixel with Ad Approach The pixel with ad approach uses the same methodology to target prospective customers to now target, or capture signals, from prospective bad bots. The most popular fraud signals in advertising and ad networks rely on the use of a single pixel in an ad call that fires JavaScript when the ad loads in the browser. The GET from the pixel in the ad call sets up a JavaScript that executes one time. The JavaScript posts data back to the pixel provider and then the fraud probability is calculated based on an analysis of the IP, a fingerprint, and patterns of requests from the paired user agent and IP address. The decision about what is labeled fraud relies on JavaScript executing, IP reputation, and an analysis of traffic patterns. Fraud signals are then provided to the ad network and the advertiser. 2 Distil Networks provided Marketing Sciences Consulting Group with a free test account for this analysis WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 11

12 Overview of Distil Networks Reverse Proxy Approach Distil sits inline with web traffic and inspects each http request in real time, determines if it s a bot or not, then passes the request to origin. Distil analyzes 40+ criterion from each client request then builds a fingerprint unique to the browser making the connection. Fingerprints are sticky to the bot even if it attempts to reconnect from random IP addresses or hide behind an anonymous proxy. Distil interrogates the browser to ensure the browser is indeed who it claims to be. Challenges to the browser are inserted and the responses reviewed dynamically to prevent pre-emptive spoofing. Distil uses behavioral analytics and machine learning to minimize false positives and optimize protection for each domain. Being inline, Distil can periodically intercept suspicious traffic that does not fit with the site s unique traffic patterns and challenge it with a hardened CAPTCHA, and then feed the response data back into its machine learning algorithms. Distil gathers attack information across customers and distributes it back out to all Distil-protected sites. Key Differences Between Single Pixel with Ad Approach and Distil Networks Reverse Proxy The heritage of the single pixel with ad approach translates into strong end-user fraud reporting and auditing capabilities. However, in analyzing Distil Networks technology, three key advantages rose to the fore. Accurate. Multiple injections with a reverse proxy service should provide a more accurate and detailed fingerprint versus a single JavaScript injection. A single pixel placed in creative executes one GET, so it can t interactively inspect the client s browser and hardware. For Distil s JavaScript tests, there are multiple injections and these are randomized across the page ensuring an unpredictable experience for the bad guys and their bots. Proactive. A pixel is a reactive signal while Distil is a proactive mechanism. A pixel will signal a suspicious request and provides advanced reporting which can be used to audit an advertising program or adjust ad rates after the fact. Distil s reverse proxy takes action on the fraudsters before a page loads. This approach has other advantages in that it protects against random spikes in bot traffic, skewed analytics, and other online threats such as web scraping, spam, transaction fraud, and brute force attacks. Domain Specific. The single pixel approach relies on data across a vast network of sites yet there is little or no domain-specific behavioral modeling or heuristics based on requested URLs, click path, or speed of navigation. Distil leverages domain-specific human behavior and machine learning to identify dangerous anomalies. This is not possible with a single pixel firing from creative. Publishers may benefit by using a layered approach to stopping digital ad fraud in which they combine the pixel in ad approach with a reverse proxy service like Distil Networks. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 12

13 CASE STUDY: WHITEPAGES PREMIUM AD PROGRAM In the current marketplace brand trust is more important than ever. Working with Distil has helped Whitepages ensure that we are delivering human traffic to our consumers and advertisers. Our partners are thrilled to know that we are eliminating non-human traffic and providing stellar viewability rates." -Andy Hibbler, VP Brand Advertising & Partnerships Andy Hibbler leads the Whitepages sales team and has been using Distil Networks solution across the company s three platforms for over a year. According to Hibbler, Distil validates and selectively blocks fraudulent traffic and works within a framework similar to a wedding cake with layers or tiers. On the bottom tier, everything is coming into the funnel for identification from Whitepages, caller IDs, and other sources. Distil Networks kicks in at the next tier to ensure that non-human traffic is blocked outright. Then the company deploys other validation tools to audit advertising inventory and measure viewability. The result is the top tier where an impression is served to an engaged consumer. According to Hibbler, we feel that this tiered approach helps us deliver on the commitment that our advertisers and customers expect from us, which is to serve an honest impression. In a typical month, Distil identifies and blocks about 50 million bad bot requests on Whitepages primary domain. To put this in perspective, they get about 180 million human requests during that same time period. Hibbler feels that one of the main benefits of the Distil solution is its accuracy. Known violators are blocked outright and suspicious requests are served a hardened CAPTCHA. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 13

14 The report below shows that over a 30-day period, Distil served 32 million CAPTCHAs but only 138K were solved or had a failed attempt. That means that Distil is serving CAPTCHAs to bots over 99.5% of the time. Whitepages reported the following results with the Distil Networks solution: Stopped comment spam and prevented scrapers from stealing data Reduced time spent detecting bad bots and eliminated false positives Allowed servers to run and load impressions at optimal speeds Created a reliable and safe environment for brands (<5% IVT) Hibbler has reported that third-party ratings companies like comscore are today giving Whitepages the lowest non-human traffic rating possible with less than 5% being questionable. According to Hibbler, Tools like Integral Ad Science are great at measurement and tracking. What we like about Distil is their ability to block the bots before they hit our site. That's why we have been able to maintain such a high human viewability score with Comscore. CONCLUSION By combining the resources of the publishing and advertising ecosystem, compliance and standards bodies, and technology innovators, the industry is poised to deliver on the promise that every advertiser gets what they pay for when they purchase digital advertising. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 14

15 COLLABORATORS About Dr. Augustine Fou Dr. Augustine Fou is an Anti-Fraud Forensic Scientist, and the founder of Marketing Science Consulting Group. In this role he advises senior executives and global brands on digital marketing strategy. Dr. Fou teaches digital and integrated marketing at Rutgers University and NYU. Dr. Fou completed his PhD at MIT in Materials Science and Engineering at the age of 23. He started his career with McKinsey & Company and previously served as SVP, digital strategy lead, McCann/MRM Worldwide and Group Chief Digital Officer of Omnicom's Healthcare Consultancy Group (HCG). About Whitepages Whitepages is the leading source for contact information in North America and is at the forefront of developing services that help people find and verify personal and business identities in the digital world. With a top 40 Website, 55 million monthly visitors and several million downloads for their suite of mobile apps, Whitepages offers access to more than 250 million personal identities. Whitepages uses its extensive data insights to monitor phone-based threats for consumers and businesses, detecting more than 1 billion robo, telemarketing and spam calls and texts to date. Whitepages also operates Whitepages Pro, a Web and API-based service which helps businesses verify the identities of their customers. To learn more, visit About Distil Networks Distil Networks, the global leader in bot detection and mitigation, is the first easy and accurate way to identify and police malicious website traffic, blocking 99.9% of bad bots without impacting legitimate users. Distil protects against web scraping, brute force attacks, competitive data mining, account hijacking, unauthorized vulnerability scans, spam, man-inthe-middle attacks and click fraud. Slash the high tax that bots place on your internal teams and web infrastructure by outsourcing the problem to the team with a maniacal focus on blocking malicious bots. For more information on Distil Networks, visit us at or on twitter. WHITE PAPER A Digital Publisher s Guide to Measuring and Mitigating Non-Human Traffic 15