Forget Big Data - Measure What Matters

Forget Big Data - Measure What Matters Katie Delahaye Paine CEO Paine Publishing www.painepublishing.com @queenofmetrics measurementqueen@gmail.com

Katie Delahaye Paine: Helping communications professionals define and measure success for 25 years. Founder of: The Delahaye Group KDPaine & Partners Paine Publishing Author of: Measuring the Networked Non-Profit Measure What Matters Measuring Public Relationships measurementqueen@gmail.com Paine Publishing: Providing communications professionals the knowledge and information they need to navigate the journey to good measurement Newsletters Training Courses Consulting painepublishing.com

Don t Measure What s Easy Cartoon by Rob Cottingham 3

Measure What Matters Cartoon by Rob Cottingham News Group International 10/11/2014 4

Welcome to the, I want what I want, not what you send me, era aka: The Relationship Era

It s the end of the world as we knew it 1,000,000 The average audience for a MyDrunkKitchen video (http://www.youtube.com/watch?v=usxqnred3is 179,000 CNN average nightly audience 27 90% The number of times per hour Digital Natives switch media every 2.2 minutes. The percent of conversation that happens OFF LINE 10% The percent of on-line conversations that are public < 5% 40% The percent of Facebook & Twitter posts that are actually seen The amount of conversations generated by bots, spammers and payper-click sites

We need a new Attribution Model Digital/On -line Media Buy Print/TV Media Buy Other Paid Marketing Media Relations/ Social Media/PR Digital/ Online Media Buy Print/TV Media Buy Media Relations /Social Media/PR ROI ROI

What s Changed? Collapse of mass media Growth of media everywhere Intolerance for messaging The Barcelona Principles Viewers are more likely to stop watching commercials at the moment in which brand logos appear on the screen - ARF Study 8

The Barcelona Principles, The Conclave & Industry Standards 1. Importance of Goal Setting and Measurement 2. Measuring the Effect on Outcomes is Preferred to Measuring Outputs 3. The Effect on Business Results Can and Should Be Measured Where Possible 4. Media Measurement Requires Quantity and Quality 5. Earned Media Value/AVEs are not the value of Public Relations 6. Social Media Can and Should be Measured 7. Transparency and Replicability are Paramount to Sound Measurement The Conclave All standards are available on http://www.painepublishing.com/st andards-central

Social Media Measurement Standards have been published for: 1 2 3 4 5 6 Content Sourcing & Methods Reach and Impressions Engagement Influence & Relevance Opinion & Advocacy Impact & Value 11/10/2014

Impressions are not awareness. Where s the So What? MSM Online Social Media Eyeball counting HITS Outcomes 11

Likes Are Not Engagement Impressions Likes Followers Trial/Consideration Commitment Advocacy 12 11/10/2014

The Engagement Process Listen Donate Purchase Achieve Goal Run

Measuring the Engagement Process Web Analytics CRM/Convio ecommerce Map My Run/Social Sharing icloud & Google+

#3: Standards for Engagement Engagement is an action that happens in response to content i.e. when someone engages with you. (not about you ) Engagement manifests differently by channel, but is typically measurable based on effort required and how it is shared with others. Engagement can be desirable or undesirable Any measure of engagement must be tied to the goals and objectives for your program Engagement occurs both off and online, and both must be considered if you intend to integrate your metrics with other marketing or communications efforts. Engagement includes such actions as likes, +1, shares, votes, comments, links, links, retweets, Facebook's Talking about you etc. Engagement actions should be counted by % of the audience that responds.

GWA s, Photos Drive Shares & Likes David Beckham & Dierks Bentley Video Share & Likes Over Time Average of PostLikes Average of PShares Mia Farrow 968 1027 914 510 188 222 234 208 41 297 261 269 215 222 145 71 50 36 25 56 30 328 204 71 45 107

#4: Influence & Relevance Influence happens when you are persuaded to change behavior or opinion that would otherwise not have changed. Influence can be attributed to either an outlet or an individual. Influence happens online and off line and both should be considered Should include some combination of the following five elements: Reach Engagement around individual Relevance to topic Frequency of posts around the topic Audience impact as measured by the ability to get the target audience to change behavior or opinion. If an individual scores a 0 on one element, they have no influence Influence cannot be expressed in a single score or algorithm

Big Numbers Influence All influence is relative Who is influential depends on your audience A computer cannot tell you who matters most To be influential requires relevance, frequency & reach

#5 Standards for Measuring Tone/Sentiment Whatever process is defined and applied, it must be used consistently throughout any analysis. Sentiment reliability varies by vendor and approach therefore coding definitions, consistency and transparency are critical. Opinions, recommendations and other qualitative measures are typically more valuable than raw sentiment and increasingly measurable: Opinions ( it s a good product ) Recommendations ( try it or avoid it ) Feeling/Emotions ( That product makes me feel happy ) Intended action ( I m going to buy that product tomorrow )

Testing the accuracy of coding % Agreement with human coding % positive %negative %neutral NetBase 88.64% UberVu 11.95% 9.46% 78.00% UberVu 68.31% NetBase 7.84% 0.32% 92.11% SDL 48.94% Beyond 58.00% 13.00% 30.00% 0.00% 20.00% 40.00% 60.00% 80.00% 100.00% SDL UberVu NetBase SDL 33.12% 26.53% 40.35%

Everyone agrees on Neutral Not so with Negative and Positive Degree to which agreement occurred No Yes Positive 72.36% 27.64% Neutral 14.82% 85.18% Negative 62.32% 37.68% Mixed 100.00% 0.00%

#6: Impact & Value Impact and value will always be dependent on client objectives Need to define outcomes in advance will likely span multiple business goals, especially for social (crosses disciplines) ROI should be strictly limited to measurable financial impact; total value can be used for financial and non-financial impact combination Value can be calculated in positive returns (sales, reputation, etc.) or avoided negative returns (risk mitigated, costs avoided) Key performance indicators and balanced scorecards are helpful to connect social media impact to business results/language

ROI ROI is: Cost savings Greater efficiency Lower legal costs Faster time to implementation Lower cost per impact vs. other marketing channels Lower cost per message exposure vs. other channels ROI is not: Impressions Earned media value or AVE Number of shares, views or likes

Social Media isn t media, it s your business Conversations PR//Mkt Prod. Mktg Product Support Mktg Research R&D CI IR HR Sales Savings, shorter cycles, more renewals, better ideas, $$$$$$$$$$$$$$$$$$$$$$$$$$ 11/10/2014 24

Good Relationships Are More Cost Effective Type I love Zappos into Google, and you find 1.19 million references Type Citibank and you get 21,000 references. Citibank spends 100 times more a year on advertising than Zappos. Cost per delegate acquired: Obama: $6,024 Clinton: $147,058 Romney: $2,389,464

So What? It s not about the media, it s about your business and your customer It s not how loud you re shouting, it s what your stakeholders do after they ve heard your message and your relationships. There are no boundaries It s not about how big your data is, it s about how you use it

6 steps to standards-compliant measurement Step 1: Define your goal(s). What outcomes is this strategy or tactic going to achieve? What are your measurable objectives? Step 2: Define the parameters Who are you are trying to reach? How do your efforts connect with those audiences to achieve the goal. Step 3: Define your benchmarks. Who or what are you going to compare your results to? Step 4: Define your metrics. What are the indicators to judge your progress? Step 5: Select your data collection tool(s). Step 6: Analyze your data. Turn it into action, measure again 1 2 Six Steps to Success 3 4 5 6 27

Definitions Monitoring process by which data are systematically and regularly collected about a program over time. Measurement a way of giving an activity a precise dimension, generally by comparison to some standard; usually done in a quantifiable or numerical manner; see also: data, scale Metrics an agreed upon number or percentage that indicates progress towards a goal Evaluation a form of research that determines the relative effectiveness of a public relations campaign or program by measuring program* *IPR Dictionary of Measurement Terms, Dr. Donald Stacks www..instituteforpr.org 28

Why do we communicate? Activities How does what you do contribute to the bottom line? Outtakes (Intermediary Effects) Awareness Knowledge/Education Understanding Outcomes (Target Audience Action) Revenue Leads Engagement Advocacy 29 11/10/2014

Step 1: Define the goals What return is expected? Define in terms of the mission. Define your champagne moment. If you are celebrating complete 100% success a year from now, what is different about the organization? 30

Goals & Suggested Metrics Goals Goal 1: Meet Sales Targets Goal 2: Reduce Risk/Threats Goal 3: Increase market share in new market Communications Contribution Expand the marketable universe Reduce sales cycle Increase Trust Increase advocacy Expand the marketable universe Metrics % Increase in desirable share of voice % in awareness % increase in undesirable voice % increase in trust scores % increase in desirable voice in new market % increase in awareness of brand in new market

Step 2: Understand the parameters. Goal: Get the cat to stop howling Strategy: Buy cat food What management s priorities? Who are you are trying to reach? How do your efforts connect with those audiences to achieve the goal? What influences their decisions? What s important to them? What makes them act? Options: Local? Cheap? Convenient? 32

Step 3: Establish benchmarks Past Performance Over Time Think 3 Whatever keeps your C-suite up at night 33

Step 4: Pick your Kick-Butt Index The Perfect KBI Is actionable Is there when you need it Continuously improves your processes & gets you where you want to go You become what you measure, so pick your KBI carefully 34

The Path to Purchase Social Media Program Awareness/Consideration Preference Purchase, Retention, Renewal

Kick Butt Quality Score Desirable Criteria Score Undesirable Criteria Score Contains a key message 3.50 No Key Message -2.0 Contains a desirable visual 1.00 Negative message -3.0 Contains a quote from your spokesperson 2.50 Contains a competitor or opposition quote -2.0 Positions you as a leader 1.00 A story or a headline that leaves the reader less likely to do business with you -3.0 The story or headline leaves a reader more likely to do business with you 2.00 Total 10.00-10.0 36

Typical Social Engagement Index Action Score Like/Follow/Opens/+1/Favorite.5 Share content (including retweets, forwards, Shares etc.) or shares a link to an ENZ site or page. 2 Signs up to receive email or other content or follows a link to content. 2 Contains one or more of the bank s key messages.5 Positive Comment (leaves a reader more likely to support ) 1 Contains a quote from an alumni or expert 1.5 Views an video 1 Is from a Tier 1 source 1.5 Total 10 37

Step 5: Pick the right measurement tools If you want to measure messaging, positioning, themes, sentiment: Content analysis If you want to measure awareness, perception, relationships, preference: Survey research If you want to measure engagement, action, purchase: Web analytics If you want predictions and correlations you need two out of three

All the tools you really need Google Analytics Ignore page views Set up conversions Excel 11/10/2014

Step 5: Selecting a measurement tool Objective KBI Tool Increase message communications Increase percent of items containing one or more messages Media Content Analysis Increase awareness/preference % of audience preferring your brand to the competition Survey Monkey, Survata Engage marketplace % increase in engagement on website and/or social sites Simply Measured, Google Analytics, Site Catalyst, Network Analysis 40

Step 6: Be Data Informed, not Data Driven Rank order results from worst to best Ask So What? at least three times Put your data into an overall framework consistent with C-Suite expectations Find your Data Geek Compare to last month, last quarter, 13-month average 41 Page 41

Very high High Medium Low Low Medium High Very High Advocacy vs. resource use High Message Content Google + Chat Webinar Level of Engagement Low Resources Ultimate Road Trip Media Day Status update High Resources Photo Event Resource Use Corporate Video Link No Message

Real-Time Metrics focus on Outcomes 43

PUBLIC AFFAIRS & BRAND DASHBOARD Below Target At Risk On Target Priority 1: Brand Awareness Priority 2: Advocacy Priority 3: New Membership Culture Current Variance Current Variance Current Variance Benchmark Benchmark Benchmark Benchmark Current Variance % increase in share of desirable voice 8.9% 2.17% 6.73 pts % increase in PRINZ share of Desirable Conversation around Digital privacy 10% 15 6.73 pts % Increase in lawyers who are extremely or very familiar with PRINZ s positioning 38% 38% Annual Data Only % increase in ratings on communications trustworthiness (from culture survey) 54% New data availabl e July 14 N/A % decrease in share of undesirable voice 1% 1% Unchanged % decrease in Government officials unaware of PRINZs position on Digital Privacy 10% 10% 6.73 pts % Increase in online engagement on issue 25% 0 1% % increase in employee knowledge of PRINZ Way We Work (From Knowledge Quizzes) 23% New data availabl e Q3 N/A % increase in awareness among lawyers (from 2014 Brand Metrics Survey) 46% 46% Next data available Q3 14 % increase in engagement on PRINZ.org.nz 202K 197K 2%s % increase in Inbound info requests for info % increase in perception of PRINZ as a leader in Digital Privacy 516(Apr) 427 (May) 5% 5% Unchanged 17% % employees strongly understanding strategic objectives (From Knowledge Quizzes) % reduction in Say/Do gap on agility metrics (average of agile & make decisions faster) 32% New data availabl e Q3 36.5% New data availabl e Q4 N/A N/A

Connecting data yield greater insights Relationship between interviews and placements 1400 1200 1000 800 600 400 1087 10.25 1331 35.03 29.13 699 848 637 8.73 8.40 1331 27.16 27.96 699 40 35 30 25 20 15 10 200 0 106 73 101 38 24 49 25 Jan Feb Mar April May June July Number of Interviews Number of placements Interview per placement 5 0

ACA has made a demonstrable difference in the quality of media coverage If an ACA or one of its programs was mentioned in a media story, it was: More likely to contain an endorsement More likely to be positive More likely to contain key messages Differences in Quality of Coverage with and without ACA Atlantic City Average of OCS Message Communication Tone Negative Visual -0.66-0.61 0.01 0.00 Less likely to be negative Positive Visual 0.08 0.22 Atlantic City Alliance 0.62 1.42 4.61 Endorsements 0.08

Average OCS Score Without ACA events, OCS Scores for Atlantic City would have been significantly lower The red line represents coverage of Atlantic City minus all mentions of ACA and its programs Average OCS Score Over Time 6 5 4 3 2 4.91 4 3.92 3.25 2.99 2.75 3.58 2.99 4.14 4.1 3.65 2.96 4.27 4.12 3.36 3.24 4.29 2.78 2.56 2.34 2.37 2.43 1.53 1.44 1.77 1 0-1 -2 1.30 0.61 0.63 0.20-0.29 0.37-0.05 0.28 0.28-1.24-1.56 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June Atlantic City OCS without ACA Atlantic City OCS with ACA

Free entertainment generated the highest OCS Scores in Q2 Top Programs By OCS Free Entertainment Lady Antebellum Concert Sand Sculpting World Cup Challenge Triathlon Boardwalk Hall Light Show Hello Summer Sand Blast Miss'd America Pageant Blake Shelton Air Show July 4th Fireworks DO AC Meet AC Miss America 5.26 5.73 8.10 8.00 7.67 7.50 7.36 7.31 7.15 6.71 6.54 6.36 8.68 9.30

When ACA programs received media coverage, traffic followed 140000 120000 100000 80000 60000 40000 20000 0 Relationship between ACA Program Mentions and Site Visits 300 250 200 150 100 50 0 Site Visits Program Mentions

ACA programs drive higher OCS scores, which correlate highly with web visits Correlations between Web Visits and PR Metrics Pearson r. value ACA OCS Scores 0.47 ACA Items 0.44 AC items 0.41

White House Volunteerism Office (CNCS) is able to connect specific social outreach to registrations on serve.gov 51

The ROI of Emily So What? 52

So What = Revenue 35,152,789 OTS Red line indicates media impressions 53

10 Ways to find the Money for Measurement 1. Don t call it measurement It s Research 2. Consolidate - Social? IR? Agencies? Research? 3. Crowdsource Who else has tools or surveys? 4. Spread the cost over 2 fiscal years 5. Reach out to universities 6. Take advantage of free stuff 7. Monitor only what matters 8. Narrow your Top Tier media list to what you CAN afford 9. Google analytics 10. Facebook Insights 11. Twitter Analytics

Remember These Points 1 It s not about the media, it s about the business and the customer 2 It s not about Big Data, but about how you use it. 3 You need to be data informed, not data-driven 4 It s not how loud you re shouting it s about relationships. 5 Standards are a reality not an excuse to hide behind

Thank You! For more information on measurement, read my blog: http://www.painepublishing.com/blog For a copy of this presentation or to subscribe to our newsletter, give me your card or email me at measurementqueen@gmail.com Follow me on Twitter: @queenofmetrics Friend me on Facebook: Katie Paine Or call me at 1-603-682-0735 56