Forget Big Data - Measure What Matters

Transcription

1 Forget Big Data - Measure What Matters Katie Delahaye Paine CEO Paine Publishing measurementqueen@gmail.com

2 Katie Delahaye Paine: Helping communications professionals define and measure success for 25 years. Founder of: The Delahaye Group KDPaine & Partners Paine Publishing Author of: Measuring the Networked Non-Profit Measure What Matters Measuring Public Relationships Paine Publishing: Providing communications professionals the knowledge and information they need to navigate the journey to good measurement Newsletters Training Courses Consulting painepublishing.com

3 Don t Measure What s Easy Cartoon by Rob Cottingham 3

4 Measure What Matters Cartoon by Rob Cottingham News Group International 10/11/2014 4

5 Welcome to the, I want what I want, not what you send me, era aka: The Relationship Era

6 It s the end of the world as we knew it 1,000,000 The average audience for a MyDrunkKitchen video ( 179,000 CNN average nightly audience 27 90% The number of times per hour Digital Natives switch media every 2.2 minutes. The percent of conversation that happens OFF LINE 10% The percent of on-line conversations that are public < 5% 40% The percent of Facebook & Twitter posts that are actually seen The amount of conversations generated by bots, spammers and payper-click sites

7 We need a new Attribution Model Digital/On -line Media Buy Print/TV Media Buy Other Paid Marketing Media Relations/ Social Media/PR Digital/ Online Media Buy Print/TV Media Buy Media Relations /Social Media/PR ROI ROI

8 What s Changed? Collapse of mass media Growth of media everywhere Intolerance for messaging The Barcelona Principles Viewers are more likely to stop watching commercials at the moment in which brand logos appear on the screen - ARF Study 8

9 The Barcelona Principles, The Conclave & Industry Standards 1. Importance of Goal Setting and Measurement 2. Measuring the Effect on Outcomes is Preferred to Measuring Outputs 3. The Effect on Business Results Can and Should Be Measured Where Possible 4. Media Measurement Requires Quantity and Quality 5. Earned Media Value/AVEs are not the value of Public Relations 6. Social Media Can and Should be Measured 7. Transparency and Replicability are Paramount to Sound Measurement The Conclave All standards are available on andards-central

10 Social Media Measurement Standards have been published for: Content Sourcing & Methods Reach and Impressions Engagement Influence & Relevance Opinion & Advocacy Impact & Value 11/10/2014

11 Impressions are not awareness. Where s the So What? MSM Online Social Media Eyeball counting HITS Outcomes 11

12 Likes Are Not Engagement Impressions Likes Followers Trial/Consideration Commitment Advocacy 12 11/10/2014

13 The Engagement Process Listen Donate Purchase Achieve Goal Run

14 Measuring the Engagement Process Web Analytics CRM/Convio ecommerce Map My Run/Social Sharing icloud & Google+

15 #3: Standards for Engagement Engagement is an action that happens in response to content i.e. when someone engages with you. (not about you ) Engagement manifests differently by channel, but is typically measurable based on effort required and how it is shared with others. Engagement can be desirable or undesirable Any measure of engagement must be tied to the goals and objectives for your program Engagement occurs both off and online, and both must be considered if you intend to integrate your metrics with other marketing or communications efforts. Engagement includes such actions as likes, +1, shares, votes, comments, links, links, retweets, Facebook's Talking about you etc. Engagement actions should be counted by % of the audience that responds.

16 GWA s, Photos Drive Shares & Likes David Beckham & Dierks Bentley Video Share & Likes Over Time Average of PostLikes Average of PShares Mia Farrow

17 #4: Influence & Relevance Influence happens when you are persuaded to change behavior or opinion that would otherwise not have changed. Influence can be attributed to either an outlet or an individual. Influence happens online and off line and both should be considered Should include some combination of the following five elements: Reach Engagement around individual Relevance to topic Frequency of posts around the topic Audience impact as measured by the ability to get the target audience to change behavior or opinion. If an individual scores a 0 on one element, they have no influence Influence cannot be expressed in a single score or algorithm

18 Big Numbers Influence All influence is relative Who is influential depends on your audience A computer cannot tell you who matters most To be influential requires relevance, frequency & reach

19 #5 Standards for Measuring Tone/Sentiment Whatever process is defined and applied, it must be used consistently throughout any analysis. Sentiment reliability varies by vendor and approach therefore coding definitions, consistency and transparency are critical. Opinions, recommendations and other qualitative measures are typically more valuable than raw sentiment and increasingly measurable: Opinions ( it s a good product ) Recommendations ( try it or avoid it ) Feeling/Emotions ( That product makes me feel happy ) Intended action ( I m going to buy that product tomorrow )

20 Testing the accuracy of coding % Agreement with human coding % positive %negative %neutral NetBase 88.64% UberVu 11.95% 9.46% 78.00% UberVu 68.31% NetBase 7.84% 0.32% 92.11% SDL 48.94% Beyond 58.00% 13.00% 30.00% 0.00% 20.00% 40.00% 60.00% 80.00% % SDL UberVu NetBase SDL 33.12% 26.53% 40.35%

21 Everyone agrees on Neutral Not so with Negative and Positive Degree to which agreement occurred No Yes Positive 72.36% 27.64% Neutral 14.82% 85.18% Negative 62.32% 37.68% Mixed % 0.00%

22 #6: Impact & Value Impact and value will always be dependent on client objectives Need to define outcomes in advance will likely span multiple business goals, especially for social (crosses disciplines) ROI should be strictly limited to measurable financial impact; total value can be used for financial and non-financial impact combination Value can be calculated in positive returns (sales, reputation, etc.) or avoided negative returns (risk mitigated, costs avoided) Key performance indicators and balanced scorecards are helpful to connect social media impact to business results/language

23 ROI ROI is: Cost savings Greater efficiency Lower legal costs Faster time to implementation Lower cost per impact vs. other marketing channels Lower cost per message exposure vs. other channels ROI is not: Impressions Earned media value or AVE Number of shares, views or likes

24 Social Media isn t media, it s your business Conversations PR//Mkt Prod. Mktg Product Support Mktg Research R&D CI IR HR Sales Savings, shorter cycles, more renewals, better ideas, $$$$$$$$$$$$$$$$$$$$$$$$$$ 11/10/

25 Good Relationships Are More Cost Effective Type I love Zappos into Google, and you find 1.19 million references Type Citibank and you get 21,000 references. Citibank spends 100 times more a year on advertising than Zappos. Cost per delegate acquired: Obama: $6,024 Clinton: $147,058 Romney: $2,389,464

26 So What? It s not about the media, it s about your business and your customer It s not how loud you re shouting, it s what your stakeholders do after they ve heard your message and your relationships. There are no boundaries It s not about how big your data is, it s about how you use it

27 6 steps to standards-compliant measurement Step 1: Define your goal(s). What outcomes is this strategy or tactic going to achieve? What are your measurable objectives? Step 2: Define the parameters Who are you are trying to reach? How do your efforts connect with those audiences to achieve the goal. Step 3: Define your benchmarks. Who or what are you going to compare your results to? Step 4: Define your metrics. What are the indicators to judge your progress? Step 5: Select your data collection tool(s). Step 6: Analyze your data. Turn it into action, measure again 1 2 Six Steps to Success

28 Definitions Monitoring process by which data are systematically and regularly collected about a program over time. Measurement a way of giving an activity a precise dimension, generally by comparison to some standard; usually done in a quantifiable or numerical manner; see also: data, scale Metrics an agreed upon number or percentage that indicates progress towards a goal Evaluation a form of research that determines the relative effectiveness of a public relations campaign or program by measuring program* *IPR Dictionary of Measurement Terms, Dr. Donald Stacks 28

29 Why do we communicate? Activities How does what you do contribute to the bottom line? Outtakes (Intermediary Effects) Awareness Knowledge/Education Understanding Outcomes (Target Audience Action) Revenue Leads Engagement Advocacy 29 11/10/2014

30 Step 1: Define the goals What return is expected? Define in terms of the mission. Define your champagne moment. If you are celebrating complete 100% success a year from now, what is different about the organization? 30

31 Goals & Suggested Metrics Goals Goal 1: Meet Sales Targets Goal 2: Reduce Risk/Threats Goal 3: Increase market share in new market Communications Contribution Expand the marketable universe Reduce sales cycle Increase Trust Increase advocacy Expand the marketable universe Metrics % Increase in desirable share of voice % in awareness % increase in undesirable voice % increase in trust scores % increase in desirable voice in new market % increase in awareness of brand in new market

32 Step 2: Understand the parameters. Goal: Get the cat to stop howling Strategy: Buy cat food What management s priorities? Who are you are trying to reach? How do your efforts connect with those audiences to achieve the goal? What influences their decisions? What s important to them? What makes them act? Options: Local? Cheap? Convenient? 32

33 Step 3: Establish benchmarks Past Performance Over Time Think 3 Whatever keeps your C-suite up at night 33

34 Step 4: Pick your Kick-Butt Index The Perfect KBI Is actionable Is there when you need it Continuously improves your processes & gets you where you want to go You become what you measure, so pick your KBI carefully 34

35 The Path to Purchase Social Media Program Awareness/Consideration Preference Purchase, Retention, Renewal

36 Kick Butt Quality Score Desirable Criteria Score Undesirable Criteria Score Contains a key message 3.50 No Key Message -2.0 Contains a desirable visual 1.00 Negative message -3.0 Contains a quote from your spokesperson 2.50 Contains a competitor or opposition quote -2.0 Positions you as a leader 1.00 A story or a headline that leaves the reader less likely to do business with you -3.0 The story or headline leaves a reader more likely to do business with you 2.00 Total

37 Typical Social Engagement Index Action Score Like/Follow/Opens/+1/Favorite.5 Share content (including retweets, forwards, Shares etc.) or shares a link to an ENZ site or page. 2 Signs up to receive or other content or follows a link to content. 2 Contains one or more of the bank s key messages.5 Positive Comment (leaves a reader more likely to support ) 1 Contains a quote from an alumni or expert 1.5 Views an video 1 Is from a Tier 1 source 1.5 Total 10 37

38 Step 5: Pick the right measurement tools If you want to measure messaging, positioning, themes, sentiment: Content analysis If you want to measure awareness, perception, relationships, preference: Survey research If you want to measure engagement, action, purchase: Web analytics If you want predictions and correlations you need two out of three

39 All the tools you really need Google Analytics Ignore page views Set up conversions Excel 11/10/2014

40 Step 5: Selecting a measurement tool Objective KBI Tool Increase message communications Increase percent of items containing one or more messages Media Content Analysis Increase awareness/preference % of audience preferring your brand to the competition Survey Monkey, Survata Engage marketplace % increase in engagement on website and/or social sites Simply Measured, Google Analytics, Site Catalyst, Network Analysis 40

41 Step 6: Be Data Informed, not Data Driven Rank order results from worst to best Ask So What? at least three times Put your data into an overall framework consistent with C-Suite expectations Find your Data Geek Compare to last month, last quarter, 13-month average 41 Page 41

42 Very high High Medium Low Low Medium High Very High Advocacy vs. resource use High Message Content Google + Chat Webinar Level of Engagement Low Resources Ultimate Road Trip Media Day Status update High Resources Photo Event Resource Use Corporate Video Link No Message

43 Real-Time Metrics focus on Outcomes 43

44 PUBLIC AFFAIRS & BRAND DASHBOARD Below Target At Risk On Target Priority 1: Brand Awareness Priority 2: Advocacy Priority 3: New Membership Culture Current Variance Current Variance Current Variance Benchmark Benchmark Benchmark Benchmark Current Variance % increase in share of desirable voice 8.9% 2.17% 6.73 pts % increase in PRINZ share of Desirable Conversation around Digital privacy 10% pts % Increase in lawyers who are extremely or very familiar with PRINZ s positioning 38% 38% Annual Data Only % increase in ratings on communications trustworthiness (from culture survey) 54% New data availabl e July 14 N/A % decrease in share of undesirable voice 1% 1% Unchanged % decrease in Government officials unaware of PRINZs position on Digital Privacy 10% 10% 6.73 pts % Increase in online engagement on issue 25% 0 1% % increase in employee knowledge of PRINZ Way We Work (From Knowledge Quizzes) 23% New data availabl e Q3 N/A % increase in awareness among lawyers (from 2014 Brand Metrics Survey) 46% 46% Next data available Q3 14 % increase in engagement on PRINZ.org.nz 202K 197K 2%s % increase in Inbound info requests for info % increase in perception of PRINZ as a leader in Digital Privacy 516(Apr) 427 (May) 5% 5% Unchanged 17% % employees strongly understanding strategic objectives (From Knowledge Quizzes) % reduction in Say/Do gap on agility metrics (average of agile & make decisions faster) 32% New data availabl e Q3 36.5% New data availabl e Q4 N/A N/A

45 Connecting data yield greater insights Relationship between interviews and placements Jan Feb Mar April May June July Number of Interviews Number of placements Interview per placement 5 0

46 ACA has made a demonstrable difference in the quality of media coverage If an ACA or one of its programs was mentioned in a media story, it was: More likely to contain an endorsement More likely to be positive More likely to contain key messages Differences in Quality of Coverage with and without ACA Atlantic City Average of OCS Message Communication Tone Negative Visual Less likely to be negative Positive Visual Atlantic City Alliance Endorsements 0.08

47 Average OCS Score Without ACA events, OCS Scores for Atlantic City would have been significantly lower The red line represents coverage of Atlantic City minus all mentions of ACA and its programs Average OCS Score Over Time Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June Atlantic City OCS without ACA Atlantic City OCS with ACA

48 Free entertainment generated the highest OCS Scores in Q2 Top Programs By OCS Free Entertainment Lady Antebellum Concert Sand Sculpting World Cup Challenge Triathlon Boardwalk Hall Light Show Hello Summer Sand Blast Miss'd America Pageant Blake Shelton Air Show July 4th Fireworks DO AC Meet AC Miss America

49 When ACA programs received media coverage, traffic followed Relationship between ACA Program Mentions and Site Visits Site Visits Program Mentions

50 ACA programs drive higher OCS scores, which correlate highly with web visits Correlations between Web Visits and PR Metrics Pearson r. value ACA OCS Scores 0.47 ACA Items 0.44 AC items 0.41

51 White House Volunteerism Office (CNCS) is able to connect specific social outreach to registrations on serve.gov 51

52 The ROI of Emily So What? 52

53 So What = Revenue 35,152,789 OTS Red line indicates media impressions 53

54 10 Ways to find the Money for Measurement 1. Don t call it measurement It s Research 2. Consolidate - Social? IR? Agencies? Research? 3. Crowdsource Who else has tools or surveys? 4. Spread the cost over 2 fiscal years 5. Reach out to universities 6. Take advantage of free stuff 7. Monitor only what matters 8. Narrow your Top Tier media list to what you CAN afford 9. Google analytics 10. Facebook Insights 11. Twitter Analytics

55 Remember These Points 1 It s not about the media, it s about the business and the customer 2 It s not about Big Data, but about how you use it. 3 You need to be data informed, not data-driven 4 It s not how loud you re shouting it s about relationships. 5 Standards are a reality not an excuse to hide behind

56 Thank You! For more information on measurement, read my blog: For a copy of this presentation or to subscribe to our newsletter, give me your card or me at measurementqueen@gmail.com Follow me on Friend me on Facebook: Katie Paine Or call me at