Redefining Measurement for Continuous Learning How to demonstrate impact when learning goes beyond L&D Todd Tauber Vice President, Learning Research Bersin by Deloitte, Deloitte Consulting LLP April 1, 2014
Contents The importance of measuring impact Persistent challenges to measuring impact New complexities in measuring impact Redefining measurement and evaluation 1 IMPACT 2014 THE BUSINESS OF TALENT
Todd Tauber Vice President, Learning Research Bersin by Deloitte, Deloitte Consulting LLP
The importance of measuring impact
Measuring impact is about demonstrating and communicating the value of L&D to the organization If you don t tell people about your success, they probably won t know about it. Donald Trump, The Art of the Deal 4 IMPACT 2014 THE BUSINESS OF TALENT
Share of S&P 500 Value Learning is increasingly important Companies are increasingly relying on human capital 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 83% 68% 32% 20% 20% 80% 80% 68% 32% 17% 1970s 1980s 1990s 2000s 2010s Tangible assets Intangible assets Source: Ocean Tomo LLC, Intangible Asset Value, 4/2011 5 IMPACT 2014 THE BUSINESS OF TALENT
% of payroll Organizations expect a lot from L&D Organizations invest a lot in L&D 4% 3% $1,169 per learner 2% 1% 0% 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 Source: ASTD, 2013 State of the Industry, 10/2013; Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014 6 IMPACT 2014 THE BUSINESS OF TALENT
Three ingredients for high-impact learning and employee satisfaction isn t one of them Alignment Effectiveness Business and organizational impact Efficiency Source: Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012 7 IMPACT 2014 THE BUSINESS OF TALENT
Learning isn t working for many organizations It s not aligned It s not effective It s not efficient 15% 10% 50% of global HR executives believe their organization is ready with the right people with the right skills in the right places. of the knowledge and skills people learn in training are actually applied on-the-job with positive results of the money spent on employee learning and development is wasted irrelevant, redundant, not absorbed, not applied. Source: Deloitte, Global Human Capital Trends, 3/2014; Chief Learning Officer, Scrap Learning and Manager Engagement, 3/2011 8 IMPACT 2014 THE BUSINESS OF TALENT
L&D staff / 1,000 learners Learning is essential, but L&D is vulnerable L&D is often one of the first things to get cut when times are tough 10 9 8 7 6 5 7.0 6.2 5.9 5.3 5.2 4.2 4 3 2 1 0 2008 2009 2010 2011 2012 2013 Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014 9 IMPACT 2014 THE BUSINESS OF TALENT
Would anyone miss you? I told them not to send [the reports] to anyone Guess what? At the end of the month, hardly anyone had asked for their reports, so they clearly weren t getting any insights from them. [But] if we had asked them if they wanted their reports, I m sure they would have said yes. The moral of the story is that overuse or misuse of data analytics is in danger of diverting attention away from the real business issues. former Chief Learning Officer, manufacturing company Source: Henley Business School, Big Data and HR, 7/2013 10 IMPACT 2014 THE BUSINESS OF TALENT
Persistent challenges to measuring impact
Three stubborn hurdles What are the biggest challenges you have in measurement? Where to start? Who / what it s for? What methodologies to use? People and relationships Systems for capturing data Tools for analyzing data Skills Funding Leadership / management support Frameworks Infrastructure Culture Source: Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012 12 IMPACT 2014 THE BUSINESS OF TALENT
The customer is always right How CEOs measure learning success We should measure this 96% We do measure this 74% 82% 78% 61% 53% 8% 4% 11% 28% 32% 22% Impact ROI Application Cost efficiency Learning Reactions Source: ASTD / ROI Institute, Measuring What Matters: How CEOs View Learning Success, 12/2010 13 IMPACT 2014 THE BUSINESS OF TALENT
Directional insights, not precise calculations Where people get messed up is in thinking that they have to measure individual training classes to make sure that participants absorbed the information and are applying it to their jobs. I'm not interested in that, and neither is our CFO. He wants to measure whether managers are performing well and whether the organization is productive. Chief Learning Officer, technology company Source: CFO, Measured Response, 6/2011 14 IMPACT 2014 THE BUSINESS OF TALENT
Value Business metrics, not busy metrics More mature L&D groups dig deeper into talent and business outcomes 100% Business leaders CLOs Instructional designers Job impact (Level 3) Business impact (Level 4) 50% Impact on business metrics ROI Manager satisfaction Learning Scores (Level 2) Costs Utilization Hours delivered Learner satisfaction (Level 1) Enrollments 0% 0% 50% 100% Frequency Source: Bersin by Deloitte, Practical and Proven Approaches to Learning Measurement, 4/2012 15 IMPACT 2014 THE BUSINESS OF TALENT
New complexities in measuring impact
Knowledge / skills Training event Continuous learning happens with and without L&D Learning isn t just programs and courses anymore Continuous learning Traditional training Reinforcement via Coaching & mentoring Peer feedback Job aids / tools Online / mobile courses On-demand content Conferences Professional networks Social network Communities of practice etc. Time 17 IMPACT 2014 THE BUSINESS OF TALENT
Continuous learning activities can't be measured as simply as events, courses and programs We learn through interaction, everywhere, all the time Performance support Websites Job rotations Interviews Teaching Collaboration Conversations Mentoring Dialogue Peers Meetings Feedback Manuals Successes Lectures Instructions Simulations Mistakes Virtual worlds Demonstrations Role Playing Debates Webinars Goals Whitepapers Coaching Experiments Books Customer data Presentations Lunch-and-learns Play Communities-of-practice After action reviews Podcasts Wikis Conferences Observation Journaling Micro-blogs Discussion boards e-learning Articles Onboarding Chalk talks Following Visual aids Video Advice Labs Social networking Case studies Role models Search Post-mortems Consulting Job shadowing Stretch goals Tagging 18 IMPACT 2014 THE BUSINESS OF TALENT
More digital learning means lots more stuff to track What % of learning hours are delivered via Other things Online self-study Live virtual Live instructor-led 2005 2006 2007 2008 2009 2010 2011 2012 2013 Source: Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014 19 IMPACT 2014 THE BUSINESS OF TALENT
Data is fragmenting way beyond the LMS There s a lot more to collect and connect Learning systems Multiple LMSs vilt systems Collaboration platforms Learning portals Business systems ERP Finance Business intelligence CRM / marketing HR / HCM systems ATS / assessment Core HRIS Workforce analytics Talent management systems Other systems Social media Web analytics Mobile apps MOOCs 20 IMPACT 2014 THE BUSINESS OF TALENT
Growing demand for better insights Are your business leaders asking for quantitative measures of HR / L&D? In 2009, 21% said yes 2009 2014 In 2014 52% are saying yes (and 29% expect them to soon) Source: KnowledgeAdvisors, The Business Drivers of Talent Analytics, 3/2014 21 IMPACT 2014 THE BUSINESS OF TALENT
Expectations outpacing capabilities Which functions have strong analytics capabilities? Finance Operations 77% 81% Sales Marketing 58% 56% HR 15% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Bersin by Deloitte, High-Impact Talent Analytics, 10/2013 22 IMPACT 2014 THE BUSINESS OF TALENT
The more things change, the more they stay the same Same old lack of know-how, systems and support Frameworks Infrastructure Culture For informal learning For on-demand learning For continuous, self-directed learning New data New data sources New systems and tools Changing demands Increasing expectations Different skills and capabilities 23 IMPACT 2014 THE BUSINESS OF TALENT
Redefining measurement and evaluation
The basics still apply Have a plan 60% of L&D leaders we surveyed said their companies don t have a real plan for measuring learning impact. Build relationships with stakeholders 1.2x companies that regularly engage with stakeholders are more likely to have high-impact measurement capabilities. Dedicate resources and manpower 2% 64% of L&D organizations spend 2% or less of their budgets on measurement. 72% think they should spend more. Source: Bersin by Deloitte, Key Trends in the Measurement of Talent Development, 3/2012; Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012 25 IMPACT 2014 THE BUSINESS OF TALENT
Rethink yesterday s tools and practices Kirkpatrick and Phillips methods are only the beginning Too simplistic Too hard And too backward Does liking a course or passing a test really translate directly into better performance or business results? There are no clear, consistent definitions and it s rarely simple or easy to isolate learning s contribution ( or L&D s). Finding out after training that people were not satisfied or that results didn t improve doesn t tell you what to change. Source: Bersin by Deloitte, Key Trends in the Measurement of Talent Development, 3/2012 26 IMPACT 2014 THE BUSINESS OF TALENT
Build (some) new capabilities Utility / usefulness Stakeholder satisfaction Experience API (xapi) Learning records stores Employee engagement Readiness / promotions Information technology Adaptive learning Employee retention Finance and accounting Web analytics Time to proficiency Marketing Social listening Activity / contributions Business intelligence CRM systems Influence and feedback New friends New tools New metrics 27 IMPACT 2014 THE BUSINESS OF TALENT
Adopt some new approaches Good enough is good enough Predictive analytics Continuous monitoring 28 IMPACT 2014 THE BUSINESS OF TALENT
Find inspiration in finance and marketing Marketers measure and optimize ad campaigns continuously Market conditions Season Consumer confidence Fuel prices Marketing actions TV advertisements Online ads Social media Competitive activity Product releases Pricing Promotions Analyze Optimize Allocate Response Search Online chatter Store visits Purchases Outcomes Unit sales Revenues Margins Market share Share of voice Customer value Source: Adapted from Harvard Business Review, A Better Way to Measure Your Ad Campaign, 3/2013 29 IMPACT 2014 THE BUSINESS OF TALENT
The payoff Better alignment which (often) leads to better relationships with business leaders More effective learning And more efficient operations which (often) leads to more credibility and opportunities which (often) makes it easier to gain access to resources 30 IMPACT 2014 THE BUSINESS OF TALENT
This publication contains general information only and Deloitte is not, by means of this publication, rendering accounting, business, financial, investment, legal, tax, or other professional advice or services. This publication is not a substitute for such professional advice or services, nor should it be used as a basis for any decision or action that may affect your business. Before making any decision or taking any action that may affect your business, you should consult a qualified professional advisor. Deloitte shall not be responsible for any loss sustained by any person who relies on this publication. As used in this document, "Deloitte" means Deloitte Consulting LLP, a subsidiary of Deloitte LLP. Please see www.deloitte.com/us/about for a detailed description of the legal structure of Deloitte LLP and its subsidiaries. Certain services may not be available to attest clients under the rules and regulations of public accounting. 36 USC 220506 Member of Deloitte Touche Tohmatsu Limited