Evaluation and Metrics Program. Global Knowledge. M R Chevalier, Market Research Analyst. Sarah Broyer, Evaluation & Metrics.

Transcription

1 Global Knowledge Evaluation and Metrics Program Sarah Broyer, Evaluation & Metrics M R Chevalier, Market Research Analyst October 6, Global Knowledge Network, Network, Inc. Inc. All All rights rights reserved. reserved. 12/19/ /19/2005 Page Page 1

2 Agenda Background on Global Knowledge Business case for using Metrics that Matter Worldwide Implementation Plan and Timeline Tactical and Strategic uses of MTM Marketing Applications & Best Practices 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 2

3 About Global Knowledge Largest privately held provider of training, enterprise learning services, and software solutions for IT and management professionals. Employs 800 employees worldwide, delivering over 700 courses in 13 languages. World s largest Authorized Cisco Learning Solutions Partner. Blended learning solution provides a combination of Instructor-led and self-paced learning to meet very specific training needs Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 3

4 Business Case for MTM Strong desire to automate evaluation collection process. Need for real-time access to customer response data and various reporting. Committed to monitoring and making improvements to products and services where opportunities lie. Interested in benchmarking our performance against the Industry Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 4

5 Deploying Metrics that Matter Define Strategy Deployment Execute Value Prop Impact Challenges Resources US & Canada UK Ireland Sweden Netherlands Germany France Italy Austria Switzerland Spain Communication Training Presentations 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 5

6 Where we are today Using MTM throughout Operations worldwide in 11 countries. Collected over 45,000 post event evaluations and about 5, day follow-up evaluations. Collecting Instructor evaluations to gather intelligence on our internal support system for Instructors in the field. Feeding data back into organization on a weekly, monthly and quarterly basis, to Product Management, Course developers, Operations, Sales and Marketing Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 6

7 Review Process Weekly Event level reporting Alarms reports Monthly Marketing Job and ROI analysis Course Developer reports Quarterly Product Management Course level reporting, to drive improvements in product quality Industry benchmarking to ensure we are achieving corporate goals 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 7

8 Sample Weekly Report Tactical Tools > Class Analysis > Class Level Detail Provides a overall performance snap-shot by class. Can be ed to stakeholders for immediate action Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 8

9 Sample Weekly Report Tactical Tools > Respondent Analysis > Class Summary Comprehensive Class Summary available real-time. Time saved in data collection, processing and reporting Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 9

10 Sample Monthly Report Tactical Reports > Class Analysis > Quick Question Aggregates data over a specified time period and shows distribution of results. Access to written comments aggregated over a time period too Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 10

11 Sample Monthly Report Tactical Reports > Class Analysis > Quick Question Data can be used to target sales efforts. Useful in predicting future business Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 11

12 Sample Monthly Report Executive > Report Card Provides roll-up of all responses and compares it to benchmarks. Motivates by example Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 12

13 Global Knowledge Evaluation and Metrics Program Part 2 Marketing Applications M R Chevalier Market Research Analyst Global Knowledge Network, Network, Inc. Inc. All All rights rights reserved. reserved. 12/19/2005 Page 13

14 Why Marketing Considerations? All initiatives, internal or external, need to be marketed in order to gain support ROI metrics are part of the value proposition Identification of student demographics, motivations, and the market factors enhance the relative value of the ROI 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 14

15 Marketing vs Training Marketing is an organizational function and a set of processes for creating, communicating and delivering value to customers and for managing customer relationships in ways that benefit the organization and stakeholders - American Marketing Association, 2004 A redefinition: Training is an organizational function and a set of processes for creating, communicating and delivering knowledge to employees and for managing employees relationships in ways that benefit the organization and stakeholders 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 15

16 Common Assumptions Individuals seek training/certification to increase $ False a wide variety of surveys illustrate that personal satisfaction is the most cited reason. Among Cisco candidates, 70% sought their certification for personal goal. Only a fifth said they did it to get a promotion or a raise from their current employer Cisco Professional Salary Survey, TCP Magazine, 2004 Training is expensive (and is an expense) False An increase of $680 in a firm s training expenditures per employee generates, on average, a six-percentage point improvement in total shareholder return in the following year. Profiting from learning: do firms investments in education and training pay off? American Society of Training and Development and Saba Software, Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 16

17 What is the objective? To ask the right people The right question At the right time To get the best answer To make the right decision We must be mindful that we are not looking for the answer that supports a decision already made 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 17

18 The right people? From a recent study: 98% of IT managers working in large companies participated in a market research survey within the past year The majority of respondents were asked to participate on more than five occasions On average, there were eight invitations Is there is a risk of survey burnout? - Answers Research, Inc - Conserve Your Resources - Quirk s Marketing Research Review - July/August Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 18

19 A growing challenge? 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 19

20 Who seeks certification? 92% male 77% between ages of % in network engineering/administration Average of 8.5 yrs of IT experience Average of 3.2 professional certifications and adding one new cert in past 12 months Spending an average of $2,715 on primary credential 95% work full time Average salary of $62,540 Certmag 2003 survey 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 20

21 The right question? Learning value? Satisfaction? Recommend? Our smile sheets simply addressed satisfaction. Learning analytics can expand on ROI metrics. Can we identify marketing metrics as well? 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 21

22 Applicable Satisfaction Questions From the standard surveys: What was your main objective for completing this training? Impact of this training on your career potential? How well did this training meet your expectations? From the Cisco surveys: I would take another Cisco class/learning product I would recommend this training to others Considering other educational experiences, I would rate this event: These provide information on buying behavior as well as external benchmarking Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 22

23 Applicable Market Questions Certification Goals Most recent training methods Reason for selecting selecting course/location/modality Training interest in next year Time frame of next training Planned company initiatives 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 23

24 Applicable Demographic Questions Job Function Current Certifications Years of experience Industry # Employees in firm 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 24

25 The Ultimate Question Simply ask the question - Would you recommend the product? Subtract neutral and negative responses from positive Proposed by Fred Reichheld, a Bain consultant who has used the method for Enterprise Rent-A-Car and other clients GE, Intuit, and Symantec are advocates Note however that such questions are often asked binary Yes/No 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 25

26 Variations in Satisfaction Analysis Regression Conjoint Chi-square Weighted values Trimmed scales such as translation to three-point Net promoter Measuring satisfaction is often referred to as the Ultimate Question 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 26

27 Examples Simple Yes/No Starting off with the simple question would you recommend this training to others (yes/no)? Ex 1) 791 responses 753 said Yes GREAT! 95.2% However, many factors can influence this such as cost, learning objectives, etc. A particular student s objectives may not have been met (low satisfaction) but he would recommend the class to others to meet their objectives 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 27

28 Examples Cross Reference Pair now with the question regarding meeting career goals Satisfaction Level 1 2 Would You Recommend? No 5.3% 18.4% Yes 0.0% 0.3% Total % 3.2% 30 Ex 2) 791 responses 753 said Yes GREAT! 95.2% Of the top 2 tiers, 91.9% said Yes Note however that nearly ¼ of those fully satisfied would NOT recommend the class to others. all yes top two brackets 4 5 Total 36.8% 23.7% % 74.0% % 91.9% Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 28

29 Examples Weighted Scale Try a weighted score Satisfaction Level Yes points Ex 3) Grading like a class test average If all were satisfied and said Yes, then the maximum score would be n*5 possible (791*5) Total "average" 89.5% 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 29

30 Examples 3 point performance score Using a weighted 3 point scale Would You Recommend? Ex 4) Collapse to a 3 point scale Low are 1-3, value = 0 Medium is 4, value = ½ ratio High is 5, full value of ratio three point scale No Yes low medium (1/2 ratio) % high (full ratio) % performance score 81.2% 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 30

31 Examples Net Promoter Calculating a Net Promoter Ex 6) Sum % of score 4/5 recommend Subtract % of all other responses Satisfaction Level Yes % of All % % % % % Total total 4/5 % recommend 91.9% % low or not recommend 8.1% net promoter 83.8% 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 31

32 Summary Comparison Simple Yes/No % Cross reference, top two brackets 91.9% Weighted test average 89.5% Performance score 81.2% Net promoter 83.8% On a 10 point letter grade scale, variance is just above a C to a solid A How satisfied are your trainees? 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 32

33 Meeting Your Market Customer satisfaction is the single most important issue affecting organizational survival..the key to competitive advantage is proactively gauging customer perceptions and aggressively acting on the findings. - Craig Cochran, Center for International Standards & Quality (CISQ) 2005 Global Knowledge Network, Inc. All rights reserved. 12/19/2005 Page 33