Constructing the Outcome Statement. Dave Vance President, Manage Learning LLC. tdrp. Talent Development Reporting principles

Transcription

1 Constructing the Outcome Statement by: Dave Vance President, Manage Learning LLC Reporting principles

2 Introduction Introduction Reporting principles (TDRp) recommends the creation of an Outcome Statement in addition to Effectiveness and Efficiency Statements. Many learning organizations already create some version of these latter two statements to review progress and manage their function. 1 The Outcome Statement, however, is new for most organizations. While simple in appearance, it may be more challenging to develop. The Outcome Statement collects the organization s key goals and learning s expected impact on achieving those goals. The statement should list the top (5-10 goals of the organization in priority order (either by priority number or by high/ medium/low) as rows in the statement. Goals should be listed even if no learning is planned to achieve them. The row(s) beneath each goal include the impact or measure of success of the planned learning programs (if any) that support the goal. For example, under a company goal to increase sales by 10%, we might have a line called Impact of Training which would indicate that the stakeholder and learning leaders expect training to increase sales by 3% (example of a level 4 impact), or to have an application rate of 70% (example of a level 3 measure), or that 80% of the participants expect to apply it (example of a level 1 predictive measure). Step 1. Columns for the high-level Outcome Statement include, at a minimum, annual data for Last Year s Actual, Current Year Plan, and Year-to-Date Actual. Your organization may also choose to generate a detailed Outcome Statement containing more frequently refreshed data (monthly or quarterly) and/or more granular information (e.g., business unit or district level). The detailed statement does not require a plan or target for each measure. 1. The Effectiveness Statement collects the Kirkpatrick-Phillips Level 1-5 measures and the Efficiency Statement collects cost, activity, utilization and cycle-time measures. Page 1

3 Constructing the Outcome Statement Tips for Getting to Agreement on the Impact % 1. Ask the stakeholder to list all the factors likely to lead to higher sales. 2. Then ask the stakeholder to prioritize them. Then explain that the next step is to assign percentages to each so they add to 100% 3. Work your way down the list starting from the most important. 4. Revise as necessary until stakeholder is comfortable. 5. At the end, the stakeholder may say something like, I guess training should contribute about 30%-40% Tho which you might say, Shall we just use 30% to be conservative? Construction of the Outcome Statement starts with identifying the key goals for the coming year, usually through a discussion by the CLO with the CEO or other senior leaders. This discussion should occur before the new fiscal year begins; consequently, a written plan for the coming year will not likely be available for review. (However, it would be good preparation to review the written plan for the current year if one is available.) Ask the CEO what the goals are likely to be for next year and their priority and who the other stakeholders will be, so you can schedule meetings with them to better understand their goals and determine what role learning has to play in achieving them. Next, meet with the stakeholder of each high-priority goal. Engage in a good discussion about the goal and what will be required in general to achieve the goal. Then explore whether learning might help achieve the goal. If learning can contribute, agree (at least in broad terms) on the types of programs, target audience, completion dates, costs, and most importantly, the expected impact of learning on the goal or at least some measure of success. Some goals will be carry over goals and you may have had these discussions with the stakeholder previously. In cases like these, you can just focus on any new programs for the coming year and their expected impact. If the goal is new or you have not worked with expected impact of learning may require multiple meetings. Ensure you conduct these initial meetings with the senior stakeholder. Gaining agreement on the broad program elements and the expected impact of learning must occur with the individual who is ultimately accountable for the organization s success in achieving the goal. Do not let the senior stakeholder (like the SVP of Sales) delegate these meetings to their staff because staff cannot speak for the stakeholder in agreeing on expected impact. Likewise, the CLO or VP of Training should be in these initial meetings. Staff from both sides may be present and can work the detail later. The goal is to engage in a healthy discussion and arrive at a reasonable and mutually agreeable measure of success for the training program. For some programs (like those in support of a goal to increase sales by 10%), it may be possible to reach agreement on the expected impact of learning on achieving the goal. For example, the stakeholder and CLO may agree that the proposed training program, properly conceived, developed, delivered, supported, and reinforced, should contribute 30% towards the goal of a 10% increase in sales. In this case, the impact of training would be a 3% (30% x 10%) increase in sales. Page 2

4 Talent Det The goal is to find a measure that is meaningful, roughly right and mutually agreed on. However, in some cases it may not be possible or practical to get agreement on a quantitative forecast for the expected isolated impact of learning. Instead, the stakeholder and CLO may agree on a qualitative measure like High, Medium, or Low impact of training. Agreement on a High impact would be appropriate if learning was expected to play the major role (e.g., more than 50% contribution) in achieving the goal. Agreement on a Low impact would be appropriate if learning was expected to play a minor role (e.g., less than 20% contribution). In this case, there are probably multiple other factors which are expected to have a greater impact on the goal. Agreement on a Medium impact would be appropriate if the contribution of learning is expected to fall in between High and Low. Again, the point is to have the discussion with the stakeholder and agree on some measure of the isolated impact of learning before the program is begun. For yet other programs, especially those that support goals only indirectly, it may not be feasible or desirable to try to forecast the isolated impact of the training (either quantitatively or qualitatively). Instead, the CLO and stakeholder may agree on a proxy for impact or some other measure of success. An example of a proxy measure for a learning program to teach leaders how to be better coaches might be the time spent by leaders coaching their employees. The assumption is that if leaders spend the time coaching they will apply their new skills and improve their coaching. Time spent coaching could be captured by a simple survey from a sample of leaders. Another example would be the use of an application rate (a level 3 measure) as a proxy for isolated impact. The assumption is that if participants apply their learning, the program will have an impact on the goal. Other proxies might include the participants predicted impact of training (a predictive level 3 measure), or customer or patient satisfaction ratings or supervisor ratings of the program s effectiveness (level 1 measures). While ideally you would like to reach agreement on Level 4 measure of impact, remember that no measure is perfect. If a Level 4 measure cannot be identified, do the best you can. The goal is to find a measure that is meaningful, roughly right, and mutually agreeable. Last, there are some programs (e.g., basic skill building for new hires) where even use of a proxy may not make sense. The stakeholder and the CLO may agree that the training is absolutely essential to the employee s performance on the job. In other words, the employee would be totally ineffective and may even cause harm to fellow employees or customers if they are not properly trained. In these cases, if no measure of isolated impact or proxy presents itself, the stakeholder and the CLO may agree to use Essential as the impact of training. Obviously, the use of Essential cannot be overused. A lot of learning, in fact, is not essential. Learning may help the organization achieve its goals faster and at less expense and is therefore worth undertaking, but in some cases, the organization could still accomplish its goal without training. Page 3

5 The agreed-upon measure(s) of success are placed below each goal on the Outcome Statement. If there are multiple programs in support of a single goal, we recommend you display the impact of the programs individually in the Outcome Statement along with their combined impact (see Figure 3). If a learning program contributes in a significant way to more than one goal, it should be listed under each goal. You might also add the number of expected participants to show the size of the planned initiative. You might also elect to show multiple measures of success for a single program. Whatever measures are chosen in the end, it is important to retain the business-centric focus of the statement by organizing the information by business goal not by learning program. Do not start with the learning programs and list the business goals underneath them. While it does require some effort, the Outcome Statement will help ensure that learning is aligned to the key goals for the organization and will clearly show the role learning is expected to play in accomplishing those goals. The Outcome Statement also is a key source document used to create the L&D Summary Report. Sample Outcome Statements Four sample Outcome Statements are provided to illustrate the possibilities. The first three are examples of high-level Outcome Statements and the last is an example of a detailed Outcome Statement. Figure 1: A Simple High-Level Outcome Statement Figure 2: An Expanded High-Level Outcome Statement Figure 3: A Complex High-Level Outcome Statement Figure 4: A Detailed Outcome Statement Each meets the TDRp requirements for an Outcome Statement: High-Level or Summary Outcome Statement Key organizational goals listed in priority order Impact of learning on achieving the goald and/or another measure(s) of success Data for last year s actual results, this year s plan, year-to-date results, and year-to-date results as a percentage of plan Detailed Outcome Statement Key organizational goals listed in priority order Impact of learning included where possible (some impact measures are not available on a monthly basis) Greater granularity than the high-level statement and/or more frequently refreshed data (monthly or quarterly rather than annual and year-to-date) Note: The detailed outcome statement cannot take the place of the high-level or summary statement. The detailed statement is optional while the summary statement is required. Page 4

6 A Simple High-Level Outcome Statement Figure 1 The High-Level Outcome Statement in Figure 1 lists six organizational goals and learning s expected impact on achieving five of those. A Simple High-Level Outcome Statement 2011 For 2012 Priority Business Outcomes and Learning Impact Actual Plan Jun YTD % of Plan Included in Business Plan 1 Revenue: Increase in Sales Corporate Goal or Actual % 10% 20% 17% 85% Application of Training: % Applying 3 key concepts % NA 80% 85% 106% 2 Leadership: Increase in EOS (1) Leadership Score Corporate Goal or Actual points 0 pts 5 pts 3 pts 60% Impact of Training: Time spent coaching direct reports min/wk NA % 3 Safety: Reduction in Injuries Corporate Goal or Actual % 10% 20% 15% 75% Impact of Training: 70% reduction in injuries due to L&D % 5% 14% 11% 79% 4 Call Center Satisfaction: Improve Score Corporate Goal or Actual points % Impact of Training: High, Medium, Low H/M/L NA High High 5 Comply with New Regulations Corporate Goal or Actual % in compliance 100% 100% 100% Impact of Training: Essential for compliance NA Essential Essential 6 Innovation: Increase in New Patents Corporate Goal or Actual # % Impact of Training NA NA NA For Priority 1, the application rate (a level 3 measure) is used as a proxy for learning s impact on increasing sales. The stakeholder believes that if 80% of the participants apply the three key concepts in the first 60 days following training, the training will have a significant impact on increasing sales. Of course, it would be better to have a forecast of isolated impact, but the application rate is a good proxy and readily measurable. For Priority 2, time spent coaching is used as a proxy for the impact of the training program on the goal to increase the leadership score from the semiannual employee opinion survey (EOS). For Priority 3, the stakeholder expects learning to deliver 70% of the planned reduction in injuries. Since the plan is for a 20% reduction in injuries, learning is expected to reduce injuries by 14% (70% x 20%). In other words, learning is expected to have an isolated impact (a level 4 forecast) of 70% on achieving the goal to reduce injuries by 20%. For Priority 4, learning is expected to have a High impact on improving call center satisfaction. Although not quantified, this implies that learning will play the major role in increasing call center satisfaction. Page 5

7 For Priority 5, the stakeholder has deemed that learning is Essential to complying with the new regulations. This means that if learning were to be withheld, the organization would definitely be out of compliance with the new regulations. Note that no learning is planned for Priority 6, but the goal is still included. It is just as important to know which goals will not be supported by learning. This may provoke someone to ask if learning could or should play a role which is always a healthy discussion. Talent Det The High-Level Outcome Statement in Figure 2 includes important goals not in the organization s high-level business plan and also adds the number of participants. An Expanded High-Level Outcome Statement The High-Level Outcome Statement in Figure 2 (on the next page) includes important goals not in the organization s high-level business plan and also adds the number of participants. (Note: The goals, priorities, and programs of Figure 2 are consistent with those in Figure 1. Thus, Figure 2 represents an expanded version of Figure 1.) Page 6

8 Figure 2 An Expanded High-Level Outcome Statement 2011 For 2012 Priority Business Outcomes and Learning Impact Actual Plan Jun YTD % of Plan Included in Business Plan 1 Revenue: Increase in Sales Corporate Goal or Actual % 10% 20% 17% 85% Application of Training: % Applying 3 key concepts % NA 80% 85% 106% Number of Participants % 2 Leadership: Increase in EOS (1) Leadership Score Corporate Goal or Actual points 0 pts 5 pts 3 pts 60% Impact of Training: Time spent coaching direct reports min/wk NA % Number of Participants % 3 Safety: Reduction in Injuries Corporate Goal or Actual % 10% 20% 15% 75% Impact of Training: 70% reduction in injuries due to L&D % 5% 14% 11% 79% Number of Participants 458 1, % 4 Call Center Satisfaction: Improve Score Corporate Goal or Actual points % Impact of Training: High, Medium, Low H/M/L NA High High Number of Participants % 5 Comply with New Regulations Corporate Goal or Actual n compliance 100% 100% 100% Impact of Training: Essential for compliance NA Essential Essential Number of Participants 4,612 5,000 2,463 49% 6 Innovation: Increase in New Patents Corporate Goal or Actual # % Impact of Training NA NA NA High Not Included in Business Plan Business Acumen: Improve EOS (1) score Corporate Goal or Actual points % Impact of Training: Increase acumen score on EOS H/M/L NA High Medium Number of Participants NA % Medium Employee Engagement: Increase in EOS (1) Total Score Corporate Goal or Actual points % Impact of Training: Provide 2 development courses per ee H/M/L Low Medium Medium Number of Participants 9,450 12,000 5,478 46% Note: 1. EOS is quarterly Employee Opinion Survey Page 7

9 A Complex, High-Level Outcome Statement The high-level Outcome Statement in Figure 3 includes multiple programs for the first and second goals. Also, notice that the leadership program supports the first and second goals as well as the leadership goal. (Note: The goals, priorities, programs, and number of participants in Figure 3 are not identical to those in Figures 1-2 and 4. Consider Figure 3 a stand-alone example.) Talent Det When there are multiple programs in support of one goal, each program should be listed separately if possible. When there are multiple programs in support of one goal, each program should be listed separately if possible. Typically, you will want to also show the combined impact of the multiple programs. For Priority 1, both the impact and number of participants may be summed for the three learning programs that contribute to increased sales. For ease of reading, this combined impact is highlighted in bold and the number of participants is italicized. For this goal, the planned combined impact of the three learning programs is a 6% increase in sales for the 260 participants. Priority 2 illustrates a situation where there are multiple programs but the impact cannot be summed. This may be the case where the stakeholder is uncomfortable quantifying the impact of a particular learning program (New Hire in this example). Since the number of participants can always be summed, it makes sense to retain the lines for combined or total impact of training and simply indicate that the combined impact will be greater (>) than that of the program(s) that can be quantified. Since the leadership program alone is expected to increase the employee engagement score by 2 points, then the combined impact of the leadership and new hire programs must surely be greater than 2 points. The remaining four goals each have a single learning program so the treatment is similar to that in the less complex outcome statements. For more on multiple programs/multiple goals, see the Multiple Learning Programs in Support of One or More Goals piece by the same name under the Resources>Tips and Tools tab on the TDRp website ( Page 8

10 Figure 3 A Complex, High-Level Outcome Statement Priority Business Outcomes and Learning Impact 2011 Actual 2012 Plan 2012 YTD % of Plan 1 Increase Sales Corporate Goal or Actual % 2% 10% 4% 40% Impact of Training Programs Consultative Selling Skills % NA 2% 0.8% 40% Total Participants Number NA % Product Features % NA 3% 1.2% 40% Total Participants Number NA % Leadership Program % NA 1% 0.4% 40% Total Participants Number NA % Total Impact of Training % NA 6% 2.4% 40% Total Participants Number NA % 2 Increase Employee Engagement Corporate Goal or Actual Points -1 pt +4 pts +2 pts 50% Impact of Training Programs Leadership Program Points NA +2 pts +1 pt 50% Total Participants Number NA % New Hire Program Points NA High High Total Participants Number NA % Total Impact of Training Points NA >+2 pts >+1 pt 50% Total Participants Number NA % 3 Increase Leadership Score Corporate Goal or Actual Points -3 pts +6 pts +3 pts 50% Impact of Leadership Program Points NA +6 pts +3 pts 50% Total Participants Number NA % 4 Reduce Injuries Corporate Goal or Actual % 10% 20% 15% 75% Impact of Safety Program % 5% 14% 11.0% 79% Total Participants Number , % 5 Improve Call Center Satisfaction Corporate Goal or Actual Points +1.6 pts +4 pts +2.9 pts 73% Impact of Training Program Points NA Medium High Total Participants Number % 6 Comply with New Regulations Corporate Goal or Actual % Compliance 100% 100% 100% Impact of Training Program Essential Essential Essential Total Participants Number % Page 9

11 A Detailed Outcome Statement A detailed statement will include more granularity, more goals, and/or more frequently refreshed data (monthly or quarterly instead of annual) and may not always include the impact of learning since it is difficult to forecast or measure at a more granular level more frequently. Detailed statements typically will not include a plan for the year, although they could. Talent Det A detailed statement will include more granularity, more goals, and/or more frequently refreshed data and may not always include the impact of learning. Figure 4 is an example with more frequently refreshed data and is a complement to, but NOT a replacement for, the high-level statement in Figure 2. Notice that the percentage changes in business outcome measures and the participant totals for both 2011 actual and 2012 YTD match the totals in Figure 2. For example, in Figure 4 the percentage change in revenues for 2011 is 10%, which matches the 10% in Figure 2. Likewise, the participant count for 2011 is 95 in both tables. Similarly, the 2012 YTD percentage change in revenues is 17% and the 2012 YTD participant count is 124 in both. The detailed outcome statement simply provides the quarterly (could also be monthly) detail behind the annual and YTD totals in Figure 2. In this example, Figure 4 also provides the source data for the percentage calculations, which is not provided in the high-level statement. So, Figure 4 shows that sales were $3,052 million in 2011 and are $1,711 through June The YTD sales of $1,711 for 2012 are 17% higher than the June YTD sales for 2011 [($1711 / ($774+$689) - 1) = 17%]. It is generally better to focus on the change in the business outcome in the high-level statement, but the level could also be shown as part of the goal in Figure 2. (e.g., the first goal could read Increase Sales to $3,662 and the second goal could read Increase Leadership Score to 73.5 ) The monthly or quarterly business outcome measures will typically be compared to their levels the previous period (e.g., 2nd quarter to 1st quarter) or to the level the same period in the prior year (e.g., 2nd quarter 2012 to 2nd quarter 2011). If there is a seasonal pattern to the measure (e.g., sales are always higher during the summer), then it is better to compare to the same period in the prior year. In this example, sales, injuries, and patents will be compared to their year-earlier levels while leadership, call center satisfaction, business acumen, and engagement scores will be compared to the previous quarter levels. The Total and YTD levels will typically be a sum (or an average if the data are annualized) of the individual quarters for measures like sales, costs, and number of injuries. For measures like scores on an employee engagement or quality survey, the Total and YTD levels will typically be the value for the last quarter (e.g., 68.5 for the 2011 leadership score). However, it could be calculated as the average for all the periods (e.g., the average 2012 leadership score of 68.0 could be reported as the 2012 Total). Whichever method is employed, just be sure to tell the reader how the change was calculated. Page 10

12 Figure 4 A Detailed Outcome Statement Priority Business Outcomes Q1 Q2 Q3 Q4 Total Q1 Q2 YTD 1 Revenue Millions of $ $774 $689 $764 $825 $3,052 $890 $821 $1,711 % change over year earlier 7.2% 8.8% 10.7% 13.3% 10.0% 15.0% 19.2% 17.0% Application Rate for Training NA NA NA NA NA 83.0% 87.0% 85.0% Number of Participants Leadership EOS (1) leadership score (points) Change over previous quarter (pts) Time Spent Coaching in Minutes NA NA NA NA NA Number of Participants Safety Number of injuries % change over year earlier -8.5% -10.5% -9.4% -11.6% -10.0% -12.6% -17.2% -15.0% Impact of Training: % Due to Training -4.5% -4.8% -5.2% -5.5% -5.0% -8.8% -12.0% -10.5% Number of Participants Call Center Satisfaction Satisfaction score Change over previous quarter (pts) Impact of Training NA NA NA NA NA High High High Number of Participants Comply with New Regulations % in compliance 100% 100% 100% 100% 100% 100% 100% 100% Impact of Training NA NA NA NA NA Essential Essential Essential Number of Participants Innovation Number of new patents Change over year earlier Impact of Training: None planned NA NA NA NA NA NA NA NA Number of Participants High Business Acumen EOS (1) business acumen score Change over previous quarter Impact of Training NA NA NA NA NA Low Medium Medium Number of Participants Medium Employee Engagement EOS (1) total score Change over previous quarter Impact of Training Low Low Low Medium Low Medium Medium Medium Number of Total Participants Number of Unique Participants Average Classes per Unique Participant Note: 1. EOS is quarterly Employee Opinion Survey Page 11

13 Conclusion Outcome Statements are an essential component of TDRp serving to bring together, on one page, the organization s key goals and the expected impact of learning on those goals. The statement shows both the alignment of learning to the organization s goals and learning s contribution to achieving those goals. After reading an Outcome Statement, there should be no doubt that learning is a strategic partner in achieving the organization s goals. The most challenging aspect of constructing the Outcome Statement is getting agreement with the stakeholder on the contribution of learning to business goals. The senior stakeholder and learning leader must agree on some measure of impact or success to include in the Outcome Statement. Ideally, this impact will be a forecast of the expected isolated impact of learning (either quantitative or qualitative). If that is not possible or practical, then select a proxy. The point is to agree on some measure of impact or success before the program is launched. Once you have the measures, construct your own Outcome Statement following the TDRp format. You may construct a detailed statement(s) if desired, but you must construct a high-level statement. Show multiple programs for a single goal if each program has significant impact on achieving the goal. You can also include the number of participants if you wish. The time spent on the Outcome Statement will pay dividends in planning your learning for the year and in your becoming a valued, strategic partner in your organization s success. Page 12