Prioritize Your Roadmap

Transcription

1 Prioritize Your Roadmap An Optimizely Online Workshop Strategy Consultant: Ryan Lillis August 12th, 2015

2 Introduction Ryan Lillis Strategy Consultant

3 Today s Takeaways

4 Today s Takeaways #1 Why Prioritize & Key Components #2 Tools to Prioritize & Effective Usage

5 Today s Tools

6 Today s Tools #1 Your List of Test Ideas

7 Today s Tools #1 Your List of Test Ideas #2 A Blank Spreadsheet

8 Agenda Key Focus Areas 1. Prioritization Components 2. Components in Action 3. Tools for Organization & Prioritization

9 Three Questions

10 #1 - Poll: Do you have a list of test ideas?

11 #2 - Poll: How many tests are in your list?

12 #3 - Poll: Do you have a prioritization framework and is it followed?

13 Prioritization Components

14 Effort Impact

15 Effort Activity: Write down what takes effort to set up a test.

16 Effort Activity: Write down what takes effort to set up a test. Technical Teams (or People) HTML CSS Javascript Backend changes (reordering pages in a checkout flow) Advanced audience targeting conditions (cookies, ad campaigns, custom tags etc.) Power user(s) Designer(s) Developer(s) Graphic designer(s) Photography team SEO and/or SEM team IT department QA department Executive sponsor

17 Impact Poll: What is the primary way a test s impact is measured?

18 Impact Poll: What is the primary way a test s impact is measured? Hard Impact - Quantifiable Higher Conversion Rate Purchases Sign ups Engagement More Revenue Pageviews Less of Something of Something Internal cost reductions

19 Impact Activity: Write down the primary ways your tests impacts will be measured? Hard Impact - Quantifiable Higher Conversion Rate Purchases Sign ups Engagement More Revenue Pageviews Less of Something of Something Internal cost reductions Soft Impact - Meaningful Internal Buy-in Executive sponsorship Excitement (test ideas from more people) Resource allocation (dedicated developer) Access (places testing is allowed to happen) Implementation (winners actually implemented)

20 Components In Action

21 Basic Framework Effort vs. Impact

22 Effort vs. Impact Priority First Second Third Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low Test Name Effort Impact Test 1 Low High Test 2 Low Medium Test 3 Low Low Test 4 Medium High Test 5 Medium Medium Test 6 Medium Low Test 7 High High Test 8 High Medium Test 9 High Low

23 Effort vs. Impact Priority Prioritized by Impact First Second Third Prioritized by Effort Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low Test Name Effort Impact Test 1 Low High Test 2 Low Medium Test 3 Low Low Test 4 Medium High Test 5 Medium Medium Test 6 Medium Low Test 7 High High Test 8 High Medium Test 9 High Low

24 Chat: What could be done to this framework to make it more useful?

25 Quantify Effort & Impact Use Numbers Instead of Words & Sum the Scores This impact framework Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low

26 Quantify Effort & Impact Use Numbers Instead of Words & Sum the Scores This impact framework becomes this mixed framework. Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low Priority Test Name Effort Impact Score 1 Test Test Test Test Test Test Test Test Test

27 Poll: Who s satisfied with this framework?

28 Objectively Quantify Effort & Impact Consider Your Key Components & Create a Rubric

29 Rubric Method #1 Create a Classic, Detailed Rubric

30 Rubric Method #1 Create a Classic, Detailed Rubric Example Effort Rubric Effort Component HTML No HTML required HTML required but can be done within Optimizely s visual editor Detailed, custom HTML needs to be created = Javascript / jquery No Javascript required Simple manipulation of jquery in Optimizely s Edit Code section Detailed, custom Javascript or jquery needs to be created Graphic Design No graphic design required Quick graphic design element needed Detailed graphic design work needed Executive Sponsor Executive sponsor not needed [intentionally blank] Executive sponsor approval needed

31 Rubric Method #1 Create a Classic, Detailed Rubric Example Impact Rubric Impact Component Conversion Rate = Excitement Implementation 3 Test has a high likelihood of directly impacting our conversion rate. 2 1 Test may or may not directly This test will likely only impact our conversion rate impact a leading indicator but should impact a leading metric. indicator metric. This test will only matter to a This test will generate a lot of This test will interest and few people and get little internal excitement for satisfy some team members. visibility regardless of the testing and build momentum. outcome. If this test has a winning variation it will definitely be implemented immediately. If this test has a winning variation it will probably be implemented or it will take some time before it is. Even if this test produces a winning variation, there is only a small chance it will be implemented due to buy-in or technical constraints.

32 Example Spreadsheet Let s Look at an Example of What a Prioritized Spreadsheet May Look Like

33 Tools for Organization & Prioritization

34 Three Tool Categories Idea Collection Idea Prioritization Workflow Management

35 Three Tool Categories Idea Collection Idea Prioritization Workflow Management

36 Poll: How can test ideas be collected in a way that incorporates rubric scoring automatically?

37 Poll: How can test ideas be collected in a way that incorporates rubric scoring automatically? #1 Impossible! It simply cannot be done!

38 Poll: How can test ideas be collected in a way that incorporates rubric scoring automatically? #1 Impossible! It simply cannot be done! #2 We ll create an exploratory committee to discover this method. We should have an answer when Tesla has a flying car.

39 Poll: How can test ideas be collected in a way that incorporates rubric scoring automatically? #1 Impossible! It simply cannot be done! #2 We ll create an exploratory committee to discover this method. We should have an answer when Tesla has a flying car. #3 Create a test idea submission form that utilizes questions asking what resources, skill-sets etc. are required for a given idea. The responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a score is generated and the list can be sorted by scores.

40 Poll: How can test ideas be collected in a way that incorporates rubric scoring automatically? #1 Impossible! It simply cannot be done! #2 We ll create an exploratory committee to discover this method. We should have an answer when Tesla has a flying car. #3 Create a test idea submission form that utilizes questions asking what resources, skill-sets etc. are required for a given idea. The responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a score is generated and the list can be sorted by scores.

41 So the answer is #3. Now what? Auto-Prioritization Recipe Forms: Google Forms Responses: Google Sheets Scoring: Common Spreadsheet Formulas 1. Create a test idea submission form 2. Include questions about required resources & skillsets etc. required. 3. Modify the spreadsheet with built in formulas that add or subtract points based on the responses. 4. Sort the spreadsheet based on the final prioritization score. = + + Scoring: Common Spreadsheet Formulas

42 Quick Chat: Name one* tool you can use to help with prioritization?

43 Quick Chat: Name one* tool you can use to help with prioritization? Forms: Google Forms Responses: Google Sheets * Google Forms uses Google Sheets automatically.

44 Three Tool Categories Idea Collection Idea Prioritization Workflow Management

45 Idea Prioritization Tools Google Sheets Microsoft Excel Automatically integrates with Google Forms, always shareable in the cloud Tried and true with advanced functionality for graphing etc., can be cloud-based Smartsheet Atlassian s JIRA Set up for seamless project management and timelines, cloud-based Ticketing system incorporating elements of form submission tool and a sortable spreadsheet

46 Three Tool Categories Idea Collection Idea Prioritization Workflow Management

47 Poll: Why would a workflow management tool help with prioritization?

48 Poll: Why would a workflow management tool help with prioritization? Choice #1 - A test can t be the first priority if it needs the expertise of someone who s on vacation. Choice #2 - A test can t be the last priority if it s part of a timesensitive marketing campaign. Choice #3 - Any test idea submitted by my boss is always the top priority. Choice #4 - Both #1 and #2 are correct but definitely not #3.

49 Workflow & Project Management Tools Smartsheet Set up for seamless project management and timelines, cloud-based Atlassian s JIRA Ticketing system incorporating elements of form submission tool and a sortable spreadsheet, also can utilize cards that track the status of each step Trello JIRA One of the most popular testing workflow management tools, incorporates cards, to-do lists and is one of the more visual systems out there

50 Recap

51 Original Agenda 1. Prioritization Components 2. Components in Action 3. Tools for Organization & Prioritization

52 Additional Resources Optiverse contains content on a variety of topics in a variety of formats including our online user community, knowledge base, and learning academy! Community Knowledge Base Learning Academy

53 Thank You For Attending!