Engineering Your Startup to Innovate at Scale. Randy linkedin.com/in/randyshoup

Size: px

Start display at page:

Download "Engineering Your Startup to Innovate at Scale. Randy linkedin.com/in/randyshoup"

Daniel Bruce
5 years ago
Views:

1 Engineering Your Startup to Innovate at Scale Randy linkedin.com/in/randyshoup

2 Background VP Engineering at Stitch Fix o Combining Art and Science to revolutionize apparel retail Consulting CTO as a service o Helping companies scale engineering organizations and technology Director of Engineering for Google App Engine o World s largest Platform-as-a-Service Chief Engineer / Distinguished Architect at ebay o Multiple generations of ebay s linkedin.com/in/randyshoup

3 Stitch linkedin.com/in/randyshoup

4 Stitch linkedin.com/in/randyshoup

5 Stitch linkedin.com/in/randyshoup

6 Stitch linkedin.com/in/randyshoup

7 Combining Humans and Data Science 1:1 Ratio of Data Science to Engineering o Almost 100 software engineers o Almost 100 data scientists and algorithm developers o Unique in our industry Apply intelligence to *every* part of the business o Buying o Inventory management o Logistics optimization o Styling recommendations o Demand prediction Humans and machines augmenting each linkedin.com/in/randyshoup

8 Inventory Styling at Stitch Fix Personal linkedin.com/in/randyshoup

9 Personalized Recommendations Inventory Algorithmic recommendations Machine linkedin.com/in/randyshoup

10 Expert Human Curation Algorithmic recommendations Human linkedin.com/in/randyshoup

11 Data-Driven Experimentation Experimenting with o Algorithms o Client Experience o Stylist Interactions o Growth linkedin.com/in/randyshoup

12 Faster is Better

13 Ability to go fast + Willingness to go fast

14 Capability + Lack of Fear

15 High-Performing Organizations Multiple deploys per day vs. one per month Commit to deploy in less than 1 hour vs. one week Recover from failure in less than 1 hour vs. one day Change failure rate of 0-15% vs % linkedin.com/in/randyshoup

16 High-Performing Organizations è2.5x more likely to exceed business goals o Profitability o Market share o Productivity linkedin.com/in/randyshoup

17 Speed vs. Stability?

18 Faster is Better

19 Innovating at Scale Organizing for Speed What to Build / What NOT to Build When to Build How to Build Delivering and Operating

20 Innovating at Scale Organizing for Speed What to Build / What NOT to Build When to Build How to Build Delivering and Operating

21 Conway s Law Organization determines architecture o Design of a system will be a reflection of the communication paths within the organization Modular system requires modular organization o o Small, independent teams lead to more flexible, composable systems Larger, interdependent teams lead to larger systems We can engineer the system we want by engineering the linkedin.com/in/randyshoup

Small Service Teams Full-Stack, 2 Pizza Teams o No team should be larger than can be fed by 2 large pizzas o Typically 4-6 people o All disciplines required for the team to function Aligned to

22 Small Service Teams Full-Stack, 2 Pizza Teams o No team should be larger than can be fed by 2 large pizzas o Typically 4-6 people o All disciplines required for the team to function Aligned to Business Domains o Clear, well-defined area of responsibility o Single service or set of related services o Deep understanding of business problems Growth through cellular linkedin.com/in/randyshoup

23 Ideally, 80% of project work should be within a team boundary.

24 Autonomy and Accountability Give teams autonomy Freedom to choose technology, methodology, working environment Responsibility for the results of those choices Hold team accountable for *results* Give a team a goal, not a solution Let team own the best way to achieve the goal Innovate and experiment to achieve the goal

25 Innovating at Scale Organizing for Speed What to Build / What NOT to Build When to Build How to Build Delivering and Operating

26 Building the wrong thing is the biggest waste in software development. -- Mary and Tom Poppendieck, Lean Software Development

27 What problem are you trying to solve?

28 A problem well-stated is a problem half-solved. -- Charles Kettering, former head of research for General Motors

29 What Problem Are You Trying to Solve? Focus on what is important for your business Problem might be solved without any technology at all o o o Redefine the problem Change the business process Implement manually for a while before automating in an linkedin.com/in/randyshoup

30 Buy, Not Build Use Cloud Infrastructure o Faster, cheaper, better than we can do ourselves o Stitch Fix has no owned physical infrastructure anywhere in the world Prefer Open Source o Kubernetes, Docker, Istio o MySQL, Postgres, Redis, Elastic Search o Machine learning models o Etc. o Usually better than the commercial alternatives linkedin.com/in/randyshoup

31 Buy, Not Build Third-Party Services o Stitch Fix uses >50 third party services o Logging, monitoring, alerting o Project management, bug tracking o Billing, fraud detection o Etc. Focus on our core competency o Use services for everything else linkedin.com/in/randyshoup

32 Soon it will be just as common to run your own data center as it is to run your own electrical power generation.

33 Experimental Discipline State your hypothesis o What metrics do you expect to move and why o Understand your baseline Run a real A B test o Sample size o Isolated treatment and control groups o No peeking or quitting early! Obsessively log and measure o Understand customer and system behavior o Understand why this experiment worked or did not

34 Experimental Discipline Listen to the data o o Data trumps hope and intuition Develop insights for next experiment Thinking of the experiment is art; evaluating it is science Rinse and Repeat o This is a journey, not a single step

35 ebay Machine-Learned Ranking Ranking function for search results o Which item should appear 1 st, 10 th, 100 th, 1000 th o Before: Small number of hand-tuned factors o Goal: Thousands of factors Incremental Experimentation o Predictive models: query->view, view->purchase, etc. o Hundreds of parallel A B tests o Full year of steady, incremental improvements è 2% increase in ebay revenue (~$120M / year)

36 ebay Site Speed Reduce user-experienced latency for search results Iterative Process o Implement a potential improvement o Release to the site in an A B test o Monitor metrics time to first byte, time to click, click rate, purchase rate è 2% increase in ebay revenue (~$120M / year)

37 Innovating at Scale Organizing for Speed What to Build / What NOT to Build When to Build How to Build Delivering and Operating

38 Prioritization Scarce resources require prioritization o We always have more to do than resources to do it o Opportunity cost -- deciding to do X means deciding not to do Y o Every decision is a tradeoff Priority Return on Investment o Impact / Effort Prioritization is a business decision, not a technical linkedin.com/in/randyshoup

39 Fewer Things, More Done

40 Fewer Things, More Done Maximize resources applied to o Priority 1, then o Priority 2 o etc. Incremental Delivery o Deliver increments along the way instead of everything at the end Deliver Value Faster o Time Value of Money o Benefit now is worth more than benefit in the linkedin.com/in/randyshoup

41 When you solve problem one, problem two gets a promotion.

42 Innovating at Scale Organizing for Speed What to Build / What NOT to Build When to Build How to Build Delivering and Operating

43 Microservices Single-purpose Simple, well-defined interface Modular and composable Independently deployable A B C D E

44 Evolution to Microservices ebay 5 th generation today Monolithic Perl à Monolithic C++ à Java à microservices Twitter 3 rd generation today Monolithic Rails à JS / Rails / Scala à microservices Amazon Nth generation today Monolithic Perl / C++ à Java / Scala à microservices

45 No one starts with microservices Past a certain scale, everyone ends up with microservices

46 If you don t end up regretting your early technology decisions, you probably overengineered.

47 Quality Discipline Quality and Reliability are Priority-0 features o Equally important to users as product features and engaging user experience Developers responsible for o Features o Quality o Performance o Reliability o Manageability

48 Test-Driven Development Tests help you go faster o Tests have your back o Development velocity Tests make better code o Confidence to break things o Courage to refactor mercilessly Tests make better systems o Catch bugs earlier, fail linkedin.com/in/randyshoup

49 Optimizing Developer Effort 75% reading existing code 20% modifying existing code 5% writing new code linkedin.com/in/randyshoup

50 Optimizing Developer Effort 75% reading existing code 20% modifying existing code 5% writing new code linkedin.com/in/randyshoup

51 We don t have time to do it right! Do you have time to do it twice?

52 The more constrained you are on time or resources, the more important it is to build it right the first time.

53 Build It Right (Enough) The First Time Build one great thing instead of two half-finished things Right Perfect (80 / 20 Rule) è Basically no bug tracking system (!) o o o Bugs are fixed as they come up Backlog contains features we want to build Backlog contains technical debt we want to linkedin.com/in/randyshoup

54 Innovating at Scale Organizing for Speed What to Build / What NOT to Build When to Build How to Build Delivering and Operating

55 You Build It, You Run It. -- Werner Vogels

56 Continuous Delivery Repeatable Deployment Pipeline o Low-risk, push-button deployment o Rapid release cadence o Rapid rollback and recovery Most applications deployed multiple times per day More solid systems o Release smaller units of work o Smaller changes to roll back or roll forward o Faster to repair, easier to understand, simpler to linkedin.com/in/randyshoup

Feature Flags Configuration flag to enable / disable a feature for a particular set of users o Independently discovered at ebay, Yahoo, Facebook, Google, etc.

57 Feature Flags Configuration flag to enable / disable a feature for a particular set of users o Independently discovered at ebay, Yahoo, Facebook, Google, etc. More solid systems o o o o Decouple feature delivery from code delivery Rapid on and off Develop / test / verify in production Dark launches Enables experimentation

58 Canaries and Dark Launches Canary Deployment o o Deploy to a small number of systems or users Avoid catastrophic failures Dark Launch o o Execute the code and systems for a feature without displaying results to the user Work out performance bottlenecks and system interactions

59 Innovating at Scale Organizing for Speed What to Build / What NOT to Build When to Build How to Build Delivering and Operating

60 Faster is Better

Thanks! Stitch Fix is hiring! o o o o o www.stitchfix.

61 Thanks! Stitch Fix is hiring! o o o o o Based in San Francisco Hiring everywhere! More than half remote, all across US Application development, Platform engineering, Data Science Please contact me o linkedin.com/in/randyshoup

Leading a Successful DevOps Transition Lessons from the Trenches. Randy Shoup Consulting CTO

Leading a Successful DevOps Transition Lessons from the Trenches Randy Shoup Consulting CTO What Is DevOps? Continuous Delivery? Rapid cycle times Automated testing and Continuous Integration Deployment