How we sped up delivery using feature branches. Agilia Conference 2018 Jan Vilímek, Platform Architect Oriflame Software

Transcription

1 How we sped up delivery using feature branches Agilia Conference 2018 Jan Vilímek, Platform Architect Oriflame Software

2 AGENDA About Oriflame Complexity we face Challenges we had when we were growing How to get effectivity and flexibility back? Take away 2

3 BUT FIRST.. A FEW QUESTIONS FOR YOU.. 3

4 Do you use agile for developing software? 4

5 Which source control do you use? 5

6 Are you already using feature branches? 6

7 WHERE ARE WE NOW? About Oriflame Complexity we face Challenges we had when we were growing How to get effectivity and flexibility back? Take away 7

8 Direct selling company 3 millions active consultants Whole globe spread 250+ million online visits per year 8

9 ORIFLAME SOFTWARE ~300 people across Europe Provides IT solutions exclusively to Oriflame Dynamic team with a family spirit in a wonderfull town of Olomouc, branch in Brno Partners in Ukraine, Poland Multinational Center of technology innovations 9

10 DEVELOPING & SUPPORTING Enterprise Resource Planning / Invoicing / Warehouse systems Data Warehouse / Business Intelligence Product catalogues DTP, pricing Digital e-commerce platform See next slides 10

11 E-COMMERCE CONSTANT TRANSFORMATION Web Digital Marketing Digital Business Autonomous Business +Smart Machines Combination of people, connected things, analytics and algorithms, providing a platform for business operations Orico Orisales Online V4 Cumulus 1-2 Servers Farm of servers Elastic cloud computing Operators only (~100s users) Online orders growing 99% orders done online (3M users) 11

12 E-COMMERCE 12

13 SEVERAL MOBILE APPLICATIONS

14 E-COMMERCE TEAMS 14 cross-functional distributed development teams 2 weeks sprint Every team: Front-End & Back-End developers Test Analyst and Engineer Product Owner, Scrum Master, Architect (guilds) Owns specific business domains 14

15 WHERE ARE WE NOW? About Oriflame Complexity we face Challenges we had when we were growing How to get effectivity and flexibility back? Take away 15

16 OUR SITUATION 60 countries (legal / functional requirements) in one monolith Central driven content Many 3 rd party systems (sms / payment providers) Legacy sales system integration 24/7 site availability, many parts (x6 regions) => complex deployment Cloud (specific development) 16

17 SALES PEAK HANDLING Page views per hour in one country One month One day One of the reasons to use cloud.. 17

18 QA ENVIRONMENT Cluster in the South Central US Cluster in East Asia 18

19 WHERE ARE WE NOW? About Oriflame Complexity we face Challenges we had when we were growing How to get effectivity and flexibility back? Take away 19

20 WHEN WE HAD ONLY SEVERAL TEAMS, DEVELOPMENT WAS EASY DEVELOPMENT LIFECYCLE OVERVIEW Smoke and functional testing Regression" testing Live deploy + post deploy support DEVTEST PRE-LIVE LIVE MAIN What was in MAIN branch in time of the merge was eventually pushed to Live Once per 2 weeks full merge Release User Story 1 User Story 2 Is it OK to check-in now or should I wait? Merge based on label Hotfix 20

21 WE HAVE BEEN GROWING More developers, but also testers, analysts More teams Wheel re-inventing How to coordinate? Architect guild was established 21

22 SOFTWARE ARCHITECTS GUILD LEAD DEV Once per week Architect Synchro ARCHITECT Dual track agile Architecture track Delivery track ARCH LEAD DEV MANAGER ARCHITECT LEAD DEV ARCHITECT Architect Lead DEV LEAD DEV 22

23 WE HAVE BEEN GROWING No stable code in Main Needs for formal Quality Assurance Quality gate: User Story can t be deployed to production unless it s tested Part of Definition of Done 23

24 SHOULD WE USE FEATURE BRANCHES? Team Foundation Server No environment for testing Environments too complex (Tamagotchi) So why not do some branching? 24

25 DEPLOYMENT WITH QA GATE Smoke testing Functional testing + acceptance Regression testing Live deploy + post deploy support DEVTEST QA PRE-LIVE LIVE DEV User Story 1 cherry picking QA cherry picking Main new branch Release/HF X.Y User Story 2 25

26 ACTIVITY FLOW: REGULAR DEVELOPMENT Developer Implement and test locally Developer Check-in to Dev branch Developer Ask for CODE REVIEW Test on DEVTEST environment (daily deploys) Cherry picker Merge to QA Task [Merge QA] Done (daily deploys) Quality Assurance Smoke test on DEVTEST environment If OK, assign MERGE task to Cherry picker Developer Code Review OK? Task [Implementation] Done Quality Assurance Full test on QA environment If OK, assign MERGE task to Cherry picker Cherry picker Merge to Main Task [Merge Main] Done

27 IT SOUNDS GREAT SO WHERE IS THE ISSUE? 27

28 TOO MANY CONFLICTS.. Former: Now: MAIN DEV QA MAIN 28

29 Code in Main not stable.. MERGE HELL DEV Ch01 (US01) Ch02 (US02) Ch03 (US01) Ch04 (US02) Ch05 (US01) QA Ch02 (US02) Ch01 (US01) Ch03 (US01) Ch05 (US01) Ch04 (US02) MAIN Ch01 (US01) Ch03 (US01) Ch05 (US01) Ch02 (US02) Ch04 (US02) ChXX: changeset XX USXX: User Story XX 29

30 TIME MATTERS Development is not able to quickly deliver new increment respectively get customer feedback 1 st day Commit to DEV Code review with peer, no changes 2 nd day Deployed to DEVTEST Smoke tests (of the feature + code integration) Merge to QA 3 rd day Deployed to QA Env Functional + integration tests Merge to Main If issue found in DEV => +1 day (+possibly broken environment) If issue found in QA => +2 days (+possibly broken environment) 30

31 3 days to deliver but usually 6 days or more in 10 days sprint Oriflame Cosmetics AG,

32 WHERE ARE WE NOW? About Oriflame Complexity we face Challenges we had when we were growing How to get effectivity and flexibility back? Take away Oriflame Cosmetics AG,

33 HOW TO GET EFFECTIVITY AND FLEXIBILITY BACK? Change starts from the people Retro, Yammer, Slack, s Software Architects Guild Technical solution Scrum Masters Guild VSTS Epic & Feature boards on Global level 33

34 Separated environments per FB FEATURE BRANCHES DEPLOYMENT MODEL Feature full scope testing Code integration Regression testing + acceptance Manual + auto regression testing Live deploy + post deploy support Feature 2 Branches Environment DEVTEST QA PRE-LIVE LIVE Feature 1 Branches Environment Feature 1 merging Feature 2 Master new branch Release X.Y Just one Master branch 34

35 Do you remember? DEV QA MAIN 35

36 DEV QA MAIN NO CONFLICT IN MASTER BRANCH Master Feature01 Feature01 Feature01 Feature01 36

37 TIME MATTERS WITH PREVIOUS SETUP You remember this 1 st day 2 nd day 3 rd day Commit to DEV Code review with peer, no changes Deployed to DEVTEST Smoke tests (of the feature + code integration) Merge to QA Deployed to QA Env Functional + integration tests Merge to Main 37

38 FROM 3-6 DAYS TO 3 HOURS 1 st hour 2 nd hour 3 rd hour Create branch First commit Code review with peer Start build + new environment Deploy changes Smoke test Run integration (auto) tests Functional tests (Quality Assurance ) Merge to Master 38

39 THE BENEFITS 39

40 WHAT HAS THE SOLUTION BROUGHT US? Deployment from 3-6 days to 3 hours Faster feedback from Product Owners / customers => Less feature redevelopment Higher team responsibility End of the merging hell, more stable environments 40

41 WHAT DO WE NEEDED FOR FEATURE BRANCHING? Isolated Continuous Integration environment per feature branch Branch master Process change Teams involvement Mindset change 41

42 CHALLENGES 42

43 FEATURE SIZE MATTERS... How big should a feature be? Environment costs money + time Let s limit environments per team (2-3) 43

44 ISOLATION OF DEVELOPMENT (MOCKS) We needed to mock 3 rd parties Legacy sales systems Sms/ /payment providers It s good: you develop independently Ultimate responsibility for environment => team it s testable But: the mock needs to be developed and kept up to date 44

45 AUTOMATION (SELF SERVICE) Environment (VM, storage ) creation with one click No image, but install scripts Database/storage snapshot Build Deployment Testing 45

46 WHERE ARE WE NOW? About Oriflame Complexity we face Challenges we had when we were growing How to get effectivity and flexibility back? Take away 46

47 Every change come with people Communicate Automate everything Separate concerns, independent development Use feature branches 47

48 Thank you for your attention! Questions? Jan 48

49 49