Decrease Build Testing Time NOT Build Quality. Refael Botbol Director of Professional Services

Transcription

1 Decrease Build Testing Time NOT Build Quality Refael Botbol Director of Professional Services

2 Our Goal To reduce the build tes/ng phase while increasing test coverage Save Time: By paralleling your tests Delegate: Enable each persona contribute to the effort Focus: Concentrate on building not on maintaining tests Scale: Execute builds as many as your business requires

3 Which type of test we are / should run? Func/onal (API / UI) Regressions Cri/cal Business Processes Performance Smoke tests

4 Open Source Tools you probably use for that Func/onal (API / UI) Selenium / JMeter / PhantomJS Regressions Selenium / JMeter / PhantomJS Cri/cal Business Processes Gatling / JMeter / Selenium Performance Gatling / JMeter / Locust / Tsung Smoke tests Selenium / JMeter / Gatling

5 Tes/ng Layers Prod Monitoring Smoke tests Full scale performance tests Save Time Staging Critical business processes Performance tests Integration Functional UI Functional backend Regression tests Critical business processes

6 Tes/ng layers (for example) Integration tests types Functional UI & Backend Regression tests Critical business processes Staging tests type Critical business processes (part 2) Performance tests Production tests type Monitoring Smoke tests Full scale performance tests

7 Divide, Parallel & Conquer Plan your tests per layer and parallel as much as possible Stop the build once a designated cri/cal test has failed and No/fy the developer regarding the issue Free resources for other builds Build once, run many (even when adding new tests)

8

9

10

11

12 Who can contribute to the build s tests? Coders By providing low level tests for API / UI and regressions Delegate Testers By providing system wide tests and regressions Opera/ons By providing cri/cal key points and crea/ng tests with pass/fail criteria for them

13 Who can contribute to the build s tests? Product By providing pass/fail criteria for business processes Delegate Marketeers By comple/ng the product view and providing marke/ng scenarios Performance By implemen/ng performance tests

14 This is how we (can) do it Coders Using code based engines or even YML/JSON DSL Delegate Testers They would probably tell you how J Opera/ons Automa/ng the scripts and providing pass/fail criteria to make sure nothing breaks

15 The how surprising part Delegate Product Providing pass/fail criteria for staging and produc/on environments (and azer several runs, even for the integra/on..) Marketeers Launching a new event? - Why not test is by using a DSL with these pages? Performance These guys already have the scripts and the produc/on values. The only thing that is lez,is to include it.

16 A simple API example Coders / Testers Examples Delegate A complex scenario Java / Python selenium tests JMeter / Gatling tests

17 Create tests and set criteria Product / Opera/ons Examples Delegate Set pass / fail criteria per environment Easily include it as part of your test execu/on

18 Delegate Marke/ng Example

19 Delegate Add a performance tests to your build

20 We can also use the DSL to set pass/fail criteria and describe the test execu/on Short summary First, take a deep breath you have made it so far, you deserve it J We can use DSL based on YML / JSON to create tests We can choose the flavor of tools which best suits us, and reuse our exis/ng scripts

21 Build Once Use Many Focus Op/on 1 : Separate everything Have your scripts represent one test at a /me Have one or more DSL describing the execu/on per build per environment Have one or more DSL describing the pass/fail criteria per environment This gives you maximum agility on controlling your build tests Adding tests is fast Criteria are applied to tests automa/cally per environment

22 Focus Build Once Use Many Op/on 2 : Have clusters of tests Group your tests together base on type, func/onality or any characteris/cs that fits you Abach the right execu/on and criteria per group This is the best op/on for special projects such as A new campaign Satellite projects which aren t part of every build Performance tests

23 Build once use many Focus Op/on 2 : have clusters of tests Group your tests together base on type, func/onality or any characteris/cs which fits you Abach the right execu/on Show option and 1 criteria per grou Best for special projects such as New campaign Satellite projects which aren t part of every build Performance tests

24 Build once use many Focus Op/on 2 : have clusters of tests Group your tests together base on type, func/onality or any characteris/cs which fits you Abach the right execu/on Show option and 2 criteria per grou Best for special projects such as New campaign Satellite projects which aren t part of every build Performance tests

25 Build once use many Focus Op/on 2 : have clusters of tests Group your tests together base on type, func/onality or any characteris/cs which fits you Abach the right execu/on Show a mix of and option criteria 1 & 2 per grou Best for special projects such as New campaign Satellite projects which aren t part of every build Performance tests

26 Use the Force Cloud to Scale Some test types can consume all of your Jenkins Worker s resources Scale Some/mes, the amount of tests is simply too high for one Worker You shortened the build /me and now everyone wants to build

27 One Worker to Test Them All The Jenkins Worker holds the files and configura/ons Scale The Worker start the tests using cloud resources to scale it Results are transimbed to back to the Worker Based on the results the Worker decides: Lunching the next pack of tests Stopping the build due to bad results Mark the build as successful and con/nue to the next step

28 Use the force cloud to scale Some test types can consume a lot resources Scale Some/mes, the amount of tests is simply too high for 1 server Show a YML file with cloud provisioning You ve shorten the build /me, now everyone wants to build

29 Use the force cloud to scale Some test types can consume a lot resources Scale Some/mes, the amount of tests is simply too high for 1 server Show a YML file with cloud provisioning Weight off Multiple scenarios You ve shorten the build /me, now everyone wants to build

30 Enough talk let s rock Workshop /me

31 Gedng Taurus Overall, we install Taurus by using pip. Scale Informa/on per OS can be found here : hbp://gebaurus.org/install/installa/on/

32 Get your Taurus up and running Scale Plug your flash drive Install VirtualBox install vagrant (.dmg file) Open terminal and create a folder Go to that folder and do vagrant box add my-taurus-box file:///d:/path/to/taurus/box.box

33 Get your Taurus up and running Scale On that folder do vagrant init my-taurus-box Vagrant up Vagrant ssh Once you re inside the vagrant, do cd TaurusScript Do: bzt TaurusScenario.yml

34 The Challenge 1. Run a JMeter script with Taurus 2. Create your own (simple) YML DSL file and run it with Taurus 3. Run a Gatling test with Taurus 4. Parallel them J 5. Run your YML DSL with 3 different executors (JMeter, Gatling and Selenium) and see the difference between them Use report to see the BlazeMeter s report

35 Working :me