Allen Hutchison Google, Inc. November 7, Automated Testing Dos and Don ts

Transcription

1 Allen Hutchison Google, Inc. November 7, 2006 Automated Testing Dos and Don ts Allen Hutchison Google, Inc. November 7, 2006

2 Automated Testing: Dos and Don ts Collection of experiences and observations A little about me: Started at a university Networking and Digital Libraries Moved to the Silicon Valley Worked at a big company as a software engineer in the test automation group Worked at several startups implementing test automation code Now at Google Have published an open source automation framework at: 3 Don t Try To Do It All Yourself Don t automate for the sake of automation We need automation Wouldn t it be cool if we used that tool Do consider outsourcing your test automation Do evaluate tools Consider both commercial and open source / free tools Sales person s goal is to get you to buy the tool Consider tools that are extensible and will allow you to grow Don t let technology slow you down Don t be afraid to throw something away 4

3 Do Start Small Manual assisted testing is a great place to start Load new code on the test bed Set each DUT to a default configuration Look around at the mundane tasks you do every day and start with those Learn from the small scripts What tools do you need How much time does it take to write a script What can you do with your tools, what can t you do Develop a framework one need at a time I wish we could... If we could only... Wouldn t it be great if we could... 5 Start Small # Prepare the test bed foreach(@argv) { my $dut = MyCompany::DUT->new(); $dut->upgrade(); $dut->exec( set password foo ); } 6

4 Do Test One Thing At A Time Tests should be reproducible Tests should run quickly You should see your test fail at least once 7 Do Test One Thing At A Time Tests should be reproducible If you run the same test 5 times and get 5 different results, you might have a problem Performance Test Run 1 Run 2 Run 3 Run 4 Run

5 9 Do Test One Thing At A Time Tests should run quickly Tests should be small enough to only test the functionality you are trying to test Eliminate as much extraneous testing as possible Combine common setup and tear down code You should see a test fail at least once Force your test to fail if you can You should get used to the idea of testing your test 10

6 Evolution Is The Key To A Thriving Ecosystem Do make sure your test suites are constantly evolving Your tests haven t changed in six months, has your code base changed in that time Never trust a developer when he says that he is just fixing bugs. He is also adding features 11 Don t Ignore Failing Tests You get used to seeing all that red Eventually it looks normal Failing Tests don t give you any information, and they desensitize everyone to the red status Red, the color of the day Weeks passed without green Are my tests talking to me? 12

7 Don t Modify A Test To Ensure It Passes Expected and Unexpected Instead of thinking in terms of Pass and Fail at the suite level Consider building a system that is flexible in results reporting Remember our job is to provide information 13 Expected vs. Unexpected # Network Suite File test 24 net/ping test 27 net/route # This test fails right now!test 25 net/ping

8 Do Store Your Test Data You never know when you re going to need it Allows for data mining Learn more about your system When a person dies unexpectedly we do an autopsy Why should your testing be any different Run your tests regularly Continuously With Every Build An idle test machine just wastes electricity 15 Do Consider your Audience Make tests self-explanatory Present consistent metrics Consider your audience Developers Project managers Executives 16

9 Developers Want to have the test code available Want to know what tests passed and what tests failed When, Where, Why, How May not have the time to learn your system That means tests have to be self-explanatory 17 Developers # Test to see if the local IP stack came up my $dut = MyCompany::DUT->new(); $dut->exec( ping ); $dut->expect( 0% packet loss ) or $dut->testfailed( Saw Packet Loss ); $dut->testpassed() 18

10 Project Managers Don t really care about individual tests Don t have time to read code Want to know how the project is going Milestones, Dates, Commitments Need statistics to generate reports 19 Project Managers Milestone Percent Passed Percent Failed

11 Executives Don t have time to read the code Don t care about individual tests Don t have time to read through statistics Need to absorb information in the least amount of time possible Will typically ask questions if they want to know more 21 Executives 22

12 Do Ask Questions? 23 Allen Hutchison Google, Inc.