Automatic Regression Testing Introducing effective testing of graphical user interfaces

Transcription

1 Automatic Regression Testing Introducing effective testing of graphical user interfaces Oscar Lund Christian Strömqvist Department of Communication Systems Lund University Advisor: Carina Andersson January 22, 2007

2 PrintedinSweden E-huset, Lund, 2007

3 Abstract Automated regression testing of software has become a more and more important part of software development. As the competition increases and the size and complexity of software systems grows, higher demands are put on the quality of the test process. The earlier defects can be detected and attended to, the less expensive it becomes for the organization. Today, many companies recruit consultants for manual testing. Introducing well chosen automated tests may greatly reduce development costs and time as well as ensuring a high quality of the product. This report describes different methods to use when introducing automated testing with primary focus on graphical user interfaces. Different test reference models has been reviewed and a number of test tools as well as scripting techniques has been evaluated. Although this subject has been treated in numerous other studies, the focus often lies on the theory and difficulties of graphical user interface testing. It is rare to find actual examples of how these automated tests are implemented. Since both testing tools and methods evolves rapidly, the reports are often outdated and not applicable on the software today. Finally, a test strategy and a test environment which suits the needs and resources of Beijer Electronics AB are presented. In conclusion, Test Process Improvement model is the reference model best suited for Beijer Electronics AB. Furthermore, TestComplete is the test tool recommended and Keyword driven scripting is the choice of scripting technique. i

4 ii

5 Preface This master thesis is the result of a project work at the company Beijer Electronics Products AB, located in Malmö, in cooperation with the Department of Communication Systems at Lund University. It has been 20 weeks of hard labour, but also very interesting, educational and fun. We would like to take this opportunity to thank several people who have had a very positive effect on this Master Thesis. Daniel Björkkvist, our supervisor at Beijer Electronics Products AB. A very patient man and a good mentor. Ola Andersson, our supervisor for the last couple of weeks at Beijer Electronics Products AB. Per Runeson, our supervisor during the first three weeks at the Department of Communication Systems. Carina Andersson, our supervisor at the Department of Communication Systems. Tommy, the chef at Beijer s restaurant for delicious food every single day. All testers and developers at Beijer Electronics AB. Malmö, January 22, 2007 Oscar Lund Christian Strömqvist iii

6 iv

7 Table of Contents 1 Introduction Background Objective Approach Scope Intended audience Glossary Outline of this report Survey Exploring Interview Results Questionnaire Interview Reference models TMM - Test Maturity Model TPI - Test Process Improvement Key Areas MTPF - Minimal Test Process Framework Responsibility Areas Summary Test tool TestComplete TestPartner AutoMate QuickTest Summary Scripting Techniques Linear scripting v

8 5.2 Structured scripting Shared scripting Data Driven scripting Keyword Driven scripting Summary Suggestion Reference model Test tool Scripting Technique Implementation Overview Test Cases Controller Supporting Scripts Projects Scripts Summary Test Support Controller Script Summary Conclusion Proposition evaluation Prototype evaluation Future improvements References 57 A Questionnaire 59 A.1 Organization Stage A.2 Tool Evaluation A.3 Tool Features Evaluation A.4 Identifying Test Cases A.5 Programming language A.6 Test/Trouble Reporting A.7 Final Comments B Interview Questions 67 B.1 Organization Background B.2 Maturity identification C Implementation 71 C.1 Controller script C.2 Supporting scripts C.3 Project Scripts vi

9 Chapter 1 Introduction 1.1 Background Beijer Electronics AB is, like many other small but growing companies, in a phase where the next natural step in testing and development is the introduction of automatic regression testing. As of today, there exists a limited number of reports and studies regarding automatic regression testing with focus on graphical user interfaces. Furthermore, these are often outdated, i.e. not applicable on today s software technologies. Others are too theoretical and therefore sometimes very difficult to apply in practice. In the present, Beijer Electronics AB has no automatic testing at system and acceptance level. The company rely on manual testing to ensure the quality of their products. Manual testing is currently performed in the end of a product s development process and do not receive as much time as it should, sometimes due to delays in the development. The goal of introducing automatic testing is to be able to perform tests as early as possible in the software development process. Beijer Electronics AB produces builds of their software on a daily basis. By testing these builds with some selected automatic regression tests every time, time is saved since certain defects are detected immediately. In this way and by having most regression tests run automatically, the organization ensures a higher quality of their product. Beijer Electronics AB develops and produces human machine interface products, i.e. terminals controlling industrial robots and machines. The terminals communicates directly with PLCs integrated with the machines. These graphical terminals is delivered to customers without any PLC controlling software installed. Instead the terminals are delivered with an application that allows the end user to build their own graphical user interface to control the PLC. At Beijer Electronics AB, this application is referred to as Neo. Neo is a design tool, similar top Visual Studio developed by Microsoft [?]. It is developed with Microsoft.NET 3.0 [?]. This is a new framework which very few test tools have support for. Neo offers the possibility to design and implement an application with buttons, text fields, meters etc. The user can then compile the application and transfer it to the terminal. In this way, a client designs a graphical user interface suiting the needs for different industrial machines or robots. A screenshot of Neo can be found in Figure

10 2 Introduction Figure 1.1: Screenshot of Neo build 920 To ensure the quality of Neo, Beijer Electronics AB is interested in introducing automatic regression testing as an additional quality assurance to the test process. This study takes place in parallel to the development of Neo and is executed with continuous feedback from testers and developers. The idea is that the developers will be of assistance during the implementation of the test environment. For a deeper description of how this assistance works in practice, refer to Section 7.2. A very important reason for choosing automatic regression testing over manual testing is that automatic regression testing offers the possibility to test a certain functionality in the exact same way over and over again. This helps in recreating an error and thereby identifying why the error occurs as well as ensuring if a defect has been attended or not. This is hard to do by manual testing since it is impossible to repeatedly perform a test case exactly in the same way.

11 Introduction Objective The main objective of this report is to offer a both theoretical and practical guide on how to introduce automatic regression testing in small size software companies. The concept of this guide has been that it should be easy to understand and be relatively manageable to apply and implement. This study presents a strategy on how an organization can evaluate and introduce test strategies, test tools and test structures suiting their specific needs. As a case study, Neo, developed at Beijer Electronics AB has been used. 1.3 Approach Thefirststepistoidentifyandformulatetheproblem. Thenextstepistoacquirethebasic knowledge needed to attack the problem in an effective way. This research incudes identifying several important areas, such as the choice of reference model, test tool and scripting techniques. To motivate the choices made when selecting reference model and test tool, a preceding research is made. By handing out forms to employees as well as having interviews and discussions with group leaders, the understanding for the company is increased and helps in giving a more well-founded proposition fitting the company s needs and goals. The results of the questionnaire is intended to give a basic understanding of the level of knowledge and experience of employees, as well as the expectations of the test tool being chosen. The evaluations of reference models, test tools and scripting techniques are made solely by the authors of this report. However, considerations form the results of the questionnaire and the interview are taken into account. Figure 1.2 shows the work procedure used to acquire the information needed to make a well-grounded decision. The initial proposition presented is by no means final. As feedback is received, the proposition is reviewed and modified. This is repeated continuously until both sides mutually agrees to a final version of the test strategy. As the final proposition is formed, the support state is entered and a test strategy is defined and presented to the company and all personal concerned. The final steps are implementation and evaluation. As the arrows in Figure 1.2 applies, there is an open and continuous dialog with all parts concerned throughout the entire process of the introduction of automatic regression testing.

12 4 Introduction Explore Survey Interview Reference Model Test Process Improvement Testabilty Maturity Model Test Maturity Approach Minimal Test Practise Framework Suggestion Feedback Review Support Implementation Evaluation Figure 1.2: Work flow 1.4 Scope This report is not to be seen as a complete reference to automatic regression testing, but rather as a guide when introducing automatic regression testing in a test environment of a small or medium-sized organization. To get a deeper understanding for certain areas like reference models and test structures, it is recommended to study the literature in Chapter References. The choices made in this report focus on Beijer Electronics AB and may not be the right solution for another organization. The implementation of the test environment is to be seen as a prototype. The number of test cases created is very limited and basic, but are intended to function as a guideline for the implementation of other and more advanced test cases. 1.5 Intended audience This report is intended for higher education students with focus on software development and basic knowledge in software testing. This study also is of interest for smaller or medium-sized companies aiming to introduce automatic testing in their test environment.

13 Introduction Glossary Subject TMM TMap TPI MTPF Regression Testing Explanation Test Maturity Model Test Management approach Test Process Improvement Minimal Test Practice Framework Retesting of a previously tested program, to ensure that modifications has not caused new defects. Table 1.1: Glossary 1.7 Outline of this report The first part of this report, Chapter 2 Survey, describes the exploring part of this master thesis. This is used to get an understanding for the organization as well as formulating and identifying the problem. The second part, Chapters 3, 4 and 5, evaluates reference models, test tools and scripting techniques respectively. The third part, Chapter 6 Suggestion, presents the results of the evaluations. Here are the choices made and motivated considering reference model, test tool and scripting techniques. The forth part, Chapter 7 Implementation, illustrates the actual implementation of the test structure. This is the result of all preceding chapters and ideas. The final part, Chapter 8 Conclusion, evaluates the prototype from both an own perspective and tester feedback view. This chapter also includes future improvements.

14 6 Introduction

15 Chapter 2 Survey The first natural step when introducing automatic regression testing is not only to identify a reference model suiting the organization s current level of maturity of the test process, but also to setup a test tool fitting the company s needs with cost, usability, functionality etc. taken into consideration. To be able to make these decisions, an extensive analysis of the organization is needed. With well-grounded and relevant questions to all employees, concerned with the test process, the idea is to create a test strategy and test environment, suitable for the organization. In this chapter the process that lies as a foundation to the proposition is described. If the management approves of this suggestion, it is used as a starting point of how the company will work with testing, test improvements and test planning. In Section 2.1 Exploring, the process of gathering the information needed to get a clear image of what the organization expects and requires, is described. In section 2.2 Interview, recommended questions for test management for understanding the level of maturity of the organization, is discussed. 2.1 Exploring To achieve a satisfying and a well functional test strategy fitting the organization it is of great importance to get a deep understanding for the activity. By having insight in the organizational structure, how the testers are working and what knowledge and experience they have as well as their expectations of the test process, a better and more well-founded decision regarding test strategy can be made. It is also important that the test strategy with minimum efforts can be integrated in the current organizational structure. The advantage of this thorough organizational understanding is minimizing the risk of time consuming and resource absorbing reconstructions. As a positive side-effect a greater trust is built among the employees, mainly because the test strategy presented is well-founded and somewhat familiar. A first step is collecting information about the activity. This is done preferably with a questionnaire sent to all employees concerned with the test process, for example software quality assurance managers, testers and developers. The main purpose with the questionnaire is to get an understanding of what test personal expects and demands of the test tool they eventually will be working with. Furthermore it is important to get an insight on how the test environment is structured initially, regarding experience, knowledge and work assignments of the employees. [1] 7

16 8 Survey The questionnaire is found in Appendix A. The first section of the questionnaire treats organizational questions with focus on the individual tester s experiences and views on the corporation. These questions do not lie as a foundation when choosing reference model, but rather gives an indication on what level a test process can be established. The questions is mainly fetched from Burnstein [1], but are reworked to be relevant for smaller companies like Beijer Electronics AB. The second and third section of the questionnaire treats the needed functionality and features of the test tool. The answers are the very foundation for the choice of test automation tool. A thorough evaluation of different test tools is made, where various functionalities and features is rated, see Chapter 4. These questions gives an idea of what properties of the test tools that is of highest importance for the testers. Examples of areas the questions handles are support, cost, stability and functionality. In chapter 4 the results are evaluated and in Section 6.2 is explained in detail how they have affected the choice of test automation tool. The questions in chapter 1 and 2 in the questionnaire are collected from Burnstein [1] and the questions in chapter 3 are from Fewster [2]. The remaining questions is the results of our own experience in software testing and to some extent feedback from software testers at Beijer Electronics AB. With a test automation tool one can create many different types of tests. This often requires some programming skills regarding the construction of the tests. This is treated by the questions in chapter 5 in the questionnaire, as what programming languages the testers have experience from and prefer. This, of course, also inflict on the choice of test tool. In the end of the questionnaire there is a chapter where the concerned employees have the opportunity to leave comments regarding test and defect reporting. These comments are a factor when choosing a reference model, since it is important to know how everything works in the present and if there is a possibility to make improvements. Finally there is a final comments field where the readers can add thoughts on the proposition and if they feel that there is something left out. By taking the feedback from the testers into consideration and letting them affect the outcome of the new test strategy, it is easier to carry out all major changes since all employees feel that they are part of the new solution. Hopefully this leads to that employees feel a responsibility to make sure the new reorganization really works. 2.2 Interview The questionnaire gives an understanding of the skills and the expectations of the employees at the test department. To get a deeper understanding of the organization, which level of maturity it is in, as well as what the management expects, an open discussion with the test managers is required. The discussion is based on several interview questions which can be found in Appendix B. These questions is primarily based on suggested interview questions from Burnstein [1]. Questions irrelevant for Beijer Electronics AB has been removed. The results will underlie the test strategy which is to be introduced in the company. An analysis of the answers can be found in section 6.1, as well as a deeper explanation on how they have affected the choice of reference model. The first part of the interview, Organizational background, treats more practical questions, such as number of full-time and part-time employees. This is of interest when deciding a reference

17 Survey 9 model fitting the organization s size. Second and last part of the interview mainly treats questions that will be a foundation when identifying which level of maturity the company has achieved. The questions has been fetched from a recommended template, developed by Burnstein [1] for reference model TMM. However, MTPF, TMM and TPI all include the process of identifying the maturity of the organization or test process. In this study, all interview questions are selected to be relevant for all three test frameworks. This part is divided into three smaller parts. The guideline for the questions is that if more than half of the answers in each part are positive, than that part s level of maturity is achieved. If more than half of all parts has reached maturity, then the total maturity level is considered achieved. [1] First part treats Testing Goals and Policies. NextpartfocusesonTest Planning Process. Final and last part treats Testing Techniques and Methods. 2.3 Results Questionnaire The questions handed out to the seven employees were answered and summarized in tables. The majority of all testers, including all test managers, have been involved in this research. From the development department, two senior software architects were selected. These seven employees were considered to have the knowledge and insight to represent the company. The reason for selecting more testers than developers is that the test department should have more to say in this matter. To get a picture of the organizational structure the first two questions focused on the individual work position of the employees, as seen in Table 2.1 and Table 2.2. Each cross represent an answer from an employee, while the last column represent the average. As Table 2.1 illustrates, approximately one third of those asked are developers and the rest is part of the test department. Which best describes your current position? Software Quality Assurance Group Leader x x 29% Software Quality Assurance Group Member x x x 43% Software Engineer (developer) x x 29% Table 2.1: Employee position Table 2.2 does not only show the responsibilities of the employees, but also hints of which responsibility areas within the test process that are covered.

18 10 Survey What is your current test-releated duties? Test policy and goal development x 14% Test planning x x 29% Test case and test procedure design x x 29% Test execution x x x x 57% Maintenance of case database 0% Standards development 0% Reviews and audits 0% Status tracking 0% Test training 0% Hiring and recruiting x 14% User/Client communications x 14% Test process control x 14% Test process evaluation x x 29% Test process improvment x x 29% Defect prevention 0% Usability testing x x 29% Tool evalutaion x x 29% Table 2.2: Employee responsibilities When evaluating different test strategies it is important to know what knowledge the testers have when it comes to different reference models. As the Table 2.3 implies, the Test Process Improvement Model, TPI is most widely known among the employees. TPI is also currently used as a reference model at Beijer Electronics AB. Are you familiar with any of the following test practise frameworks. TMM (Test Maturity Model) x x 29% TPI (Test Process Improvment model) x x x 43% MTPF (Minimal Test Practice Framework) 0% TMap (Test Management approach) x x 29% Table 2.3: Reference models The next question treats the amount of work experience there exists among the employees. The Table 2.4 indicates that the testers of Beijer Electronics AB are well trained and experienced. Therefore no major difficulties is expected when introducing a more complex test environment. The basics of testing has also been left out of the proposition and this study, since it is considered that the target group already has acquired this knowledge.

19 Survey 11 What is the extent of your experience in software testing? Less then 1 year 0% 1 to 2 years x x 29% 2 to 3 years 0% More than 3 years x x x x x 71% Table 2.4: Work experience The next question, as seen in Table 2.5, gives an idea of how structured the test process is in the present. This gives a hint of difficult it can be to introduce a highly structured test environment, compared to the current structured level. As seen in Table 2.5 most of the employees consider the test environment to be somewhat structured, which helps when introducing automatic regression testing. How would you generally characterize the nature of your testing process? Ad Hoc 0% Informal x 14% Somewhat structured x x x x x 71% Highly structured x 14% Table 2.5: Level of structure When evaluating test automation tool it is important to know how the testers prioritize different properties of the tool. As Table 2.6 shows, the most prioritized properties are Ease of Use and Functionality. Later on, when evaluating test tools, see Chapter 4, the average score of each property is used as a weight. For a description of each property, refer to Table 4.2. The Functionality category consists of several different tool features as described in Table 2.7. The employees rated the importance of each property on a scale from 1 (the lowest) to 7 (the highest). Tool Evaluation Ease of use ,1 Power ,7 Robustness ,6 Functionality ,1 Ease of insertion ,6 Quality of Support ,3 Cost ,3 Table 2.6: Tool property evaluation As seen in Table 2.7, the most valued features are GUI testing and Script support, whichwas expected. The employees rated each feature on a scale from 1 (the lowest) to 5 (the highest).

20 12 Survey Tool Features Evaluation Remote handlig ,1 Recording ,1 Native.NET testing GUI Testing ,7 Script support ,3 Plugin support ,6 API support ,6 COM support Distribuation of test reports ,6 Compare test results ,6 Other Table 2.7: Tool functionality evaluation The next question treats specific test areas for the product (Neo) being developed at Beijer Electronics AB. It is not always the case that these areas are monitored together with automatic regression testing. However, it is of interest to find out if there is a need to cover these test areas in the test environment. As seen in Table 2.8 memory leaks has a high priority to be tested. The employees rated each area on a scale from 1 (the lowest) to 5 (the highest). Identifying Test Cases Execution time of NEO ,1 Execution time of generated binaries Response time of NEO ,6 Response time of generated binaries Memory usage of NEO ,7 Size and memory usage of generated binaries Memory leaks in NEO ,1 Memory leaks in generated binaries ,5 Table 2.8: Additional test areas The last two questions determines the level of experience and skills testers have in different scripting languages. This can be of value when choosing and evaluating test automation tool, depending on how test cases are created, for example by recording or writing test scripts. As seen in Table 2.9 and 2.10 the testers have the most knowledge and skills in Visual Basic script, which can be worth noticing when evaluating test automation tools. The employees rated their experience and knowledge in each programming language on a scale from 1 to 5, where 1 is no knowledge and 5 is experienced.

21 Survey 13 Programming language (prefer) VBScript x x x x 57% JScript 0% C#Script x x 29% DelphiScript x 14% Table 2.9: Preferred programming language Programming language (skills) Visual Basic ,7 Java ,4 C# ,2 Delphi ,2 Table 2.10: Programming skills Interview The interview took place approximately five weeks from project start. Fredrik Persson, Software Quality Assurance Manager, was the objective for the interview. The interview is intended to give an understanding of the organizational structure and current maturity level. This helps when deciding an appropriate reference model and to what level of difficulty the test environment could be built. All questions in this interview were gathered from Burnstein [1], with some questions slightly modified. The number of questions are also decreased, so that only the most relevant questions, suitable for a smaller company, are used. The first section of questions according to Table 2.11 treats size and structure of the organization. According to Burnstein [1] these questions are important when identifying the maturity level and deciding possibilities of improvements. As seen in Table 2.11, the human resources in testing of Beijer Electronics AB is rather limited. It is therefore important to develop a test system which do not requires to much costs and resources in maintenance. By introducing automatic regression testing a large portion of the time caused by manual testing can be avoided and thereby allowing testers to concentrate on less tedious tasks. Accordingly to Table 2.11, compared to software development, a lot of resources are put into testing. However, it is important to emphasize that testers not only works with software testing, but also performs hardware and driver testing.

22 14 Survey How many people are employed in the organization being assessed? Total employees 70 Number engaged in software development 12 Number engaged in software testing 7 Please describe the percent of staff engaged in testing as follows? Fulltime 43% Part time 14% Consultants 43% Are the responsibilities for test process improvement clearly defined and supported? Very well defined How is the testing group organized? Developers do the testing Test group within development report to project manager Separate test group, report to test manager Part of Software Quality Assurance group No No Yes Yes Table 2.11: Organizational background Next three sections focuses on identifying the maturity level of the company. Questions seen in Table 2.12 focus on testing and debugging goals. It can be stated that the testing goals and policies are either well defined or under development. This means that the organizational basis for a functional test environment has been established. Develop testing and debugging goals and policies Has a committee on testing and debugging been established? Have policies, goals, activities, and tools for the testing processbeen identified, documented and approved? Do the software testers follow a written organizational policy for testing when test planning? Are basic measurements used to determine achievement of testing goals? Have the testing policies and goals been developed with inputs from user/client groups with respect to their needs? Do developers/testers log defects into the repository on a consistent basis? Are testing/debugging policies and goals periodically reviewed? Yes Under development Under development Yes Yes Yes Yes Table 2.12: Maturity identification, part 1 Next section is concentrated on test planning. According to Table 2.13 there is room for some improvements in the area of test planning.

23 Survey 15 Initiate a test planning process Has an organizationwide test planning commitee or group been established? Are there procedures for test planning? Is there adequate support and funding for test planning for all projects? Are test goals/objectives used as a basis for test planning? Have test plan templates been developed? (for earlier projects?) Are there appropriate planning tools available for test planning? Are test managers trained properly to develop test specifications, test designs and test cases? Are testers trained properly to develop test specifications, test designs and test cases? Are test-related risks considered when developing test plans? Are estimates (time, budget, tools) available from past projects for use in test planning? Is test planning done at the acceptance level? incident reports, and test summary reports defined in organizational documents? Are other test-related items such as test transmittal reports, test logs, test incident reports, and test summary reports completed for each project? Do testers collect and store basic test-related measurements? Are the basic test measurements used to ensure that basic testing goals have been met? No Yes Yes No Yes Yes Yes No Yes No No Yes Yes No Yes Table 2.13: Maturity identification, part 2 Final section of the maturity identification questions evaluates the current situation regarding different test techniques and methods. As Table 2.14 shows, quite a few improvements are needed here also.

24 16 Survey Institutionalize basic testing teqniques and methods Has a committee or group been formed to evaluate and recommend a set of basic testing techniques, methods, and tools? Have the recommendations of the group been documented, distributed, and approved? Have basic tools and techniques been included in test policies? Have appropriate forms and templates been designed to support basic testing techniques? Are adequate resources provided by upper management to support the use of basic testing techniques and methods, as well as basic testing tools? Have testers been trained to apply the basic tools, forms, and methods? Are basic testing techniques, and methods applied on an organizationwide basis? Are basic testing techiques and methods reviewed periodically? Is testing planned and implemented at multiple levels (unit, integration, system, etc.)? Yes No No No Yes Yes No Yes No Table 2.14: Maturity identification, part 3 From the results of these questions a reference model is chosen and the maturity level is identified. This is presented in Section 6.1.

25 Chapter 3 Reference models This chapter gives an overview of a selection of widely used reference models. The choice of reference model describes how the company or organization will work with the test environment. In this chapter, a number of different test improvement models are treated. The focus in this study is on the models TMM, TPI and MTPF. All test improvement models strives for a more structured and effective test process and organization. According to Pol, the goal of a test improvement model is a test process method called TMap. For further reading on TMap, refer to Pol [3] 3.1 TMM - Test Maturity Model TMM or Test Maturity Model is a test strategy built upon five levels of maturity. When a test process has achieved a number of specified demands on one level, the test process can be considered being on that level. [1] An organization that does not have any testing or a highly unstructured test environment mainly based on ad hoc is in the initial state at level 1. For a company to climb to the next level it is required to introduce a more structured testing. This involves different test techniques, like White-Box and Black-Box testing, but also development of test- and debugging goals as well as atestplanningprocess. To climb further up in the level hierarchy additional measures are needed. In term this means that testing no longer is a part of the software development process, but rather an independent process starting as early as the software development itself. This means introducing different test activities at various states in the software development and having skilled groups established, each responsible for different parts of the test process, such as planning, execution and evaluation. 17

26 18 Reference models Level 5: Optimization/Defect Prevention and Quality Control Test process optimization. Quality control. Aplication of process data for defect prevention. Level 4: Management and Measurement Software quality evalutaion. Establish a test measurement program. Establish an organizationwide review program. Level 3: Integration Control and monitor the testing process. Integrate testing into the software life cycle. Establish a technical traning program. Establish a software test organization. Level 2: Phase Definition Institutionalize basic testing techniques and methods. Initiate a test planning process. Develop testing and debugging goals. Level 1: Inital Figure 3.1: TMM maturity level structure A smaller company or an organization with limited resources allocated for testing, may have trouble reaching level 2 and even more difficulties reaching higher levels. The magnitude of these steps not only makes it difficult and demanding but also less motivating. 3.2 TPI - Test Process Improvement Test Process Improvement, TPI, is a test practise framework developed with the test method TMap [3] in mind. The main purpose is to allow a more balanced test improvement strategy. This means that companies continuously works with the improvements and can follow the levels of maturity through a so called test maturity matrix [4], see Figure 3.2

27 Reference models 19 Key Area Scale Test strategy A B C D Lifecycle model A B Moment of involvement A B C D Estimating and planning A B Test-specification techniques A B Static test techniques A B Metrics A B C D Test automation A B C Test environment A B C Office environment A Commitment and motivation A B C Test functions and training A B C Scope of methodology A B C Comunication A B C Reporting A B C D Defect management A B C Testware management A B C D Test process management A B C Evaulation A B Low-level testing A B C Controlled Efficient Optimizing Figure 3.2: TPI Test maturity matrix The matrix consists of twenty different key areas, each representing areas all open for continuous improvements. Every area has a number of maturity levels. The grading goes from A to D, wherea is the lowest level and D is the highest level. Each maturity level is placed on a scale from 1 to 13, illustrating the significance of fulfilling every individual level. The space in between the different levels describes the relation to other key areas. It is a wise choice to have different levels of maturity evenly distributed over the matrix. For example, it is not particulary effective to have a high level of evaluation without a well defined test strategy Key Areas This section gives a short summary of all the key areas in the TPI maturity matrix, see Figure 3.2. For further explanations, refer to [4]. Test strategy This area focuses on how well defined the test strategy is on different levels of testing. It is important to detect defects in a stage as early as possible. Life-cycle model A test process consists of different phases, such as planning, preparation and execution. Each phase contains predefined activities. Worth mentioning are purpose, techniques and tools. When using a life-cycle model one gets a much better overview of the test process and thereby can plan

28 20 Reference models and execute it in a more efficient way. Moment of involvement Defines when the test process should start in relation to the software development. The test process consists of more than just the test execution. Estimating and planning Defines when different activities within the test process is to be carried out and what necessary resources that should be available. Test-specification techniques Describes how different test cases are generated from various testing goals. Static test techniques Static tests means that inspections of the product are made without running it, for example by reviewing the source code. Metrics By continuously examine the product and the test process, metrics is introduced. This gives an opportunity to measure the quality of the entire test process. Test automation Describes to what degree supporting tools is used to automate different parts of the test process. Test environment Describes to what extent hardware, software, communication means, facilities and procedures fulfill the needs of the testers. Office environment The test staff need adequate working conditions to be as efficient as possible in test process. Commitment and motivation Without a committed and motivated test staff, an efficient test process can not be expected. There are several ways to increase the commitment and motivation, for example by assuring thereisadequatetimeandresources. Test functions and training It is important to have a set of well-defined test techniques, test methods and test functions, as well as experienced and skilled testers with concrete roles within the test team. Scope of methodology For every test process in an organization a specific test method is used. If these test methods differs from one process to another, this results in a decreased efficiency. It is desirable to use a sufficient generic method that can be applied on each test process, but still contains enough detail to be efficient each time.

29 Reference models 21 Communication The communication between the people involved in a test process are extremely important, not only for creating good conditions, but also to ensure the quality of the product Reporting Testing is mainly about quality assurance. Reporting aims at giving the client an insight in the test process and the quality of it. Defect management Administration of defects is important to be able to follow the life-cycle of a reported error and thereby analyze the quality trend in detected defects. Testware management Version control systems, structuring of test plans and specifications are important parts of a testware system, to make it run as efficient as possible. Test process management There are several steps and activities to be carried out in a test process, to make it well-structured and effective. The most important are planning, execution, checking and action. Evaluation By continuously inspecting intermediate products, for example requirements and functional design, defects can be detected at an earlier stage. Low-level testing Low-level testing, such as unit tests and integration tests, are performed exclusively by developers. This is important in helping finding defects as early in the test process as possible. 3.3 MTPF - Minimal Test Process Framework Smaller and medium sized companies often have difficulties in applying well known and large reference models. These are in most cases intended for larger organizations with resources, development teams and test groups. For smaller expanding companies it can be difficult to apply the recommendations suggested in other test improvements models to increase in level of maturity. Withthisinmind, MTPF was developed, which can be seen as a simpler alternative to other more known test improvement models. MTPF was developed by Karlström, Runeson and Nordén and has been evaluated by local companies in Lund and Malmö, Sweden. [5]. MTPF is a model suited for growing companies mainly practicing Ad Hoc in their testing environment. As Figure 3.3 shows, the test improvements are applied as the company is growing. Thereisnospecifiedtimeplanwhenadvancementsshouldbedone. Theseareperformedonly when the organization feel they are ready and all requirements are fulfilled.

30 22 Reference models Phase 3 (30+) Phase 2 (~20) Maintain & Evolve System Create System Define Teams Perform Inspections Risk Management Define Goals Perform Walkthroughs Test Cases Coordinate Software Quality Assurance Phase 1 (~10) Define Syntax Define Responsability User Checklists Basic Administration of Test Environment Test Plan Category Problem & Experience Reporting Roles & Organizations Verification & Validation Test Administration Test Planning Figure 3.3: Overview of MTPF Responsibility Areas In this section, the different areas in the MTPF matrix, according to Figure 3.3 is described. Problem and experience reporting Phase 1 Phase 2 Phase 3 Define Syntax Define a standard and a syntax to follow when reporting defects. This minimizes the risk of misunderstandings. Create System Introduce a centralized system for defect reporting. Maintain & Evolve System Verify that error reporting works satisfying and continuously improve the measures taken in the earlier phases, to further raise the quality. Table 3.1: Problem and experience reporting

31 Reference models 23 Roles & Organization Phase 1 Phase 2 Phase 3 Define Responsibility Testing responsabilty is defined within the company. Define Roles The roles for the manager, testers and developers are assigned. Everyone has clear responsibilities. Define Teams An independent test team is created and separated from development. Table 3.2: Roles & Organization Verification & Validation Phase 1 Phase 2 Phase 3 Use Checklists Checklists are used to perform the most important activities such as GUItesting and platform testing. Perform Walkthroughs Walktroughs should be done before software is ready for execution. This is preferably done in the design phase. Perform Inspections Inspections is more often used rather than walkthroughs. When an organization grows, a more formal method for evaluation is required. Table 3.3: Verification & Validation Test Administration Phase 1 Phase 2 Phase 3 Basic Administration of Test Environment Test environment must always be available. Administration is responsible for test environment for each and every single project, as well as documentation and updates. Test Cases Test cases is developed to ensure that the most common situations and activities are handled in a correct way. Risk Management Risk managment focuses on identifying problems and risks in the beginning of each project. Table 3.4: Test Administration

32 24 Reference models Test Planning Phase 1 Phase 2 & 3 Test Plan A test plan is created to gather all test related information. Coordinate Software Quality Assurance Routines to ensure quality is introduced. This assures that software has reached a certain level of quality before release. Table 3.5: Test Planning 3.4 Summary In this chapter three different test process improvement models has been reviewed. As already concluded it is important to choose a reference model fitting the company s needs. In chapter 6 a reference model is chosen based on the conclusions made in this chapter. In other words, this chapter only gives a review of each test framework and in chapter 6 the models are summarized and the most appropriate is recommended.

33 Chapter 4 Test tool In this chapter different test automation tools are described. A thorough review of each test tool is done in order to make a decision, of course with some requests from the employees in mind, of which test tool that will be used in the automatic regression testing environment. All the evaluated test applications are specialized in Graphical User Interface (GUI) testing. A summary of the test tool evaluation is presented in Figure 4.1. TestComplete were evaluated first and the ratings were set relatively to the expectations. These values were then considered to be reference values and the remaining three tools were rated relatively to TestComplete. The summarized rating for each test tool is calculated by Formula (4.1), where r i is the rating for a category and w i is the weight for that category. ri w i S = (4.1) wi 25

34 26 Test tool Category Test Complete Test Partner AutoMate 6 Quick Test Weight Ease of use ,1 Power ,7 Robustness ,6 Functionality 4,1 3,4 2,9 3,1 6,1 Ease of insertion ,6 Quality of support ,3 Cost ,3 Summary 3,6 2,8 3,4 2,8 Functionality Test Complete Test Partner AutoMate 6 Quick Test Weight Remote handling ,1 Recording ,1.NET GUI-Testing ,7 Script support ,3 Plugin support ,6 API support ,6 COM support Distribution of reports ,6 Compare ,6 Average 4,1 3,4 2,9 3,1 Table 4.1: Test Tool Evaluation. Scale 1 to 5 The ratings found in the category Functionality comes from the results of the functionality evaluation, see second part of Table 4.1. The column Weight is the average of values fetched from the questionnaire, i.e. how the testers rated each and every category and functionality. An explanation of each category can be found in Table 4.2

35 Test tool 27 Category Ease of Use Power Robustness Functionality Ease of Insertion Quality of Support Cost Description The tool has an intuitive interface, which makes it easy for an unexperienced user to be able to work with it. How well the program handles both predictable and unpredictable errors as well as exceptions in test scripts are evaluated. Furthermore, steepness of the learning curve is taken into consideration. Number of features, usefulness of the commands are both important factors of this category. How versatile the program is and the possibility to validate objects and structures are other important aspects. The reliability and the possibility to recover from program failures are considered. How compatible different versions of the program are are also important. Ten specific functionality requests from the testers are evaluated. These are Remote handling, Recording,.NET support, GUI-testing, Script support, Plugin support, API support, COM support, Distribution of Reports and Comparison of Files and Logs. The difficulty of introducing the software in the company, i.e. how well does it fit with the organization s present situation. Another important aspect is if the employees already have the experience and proper training needed to manage the test tool. Does the company developing the test software have a history of good support? That is for example, does the company release updated versions on regular basis? Does the company offer mail, phone or online support? Is the software well documented and is there supportive information to be found on other sites? License costs and other costs connected with the tool, such as training, maintenance and updates. Table 4.2: Categories in Tool Evaluation Following sections is a summarized evaluation of the test tools. Only the most differential and characteristic features are described in every category.

36 28 Test tool 4.1 TestComplete 4 TestComplete [7] is developed by Automated QA. Ease of Use TestComplete has a very intuitive user interface. It is easy to create and maintain test cases. The basic features is very easy to learn by just using the recording function, which translates mouse movements and clicks to various scripting languages. The documentation and samples are very useful to get familiar with the program. However, when it comes to the more complex features of TestComplete it is difficult to find relevant information. Power TestComplete is an extremely powerful testing tool. Due to its.net and scripting support it has very few limitations. Currently there is only a Beta version released of TestComplete version 5. It is very unstable, however it offers full support for.net 3.0, which is the framework the test object is built upon. Robustness TestComplete 4 is very stable. Although the few times the program crashed, no information in theprojectwaslost. Functionality A very useful feature of TestComplete is the possibility to browse all objects, methods, properties and fields in basically every application based on the.net framework. This information is presented in very easy to use tree list view. This makes really easy to find objects and methods within the application. Ease of Insertion TestComplete has a rather easy installation procedure, but nothing out of the ordinary. However, the supporting tool TestExecute allows test scripts to be run on a user s or customer s computer without having TestComplete installed, which is a quite useful feature. Quality of Support There is an extensive documentation and solutions to various problems on Automated QA s online forum. They have free mail support and access to the online message boards. TestComplete and Automated QA is well promoted on other web sites. Cost The enterprise version of TestComplete costs about $900 per user. The phone support costs $175 a month. Conclusion TestComplete is stable and versatile software with the functionality currently needed for introducing automatic regression testing.

37 Test tool TestPartner 5 TestPartner [8] is developed by Compuware. Ease of Use TestPartner is a tool easy to get started with. Many basic functions are presented clearly and are easy to locate in an intuitive user interface. With help of the documentation s step-by-step guide, it only takes a few minutes to set up the first test case. However, when it comes to more advanced functionality and larger test projects, it soon becomes very difficult and hard to stay organized. The documentation is very limited and the samples which comes with the program only demonstrates a small part of TestPartner s functionality. Power TestPartner has a great support for data driven testing. Input data is managed through tables with a user-friendly interface. Important features such as accessing.net classes and objects does exists in the latest version, however it is a very lengthy procedure writing test cases using this functionality. The scripting engine has been rewritten and in the latest version it is based on Microsoft Visual Basic Script Engine, which is a familiar language to many testers and programmers. Robustness TestPartner did hang a few times when testing simple programs, but no data was lost. Functionality As with most test tools on the market. TestPartner lacks support of.net 3.0 GUI testing. No open SDK was found to assist in the development of custom plugins. TestPartner has no native support for comparing files and logs, however it is possible by writing custom scripts. Ease of Insertion The installation of TestPartner was simple, but nothing out of the ordinary. As the test projects grow it soon becomes very difficult and therefore expensive to maintain. Quality of Support To access the online knowledge database one must register an account. The impressions is that it is more difficult to find support from third party sites compared to other test tools. Cost $9.000 per user, including 12 months of support. Supporting costs was not found on web page. Conclusion TestPartner is a powerful test tool. Writing advanced test scripts is difficult and time demanding.

38 30 Test tool 4.3 AutoMate 6 AutoMate [9] is developed by Network Automation Inc. Ease of Use This program is really easy to use. Just after looking at some of the examples that comes with the installation one can learn it quite fast. The method of creating test cases consists of dragging and dropping different tasks to the main script, which is really easy for a beginner. Power AutoMate 6 is very powerful program. It can identify every object on the screen, even if the object is based on the.net 3.0 framework. This makes it very useful to create test cases. Robustness Like other applications, AutoMate 6 is a stable software and test projects do not get lost if the program somehow crashes, which happens hardly ever. Functionality What AutoMate 6 gains in power, it lacks in functionality. Although it has support for finding objects on the screen and identifying them, it is hard to use them in test scripts due to the fact that the program has a hard time identifying them as unique. This makes it hard or almost impossible to run more advanced test scripts. The recording function is also very inferior. It records every mouse move and it is impossible to alter the recorded script without doing the whole recording all over again. Ease of Insertion AutoMate 6 has an easy installation procedure and comes with a service that can be modified to run test scripts upon actions in the operating systems, such as when starting a certain software etc. Quality of Support AutoMate 6 comes with great support to get started and even more advanced topics online. Cost The professional edition tried and evaluated costs about $1000. Conclusion In a conclusion one can say that AutoMate 6 is well suited for simpler tasks such as starting different applications, sending keystrokes, mouse clicks etc., but when it comes to more advanced tests AutoMate 6 do not have the functionality needed to implement these.

39 Test tool QuickTest 9 QuickTest [10] is developed by Mercury. Ease of Use QuickTest is very easy to use and has an interface easy to understand. It is effortless to record scripts and follow the recorded scripts. It has both a task view and a script view which is very useful. However, it is hard to find information in the application and huge test projects are very difficult to maintain and get an overview of. It is also difficult to change the test script since mapping of objects on the screen is done automatically and do not always work in a satisfying way. Power QuickTest has a powerful keyword driven language which means that an object deep down in the hierarchy of an application can be mapped with a short name without the test script author having to know the path to the object. The recording functionality is also easy to learn and maintain. Robustness No crashes was experienced during the evaluation period. Functionality Although the recording was easy to do, the playback did not always work. Since the mapping of objects is done automatically the software sometimes has problems of identifying some objects as unique. It also have very limited or no plugin support. Ease of Insertion The program is very advanced. If a common database for error logs is to be used, another software must be installed as a web service. This makes it a little bit too advanced for the current needs at Beijer Electronics AB. The installation procedure crashed when trying to install the.net Framework 1.1. Quality of Support It is one of the most widely used test tool which makes it well documented on different web sites. However, it is difficult to find information about the support available from Mercury. Cost QuickTest 9 professional, which was the software tested and evaluated costs about $ Conclusion QuickTest is a popular and commonly used test automation tool. It can be hard to motivate such a licence cost for a small company or organization.

40 32 Test tool 4.5 Summary In this chapter four different test automation tools has been reviewed. As already concluded it is important to choose a test tool fitting the company s needs. In chapter 6 a test tool is chosen based on the conclusions made in this chapter. In other words, this chapter only gives a review of each automation tool and in chapter 6 the tools are summarized and the most appropriate is recommended.

41 Chapter 5 Scripting Techniques According to Fewster [2] there are several measures to be taken into consideration when introducing a cost and resource effective test environment. One of the more important parts is the choice of an appropriate and well defined scripting technique. A scripting technique is the structure and implementation of the test scripts. It is important to be consequent in the use of a scripting technique. In this way the test scripts are ensured to be maintainable. If different techniques are used, not only is the code difficult to maintain, it is almost impossible for another person to take over the project, and of course this is not desirable. The higher the level of ambition the organization has for the test structure, the cost in time and resources to introduce automatic regression testing will increase. A simple test structure is easy to implement but will not be very cost-effective as the number of test cases increases and the project evolves and changes over time. A well defined test structure, however, may be difficult to introduce but a lot easier to maintain over time and stays effective throughout the whole test process. To illustrate this, take a test case written/recorded in strictly linear scripting. This takes about 5 times longer than doing the test manually. However, shared and data-driven test cases can take up to 40 times longer than manual testing. Looking at a long perspective the second alternativeismuchmoreefficientsincethereisnoneedtorewriteanentirescriptastheproject evolves. The principle of each scripting technique is described in the sections below. [2] 33

42 34 Scripting Techniques 5.1 Linear scripting Linear scripting can basically be described as when every test case is written in its own script from the beginning to the end. No scripts are reused and all code is static. Often a linear script is the result of recording a script performed manually. The source code below is an example of linear scripting which creates a new project in the software Neo, that is the test object at Beijer Electronics AB. As seen, the code is difficult to read and maintain. Dim w1 Dim p1 Set w1 = NameMapping.Sys Set p1 = w1.neo Set w1 = p1.mainform Call w1.winformsobject("m_ribboncontrol").winformsobject("ribbonstrip", "", 3).Click(27, 28) Call w1.winformsobject("menupanel", "").Click(38, 33) Call p1.savedialog.no.clickbutton Set w1 = p1.window("hwndwrapper[neoide.exe;;d73fd5fd-23cf-4e2d-b933-eeb5a371ea05]", "New Project Wizard") Call w1.click(449, 527) Call w1.click(460, 530) Call w1.click(707, 524) Furthermore, no error checking or unexpected events are taken into consideration when using linear scripting. 5.2 Structured scripting Linear scripting seldom requires programming skills, since it often is the result of a recording. Structured scripting calls for some basic knowledge in any of the most basic scripting languages. Since most test tools support syntax for the most popular scripting languages today, it is fairly easy to write structured test cases if the tester is familiar with any programming language. Structured testing is basically test cases which adapts to different situations. These situations is either defined or created by e.g. if-, for- and while-blocks. Used in an appropriate way, the quality of test cases can be increased considerable. Structured scripting can be used for example to verify that a specific message box is displayed on the screen or to repeat certain tasks multiple times. This can be useful when stress testing a program or handling unpredicted events. An example of structured scripting is illustrated in the code below. Dim w1 Dim p1 Set w1 = NameMapping.Sys Set p1 = w1.neo Set w1 = p1.mainform Call w1.winformsobject("m_ribboncontrol").winformsobject("ribbonstrip", "", 3).Click(27, 28) Call w1.winformsobject("menupanel", "").Click(38, 33) if SaveDialog.Exists then Call p1.savedialog.no.clickbutton end if Set w1 = p1.window("hwndwrapper[neoide.exe;;d73fd5fd-23cf-4e2d-b933-eeb5a371ea05]", "New Project Wizard") Call w1.click(449, 527) Call w1.click(460, 530) Call w1.click(707, 524)

43 Scripting Techniques Shared scripting The introduction of shared scripting allows a tester to reuse already existing code in several different test cases. This is achieved by dividing each test case in smaller tasks. Each task is categorized and structured in a test library. These tasks can then be called from a test script to build up a whole test case. The main advantage of shared scripting is that they can easily be adapted to architectural changes, since it is only necessary to alter one task to make changes in all scripts using that task. To illustrate this, imagine a couple of hundred test cases using the task Create New Project. If the program changes so that the task does not work anymore, it is only necessary to change the task in one place and it will affect all test cases using that task Shared scripts may be more time-consuming and more difficult to write, but in the long run as the program changes, the maintenance and cost will be much less as if linear and structured wouldbeused. Anexampleofsharedscriptingcanbeviewedinthecodebelow. Asseen,every test case uses predefined functions that are reused when they are needed. The advantage is that the method CreateNewProject only needs to be altered in one place if any changes are required. sub TestCase1 CreateNewProject CreateButton(50,50) SaveProject CloseProject end sub sub TestCase2 CreateNewProject CreateButton(100,20) CloseProject end sub sub CreateNewProject Dim w1 Dim p1 Set w1 = NameMapping.Sys Set p1 = w1.neo Set w1 = p1.mainform Call w1.winformsobject("m_ribboncontrol").winformsobject("ribbonstrip", "", 3).Click(27, 28) Call w1.winformsobject("menupanel", "").Click(38, 33) if SaveDialog.Exists then Call p1.savedialog.no.clickbutton end if Set w1 = p1.window("hwndwrapper[neoide.exe;;d73fd5fd-23cf-4e2d-b933-eeb5a371ea05]", "New Project Wizard") Call w1.click(449, 527) Call w1.click(460, 530) Call w1.click(707, 524) end sub 5.4 Data Driven scripting Data Driven testing allows the user to define all test input data in separate text files, e.g. size and layout of different components. This makes it possible to run several tests using only one test script but with different data as input. For example, a test that put fifty components on the screen with different positions and sizes can be run this way and later also be verified using thesameinputdata.thisisillustratedinfigure5.1.

44 36 Scripting Techniques open x y w h save & close project1.neo project1.neo project1.neo project1.neo project1.neo project1.neo project1.neo project2.neo Figure 5.1: Data File 1 The script using this data file opens a project, places a button at the coordinates x, y with the width w and height h and then saves the project and closes it. This data file represents having a test script running four different tests with different data. Data Driven testing can also be taken to the next level by building in some knowledge of the application into the data file. This is done by representing different fields in the data file by a certain function or a type of input. An illustration of this is shown in Figure 5.2. The script using this data file checks weather to open a new project, create a new button with specific position and size, or save and close the project. In Figure 5.2 the test script opens a new project called project1.neo, then creating a button at 20, 40 with the width 200 and the height 200, and then saves and closes the project. The next step opens the same project, but this time three different buttons with unique positions and sizes are created and then the project is saved under a different name. open x y w h save & close project1.neo project1.neo project1.neo project2.neo Figure 5.2: Data File 2 The main difference can be described like this. Data file 1 describes how the script should perform different actions, but date file 2 both describes when and how the script should perform these actions. 5.5 Keyword Driven scripting The final step towards a well structured test environment is the introduction of Keyword Driven testing. The principle is the same as in Data Driven testing with the main difference that the test tasks is specified with keywords rather than predefined in columns as in Data Driven

45 Scripting Techniques 37 testing. The main advantage of Keyword Driven testing is that a tester does not have to have great programming skills to write test cases. However, the supporting scripts, i.e. the scripts corresponding to different keywords, still have to be written by an experienced programmer. An example of how test cases are written is illustrated in Figure 5.3. OpenProject project1.neo CreateButton SaveAndCloseProject project1.neo Figure 5.3: Keyword Driven script Each keyword (OpenProject, CreateButton, SaveAndCloseProject) in this script corresponds to a function in another script where the other words on each line are sent as parameters. Another advantage of keyword driven testing is that since the test cases are written in a predefined language it is possible to change the whole test environment, e.g. OS, test tool, scripting language. Of course it is then necessary to change the functions that the keywords refer to, but the test cases can be kept intact. 5.6 Summary In this chapter several different levels of scripting techniques has been reviewed. As already concluded it is important to understand different ways of implementing test cases and be consequent when coding. In chapter 6 an appropriate level of scripting is chosen based on the conclusions made in this chapter. In other words, this chapter only gives a review of each scripting technique and in chapter 6 the techniques are summarized and the most appropriate is recommended.

46 38 Scripting Techniques

47 Chapter 6 Suggestion It is intended to give a full proposition of the implementation of the testing strategy. This includes choice of reference model, test tool, test structure and scripting technique. The proposition is handed out to all involved managers in the testing and development departments. This together with a practical demonstration of a prototype is a foundation, that managers will give feedback on. This feedback is considered and discussed with all involved parties. When all parties agrees on a final test strategy and environment, this is implemented and presented. This chapter contains information on how decisions are made regarding reference model, test tool and scripting techniques. The proposition can be found at Department of Communication Systems at Lund University [6]. The last section in this chapter describes the feedback received and the final changes made to the proposition. 6.1 Reference model As explained in section 3.1 TMM is a reference model suitable for larger organizations. Following the improvements recommended in the framework is very difficult, since the improvements steps is very large. This is also sometimes very unmotivating since improvements made in the test department is not traceable in the maturity scheme, see Figure 3.1. An interesting test framework is the MTPF, section 3.3. This test practise aims towards smaller companies and the improvement steps are smaller than TMM which makes it easier for a company of Beijer s size to follow the test practice framework. The main disadvantages with MTPF seen from a Beijer perspective, are that improvements are executed in parallel company expansion. Looking at Figure 3.3, Beijer Electronics AB currently is somewhere between 2 and 3. However, there is no plans to hire new employees, which makes the test framework difficult to follow. It is a disadvantage that Beijer Electronics AB already has reached phase two of this test framework, since this leaves little room for further improvements. Finally, none of the test or developer managers are familiar with MTPF, which makes training and understanding of the reference model necessary. The last of the evaluated improvement models is TPI, see section 3.2. This framework offers improvements to be made in small steps, which is considered valuable when used in a smaller company, that have limited resources and works continuously to improve the test environment. According to the results of the interview and questionnaire in Section 2.3 TPI is already familiar to the test managers and to some extent the testers, which makes it easier to apply the model. In addition to this, TPI is already used as a test improvement model at Beijer Electronics AB, 39

48 40 Suggestion which is a very strong argument for this test practice framework. Key Area Scale Test strategy A B C D Lifecycle model A B Moment of involvement A B C D Estimating and planning A B Test-specification techniques A B Static test techniques A B Metrics A B C D Test automation A B C Test environment A B C Office environment A Commitment and motivation A B C Test functions and training A B C Scope of methodology A B C Communication A B C Reporting A B C D Defect management A B C Testware management A B C D Test process management A B C Evaulation A B Low-level testing A B C Controlled Efficient Optimizing Figure 6.1: TPI Maturity Matrix for the organization The TPI Maturity matrix, as seen in Figure 6.1 is a representation of how well established different areas of the automatic test environment are. This is derived from the interview and verified by test manager Fredrik Persson. As seen in the matrix there is still a need for some improvements, especially in those areas that does not fulfill the requirements for the lowest level A. TheconclusionisthatTPIisthemostsuitabletestframeworkforBeijerElectronicsAB and thereby the recommended choice of reference model for this study. 6.2 Test tool The test software TestPartner is, as already been established, a versatile tool. However it soon becomes to complicated when working in larger test projects. Another factor is that it is expensive both to purchase and to maintain. As seen in Figure 4.1 TestPartner along with QuickTest receives the lowest overall rating of the test tools. QuickTest is very advanced, but far from flawless. It has inconsistent object identification and lacks the ability to browse objects in a hierarchial tree structure, which this study found to be one of the most helpful features when creating test cases for graphical user interface applications. AutoMate 6 is an interesting candidate. It has a really reasonable learning curve and test cases are implemented in a very intuitive way. However, it lacks support of.net objects

49 Suggestion 41 which is important for this study. When doing more advanced test cases it soon becomes quite difficult to maintain them and it is desirable to have more script support than what exists today. AutoMate 6 although receives a quite high overall rating but its functionality is not enough for the application to be used in the test environment at Beijer Electronics AB. TestComplete has two major advantages over the other evaluated applications. Firstly it got the highest overall rating of the tested program and secondly it had already been purchased by Beijer Electronics AB before the start of this study. However, this was not not a factor when the test tools were evaluated. Normally the second advantage should not be considered, but Beijer Electronics AB has limited resources and this combined with the high purchase cost of the other tools makes this a factor to be considered. TestComplete has all the features valued high by the testers, such as.net support and script support. It has also the ability to browse objects in a hierarchical tree structure, which has been explained earlier to be very useful. TheconclusionisthatTestCompleteisareallystrongcandidatethatwillcovertheneeds for a functional test environment, and therefore the recommended choice of test automation tool for this study. 6.3 Scripting Technique As concluded in Chapter 5, linear scripting takes the smallest amount of time to implement. Structured, shared, data-driven and keyword-driven scripting, each takes more and more time to implement. Looking at maintenance the scale is reversed, where keyword-driven testing is the easiest and less resource costing alternative. This means that when the test scripts and test cases has been developed, keyword-driven testing soon becomes the most valuable test structure. Another factor to consider is that linear scripting requires practically no programming skills and very limited testing experience. When using keyword-driven testing, experience in programming is a must. Since the intent of the introduction of automatic testing is very long termed, keyword-driven scripting is the most useful and beneficial test structure and is therefore the recommended choice of this study.

50 42 Suggestion

51 Chapter 7 Implementation This chapter presents a prototype, which is to be considered as the base for the implementation of automatic regression testing at Beijer Electronics AB. First section covers the recommended structure of test cases and scripts to be used in the test environment. Next section gives a deeper explanation of the controller script, which is the main scripts that controls and runs all test cases. Finally the implemented test environment is evaluated. 7.1 Overview This section describes an overview on how test cases are implemented and structured. As a practical example, Beijer Electronics project Neo has been used. There is several considerations to be made when designing the test case structure. It is of great importance that the structure, i.e. how the test cases are organized, is well defined and clear. This makes the maintenance a lot simpler and new test cases can easily be added. The proposed Test Structure is described in Figure

52 44 Implementation Controller Test Cases Test Cases Test Cases Keyword Driven Testing Device Library GUI Library Project Library Design Library Supporting Scripts, catagorized by function Window Manager Toolbox Manager Undo Manager Command Facade Script Manager Script Editor Socket Server Socket Client Toolbar Manager Link Library Test projects, each corresponding to a module in Neo Figure 7.1: Test Case Structure Test Cases The test cases are written in a very simple keyword-like language, preferable in a well known environment like Excel, for example as illustrated in 7.2. The advantage of keeping the test cases simple and well structured is that testers with virtually no programming skills still can maintain and write these test cases. The test cases are then interpreted by the control script in the controller. As seen in the Figure 7.2, the test case exists of several predefined commands, i.e. keywords, and one or more possible parameters. These should be easy to understand and well documented.

53 Implementation 45 Figure 7.2: Excel Test Case The test program used in the prototype, TestComplete, offers great support for Excel with built in functions for reading worksheets. Since Excel is a well known and easy to use program, it is a good reason for choosing it as a test case writing tool. Test Cases Test Cases Test Cases Script files Script files Script files Figure 7.3: Test Case Structure The test cases are categorized depending what they are designed to perform. It then becomes easier to find and alter a specific test case and to run scripts testing certain functionalities in

54 46 Implementation Neo Controller The purpose of the control script is interpretation of test cases. The controller scans through predefined test cases row by row and calls the functions associated with each keyword along with parameters specified. These functions are located in the supporting scripts. As seen in Figure 7.4, the controller consists of a control script, a configuration file and a function reference file. The configuration file defines which test cases to execute and in which order. The control script is the main script that interprets the predefined test cases and uses the function reference file to call functions in the supporting scripts corresponding to different keywords. There are also documentation files explaining how to use the files in the controller library. A deeper explanation of this is found in Section 7.3. Controller Documentation Function reference Config Control Script Figure 7.4: Controller Structure Supporting Scripts The supporting scripts are divided into different categories depending on what type of functionality is being tested. For example, all functions handling the project are located in the Project Library. It is important to point out that the functions in these functional libraries do not call internal methods in Neo. Instead it calls functions in the scripts in the test projects, as illustrated in Figure 7.1. The main reason for this is that if several different functions in the supporting script uses the same internal method in Neo, and the test for this method needs to be changed, i.e. due to program changes, it is necessary to change in more than one place. The maintenance then becomes much more difficult.

55 Implementation 47 Library Utilities Documentation Script files Third party utilities Figure 7.5: Supporting Script Structure Projects Scripts Each module in Neo, relevant to the test environment is represented by a corresponding test project. By using this structure, a change in any module in Neo that require a change in the tests, can easily be handled since the change only need to be carried out in one place. As described earlier, a structure with a more direct approach would require changes in multiple scripts. Test Project Documentation Script files Figure 7.6: Test Project Structure The test projects are the only projects that call internal methods in Neo and each project only call methods within the same module or same group of modules. This makes it easy to know which project is responsible for which part of Neo and if a change is needed, it is effortless to locate areas to change Summary The test structure has been developed with the consideration that also testers without or with limited programming skills are qualified to write test cases. At least one programmer is needed

56 48 Implementation to write the supporting scripts and project scripts in parallel with the development of Neo. 7.2 Test Support There are several considerations to be taken into account when testing a graphical user interface automatically. Since Neo is in an early stage of development it is most likely to change in appearance and structure over time. This is illustrated in Figure 7.7 and Figure 7.8. The screenshots differs in both layout and functionality, and this is only over a period of six weeks. Obviously, this makes it difficult to do automatic GUI testing, e.g. by recording mouse events and coordinates. Figure 7.7: Screenshot of Neo build 920

57 Implementation 49 Figure 7.8: Screenshot of Neo build 736 Another consideration to be taken into account is that Neo is built upon.net Framework 3.0 which in the present is very recently released and therefor it does not yet exists any fully developed commercial test tools with full support for this platform. This means that the tool of our choice, TestComplete, does not recognize objects, such as components on the screen and toolbar buttons. It makes it almost hopeless in relaying on mouse clicks and coordinates when it comes to automation testing of the GUI. It is therefore vital to be able to get under the shell, i.e. by directly accessing internal methods

58 50 Implementation and properties. In this way, the testing process is not sensitive to graphical user interface design changes, but only to major architecture structure changes, which are not very common. This is mainly due to facts that the developers uses an architecture design called SOA Light, Service Oriented Architecture Light. SOA is a design approach where the project is based on numerous different modules, which all has its own responsibility. These modules communicate with each other through services, which is a certain interface and is independent of how the other modules are implemented. To be able to access the functionality of each and every service, there has to be some sort of communication between the test program and the test object. Due to the fact that TestComplete has great support for.net applications and Neo is built upon the.net framework the process of accessing methods and functionality from Neo is significantly simplified. TestComplete allows theusertoaccesspropertiesandinternalmethodsofneodirectlyoutofthebox. However,by adding a property containing a handle to a helper class the access to properties is simplified even more. The helper class is a class which only responsibility is to provide simple methods to perform more advanced tasks, e.g. accessing objects on the screen, handling projects and calling internal methods helpful when writing test cases. Due to lack of time, the developers have not had the possibility to fully implement the helper class. This has limited the number of test cases presented. It is very important to allocate sufficient time and resources for the development of test support in the software. Also important to say is that a continuous development of the test support is needed to ensure that the automatic regression testing is at a functional and efficient state at the same time as the software. 7.3 Controller Script The controller script is the most essential part of the implementation of the test structure. This script loads and executes all test cases, defined in a single Excel file, see Figure 7.9. A test case is specified in an Excel file which is interpreted by the controller script and a corresponding method in the supporting scrips is executed. TheTestConfigfilecontainsalltestcaseswhichistoberun. Firstcolumnsimplydefines if a specific test case is to be executed or not. If this column contains a # 1 thetestcaseis skipped. Next column specifies where the test case file is located, i.e. path to folder. The reason for having the possibility to place various test cases in different folder is that the test cases stays structured as the number of test cases increases. Third and last column, contains the actual test case to be run. This name corresponds to an Excel file in the specified folder. 1 The universal standard sign for defining comments

59 Implementation 51 Figure 7.9: Config file As seen in Figure 7.10, the Test Cases are very intuitive and simple to read and write. The first column always contains the keyword to be executed. The following columns specifies the parameters to be sent with the keyword. For example, looking at the figure, we see that the first command is CreateButton, which intuitively creates a button. The button is named Button1 and also gets the caption Button1. And is created at position 50 (x), 50 (y). All of these parameters is specified in an reference file and a more extensive documentation. Figure 7.10: Test Case file All existing keywords and the corresponding namespace 2 is defined in a the Functional Reference file, see Figure It is a useful resource for test case writers, which needs a list of available keywords. 2 Defines the location of a function

60 52 Implementation Figure 7.11: Functional Reference file 7.4 Summary As seen in this chapter a foundation for the test structure is presented. The main idea is to keep the test cases as simple as possible which allows testers with virtually no programming experience to write test cases. Only one experienced programmer is needed to maintain and evolve the test structure with new functionality in parallel with the development of Neo.