Mobile Application Testing in Pakistan: A Survey

Transcription

1 MUHAMMAD SAQIB NAWAZ*, ZAFAR IQBAL**, MUHAMMAD IKRAMULLAH LALI***, AND RAZA UL MUSTAFA** RECEIVED ON ACCEPTED ON ABSTRACT Reseach egading MAT (Mobile Application Testing) in Pakistan is had to discove and to the best of ou knowledge, no wok has been done in suveying MAT in Pakistan. In this wok, we have examined the cuent tend and status of MAT in Pakistan. Main objective was to investigate to what extent MAT is cuently applied in Pakistan softwae companies and what expeience the companies have with using MAT. Futhemoe, effots wee made to find out what testes think about MAT, e.g. issue, advantages and disadvantages of MAT, what factos affects MAT and how they plan to impove MAT. In ode to achieve ou objectives, we used a compehensive online suvey so we conveted ou eseach questions into coespondence suvey questions. We seved a questionnaie of the suvey to 66 testing elevant officials of leading softwae companies in diffeent cities of Pakistan to develop a model study about geneal tend and status of MAT which can be genealized all ove Pakistan. We eceived eplies in total afte ove 2 months. Afte that, we used SPSS tool to analyze the eplies of this questionnaie. Coss-Tabulation Analysis and Peason Chi-squae tests have been computed to examine the esults. We found some inteesting esults on cuent status and pactice of MAT in Pakistan softwae companies. Key Wods: Mobile Applications, Suvey, Statistical Analysis, Testing. 1. INTRODUCTION The futue of computing is moving towads mobile devices with apid advancements in ICT (Infomation & Communication Technology). The applications which ae unning on mobile devices ae known as mobile applications. Mobile applications ae becoming vey famous as these ae showing evolution in the field of infomation technology. Initially, mobile applications wee developed mostly in the entetainment secto; howeve, they ae now also available in othe domains such as media, games, etail, tavel, education, healthcae, finance and social media platfoms. In 2012, thee wee aound 1.1 Billion uses of mobile applications. Based on the foecast, the above said numbe will aise up quickly to almost 30% annually to get in touch with 4.4 Billion in 2017 [1]. So this huge demand fo mobile devices, along-with mobile applications equies bette mobile application testing tools [2]. In softwae quality assuance activities, testing is consideed a key pat in the life cycle of a softwae ceation. Testing accomplish following two pime puposes [3]: Coesponding Autho ( msaqibnawaz@pku.edu.cn) * Depatment of Infomation Science, Peking Univesity, Beijing, China. ** Depatment of Compute Science, Univesity of Lahoe, Sagodha Campus. *** Depatment of Compute Science & Infomation Technology, Univesity of Sagodha. 15

2 It expesses the quality o accuate behavio. It identifies and fixes the poblem. Mobile testing efes to diffeent types of testing, such as native and Web mobile application testing and testing of mobile devices [6]. We use MAT to efe to testing activities fo mobile applications (native, hybid and Web applications) on mobile devices with softwae test methods and tools in ode to ensue the coectness of functionality, behavios, quality of sevice, usability, secuity and pivacy of mobile applications. Native mobile applications ae coded in specific pogamming language (Java fo Andoid platfom and Object C fo ios) and executed on mobile devices.native applications depend on device featues, such as camea, contact list, GPS (Global Positioning System) and so on. Web mobile applications ae un by a bowse and geneally witten in HTML5 (Hype Text Makup Language). It consists of an application seve and client softwae. Uses can access application though client softwae executed ove bowses. Hybid application is the combination of both the native and web applications. Hybid is basically a web application that is built using HTML5 and JavaScipt. It is then wapped in a thin native containe that offes access to device featues. MAT is cuently applied in many counties. Howeve, actual facts and figues egading MAT in Pakistan softwae companies ae not collected efficiently. In this wok, we examined the cuent tend and status of MAT in Pakistan softwae companies. Ou main focus is on the following eseach question: How to optimize the pefomance of MAT in Pakistani softwae companies? The aim is to exploe the benefits of MAT as well as to impove the quality and pomote MAT in Pakistan. Objective is to examine that how much MAT is pefomed in Pakistani softwae companies and what advantages the companies have of using MAT. We have also examined affiliation among the oles, pofessional expeience, histoy (age), location and size of the company. We have collected the data fom softwae companies located in diffeent cities including ISD (Islamabad), SGD (Sagodha), LHR (Lahoe), FSD (Faisalabad), Multan and Bahawalpu. Fo this pupose, a compehensive online suvey is conducted to collect esponses fom testing elated pesons. Rest of the aticle is oganized as follows: Related wok is discussed in Section 2. Ou methodology, suvey design and execution ae stated in Section 3. Data collection fom the suvey, demogaphic infomation obtained fom suvey and main findings ae pesented in Section 4. Discussion and suvey validity is discussed in Section 5, followed by conclusion in Section RELATED WORK In the IT industy and in academia, MAT has ecently eceived moe attention by testing pofessionals due to incease in the numbe of testing tools and testing magazines ae publishing issues exclusively on MAT [7]. Liteatue on MAT concepts, testing techniques, issues and challenges can be found in [4,6,8,9]. Yin [10] conducted a suvey (executed in Finland and Estonia in the ealy of 2014) egading softwae exploatoy testing [11]. ET (Exploatoy Testing) is a concuent leaning pocess in which the design and execution of the tests ae not declaed ealie. Design of softwae tests, execution of softwae tests and modification of softwae tests ae made at un time. The objective and main eseach question of the suvey [16] was to know the tend and status of existing pactice about ET in Finland and Estonia. The questionnaie of the suvey on softwae ET was seved to 61 diffeent oles such as developes, testes and test manages in vaious softwae companies in Finland and Estonia. Main findings of the suvey wee: 16

3 Mostly testes (48%) and test manages (36%) apply ET in Estonian and Finnish companies. Appoximately half of the espondents (50%) have moe than five yeas of woking expeiences. Testes in Estonia (50%) use ET moe than Finnish testes (38%). In small companies, ET has been applied elatively moe often (less than 50 employees, %) than in lage companies (32%). Companies olde than 5 yeas that use ET ae ove 50%. Appoximately 76% of the espondents who use ET ae membes of a sepaate testing oganization within thei company. Madhushani et. al. [12] pesented a mobile application testing pocedue fo Si Lankan softwae development companies. They have also explained main challenges in MAT fom Si Lankan pespective. A questionnaie of 10 questions was distibuted among vaious IT individuals and softwae companies in Si Lanka. Accoding to analysis: 93% IT industials believes that MAT is vey impotant fo the customes. In Si Lanka, 92% developes develop Andoid mobile application and 54% develop ios based mobile application. Consideing the level of testing, most of QA(Quality Assuance) enginees o developes use pefomance and secuity testing. Futhemoe, few companies develop and test same application fo diffeent opeating systems and devices. 92% testes use open souce tools fo the testing. Main challenges in MAT ae vaiety of mobile devices and diffeent opeating systems. 3. METHODOLOGY We have done a qualitative eseach and conducted an online suvey fom Octobe 25, 2015 to Decembe 31, 2015 in ode to investigate the cuent status of MAT in Pakistan. Main objective was to investigate to what extent MAT is cuently applied in Pakistan softwae companies and what expeience the companies have with using MAT. Futhemoe, we aimed to find out what factos affects MAT, such as specific oles in a company use MAT moe often than othes, and whethe enginee expeience, company size, company age, and location has any influence on the usage of MAT. In addition, what ae the chaacteistics of MAT accoding to enginees and what they think about MAT, e.g. advantages and disadvantages of MAT and how they plan to impove MAT. Ou main RQ is fomulated to 3 eseach questions and each eseach question is futhe extended into elated set of SQs (Suvey Questions): RQ-1: What ae the abilities of those who pefom MAT in Pakistan and what ae the benefits of MAT fo a company? RQ-1.1: Which mobile platfom enginee pefe fo mobile application development and testing? RQ-1.2: What ae the main chaacteistics of mobile application? RQ-1.3: What type of mobile applications SW (Softwae) enginees develop and test? RQ-1.4: What types of tools ae pefeed by sw enginees to use in MAT? RQ 1.5: What is sw enginee tend to pefom diffeent types of testing? RQ-1.6: In what cicumstances, sw enginee uses MAT fo thei company? RQ-1.7: What ae the key advantages to the company of using MAT? 17

4 RQ-2: To what extent MAT is pesently pefomed in Pakistani softwae companies and what ae the influential factos on the usage of MAT in Pakistanis softwae companies? RQ-2.1: Do testes pefom MAT much commonly as compae to developes o pogammes etc? RQ-2.2: Do sw enginee having much pofessional expeience pefom MAT much commonly? RQ-2.3: Does the localities (SGD, LHR, ISD and FSD) have diffeent outcomes on pefoming MAT? RQ-2.4: Do employees of big company pefom MAT much fequently as compae to the small companies? RQ-2.5: Is MAT pefomed much commonly in old companies? RQ-2.6: Do enginees of sepaate testing oganization pefom MAT much commonly? RQ-2.7: What factos ae dissimila when pefoming MAT in SGD, LHR, ISD and FSD? RQ-3: What ae the opinions of SW enginees egading MAT and how impovements can be bought in existing pactice of MAT in Pakistan? RQ-3.1: What ae the basics, sw enginees think as necessay fo defining MAT? RQ-3.2: What uniqueness do sw enginees feel about MAT? RQ-3.3 and 3.4: What ae the opinions of sw enginees egading advantages and disadvantages of MAT? RQ-35 and 3.6: What ae the issues in MAT and what ae the key steps fo impovement in existing pactice of MAT in Pakistan? Fom these thee RQ s, we have designed SQ s. SQ s and thei mapping to RQ s is shown in Appendix-A. 4. SURVEY DISTRIBUTION AND RESULTS Online Suvey questionnaie was seved to vaious mobile applications developes, testes, manages and othe test elevant pesons that coopeate with manages o testes. Fo this pupose, we choose Google Docs platfom to seve and eceive online eplies fom espondents. We have also posted the suvey link though othe channels such as mailing list, LinkedIn, Twitte and Facebook. We collected online eplies of ou suvey questionnaie. The esults of data collection of suvey questionnaie in gaphical epesentations ae discussed next. 4.1 Demogaphic Infomation The demogaphic infomation of suvey espondents is shown in Figs Figues in the gaphs show fequencies and pecentage (fo example, 24, 46% means that 24 espondents of suvey select that choice and its pecentage is 46). Respondents ole and the total woking expeience of the espondent in cuent oganization is shown in Figs. 1-2 espectively. 73% espondents ae eithe woking as manage o develope in the company. Wheeas, 9 out of espondents (16%) ae woking as testes. Most of the espondents (41 out of ) have moe than 3 yeas of woking expeience in thei cuent ole and 15 espondents (27%) have less than 3 yeas of woking expeience. Company location, size and age of the company is shown in Figs. 3-5 espectively. In esponses, 49 espondents belong to fou cities (LHR, FSD, ISD and SGD) wheeas 7 espondents belong to othe cities (Multan and Bahawalpu). The chaacteistics of the mobile applications that espondents oganization deals with ae shown in Fig. 6. The most fequently mentioned mobile application chaacteistics is high pefomance (30%), followed by high secuity (23%) and complex GUI (18%). 18

5 4.2 Main Findings We used SPSS [14] tool to analyze the eplies of this questionnaie. Statistical tests such as Coss- Tabulation, Fishe Exact Tests and Peason Chi-Squae Tests [5,15] have been computed to examine the esults. Statistical tests give the infomation that can be used to assess the significances of inceases o deceases in the esults. Coect selection of statistical test depends on data type as well as aim of the analysis. In case of wong selection of statistical test, equied esults may not be valid and can mislead to conclusion of the eseach. In this section, suvey esults elevant to each RQ ae epoted RQ-1 What ae the abilities of those who pefom MAT in Pakistan and what ae the benefits fo thei companies of MAT? In the following subsections, esults elated to RQ-1 fo each sub-question is epoted. RQ-1.1 Which mobile platfom enginee pefe fo mobile application development and testing? Table 1 shows the distibution of mobile platfoms that suvey espondents use fo mobile application development and testing. 54 out of (96%) espondents FIG. 1. RESPONDPENTS ROLE FIG. 4. RESPONDENTS COMPANY SIZE FIG. 2. RESPONDENTS WORKING EXPERIENCE FIG. 5. AGE OF RESPONDENTS COMPANY FIG. 3. RESPONDENTS COMPANY LOCATION FIG. 6. MOBILE APPLICATION CHARACTERISTICS 19

6 develop and test mobile application in Andoid and ios. This is because 82% mobile uses in Pakistan use Andoid based Mobiles and 12.9% use ios, accoding to online epot [13]. RQ-1.2: What ae the main chaacteistics of mobile application? Main chaacteistics of mobile applications accoding to suvey espondents ae listed in Table 2. 30% espondents (17 out of ) believe that high pefomance is the main chaacteistic of thei application. 13 espondents (23%) said that high secuity and safety is the main chaacteistics of mobile applications that they developed and test. RQ-1.3: What type of mobile applications SW enginees develop and test? Table 3 lists the distibution of mobile applications that suvey espondents geneally develop and test in thei company. Almost 80% enginee test eithe web application o hybid application. RQ-1.4: What types of tools ae pefeed by SW enginees to use in MAT? Fom RQ-1.1, 54 espondents (98%) said that eithe use Andoid o ios platfom fo mobile application development and testing. So we have asked espondents that what types of tools they pefe fo MAT in Andoid and ios. In Table 4, tools used by enginees fo Andoid and ios application testing is listed. TABLE 1. DISTRIBUTION OF MOBILE PLATFORMS FOR DEVELOPMENT AND TESTING Mobile Platfom Sagodha Lahoe Faisalabad Islamabad Othe City A ndoid 6 (16.5%) 10 (28%) 6 (16.5%) 9 (25%) 5 (14%) 36 (64%) Blackbey 0 1 (100%) (2%) Windows (100%) 0 1 (2%) i OS 3 (16.5%) 5 (27.5%) 5 (27.5%) 3 (16.5%) 2 (11%) 18 (32%) T otal 9 (16%) 16 (29%) 11 (20%) 13 (23%) 7 (12%) TABLE 2. MAIN CHARACTERISTICS OF MOBILE APPLICATIONS Chaacteistics Fequenc y Pecent High Secuity and Safety Complex GUI High pefomance Enegy Saving 7 13 Othe 9 16 TABLE 3. DISTRIBUTION OF MOBILE APPLICATIONS SURVEY CORRESPONDENTS DEVELOP AND TEST Mobile App Type Sagodha Lahoe Faisalabad Islamabad Othe City N ative App 2 (16.5%) 3 (25%) 2 (16.5%) 3 (25%) 2 (17%) 12 (22%) H ybid App 3 (13%) 7 (32%) 5 (23%) 4 (18%) 3 (13.5%) 22 (39%) W eb App 4 (17.5%) 6 (27.5%) 4 (17.5%) 6 (27.5%) 2 (9%) 22 (39%) T otal 9 (16%) 16 (29%) 11 (20%) 13 (23%) 7 (12%) 20

7 13 out of 36 espondents (36%) use SOASTA tool to pefom MAT in Andoid platfom. Wheeas, 44% espondents (8 out of 18) used Ranoex tool fo ios platfom. RQ-1.5: What is SW enginee tend to pefom diffeent types of testing. Data collection fom SQ-1.10 and SQ-2.3 shows that system testing (25 out of ) is pefomed moe times as compae to othe types of testing. On the othe hand acceptance testing is pefomed vey less (9 out of ). Suvey esults fo RQ-1.5 ae shown in Fig. 7. RQ-1.6: In what cicumstances, SW enginee uses MAT fo thei company? Testing of mobile application is geneally done duing application development o afte the development. Almost half of espondents (27 out of ) pefom MAT duing the development and othe half pefoms MAT afte application development (as shown in Fig. 8). In ode to check whethe thee is any diffeence by applying MAT duing application development o afte the development, we stated following null hypothesis. FIG. 8. RESPONSES FROM RESPONDENTS REGARDING TESTING TIME OF MAT TABLE 4. TOOLS USED FOR ANDROID AND IOS APPLICATIONS TESTING Tools fo MAT Andoid ios P efecto Mobile 3 (8%) 2 (11%) 5 (9%) D evice Anywhee 6 (17%) 3 (17%) 9 (17%) S OASTA 13 (36%) 1 (5%) 14 (26%) J amomuex 2 (5%) 1 (5%) 3 (5%) Z apfix 2 (6%) 2 (11%) 4 (7%) R anoex 8 (22%) 8 (44%) 16 (30%) N o Tool 2 (6%) 1 (6%) 3 (6%) FIG. 7. TRENDS TO PERFORM DIFFERENT TYPES OF TESTING 21

8 H0: Thee is no diffeence of pefoming MAT with espect to time. H1: Thee is a diffeence of pefoming MAT with espect to time. To test these hypotheses, we used Peason Chi squae test. We used SPSS tool fo all calculations and the value of significant level alpha (α) is set to Table 5 shows the test esults. Table 5 shows that p= > α (0.05), hence accept hypothesis H0 (Thee is no diffeence of pefoming MAT with espect to time) and eject hypothesis H1. RQ-1.7: What ae the key advantages to the company of using MAT? Table 6 shows that 19 out of espondents (4 manages, 11 developes, 3 teste, 1 othe) believes that most impotant benefit to the company of pefoming MAT is that the company poduces high quality mobile application. 15 espondents believe that company gets the attention of custome by using MAT RQ-2: To what extent MAT is pesently pefomed in Pakistani softwae companies and what ae influential factos on the usage of MAT in Pakistanis softwae companies? Fo RQ-2.1 to RQ-2.6, we fomulated and tested following null hypothesis: [RQ-2.1] H0-1: Thee is a diffeence in fequencies of pefoming MAT fo testes and developes. [RQ-2.2] H0-2: Thee is no diffeence in pefoming MAT to dissimila level of pofessional expeience. [RQ-2.3] H0-3: Thee is no diffeence of pefoming MAT in diffeent cities. [RQ-2.4] H0-4: Thee is no diffeence of pefoming MAT in diffeent sizes of companies. [RQ-2.5] H0-5: Thee is no diffeence of pefoming MAT in diffeent ages of companies. [RQ-2.6] H0-6: MAT pefomance is not affected by the existence of a sepaate testing company. Peason Chi-Squae TABLE 5. PEARSON CHI SQUARE TEST RESULTS Value df Asymp. Significant (2-Sided) 0.622a Likelihood Ratio Linea-by-Linea Association N of Valid Cases Benefits to Company Company meets the standads of mobile app development as maket's demand a. 4 cells (50.0%) have expected count less than 5. Manage Develope Teste Othe 3 (27%) 5 (46%) 2 (18%) 1 (9%) 11 (20%) C ompany gets the attention of custome 2 (14%) 9 (60%) 2 (13%) 2 (13%) 15 (27%) Company poduces high quality mobile applications. TABLE 6. COMPANY BENEFITS BY USING MAT 4 (21%) 11 (58 %) 3 (16%) 1 (5%) 19 (33%) Company gets financial benefits (11%) O the 1 (16%) 2 (50%) 1 (17%) 1 (17%) 5 (9%) 22

9 All the suvey espondents wee familia with MAT. Data elated to RQ 2.1 and RQ 2.2 is listed in Table 7. Responses fom suvey espondents egading whethe they wok as sepaate teste ae listed in Table 8. Suvey espondents company location, company size and company age is aleady shown in Figs. 3-5 espectively. Fo hypothesis testing of RQ-2, we pefomed Cosstabulation analysis and Peason Chi-squae tests in SPSS. We set the significance level alpha (α) = Test esults ae shown in Table 9. It tuns out that only hypothesis H0-1 could be ejected. Thee is no diffeence in fequencies of pefoming MAT fo testes. RQ-2.7: What factos ae dissimila when testes pefom MAT among SGD, LHR, ISD and FSD? Some dissimila factos wee also found in suvey eseach as esults of SQ-1.1 and SQ-1.3 shows that 30% of espondents of ISD ae manages of softwae companies. Thee is no pofessional teste found in FSD in ou suvey. Mostly developes and manages pefom testing of applications by themselves. Results of SQ-1.3, SQ-1.4 and SQ-1.5 show that biggest and oldest companies ae located in ISD. Results of SQ-1.2 show that most expeienced developes and manages also belong to ISD RQ-3: What ae the opinions of testes egading MAT and how impovements can be bought in existing pactice of MAT in Pakistan? RQ-3.1: What ae the basics, SW enginees think as necessay fo defining MAT? In RQ-3.1, we asked espondents to define MAT by selecting thee choices. Result is shown in Table out of espondents (31%) think that tained and pofessional team and skill ae the basics of MAT. 36 espondents (21%) have the opinion that quality assuance methodologies fo MAT ae the basics. TABLE 7. RESPONSES FOR RQ-2.1 AND RQ-2.2 Do You o You Colleagues Pefom MAT Manage Develope Teste Othe I pefom MAT 6 (24%) 10 (40%) 6 (24%) 3 (12%) 25 (45%) I pefom MAT and my colleagues also pefom. 3 (14%) 16 (76%) 1 (5%) 1 (5%) 21 (37%) I pefom MAT but my colleagues do not pefom. 2 (20%) 4 (40%) 2 (20%) 2 (20%) 10 (18%) T otal 11 (20%) 30 (53%) 9 (16%) 6 (11%) TABLE 8. RESPONDENT WORKS AS SEPARATE TESTER Respondent Woks as Sepaate Teste Fequenc y Pecent Yes 38 68% No 18 32% 100% TABLE 9. TEST RESULTS FOR RQ-2.1 TO RQ-2.6 Chi-squae Tests RQ-2. 1 RQ-2. 2 RQ-2. 3 RQ-2. 4 RQ-2. 5 RQ-2. 6 Df p-value N 46 23

10 RQ-3.2: What uniqueness do SW enginees feel about MAT? We asked espondents what uniqueness they fell about MAT by selecting thee choices. Based on the esults (shown in Table 11) of SQ-1.1 and SQ-3.2, mobile application testing is efficient is the highly appeciated chaacteistics of MAT. RQ-3.3 and 3.4: What is the opinion of testes egading advantages and advantages of MAT? In RQ-3.3, and RQ-3.4, we asked espondents to give thei opinions on advantages and disadvantages of MAT by selecting at most 5 options (shown in Table 12). 31% espondents believe that effectiveness and efficiency ae the main advantage of MAT. On the othe hand, 26 % espondents have the opinion (shown in Table 13) that MAT is had to epot and is inflexible. RQ-3.5 and RQ-3.6: What ae the issues in MAT and what ae the key steps fo impovement in existing pactice of MAT in Pakistan? Data collection fom SQ-3.5 (issues in MAT) is listed in Table 14. Out of espondents, 34% believe that the main issue in MAT is its secuity. In RQ-3.6, we asked espondents to give thei opinions in connection with the impovements in MAT by selecting 3 options. Results ae listed in Table % espondents believe that testes may be tained in ode to impove cuent status of MAT in Pakistan. 18% believes that adapt latest technology in ode to impove MAT. 5. DISCUSSION AND VALIDITY OF SURVEY Results pesented in pevious section ae discussed in this section. Futhemoe, the most pessing issue of ou study is also discussed. Basics of MAT TABLE 10. RESPONSES REGARDING MAT BASICS Pecent Tained and Pofessional Team and Skills Quality Assuance methodologies fo MAT MAT tools MAT technology vision MAT development 14 8 MAT envionment 9 5 Othe 5 3 Chaacteistics of MAT Manage TABLE 11. MAT UNIQUENESS Develope M AT suppots ceativity 5 (20%) 15 (60%) 4 (16%) 1 (4%) 25 (15%) M AT is focused 1 (12.5%) 6 (75%) 1 (12.5%) 0 8 (5%) M AT is flexible 5 (22%) 13 (%) 3 (13%) 2 (9%) 23 (14%) M AT is efficient 11 (25%) 20 (46%) 9 (20%) 4 (9%) 44 (26%) M AT is effective 6 (19%) 17 (55%) 3 (10%) 5 (16%) 31 (18%) M AT is easy to use 3 (15%) 10 (50%) 3 (15%) 4 (20%) 20 (12%) M AT makes testing inteesting 2 (12%) 9 (53%) 4 (23%) 2 (12%) 17 (10%) Teste Othe 24

11 Regading RQ-1, we found the following: 64% softwae enginee uses Andoid platfom and 32% uses ios fo mobile applications development and testing. Advantages of MAT Manage TABLE 12. ADVANTAGES OF MAT Develope 30% (17 out of ) espondents believe that main chaacteistics of mobile application is high pefomance. 78% enginees develop and test hybid and Web mobile applications. Teste E ffective 7 (15%) 27 (%) 9 (19%) 5 (10%) 48 (17%) E mphasizes teste 5 (29%) 6 (35%) 4 (24%) 2 (12%) 17 (6%) E fficient 9 (22%) 23 (58%) 6 (15%) 2 (5%) 40 (14%) S uppot Ceativity 6 (19%) 20 (62%) 5 (16%) 1 (3%) 32 (11%) S uppots Leaning 3 (11%) 15 (54%) 6 (21%) 4 (14%) 28 (10%) T ime saving 6 (25%) 12 (50%) 3 (12%) 3 (13%) 24 (9%) E asy 6 (22%) 13 (48%) 5 (19%) 3 (11%) 27 (10%) I nteesting 3 (15%) 11(55% ) 1 (5%) 5 (25%) 20 (7%) F lexible 3 (22%) 6 (43%) 2 (14%) 3 (21%) 14 (5%) F ocused 2 (14%) 10 (72%) 1 (7%) 1 (7%) 14 (5%) I ndependent 5 (31%) 7 (44%) 3 (19%) 1 (6%) 16 (6%) TABLE 13. DISADVANTAGES OF MAT Othe Disadvantages of MAT Manage Develope Teste Othe I nflexible 5 (15%) 18 (53%) 6 (17%) 5 (15%) 34 (12%) I neffective 6 (24%) 12 (48%) 4 (16%) 3 (12%) 25 (9%) I naccuate 3 (17%) 10 (59%) 1 (6%) 3 (18%) 17 (6%) N o focus 3 (15%) 12 (60%) 3 (15%) 2 (10%) 20 (7%) U nnecessay 4 (31%) 7 (54%) 1 (7%) 1 (8%) 13 (5%) T ime limit 6 (19%) 17 (54%) 6 (19%) 3 (9%) 32 (11%) H ad to epot 9 (24%) 18 (47%) 7 (18%) 4 (11%) 38 (14%) H ad to compae esults 5 (21%) 14 (58%) 3 (13%) 2 (8%) 24 (9%) C omplicated 6 (20%) 16 (53%) 6 (20%) 2 (7%) 30 (11%) C onfusing 3 (17%) 11 (61%) 2 (11%) 2 (11%) 18 (6%) T ime consuming 5 (17%) 15 (52%) 6 (21%) 3 (10%) 29 (10%) Issues in MAT TABLE 14. RESPONSES REGARDING ISSUES IN MAT T otal Fequency (%) Use expeience 9 16 Coveage of available mobile hadwae Connectivity Secuity

12 Fo Andoid based applications, SOASTA is the pefeed testing tools fo enginees wheeas Ranoex is the pefeed tool fo ios. Thee is no diffeence of pefoming MAT with espect to time. 48% enginees use testing duing mobile application development and 52% uses it afte mobile application development Main advantage of MAT fo the company is that the company poduces high quality mobile applications. Regading RQ-2, we found the following: Thee is no diffeence in fequencies of pefoming MAT fo sw enginees. Thee is no diffeence in pefoming MAT to dissimila level of pofessional expeience. Thee is no diffeence of pefoming MAT in diffeent cities. Thee is no diffeence of pefoming MAT in diffeent sizes of companies. Thee is no diffeence of pefoming MAT in diffeent ages of companies. MAT pefomance is not affected by the existence of sepaate testing company. 30% of espondents of ISD ae manages of softwae companies. Thee is no pofessional teste found in FSD in ou suvey. Mostly developes and manages pefom testing of thei applications themselves. Regading RQ-3, we found the following: Moe than half of the espondents (52%) has the opinion that that tained and pofessional team and skill and Quality Assuance methodologies ae the basics fo defining MAT. MAT is effective and efficient (advantages) and it is had to epot and inflexible (disadvantages). Main issue in MAT is secuity and 38% espondents believe that testes may be tained and adapt latest technology in ode to impove MAT Validity of Suvey Evey empiical study has shotcomings, so does ou study too. Some issues elated to ou study ae discussed next. TABLE 15. RESPONSES ON IMPROVEMENTS IN MAT Impovements in MAT Manage Develope Teste Othe M aintain ecod fo MAT 4 (25%) 8 (50%) 2 (12%) 2 (13%) 16(10% ) S elect MAT methods accoding to app 3 (16%) 10(52% ) 3 (16%) 3 (16%) 19(11% ) R epots may be impoved 3 (27%) 5 (46%) 3(27% ) 0 11(7% ) M ake epots moe effective and efficient 2 (22%) 5 (%) 1 (11%) 1 (11%) 9(5% ) I mpove the testing pocess all the time 5 (28%) 7 (39%) 3 (16%) 3 (17%) 18(11% ) M oe isk-based testing 1 (14%) 5 (72%) 1(14% ) 0 7(4% ) U se best tools 3 (12%) 16(67% ) 3 (13%) 2 (8%) 24(14% ) A dopt latest technology elated to MAT 7 (23%) 15(50% ) 5 (17%) 3 (10%) 30(18% ) T estes may be tained fo MAT 5 (15%) 19(65% ) 6 (17%) 4 (12%) 34(20% ) 26

13 Unfainess Selection: Fistly, ou eseach consists poweful petinent. Ou goal was fully concened to the testing of mobile applications. This is why we could not appoach the vaious espondents at initial stages. Fo instance, a develope o teste who is not familia with MAT heas at a glance the suvey egading MAT; he/ she might want to ignoe the suvey as soon as possible. That is the main cause of achieving 100% encouaging eplies (espondents knows about MAT ealie). Secondly, we conducted ou suvey mostly in fou cities (SGD, LHR, ISD and FSD) but we still obtained five eplies othe than above mentioned cities. In the meantime, we wee not successful to get maximum esponses due to esticted dispesion of the suvey. Extenal Validity: An additional flaw of the suvey is the small amount of eplies. This shows a vast hazad to extenal soundness of the suvey. Moe than 500 mobile applications developes o teste could be found in above mentioned fou cities but we got only eplies. Moeove, we could not identify accuately how much pecentage of all the mobile applications developes o testes of above said fou cities we appoached and how much is the ate of thei esponses. Intenal Validity: Stength of suvey depends on design of suvey questions. If eseach questions ae pecisely conveted into suvey questions and evey one of the possibilities is enclosed by answe choices thus the quality of suvey enhanced. It means, each suvey question and its answe must be undestandable by the espondents. So, to ovecome the intenal validity theats of ou suvey, we evaluated and eoganized ou suvey questions many times. Fo this pupose, we got help fom expets/pofessos fom academia and institutions to make ou suvey questionnaie easy, inteesting, elevant to the topic, compehensive as well as clea fo espondents. Thus, we enhanced the suvey based on time to time evaluation to move up its intenal stength. Statistical Conclusion Validity: Thee is a poweful link among eliability of statistical esults, size and coveage of the sample. As quantity of sample inceases, ultimately, boad coveage of sample also inceases so statistical esults becomes much valuable. In ou suvey, the size of the sample is not vey big that is why thee is a geat theat to ou eseach on the subject of validity of statistical esults. This theat can be minimized in upcoming time by gatheing a geat numbe of samples. 6. CONCLUSION In this wok, an online suvey is conducted to investigate the cuent status and tends of MAT in Pakistan and we found some inteesting answes on the pactice of MAT. In total, we eceived eplies fom softwae enginees out of which 51 wee fom softwae companies located in Islamabad, Lahoe, Faisalabad and Sagodha. It comes to ou notice that thee ae few pofessional testes in all ou dataset, this is why, most of the developes and testes even manages of softwae companies pefom testing themselves. Results ae vey fuitful to know that what kind of tools, mobile platfoms and the test stategies ae consideed as best fo MAT along with advantages, disadvantages and how impovements can be made in cuent MAT pactice in Pakistan so it can be valuable and poficient to manages, develope and testes as well as companies. esponses in total is not a vey good statistic amount. In futue we would like to implement the suvey again and ty to get moe esponses and include othe big cities in ou suvey such as Kaachi and Peshawa. Futhemoe, in ode to get a highe esponse ate and a moe epesentative set of esponses, we would like to have an ageement with some companies and make sue that employees of those companies answe ou suvey. 27

14 ACKNOWLEDGEMENT Authos would like to thank all the espondents of suvey and seveal expets/pofessionals fom academia who helped in the design of ou suvey questionnaie. REFERENCES [1] Schweighofe, T., and Heicko, M., Mobile Device and Technology Chaacteistics: Impact on Mobile Application Testing, In Poceedings of 2nd Intenational Confeence on Softwae Quality Analysis, Monitoing, Impovement and Applications, pp , [2] Mewe, H., Mewe, B., and Visse, W., Veifying Andoid Applications using Java PathFinde, ACM SIGSOFT Softwae Engineeing Notes, Volume 37, No. 3, pp. 1-5, [3] Tian, J., Softwae Quality Engineeing: Testing, Quality Assuance, And Quantifiable Impovement, John Wiley & Sons Publishes, Hoboken, New Jesey, USA, [4] Gao, J., Bai, X., Tsai, W.T., and Uehaa, T., Mobile Application Testing, Compute, Volume 47, No. 2, pp , [5] Gadne, E.M., Which is the Coect Statistical Test to use?, Bitish Jounal of Oal and Maxillofacial Sugey, Volume 46, No. 1, pp , [6] Halle, K., Mobile Testing, ACM SIGSOFT Softwae Engineeing Notes, Volume 38, No. 6, pp. 1-8, [7] Testing expeience Special Issue on Mobile App Testing,Septembe, [8] Muccini, H., Di Fancesco, A., and Esposito, P., Softwae Testing of Mobile Applications: Challenges and Futue Reseach Diections, In Poceedings of 7th Intenational Wokshop on Automation of Softwae Test, pp , [9] Zhang, D., and Adipat, B., Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, Intenational Jounal of Human- Compute Inteaction, Volume 18, No. 3, pp , [10] Yin, H., Suvey on Exploatoy Testing, Maste s Thesis, Univesity of Tatu, Estonia, [11] Whittake, J.A., Exploatoy Softwae Testing: Tips, Ticks, Tous, and Techniques to Guide Test Design, Peason Education Pess, [12] Madhushani, B.A.L., De Silva, P.H.A.M., Madushanka, W.A.L., Malalagama, M.G.T.H., and Manawadu, D., Challenges in Mobile Application Testing: Si Lankan Pespective, Compusoft, Volume 3, No. 10, pp , [13] Pakistani Online Buyes Pefe Andoid Phones ove Othe Mobiles [online], Available at: on Januay 17, [14] Field. A., Discoveing Statistics using IBM SPSS Statistics, Sage Publications, [15] Kanji, K.G., 100 Statistical Tests, Sage Publications, [16] Pfahl, D., Yin, H., Mäntylä, M.V., and Münch, J., How is Exploatoy Testing Used? A State-of-the-Pactice Suvey, In Poceedings of the 8th ACM/IEEE Intenational Symposium on Empiical Softwae Engineeing and Measuement, pp. 5, Appendix-A: