Evaluation of Software Quality in the TopCoder Crowdsourcing Environment

Transcription

1 Evaluation of Software Quality in the TopCoder Crowdsourcing Environment Xin Wang, Wenjun Wu, Zhenghui Hu State Key Lab of Software Development Environment Department of Computer Science and Engineering Beihang University Beijing, China {wangxin, wwj, Abstract Many practitioners use rules of thumb to take participate in tasks from online labor markets. TopCoder is such a crowdsourcing platform in which users select among and compete in a collection of contests offering various rewards. Most research efforts on crowdsourcing focus on competitive behaviors of programmers and performance of contests without discussion on the quality of a complete software project that often involves several phases in a group software production process. The main objective of our research is to capture the essential features of TopCoder projects and introduce a new approach to evaluate the quality for each TopCoder project. This paper makes two major contributions: First, we model the effort level using intrinsic properties of contest and validate the definition. We evaluate quality of each contest following the definition of effort level. Subsequently we extend the quality definition to each phase of a TopCoder project and finally obtain the whole project quality. We also perform necessary validation on our model using empirical data and show some important discovery. Keywords Software Quality; Crowdsourcing based Software Development; Effort Level I. INTRODUCTION Crowdsourcing, an emerging distributed problem-solving model, has been widely used in many aspects of human life and social activities. Thus it s reasonable and tempting to adopt crowdsourcing in the area of software engineering, in which software development tasks such as requirement analysis, software design, coding, testing and verification can be conducted by workers from online labor markets instead of in-house software developers or contractors. Crowdsourcing based software engineering is an attractive research area with great potential. Most research efforts focus on studying competitive behaviors of programmers and the best methods for quality outcomes out of different kinds of software tasks. No previous work has been done on evaluating the quality of crowdsourcing-based software development in the project level. This motivates us to carry out further research on the characteristics of crowdsourcing-based software engineering. This article is the first to study crowdsourcing-based software development in the perspective of software projects. Our major contribution is to define an evaluation framework for assessing the quality of TopCoder software projects. The rest of the paper is organized as follows: Section 2 compares related work with our framework. Section 3 describes the data set that we prepare for building the quality evaluation framework. Section 4 introduces a GMM-based method to categorize the effort level of TopCoder contests. Section 5 elaborates the framework for evaluating quality of TopCoder software project. The conclusion is given in Section 6. II. RELATED WORK Many researchers have already pointed out that software quality issues in the crowdsourcing circumstance are vital to the success of a software crowdsourcing project [2][3]. In fact, a few papers have published for analyzing the quality of software developed in crowdsourcing-based software engineering[1][4]. Mao Ke et al have identified six core factors impacting software quality and suggested four aspects for enterprises to develop software with high quality [5]. Archak found that individual specific traits, project payment and the number of project requirements are three significant predictors of the quality of the final winner /17/$ IEEE

2 in a TopCoder programing contest [7]. Additionally, some previous work has been done on analyzing the quality or success of non-software tasks. Yang introduced content-based recommendation techniques for developer recommendation as an approach to guarantee the quality of software deliverables [6]. Nag et al. collaborated with TopCoder on the SPHERES Zero Program and reported this experience in [10][11]. Rogstadius et al. made the conclusion that intrinsic and extrinsic motivations had different influence on image sorting task performance, and Edelman et al. found that the larger community, greater participants, and higher payment [4]. Wu et al. made some effort on TopCoder community platform, such as describing crowdsourcing-based software development processes, they also proposed a game-theoretical model for peer software production, and analyzing the competitive behaviors occurred in algorithm contests on TopCoder[13][14][15]. III. DATA SET By crawling data from the TopCoder website, we collect 14,631 contests information totally from Feb, 2003 to Sep, 2015, except for algorithm and high school contests such as SRM and marathon match, which are not relevant to our further analysis. Our research concentrates on complete software projects on TopCoder. However, due to the unavailability of project management information stored at the backend of the TopCoder platform, we only get the limited metadata of each TopCoder programming contest without explicit and well-defined documents for these software projects. So we have to extract some common text terms as the keywords of these projects out of the available metadata of the contest. By using the keywords of the projects, we filter programming contests posted in the term of competitions on TopCoder to identify whether a contest belongs to a particular project or not. To implement the extraction process, we adopt a semi-automation and semi-artificial method. Suppose that terms such as project topic and company name could be found in contest name, we pick out these terms by Term Frequency-Inverse Document Frequency (TF-IDF) Fig.1. Features for Effort Level to Modeling and Validation in Different Categories algorithm at the first step according to (1): tf i D w * log i T df 1 (1) i Where tf i is frequency of term i, while T indicates the length of all contests title. Subsequently, we define df i as the number of contest which includes term i in its title while D refers to the total number of contests. Notice that we add offset to df i in case of the denominator to be zero. Meanwhile, it s necessary to ignore some irrelevant words like a, the, of, for, etc. We calculate the value of w for each term. Clustered by these terms, original software projects are collected and we verify the correctness and independence of each software project by rechecking carefully to filter result. Finally, 62 software projects were identified and they totally involved 1688 contests. IV. DEFINITION OF CONTEST EFFORT LEVEL The effort level of software development tasks is a very important indicator for project quality evaluation, giving a quantitative measurement about how much manpower needed for a specific development task. TopCoder website doesn t publish the effort level for each contest. Researchers often need to conduct surveys among the project participants to obtain subject estimation of every TopCoder contest, but the long timespan of these contests makes it time consuming and even impossible to some extent. Thus, we calculate the effort level of each contest through statistic methods and assume the larger value it is, the more difficult contest will be.

3 TABLE I. VALIDATION OF EFFORT LEVEL USING CORRELATION TEST The process of effort-level modeling and validation is illustrated in Fig. 1. There are some intrinsic properties fixed before a contest starts, such as payment, time for development, specification text length and the number of technology used. We model the effort level for each contest based on these factors and decide to perform our analysis separately in different categories to avoid the gap between them. Except for the factors mentioned above, the number of link in contest specification text is also considered to be added in as a new factor. To verify the correctness of definition over effort level, we apply other statistic variables that could only be obtained after contests. Data from intrinsic properties are normalized by Z-score method before modeling. These data become the input of GMM (Gaussian Mixture Model) to cluster contests. Factors listed in Fig. 1 for modeling are positively correlated with contest effort level and the sum of five factors value for each cluster center is deemed as its effort level. Hence, contest effort level can be calculated by (2). E k P E (2) C i i i 1 Where E c is contest effort level and E i is the effort level for each cluster center. P i describes possibility that contest belongs to particular cluster, which is calculated in the GMM model. To validate the definition of effort level, we adopt three factors that can be collected when the contest is finished the highest score, average score and the contest participants who passed among all registrants in the contest. According to TABLE 1, these factors dramatically present significant correlation with effort value, but slightly negative in coefficient. V. EVALUATION OF PROJECT QUALITY Based on the definition of effort level, we develop a project-level assessment framework to appraise the quality for TopCoder projects, which is shown in Fig. 2. Initially we start with the contest level by defining a new quality metric for every TopCoder contest. Subsequently, one can sum up all the contests in every phase of a project to calculate the quality of the phase. Lastly, one can evaluate Fig.2. Evaluation of Phase and Project Quality the whole project by combining the quality of each phase. Definition of Contest Quality In a TopCoder contest, each submission from contestants is graded by three reviewers according to a specified scorecard on the dimensions varying from technical correctness and clarity of documentation to flexibility of the solution. The sponsors for the contest only select the solution from the top two winners and award them contest bonus. Therefore, the score of the winner is related to the quality of the outcome of the contest. The other vital factor in quality assessment is the effort level of TopCoder contests defined in previous section. Apparently, a contest with a higher effort level demands more skills for developers to finish the required tasks. To evaluate the quality of the contest outcome in a balanced and objective way, we need to build a quality metric by aggregating both the winner score and the effort level of the task in the contest. Equation(3) presents the formula to calculate the quality of contest. Qc * Sc * Ec (3) Q c is the quality of contest in which highest score is S c, effort for the contest is E c. Where the coefficient α and β is to control proportion and adjust the quality lies between zero and one hundred. Definition of Phase Quality Based on the definition of the contest quality metric, one can further derive the quality metric for phases of a TopCoder project. Equation(4) calculates the average value of all the contests Q i (i=1,2 n) in a phase as the quality of this phase Q p. Q P 1 n Q n (4) i i 1 Based on the phase quality metric, we perform correlation analysis among different phases in all projects to reveal how TopCoder phases affect each other s quality. TABLE 2 displays the analysis result, in which we disregard phases such as Content Creation, Code, UI

4 TABLE II. CORRELATION TEST IN QUALITY AMONG DIFFERENT PHASES Prototype and RIA Build due to the limited data samples of these phases. From the TABLE 2, one can see that the quality of every phase, except for Test suites, is positively correlated with its adjacent phase from Conceptualization to Assembly. The quality of the Specification phase has impact on nearly all the other phases because the design documents generated by the Specification phase are of great importance to the other TopCoder phases. Our research finding also confirm that the quality of the Design phase significantly influences the quality of the Development phase as Nikolay have mentioned in his paper [7]. Moreover, the quality of the Assembly phase that is responsible for integration and deployment, is impacted by precede phases including Specification, Architecture, Design and Development, except Conceptualization. The quality of Test scenarios displays strong relation to Specification, Architecture and Assembly. Only Test Suites have no influence on any other phases. Definition of Project Quality The quality of a TopCoder project is defined as the aggregation of all the quality metrics of each phase in this project with the appropriate weights. Assume that the project includes m phases and W p means the payment of phase Q p. Equation(5) introduces a weight scheme for the project quality by assigning the ratio of each phase s payment over the total payment as the weight coefficient. Q Project w Q w Q w w p1 p1 pm pm p1 We sort the quality values of all the available projects in the ascend order in Fig. 3. Omicron-breeding becomes champion with the highest quality, On the other side, EPA Android App has the worst quality that only gets We also calculate the variance of quality between different phases for each project. Generally, projects with higher quality demonstrate lower variance through its phases. It indicates that the high quality variance between different phases in a project exhibits an unstable development process and easily leads to a poor software production. pm (5)

5 In order to validate our definition of the quality metric, we collect data from TopCoder in Bug Hunt that aims at seeking defects in the outcome of TopCoder projects as the number and density of software defects are often regarded as criterion to evaluate project quality. Only 27 projects among 62 projects post 44 bug hunt phases in total on the TopCoder website. Given the fact that we are not able to get exact number of bugs from the downloaded Bug Hunt data, we have to rely upon the observable properties to illustrate whether the quality metric is related to the effort of Bug Hunt. TABLE 3 shows the three factors including Payment, text length and Duration time. The factors of Payment and text length indicate the total cost and length of specification in the Bug Hunt contests for specific projects respectively while duration time implies the total time required to fix bugs. Moreover, we also perform the comparison between our quality metric and traditional method to demonstrate the effectiveness of our quality framework. In the traditional method named by TM as comparison in TABLE 3, the highest score is selected to describe the quality of contest, and the average score of all the contests in a project is calculated as the project quality. The calculation is TABLE III VALIDATION OF CORRELATION BETWEEN PROJECT QUALITY METRIC AND BUG HUNT EFFORTS Fig.3. Quality of Each Project together with Variance between Phases described in (6). Q Q Q Q 1 2 n (6) Project Q project indicates quality of the project including n contests. Q i (i=1,2 n) is the quality of contest. The result confirms that the three properties (payment, text length, and development duration) of projects present negative correlation to the project quality metric defined in Eq(5). More importantly, our quality metric show more consistent and significant correlation with the three major properties than the traditional method. It makes clear that our quality assessment framework is reasonable and performs better than the traditional way of quality metric aggregation. VI. CONCLUSIONS In this paper we investigate the issue of project quality evaluation in the TopCoder, one of the largest platforms for software crowdsourcing. We address the issue from the following aspects: (1) We design a clustering-based method to rate the effort level for each TopCoder programming contest. (2) We introduce a new quality metric for the software artifact delivered by each TopCoder contest and then extend the aggregated quality metric to the level of a TopCoder project. (3) We validate the framework of the project quality assessment by performing correlation analysis between the calculated quality metric and the effort level of bug hunting contests in TopCoder. The empirically analysis results show that there is significant correlation in quality between consecutive development phases in TopCoder projects. Further n

6 investigation reveals that high quality software product has low variance through each phase. These findings suggest that if a project manager plans to have a high quality deliverable from a program competition platform such as TopCoder, he needs to not only assure the good quality in each program contest but also attempt to maintain a stable and smooth process across all the phases in his project. ACKNOWLEDGMENT This work was supported in part by grant from State Key Laboratory of Software Development Environment (Funding No. SKLSDE-2015ZX-03) and NSFC (Grant No ). REFERENCES [1] Jan Bosch, "Speed, Data, and Ecosystems: The Future of Software Engineering", IEEE Software, vol.33, no. 1, pp , Jan.-Feb. 2016, doi: /ms [2] Thomas D. LaToza, Andre van der Hoek, "Crowdsourcing in Software Engineering: Models, Motivations, and Challenges", IEEE Software, vol.33, no. 1, pp , Jan.-Feb. 2016, doi: /ms [3] K. Mao et al., A Survey of the Use of Crowdsourcing in Software Engineering, research note RN/15/01, Dept. Computer Science, University College London, [4] Ke Li, Junchao Xiao, Yongji Wang, Qing Wang, "Analysis of the Key Factors for Software Quality in Crowdsourcing Development: An Empirical Study on TopCoder.com," 2014 IEEE 38th Annual Computer Software and Applications Conference, pp , 2013 IEEE 37th Annual Computer Software and Applications Conference, [5] K. Mao, Y. Yang, M. Li and M. Harman, "Pricing crowdsourcing-based software development tasks," th International Conference on Software Engineering (ICSE), San Francisco, CA, 2013, pp [6] K. Mao, Y. Yang, Q. Wang, Y. Jia and M. Harman, "Developer Recommendation for Crowdsourced Software Development Tasks," Service-Oriented System Engineering (SOSE), 2015 IEEE Symposium on, San Francisco Bay, CA, 2015, pp [7] Nikolay Archak Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder.com. In Proceedings of the 19th international conference on World wide web (WWW '10). ACM, New York, NY, USA, [8] Lakhani, Karim R., David A. Garvin, and Eric Lonstein. "TopCoder (A): Developing Software through Crowdsourcing." Harvard Business School Case , January (Revised May 2012.) [9] Klaas-Jan Stol and Brian Fitzgerald Two's company, three's a crowd: a case study of crowdsourcing software development. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, [10] S. Nag, I. Heffan, A. Saenz-Otero and M. Lydon, "SPHERES Zero Robotics software development: Lessons on crowdsourcing and collaborative competition," Aerospace Conference, 2012 IEEE, Big Sky, MT, 2012, pp [11] S. Nag, Collaborative competition for crowdsourcing spaceflight software and STEM education using SPHERES Zero Robotics, Master s thesis, Massachusetts Institute of Technology, [12] H. Tajedin and D. Nevo, "Value-Adding Intermediaries in Software Crowdsourcing," th Hawaii International Conference on System Sciences, Waikoloa, HI, 2014, pp [13] Wu, W., Tsai, W-T. and Li, W. (2013) Creative software crowdsourcing: from components and algorithm development to project concept formations, Int. J. Creative Computing, Vol. 1, No. 1, pp [14] Wenjun Wu, Wei-Tek Tsai, Zhenghui Hu, and Yuchuan Wu. Towards a game theoretical model for software crowdsourcing processes. In Wei Li, Michael N. Huhns, Wei-Tek Tsai, and Wenjun Wu, editors, Crowdsourcing, Progress in IS, pages Springer Berlin Heidelberg, [15] Z. Hu and W. Wu, "Game Theoretic Analysis for Offense-Defense Challenges of Algorithm Contests on TopCoder," Service-Oriented System Engineering (SOSE), 2015 IEEE Symposium on, San Francisco Bay, CA, 2015, pp