WHITE PAPER. Development Testing for Agile Enterprises Helping Teams Maximize Velocity

Transcription

1 Development Testing for Agile Enterprises Helping Teams Maximize Velocity

2 The Need for Speed Companies in almost every industry use software to drive innovation and compete in today s marketplace. And competing today isn t just about being better but delivering faster. In fact, 81% of CEOs believe that innovation requires getting to market quickly. 1 The race to market with software innovation has resulted in an explosion of code being developed. Of course, no one wants to sacrifice quality or security for speed, but poor software quality is costing $60 billion per year in the US alone. 2 In addition, 80% of software development budgets are currently spent finding and fixing defects, NOT on innovation and getting to market faster. As a result, most companies want to increase efficiency across the board so they can improve both quality and time to market. This has led to more and more adoption of Agile methodologies. In fact, in a recent survey by VersionOne, 84% of the organizations surveyed are using Agile methods and practices. 3 This is because most of the Agile principles 4 revolve around key concepts that resonate in today s fast-paced, highly competitive market. These key concepts correspond directly to the top reasons organizations cited for adopting Agile in the VersionOne survey. Key Agile Concept Reason for Adopting Agile * Release frequently: Get software in customer hands quickly. Provide high quality: Deliver secure software that works. Adapt to change: Plan ahead but maintain the flexibility to respond quickly to market demands. Collaborate and provide visibility: Open and frequent communication is critical, both within the organization and with the customer/end user. Accelerate time to market (95%) Enhance software quality (93%) Manage changing priorities (97%) Project visibility (90%) Over the years, many different Agile methodologies have emerged, and they generally fall into one of two categories: fully Agile or some sort of hybrid Agile. Fully Agile environments tend to work best for consumer-driven and web-based applications where getting new features out quickly can provide a distinct competitive advantage. However, in some environments, such as enterprise software, where the customers are large corporations with applications installed onsite in complex environments, frequent releases may not be an option because IT departments don t have the resources to continuously roll out new versions. In this type of environment, it is more common to integrate some Agile techniques into an existing waterfall or other predictive methodology. Agile Pure Agile development doesn t work for all software, but it fits very well when frequent software updates are required. Getting new features out quickly can be critical in highly competitive markets such as: Consumer applications: These are typically installed on laptop and desktop systems that can automatically update over the internet to add new features. Mobile applications: These are installed on smartphones and tablets which can easily accommodate frequent updates over wi-fi or a user s mobile network. Embedded software for online devices: This generally refers to the underlying operating system on devices such as smartphones, tablets, cable boxes, gaming systems and other devices that maintain network connectivity for easy updating. Web applications: This refers to software accessed through a browser that can be updated on the back end without any action by the end-user. * % of respondents who consider this an important reason for change 1 Forbes: Global CXO Outlook, NIST: The Economic Impacts of Inadequate Infrastructure for Software Testing, VersionOne: 7th Annual State of Agile Development Survey, Principles behind the Agile Manifesto,

3 Joe Little 5 recently made a great analogy in his blog. The customer is thirsty and wants a drink now. The customer doesn t want to wait three months for a delivery of 200 water bottles just a drink now, please. This is practically the definition of Agile. Release a small set of features in a few days or weeks instead of a huge set of features six months or a year from now. Hybrid Agile More common than pure Agile development, hybrid Agile development is often used when frequent updates are not an option. The hybrid is typically a combination of Agile, which incorporates adaptive planning, and Waterfall, the most common predictive planning method. Organizations often use this type of hybrid methodology to take advantage of many of the principles of Agile development while still maintaining a longer release cycle that may be six months to a year or even longer. Software using this type of hybrid methodology may include: Enterprise software that requires the customer to install locally: This type of software often has a complex upgrade process that requires a planned roll-out by the IT department. Consumer software that requires the user to purchase the latest version: Some consumer software may be set up for autorenewal and thus may still fall into the pure hybrid category, but other software may require purchase of a new or upgrade license to get a new version with updated features. This may require a manual software upgrade and will typically have a longer release cycle. Embedded software for offline devices: Automobiles, pacemakers, appliances and other devices may rarely be updated, if ever. In many cases, the device will require authorized personnel for updates, and in some cases, the only way to get the latest software version may be to buy the latest model of the device. The Need for Development Testing Whether purely Agile or a hybrid, quality and security must be as important as speed, which means it s critical to test early, test often and know when the product is done. To meet aggressive sprint cycles, testing must be done in conjunction with code development. Teams can t afford to wait until a formal Quality Assurance (QA) cycle to begin testing. If issues can be identified earlier, they can be fixed easily, while they are still fresh in the developers minds and much less costly to fix. And if issues aren t addressed early, technical debt will accumulate which will have to be repaid at some point or velocity will slow. Development testing comes in many forms including static analysis, unit testing, regression testing and other types of testing that may be performed during the development stage before moving on to QA. Static analysis can be easily implemented along with continuous integration so that code is analyzed as part of the check-in and build processes. This ensures that issues are addressed as soon as they are introduced. Unit testing refers to automated testing using tests written by developers as they code to be sure that specific sections of code function as expected, and regression testing is used to confirm that anything previously working doesn t stop working because of code changes. Integration testing takes the individual modules that have been unit tested and aggregates them into larger modules to test the functionality, performance, and reliability of the major design items. These types of automated tests can be very valuable in identifying issues early, and unit testing, in particular, has become a core Agile tenet with 74% of development organizations adopting unit testing as part of an Agile method. 6 One of the biggest issues in any agile environment is often determining when the product is done. Development testing enables teams to define specific criteria that determine when the product is ready for release. Some example criteria may be: all new code automatically reviewed and analyzed, all critical defects fixed, all critical code tested, and other criteria deemed vital to the organization. With this criteria clearly defined, organizations have the visibility and predictability necessary to increase development velocity for rapid and resilient software delivery. The Coverity Development Testing Platform The Coverity Development Testing Platform enables developers to build quality and security testing into their Agile development process and improve the overall efficiency of their automated testing efforts. The platform provides code intelligence, transforming raw source code into meaningful and actionable information. Code intelligence provides deep analytics into your code, automatically analyzing large amounts of code with a 5 Agile & Business blog by Joe Little, an Agile coach and Certified Scrum Trainer (CST) through Scrum Alliance. 6 VersionOne: 7th Annual State of Agile Development Survey,

4 deep understanding into the ripple effect of a code change, and delivering meaningful metrics and actionable information that is critical to making your software development release fast and predictable. It identifies where there is risk in the code, provides remediation advice to fix issues and reduce risk, detects testing gaps, and tells when code is sufficiently tested. This code intelligence comes from Coverity SAVE, the awardwinning analysis engine for the Coverity Development Testing Platform. Based on more than a decade of research, development and analysis of more than 5 billion lines of proprietary and open source code, Coverity SAVE applies multiple patented analysis techniques to automatically test code as it is written. It can identify hard-to-spot, traditionally untestable issues with accuracy and precision, minimizing false positives. Coverity SAVE provides full path coverage, ensuring that every line of code and every potential execution path are tested. Coverity Quality Advisor and Coverity Security Advisor help developers reduce their risk and improve their efficiency by testing their code for critical defects such as null reference pointers, memory leaks, and potentially exploitable security defects such as SQL injection and cross-site scripting from within their IDE or as part of the continuous or central build system. These solutions enable users to find hard-to-spot issues in the largest, most complex code bases. They complement solutions designed to find coding style and standard issues, such as FindBugs and Microsoft Visual Studio Static Analyzer, by enabling developers to incorporate and correlate the issues found by these tools so all issues and defects can be managed within the Coverity platform. This provides users with one process and one workflow for remediating all of the defects in the code. Plus, the earlier defects are found in the lifecycle the less they cost to remediate. With the compression of testing cycles in Agile environments, development teams can t afford to waste cycles testing code that is not important. Coverity Test Advisor improves the efficiency of automated testing such as unit testing, regression testing and integration testing by focusing time and resources on the most critical parts of the code. Instead of chasing a naïve coverage number such as 60% code coverage, which may include debugging and exception-handling code, organizations can ensure that their high-risk code, such as recently changed code, code impacted by a change, or the code of critical features, is tested sufficiently. With Coverity Test Advisor, critical areas of the code that are missing unit tests are automatically flagged. Then testing violations are generated and automatically assigned to developers for remediation. Coverity Test Advisor also provides a list of recommended tests based on changes in the source code since the last analysis was run, and identifies which tests to run first. With improved visibility and focus, development teams can also minimize the number of unit tests they generate with solutions such as JUnit. Another area that can impact velocity is code complexity as well as architectural organization. Most legacy codebases lack documentation and may not be organized according to current design rules and specifications. Adding new features takes longer and longer while developers get more and more frustrated. Coverity Architecture Analysis provides a high degree of visibility into code complexity with a simple and actionable measurement framework. It discovers and maps the architecture to the actual implementation providing a view of the overall code structure to identify areas of excess complexity and enforce design rules. To maintain velocity, developers must be able to quickly resolve quality and security issues. Coverity Connect provides developers with a single console to quickly remediate issues identified by Coverity Quality Advisor, Coverity Security Advisor, Coverity Test Advisor, and third party solutions such as FindBugs to resolution within a unified workflow. Developers are presented with accurate and actionable information about the severity of issues, potential impact and where they occur in the code. This detailed and intuitive information helps developers get to work quickly on resolving the most critical issues and keep the product schedule on track. Plus, developers can identify all of the places defects exist in the code base across projects and products, which saves valuable time. Through Coverity Policy Manager (See Figure 1), managers and executives get visibility into the overall quality and security of internally developed code and third party code across 4

5 Figure 1: Coverity Policy Manager provides visibility into risk so management can make better decisions and release with confidence. development sprints. Deciding when a project is ready for a release is a key principle in Agile. In Scrum, one of the most common frameworks for project management used in Agile software development, teams hold daily meetings to report progress and make decisions. Coverity Policy Manager provides rich data and metrics about the quality and security of each sprint. This visibility is critical when trying to balance the need for rapid time to market with the need for acceptable quality levels. Integration into Software Development Lifecycle With so many extra pressures on developers today, they need solutions that will tie into their existing software development tools and processes. The Coverity Development Testing Platform integrates seamlessly with common Agile development practices and market-leading tools. Continuous Integration One of the most common practices in Agile development is continuous integration (CI). It has been embraced by the majority 7 of the Agile community because it enables a faster, automated central build process. By increasing the frequency of integration that CI provides, delivery teams improve their visibility of the overall quality of the software. Integration issues, build problems and code conflicts are surfaced more quickly, allowing faster remediation. In order for a development testing solution to work in an Agile environment, it is essential that the analysis is done as frequently as the source integration happens. The analysis needs to be automated, fast and scalable, especially when the development team is large. 7 56% according to VersionOne: 7th Annual State of Agile Development Survey,

6 The Coverity platform provides integration with many continuous integration and automated build tools such as Jenkins (See Figure 2), CruiseControl, Bamboo and others. Each time a central build is generated, Coverity can automatically test the code for high-severity defects. Users have the flexibility to adjust the testing to conduct the most complete analysis or only identify the highest-risk defects. Once defects are identified, they are published to the defect tracking database in Coverity Connect for tracking and management. Developers can receive automatic notification so they can quickly begin the triage or inspection process and fix new defects. Unit Testing Unit testing is valuable in concept but can often be inefficient in practice because code coverage is commonly used as the metric to determine adequate vs. inadequate testing, and this metric can be misleading. Without the proper code intelligence, or insight into the code, coverage is just an arbitrary number that can waste developer time and increase risk. According to Forrester, organizations should test smarter to test less. 8 This means focusing testing on the most important and riskiest areas of code. Code supporting critical features, new code, or legacy code that is impacted by a change are examples of critical code that should be thoroughly tested, while legacy and debugging code are lower risk and may not require testing. Coverity Test Advisor integrates with code coverage tools to provide the code intelligence necessary to identify high-risk code and prioritize tests with insight on what needs attention and what can be ignored. Users can establish and enforce consistent testing rules across projects and teams based on high-risk criteria. Testing rules guide developers on what tests they need and don t need to write. This ensures high-risk and changed code is tested sufficiently as part of a standardized process. Figure 2: The Coverity Plug-in for Jenkins (available on the Jenkins website) 8 Forrester, Seven Pragmatic Practices To Improve Software Quality

7 Desktop Analysis Coverity enables developers to analyze their code in minutes, depending on the code size and complexity, and remediate any defects found from within Eclipse (See Figure 3) or Visual Studio on their desktop. Developers can view the defects, understand their severity and impact, and link to the Common Weakness Enumeration, an industry-standard knowledge base, for additional information. Once the code has been analyzed on the desktop, the developer can check it into the continuous integration server or central build system where the analysis engine can evaluate the cumulative changes of the entire development team. To save time, developers can also choose to utilize incremental analysis, which only analyzes the files that have been changed or those affected by the change instead of the entire code stream. By scanning the code from the desktop, developers are able to address security and quality issues immediately. Extensible Platform The Coverity Development Testing Platform is extensible, which means that in addition to the out-of-box integrations provided, the platform also allows rapid integration of additional continuous integration systems, build systems and other components in your development environment such as source control management systems, bug tracking systems, and application lifecycle management (ALM) solutions such as HP ALM and IBM Rational Team Concert (RTC). The extensible platform even enables integration with third party analysis tools so that the results are visible in Coverity Connect. This provides a unified system for simplified code analysis and management. Figure 3: Coverity Plug-in for Eclipse (also available for Microsoft Visual Studio) 7

8 How Development Testing Fits We spoke with some of our customers to understand how they incorporate development testing into the software development lifecycle. Between Agile and hybrid Agile, we found some key differences. Agile Hybrid Agile Release cycle <week 2 months 6 months 2 years Testing Mostly automated Some automation Sprint duration A few days 2 weeks 2 4 weeks But regardless of whether they are fully Agile or a hybrid, we found that most customers use development testing as a stage gate (See Figure 4) with clearly defined defined criteria based on code intelligence metrics. This measurable gate ensures that code meets specific requirements before it ever leaves development, provides QA with predictability in the quality of the code, and minimizes feedback loops. Customer: Global Engineering Software Company Agile-Waterfall hybrid, release cycle over 12 months This company began adopting Agile methods in 2009 primarily to improve quality. They brought in an Agile scrum coach an started bringing teams under Agile one by one. The release cycle can t be less than 12 months because their customers can t implement more frequently, so they use sprints to improve time to market for patches and fixes. The company has not yet implemented continuous integration, and unit testing is still not prevalent though some teams are better than others. Sometimes testing piles up to the end of a sprint, turning it into a miniwaterfall. However, they are moving toward test-driven design and have recently implemented an Automated Test Plan (ATP) program at the team level which specifies all code is supposed to have an automated test written. Current process: (not a strict stage gate but used to determine when a release is done ) Developers run Coverity analysis (in Visual Studio) and unit tests on the desktop. Each team runs Coverity analysis before checking in code. Coverity analysis runs again for weekly full build, and all tests are re-run. Benefits realized from Agile and development testing: Improved quality and product stability More effective communication Accountability and teamwork entire team is responsible and will help others who are stuck or delayed Figure 4: Development testing in Agile environments 8

9 Customer: A Leading Provider of Set Top Boxes Agile, varying release cycles This company adopted Agile to drive software quality and development testing to force developers to fix bugs. Interestingly, the sponsors are not part of development. They are part of Global Configuration Management, the team responsible for implementing the new process. The company uses several Atlassian tools, Bamboo for continuous integration, JIRA for bug tracking and Crucible for code review, along with ElectricAccelerator from Electric Cloud for build automation. Currently Bamboo calls ElectricAccelerator to generate the build, and with continuous integration, this happens every time someone checks in code. The plan is to implement a stage gate so that when a problem is detected, the change is backed out and the developer is notified. The stage gate is designed so that the build fails if any of these criteria are met: Compiler failure JIRA ticket not set to proper state No code review by Crucible Test suite failure (using a quick smoke test) Coverity identifies any severity 1 or 2 defects Black Duck identifies any compliance issues (integration reports defects in Coverity Connect) Customer: A Leading Oil & Gas Exploration Company Hybrid Agile, varying release cycles This company was struggling to determine when the software was done enough to make a release, and they came up with a unique way to address it. The development testing process is fairly typical. A developer checks in code for a build in Jenkins, and the Coverity analysis runs. A list of new defects introduced goes out to the developer, and a trouble ticket is automatically opened in ClearQuest. The developer then has to fix these issues and check the code back into Jenkins. As they near the release date, it gets more interesting. The company has structured the process such that one of their development sprints is purely focused on eliminating highimpact defects found by the Coverity platform. At the end of that sprint, they use a voting tool for the team members to indicate whether they think the product is ready for release. If the number of defects found by the Coverity platform exceeds a certain threshold, they vote no on the readiness of the release. Customer: Direct Edge Agile, 4-6 week release cycle To keep pace with the rate of change and service delivery requirements, Direct Edge realized that traditional QA practices were simply too reactive, as finding an issue late in the cycle impacts cost, time to market, and risk. To transform software testing from reactive to proactive they implemented Edgile a customized development model based heavily on Agile principles and methods. 9 The intent was to improve the agility of the release management and software development lifecycle and reduce the QA cycle by shifting software testing upstream into development. It tightly integrates QA and Development together and shifts testing from a reactive to a proactive process to detect issues as early as possible. They automated traditional QA regression testing and introduced development testing which has shortened the downstream QA cycle by surfacing and remediating issues early in the process. Direct Edge also implemented a daily stage gate process, meaning a failure at any step resets the flow back to the initial code modification step. Developer local build and verification with Microsoft Visual Studio Code commitment to Subversion project repository via VisualSVN Peer code review with Trac Unit testing with Microsoft Visual Studio Full integrated build with CruiseControl Static code analyses with Coverity solution Coverity: Our Own Agile Processes Agile-Waterfall hybrid, 6 month release cycle Here at Coverity, we embarked on a journey in 2011 to automate testing as part of the development lifecycle. 10 We already used some Agile methods such as scrum and two-week sprints for building features, but we relied heavily on outsourced, manual QA operations. One uncontroversial tenet of Agile is to test code as it is written, and our goal was to transform our own organization to follow this tenet. We are still going through this transformation 9 Saro Jahani, CIO of Direct Edge, Coverity Exchange NYC event, Coverity: Transforming Testing Through Automation,

10 process, but we have come a long way in two years and are now automating 98% of our end-to-end testing. Many organizations struggle to determine when a product is done, and we faced the same struggle. Ultimately, a set of passing tests should be in place before a feature is declared accepted in the Agile development process. The problem is how to define that set of tests and know that the code is sufficiently tested. It quickly became clear that we needed a way to direct our test development efforts to the most needed areas. Code coverage is a common metric, and our code coverage for the front-end and analysis components was about 83%, which is very good. But the concern is that we had no intelligence about what was hidden in the uncovered 17%. This led to the development of an in-house solution that enables development teams to specify a test policy, defining which code is high risk so testing efforts are focused in the right place. This solution was so useful that it became a new product: Coverity Test Advisor. Our current process is similar to some of the others outlined above. For the build process, we use continuous integration with the Jenkins platform, and we perform continuous Coverity analysis with scans running every few check-ins. We have stage Figure 5: Testing stage gate gate for testing (See Figure 5). Coverity Clean which means all critical defects have been addressed. Previously this only referred to quality and security defects, but for the upcoming release we added another stage gate for testing. Coverity Clean, at Coverity, now means free of quality and security defects AND no test policy violations. We see some customers starting to add this testing stage gate as well. Summary Companies are adopting Agile development methods primarily to manage changing priorities and delivery higher quality, more secure software faster. As a result, development testing is rapidly increasing in adoption due to the need for maximum development velocity. Velocity isn t only about speed and delivering more, better and faster. It is also about productivity and minimizing rework as well as predictability to deliver quality software that provides business value and increases customer satisfaction. Regardless of the development method, the cost of finding and fixing defects increases exponentially as software moves through the development cycle, from design and coding to release. Development organizations are being held accountable for quality and security, so they can no longer afford to wait until the end of a formal QA cycle or security audit to be informed of defects that need to be addressed. They need an automated solution for assuring the quality, safety and security of their software that keeps up with the rapid iterative development process. Coverity development testing enables developers to test early and often so they can assure code quality and security at each development sprint. The code intelligence provided by the Coverity platform delivers the accountability, visibility and predictability needed to maximize velocity. About Coverity Coverity, Inc., the leader in development testing, is the trusted standard for companies that need to protect their brands and bottom lines from software failures. More than 1,100 Coverity customers use Coverity s development testing platform to automatically test source code for software defects that could lead to product crashes, unexpected behavior, security breaches or catastrophic failure. Coverity is a privately held company headquartered in San Francisco. Coverity is funded by Foundation Capital and Benchmark Capital. For More Information Find out how Coverity can help your organization improve the quality and security of your software, improve the effectiveness of your unit testing efforts and enhance your Agile development initiatives. To learn more, contact your Coverity representative or visit us at For More Information info@coverity.com Coverity Inc. Headquarters 185 Berry Street, Suite 6500 San Francisco, CA USA U.S. Sales: (800) International Sales: +1 (415) sales@coverity.com 2013 Coverity, Inc. All rights reserved. Coverity, the Coverity logo and Coverity SAVE are registered trademarks of Coverity, Inc. in the U.S. and other countries. All other company and product names are the property of their respective owners