Software Process Improvement

Transcription

1 Abstract This paper discusses how to perform a test process improvement within an organisation and the different types of process improvement models you can use. The paper defines a process to use in performing a review and discusses how to implement the recommendations made as a result of the review. As part of the implementation of the recommendations, there are considerations around organisational cultural change and the impact change has on the staff. Like most change initiatives, there are a number implementation issues discussed that should be avoided in order to have a successfully implemented set of process changes. Software Process Improvement Introduction We all would agree, that no matter what type of software we are developing or building, some level of software testing is required to assess whether the software has met its stated requirements. The result of the software testing we have performed at any point in time provides an indicator of the level of risk we have mitigated. While we try to determine how much test execution needs to be performed based on risk, there are other software quality activities that can be performed or improved to build quality in across the entire Software Development Life Cycle (SDLC). These additional software quality activities are focused on other areas of the SDLC and can employ either static and/or dynamic testing techniques, e.g. requirements reviews. Software Testing and Context Software development is contextual and therefore the level of perceived risk in the project is determined by a number of attributes including: The nature of the software development being undertaken The size and complexity of the project The predicted project costs The industry segment in which the software will be used e.g. Health based systems versus internet online shopping site Building Quality In The level of process maturity can vary considerably between organisations. Immature organisations generally rely on the tacit knowledge and skills of their staff to get the job done; while other organisations have complete process frameworks in place which are tailored based on the projects context and government regulations. Figure 1 - Balance between Time, Functionality and Cost The tailoring of the SDLC is generally based on the level of perceived risk that the project has and this then drives the level of quality that is acceptable. The ability to tailor your process gives you the flexibility to; for example: write a high level Test Plan, which may cover a group of small projects, instead of writing one for each project.! The level of perceived risk plays an important role in deciding what level of process needs to be followed; this in turn determines the type of quality assurance and control processes that must be applied. The industry segment the software will be deployed in can also dictate the level of process required due to government legislation and the mandatory requirements that it places on certain software development projects. The concept of quality in a project is a balance between time, functionality and cost as shown in Figure 1. The choice of SDLC is now a common aspect of process tailoring, especially with the rapid move towards adopting an agile method. Different SDLC s have different levels of formality attached to the selected SDLC. The level of formality dictated by the SDLC is generally driven by the need to show process compliance and traceability, but even formal process can adopt agile techniques if the project context is appropriate. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 1 OF 11

2 Impact of Different Software Development Methodologies If the project is following an agile approach, then a lighter weight approach is taken in which the focus will be on team ownership and collaboration instead of creating a set of heavy documentation. The feedback on the effectiveness of the process and the approach adopted will be ascertained as part of the iteration retrospective. The team can then decide to modify and tune the approach and the process based on the outcomes of the iteration. For a traditional waterfall/v-model project, which parts of the test process framework will be used is generally defined within the Test Strategy for the project or programme. Once the process is defined, the testing approach and process very rarely change or deviate from what has been agreed. Any feedback on the effectiveness of the testing process and therefore any changes that could be made to remedy the issues have a much longer lead time in a traditional project than that of an agile based method. While an agile project actively seeks feedback at the end of the iteration a traditional project will not normally seek improvement suggestions until a Post Implementation Review (PIR) is held. Changing the processes on a traditional project is also harder given the large numbers of touch points between development phases and teams. While an agile team is a coherent entity consisting of all skills and roles, e.g. testers, BA s and developers, a traditional SDLC, like waterfall or the V-Model, has the project divided into teams based on core responsibilities e.g. the BA s belong to the Business team, the testers belong to the enterprise testing team. For this reason, organisations that want to improve their processes usually look at employing some form of software test process improvement review. These should be performed using one of the various testing process improvement models available, which we will discuss later in this paper. While these models can be seen as a formal set of tools for assessing your SDLC processes, it does not mean that these models are not applicable for assessing aspects of your agile processes. Testing Processes and the SDLC Software testing does not occur in isolation and therefore any touch points between software testing and other aspects of the development process also need to be considered when completing any review of the process you follow. The most common touch points with software testing are project management, software development and requirements management. Any process changes made to software testing will have impacts on the touch points. If consideration is not given to the impact on these touch points, it is unlikely that the recommended improvement benefits will be fully realised. The boundaries between the different SDLC teams and their individual process activities are often one area where significant improvements can be made. Any interface where there is the hand-off of information has the potential to create inefficiencies and result in a bottleneck. Test Process Improvement As software testing professionals we should always be reviewing the development processes we follow and asking the question: By asking questions and challenging the status quo, we can continue to improve our testing processes, methods and approaches by identifying areas of potential improvement. While asking these types of questions can help identify areas of improvement, it is often advantageous to utilise one of the various process improvement models to assist with performing a more structured review. Once it has been decided to carry out a test process improvement review, there is a systematic set of steps defined that provide guidance on performing a process improvement initiative. This process is discussed in section 4.2. Types of Test Process Improvement Models Process Improvement models are divided into two key types: Process Reference Models Content Reference Models A number of the models combine both types in the one model framework while others only contain one type of model. Reference Models Explained In understanding these two model types we need to understand what a reference model actually is. The definition provided by Wikipedia, shown in figure 2 provides a good description of what a reference model is in relation to computer engineering. It also describes the four key concepts that reference models embody. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 2 OF 11

3 In summary, a reference model is a model that: Describes abstract concepts i.e. it doesn t describe real things Defines entities (real things) and there relationships Only describes the things it models in the problem space or the environment in which it applies, i.e. it is contextualised to a specific environment Is technologically agnostic, i.e. it doesn t make assumptions about technology A Generic Approach to Making Process Improvements Once it has been agreed that testing processes should be reviewed, with a view to identifying areas for possible improvement, the process steps to be adopted for this activity should be as follows (as defined by the ISTQB Advanced syllabus): Initiate Measure Prioritise and Plan Define and Redefine Operate Validate Figure 2 - Definition of a Reference Model Process Reference Models Process reference model provide a framework under which the assessment can be performed and enables the reviewer to evaluate the organisations process against the model. Relating this description of a process reference model back to the definition of reference models, we can relate the four concepts of reference models to the following for a testing process reference model: The abstract things are the processes The entities are the process artefacts, like test strategies and test plans The problem space is either project or enterprise depending on the focus of the review It makes new assumptions about the technology being utilised within the organisation as the process apply across the domain Content Reference Models A content reference model is used to provide guidance on how to improve the processes once the assessment has been completed. Content models can describe the types of actions that could be performed to remedy the situation based on the findings of the process reference model. Evolve The process improvement activities outlined are applicable to all process improvement initiatives regardless of the assessment model used and who has been approached to perform the review. Initiate In this activity the confirmation of the stake-holders, the objectives, goals, scope and coverage of the process improvements is agreed. The choice of which process model will be used to identify the improvements will be made during this activity. Before the process improvement activity starts, success criteria should be defined and a method by which they will be measured throughout the improvement activity should be implemented. Measure The agreed assessment approach is undertaken culminating in a list of possible process improvements. The assessment is completed using one or more of the following methods to gather the information: Interviews with stake-holders (this may be stakeholders that are outside of the testing team, but which interact with it such as the development team, BAs and the Business representatives) Review of testware (test plans, test cases etc) Enablers such as questionnaires, workshops, demos or walkthroughs PLANIT TEST MANAGEMENT SOLUTIONS PAGE 3 OF 11

4 The next activity is to analyse all the collected information. This activity can be broken down into the following steps: Identify the gaps Validate the gaps with stake-holders, including detailed analysis where stake-holders may vary in their views Review opportunities to implement or improve best practice and reduce test cycles and areas of wastage etc. Detail any issues and investigate what is generating this Prioritise & Plan The list of possible process improvements are put in order of priority. The order could be based upon: The perceived return on investment, based on measuring the quantitative and/or qualitative benefits The level of perceived risk associated with the change Alignment to the organisational strategy Having established the priority order, a plan for the delivery of the improvements should then be developed and deployed. Define & Redefine Based upon the process improvement requirements identified, any new processes are defined, and where existing processes require updating, they are redefined and made ready for deployment. Operate Once developed, the process improvements are deployed. This could include any training or mentoring required, piloting of processes and ultimately the full deployment of them. Validate Having fully deployed the process improvements, it is key that any benefits that were agreed before the improvement(s) were made are validated i.e. benefit realisation. It is also important that any success criteria stated for the process improvement activity have been met. Evolve Dependent on the process model used, this stage of the process is where monitoring of the next level of maturity starts and a decision is made to either start the improvement process again, or to stop the activity at this point. The use of assessment models is a standardised approach to improving test processes using tried and trusted practices. Test process improvement can however also be accomplished without models by using, for example, analytical approaches and retrospective meetings. Test Process Improvement models The IT industry has started to work with test process improvement models as it seeks to reach a higher level of maturity and professionalism. Industry standard models are helping to develop cross organisation metrics and measures that can be used for comparison. A number of the test process improvement models require the reviewer to have completed some form of upfront training. The training provides the reviewer with the knowledge on how to use the model. When using these models the reviewer needs to understand the type of information to collect and then how this information can be analysed and interpreted to form the basis of a set of test process improvement recommendations. Some models are more complex then others and therefore require a significant investment of time to collect the information the models uses to enable the assessment to be performed. Deciding on who should perform the review will depend on which model you decide to adopt as the basis of your review, but you essentially have the following three choices when choosing who should perform the review: An external consultant to run the review, which is effective in also providing comparison with other organisations, or benchmarking An internal expert from within the organisation A member who is part of the testing team itself The staged models, like Testing Maturity Model Integrated (TMMi) and the Capability Maturity Model integrated (CMMI) provide standards for comparison across different companies and organisations. The continuous models allow an organisation to address its highest priority issues with more freedom in the order of implementation. Out of the need for process improvement in the software testing industry, several standards have materialised. The following are the most commonly applied test improvement models: Test Maturity Model (TMM) Systematic Test and Evaluation Process (STEP) Critical Testing Processes (CTP) Test Process Improvement (TPI) TPI and TMM, in particular, can be used for benchmark comparisons with other organisations due to the greater degree of cross-organisation metrics that they measure. Other models available but not as frequently used include Test Organisation Maturity (TOM), Test Improvement Model (TIM) and Software Quality Rank (SQR). Testing professionals should research all available models to determine which model best fits their situation. Some of the most commonly used test process improvement models are discussed in the following paragraphs at a high level. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 4 OF 11

5 Improving the Test Process with TMM The Testing Maturity Model is composed of five levels and is intended to complement CMM. The TMM level cannot be greater than the company CMM level as some of the processes may impact teams outside of the testing team and therefore they cannot be more mature than the company capability model. Each of the levels contains defined process areas that must be completely defined before the organisation can advance to the next level (i.e. staged representation). TMM provides both a process reference model and a content reference model. To achieve a particular level in the model a number of predefined maturity goals and sub-goals must be achieved. These goals are defined in terms of activities, tasks and responsibilities and assessed according to specified views for manager, developer/tester and customer/user. As an example, one of the areas covered in TMMi is test planning and for level 2 there is a goal Perform a product risk assessment. This goal is broken down into three specific practices: Define product risk categories and parameters Identify product risks Analyse product risks Improving the Test Process with TPI TPI (Test Process Improvement) uses a continuous representation rather than the staged representation utilised by TMM. The test process optimisation plan involves a set of key areas that are set within the four cornerstones of the model which are: Life cycle the life cycle of test activities related to the development Organisation the right level of organisation structure Infrastructure and tools use of the appropriate life cycle support tools Techniques the application of appropriate techniques when performing the life cycle activities The cornerstones are universal and within each test process some degree of attention must be given to each cornerstone. An example of a TPI cornerstone is shown in table 1. Key areas can be evaluated at a level between A to D with A being the lowest level and is referred to as Level A. It is also possible, within this model, for an immature organisation to not achieve the initial level A. i.e. no level has been achieved. The number of levels is not the same for each area. Each higher level is better than its preceding one in terms of time, money and/or quality. By using levels we can determine the current situation of a test process and provide better targets for step-by-step improvement. Cornerstone Key Area Description Life cycle Test Strategy The test strategy has to be focused on detecting the most important defects as early and as cheaply as possible. The test strategy defines which requirements and (quality) risks are covered by what tests. The better each test level defines its own strategy and the more different test level strategies are adjusted to each other, the higher the quality of the overall test strategy. Table 1 - TPI Test Strategy Corner Stone For each area of the model, the definition of level can be different. If we continuing on using our Life cycle cornerstone and Test Strategy key area example, table 2 lists the defined levels and their descriptions for test strategy within TPI. The level achieved for a given key area is assessed by evaluating checkpoints defined in the TPI model. If all checkpoints for the key area reporting are answered positively at level A and B, then level B is achieved for this key area. These checkpoints are achieved by answering positively to a number of questions. Level A Level B Level C Level D Strategy for a single high level test Combined strategy for high level tests Combined strategy for high level tests plus low level tests or evaluations Combined strategy for all test and evaluation levels Table 2 - TPI Life Cycle Cornerstone for Test Strategy TPI is primarily a process reference model that defines dependencies between the various key areas and levels. These dependencies ensure that the test process is developed evenly. It is not possible, for example, to achieve level A in key area metrics without also achieving level A in key area reporting, i.e. what is the use of taking metrics if they are not reported. The use of dependencies is optional in the TPI model. The approach to a TPI assessment is via capturing quantitative metrics and performing qualitative interviews to establish the level of test process maturity. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 5 OF 11

6 Improving the Test Process with CTP The Critical Testing Process (CTP) is a content reference model with the basic premise that certain testing processes are critical. These critical processes, if carried out well, will support successful test teams. Conversely, if these activities are carried out poorly, even talented individual testers and test managers are unlikely to be successful. The model groups twelve critical testing processes, under the following four categories: Plan Prepare Perform Perfect. The critical testing processes model is a context sensitive approach that allows for tailoring the model including: Identification of specific challenges Recognition of attributes of good processes Selection of the order and importance of implementation of process improvements Process improvements using CTP begin with an assessment of the existing test process. The assessment identifies which processes are strong and which are weak, and provides prioritised recommendations for improvement based on organisational needs resulting in the creation of an improvement plan. Generic improvement plans are provided by the model for each of the critical testing processes, but the assessment team is expected to tailor those heavily. Improving the Test Process with STEP STEP (Systematic Test and Evaluation Process), like CTP, is a content reference model and unlike TMMi and TPI, does not require that improvements occur in a specific order. The STEP methodology is based upon the idea that testing is a life-cycle activity that starts at the beginning of the lifecycle, during requirements formulation and continues until retirement of the system. The STEP methodology stresses test then code" by using a requirements-based testing strategy to ensure that the early creation of test cases validate the requirements specification prior to design and coding. The methodology focuses on improvement in three major phases of testing: Planning Acquisition Measurement Other concepts embodied in STEP are: Testware design leads software design Defects are detected earlier or prevented altogether Defects are systematically analysed Testers and developers work together STEP consists of specified tasks, work products and roles packaged into a system with proven effectiveness for consistently achieving software quality. During a STEP assessment, quantitative metrics and qualitative interviews are used. Implementing the Recommendations There are many ways that success can be achieved in implementing the recommended improvement activities. Successful implementation is dependent on having an appropriate change management programme in place. Here are a few examples that I have found help with the success of these improvement processes: The sense of urgency for improvement Stake-holder support Bottom up introduction Empathy and support for the staff whilst learning and applying the changes Identification of champions Start small Use a non critical project to pilot the improvements Outside consultants / new management Monitor improvements and celebrate the successes Know what you are aiming for, it is not always desirable to strive for NASA like processes When making changes to any organisation, it is important to consider how we bring about the change and move the people through the change to the point it becomes embedded behaviour. Stages of Change Kurt Lewin proposed that a successful change initiative needed to have three core stages: Unfreeze The changes are explained to the team or group and change agents are identified to assist with unfreezing the team moving forward. Engaging consultants starts the unfreeze process as well Implement the Change We now start making the recommended changes by supporting the staff through the implementation Refreeze The process is now embedded and is part of the way we do things around here PLANIT TEST MANAGEMENT SOLUTIONS PAGE 6 OF 11

7 While Lewin identified the physical stages of the change process, it is also important to understand that the people affected by the change are actually experiencing emotional responses to the changes. Emotional Reactions to Change Staff affected by the change will all experience some level of emotional reaction to it. This is shown in the graph in figure 3 Emotional Reaction to Change. The amount of time people spend in each of the emotional stages is dependent on a number of factors, mainly the size of the change and its level of impact on the individual. The sense of urgency is also important for moving people quicker through the emotional response curve (figure 3). When people know that change has to happen they are less likely to dwell on the implications of the change and are generally more willing to adopt the recommendations. This does not mean your change implementation will go any smoother or not run into people related issues, but the urgency created can certainly be utilised in a positive way to get everybody moving forward and it clearly communicates the major driver for change. Stake-holder Buy-in Stake-holders buy-in is critical to be successful and you should utilise stake-holder management processes to understand where your stake-holders sit in terms of their support for the initiative. You can classify stake-holder support by assessing what your stakeholders' attitude is towards the initiative and then classify them accordingly. The following is one classification scheme suggested you could use: Champion Work for the success of the initiative Figure 3 - Emotional Reaction to Change Implementing the Changes The implementation of change is critically dependent on developing strategies that support and enable the organisation and it s staff to move forward. Successfully implementing cultural changes in an organisation has been the subject of a number of studies and has been written about extensively. The following is a list of the most common principles considered necessary to successfully implement organisational change: Create a sense of Urgency Ensure you have active stake-holder buy-in Identify a champion Facilitate staff buy-in Start small to demonstrate success, i.e. get runs on the board Trial the process changes on a pilot project Creating a Sense of Urgency Change is much easier to make when the organisation has a sense of urgency that change needs to be made. This urgency can be created as a result of having a disaster in production, a major project / programme has gone into crisis or slipped significantly behind schedule. The urgency created by the situation can then assist stake-holders to move to identifying process changes that can remedy the problems. Supporter In favour of the initiative but not very active Neutral Sit in the middle and neither support nor oppose the initiative (but you should watch these stake-holders as they tend to move) Critic Not in favour of the initiative but not actively working against it either. These stakeholders tend to be vocal about why the change should not occur Opponent These stake-holders will work to try and derail or undermine the initiative Blocker The stake-holder is actively blocking the initiative from progressing It s important to also remember that actions speck louder then words in any change related process. If stake-holders speak about the need for change but don t actively make it themselves or enable the people implementing the change to be successful, your change initiative will fail. The issue is clearly stated in the expression, You have to walk the talk! If anybody in the change team or the stake-holders don t proactively show their support through their actions then nobody else in the organisation will buy-in to the changes. If the testing team is performing the review and implementing the recommendations, then buy-in is essential from their immediate stake-holders, their management and/or direct executive management team. Their support is required so that the testing staff involved can focus on successfully completing the process improvement related activities while not being distracted by any project related workload. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 7 OF 11

8 Implementing changes initiatives is a full time role for the period of the initiative for the following reasons: It takes time to complete the review. The review of testing artefacts is a time consuming and focused activity You need time to create a plan on how the recommended changes should be made including how to manage the people related issues Implementing the recommendations can involve a significant amount of work such as making changes to templates, and supporting staff through the changes Identify a Champion amongst the Team The use of the word champion in this context is similar to that of the category defined for stake-holders previously. While a stake-holder champion is proactively supporting and promoting the initiative at a management level, you also need champions who operate at the grass roots level to perform the same role, i.e. proactively supporting and promoting the change. If you can identify a champion within the testing team and introduce change using a bottom up technique this can help with your success. We have found that if you find during the review process, that you have identified very simple changes that are able to be implemented immediately, you will start generating more positive interest and acceptance of the overall change initiative. This supports the identified champions by providing a head start in selling the change initiative to their peers. The champion for the process improvement activities often turns out to be the person that made the suggestion to management that a process improvement exercise should be undertaken. If this is not the case, a champion will frequently appear during the various interviews held with the staff as you investigate the situation. The person chosen for the role of a champion will usually show their enthusiasm about the proposed changes and be proactive in trying to get the changes made. If this approach is taken to identify your champions from within the team, we would discuss the identified staff with the stake-holders about the opportunity of engaging the identified staff as champions or change agents in the overall initiative. Generating Staff Buy-in Generating and obtaining Staff buy-in is one of the most difficult aspects of any change initiative, but this should not stop the initative from generating creative ideas on how to get the staff to buy-in to the proposed changes. In most cases, staff buy-in can be achieved simple by involving them actively in the review and implementation aspects of the initiative. Staff are no longer willing to simply accept any change pushed onto them and generally want to be part of the processes. Looking at a group of testers, you will generally find a group of individuals who are extremely happy to provide feedback, criticisms and suggestions on what they see as the problems and how they believe they can be corrected. They essentially want to have a voice and be heard. Employing a consultant, defined as somebody from outside the organisation, to perform the review will generate lots of open and candid discussions about the problems and issues. Staff feel like they now have a voice or a conduit to express their problems, issues and concerns, but more importantly, they usually have some good ideas on what the solutions could be. If you use an internal person, i.e. somebody from the testing team or an internal process or project office group to perform the review, you are likely to find staff less open to discussion about the problems with the processes or the issues that they face, for fear of retribution. Process improvement initiatives have to create an environment where discussing the issues is seen as a positive contribution in changing the organisation and moving it forward. No staff member will buy-in if speaking out about the problems is seen as a black mark against them. These types of environments are often toxic and implementing any fundamental change is extremely difficult. If you can introduce a number of small changes then they will soon add up, generating a positive view of the change initiative and subsequently generating more buy-in from the team. Start Small to Demonstrate Success Often making suggestions that require little change to the current process, but that have significant savings (normally in budget and resource time) will allow the team to see immediate benefits. Once they start to see these quick wins they are more likely to be willing to implement further change. Some examples that I have found in conducting this type of exercise are listed below: Any fields within the defect management tool that are required for the reporting of critical project metrics should be made mandatory. Metrics are fundamental to management of a project. If the data used to create the metrics is incomplete or incorrectly entered then all the data abnormalities will need to be corrected to ensure that the metrics being reported are correct. If this is not done then planning may be impossible or the wrong corrective action may be taken. Consideration should be given to ensuring that you are only collecting the metrics that you need to report the amount of residual risk remaining in the system or product. Metrics collection is time consuming and there is no point wasting time collecting metrics data that is no longer required or not used. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 8 OF 11

9 The introduction of templates for test artefacts can greatly increase the quality of the testing effort by ensuring that consistent information is captured and presented to the Business stake-holders. It should be pointed out that templates are not the replacement for the critical thinking and analysis that needs to go into the information captured by the template. Too frequently, the template is the process and the template s intent changes from one of prompting the author about areas or subjects they should consider to simply copying and pasting information from one document to another. At this point in the change process we are trying to refreeze the group so that the processes become embedded in the organisation. There are a few challenges to refreezing, often brought about by the implementation approach for the changes. Use a Pilot Project to Implement the Process Changes When implementing a change, it is best to take a pilot approach and pick a non-critical project in which the staff on the project have the time to actually implement the changes and get used to the new way of doing things. Not only does this get a group of people up skilled it also creates a pool of champions to go out into other projects and assist the team rollout the changes. Often we find that there is a big bang approach to implementing change, and whilst we do not want to discourage eagerness to improve, teams try to make the changes while they are running mission critical projects. When the pressure is on to complete these projects, often in less time than required due to other project overruns such as late delivery of requirements, testers often revert back to what they know or are most familiar with. One of the most powerful motivators is to see other team members achieving efficiencies and becoming more effective by following the improvement recommendations. Part of the recommendation plan must include metrics by which we can measure, monitor and validate improvements. As the team can see success they are more likely to embrace more change. Consultant versus Internal Staff for the Test Process Review Often the most important decision after agreeing a review should be performed is determining who should actually perform the review. The answer to this question is based on two options: Use the staff from within your organisations testing team Bring in an external consultant to perform the review One aspect of potential drivers for change is the impact of a new management team and/or management structure. These changes can have a similar effect as that of engaging an external consultant to perform the review. When new management join a team, they will bring with them ideas from their previous organisations which they will want to implement. If they are any manager worth their salt they will speak to their team and find out where they think improvements can be made. When there is new management they will be given some time to settle in and bring about change, in fact this is often what is expected. Using the Team to Perform the Review When the test process improvement exercise is carried out from within the testing team there are both positives and negatives that need to be realised before the exercise starts. The positives aspects of using the team to perform the review are: In many cases the testing team know what is wrong with the processes that they are using The team is able to make recommendations about which improvements will fit most appropriately within the company culture and environment The following are the negative aspects to be aware of when the team performs the review: Self initiated improvement programs by the team may suffer individual bias, i.e. their level of independence (from the organisation) is not high enough The team doesn t have the time to review and implement the recommendations due to project commitments Unless the internal staff member has significant respect in the organisation the review may not have any substance and simply dismissed by stake-holders The improvement initiative is likely to be impacted by internal organisation politics more when the reviewer is a staff member Unable to see the forest for the trees is the problem you suffer when trying to review your own documents. You can t see the errors as you are to close to the product The review and recommendations will be impacted by the experience and exposure of the individual to other organisations and their knowledge of current industry best practice. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 9 OF 11

10 Using a Consultant for the Review One of the primary reasons behind hiring a consultant is to bring in outside ideas to simulate change. Unfortunately, this also comes with the inference that the team is unable to provide this perspective or add value through the suggestion of fresh ideas. Any consultant engaged to perform a review needs to understand and be aware of the sensitivities that are created by this inferred perception. Some team members may be rather demoralised as they perceive that there views are unimportant or simply being dismissed by management. Using an external consultant certainly has a range of positive factors that need to be considered, some of these factors are: A Consultant has a the level of independence versus that of the team member While staff may be fearful of making comments or improvement suggestions to stake-holders, they often feel empowered to make them to a consultant who isn t tired to the organisation. Anonymity is usually a requirement for this to occur An external consultant is not impacted directly by an individual s bias or the political landscape they are working in. That said, the consultant needs to understand the political landscape to ensure that the report and recommendations are suitably framed A consultant brings a broader range of experiences from multiple organisations and industry areas thus enabling them to provide a broader based review and comparison against a number of organisations and industry best practice Some of the negatives aspects that can arise from using a consultant are: Staff maybe reluctant to provide information to an outside consultant for fear of retribution. This is highly dependent on the organisation s culture The cost of using an external consultant can be potentially more expensive then using internal staff Unless engaged to assist the organisation to implement the recommendations, consultants have limited or no accountability for the outcome They are able to influence the management team to ensure that the testing team is given the time to implement these changes and address the issues The Power of Post Implementation Reviews (PIR) Post Implementation Reviews (PIR), are one of the process activities that is significantly undervalued in all of the test process reviews I have completed. There is often pressure in the form of test execution is now complete so all resources can be moved to another project mentality rather than lets have a period of reflection on successes and lessons learnt. These closure activities can lead to significant improvements in subsequent projects. For example reviewing your regression pack of test cases are they still relevant, have new features been included or is the correct risk profile for the system under test been maintained. Conducting a PIR, which includes all the team (here the team is testers, developers, BA s, the business, support etc), will ensure that successes are reinforced, which are equally as important as identifying failures and what needs to be done to rectify them. Sadly, while a PIR can provide significant insight into the successful aspects of the project as well as the not so successful aspects, many organisations simply fail to learn! They continually repeat the same mistakes project after project even when each subsequent PIR highlights the same set of issues. This is the single biggest issue faced by the PIR process and the major reason why most staff within an organisation where this behaviour is evident view the process as a waste of time. Ironically, these are also the organisations that request process improvement reviews because they don t actually believe the information coming out of the PIR. As Winston Churchill said, Those who fail to learn from history are doomed to repeat it. Common Pitfalls to Avoid Like all initiatives, there are some common pitfalls that these types of improvement processes can fall into. You should avoid these at all costs, but awareness of the potential pitfall is the first step in avoiding them. The Blue-Sky Scenario One of the primary reasons why process improvement exercises fail is that the recommendations are aimed at blue sky scenarios, i.e. the perfect picture of software development, rather than what is required for the context in which the process improvements will be applied. The types of projects that the company is undertaking will have an influence on the rigour that needs to be applied to the processes that are improved or implemented. For example a small development shop that has maybe a few small clients will not need to have the same level of process as a enterprise wide corporate financial institution dealing with millions of client transactions or a company developing medical equipment. Know what you are aiming for, it is not always desirable to strive for NASA like processes. PLANIT TEST MANAGEMENT SOLUTIONS PAGE 10 OF 11

11 The more process that you put in place and have to maintain then the likelihood is that the cost will increase. You may be incurring costs for no benefit and in fact even worst having a negative impact by over-engineering process. Also testing is likely to be inefficient, compared to the risks identified in the project being undertaken. The key is to have flexible processes which employ just the right level of management and controls for the type of project and cultural environment. Inappropriate or Aggressive Time-scales Often the time-scales to implement the recommendations are fast tracked i.e. an aggressive implementation schedule is created. This is usually done as a result of projects in progress or pending that still need to be serviced with testing resources. Staff need time to learn the new processes and use the new practices. If pressure is applied and the process and practices are not embedded (i.e. Lewin s re-freeze change stage hasn t occurred), the staff will revert to what ever method they know will get the job done in the time frame required. Process change activities can range from updating or creating templates through to documenting the entire refactored test processes. All these types of process activities take time both to implement and for staff to learn. Conclusion Process improvements are relevant to the software development process as well as for the testing process. Learning from one's own mistakes, and those of others for that matter, as well as reinforcing the successes, makes it possible to improve the process organisations are using to develop and test software. One premise for process improvement is the belief that the quality of a system is highly influenced by the quality of the process used to develop the software. Improved quality in the software industry reduces the need for resources to maintain the software and thus provides more time for creating more solutions in the future. A culture where the team understands what Quality means to them and how to make it happen will engender continuous improvements. Process models provide a place to start improving, by measuring the organisation s maturity processes with the model. The model also provides a framework for improving the organisation s processes based on the outcome of an assessment. Once an assessment is performed, TMMi and TPI provide a prescriptive roadmap for improving the test process while STEP and CTP provide the organisation with a means to determine where its greatest process improvement return on investment will come from and leave it to the organisation to select the appropriate roadmap. Change management is more than simply changing processes; it is also about changing people, as highlighted by the definition shown in figure 4. It is often poor people change management processes which limit the value that can be gained from test process improvement activities and not resistance to the recommendations themselves. Figure 4 - Definition of Change Management Organisational change management processes include techniques for: Creating a change management strategy (readiness assessments) Engaging senior managers as change leaders (sponsorship) Building awareness of the need for change (communications) Developing skills and knowledge to support the change (education and training) Helping employees move through the transition (coaching by managers and supervisors Methods to sustain the change (measurement systems, rewards and reinforcement) In order to be successful, most of the above list needs to be in place to realise the benefits of the test process improvement process. Leanne Howard is a Principal Consultant with Planit Test Management Solutions. She has conducted a number of process reviews over her 20 years experience in the industry PLANIT TEST MANAGEMENT SOLUTIONS PAGE 11 OF 11