BCS HIGHER EDUCATION QUALIFICATIONS Level 6 Professional Graduate Diploma in IT. October 2012 EXAMINERS REPORT. Software Engineering 2

Transcription

1 General Comments BCS HIGHER EDUCATION QUALIFICATIONS Level 6 Professional Graduate Diploma in IT October 2012 EXAMINERS REPORT Software Engineering 2 The pass rate was significantly below that of the summer examinations. There were some very good scripts, but the majority were poor. Given that this is the second set of examinations under the new syllabus, it would appear that the preparation of candidates for the examinations may be an issue. Of concern is whether recommended texts are being used to cover the new topic areas such as software measurement; or, the learning is primarily targeted at past exam papers. Question ONE looked for breadth in the subject area of software measurement, but many candidates only exhibited some knowledge and awareness in a few specific topics. In contrast, question TWO sought depth in the specific area of OSSE, and many exhibited that depth of knowledge and awareness. Question THREE sought to determine students understanding design patterns and their application in facilitating reuse as well as their understanding of how claims regarding design patterns could be established empirically; many students lacked even a basic understanding of design patterns and few were able to provide a satisfactory answer. Question FOUR addressed to two topics: principles of user interface design and reliability metrics; very few students were familiar with the later. The final Question FIVE aimed to assess students understanding of software process improvement and developments such as the CMMI and the role that metrics play evaluating improvements; some excellent answers were given but many answer failed to show any depth of understanding. The following issues (identified in previous examiner reports) need to be addressed to improve the performance of candidates: 1. Coverage of the syllabus. Centres should encourage candidates to adopt a staged approached to their learning, by completing the Software Engineering 1 module, before attempting Software Engineering 2; 2. Subject awareness. A successful learner should provide more breadth in responses given to all parts of the question by reading more widely publications within the profession as well as recommended text; 3. Examination techniques. Some candidates answered too many questions resulting in very shallow answers ; or too few, which resulted in too detailed an answer but insufficient to compensate for the marks lost by not completing a third question; 4. Answers should be legible, well structured and formatted. It is important that candidate responses to questions are: comprehensible, exhibit breadth and depth in knowledge and application to the subject matter and have relevance and appropriateness to the questions and rubric of the paper itself.

2 Question 1 a) Discuss how the following concepts differ from one another, and give examples of how each might be measured: (i) (ii) (iii) software process and product metrics; software quality characteristics and sub characteristics; and coupling, cohesion and complexity metrics. A good answer should recognize that: (i) Process metrics are concerned with quantifying the characteristics of the software production process in an attempt to understand and so improve software development and maintenance. Example, Defect Removal Efficiency, defined as the number of defects found divided by the number of defects hidden in the product; Product metrics are concerned with quantifying the characteristics of the product itself, such as size, complexity, and performance. Example, the size of the product can be calculated on the basis of a count of its total number of function points or lines of code. (ii) A software quality characteristic represents the behaviour of the system. Examples of characteristics could include such things as usability, portability, and efficiency where usability according to the ISO9126 standards represents the ease of use of a given function. The measurement of usability would need to combine metrics for software sub characteristics of understandability, learnability, and operability. Sub characteristics are attributes of software development that effect behavioural characteristic. For example, sub-characteristics such as understandability, learnability, and operability effect the usability characteristic. The learnability sub characteristics metric value could be achieved through Case study design and experimentation. (iii) Coupling is concerned with the degree of external interaction exhibited by a software component such as function. In this case the number of input and output parameters, global variables, and modules called will determine its value; Cohesion is the degree of interdependency between the internal processing elements and the data objects of the component. Thus, the flow graph connectivity between the processing elements of the software object, can determine the level of cohesion. Complexity metrics are mainly concerned with program flow. These metrics can provide insights into such things as the degree of difficulty to test or maintain a software component. The CCM is a simple count of decision paths through a program the higher the value the greater the degree of complexity. In relation to coupling and cohesion, a component with low cohesion and high coupling values might be considered complex and notoriously difficult to test. (16 marks)

3 b) Discuss whether there exists an optimal size for projects (measured using function or object points), after which the likelihood of budget overrun, delay, and cancellation are very high. The answer should give a brief overview of the metric, highlighting its emphasis on weighting and counting the number of functions (input, process, and output) or objects, the result of which provides a useful indicator of the development effort required. Discuss published research (e.g. Patterns of Software Systems Failure and Success, by Capers Jones) highlighting project size (measured in function points) with success rate (in terms of delivery early, on time, or late), should identify the failings of large scale projects (with thousands of function points) compared to smaller projects (a few hundred function points. The answer should conclude by suggesting that projects grow beyond a certain size (say 100 FPs) are likely to experience exponential growth and complexity in such things as the communication between and interaction with project resources. Thus it is recommended that large scale projects be divided into increments considered to be ideal for delivery on time and within budget. (9 marks) Examiner s Guidance Notes: This question assesses a candidate s knowledge and awareness of differences in software measurement concepts and processes. A variety of concepts was presented and over 50% of candidates attempted to answer the question. Disappointingly, many candidates demonstrated depth of knowledge in one or two concepts such as coupling and cohesion, but limited knowledge of others (such as software complexity). Function points represented the second part of the question, and the maximum and minimum marks for the question as a whole are indicative of the lack of awareness of published material in this area.

4 Question 2 a) Write a report that explains the process of open source software engineering and presents a general discussion of the progress made through its practice within industry. Your answer should cover methods, tools, resource availability, and development successes. A good answer should produce: A structured report with appropriate headings, including introduction and conclusion; Introduction and Overview that defines key concepts, gives an overview of OSSE, and highlight the progress made in the adoption of OSS commercially. For example, the development life cycle, its emphasis on incremental and early release of product; publicly (freely) available source code building blocks; project teams composed of large numbers of independent, self-organised developers and testers; and the widespread use of products for development environments, office and web technologies, Body of report that discusses: teams are self-selecting, self-organising based on skill, availability, and commitment. Further, early release of software and regular peer review via the community of users and developers alike Examples of methods agile methods (such as Extreme Programming) are the most applicable given the iterative and incremental nature of development; a varied range of design and development tools of which the Eclipse Integrated Development Environment is an example. A large pool of easily accessible human and software resources exist, which is an important key to OSSE success and failure. Some of the successes might include development environments (Eclipse), version control (CVS), browser (Mozilla/Firefox), and office (Open Office) software. Less popular software may suffer from fragmented and erratic development, maintenance, and support during the lifetime of the project. 7 marks Conclusion that draws attention to progress, but some issues of the software crises that still remains - in particular, the application backlog and project management. (17 marks) b) Discuss briefly the view that Open Source Software has decreased the productivity of developers and the quality of the systems produced. A good answer might consider productivity within the context of developers ability to produce code more quickly and this can be effected by creation and availability of extensive libraries of reusable code, and greater use of productivity tools such as code generators. It is widely recognised that Open Source has been a major contributor to this resource pool. Likewise, the qualities of such components have been assured by public scrutiny, extensive testing and feedback. However, it can be argued that the OSS may have reached its critical mass, the effect of which is that productivity is hindered by slow and difficult searches for the

5 right components. Further, the pool of resources to scrutinise publications are directed more towards those products with large user groups or commercial success. Therefore, the majority of OSS are poorly documented and poorly maintained. Examiner s Guidance Notes: (8 marks) This question proved to be the most popular amongst candidates and had the highest pass rate (53%) of all the questions. It was clear from the answers submitted, that many candidates in their answer to part a) had good knowledge, awareness, and experience of open source development. It is of concern, that there were a significant minority of candidates that had no concept of the development process itself. In contrast to the relatively good (and sometimes over-zealous writing) performance in part a), many candidates found part b) difficult in terms of structuring a meaningful discussion of the viewpoint expressed. Question 3 Erich Gamma, one of the popularisers of Design Patterns, has claimed that reusing designs through patterns yields faster and better maintenance. a) Explain the concept of a Design Pattern in Software Engineering. (5 marks) b) Explain how designs can be reused through patterns in the context of software development. (10 Marks) c) Discuss reasons why Gamma s claim could be true and how this claim could be empirically established. (10 marks) A good answer to part a) is as follows: A design pattern in Software Engineering is a way of encapsulating a well established solution to a range of common recurring problems in the design of systems in such a way that it can be reused by other designers when they meet those problems. Typically the design pattern description covers the following points: Pattern Name and Classification: A descriptive and unique name that helps in identifying and referring to the pattern. Intent: A description of the goal behind the pattern and the reason for using it. Also Known As: Other names for the pattern. Motivation (Forces): A scenario consisting of a problem and a context in which this pattern can be used. Applicability: Situations in which this pattern is usable; the context for the pattern. Structure: A graphical representation of the pattern. Class diagrams and Interaction diagrams may be used for this purpose. Participants: A listing of the classes and objects used in the pattern and their roles in the design.

6 Collaboration: A description of how classes and objects used in the pattern interact with each other. Consequences: A description of the results, side effects, and trade offs caused by using the pattern. Implementation: A description of an implementation of the pattern; the solution part of the pattern. Sample Code: An illustration of how the pattern can be used in a programming language. Known Uses: Examples of real usages of the pattern. Related Patterns: Other patterns that have some relationship with the pattern; discussion of the differences between the pattern and similar patterns. (These are covered in the book, Design Patterns, by Gamma et al. and a good answer should mention a few of these or name some example design patterns.) A good answer to part b) should cover the following points: 1. If designers are familiar with a range of design patterns and understand specifically the sections in the pattern labelled Motivation (Forces), Applicability and Consequences, then in when they are developing a software design where the problem addressed by the pattern occurs or situations where the pattern is applicable and has been used in the past, they can evaluate its tradeoffs, results and decide whether or not to reuse it. 2. Having a large, well documented collection of reusable design patterns provides a repository of designer experience and if designers consult such a repository during development, they can potentially avoid re-inventing known solutions and saving time and effort. 3. During design of a large system, individual designers must communicate their design solutions to one another. The use of reusable design patterns makes this task easier. 4. Using a standard way of describing designs such as that employed in the design pattern literature, means that their reuse is made easier. 5. New design patterns when they are developed will be more easily reused if they are described in a standard way and added to existing repositories. 10 Marks A good answer to part c) should address some of the following points: 1. One reason why reusing design patterns makes maintenance easier is that design patterns can improve program comprehension and program comprehension is a pre-requisite for effective software maintenance. 2. Maintenance of software is often carried out by software engineers who were not involved in the original development. If reusable design patterns that are well known throughout the software industry are used, then maintainers can recognize them by name and understand the system they are maintaining more effectively. 3. Programmer productivity but also potentially program quality is improved through the use of patterns. 4. Communication amongst designers and maintainers is improved through the use of the common design patterns in code.

7 5. Maintainers may be less skilled than the original software designers and so may improve their programming skills when maintaining code developed with design patterns. This claim could be empirically established through a controlled experiment involving two groups. One group is asked to make a change to program with highlighted and commented design patterns, while the control group is set the same task with any highlights and comments. The time they take to make the change and the correctness of their changes could then be determined and compared. Evidence for the claim could also come from data collected in an organization pre and post the introduction of design patterns if data is available regarding time taken to make changes and regarding successful changes in both periods. Examiner s Guidance Notes: If students lacked a clear understanding of the concept of design patterns, then this question proved difficult for them to answer. Although some were able to provide a generic account of the benefits of software reuse, this was not what was required for part b of the question. Few students were able to demonstrate any understanding of empirical software engineering and its potential to establish a claim such as that made by Gamma. Nevertheless, there were a few students who gave excellent answers. Question 4 a) Discuss the relevance of the three principles of User Interface design known as Minimal Surprise, Recovery, and Consistency and give an example of how each would be applied in the design of a simple word processor. (15 marks) b) Outline appropriate reliability metrics for two systems: (1) a simple word processor and (2) a medical monitoring system to be used with critically ill patients. Give reasons for your choice of metrics based on predicted usage of each system. (10 marks) A good answer to part a) should cover the following points: 1. Minimal Surprise is the principle that the user should not be surprised by the behaviour of a system. Any surprises could be kept to a minimum if they cannot be avoided totally. People become annoyed when a system does not behave as they expect. They usually develop a mental model of how the system works and expect that comparable actions on their part will have comparable effects. If something different from they are expecting happens, then they are likely to be confused as well as surprised. In a simple word processor, cut and pasting text from one part of the document to another, should not have the surprising effect of changing the format of the document.

8 2. Recovery is an important principle because when users are working with a system, they will inevitably make mistakes and so it is very important to provide some means if possible that allows the user to recover from their mistake. There are various tactics that can be employed, such as always getting the user s confirmation before a destructive action is carried out, implementing an undo function that returns the system to its previous state before the mistake occurred, and checkpoint and saving the state of a system so it can be restarted from the checkpoint. In a simple word processor, recovery could be achieved by always allowing the user to undo their last action which changed the text. 3. Consistency is the principle that system commands and menus etc all have the same look and feel, e.g. use colour in the same way, have the same format, obtain parameters in a uniform way. It is important because it reduces the time a user needs to take in learning the system and where consistency is applied across all applications on an operating system, then users can learn even more quickly, e.g. the Windows look and feel. In a simple Word processor, consistency could be achieved by always using the same dialogue box to obtain a named file from the user. A good answer to part b) should address the following points: 1) An appropriate reliability metric for a simple word processor would be Mean Time to Failure (MTTF). In a simple word processor, people carry out long transaction preparing a document and use the system for long periods. Ideally the MTTF should be greater than the average time spent by typical users preparing a document. If it is less, then they are likely to lose all their work on their document through the failure. In the case of long transactions, to ensure correct delivery of functions, Rate of Occurrence of Failures (ROCOF) may also be relevant. 2) An appropriate reliability metrics for medical monitoring system would be Rate of Occurrence of Failures (ROCOF) and Probability of Failure on Demand (POFOD). The monitoring system may have regular demands made on it if say heart rate is being monitored and must be measured regularly and so it must do this correctly and have a low ROCOF. It may also unpredictably have to deal with unpredictable service demands when an emergency condition arises in the patient and here POFOD is relevant. Examiner s Guidance Notes: There were almost no satisfactory answers to this question taking both parts together. Many students appeared to answer the first part based simply on their understanding the English terms naming each principle although some showed familiarity with the basic principles. Many were under the mistaken impression that recovery was concerned with recovery from system failures. Almost no students were able to demonstrate any knowledge of reliability metrics and therefore the selection of appropriate metrics was beyond them.

9 Question 5 a) Explain what is meant by software process improvement and discuss the role that a metrics programme plays in software process improvement. (10 marks) b) Describe two types of software metric and relate each to the measuring specific process improvements. In your answer, you should give specific examples of each type of metric. (10 marks) c) Discuss the advantages and disadvantages of taking an approach to process assessment and improvement based on a staged approach such as the CMMI staged model. (5 marks) A good answer to part a) should address the following points: Software process Improvement focuses on identifying potential improvements to the software engineering practices in a software producing organization. It recognizing that the quality of the company s software processes, i.e. its software development practices, affect the quality of the software products it produces; so when a problem or fault is identified in a product, its cause needs to traced back to the processes that went wrong in its production, e.g. inadequate testing, or the staff who failed to carry out the process correctly. In that latter case, it may mean that the company s training needs to be reviewed. In the former, where the process is at fault, measures need to taken to improve that process so that the needed product quality is achieved in future. Usually process improvement is carried out in a programme of staged improvements and is based on an assessment of the company s current level using the 5 level of the Capability Maturity model: Level Focus Key Process Areas 5. Optimising Continuous process improvement Defect prevention, Technology innovation, process change management 4. Managed Product and process quality Process measurement and analysis Quality management 3. Defined Engineering process Organisation process focus: OP definition Peer reviews, Training program, Intergroup coordination, Software product engineering, Integrated software management 2.Repeatable Project management Software project planning and tracking, S/w subcontract mng, SQA, SCM, Req Mgt. 1. Initial Heros

10 Metrics are important in process improvement especially from the 4 th level onwards. A metrics programme comprises a defined set of metrics, with a rationale and strategy for collecting these metrics within the company and audit and regular review of metrics data to guide the workings of the organization. Metrics are particularly important when implementing managed process change. However, it is important that software product metrics and process metrics are recorded through an established metrics programme prior to and following any improvement so that its success and impact can be determined. A good answer to part b) should address the following points: Two main types of metrics are software process metrics relating to measures associated with specific software processes, e.g. average time taken by maintenance team to correct a reported software fault, and software product metrics, measures associated with the product itself, e.g. software complexity measured using McCabe s Cyclomatic Complexity measure based on a control flow representation of a program. For example, if an organization wants to improve its testing process, and previously has been recording the errors uncovered by testing, then if they continue to do this and introduce a new testing tool and process, they ll be able to gauge if the improvements have resulted in a lower error rate. They might also investigate whether the complexity of the software is related to the error rates found in testing and initial a process improvement to reduce software complexity. For this, they would need to ensure suitable complexity metrics were identified and applied before and after the improvement. 10 marks A good answer to part c) should address the following points: Advantages of taking an approach to process assessment and improvement based on a staged approach such as the CMMI staged model: The CMMI like the CMM provides a clear pathway for software producing organizations; it can be implemented in stages; goals at each level can be tackled in turn; resources can be focused on the improvements giving the most needed benefits. Disadvantages of taking an approach to process assessment and improvement based on a staged approach such as the CMMI staged model: The staged model may mean that companies do not get needed benefits of introducing a goals and practices from a higher level before trying to achieve goals and practices at the lower-level, for example, focusing on technology innovation which is level 5 give immediate benefits even to companies who have not yet achieved level 3 as many level 2 practices can benefit from technological support; where companies implement improvements out of order then assessing them using the CMM/CMMI may given a false picture of their maturity. Examiner s Guidance Notes: This topic which has been well covered in recent Software Engineering literature proved difficult for many students especially the role that metrics play in software process improvement appeared beyond the grasp of students. Their understanding of a staged approach such as the CMMI was lacking except in a few cases.