Medical Informatics Testing. Framework (MIT-F)

Transcription

1 Joint Medical Service Norwegian Armed Forces Military Medical Research and development Medical Informatics Testing Framework (MIT-F) 1 Executive Summary In June 2002 USA (Department of Defence) and Norway (Ministry of Defence) signed a Memorandum of Understanding (MoU) agreement in the area of telemedicine. The objectives were to share experience/lessons learned and together do some research in new concepts/architectures that involves the use of the US PIC 1. Since June 2002 the MOU project has developed a prototype system called Evacuation Support System (EvacSys). This system uses the PIC as an "electronic field medical card" and addresses the problems of capturing, storing and distributing medical treatment and patient tracking information during the evacuation of a wounded soldier from field. In order to have an efficient iterative development process and to document the benefits and improvements of EvacSys, a framework for testing and evaluation has been developed namely the MIT-F. The MIT-F defines: Why to test (rationale) Figure 1 Field medics performing and documenting treatment. What tests can be performed (test types) What documentation can be produced (test specification) How to test (methodology) 1 Personal Information Carrier:

2 2 of 11 We believe the framework can easily be applied to other domains and systems that face similar operational challenges. This whitepaper introduce the framework and how it can be applied. 2 Why use MIT-F? 2.1 Testing The purpose of testing is to unveil and remove as many deviations as possible in the system within a given time- and resource-frame, and to ensure that the system complies with the test s pass criteria. A test should also verify that the system complies with its requirement specification. But testing should also find out if the system is suitable for its purpose. "Suitable" is tightly connected to usability, thus testing should also evaluate and contribute to improve the overall usability of the system. Glen Myers [3] states: 1. Testing is a process of executing a program with the intent of finding an error 2. A good test case is one that has high probability of finding an as-yet undiscovered error. 3. A successful test is one that uncovers an as-yet undiscovered error Testing can only find errors; it cannot prove the absence of errors. Consequently, exhaustive testing is impossible, and a testing never ends- it is being stopped when deemed sufficient. 2.2 Verification, Validation and Usability When planning system testing, one should make a distinction between system verification and system validation. According to the IEEE Standard Glossary of Software Engineering Terminology 2, verification is defined as "The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase." Validation, on the other hand, is defined as "The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements." Verification checks that the system conforms to the specification (System Requirements), while validation checks that the system conforms to its intended use. It may be the case that the system is verified (all system requirements are fulfilled), but the system does not validate (does not conform to intended use). In this case the system requirements must be changed / updated. In summary: Verification: is the system correct? Validation: is it the correct system? MIT-F addresses all aspects of software testing (see section 3.1), but the primary focus is on validation and usability. This is because bad usability can kill patients [1]. We argue that in systems with safety concerns (i.e. systems that have implications on people s safety), high usability is key. Usability can be defined as [2]: A measure of the ease of use with which a system can be learned or used, its safety, effectiveness and efficiency, and the attitude of its users towards it. 2

3 3 of 11 The only feasible way to ensure that you achieve a sufficient degree of usability, is to validate the system using relevant end users in a realistic operational environment that capture the aspects of unfriendly environmental conditions, stressful work situations and unexpected events that are likely phenomena in the environment. This is also referred to as Site Acceptance Testing (SAT) and User Acceptance Testing (UAT) Figure 2: The relation between verification, validation and usability Figure 2 shows the relations between usability, verification and validation in a system perspective. The system has an "Intended use" that is highly dependent on good "Usability". These usability requirements are specified in the "System Requirements", but can only be tested during "Validation". MIT-F is a testing framework designed to be able to implement validation using efficient acceptance tests where real users use the system is real environments. 3 Framework Description 3.1 Tests Categories Four kinds of tests are being performed in MIT-F: Internal test Module/component testing Conducted in parallell with development Stepwise integrationtest Performed in internal test environment FAT Test of complete solution with solution provider Performed in suppliers environment SAT Test of complete solution in customer's environment Ensure that no errors are introduced during shipment and installation UAT Testing involving relevant end users in realistic environments

4 4 of 11 Internal test, including developer tests, unit tests, and internal integration tests Factory Acceptance Tests (FAT), testing the system with the solution provider Site Acceptance Test (SAT), testing the system functionality at the customer User Acceptance Tests (UAT), testing with relevant end users in realistic environment 3.2 Methodology Scenario based Acceptance Testing In MIT-F we suggest the use of Scenario-based testing (SBT) [4]. SBT concentrates on what the user does, and not what the product does. It means capturing representative tasks (via use cases and user stories) that the user will perform, and then applying them and their variants to as tests. Scenarios uncover interaction errors. To accomplish this, test scenarios must be realistic. Scenario based testing exercise multiple sub-systems in a single test (users do not limit themselves to one sub system at a time). MIT-F scenarios contain the following: Unique identifier Short description of scenario Precondition for the scenario What is documented using video What is documented as written log data What is documented using questionnaire, what questionnaire is used Variations of scenario Roles and responsibilities of evaluators Roles and responsibilities of testers Post conditions Acceptance Testing Evaluation Techniques This section describes the techniques we suggest to evaluate the tests we perform. The core task is to collect as much data as possible, to best answer the questions that are the rationale to perform the tests. We will use two complementary techniques: Observing usage and Collecting users opinions. Pretesting Questionnaires and Interview Guidelines Questionnaires and interview guidelines needs to be tested. This is done to ensure that they are fully understood by the target group, and to remove ambiguities and possible misunderstandings. Training When dealing with systems that require training, one need to train the users in the system before testing commences. This is done to reduce the amount of feedback on the system that originates from lack of training. Comparative Studies A computer system is introduced as a solution to a problem. The problem can be another computer system or not any system at all. In order to evaluate whether or not the new computer system, is solving the problem, we need to compare the old approach to the new. This involves designing two tests, one for the old approach, and another for the new approach involving the new system. Then we are able to compare (e.g. comparing time used to solve a task, user

5 5 of 11 satisfaction etc.) the two approaches and determine whether or not the new system constitutes an improvement. Direct Observation by Logging At least one evaluator is always observing the activities of the test. The evaluator is equipped with a schema for logging significant events. The schema includes a detailed specification of what is to be observed. Users are encouraged to think aloud. Even though this is considered awkward for most users, it provides valuable input for the evaluation. If it is impossible to achieve this, it is important for the observer to log as much incidents as possible, so one can have the user explain his thoughts in the interview. Indirect Observation using Video A human evaluator will have problems registering and logging everything that happens in a test scenario. It is therefore beneficial to supplement the logging using video recordings. There should always be one evaluator observing and one filming. This is because the person filming will not be able to observe, and vice versa. Analysing video is a very time consuming task, and what to be filmed must be carefully specified. It is recommended that video is used only for documentation, and not primary for the analysis. Collecting User Opinions In addition to observation, we recommend to collect data through the use of interviews and questionnaires. For the interview we are interchangeably using structured and unstructured interviews. Unstructured interviews will typically be used in the first tests to collect user s opinions about the system. In later tests one will have more concrete questions that need answering, and one can use structured interviews. Interviews are specified for each test specification. Questionnaires are used to collect a user s opinions after having performed test scenarios. Questionnaires will be specified for each test specification. Analysing Collected Data The analysis phase includes registering results with statistical software (such as SPSS 3 ). One need to group questions and run factor analysis. In order to obtain statistically significant results it is important to have a sufficient amount of test runs and testers. Other data sources such as observation logs, video recordings and interviews should be analysed in order to prove or disprove findings from statistical analysis. Findings must be unambiguous and must be documented carefully. All sources of error should be commented and investigated. 3.3 Test Process Testing conducted with MIT-F consists of five main activities 4 : The documents produced comply with IEEE Std , Standard for Software Test Documentation.

6 6 of 11 Start Project doc Develop Test Plan Test plan Preparing test environment Test Specification Test design spec Test case spec Perform test Test procedure specification Test data Test log Overall Test Report Test incident report Stop Test Reports Figure 3: Activities and documents in MIT-F 1. Planning is the process of developing the test plan, approve it, and distribute it to target group. Project documents such as requirements specifications, architecture, and design specifications are input to the plan. The test plan prescribes scope, approach, resources and schedule of testing activities. Focus is on identifying the test items, and test items should include reference to other documentation (e.g. requirement specification) for the item, if such documentation exists. 2. Specification is the process of specifying a test design, test cases and test procedures. The specifications should cover both functional and non-functional features of the system as described in the system design. The test design identifies features to be tested by the design, and its associated tests. The test design identifies test case specifications, which in turn identifies test procedure specifications. Test procedure specifications are the steps involved in executing a set of test cases. 3. Test environment and test data- addresses the task of establishing a test environment that is separated from the development environment and simulates/is the environment in which the system will be put into operation.

7 7 of Testing- is the task of testing the system based on test design specifications, test cases, and test procedure specifications. The test log is a record of relevant details about the execution of a test. The test incident report documents any event that occurs during the testing procedure that requires investigation. 5. Reporting and approval- involves developing the test reports and making sure the test is approved by proper authorities. The test report summarizes the results of the testing activities and provides evaluations based on these results. 4 Case Study: Medical Battalion during Exercise "Interaction" The evaluation framework described above was first applied on EvacSys during the military exercise Interaction in northern Norway in December This was a 5-day realistic exercise that involved both land and air units, and the EvacSys system was tested by the medical battalion in the 6th Division in the Norwegian Army. Several complete evacuation run-throughs were conducted and each step was evaluated using the framework. 4.1 Test preparation The following activities were carried out as a preparation to the test in real environments: We did a pre-test on a few of the participant to ensure that terminology and language of questions were easily understandable. We made a scenario overview that identified what key factors the observers should focus on. We gathered context information about all the participating users by using questionnaires. It is important to be aware of the technological competence when introducing new technology. We gave a brief introduction to the system and made sure that everyone had the opportunity to learn about the system and their interface to it 4.2 Testing in real environments The following activities were carried out during the exercise: A dynamic observer followed the patient throughout the evacuation chain (Battlefield, 1 st Transport, Level 1 Mobile Aid Post, 2 nd Transport, and Level 2 Light Field Hospital). Two observers were constantly located at each of the evacuation positions (static observers). Observers used log-schemas to log the activities on each of the locations before, during and after an incident occurs. Free notations were also documented in addition to the logschemas. Video recordings and photographs were used to document how specific actions were performed. The participants/medics were instructed to think aloud when performing their tasks. Evacuation run-troughs without the introduced technology and in dual-mode (new and old at the same time) were conducted for comparison studies. All participants were interviewed after each evaluation run-through. The interviews were recorded and typically took about 10 minutes. All participants filled out questionnaires regarding usability after each run-through and at the end of each day. The questions were based on previously validated scales from the

8 8 of 11 Technology Acceptance Model (TAM) [5] and The Perceived Characteristics of Innovating (PCI) [6] Figure 4: An example observer configuration. 4.3 After the Test After the exercise the gathered data were properly analysed. This resulted in new usability knowledge and evidences that can be used for improvement and comparison to other systems (or pre-system state). The analysis included the following activities: Debriefing meeting between observers and central participants. Transcription of interviews. Transcription and analysis of observations from log-schemas, notes and video. Analysis of information registered with the new technology and comparison with information gathered without it. This is typically amount of information, relevance and correctness. Registering results with statistical software (such as SPSS). Grouping questions and running factor analysis. Writing reports presenting the most important discoveries, statistics, and pictures, and that discusses further work. It is also advisable to make a short movie from the video material that can be used as a visual project presentation. In order to obtain statistically significant results it is important to have enough run-throughs and participants/medics. A medic should participate in more than one run-through and identifiable in order to observe improvements as the new technology becomes more familiar. Interviews and questionnaires should be applied immediately after a participant has completed his or her tasks.

9 9 of 11 5 Evaluation Framework Benefits By applying this evaluation framework one is able to identify and evaluate factors that would be difficult to observe in other ways. With real users in a realistic environment unexpected events are bound to happen, and new improvements and ways of use not originally thought of can be implemented. Working under pressure and in difficult conditions, a few creative users may come up with brilliant new ideas, and larger numbers of users usually indicate which way to go. The evaluation framework considers both these qualitative and quantitative aspects. The framework is also designed for full scale, complete run-throughs, and not just on isolated events. This also contributes to the relevance of the results.

10 10 of 11 6 Target Market Even thought the framework was designed for a specific project is can easily be applied to similar environments, not only related to military and healthcare. If you want to introduce new technology into an environment that conforms to some of the following concerns you should consider taking advantage of the EvacSys evaluation framework: You want to introduce and evaluate new technology that must be tightly integrated with existing work procedures and/or adds new functionality. Unpredictable incidents may occur and the system/technology must adapt. The work is carried out in a sometimes hectic and stressful environment and the weather conditions may be unfriendly. The users follow predefined work procedures. The new system is just a tool for accomplishing their main responsibilities, thus education and training in using the new system might be limited. User collaboration and communication is vital, and concurrent activities occur at several physical locations. You need to secure high usability and efficiency. You need high-quality evaluation documentation. You need to minimize the risk of failure and maximize the chance for smooth integration immediate return of effort. 7 References 1. Nielsen, J. Medical Usability: How to kill Patients through Bad Design. Jakob Nielsen s Alertbox. 2. Preece J. et.al. (1999) Human Computer interaction, Addison-Wesley. 3. Myers, G. (1979), The art of Software Testing, Wiley, 4. Pressman, R., S. (1997), Software Engineering, A Practitioner s Approach, McGraw- Hill. 5. Davis, F.D. (1989), Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly, Vol. 13, pp Moore, G. C. and I. Benbasat (1991) Development of an instrument to measure the perceptions of adopting an information technology innovation, Information Systems Research Vol 2 No 3 pp

11 11 of 11 8 Contact us For all questions regarding the evaluation framework (or the EvacSys system and concept) please contact: John Ivar Brevik MD, Chief Physician Norwegian Joint Medical Service joivar@online.no Office phone: Cell phone: Marius Mikalsen Research Scientist SINTEF ICT Ståle Walderhaug Research Scientist Norwegian Joint Medical Service swalderhaug@mil.no stale.walderhaug@sintef.no Office phone: Cell phone: Fax: Per H Meland Research Scientist SINTEF ICT Marius.mikalsen@sintef.no Office phone: Cell phone: Fax: Per.H.Meland@sintef.no Office phone: Cell phone: Fax: