Intelligent Assistant Systems: Support for Integrated Human-Machine Systems

Transcription

1 Knowledge Systems Laboratory February 1990 Technical Report KSL Intelligent Assistant Systems: Support for Integrated Human-Machine Systems by Guy Boy and Thomas R. Gruber Appeared in the proceedings of 1990 AAAI Spring Symposium on Knowledge-Based Human-Computer Communication, March 1990, Stanford University. KNOWLEDGE SYSTEMS LABORATORY Computer Science Department Stanford University Stanford, California 94305

2 Intelligent Assistant Systems: Support for Integrated Human-Machine Systems Guy Boy 1 NASA Ames Research Center Artificial Intelligence Research Branch Mail Stop , Moffett Field, CA boy@pluto.arc.nasa.gov Thomas Gruber 2 Stanford University Computer Science Department 701 Welch Road, Palo Alto, CA gruber@sumex-aim.stanford.edu 1. Introduction The increasing automation of engineered systems has been accompanied by increased complexity of the interactions between human users and the machines. The well-known dangers of human error in complex systems such as power plants, aircraft, and space systems present a challenge to the design of these machines and the procedures for operating them. With the increased automation of intelligent systems, one might expect that humans play less important roles in the function of the system. In reality, humans need to stay in the loop as automated systems take on more intelligent tasks. Machines can be built to perform many of the routine tasks allocated to humans today. However, since it is difficult to design machines that can handle unexpected situations, humans will remain in the control loop to take care of such situations. The current practice of design engineering is often machine-centered: optimized for the tasks that can be performed by the machine itself, neglecting to support the complementary roles of the human in the loop. This paper takes an alternate view of design: that the designed artifact is an integrated human-machine system (IHMS). In this view, the human operator is a functional component of an intelligent system, contributing to the overall performance of the system. Performance often includes intelligent activity, where the human and machine share responsibilities and perform complementary tasks. A class of programs that support the human in the loop called intelligent assistant systems (IAS) mediate interactions with the machine and perform some of the intelligent functions required for the system. In this paper we present an analysis of the properties of IHMSs and how they differ from machine-centered systems. These properties motivate three important design problems for the design of IASs: how to adapt to changing operator knowledge and skills, how to distribute intelligent functions, and how to share autonomy between human and machine agents. We describe the roles of IASs in the context of these problems, and lay out some of the design objectives to consider when building them. We illustrate the ideas with the example of an IAS for cooperative fault diagnosis in the space shuttle. Experience with this application suggests areas for future research and development. 2. The Problem of Machine-centered Design Traditional design is optimized for machine-centered automation. Operations procedures, for example, typically prescribe what the user needs to do with the controls of the machine to perform the system s intended task, rather than how the operator can achieve his or her goals using the resources made available by the machine. A design that neglects to account for the way a human operator is integrated into the system is not optimal for the overall system task. For example, if operators serve only backup roles for a largely automated system, then they may fail at this subtask, leading to system failure. There is evidence that humans operators are very bad at monitoring when they are not cognitively involved and focused on the overall system task. For instance, recent accidents in aerospace and nuclear power plants have dramatically pointed out that excessive automation tends to lower the level of vigilance of human operators. Conversely, it has been shown that when human operators participate in the control of a system, they are also very good monitors (Nagel 1986). Such results indicate that the success of the overall system depends on the design of the roles of the operators and the human factors that influence operator performance. Designers use implicit and explicit models of a device as they design it. If humans are part of the device, their actions need to be modeled as well. In standard engineering practice, however, interactions between machines and operators are generally not modeled, or are modeled in the same way as physical components designed to perform a given function. For example, in models of operating systems for mainframe computers, the human operators are sometimes modeled as tape loaders; their only relevance to the system is the manual service they provide. The machine-centered bias is further reflected in the fact that the behavior of end-users is rarely observed or simulated at design time. We advocate the view that the system to be designed should be modeled as an integrated human-machine system (IHMS): that the structure, behavior, and intended

3 function of the system be modeled in terms of both the machine and the human operator in the context of use. In other words, since some of the functional components of the system are people, engineering models should predict the ways in which these parts interact with other parts to achieve overall performance. Obviously, human behavior cannot be predicted or controlled with the precision of engineered devices. Since people can't literally be modeled or designed as device components, the IHMS view requires a modification of the traditional paradigm for modeling and design. The shift in modeling methodology is to focus on interactions among agents in the system with differing functions and goals, rather than modeling the behavior of all components in a system-driven way. The shift in design methodology is to use information at design time about the behavior of the human in the loop, and to devise new components for the IHMS to help mediate humanmachine interactions (e.g., intelligent assistant systems). 3. Properties of Integrated Human-Machine Systems Several properties of integrated human-machine systems follow from the fact that people and machines have different capabilities. When aware of the current situation, humans can solve unanticipated problems. By definition, solutions for unanticipated problems cannot be given machines at design time. People also learn from experience. They appear to build chunks of knowledge which are easily retrieved and adapted in analogous problems (Anderson 1983). However, they can fail on routine tasks. Humans can be distracted, overloaded, and their vigilance can drop dramatically if the nature of the job is boring for them. They are poor at numerical calculations or reasoning in a short period of time, unless they have compiled tricks to do the corresponding job. Conversely, machines can be made very reliable for routine tasks requiring numeric computation and rapid response-time. A design goal for IHMSs is to provide mutually complementary roles for humans and artificial components of the system. Some tasks can only be performed by machine, and some by the human. With improved AI techniques, some intelligent tasks may be performed by either human or automated by a program. However, maximum automation is not necessarily desirable or achievable. The human operator may be responsible for overriding the normally automated functions in certain situations. And, as stated above, leaving only backup functions to humans can lead to failure due to lack of attentiveness. On the other hand, it is possible for humans cause failures by moving a device out of its intended operating range. Thus to achieve robustness in overall system performance, the human role should facilitate recovery from unexpected error conditions while not compromising the safe operation of the machine. A final property of IHMSs is that they can contain IASs to mediate between human and mechanical constituents of systems and to perform some of the intelligent functions of the system that can be shared with human operators. Section 4 will describe an IAS and show how its development took into account the properties of human-machine systems. Figure 1 illustrates the structure of an IHMS, and the corresponding models. User Machine Environment Integrated Human- Machine System Intelligent Assistant System User Model Device Model IAS Prototype Model of Environment Models of IHMS Figure 1. The structure of an IHMS, and the corresponding models 4. Design Parameters for IHMSs The special properties of having humans in an integrated system leads to a different design methodology. First, it suggests that at design time we need to find out more about what to expect from human operators in the context of using the designed device. This suggests a move from the traditional sequence of specification-develop-testproduce, to a methodology that brings quick, repeated feedback to the designer about the utility of design decisions. Rapid prototyping is such a methodology. Rapid prototyping of IHMSs involves observing test subjects operating in prototype system environments, simulated or real. This is difficult in practice, because the construction of realistic environments is itself as designto-production process. Imagine, for example, using a cockpit simulator to design cockpits! New techniques for prototyping simulation environments will be required. Based on general information and knowledge about human-machine interaction, the designer can anticipate some properties of integrated systems. In this section we identify three parameters to consider in the design of IHMSs. They are: how to adapt to changing operator expertise, how to distribute intelligent functions among human and machine agents, and how to share autonomy in the control of the system. 4.1 Adaptation to changing operator expertise People learn when in interesting environments, and their behavior changes accordingly. In IHMSs, human

4 operators adapt to the contingencies offered by the environment, include the controls, information displays, etc. They get better at sensory motor tasks, and develop strategies that improve performance on cognitive tasks. How can a system be designed to adapt to the changing expertise of the user? One approach is to try to track the user s knowledge or behavior with an adaptive user model. User modeling is still a problem for research. No satisfactory general theories exist, and there is little reason to believe that a general theory would ever be found (in light of the history of psychology). In practice, building a user model requires data about real users in authentic or realistic simulations of the environment in which they are users. An adaptive model can be induced, if at all, from data on users who have gone through stages of using the system and learned to use it. The fact that people s interactions with machines can change with use has implications for the design of operations procedures and training methods. If the designer of the operations procedures assumes novice users, then the procedures are that are designed will be non-optimal as users adapt. Similarly, training methods, which are also designed (often at the same time as the device), will need data on how users are expected to adapt to the system. 4.2 Distributing intelligent function In intelligent systems of the future, artificial agents and human operators will both perform intelligent tasks in the context of an integrated system. Each has responsibilities, requires access to resources, and has particular knowledge appropriate to tasks. Some tasks may be done in parallel, others may require results from other tasks performed by agents. The balance of sharing in intelligent functions can be characterized as a continuum: Manual interactive Automated In completely manual and totally automated systems, the user and machine are effectively decoupled. In the intermediate range, where IHMSs lie, the human and artificial agents must interact by communication. When designing integrated systems the designer needs to consider the nature of communication among the agents, human and artificial. The type of interaction depends, in part, on the knowledge each agent has of the others. An agent interacting with another agent, called a partner, can belong to two classes: (1) the agent does not know its partner, (2) the agent knows its partner. The second class can be decomposed into two sub-classes: (2a) the agent knows its partner indirectly (using shared data for instance), (2b) the agent knows its partner explicitly (using communication primitives clearly understood by the partner). This classification leads to three relations between two agents communicating: (A) competition, (B) cooperation by sharing common data, (C) cooperation by direct communication. In the competition case, the agent is totally ignorant of the existence of other agents. This can lead to conflicts for existing resources. Thus, it is necessary to define a set of synchronization rules for avoiding any problems of resource allocation between agents. Typically, these synchronization rules have to be handled by a supervisor. The supervisor can be one of the partners or an external agent. Obviously, if available resources exceed the requirements of all the agents, conflicts are automatically avoided. In the real world, we do not usually know all these requirements and it is impossible to create physical resources on demand. In the case of cooperation by sharing common data, the agent knows that its partner exists because it is aware of the results of (at least) some of the partner actions. Both of them use a shared data base. Agents use and update this data base. An example would be both agents noting all their actions on a blackboard to which the other agent refers before acting. This is no longer a problem of resource allocation, but a problem of agents cooperating to manage the shared data base. Some system has to maintain the consistency of the shared data. Also, cooperative relations between agents do not exclude competitive relations. Shared data are generally supported by resources for which the corresponding agents may be competing. In this case, synchronization rules have to deal with resolution of resource allocation conflicts and corresponding data consistency checking. In the previous cases, the interaction is always indirect. In the case of cooperating by direct communication, agents share a common goal and a common language expressed by messages, e.g., experts in the same domain cooperating to solve a problem. 4.3 Sharing autonomy between IAS and operator As noted earlier in the paper, human operators do poorly at monitoring tasks if they are not involved in cognitive activity related to the system being monitored. On the other hand, humans are not as good as machines at highly repetitive, complex, or very time-critical tasks. Thus a designer must strike a balance between total user control and autonomous machine control. Sharing autonomy between humans and machines is very important and poorly understood. In our analysis of this problem, we extend the concept of levels of automation introduced by Sheridan (1984). Figure 2 represents a model of the performance of an IHMS with differing levels of automation and under various assumptions of what is known about the overall task. The horizontal axis represents a continuum of levels of autonomy from manual control to complete automation. The vertical axis represents the performance of the human-machine system. Performance could be related to time, precision of results, costs to solve a given problem or some other heuristic performance measure.

5 Each curve of figure 2 corresponds to a level of knowledge the designer has about the task. In general, better designer knowledge leads to higher performance of the designed system, as illustrated by the tradeoff curves moving upward. Overall System Performance Human (Manual) Designer's knowledge of the task Optimum Balance of Autonomy Machine (automated) Limit on what can be automated Figure 2. Human-Machine System Performance versus Levels of Autonomy (Boy, 1986). Following the above model, the performance of the human-machine system increases with the autonomy of the machine, but only until some optimum, after which it decreases. That is because if the autonomy of the machine is further increased, the human operator is likely to lose control of the situation. Note that the level of autonomy can be increased to a certain limit fixed by technological limitations what can be automated. At present, typical engineering methods strive for maximum automation, up to the technical limitations. In the IHMS view, optimization criteria are centered on the end-user/machine performance. Such an approach necessitates simulations and real-world experimentation. Human-machine system performance optima can be shifted to the right if human operators are very well trained. As the knowledge and experience with the IHMS and the task increases, designers can achieve performance curves with optima shifted up (better performance) and to the right (greater automation). Finally, if everything can be known about the system in its environment (i.e., there are no unanticipated situations), then the corresponding optimum may be found at full automation. 5. The Role of Automation: An Intelligent Assistant System IASs are electronic extensions of users' capabilities. They mediate between the mechanical parts of the system, and share some of the intelligent functions with the user. Automated co-pilots, on-line intelligent user manuals, and intelligent notebooks are examples of IASs. We will use the example of the design of an existing IAS to illustrate the design parameters outlined above. As an example of an IHMS, consider the Orbital Refueling System (ORS). The ORS was used in the space shuttle to refuel satellites. It is a system of machinery controlled by astronauts. An IAS, called the Human- ORS-Expert System (HORSES), was developed to perform interactive fault diagnosis (Boy 1986). In a series of experiments, HORSES was used with users working in a simulated ORS environment. At the appropriate times during simulations, a malfunction generator generated fault scenarios for the ORS. Operators were observed using HORSES to assist with real-time fault diagnosis. The experiments produced data showing how users adapted to the task, and how their performance varied with different parameters. From the data, Boy developed the following model of user behavior. Fault identification can be represented by the Situation Recognition / Analytical Reasoning (SRAR) model (Figure 3). SRAR is an evolutionary model of the user. From the experiments to date, it appears that subjects use chunks of knowledge to diagnose failures. A chunk of knowledge is fired by the matching of a situation pattern with a perceived critical situation. This matching is either total or partial. After situation recognition, analytic reasoning occurs. This analytical part gives two kinds of outputs: diagnosis or new situation patterns. The chunks of knowledge are very different between beginners and experts. Beginners situation patterns are poor, crisp and static, e.g., The pressure P1 is less than 50 psia. Subsequent analytical reasoning is generally important and time-dependent. On the one hand, when a beginner uses an operation manual to make a diagnosis, his or her behavior is based on the pre-compiled engineering logic he has previously learned. On the other hand, when he or she tries to solve the problem directly, the approach is very declarative, i.e., using the system first principles. With practice, beginner subjects were observed to get a personal procedural logic (operator logic), either from the pre-compiled engineering logic or from a direct problem solving approach. This process is called knowledge compilation. Conversely, experts' situation patterns are sophisticated, fuzzy and dynamic, e.g., During fuel transfer, one of the fuel pressures is close to the isothermal limit and this pressure is decreasing. This situation pattern includes many implicit variables defined in another context, e.g., during fuel transfer means in launch configuration, valves V1 and V2 closed, and V3, V4, V7 open. Also, a fuel pressure is a more general statement than the pressure P1. The statement isothermal limit includes a dynamic mathematical model, i.e., at each instant, actual values of fuel pressure are compared fuzzily ( close to ) to this time-varying limit [P isoth = f(quantity, Time)]. Moreover, experts take this situation pattern into account

6 only if the pressure is decreasing, which is another dynamic and fuzzy pattern. It is obvious that experts have transferred part of analytical reasoning into situation patterns. This part seems to be related to dynamic aspects. Beginner Expert SITUATION PATTERNS s1 s2 sn S1 S2 SN ANALYTICAL KNOWLEDGE Figure 3. Situation Recognition / Analytical Reasoning (SRAR) Model (Boy, 1986). Thus, with learning, dynamic models are introduced in the situation patterns. It is also clear that experts detect broader sets of situations. First, experts seems to fuzzify and generalize their patterns. Second, they have been observed to build patterns more related to the task than to the functional logic of the system. Third, during the analytical phase, they disturb the system being controlled to get more familiar situation patterns, which are static most of the time, e.g., in the ORS experiment, pilots were observed to stop fuel transfer after recognizing a critical situation. The lessons of the HORSES experiments include the following. By analyzing the human-machine interactions in the simulated system, by simulation and protocol analysis, the developers could design an instrument that presented the information that the experts had learned to attend to (i.e., a monitor showing the relevant isothermal bands). This improved user and system performance. Second, the HORSES assistant strikes a balance in the sharing of autonomy. The original system designer did not anticipate the way that the operators would use the system, so letting them have control over the assistant allowed them to do accomplish what they had learned to do well. A1 A2 An a1 a2 an 6. Discussion IASs are enabled by AI technology, and will require continued advances to work. But the old view of intelligent agents as rational, all-knowing beings that search for paths to goals will need to be modified, since (i) operators and system (i.e., designer's) goals will differ; (ii) global system performance is not always the result of a sequence of well-defined actions, but arises from the interaction of agents with different abilities; and (iii) the closed-world assumption is invalid, since the space of possible behaviors of agents, their effects on the environment, and the relevant features of situations are never completely known. Modeling integrated systems containing humans and IASs will require a change in modeling methodology. Lacking first principles of human-machine interaction, empirical study using simulation of the IHMS is a necessary and important activity for the design of systems. Some areas ripe for research include: automated observation of user actions, formalisms for modeling human/machine interaction at design time (in the design space), model formulation tools to support the modeling, and incorporating results from human factors research into the design of intelligent assistant systems. References Anderson, J. R. (1983). Acquisition of proof skills in geometry. Pages in Machine Learning: An Artificial Intelligence Approach, R. Michalski, J. G. Carbonell, and T. M. Mitchell, editors. Palo Alto, CA: Tioga. Boy, G. B. (1986). An expert system for fault diagnosis in orbital refueling operations. AIAA 24th Aerospace Sciences Meeting, Reno, Nevada, January. Nagel, D. C. (1986). Pilots of the future: Human or Computer? Conference on Human-Machine Interaction and Artificial Intelligence in Aeronautics and Space, Toulouse, October. Sheridan, T. B. (1984). Supervisory control of remote manipulators, vehicle and dynamic processes: experiments in command and display aiding. In Advances in Man-Machine Systems Research, Vol 1, , J.A.I. Press, Inc.