Understanding User Interactions with a Chatbot: A Self-determination Theory Approach

Transcription

1 Understanding User Interactions with a Chatbot: A Self-determination Theory Approach Emergent Research Forum (ERF) Quynh N. Nguyen University of North Texas Quynh.Nguyen@unt.edu Anna Sidorova University of North Texas Anna.Sidorova@unt.edu Abstract Artificial intelligence (AI) technology is advancing, and its application are widely used in many fields. AI chatbots powered by natural language processing are being integrated in numerous industries. AI chatbots help to improve customer services and enhance customer experiences. There is lack of study about the interactions between human and chatbots. As chatbots are designed to make interactions that are closer and more personal to the user, the way users communicate and interact with chatbots will differ from non- AI machines. Based on self-determination theory, this research aims to study the differences in system satisfaction between chatbot system and website system, and what factors determine satisfaction. The proposed research model hypothesizes the relationship between perceived autonomy, perceived competence, cognitive load, performance satisfaction, process satisfaction, and system satisfaction. It is proposed that the model will be tested using experimental survey. Keywords Chatbot, human-computer interaction, self-determination theory. Introduction The rise of artificial intelligence (AI) is dramatically transforming business world and human society. Over 85% of global executives believe that AI will allow their companies to gain or sustain a competitive advantage (Ransbotham et al. 2017). Chatbots, AI systems that communicate with users and perform basic tasks via chat or speech interface, are one of the most popular applications of AI. AI-powered bots are being integrated in numerous industries and business such as customer service, financial banking, airlines, retails, etc. Customers also find chatbots are easy and convenient medium to contact with companies. While human-computer interaction (HCI) is a popular topic in Information Systems (IS) literature, there is still a need for a better understanding about the nature of the interactions between human and chatbots. Chatbots are designed to make interactions with the user more similar to person-to-person conversation. Therefore, the way users communicate and interact with chatbots will differ from non-ai applications. This research seeks to examine the difference in satisfaction between chatbot system and website system. Using self-determination theory (SDT), we propose a research model to study the factors affect system satisfaction. Specifically, the study aims to address the following research questions: What factors affect system satisfaction in a chatbot system? Does the level of system satisfaction differ between a website system and a chatbot system? The study is expected to contribute to human-computer interaction research by highlighting the role of autonomy, competence and cognitive effort in explaining the difference in user experiences with natural language processing (NLP) based chatbots vis-à-vis traditional menu based websites. Twenty-fourth Americas Conference on Information Systems, New Orleans,

2 Literature Review AI Chatbot Chatbot can be defined as an AI application that uses NLP to understand and enable conversation between a human and a machine (Abdul-Kader and Woods 2015). Chatbots have become a crucial part of corporate strategy as they provide solutions for a variety of industries, from providing better customer service to automating enterprise IT help desks. To improve customer satisfaction, chatbots are programmed to analyze words, phrases and sentence constructions of customers so they can predict customer personality, figure out desirable product preferences of customers and adjust the conversations based on that. While carrying out conversations, chatbots collect data, which enables them to provide users with more satisfactory answers in the future, recommend personalized products and services. In the e-commerce context, chatbots handle communication with customers and assist them during the sales process, which is expected to have positive impact on online business (Horzyk et al. 2009). More experienced users are more satisfied with conversational agents compared to less experienced users on all the evaluated dimensions including impression, control, effectiveness, navigability, learnability etc. (Semeraro et al. 2008). Self-determination theory Self-determination theory represents a macro theory of human motivation and personality (Ryan and Deci 2000). SDT examines the motivation behind the choices people make, and the degree to which individual behavior is self-motivated and self-determined. According to SDT, people are active organisms that have growth tendencies, i.e. they repeatedly show effort, master challenges, and integrate new experiences. These natural development tendencies require social nutriments and supports. SDT identifies three basic psychological needs, which, if satisfied, allow healthy development and function. These needs are competence, relatedness, and autonomy. When these needs are fulfilled, they foster motivation for activities and enhance performances. In IS research, researchers have studied the effect of SDT constructs in the context of user s motivation to continue e-learning technology (Roca and Gagné 2008; Sørebø et al. 2009). An SDT-based extension of the Technology Acceptance Model showed that users who feel autonomous and competent are more likely to continue using IT in a workplace context (Roca and Gagné 2008). Natural Language Processing and User Interface User interface (UI) includes hardware and software components of a computer system, which allow users to interact with a software application and determine how users control the system. A good design UI is expected to result in positive user experience (UX) by allowing users to interact with a computer system in an easy and effective way. UX refers to the overall experience users have when they interact with the system, the company, the products or services by means of a user interface. AI is being employed by to improve UI/UX design as it helps to bring more human-like interactions into existing interfaces (Accenture 2017). NLP is a branch of AI that explores how computers can be used to analyze, interpret, understand and manipulate natural language text or speech in order to accomplish certain tasks (Chowdhury 2003). NLP, which can receive commands in natural language and process natural language statements into appropriate actions for the systems, enhances UX by providing a friendly, easily learned interface for users. Researchers have been using NLP to empower chatbots as well as specially design NLP based UI hardware (Ghose and Barua 2013). Research Model and Hypotheses For the purpose of this study, we focus on search tasks performed on travel website and via travel chatbots. A travel website interface normally includes graphical UI elements, such as forms, input controls (text fields, buttons, checkboxes), navigational components (search field, slider), graphic components (icons, banner), etc. When using a travel website, a user first formulates a specific goal, which involve finding accommodation or flights for a destination or travel dates. A user then follows a sequence of steps and interacts with different website components in order to transform their goals into commands for the system. Instead of displaying a set of menus, a chatbot uses NLP interface to bridge the gap between user s intent and the underlying language of the computer system. Previous research indicates that user performance using the natural language front-end is better than performance using the regular command language or Twenty-fourth Americas Conference on Information Systems, New Orleans,

3 standard UI (Napier et al. 1989). Perceived autonomy is defined as the extent a person feels about their ability to control their tasks freely within the system (Nikou and Economides 2017). When interacting with a menu-based website users have control over translating their intent into system commands through menu selections. They can click on the dropdown menu to choose one value from a list, check/uncheck a check box, type words on the search bar, etc. In the NLP-based interaction with a chatbot, users delegate the task of understanding their intent and translating it into system commands to the chatbot agent. They can tell the chatbot I d like to take a vacation from Austin to San Diego, from Jun 10 to Jun 15, the chatbot is designed to understand user s intent, translate it into system commands and produce an appropriate response. As a result, when interacting with a chatbot users have less control over task input step and consequently lose their autonomy. Cognitive effort refers to the amount of attentional capacity allocated to obtain and process information in order to perform a task (Hong et al. 2004). Webpage design with many elements such as animations, layout, etc. can impact user s visual processing, and the more visually complex a website is, the more likely user s cognitive load increases (Harper et al. 2009). Using a menu-based website is a complex task that requires understanding of existing system menus and making decisions about the appropriate menu choice, whereas an interaction with a NLP-based chatbot simply requires stating the desired outcome from the user s point of view. A complex task would require more information processing capability and increase the cognitive work (Wang et al. 2014). Therefore, NLP-based interaction is expected to require less attentional capacity as compared with the menu-based system. Hence, we propose: Hypothesis 1: Chatbots with natural language processing interface will lead to lower level of perceived autonomy than websites with menu-based interface. Hypothesis 2: Chatbots with natural language processing interface will lead to lower level of cognitive effort than websites with menu-based interface. Perceived competence is defined as the extent to which a person feels confident in their ability to complete a task, and it reflects the user s perception on whether they can interact with the system effectively to perform the task (Nikou and Economides 2017). When a user feels they are in control of the system, they are more likely to feel that they are able to meet the challenge of completing the task. Thus, higher perceived autonomy is expected to lead to higher perceived competence. As users expend less cognitive effort on the task, they perceive the task as less difficult and thus perceive themselves as more competent in completing the task. We propose the following: Hypothesis 3: Perceived autonomy has a positive effect on perceived competence. Hypothesis 4: Cognitive effort has a negative effect on perceived competence. Performance satisfaction is the extent to which a person is pleased with their performance in completing the task. Process satisfaction is the extent a person is pleased with the series of actions they execute while completing the task. When users feel in control of the system and their basic autonomy need is satisfied, the receive pleasure from the activity itself and are expected to be satisfied with the process. In philosophical studies view, a person s responsibility for something is determined by whether he has control over that thing or not (Smith 2008). Because users who feel in control of their interaction with the system will responsibility for the outcomes of the interaction, and therefore are more likely to judge such outcomes as satisfactory. Similarly, users who see themselves as competent at a task are more likely to enjoy and engage in the task. As users are more engaged in completing task, they are expected to be more satisfied with the process they follow. Hence, we propose the following: Hypothesis 5: Perceived autonomy has a positive effect on performance satisfaction. Hypothesis 6: Perceived autonomy has a positive effect on with process satisfaction. Hypothesis 7: Perceived competence is positively associated with process satisfaction. Research suggests that the amount of effort user expends to complete the task affects how they evaluate the system and the results (Al-Maskari and Sanderson 2010). In other words, user satisfaction with an information system is influenced by the level of benefit received and their evaluation on whether the required effort to use the system is worthy or not (Au et al. 2008). Al-Maskari and Sanderson (2010) found that the higher the effort users exert, the less satisfied they become. Other researchers revealed that cognitive load has greater direct effect on satisfaction than performance outcomes do (Hu et al. 2017). Hence, we predict that cognitive effort has direct impact performance satisfaction and process satisfaction. When a user spends high cognitive effort on the task, they are less likely to be satisfied with their performance on the task and the overall process. Hence, we propose the following: Hypothesis 8: Cognitive effort has a negative effect on performance satisfaction. Hypothesis 9: Cognitive effort has a negative effect on process satisfaction. Twenty-fourth Americas Conference on Information Systems, New Orleans,

4 System satisfaction refers to an extent an individual feel pleased with respect to the system and the mechanics of interaction (Wixom and Todd 2005). When a user uses a system to complete task, if they are satisfied with their performance, they would more likely evaluate the system as good system and be satisfied with the system. However, if a user has to follow complex steps to complete the task, they would be frustrated with the process and find that the cost of using the system, the effort they spend to execute tasks, outweigh the benefit. They may perceive that the system quality is low, which will shape their unfavorable attitude toward the system. Their satisfaction with the system would decrease. Thus, a higher process satisfaction would lead to higher system satisfaction. Hypothesis 10 and 11 are presented below: Hypothesis 10: Performance satisfaction has a positive effect on system satisfaction. Hypothesis 11: Process satisfaction has a positive effect on system satisfaction. Figure 1 summarizes the research hypotheses. Perceived Autonomy H5 (+) H1 (-) H3 (+) Performance H6 (+) H10 (+) UI type: NLP (Chatbot)* Perceived Competence H7 (+) System H2 (-) H4 (-) H8 (-) Process H11 (+) H9 (-) Cognitive effort *As opposed to menu-based website Figure 1. Research model Methodology We will collect data using an online experiment involving students from a public university as participants. The students will be asked to use Hipmunk website and Hipmunk chatbot to complete two information search tasks, including flight ticket and hotel room search. Hipmunk chatbot is an AI-powered virtual travel-planning assistant that not only helps search for flight and accommodation options, but also gives advice and recommendations. Participants will be randomly assigned to one of four treatments, each containing two scenarios in a same category and two user instructions (website vs chatbot), the orders of the scenarios and instructions are randomized. At the end of the search tasks, the participants will be asked about their search result. Performance satisfaction, process satisfaction, and system satisfaction will be measured by item developed by author. Perceived autonomy and perceived competence are measured using the scales adapted from Deci and Ryan (2000); Gagné (2003). Cognitive effort scale will be adapted from existing literature. Data will be analyzed using ANOVA and structural modeling. Conclusions This research applies SDT theory and HCI literature to understand the factors influencing user satisfaction with NPL based systems. The proposed process research model, if supported, would shed light on how factors such as perceived autonomy, competence and cognitive effort together influence user satisfaction with the interaction process, task performance and ultimately the systems itself. Baskerville and Myers (2009) use the term fashion wave to describe a periods of interest in IS topics. For IS discipline to be relevance, IS research should participate more directly at the beginning of the IS fashion-setting process. AI technology is considered the next big thing, it changes the relationship between human and machine, create a new type of interaction between human behavior and automation. This study is expected to provide new understanding about human-ai interaction, and the findings can be applied to practice to improve AI chatbot design and chatbot capabilities. Therefore, it will improve the relevance and importance of IS research to practice. Twenty-fourth Americas Conference on Information Systems, New Orleans,

5 References Abdul-Kader, S. A., and Woods, J Survey on Chatbot Design Techniques in Speech Conversation Systems, IJACSA) International Journal of Advanced Computer Science and Applications, (6:7) Accenture AI is the New UI, (available at Al-Maskari, A., and Sanderson, M A Review of Factors Influencing User in Information Retrieval, Journal of the Association for Information Science and Technology, (61:5), pp Au, N., T Ngai, E. W., E Cheng, T. C., Kong, H., Croteau, A.-M., and DeLone, W Extending the Understanding of End User Information Systems Formation: An Equitable Needs Fulfillment Model Approach, Source: MIS Quarterly MIS Quarterly, (32:1), pp Baskerville, R. L., and Myers, M. D Fashion Waves in Information Systems Research and Practice Qjarterly Fashion Waves in Information Systems Research and Practice1, Source: MIS Quarterly, (33:4), pp (available at Chowdhury, G. G Natural Language Processing, Annual Review of Information Science and Technology, (37:37), pp (doi: /aris ). Deci, E. L., and Ryan, R. M The what and why of goal pursuits: Human needs and the selfdetermination of behavior, Psychological Inquiry, (11:4), pp Gagné, M The Role of Autonomy Support and Autonomy Orientation in Prosocial Behavior Engagement, Motivation and Emotion, (27:3), pp (doi: /A: ). Ghose, S., and Barua, J. J Toward the implementation of a topic specific dialogue based natural language chatbot as an undergraduate advisor, in 2013 International Conference on Informatics, Electronics and Vision (ICIEV), IEEE, May, pp. 1 5 (doi: /ICIEV ). Harper, S., Michailidou, E., and Stevens, R Toward a Definition of Visual Complexity as an Implicit Measure of Cognitive Load, ACM Trans. Appl. Percpt, (6:10) (doi: / ). Hong, W., Thong, J. Y. L., and Tam, K. Y The Effects of Information Format and Shopping Task on Consumers Online Shopping Behavior: A Cognitive Fit Perspective, Journal of Management Information Systems, (21:3), pp (doi: / ). Horzyk, A., Magierski, S., and Mklaszewski, G An Intelligent Internet Shop-Assistant Recognizing a Customer Personality for Improving Man-Machine Interactions, Recent Advances in Intelligent Information Systems, pp Hu, P. J.-H., Hu, H., and Fang, X Examining the mediating roles of cognitive load and performance outcomes in user satisfaction with a website: A field quasi-experiment, MIS Quarterly, (41:3), pp Napier, H. A., Batsell, R. R., Guadango, N. S., and Lane, D. M Impact of a restricted natural language interface on ease of learning and productivity, Communications of the ACM, (32:10), ACM, pp (doi: / ). Nikou, S. A., and Economides, A. A Mobile-Based Assessment: Integrating acceptance and motivational factors into a combined model of Self-Determination Theory and Technology Acceptance, Computers in Human Behavior, (68), pp (doi: /j.chb ). Ransbotham, S., Kiron, D., Gerbert, P., and Reeves, M Reshaping Business With Artificial Intelligence, MIT Sloan Management Review Roca, J. C., and Gagné, M Understanding e-learning continuance intention in the workplace: A selfdetermination theory perspective, Computers in Human Behavior, (24:4), pp Ryan, R. M., and Deci, E. L Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being., The American psychologist, (55:1), pp Semeraro, G., Andersen, V., Andersen, H. H. K., de Gemmis, M., and Lops, P User profiling and virtual agents: A case study on e-commerce services, Universal Access in the Information Society, (7:3), pp (doi: /s ). Smith, A. M Control, responsibility, and moral assessment, Philosophical Studies, (138:3), Springer Netherlands, pp (doi: /s x). Sørebø, Ø., Halvari, H., Gulli, V. F., and Kristiansen, R The role of self-determination theory in explaining teachers motivation to continue to use e-learning technology, Computers and Education, (53:4), pp (doi: /j.compedu ). Wang, Q., Yang, S., Liu, M., Cao, Z., and Ma, Q An eye-tracking study of website complexity from cognitive load perspective, Decision Support Systems, (62), North-Holland, pp Wixom, B. H., and Todd, P. a A theoretical integration of user satisfaction and technology acceptance, Information Systems Research, (16:1), pp (doi: /isre ). Twenty-fourth Americas Conference on Information Systems, New Orleans,