Democratizing Chatbot Debugging: A Computational Framework for Evaluating and Explaining Inappropriate Chatbot Responses

Xu Han, Michelle Zhou, Yichen Wang, Wenxi Chen, Tom Yeh

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Evaluating and understanding the inappropriateness of chatbot behaviors can be challenging, particularly for chatbot designers without technical backgrounds. To democratize the debugging process of chatbot misbehaviors for non-technical designers, we propose a framework that leverages dialogue act (DA) modeling to automate the evaluation and explanation of chatbot response inappropriateness. The framework first produces characterizations of context-aware DAs based on discourse analysis theory and real-world human-chatbot transcripts. It then automatically extracts features to identify the appropriateness level of a response and can explain the causes of the inappropriate response by examining the DA mismatch between the response and its conversational context. Using interview chatbots as a testbed, our framework achieves comparable classification accuracy with higher explainability and fewer computational resources than the deep learning baseline, making it the first step in utilizing DAs for chatbot response appropriateness evaluation and explanation.

Original languageEnglish
Title of host publicationProceedings of the 5th International Conference on Conversational User Interfaces, CUI 2023
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9798400700149
DOIs
StatePublished - 19 Jul 2023
Event5th International Conference on Conversational User Interfaces, CUI 2023 - Eindhoven, Netherlands
Duration: 19 Jul 202321 Jul 2023

Publication series

NameProceedings of the 5th International Conference on Conversational User Interfaces, CUI 2023

Conference

Conference5th International Conference on Conversational User Interfaces, CUI 2023
Country/TerritoryNetherlands
CityEindhoven
Period19/07/2321/07/23

Bibliographical note

Publisher Copyright:
© 2023 Owner/Author.

Keywords

  • Automatic Chatbot Evaluation and Explanation Framework
  • Chatbot Debugging
  • Conversational AI Agents
  • Interview Chatbot

Fingerprint

Dive into the research topics of 'Democratizing Chatbot Debugging: A Computational Framework for Evaluating and Explaining Inappropriate Chatbot Responses'. Together they form a unique fingerprint.

Cite this