Commercial Chatbot: Performance Evaluation, Usability Metrics and Quality Standards of Embodied Conversational Agents

Professionals Center for Business Research 02(2015)

16 Pages Posted: 27 Feb 2015 Last revised: 3 Mar 2015

See all articles by Karolina Kuligowska

Karolina Kuligowska

University of Warsaw - Faculty of Economic Sciences

Date Written: February 5, 2015

Abstract

The aim of this paper is to explore commercial applications of chatbots, as well as to propose several measurement metrics to evaluate performance, usability and overall quality of an embodied conversational agent. On the basis of these metrics we examine existing Polish-speaking commercial chatbots that a) work in the B2C sector, b) reach the widest possible range of users, and c) are presumably the most advanced commercial deployments of their creators. We analyse various aspects of functioning of each embodied conversational agent: visual look, form of implementation on the website, speech synthesis unit, built-in knowledge base (with general and specialized information), presentation of knowledge and additional functionalities, conversational abilities and context sensitiveness, personality traits, personalization options, emergency responses in unexpected situations, possibility of rating chatbot and the website by the user. Our study reveals the current condition of Polish market of commercial virtual assistants and emphasizes the importance of a multidimensional evaluation of any commercial chatbot deployment.

Keywords: commercial chatbot, virtual assistant, virtual agent, embodied conversational agent, chatbot developer

Suggested Citation

Kuligowska, Karolina, Commercial Chatbot: Performance Evaluation, Usability Metrics and Quality Standards of Embodied Conversational Agents (February 5, 2015). Professionals Center for Business Research 02(2015). Available at SSRN: https://ssrn.com/abstract=2569637

Karolina Kuligowska (Contact Author)

University of Warsaw - Faculty of Economic Sciences ( email )

Dluga Street 44/50
Warsaw, 00-241
Poland

Register to save articles to
your library

Register

Paper statistics

Downloads
1,596
rank
9,858
Abstract Views
3,324
PlumX Metrics