Exploring the Impact of User-Participated Customization in Experiencing Chatbot Failure
Extended Abstracts of the CHI Conference on Human Factors in Computing Systems 2024
ChatbotCustomizationFailureUser Experience
Does building your own chatbot change how you react to its mistakes?
Personalization can happen automatically, or it can involve users directly in shaping a chatbot. We explored whether participating in that process changes what people do when the chatbot fails.
Compare two routes to personalization
The study involved 22 participants: fourteen in a user-participated customization group and eight in a data-driven personalization group. We examined their reactions to failures and the kinds of customization they preferred for task-oriented and social-oriented chatbots.
Both groups were frustrated, but their intentions differed
Participants who customized their chatbots more often expressed an intention to try again. Those using pre-personalized chatbots more often expressed an intention to give up. The results suggest that investing effort in customization may change the relationship people have with the system, even when the failure itself remains frustrating.
Participation can affect the experience of failure
The paper discusses the IKEA effect: the tendency to value something more after helping create it. In this small exploratory study, the difference concerned reported intentions; it does not demonstrate long-term retention or improved chatbot reliability. Customization should be considered alongside the work of preventing and recovering from failures.
Read the original abstract
Traditionally, personalized AI systems have offered customized experiences by inferring user preferences from their system usage data, which can be referred to as data-driven personalization. With the advent of GPTs, opportunities for users to directly engage in the customization process have increased, which we term user-participated customization. In this study, we explored the impact of user-participated customization on user reactions to chatbot failures. We involved twenty-two participants in total, with fourteen for user-participated customization and eight for data-driven personalization. Both groups experienced frustration with chatbot failures, yet their responses differed. Those who customized their chatbots predominantly displayed retrying intentions, in contrast to those using pre-personalized chatbots, who primarily showed giving-up intentions. Moreover, we noted different customizations were preferred for task-oriented and social-oriented chatbots. This study suggests that user-participated customization has the potential to foster the Ikea effect. This effect is expected to mitigate negative experiences with chatbots.