Adapting a student-facing chatbot to the needs of first generation students: a user experience study

Authors: Maria K. Wolters, Tatjana Kukic, Stefan Hillmann

Abstract:

First-generation students often face challenges in navigating university structures due to a lack of familial academic experience. The presented study investigates the adaptation of the student-support chatbot CHATU to better serve FGS at Technische Universität Berlin. In a two-stage evaluation, we first assessed the usability and effectiveness of the existing CHATU chatbot with FGS participants, revealing below-average usability ratings in perspicuity and dependability, as well as negative ratings in attractiveness, efficiency, stimulation, and novelty. Based on these findings, we developed CHATU-RAG, integrating Generative AI for improved interaction and adaptability. While perceived usability improved significantly, actual task success declined. User impression of the chatbot increased substantially (p<0.0005), highlighting the trade-offs between AI-driven interaction improvements and reliable information retrieval in an academic support system.