WHAT KIND OF EDUCATIONAL RELATIONSHIP DOES A LARGE LANGUAGE MODEL CONSTRUCT? A QUALITATIVE ANALYSIS OF HUMAN ANNOTATIONS IN UNIVERSITY GUIDANCE DIALOGUES

Abstract

The adoption of Large Language Models (LLMs) in university guidance is reshaping how students explore academic pathways, raising a key pedagogical question: what kind of educational relationship does conversational AI construct during guidance dialogues? Developed within the ELOQUENCE project (HORIZON-CL4-2023-HUMAN-01-CNECT), this study extends the framework of synthetic personas (Casoria et al., 2025) and synthetic dialogues for bias elicitation (Augello et al., 2025), shifting the focus from algorithmic fairness to educational relationship quality. Through a thematic analysis of human annotations on university guidance dialogues, six pedagogical dimensions emerged: pedagogical listening, recognition of student uniqueness, continuity, agency, reflective scaffolding, and educational reciprocity. Findings show that, although LLMs produce coherent interactions, they remain limited in supporting authentic listening, personalization, and students' autonomous decision-making, providing a pedagogical framework for evaluating AI-supported guidance systems.