Evaluating Large Language Models as Academic Tutors: A Human-AI Collaboration Approach for Structured Literature Review

The rise of Large Language Models (LLMs) such as ChatGPT, Microsoft Copilot, and Claude has generated growing interest in their application as educational tools. Recent studies have explored their use not only for content generation but also as instructional partners, particularly in cognitively demanding academic tasks such as literature reviews. However, limited empirical work has assessed their effectiveness as pedagogical agents within structured, dialogic educational settings. This study presents a qualitative, comparative analysis of three advanced LLMs—ChatGPT 5.3, Microsoft Copilot, and Claude 3.5 Haiku—each engaged as a tutor in accordance with Mollick’s AI role taxonomy. Using a 14-step interaction protocol grounded in the ram framework, each model facilitated a dialogically structured reflection in which students collaboratively constructed a SWOT analysis evaluating the model’s own instructional effectiveness. Claude 3.5 Haiku demonstrated the highest levels of protocol compliance, adaptability, and critical depth, consistently providing personalized and reflective feedback. ChatGPT offered accurate and efficient technical support but showed limitations in individualization and procedural fidelity. Microsoft Copilot struggled with dialogic continuity and adherence to the protocol, resulting in weaker overall tutoring performance. These findings underscore the importance of structured prompting and clearly defined pedagogical roles in LLM-based tutoring and highlight the potential of nextgeneration models like Claude to scaffold high-level academic tasks.