Advancing Academic Chatbots: Evaluation of Non-Traditional Outputs

Abstract

Most studies still judge large-language-model (LLM) chatbots only on classic tasks such as factual question answering (Q&A) and short summaries. This thesis broadens that focus by testing whether LLMs can also produce complete slide decks and podcast scripts, and by asking whether a knowledge-graph retrieval strategy (Graph RAG) improves Q&A compared with a lighter hybrid search that blends keyword and semantic retrieval (Advanced RAG). We built a prototype chatbot for Copenhagen Business School that draws on the university’s research archive and compares two contrasting models: Meta LLaMA-3-70B, an openly released model that can run on local hardware, and OpenAI GPT-4o-mini, a commercial model accessed through an API. Each model is coupled with both retrieval pipelines so their interaction can be analysed systematically. Quality is judged in two stages. First, a panel of Copenhagen Business School evaluators scores every output on eleven dimensions spanning fluency, factual accuracy, coherence, usefulness, and hallucinations. Their ratings are then cross-checked by two external LLM “judges”, giving a scalable second opinion. Across ten representative papers, GPT-4o-mini with Advanced RAG delivers the most reliable answers and the most classroom-ready slides and podcasts, while LLaMA-3 approaches GPT-4o-mini in narrative tasks, such as podcasts, but falls behind on strict factual precision. Graph RAG provides no consistent gain over Advanced RAG; the extra structure helps a handful of multi-hop questions, but also increases hallucination rates and, because of time and computational constraints, requires manual curation of entities. Human and LLM judges ranked the systems in a similar way, but the human evaluations showed much more variation between individual scores. While this makes human assessment less consistent, it also has an important advantage: human evaluators were able to notice layout and stylistic issues in the slide decks and podcasts that the LLMs did not. This shows that combining the stability of LLM-based evaluation with the human ability to catch design and presentation flaws is the most effective way to assess the quality of non-traditional academic outputs. The entire codebase used to build the pipeline of the project and examples of outputs are openly accessible on GitHub at the link Chatbot in Academia - GitHub Repository.