Agentic education · Learning science

Actionable companion · Motivation & engagement

From AI evidence to responsible practice: Transforming education: tackling the two sigma problem with AI in journal clubs – a proof of concept

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participants, intervention and comparator before applying this study in a new setting.

Original authors’ abstract, reproduced under the linked source license:

Introduction Journal clubs are integral to continuing medical education, promoting critical thinking and evidence-based learning. However, inconsistent engagement, reliance on faculty expertise, and the complexity of research articles can limit their effectiveness. Generative Artificial Intelligence (Gen AI), particularly Large Language Models (LLMs) offers a potential solution, but general-purpose LLMs may generate inaccurate responses (“hallucinations”). Retrieval-Augmented Generation (RAG) mitigates this by integrating AI-generated content with curated knowledge sources, ensuring more accurate and contextually relevant responses. This study explores the development and preliminary evaluation of a RAG-enhanced LLM to support journal club discussions. Materials and methods A specialized LLM was deployed using Microsoft Azure’s GPT-4o. A vector database was created by embedding journal club articles using text-embedding-ada-002 (Version 2) for efficient information retrieval. A dedicated website provided user-friendly access. The study followed a design-based research (DBR) approach, engaging residents and faculty who interacted with the LLM before and during journal club sessions. Data collection included focus group discussions (FGDs) and questionnaires assessing engagement, usability, and impact. Results The study involved a total of 13 residents and three faculty members as participants. 50% of residents reported a positive experience, while the rest had a neutral response, citing both advantages and limitations. The LLM improved article summarization, query responses, and engagement as reported by residents. Moreover, the faculty observed enhanced discussion quality and preparation whereas overall challenges included the need for precise prompts and occasional misleading responses. Conclusion The study highlights the potential of a RAG-enhanced LLM to improve journal club engagement and learning. Future advancements in AI and open-source models may enhance accessibility, warranting further research.

What kind of evidence is this?

Supporting AI learning science · Qualitative / mixed methods

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

A specialized LLM was developed to assist with residents’ journal club sessions using RAG. Microsoft Azure (azure.microsoft.com) platform was utilized to deploy GPT-4o (https://openai.com/index/hello-gpt-4o/). Development was conducted in the Chat Playground by FU, which is an interactive environment provided by Microsoft Azure for configuring language models before deployment. This platform allows developers to experiment with various parameters and assess model behavior in a controlled setting.

Limitations and uncertainty · source passage

Despite these advantages, the integration of Gen AI in education, especially in teaching and learning, remains relatively underdeveloped in practical, real-world settings [6]. One major challenge is the general-purpose nature of many current LLMs [7]. These models are typically designed to be versatile, and capable of handling a wide array of tasks across various domains. While this broad applicability is beneficial in many contexts, it can also lead to significant limitations when it comes to specialized fields like medical and dental education [11].

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Measure engagement separately from achievement. Check whether novelty-driven participation persists and whether it translates to independent performance.
  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.
  • Co-design accommodations with learners and specialists. Check access and error burden across groups without inferring subgroup effectiveness from small or unreported samples.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Fahad Umer; Nighat Naved; Azra Naseem; Ayesha Mansoor; Syed Murtaza Raza Kazmi. Transforming education: tackling the two sigma problem with AI in journal clubs – a proof of concept. BDJ Open (2025-05-08). https://doi.org/10.1038/s41405-025-00338-4

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.