Agentic education · Learning science

Actionable companion · Assessment & reasoning

Learning with agency: The Effectiveness of Artificial Intelligence in Undergraduate Health Professions Education: Systematic Review and Meta-Analysis of Randomized Controlled Trials

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background Health professions education faces increasing challenges from rising health care complexity, pedagogical shifts, and constrained curricular space, and rapidly expanding knowledge and technological advances. While artificial intelligence (AI) shows promise for transforming health professions education, evidence of its effectiveness remains unclear. Objective This study synthesized evidence from randomized controlled trials (RCTs) on the effectiveness of AI in undergraduate health professions education. Methods We included RCTs, randomized crossover trials, and cluster RCTs comparing AI against standard educational interventions at the undergraduate level. We excluded quasi-experimental studies and those without clear AI components. We searched PubMed, Cochrane, Embase, Educational Resources Information Center, and Web of Science up to January 26, 2026. Outcomes were categorized according by Kirkpatrick levels; risk of bias was assessed using the Risk Of Bias Instrument for Use in Systematic Reviews for Randomised Controlled Trials tool; random-effects meta-analysis was conducted in RevMan (Cochrane); and certainty of evidence was rated using the Grading of Recommendations, Assessment, Development, and Evaluation approach. AI interventions were subcategorized by technology type and educational functions, yielding 13 subcategories. Results Of 39,783 records identified, 66 RCTs (N=4911 participants; 2020-2026) were included. Subcategorized analyses across 7 outcome domains yielded 48 comparisons. Most studies had high risk of bias, mainly due to poor allocation concealment and blinding, and certainty of evidence ranged from low to very low. Large language model (LLM)–based personalized learning aids comprised the largest evidence base and showed positive effects for satisfaction (standardized mean difference [SMD] 0.93, 95% CI 0.40-1.46; 7 studies; 430 participants; I²=74%), confidence (SMD 0.91, 95% CI 0.54-1.29; 7 studies; 609 participants; I²=64%), and theoretical knowledge (SMD 0.53, 95% CI 0.13-0.94; 12 studies; 955 participants; I²=86%), all with very low certainty. Other AI subtypes, including LLM content generators, natural language processing (NLP) chatbots, and non-LLM adaptive learning platforms, showed generally favorable point estimates but substantial heterogeneity and wide CIs, often included no effect. Prediction intervals frequently crossed the null, indicating uncertainty across educational setting. No studies assessed Kirkpatrick levels 3 or 4. Conclusions This review synthesized RCT evidence on AI in undergraduate health professions education by technology type and function, incorporating evidence certainty. Despite the large number of included studies, evidence remains insufficient to inform educational practice. Some AI interventions may improve some learning outcomes, but effects are inconsistent and not reliably reproducible. High risk of bias, heterogeneity, imprecision, and absence of higher-level outcomes limit conclusions. AI applications should therefore be used cautiously and on a trial basis.

What kind of evidence is this?

Conversational & pedagogical agents · Evidence synthesis

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

We included RCTs, randomized crossover, and cluster-randomized trials that enrolled undergraduate students of health sciences, including medicine, pharmacy, nursing, and allied health. We accepted studies conducted in any health training setting, including clinical teaching that involved patient engagement, skills training, classroom learning in the form of problem-based, task-based, or case-based learning, written assignments, as well as any form of assessment across all periods of follow-up. We excluded studies that evaluated postgraduate or residency training to limit the scope of our review, given the sufficient number of studies published at the undergraduate level. We imposed no restrictions on publication year, language, or report format (abstract or full paper).

Limitations and uncertainty · source passage

We acknowledge several limitations of the evidence gathered and the review process. In terms of the evidence gathered, despite comprehensive and up-to-date searches, several limitations require acknowledgment. First, even after subcategorization, substantial residual heterogeneity remained within several comparisons (I²: 64%‐98%), which was not adequately explained by the study characteristics reported. This heterogeneity likely reflects the inherently context-dependent nature of educational interventions, where learner characteristics, instructor expertise, institutional culture, implementation fidelity, and assessment approaches interact in ways that are rarely reported in sufficient detail [119120]. Rather than invalidating the findings, this suggests the effectiveness of AI tools may be context-sensitive, shaped by local factors rarely reported in sufficient detail. The consistent direction of effect (most studies favoring AI) provides a preliminary signal of benefit, but the magnitude of any effect in a given setting remains uncertain.

Design the system before the pilot

Define whether the agent asks questions, gives hints, explains or assesses. Test hallucinated feedback, excessive praise, misleading confidence and responses when the learner is confused. A conversational interface alone does not establish autonomous agency.

  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Nai Ming Lai; Yin Sear Lim; Min Thein Win; Prabal Bhargava; Paraidathathu Thomas; Qi Chwen Ong. The Effectiveness of Artificial Intelligence in Undergraduate Health Professions Education: Systematic Review and Meta-Analysis of Randomized Controlled Trials. JMIR Medical Education (2026-05-05). https://doi.org/10.2196/88933

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.