Agentic education · Learning science

Actionable companion · Assessment & reasoning

From AI evidence to responsible practice: The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participants, intervention and comparator before applying this study in a new setting.

Original authors’ abstract, reproduced under the linked source license:

Abstract Purpose Large language models (LLMs) have the potential to change medical education. Whether LLMs can generate multiple-choice questions (MCQs) that are of similar quality to those created by humans is unclear. This investigation assessed the quality of MCQs generated by LLMs compared to humans. Methods This review was registered with PROSPERO (CRD42025608775). A systematic review and frequentist random-effects network meta-analysis (NMA) or pairwise meta-analysis was performed. Ovid MEDLINE, Ovid EMBASE, and Scopus were searched from inception to November 1, 2024. The quality of MCQs was assessed with seven pre-defined outcomes: question relevance, clarity, accuracy/correctness; distractor quality; item difficulty analysis; and item discrimination analysis (point biserial correlation and item discrimination index). Continuous data were transformed to a 10-point scale to facilitate statistical analysis and reported as mean differences (MD). The MERSQI and the Grade of Recommendations, Assessment, Development and Evaluation (GRADE) NMA guidelines were used to assess risk of bias and certainty of evidence assessments. Results Five LLMs were included. NMA demonstrated that ChatGPT 4 generated similar quality MCQs to humans with regards to question relevance (MD −0.13; 95% CI: −0.44,0.18; GRADE: VERY LOW), question clarity (MD −0.03; 95% CI: −0.15,0.10; GRADE: VERY LOW), and distractor quality (MD −0.10; 95% CI: −0.24,0.04; GRADE: VERY LOW); however, MCQs generated by Llama 2 performed worse than humans with regards to question clarity (MD −1.21; 95% CI: −1.60,-0.82; GRADE: VERY LOW) and distractor quality (MD −1.50; 95% CI: −2.03,-0.97; GRADE: VERY LOW). Exploratory post-hoc t-tests demonstrated that ChatGPT 3.5 performed worse than Llama 2 and ChatGPT 4 with regards to question clarity and distractor quality (p < 0.001). Conclusion ChatGPT 4 may create similar quality MCQs to humans, whereas ChatGPT 3.5 and Llama 2 may be of worse quality. Further studies that directly compare these LLMs to human-generated questions and administer MCQs to students are required.

What kind of evidence is this?

Supporting AI learning science · Model benchmark / tool evaluation

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

This systematic review was reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement [14]. (S1 Table) and was registered with PROSPERO (CRD42025608775). Ethics approval was not required for this systematic review, as all analyses were performed using study data that had previously obtained informed consent and ethics approval.

Limitations and uncertainty · source passage

Our review has several limitations. There was variability in how MCQ quality was defined and measured across studies. Only a minority of studies were comparative; the remainder relied on single-group cross-sectional or non-randomized two-group designs, usually from a single institution and with small sample sizes. Such characteristics underpin the “very low” GRADE ratings. There are no published randomized control trials evaluating the quality of MCQs written by LLMs to those written by humans. Two-thirds of the studies included in our review did not perform head-to-head studies evaluating the quality of LLM-generated MCQs to those written by humans. Approximately half of the studies did not administer the MCQs to students; thus, the conclusions regarding the ability of LLMs to generate high-quality MCQs are drawn from a relatively small and contextually heterogenous set of studies. The certainty of evidence is very low and was limited by risk of bias within the studies, inconsistency, and imprecision; this was particularly evident with limited data to perform the network meta-analysis. Although inter-rater reliability among experts may be adequate, the limited validity evidence warrants restraint in inferring meaningful distinctions from variations in Likert-scale ratings of clarity, relevance, and distractor quality. One notable limitation is that most included studies focused on undergraduate and postgraduate medicine, with relatively few conducted in other allied health care professions. The extent to which our findings generalize beyond the field of medicine remains unclear. Finally, given the rapid evolution of LLMs, future or novel models may not align with the patterns observed in this systematic review and network meta-analysis. Nevertheless, the findings provide meaningful insights and highlight the need for further high-quality research across a broader range of health care fields to better understand the role of LLM-generated MCQs in medical education.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Have a subject specialist reproduce a small set of the paper’s tool-evaluation tasks using the documented model and prompt conditions. Keep this as a tool-reliability exercise until a separate learner study is designed. Require learners to attempt equivalent tasks independently before any AI comparison. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Report tool accuracy and error severity separately from learner outcomes. Measure learner understanding and transfer with unaided parallel tasks; model examination scores are not evidence of improved student learning. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Lauren Riehm; Keean Nanji; Moiz Lakhani; Evelina Pankiv; Dean Hasanee; Wesla Pfeifer. The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis. PLOS One (2026-01-02). https://doi.org/10.1371/journal.pone.0340277

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.