Actionable companion · Feedback & scaffolding
Learning with agency: Assessing LLM-generated vs. expert-created clinical anatomy MCQs: a student perception-based comparative study in medical education
AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.
Evidence and context
Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Health-professions learning must be distinguished from K–12 instruction.
Original authors’ abstract, reproduced under the linked source license:
ABSTRACT Large language models (LLMs) such as ChatGPT and Gemini are increasingly used to generate educational content in medical education, including multiple-choice questions (MCQs), but their effectiveness compared to expert-written questions remains underexplored, particularly in anatomy. We conducted a cross-sectional, mixed-methods study involving Year 2–4 medical students at Qatar University, where participants completed and evaluated three anonymized MCQ sets—authored by ChatGPT, Google-Gemini, and a clinical anatomist—across 17 quality criteria. Descriptive and chi-square analyses were performed, and optional feedback was reviewed thematically. Among 48 participants, most rated the three MCQ sources as equally effective, although ChatGPT was more often preferred for helping students identify and confront their knowledge gaps through challenging distractors and diagnostic insight, while expert-written questions were rated highest for deeper analytical thinking. A significant variation in preferences was observed across sources (χ² (64) = 688.79, p < .001). Qualitative feedback emphasized the need for better difficulty calibration and clearer distractors in some AI-generated items. Overall, LLM-generated anatomy MCQs can closely match expert-authored ones in learner-perceived value and may support deeper engagement, but expert review remains critical to ensure clarity and alignment with curricular goals. A hybrid AI-human workflow may provide a promising path for scalable, high-quality assessment design in medical education. KEYWORDS: AI-generated content, medical education, anatomy, large language models, assessment quality
What kind of evidence is this?
Supporting AI learning science · Evidence synthesis
Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.
Study methods · source passage
This study adopted a cross-sectional, mixed-methods design to evaluate and compare the quality of clinical anatomy MCQs generated by large language models (LLMs) – namely ChatGPT 3.5 (OpenAI) and Gemini 1 (Google DeepMind) – against those created by several experienced academic anatomists for 6 medical modules using principles of Team-Based Learning (TBL). To put it simply, TBL in anatomy uses pre-class preparation, individual and team assessments, and clinically focused group application exercises to promote deep understanding and collaborative problem-solving [17]. In this study, the MCQs generated by ChatGPT and Gemini were developed using a standardized prompt to ensure consistency across both platforms. The prompt was carefully crafted to reflect explicit instructions aligned with United States Medical Licensing Examination and National Board of Medical Examiners (USMLE/NBME-style) expectations, asking each model to create six long-form, clinically oriented MCQs based on specified learning objectives within a given anatomical system. Each question was required to include five answer choices, a clear correct response, and detailed explanations justifying both the correct and incorrect options. Importantly, both LLMs were used in their native, unmodified forms without fine-tuning or constraints tied to curricular content. This design choice allowed us to evaluate the models’ baseline capabilities in generating exam-quality questions and their alignment with established assessment standards in medical education. The primary objective was to assess undergraduate medical students’ perceptions of each MCQ set across key pedagogical dimensions, including clarity, clinical realism, scientific accuracy, cognitive difficulty, and overall learner-perceived educational value.
Limitations and uncertainty · source passage
This study addresses that need by focusing on clinical anatomy, a foundational discipline that presents unique demands in MCQ design – precision, spatial reasoning, and clinical contextualization. We compare AI-generated anatomy MCQs (using ChatGPT/Gemini) to expert-written items across dimensions such as realism, clarity, distractor quality, interdisciplinary integration, and cognitive depth. By evaluating both expert ratings and item performance with students, we aim to assess whether LLM-generated questions meet the pedagogical standards expected in medical education. Our findings will inform best practices for incorporating AI in assessment – identifying strengths, limitations, and how human-AI collaboration can best support high-quality question development.
Design the system before the pilot
Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.
- Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
- Measure engagement separately from achievement. Check whether novelty-driven participation persists and whether it translates to independent performance.
- Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.
A 90-day district pilot
- Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
- Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
- Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
- Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.
The board decision brief
Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.
Attribution and reuse
Maram Elzayyat; Janatul Naeim Mohammad; Sami Zaqout. Assessing LLM-generated vs. expert-created clinical anatomy MCQs: a student perception-based comparative study in medical education. Medical Education Online (2025-08-30). https://doi.org/10.1080/10872981.2025.2554678
Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.