Generative AI · Education leadership

Actionable companion · Assessment and evidence of learning

From evidence to district action: Which curriculum components do medical students find most helpful for evaluating AI outputs?

AI-assisted editorial implementation guide. The district actions are proposed applications, not findings or recommendations attributed to the original authors. No additional peer review is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Health-professions education context: test transfer to K–12 rather than assuming it.

Original authors’ abstract, reproduced under the linked source license:

Introduction The risk and opportunity of Large Language Models (LLMs) in medical education both rest in their imitation of human communication. Future doctors working with generative artificial intelligence (AI) need to judge the value of any outputs from LLMs to safely direct the management of patients. We set out to investigate medical students’ ability to evaluate LLM responses to clinical vignettes, identify which prior learning they utilised to scrutinise the LLM answers, and assess their awareness of ‘clinical prompt engineering’. Methods Final year medical students were asked in a survey to assess the accuracy of the answers provided by generative pre-trained transformer (GPT) 3.5 in response to ten clinical scenarios, five of which GPT 3.5 had answered incorrectly, and to identify which prior training enabled them to evaluate the GPT 3.5 output. A content analysis was conducted amongst 148 consenting medical students. Results The median percentage of students who correctly evaluated the LLM output was 56%. Students reported interactive case-based and pathology teaching using questions to be the most helpful training provided by the medical school for evaluating AI outputs. Only 5% were familiar with the concept of ‘clinical prompt engineering’. Conclusion Pathology and interactive case-based teaching using questions were the self-reported best training for medical students to safely interact with the outputs of LLMs. This study can inform the design of medical training for future doctors graduating into AI-enhanced health services. Practice points Students reported pathology and interactive case-based teaching using questions as the best training to evaluate the LLM outputs. Students need to be adequately equipped to engage with clinical prompt engineering to optimise the value of LLM outputs they interact with in their future practice.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Redesign one low-stakes assessment: require a first attempt, an AI-assisted revision, a source-check log and a short oral explanation. Use the same rubric for the comparison task. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning, factual accuracy and unaided transfer on a fresh task. Record suspected misconduct separately from demonstrated evidence. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

William J. Waldock; George Lam; Ana Baptista; Risheka Walls; Amir H. Sam. Which curriculum components do medical students find most helpful for evaluating AI outputs?. BMC Medical Education (2025-02-06). https://doi.org/10.1186/s12909-025-06735-5

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.