Agentic education · Learning science

Actionable companion · Assessment & reasoning

Learning with agency: Large language model-generated versus teacher-written objective structured clinical examination stations for medical students: a blinded comparative pilot study

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Health-professions learning must be distinguished from K–12 instruction.

Original authors’ abstract, reproduced under the linked source license:

Developing objective structured clinical examination (OSCE) stations is time-consuming for medical teachers. We aimed to evaluate the ability of a large language model (LLM) to generate ready-to-use OSCE stations. Five OSCE stations generated by the LLM GPT-4o were evaluated by 7 expert assessors using a 5-point Likert scale and compared with 5 teacher-written stations targeting similar learning objectives. A station was considered to be of good quality if most assessors responded “agree” or “strongly agree” to the statement “The station is good enough to be used by students.” All teacher-written stations were rated as being of good quality, compared with only one GPT-4o-generated station. The LLM produced adequate clinical scenarios when reference knowledge was provided and tasks were clearly ordered, but it failed to generate reliable assessment grids. Careful review by teachers remained essential. GPT-4o failed to consistently produce fully ready-to-use OSCE stations.

What kind of evidence is this?

Supporting AI learning science · Model benchmark / tool evaluation

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The study used the METRICS checklist (Supplement 1, Table S1) to guide the design and reporting of the use of generative artificial intelligence in medical education [5]. Five GPT-4o-generated OSCE stations were blindly compared with 5 “mirror” stations written by a medical teacher and targeting the same learning objectives. The study followed 4 sequential steps: prompt engineering, station writing by the medical teacher, station generation by GPT-4o (up to 3 attempts), and blinded assessment of the stations. Writing and generation times were recorded. The study involved 1 assistant professor responsible for prompt engineering and for writing and generating the OSCE stations, as well as 7 academic physicians (1 full professor and 6 assistant professors). To ensure consistency in evaluation, each teacher was selected based on training and experience in OSCE development and evaluation; all were certified after completing the French national OSCE evaluator training course (https://formation.uness.fr/formation/course/view.php?id=20081). The paid version of GPT-4o was selected because of its availability and because it outperformed other LLMs in prior evaluations [6].

Limitations and uncertainty · source passage

Our study has several limitations. The success rate for generating ready-to-use OSCE stations was lower in our study (1 out of 5) than in previous studies [4,9]. This difference may be explained by our prompting strategy, which aimed to generate a complete OSCE station in a single output, whereas previous studies relied on sequential, step-by-step case generation. As inputs become longer, LLMs tend to use the available information less consistently [10]. Furthermore, prompt engineering, station generation, and station writing were all performed by the same teacher, which may have introduced bias. The performance of LLMs is rapidly evolving, and future improvements may affect the reproducibility and comparability of results obtained using identical prompts. Our a priori objective was not to modify the generated content and to obtain a fully “ready-to-use” OSCE station in each case, which may have been overly ambitious. Rather than acting as a complete substitute for OSCE writing, LLMs may be better suited as tools for saving time on specific components of OSCE construction, such as drafting clinical scenarios or standardized patient scripts. However, GPT-4o outputs are not reproducible, even when the same prompt is used repeatedly.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Piotr Szychowiak; Jonathan Wong-So; Hélène Messet; Mélanie Faure; Isaure Breteau; Simon Jamard; François Barbier; Maxime Desgrouas. Large language model-generated versus teacher-written objective structured clinical examination stations for medical students: a blinded comparative pilot study. Journal of Educational Evaluation for Health Professions (2026-05-26). https://doi.org/10.3352/jeehp.2026.23.9

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.