Actionable companion · Feedback & scaffolding

Learning with agency: Artificial intelligence-assisted feedback in pharmacology education: a pilot evaluation of a custom generative model

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background Timely, individualised feedback is central to effective learning in medical education but remains resource intensive, particularly when based on short answer questions (SAQs). Large language models (LLMs) offer potential to support feedback provision, yet prospective evaluations of their accuracy and educational value remain limited. This pilot study evaluated the feasibility, accuracy and student perceptions of AI-generated pharmacology SAQ feedback. Methods A prospective pilot study was conducted within a UK Physician Associate MSc programme between March and October 2025. Students were invited to complete voluntary four-question formative pharmacology SAQs. A bespoke custom Generative Pre-trained Transformer (GPT), built using GPT-4 family LLM, guided by a standardised rubric and references, generated structured feedback. All outputs underwent faculty moderation prior to release. Student perceptions were assessed using a 5-point Likert-scale survey. 25% of student responses were double marked and independently reviewed in a blinded comparison of AI and faculty feedback. Results Twenty students submitted complete responses, generating feedback for a total of 80 questions. Mean AI feedback generation time was 34 s per quiz, compared with 508 s for faculty marking, representing a 15-fold reduction. Twelve of 20 (60%) feedback files required no modification before release, while eight required amendments, including four major corrections, even with faculty modification there was substantial efficiency gains from AI generated feedback. Eleven students (55%) completed the survey, reporting favourable perceptions of clarity, actionability, confidence and overall usefulness (median 4 out of 5) for the moderated feedback. In an exploratory blinded review of 5 double-marked submissions, no statistically significant differences were detected between AI-generated and faculty-generated feedback across rated domains (p > 0.23 for all domains). Conclusions A custom GPT-based LLM delivered rapid, structured pharmacology feedback with substantial efficiency gains and positive student perceptions. However, clinically important errors occurred, necessitating consistent faculty oversight. Generative models may augment formative assessment in medical education, but rigorous calibration, moderation and ethical safeguards remain essential for safe implementation. Supplementary Information The online version contains supplementary material available at 10.1186/s12909-026-09300-w. Keywords: Generative AI, Pharmacology education, Assessment feedback, Large language models

What kind of evidence is this?

Supporting AI learning science · Survey / observational study

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The study protocol was approved by our institutional research governance and ethics committee (committee reference number: ER/BSMSA24E/1). Participation was voluntary and students provided written informed consent electronically prior to enrolling in the study.

Limitations and uncertainty · source passage

In terms of limitations, the pilot nature and small sample size limit generalisability. No formal a priori sample size calculation was performed, as the study was designed as a pilot evaluation intended to assess feasibility and generate preliminary data. Only 20 students participated, and survey responses were obtained from 11, introducing potential response bias. As completion of both the formative quizzes and post-feedback survey was voluntary, students who found the feedback more useful or engaging may have been more likely to respond. Error classification relied on researcher judgement, and the threshold for major versus minor amendments may not be generalisable.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Edward Stephenson; Katie Morrigan; Ayesha Irfan; Kate Bascombe; Michael Okorie. Artificial intelligence-assisted feedback in pharmacology education: a pilot evaluation of a custom generative model. BMC Medical Education (2026-05-30). https://doi.org/10.1186/s12909-026-09300-w

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.