Agentic education · Learning science

Actionable companion · Learning design & outcomes

From AI evidence to responsible practice: Short-term learning effect of ChatGPT on pharmacy students' learning

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participants, intervention and comparator before applying this study in a new setting.

Original authors’ abstract, reproduced under the linked source license:

Introduction Students in pharmacy are positive towards integrating artificial intelligence and ChatGPT into their practice. The aim of this study was to investigate the direct short-term learning effect of using Chat GPT by pharmacy students. Methods This was an experimental randomized study. Students were allocated into two groups; the intervention group (n = 15) used all study tools and ChatGPT, while the control group (n = 16) used all study tools, except ChatGPT. Differences between groups was measured by how well they performed on a knowledge test before and after a short study period. Results No significant difference was found between the intervention and control groups in level of competence in the pretest score (p = 0.28). There was also no significant effect of using ChatGPT, with a mean adjusted difference of 0.5 points on a 12-point scale. However there was a trend towards a higher proportion of ChatGPT participants having a large (at least four point) increase in score (4 out of 15) vs control group (1 out of 16). Conclusion There is a potential for positive effects of ChatGPT on learning outcomes in pharmacy students, however the current study was underpowered to measure a statistically significant effect of ChatGPT on short term learning. Highlights • ChatGPT has potential as an educational tool for pharmacy students. • Pharmacy students who use ChatGPT showed an improvement in test scores in an experimental study setting. • Studies that investigate the effect of ChatGPT as an educational tool on long-term learning are warranted. • Pretest scores were positively associated with improvement in test scores.

What kind of evidence is this?

Supporting AI learning science · Randomized study reported

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The participants' characteristics are displayed in Table 1. Of 31 students, 2/3 were females and the participants median age were 23. Fourteen students attended the bachelor program, while 17 attended the master program.

Limitations and uncertainty · source passage

Artificial Intelligence has been used in teaching future health care professionals in communication skills by being virtual patients and physicians.1 In addition, AI chatbots have shown positive effects on learning outcomes for students in general.2 These studies have all been conducted on older AI/chatbot tools while students since November 2022 have had the chance to use generative AI tools such as the Chat Generative Pre-trained Transformer (Chat GPT). The introduction of ChatGPT at universities has been considered as a paradigm shift.3 The research community has begun to untangle how well this tool can provide information in pharmacy related topics. In one study it was shown that ChatGPT can supply answers to medicine therapy management cases4 and in another study that it can provide answers to questions related to the treatment of hypothyroidism in pregnancy fairly well.5 Moreover, a recent study reported that ChatGPT produces accurate answers to medical queries as judged by academic physician specialists, although the answers had some important limitations.6 Finally, a study from Thailand showed that ChatGPT performed worse than pharmacy students in an exam in the pharmacotherapy of shock and electrolyte disorders.7

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Specify a measurable learning objective, a comparator and a task without tool assistance before adopting a system.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Kristian Svendsen; Mohsen Askar; Danial Umer; Kjell H. Halvorsen. Short-term learning effect of ChatGPT on pharmacy students' learning. Exploratory Research in Clinical and Social Pharmacy (2024-07-23). https://doi.org/10.1016/j.rcsop.2024.100478

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.