Agentic education · Learning science

Actionable companion · Learning design & outcomes

From AI evidence to responsible practice: The role of artificial intelligence in medical education: an evaluation of Large Language Models (LLMs) on the Turkish Medical Specialty Training Entrance Exam

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Health-professions education findings do not establish effectiveness in K–12 classrooms.

Original authors’ abstract, reproduced under the linked source license:

Objective To evaluate the performance of advanced large language models (LLMs)—OpenAI-ChatGPT 4, Google AI-Gemini 1.5 Pro, Cohere-Command R + and Meta AI-Llama 3 70B on questions from the Turkish Medical Specialty Training Entrance Exam (2021, 1st semester) and analyze their answers for user interpretability in languages other than English. Methods The study used questions from the Basic Medical Sciences and Clinical Medical Sciences exams of the Turkish Medical Specialty Training Entrance Exam held on March 21, 2021. The 240 questions were presented to the LLMs in Turkish, and their responses were evaluated based on the official answers published by the Student Selection and Placement Centre. Results ChatGPT 4 was the best-performing model with an overall accuracy of 88.75%. Llama 3 70B followed closely with 79.17% accuracy. Gemini 1.5 Pro achieved 78.13% accuracy, while Command R + lagged with 50% accuracy. ChatGPT 4 demonstrated strengths in both basic and clinical medical science questions. Performance varied across question difficulties, with ChatGPT 4 maintaining high accuracy even on the most challenging questions. Conclusions GPT-4 and Llama 3 70B achieved satisfactory results on the Turkish Medical Specialty Training Entrance Exam, demonstrating their potential as safe sources for basic medical sciences and clinical medical sciences knowledge in languages other than English. These LLMs could be valuable resources for medical education and clinical support in non-English speaking areas. However, Gemini 1.5 Pro and Command R + show potential but need significant improvement to compete with the best-performing models. Supplementary Information The online version contains supplementary material available at 10.1186/s12909-025-07148-0.

What kind of evidence is this?

Supporting AI learning science · Model benchmark / tool evaluation

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The questions were asked to the LLMs in Turkish. The questions were obtained from the official website of Student Selection and Placement Centre in multiple-choice (with five options from A to E) with a single best-answer format. The answers were provided by the LLMs in Turkish. This is consistent with the questions being in Turkish and the exam being a Turkish medical examination. You can reach all questions details in Supplementary Files.

Limitations and uncertainty · source passage

This study provides valuable insights into the performance of large language models (LLMs) on the Turkish Medical Specialty Training Entrance Exam (TUS), but several important limitations must be acknowledged to contextualize the findings and guide future research. First, it is uncertain whether the TUS questions were included in the training data of the AI models evaluated in this study. Since past TUS questions are publicly available, it is possible that the questions used in this study were part of the models'training data. This raises concerns about whether the models'performance reflects genuine understanding or simply the ability to memorize specific questions. Future studies should develop methods to assess whether AI models are demonstrating true reasoning capabilities or relying on memorized information.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Specify a measurable learning objective, a comparator and a task without tool assistance before adopting a system.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Have a subject specialist reproduce a small set of the paper’s tool-evaluation tasks using the documented model and prompt conditions. Keep this as a tool-reliability exercise until a separate learner study is designed. Require learners to attempt equivalent tasks independently before any AI comparison. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Report tool accuracy and error severity separately from learner outcomes. Measure learner understanding and transfer with unaided parallel tasks; model examination scores are not evidence of improved student learning. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Murat Koçak; Ali Kemal Oğuz; Zafer Akçalı. The role of artificial intelligence in medical education: an evaluation of Large Language Models (LLMs) on the Turkish Medical Specialty Training Entrance Exam. BMC Medical Education (2025-04-25). https://doi.org/10.1186/s12909-025-07148-0

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.