Actionable companion · Learning design & outcomes
From AI evidence to responsible practice: Performance benchmarking of LLMs on Chinese national medical licensing education: Cross-lingual and question-type effects
AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.
Evidence and context
Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Health-professions education findings do not establish effectiveness in K–12 classrooms.
Original authors’ abstract, reproduced under the linked source license:
Abstract Background The cross-lingual and question-type variations affecting large language models (LLMs) accuracy on the Chinese national medical licensing educations remain insufficiently explored. Methods In this cross-sectional study (May 13–20, 2025), 396 educational questions (198 English–Chinese pairs) were extracted from the Chinese national medical licensing examination. ChatGPT-4o, ChatGPT-o3, Gemini-2.5-pro, Deepseek-V3, Deepseek-R1, and Doubao-1.5-pro were prompted to provide answers. Responses were compared against reference answers, and accuracy was computed for three question types: basic knowledge (Type A), case analysis (Type B), and integrative judgment (Type C). Results Across all question types and languages, Doubao-1.5-pro achieved the highest accuracy at 92.0% ± 1.3%, whereas ChatGPT-4o had the lowest accuracy at 82.8% ± 3.7%. There was a significant main effect of question type (P = 0.0038) but no main effect of language (P = 0.56). Post hoc tests confirmed that Type A performance exceeded Types B and C (P < 0.01), while B vs. C did not differ. Among the models, Doubao-1.5-pro, Deepseek-R1, and Deepseek-V3 demonstrated notable cross-lingual stability, with accuracy differences between Chinese and English versions remaining below 5%. Conclusion The question type was a key factor affecting LLMs performance on Chinese medical licensing exam questions, whereas language had no significant impact. Doubao-1.5-pro, Deepseek-R1, and Deepseek-V3 demonstrated particularly strong cross-lingual consistency. These findings point to the potential value of specialized LLMs for enhancing medical education in China.
What kind of evidence is this?
Supporting AI learning science · Survey / observational study
Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.
Study methods · source passage
The Chinese medical licensing examination utilizes three standardized formats: Type A (independent items), Type B (shared-stem items), and Type C (shared-option items). While Type A items focus on foundational knowledge retrieval, Type B and Type C formats introduce significant contextual complexity. Type B items require the model to sustain clinical reasoning across a multi-stage vignette, while Type C items assess its precision in differential diagnosis when faced with a static set of competing distractors. Since official examination papers are not publicly released, all items for this study were sourced from widely recognized, publicly available preparatory materials provided by Beijing Medical Examination Assistance Technology Co., Ltd. We selected materials from 2020−2022 to ensure the inclusion of the most comprehensive and contemporary collections with verified answer keys. To ensure the statistical validity of this sample size, we calculated that 198 items drawn from an annual population of approximately 600 questions provide a margin of error of approximately ±5.8% at a 95% confidence interval (assuming a 50% response accuracy for maximum variance). This degree of precision is widely considered sufficient for the evaluative analysis of LLMs. To maintain consistency for cross-lingual evaluation, only text-based items were included, while questions containing images, tables, or chemical formulas were excluded. From this pool of available text-based items, we employed a stratified random sampling approach to select 198 questions, strictly preserving the original distribution of Type A, B, and C formats. This resulted in a dataset comprising 76 Type A, 62 Type B, and 60 Type C items. The English versions were generated through a standardized forward-translation process conducted by two authors (YX Tang and SJ Wang), both of whom possess over ten years of clinical experience and have completed more than one year of formal medical training in the United States. The study was conducted from May 13–20, 2025. The LLMs include ChatGPT-o3 (released April 17, 2025), ChatGPT-4o (released May 13, 2025), Gemini-2.5-pro (released March 25, 2025), Doubao-1.5-pro (released January 22, 2025), Deepseek R1 (released January 20, 2025), and Deepseek V3 (released December 26, 2024).
Limitations and uncertainty · source passage
This study has several limitations. Although the passing standard for this examination is 60% accuracy, the results of this study do not indicate that these LLMs can pass the real examination. This is because the analysis was based on only 198 questions rather than the complete examination set, and it was limited to text-based single-choice questions, excluding items containing images, tables, chemical formulas, and other non-text modalities that are typically included in the full examination. This study only focused on accuracy, neglecting consistency, rationale, educational value, and clinical validity. Future studies are needed to address these issues. Extending assessments to various dimensions and real-world scenarios will better establish LLMs’ utility and reliability in clinical education.
Design the system before the pilot
Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.
- Specify a measurable learning objective, a comparator and a task without tool assistance before adopting a system.
A 90-day district pilot
- Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
- Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
- Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
- Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.
The board decision brief
Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.
Attribution and reuse
Yuxia Tang; Jian Chen; Shouju Wang. Performance benchmarking of LLMs on Chinese national medical licensing education: Cross-lingual and question-type effects. PLOS One (2026-04-08). https://doi.org/10.1371/journal.pone.0346518
Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.