Agentic education · Learning science

Actionable companion · Assessment & reasoning

Learning with agency: Integrating ChatGPT in Orthopedic Education for Medical Undergraduates: Randomized Controlled Trial

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Health-professions learning must be distinguished from K–12 instruction.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background ChatGPT is a natural language processing model developed by OpenAI, which can be iteratively updated and optimized to accommodate the changing and complex requirements of human verbal communication. Objective The study aimed to evaluate ChatGPT’s accuracy in answering orthopedics-related multiple-choice questions (MCQs) and assess its short-term effects as a learning aid through a randomized controlled trial. In addition, long-term effects on student performance in other subjects were measured using final examination results. Methods We first evaluated ChatGPT’s accuracy in answering MCQs pertaining to orthopedics across various question formats. Then, 129 undergraduate medical students participated in a randomized controlled study in which the ChatGPT group used ChatGPT as a learning tool, while the control group was prohibited from using artificial intelligence software to support learning. Following a 2-week intervention, the 2 groups’ understanding of orthopedics was assessed by an orthopedics test, and variations in the 2 groups’ performance in other disciplines were noted through a follow-up at the end of the semester. Results ChatGPT-4.0 answered 1051 orthopedics-related MCQs with a 70.60% (742/1051) accuracy rate, including 71.8% (237/330) accuracy for A1 MCQs, 73.7% (330/448) accuracy for A2 MCQs, 70.2% (92/131) accuracy for A3/4 MCQs, and 58.5% (83/142) accuracy for case analysis MCQs. As of April 7, 2023, a total of 129 individuals participated in the experiment. However, 19 individuals withdrew from the experiment at various phases; thus, as of July 1, 2023, a total of 110 individuals accomplished the trial and completed all follow-up work. After we intervened in the learning style of the students in the short term, the ChatGPT group answered more questions correctly than the control group (ChatGPT group: mean 141.20, SD 26.68; control group: mean 130.80, SD 25.56; P=.04) in the orthopedics test, particularly on A1 (ChatGPT group: mean 46.57, SD 8.52; control group: mean 42.18, SD 9.43; P=.01), A2 (ChatGPT group: mean 60.59, SD 10.58; control group: mean 56.66, SD 9.91; P=.047), and A3/4 MCQs (ChatGPT group: mean 19.57, SD 5.48; control group: mean 16.46, SD 4.58; P=.002). At the end of the semester, we found that the ChatGPT group performed better on final examinations in surgery (ChatGPT group: mean 76.54, SD 9.79; control group: mean 72.54, SD 8.11; P=.02) and obstetrics and gynecology (ChatGPT group: mean 75.98, SD 8.94; control group: mean 72.54, SD 8.66; P=.04) than the control group. Conclusions ChatGPT answers orthopedics-related MCQs accurately, and students using it excel in both short-term and long-term assessments. Our findings strongly support ChatGPT’s integration into medical education, enhancing contemporary instructional methods. Trial Registration Chinese Clinical Trial Registry Chictr2300071774; https://www.chictr.org.cn/hvshowproject.html ?id=225740&v=1.0 Keywords: ChatGPT, medical education, orthopedics, artificial intelligence, large language model, natural language processing, randomized controlled trial, learning aid

What kind of evidence is this?

Supporting AI learning science · Randomized study reported

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

A parallel-design randomized controlled trial was used for this investigation. First, the accuracy of ChatGPT-4.0’s responses to orthopedics-related MCQs was examined. In addition, for a group experiment, 129 third-year medical students from Jinan University’s Medical College were recruited. They were divided into 2 groups at random, namely, the control group and the ChatGPT-4.0–assisted learning group (ChatGPT group). The internet-using students in the control group were not permitted to use any OpenAI-related software or programs, whereas those in the ChatGPT group used only ChatGPT-4.0 as the learning tool. Only after completing the orthopedics course, the orthopedics exercises, the review of the fundamental concepts in orthopedics, using the internet or ChatGPT, the orthopedic examination, and the final examination for the semester’s teaching task did the participants finish the experiment. The detailed process of experimental arrangement is shown in Figure 1. Both the CONSORT (Consolidated Standards of Reporting Trials) checklist (Multimedia Appendix 1) and the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and online TeleHealth; version 1.6.1) checklist (Multimedia Appendix 2) were used for this trial [30,31].

Limitations and uncertainty · source passage

First, although MCQs are often used to gauge medical students’ fundamental theoretical knowledge in many nations, China may have a distinct custom for framing questions and placing a different priority on knowledge points than other nations. Second, this study focuses more on orthopedic intervention than on multidisciplinary intervention, which might result in some limitations in the research findings. Third, during the preparation stage of this study, while assessing ChatGPT-4.0’s proficiency in answering MCQs pertaining to orthopedics, we made the decision to omit MCQs that included visual elements. This choice might have, to some degree, constrained the breadth of our investigation and consequently fell short of delivering an all-encompassing appraisal of ChatGPT-4.0’s aptitude in tackling a wide spectrum of medical inquiries. This study was a single-center randomized controlled trial; thus, more prospective multicenter and interdisciplinary investigations are required to fully examine ChatGPT’s potential as a teaching tool for medical education.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Wenyi Gan; Jianfeng Ouyang; Hua Li; Zhaowen Xue; Yiming Zhang; Qiu Dong; Jiadong Huang; Xiaofei Zheng; Yiyi Zhang. Integrating ChatGPT in Orthopedic Education for Medical Undergraduates: Randomized Controlled Trial. Journal of Medical Internet Research (2024-08-20). https://doi.org/10.2196/57037

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.