Actionable companion · Assessment & reasoning

From AI evidence to responsible practice: ChatGPT as a prospective undergraduate and medical school student

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Health-professions education findings do not establish effectiveness in K–12 classrooms.

Original authors’ abstract, reproduced under the linked source license:

Abstract This article reports the results of an experiment conducted with ChatGPT to see how its performance compares to human performance on tests that require specific knowledge and skills, such as university admission tests. We chose a general undergraduate admission test and two tests for admission to biomedical programs: the Scholastic Assessment Test (SAT), the Cambridge BioMedical Admission Test (BMAT), and the Italian Medical School Admission Test (IMSAT). In particular, we looked closely at the difference in performance between ChatGPT-4 and its predecessor, ChatGPT-3.5, to assess its evolution. The performance of ChatGPT-4 showed a significant improvement over ChatGPT-3.5 and, compared to real students, was on average within the top 10% in the SAT test, while the score in the IMSAT test granted admission to the two highest ranked Italian medical schools. In addition to the performance analysis, we provide a qualitative analysis of incorrect answers and a classification of three different types of logical and computational errors made by ChatGPT-4, which reveal important weaknesses of the model. This provides insight into the skills needed to use these models effectively despite their weaknesses, and also suggests possible applications of our analysis in the field of education.

What kind of evidence is this?

Supporting AI learning science · Qualitative / mixed methods

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The reasons for selecting the previously described tests are varied. First and foremost, they provide a wide variety of questions across a range of problem categories that span several macro-areas, including logic, language production, critical thinking, mathematics, biology, and chemistry. These tests are designed to assess a wide range of skills and are the result of many years of collaboration between experts with diverse expertise in the disciplines studied, in learning processes, and in educational practice in general.

Limitations and uncertainty · source passage

To achieve this new version, OpenAI rebuilt its entire deep learning stack and co-designed a supercomputer from the ground up with Azure for its workload, resulting in a highly stable training run for GPT-4. The model’s text input capability is available through ChatGPT and the API, while image input capability is currently being prepared for wider availability. GPT-4 has been used internally to demonstrate strong performance in support, sales, content moderation, programming, and AI output evaluation. However, developers still acknowledge some limitations, particularly in making reasoning errors. OpenAI describes GPT-4 as an ongoing system, with further developments to be provided soon; [2] is a detailed technical report on GPT-4’s new features and differences from GPT-3.5. It is worth noting that, as demonstrated in the experimental sections of the paper, although GPT-4’s average performance generally surpasses GPT-3.5, it is not uncommon for some questions to be answered correctly by GPT-3.5 but not by GPT-4.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Marco Giunti; Fabrizia Giulia Garavaglia; Roberto Giuntini; Giuseppe Sergioli; Simone Pinna. ChatGPT as a prospective undergraduate and medical school student. PLOS ONE (2024-10-23). https://doi.org/10.1371/journal.pone.0308157

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.