Agentic education · Learning science

Actionable companion · Feedback & scaffolding

From AI evidence to responsible practice: ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participants, intervention and comparator before applying this study in a new setting.

Original authors’ abstract, reproduced under the linked source license:

Abstract Authoring of help content within educational technologies is labor intensive, requiring many iterations of content creation, refining, and proofreading. In this paper, we conduct an efficacy evaluation of ChatGPT-generated help using a 3 x 4 study design (N = 274) to compare the learning gains of ChatGPT to human tutor-authored help across four mathematics problem subject areas. Participants are randomly assigned to one of three hint conditions (control, human tutor, or ChatGPT) paired with one of four randomly assigned subject areas (Elementary Algebra, Intermediate Algebra, College Algebra, or Statistics). We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help. Notably, ChatGPT-generated help failed quality checks on 32% of problems. This was, however, reducible to nearly 0% for algebra problems and 13% for statistics problems after applying self-consistency, a “hallucination” mitigation technique for Large Language Models.

What kind of evidence is this?

Supporting AI learning science · Randomized study reported

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

We selected algebra and statistics as the academic subject learning areas for this study. Algebra acts as a bridging subject towards many higher level math classes for STEM majors, particularly calculus. Additionally, due to the continuing struggles with math achievement in the U.S. [54], many efforts have been put towards improving algebra outcomes such as increases in commercial investments and public funding. This has also resulted in algebra becoming one of the most common subjects for tutoring systems. Secondly, statistics was chosen due to it often being a prerequisite course for data science, one of the world’s fastest growing majors [55]. Statistics is often viewed as a foundational and versatile subject, beneficial to a multitude of STEM majors. It also allowed us to experiment with how ChatGPT and our participant pool interacted with a more advanced academic subject. Pre-authored questions and hints for these subjects were available under a CC B-Y license from the OATutor Project. Each textbook subject is comprised of chapters, which contain learning objectives and sets of problems belonging to those learning objectives. To decide which learning objectives would be utilized in the study, each OATutor question was examined in terms of its associated learning objective. Within Algebra, we decided to choose one learning objective from Elementary Algebra, one from Intermediate Algebra, and one from the College Algebra textbook. We also chose one learning objective from the Statistics textbook.

Limitations and uncertainty · source passage

We investigate whether ChatGPT-generated hints can be beneficial to algebra and statistics learning by conducting an online study with 274 learners from Mechanical Turk. Participants are randomly assigned to the human tutor hint, ChatGPT hint, or no hint condition and are further randomly assigned to one of four mathematics tutoring subjects with questions adopted from OpenStax Elementary Algebra, Intermediate Algebra, College Algebra, and Statistics Creative Commons-licensed textbooks (https://openstax.org/subjects/math). We use an open-source tutoring system, Open Adaptive Tutor (OATutor), as the base platform to deliver the OpenStax derived questions with hints authored by human tutors from the OATutor Project [11]. This “human tutors” condition is compared to one in which the same questions are asked but the hints are replaced by worked solution hints generated entirely by ChatGPT. A three item repeated pre- and post-test is used to measure the learning gained from the randomly assigned condition. To an individual who has previously learned the concepts presented in their assigned subject, the acquisition phase may act as a review and thus aid in recall. To distinguish which gains are attributable to learning from the hint conditions versus exhibiting pre-post test improvement due to memory recall, a third, no-hint condition was included. This study remedies the limitations, in particular limited participation (i.e., N = 77) and ceiling effects, of a previous pilot study [12].

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Zachary A Pardos; Shreya Bhandari. ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLOS ONE (2024-05-24). https://doi.org/10.1371/journal.pone.0304013

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.