Actionable companion · Assessment & reasoning

Learning with agency: More polished, not necessarily more learned: LLMs and perceived text quality in higher education

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

The use of Large Language Models (LLMs) such as ChatGPT is a prominent topic in higher education, prompting debate over their educational impact. Studies on the effect of LLMs on learning in higher education often rely on self-reported data, leaving an opening for complimentary methodologies. This study contributes by analysing actual course grades as well as ratings by fellow students to investigate how LLMs can affect academic outcomes. We investigated whether using LLMs affected students’ learning by allowing them to choose one of three options for a written assignment: (1) composing the text without LLM assistance; (2) writing a first draft and using an LLM for revisions; or (3) generating a first draft with an LLM and then revising it themselves. Students’ learning was measured by their scores on a mid-course exam and final course grades. Additionally, we assessed how the students rate the quality of fellow students’ texts for each of the three conditions. Finally we examined how accurately fellow students could identify which LLM option (1–3) was used for a given text. Our results indicate only a weak effect of LLM use. However, writing a first draft and using an LLM for revisions compared favourably to the ‘no LLM’ baseline in terms of final grades. Ratings for fellow students’ texts was higher for texts created using option 3, specifically regarding how well-written they were judged to be. Regarding text classification, students most accurately predicted the ‘no LLM’ baseline, but were unable to identify texts that were generated by an LLM and then edited by a student at a rate better than chance.

What kind of evidence is this?

Supporting AI learning science · Design requires full-text appraisal

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

In total the course had 166 students and out of these, 118 students chose to participate in the study (26 = women and 92 = men). Five were removed for the mid-course exam analysis (due to not completing the exam) four out of these five were removed from the final grade analysis (due to not completing the course).

Limitations and uncertainty · source passage

Overall, the absence of strong differences between the conditions is itself interesting because it indicates that using LLMs makes little difference in either direction except for how well written (linguistically) a text is perceived. Hence in the context of this particular study, LLMs appear to yield benefits as a form of advanced autocorrect, but do not affect the actual content of the text very much. Further on, even if there was a weak and uncertain effect for the “predicted competency” (estimated effect −0.21 [−0.61, 0.23] compared to No LLM baseline, Appendix Table A3) and “use concepts correctly” (estimated effect −0.14 [−0.48, 0.21] compared to No LLM baseline, Appendix Table A3), our analysis could detect little difference between the LLM conditions. We also acknowledge the limitation of operationalising outcomes as measured scores, which leaves the question of actual learning open. A future study could thus combine grades and scores with questionnaires about study material to possibly gauge the learning aspect more precisely.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Betty Tärning; Trond A. Tjøstheim; Annika Wallin. More polished, not necessarily more learned: LLMs and perceived text quality in higher education. Frontiers in Artificial Intelligence (2025-12-01). https://doi.org/10.3389/frai.2025.1653992

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.