Actionable companion · Feedback & scaffolding

Learning with agency: Measurement of Ability in Adaptive Learning and Assessment Systems when Learners Use On-Demand Hints

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Adaptive learning and assessment systems support learners in acquiring knowledge and skills in a particular domain. The learners’ progress is monitored through them solving items matching their level and aiming at specific learning goals. Scaffolding and providing learners with hints are powerful tools in helping the learning process. One way of introducing hints is to make hint use the choice of the student. When the learner is certain of their response, they answer without hints, but if the learner is not certain or does not know how to approach the item they can request a hint. We develop measurement models for applications where such on-demand hints are available. Such models take into account that hint use may be informative of ability, but at the same time may be influenced by other individual characteristics. Two modeling strategies are considered: (1) The measurement model is based on a scoring rule for ability which includes both response accuracy and hint use. (2) The choice to use hints and response accuracy conditional on this choice are modeled jointly using Item Response Tree models. The properties of different models and their implications are discussed. An application to data from Duolingo, an adaptive language learning system, is presented. Here, the best model is the scoring-rule-based model with full credit for correct responses without hints, partial credit for correct responses with hints, and no credit for all incorrect responses. The second dimension in the model accounts for the individual differences in the tendency to use hints.

What kind of evidence is this?

Intelligent tutoring & adaptive learning · Design requires full-text appraisal

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The set of models described in the previous section were fitted to the English-from-Portuguese data set. First, the initial model was fitted (Equation 5). Second, the model extension with the differences in discrimination was fitted (Equation 6). Third, the further model extension with the category-specific parameters δ is was fitted (Equation 9), followed by the final extension with the additional latent variable (Equation 12). In the first four models, the scoring function a = [0, 1, 2, 3] was used. In addition to that, four multidimensional nominal response models with alternative scoring functions were fitted. Finally, five different IRTree models were fitted with the following specifications on the differences between the second and the third nodes of the IRTree: 1. θ1 = θ2, α2i = α3i, β2i = β3i 2. θ1 = θ2, α2i ≠ α3i, β2i = β3i 3. θ1 = θ2, α2i = α3i, β2i ≠ β3i 4. θ1 = θ2, α2i ≠ α3i, β2i ≠ β3i 5. θ1 ≠ θ2, α2i ≠ α3i, β2i ≠ β3i

Limitations and uncertainty · source passage

An important limitation of the current study is that the data were collected in a learning context, that is, where ability is changing, but the models assume that ability is constant. The small positive correlation, between ability and the hint dimension can be possibly explained by fact that those students who ask for more hints learn more and thus their average ability over the whole period is higher than students who ask for fewer hints and learn less. It would be interesting to apply the models in settings where no change in ability can be assumed and compare the results in terms of the relationship between the latent variables. Furthermore, the proposed models could be extended with a component that accounts for learning similar to the Additive Factor Model (Cen et al., 2006).

Design the system before the pilot

Document the learner model, mastery rule, hint sequence and adaptation signal. Compare adaptive routing with the same content in a non-adaptive sequence. Check whether early errors lead to persistent low-expectation pathways.

Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Maria Bolsinova; Benjamin Deonovic; Meirav Arieli-Attali; Burr Settles; Masato Hagiwara; Gunter Maris. Measurement of Ability in Adaptive Learning and Assessment Systems when Learners Use On-Demand Hints. Applied Psychological Measurement (2022-04-18). https://doi.org/10.1177/01466216221084208

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.