Actionable companion · Learning design & outcomes

Learning with agency: Tracking a multitude of abilities as they develop

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Primary school. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Recently, the Urnings algorithm (Bolsinova et al., 2022, J. R. Stat. Soc. Ser. C Appl. Statistics, 71, 91) has been proposed that allows for tracking the development of abilities of the learners and the difficulties of the items in adaptive learning systems. It is a simple and scalable algorithm which is suited for large‐scale applications in which large streams of data are coming into the system and on‐the‐fly updating is needed. Compared to alternatives like the Elo rating system and its extensions, the Urnings rating system allows the uncertainty of the ratings to be evaluated and accounts for adaptive item selection which, if not corrected for, may distort the ratings. In this paper we extend the Urnings algorithm to allow for both between‐item and within‐item multidimensionality. This allows for tracking the development of interrelated abilities both at the individual and the population level. We present formal derivations of the multidimensional Urnings algorithm, illustrate its properties in simulations, and present an application to data from an adaptive learning system for primary school mathematics called Math Garden.

What kind of evidence is this?

Intelligent tutoring & adaptive learning · Design requires full-text appraisal

Stage: Primary school. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

We first briefly describe the original Urnings algorithm proposed by Bolsinova et al. (2022). Under the Rasch model, the response of learner i to item j can be conceptualized as an outcome of the following process: one ball is sampled from an infinite urn of learner i with the proportion of green balls equal to πi=expθi/1+expθi (with others being red) and another ball is sampled from an infinite urn of item j with the proportion of green balls equal to πj=expδj/1+expδj until the balls are of different colour; the colour of the ball from the learner's urn determines the outcome: green for correct, red for incorrect. We can express this algorithmically as follows:

Limitations and uncertainty · source passage

While the Urnings rating system combines the benefits of Elo with desirable statistical properties, in its current form an important limitation prevents it from being optimally suited for ALSs. The system has currently been developed for dealing with a single ability dimension, and hence is tied to unidimensional applications. ALSs generally consider a wide range of abilities with both between‐ and within‐item multidimensionality. Hence, developing a multidimensional extension of the Urnings algorithm is of importance for improving its feasibility and practical usefulness of working with ALSs.

Design the system before the pilot

Document the learner model, mastery rule, hint sequence and adaptation signal. Compare adaptive routing with the same content in a non-adaptive sequence. Check whether early errors lead to persistent low-expectation pathways.

Specify a measurable learning objective, a comparator and a task without tool assistance before adopting a system.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Pilot one teacher-led unit with short guided turns. The tutor asks for an explanation and offers a graduated hint before any worked solution. Use concrete examples and an equivalent task without AI. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Use an age-appropriate independent post-task and a delayed parallel task. Score explanation quality and transfer, not the number of agent messages or completion rate alone. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Maria Bolsinova; Matthieu J S Brinkhuis; Abe D Hofman; Gunter Maris. Tracking a multitude of abilities as they develop. The British Journal of Mathematical and Statistical Psychology (2022-06-05). https://doi.org/10.1111/bmsp.12276

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.