Actionable companion · Feedback & scaffolding

Learning with agency: AI-related measures and self-reported lesson-aim attainment: an explainable machine-learning analysis of TALIS 2024

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Introduction Artificial intelligence (AI) is increasingly used to support teachers' lesson planning, feedback, assessment, and learning-material adaptation in school settings. Effective teaching practices may also support health literacy, social-emotional development, and health-promoting learning environments, although TALIS does not directly measure public-health education outcomes. Existing studies often treat AI use as a simple adoption variable and provide limited cross-national evidence on whether AI-related measures provide incremental predictive information after accounting for teacher background, teachers' work-related stress, and education-system context. Methods A nested explainable machine-learning framework was developed using TALIS 2024 teacher data to test the incremental predictive information supplied by AI-related measures. The participants were 88,991 primary, lower-secondary, and upper-secondary teachers (ISCED 1–3) from 55 countries/economies who were eligible for the AI-module analysis and had a valid outcome. The seven-item outcome captured teachers' reports of how far lessons taught in the previous week achieved specified instructional and classroom-management aims; a relatively high indicator denoted scores at or above the 75th percentile within the same country/economy and ISCED stratum. Matched baseline and AI-added regularized LightGBM models were compared using country-by-school grouped validation and early stopping. Threshold, continuous-outcome, survey-weighted-evaluation, missingness, and country-heterogeneity sensitivity analyses were conducted. Results In the final matched analysis, the AI-added model increased independent-test AUROC from 0.6021 to 0.6299 and AUPRC from 0.4138 to 0.4476, while reducing the Brier score from 0.2114 to 0.2071 and log loss from 0.6126 to 0.6029. Paired bootstrap intervals indicated statistically stable, but practically modest, gains (observed ΔAUROC = 0.0277, 95% CI 0.0215–0.0334; observed ΔAUPRC = 0.0338, 95% CI 0.0262–0.0411). AUROC gains were similar at the 70th and 80th percentile thresholds, and the continuous-outcome R2 increased from 0.1745 to 0.1911. Country-specific AUROC gains were positive in 48 of 55 countries/economies but varied substantially. Discussion AI-related perceptions and breadth of task use supplied modest incremental predictive information for predicting relatively high self-reported lesson-aim attainment. “Relatively high” denotes only a within-country/economy and ISCED ranking and does not indicate objectively superior teaching or better student learning. The modest discrimination, cross-sectional design, routed missingness, and self-reported outcome preclude causal interpretation and use for teacher screening or other high-stakes individual decisions. Public-health relevance is presented as a potential extension to future research in health-promoting school settings, not as a directly measured outcome. Keywords: artificial intelligence, explainable machine learning, health-promoting schools, LightGBM, self-reported lesson-aim attainment, SHAP, TALIS 2024

What kind of evidence is this?

Supporting AI learning science · Survey / observational study

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

This secondary cross-sectional study used the TALIS 2024 teacher public-use file. TALIS employs stratified two-stage probability sampling: schools are sampled within explicitly and implicitly defined strata, followed by random sampling of eligible teachers within participating schools; participation and response-rate procedures are documented by the OECD (18, 25). Participants were primary, lower-secondary, and upper-secondary teachers in the ISCED 1, 2, and 3 populations. The primary objective was conditional prediction in the assembled analytic sample, so models were fitted without final teacher weights and are not design-based population estimates. As a sensitivity analysis, all test metrics were recalculated using the final teacher weight. Country/economy and school identifiers were concatenated to form 16,799 unique school groups, preventing schools from appearing in both training and test data. Country/economy was included as context, and country-specific performance was examined. Survey clustering, unequal selection probabilities, and system-specific response patterns remain limitations.

Limitations and uncertainty · source passage

This study has several limitations. First, all focal variables were measured at one time point, so temporal ordering and causality cannot be established; reverse associations and common-method bias are plausible. Second, SLAA is a recent, class-specific teacher report rather than student outcomes, observations, or supervisor ratings. The upper-quartile label denotes only relative position within country/economy and ISCED stratum; it is not objectively superior teaching performance or a common cross-national standard. Third, AI-module routing and missingness reduced the raw file substantially. In particular, 59.1% missingness in task breadth was largely routing-related, so its SHAP attribution can conflate observed breadth with eligibility or questionnaire-pathway information. Train-only imputation and missingness indicators retain records for prediction but do not correct selection bias; the low observed background standardized differences do not rule out unmeasured selection. Fourth, unweighted model fitting targets the analytic sample rather than design-based TALIS populations. The direction of weighted results was similar, but baseline performance and the incremental magnitude changed meaningfully; clustering, unequal selection probabilities, country sample-size imbalance, response styles, and cultural/linguistic measurement comparability remain concerns. Fifth, the 20-item stress index covers heterogeneous sources and experiences. Its high alpha and omega may partly reflect item count, and its mean communality of 0.350 does not establish a single latent stress dimension; its use as one broad adjustment variable may obscure dimension-specific patterns. Full measurement invariance, convergent validity, and discriminant validity were not established. Sixth, the Health/PE scenario mainly represents school physical-education/health teachers, is not representative of public-health, medical, or university education, and is a same-source descriptive subset rather than external validation. The study did not measure public-health curricula or health outcomes. Seventh, modest discrimination precludes individual decision use. Finally, SHAP explains fitted predictions, can redistribute importance across correlated variables, and establishes neither statistical association nor causal or actionable effects.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

ZhuFang Mao. AI-related measures and self-reported lesson-aim attainment: an explainable machine-learning analysis of TALIS 2024. Frontiers in Public Health (2026-08-28). https://doi.org/10.3389/fpubh.2026.1911565

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.