Agentic education · Learning science

Actionable companion · Feedback & scaffolding

Learning with agency: Automated Fidelity Monitoring of Lay-Delivered Mental Health Interventions Using Large Language Models: Development and Pilot Validation of shamiriAI in Kenya

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Secondary school. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background Task-shifting can help close the mental health treatment gap in low- and middle-income countries, but its effectiveness depends on ongoing supervision, which is hard to scale. AI tools that process session recordings and generate structured fidelity feedback could offer a scalable alternative; yet, to our knowledge, none have been developed or validated for lay-delivered, multilingual, group-format interventions in low-resource settings. Objective We developed and pilot-validated shamiriAI (ShamiriAI Institute), an automated fidelity-monitoring tool for lay-delivered mental health interventions, embedded within the Shamiri school-based program in Kenya. Methods Across 6 secondary schools in Ngong Hub, Kajiado County, Kenya (May-September 2025), shamiriAI processed session audio from 47 lay providers through a 5-stage pipeline: ingestion, multilingual automatic speech recognition (ASR) with prosodic feature extraction, personally identifiable information scrubbing, large language model–based fidelity inference, and supervisor reports. The following two aims were assessed: (1) ASR performance on a held-out test set of manually transcribed sessions and (2) interrater reliability between shamiriAI and independent human supervisor ratings across 52 sessions (38 AI-augmented and 14 standard) on 6 domains (Required Contents, Specifics, Thoroughness, Clarity, Skill, and Purity; 1‐7 scale). Reliability used intraclass correlation coefficients, Bland-Altman analysis, adjacent-agreement rates, paired t tests with Holm-Bonferroni correction, and Gwet AC2 (ordinal weights) across 3 formulations of the human reference. Results The ASR model achieved a character error rate of 0.19, word error rate of 0.34, and cosine semantic similarity of 0.77, indicating strong meaning preservation in code-switched speech. AI fidelity scores were systematically lower than the human composite overall (mean 5.14, SD 0.77 vs mean 5.93, SD 0.57 ); Δ=−0.79; d=−1.16; P<.001). Primary intraclass correlation coefficients ranged from −0.06 to 0.20 across the 6 domains, and AC2 sensitivity analyses (against each individual rater and the rounded composite) corroborated this dimension-level ordering. Three patterns emerged: large systematic underrating on holistic dimensions (Required Contents: d=−3.48; Clarity d=−1.56); bidirectional medium-effect bias on facilitation dimensions (Thoroughness d=−0.99; Skill d=+0.87); and substantial agreement on Specifics, Skill, and Purity (Gwet AC2 0.69‐0.76 against the rounded composite), relative to a human-human AC2 ceiling of 0.42‐0.60 estimated in the same dataset. Exploratory, underpowered subgroup and per-arm checks found no preliminary evidence of bias by lay-provider sex or age band or of arm-level differences. Conclusions Within this 52-session pilot, shamiriAI shows technically feasible multilingual ASR and a coherent, dimension-dependent reliability profile. Specifics, Purity, and Skill already reach substantial agreement, while underperformance on holistic dimensions (Required Contents and Clarity) reflects diagnosable misalignments in rubric interpretation and prompt design, specifying a concrete agenda for shamiriAI (version 2; Shamiri Institute). Whether AI-augmented supervision improves provider skill or student mental health outcomes will be tested in a planned cluster-randomized noninferiority trial.

What kind of evidence is this?

Supporting AI learning science · Randomized study reported

Stage: Secondary school. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

We conducted a pilot validation study of shamiriAI, an AI-based supervision support tool, embedded within routine delivery of the Shamiri intervention across 6 secondary schools in Ngong Hub, Kajiado County, Kenya, between May and September 2025. The study evaluated the following two pilot aims: (1) the technical performance of shamiriAI’s ASR pipeline on a held-out test set of manually transcribed sessions, and (2) interrater reliability between shamiriAI-generated fidelity ratings and independent human supervisor ratings across 52 recorded sessions. These aims concern the technical properties of the system, not between-arm comparisons.

Limitations and uncertainty · source passage

The largest discrepancies were observed for Required Contents (d=−3.48) and Clarity (d= −1.56). The AI differed from the panel by more than one scale point in 94% of Required Contents sessions, and Clarity adjacent agreement was 40%. Both dimensions require holistic, integrative judgment about whether a session was complete and clear “in spirit,” not whether scripted elements appeared verbatim. Human supervisors credit spirit-compliant delivery—paraphrase, local idiom, and language mixing—as meeting the standard, whereas the AI appears to apply a more literal rubric, penalizing missing scripted phrases [84,85]. Required Contents may have a further diagnosable cause: ASR errors in group audio destroy the short transitional utterances that signal element completion, and required elements are often spread across turns and speakers rather than delivered in one block. These are architectural, not prompt-tuning, problems and point to specific version 2 fixes, including required-element checklists in the prompt, sequential per-element evaluation before an overall rating, behavioral anchors from calibrated annotations, and few-shot examples spanning acceptable delivery styles.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Ask learners to attempt a problem unaided, compare an agent hint, explain a revision and verify the final reasoning against a trusted source. Make the permitted use explicit for each assignment. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Measure unaided transfer and delayed retention, calibration of confidence and the ability to identify an incorrect AI explanation. Record whether help-seeking becomes answer copying. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Shadrack Lilan; Brandon Mochama; Tom Osborn; Wendy Mmbone; Rachael Kilonzo; Faith Kamau; Rahim Daya; Christine Wasanga. Automated Fidelity Monitoring of Lay-Delivered Mental Health Interventions Using Large Language Models: Development and Pilot Validation of shamiriAI in Kenya. JMIR AI (2026-07-23). https://doi.org/10.2196/95063

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.