Actionable companion · Feedback & scaffolding

Learning with agency: Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background Canadian psychiatry residents must demonstrate consultation competency, assessed using the standardized assessment of a clinical encounter report (STACER). However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings. Objective This study aimed to evaluate the technical feasibility of an agentic AI system designed to support psychiatry residents’ consultation competence through simulated patient encounters with a patient agent and structured feedback from a rater agent. Methods We conducted a two-phase technical feasibility prospective single-arm cohort study of the STACER Agentic System, a large language model–based platform integrating a patient agent and a rater agent. Phase 1 involved automated evaluation of the patient agent using a psychiatrist agent across 227 synthetic major depressive disorder cases. Performance was assessed using DeepEval metrics (correctness, clarity, medical faithfulness, turn relevance, and role adherence) with descriptive statistics and 95% CIs. Phase 2 involved a preliminary user study with 14 convenience-sampled participants: a total of 5 members of the clinical research team and 9 psychiatry residents from the University of Alberta. Participants completed simulated diagnostic interviews and case presentations. Performance was evaluated using STACER-based scoring by the rater agent and 2 psychiatrists. Interrater reliability was assessed using intraclass correlation coefficients (α=.05). Participants rated realism, behavioral consistency, psychiatric nuance, and feedback utility using Likert scales and free-text answers. Results The patient agent demonstrated high behavioral (51/56, 91.07%) and symptom fidelity (105/110, 95.45%), with strong automated performance (medical faithfulness mean 0.99, 95% CI 0.99‐1.00; turn relevance 0.99, 95% CI 0.986‐0.992). Participants rated simulations as psychiatrically plausible and diagnostically useful, particularly for depressive symptom representation, although rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues. The rater agent generated structured STACER-aligned feedback with high intrarater consistency, especially at the section subtotal level. Interrater reliability with psychiatrists was poor at the item level (intraclass correlation coefficient range=0.25‐0.49) but improved to good-to-excellent agreement at the section level for psychiatry resident sessions (intraclass correlation coefficient range=0.89‐0.93). The rater agent’s scores fell between those of the 2 psychiatrists for the clinical research team and were lower than both human raters for psychiatry residents. Conclusions The STACER Agentic System demonstrates the technical feasibility of using agentic AI to simulate psychiatric consultations and deliver STACER-aligned formative feedback. By combining adaptive multiturn psychiatric simulation with competency-based evaluation, it shows promise in supporting cognitive aspects of consultation, though it remains limited in facilitating relational skills such as rapport building. These findings suggest agentic AI could expand scalable, low-risk opportunities for deliberate practice and formative feedback in competency-based psychiatric education. Further controlled studies are needed to evaluate educational effectiveness and integration into residency training.

What kind of evidence is this?

Agentic & multi-agent systems · Evidence synthesis

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The SAS is an LLM-powered educational platform designed to provide psychiatry residents with structured, simulation-based practice in consultation interviewing. Grounded in principles of simulation-based education [34], the system operationalizes the Kolb experiential learning cycle [31] by coordinating 2 complementary agents: a patient agent, which provides the concrete experience of a simulated psychiatric interview, and a rater agent, which supports reflective observation and abstract conceptualization through structured formative assessment, narrative feedback, and guided cross-examination [10]. Figure 1 illustrates how these agents collectively enact the 4 stages of the Kolb cycle to support competency-based medical education (CBME) [3536].

Limitations and uncertainty · source passage

Several limitations must be acknowledged when interpreting these findings. First, phase 1 relied on automated testing using the LLM-as-a-judge paradigm [5657], enabling high-volume evaluation but limiting the ecological validity of results. Automated benchmarking is useful for characterizing agentic behavior but cannot substitute for human educational outcomes. Second, although psychiatry residents were included, the sample size was small, and variability in interface use (text vs voice) introduced heterogeneity in the user experience, limiting generalizability and precluding conclusions about educational efficacy. Future studies should use randomized controlled designs with larger samples focused on psychiatry residents to evaluate competence development, diagnostic accuracy, and changes in consultation confidence. Third, and most notably, the limited capacity for rapport formation and nonverbal communication directly impacts multiple STACER domains, including affect, engagement, and relational skills. These findings reinforce that current text- and voice-based agentic systems cannot fully replicate the interpersonal complexity of psychiatric encounters. Future development should prioritize multimodal enhancements to better capture both patient and learner nonverbal behaviors and improve ecological validity.

Design the system before the pilot

Document the agent goal, available tools, memory, planning loop and stopping rule. Test tool calls in a sandbox and require human approval for external actions. A multi-agent debate is not independent verification when agents share the same failure modes.

Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Alice Rueda; Huda F Al-Shamali; Zack Cote; Niloy Roy; Reinhard Janssen-Aguilar; Jithin Joseph; Bazen Gashaw Teferra; Mohammad Amin Kamaleddin; Lisa Burback; Olga Winkler; Bill Kapralos; Andrei Torres; Divya Sharma; Sridhar Krishnan; Guillaume Dumas; Andrew Greenshaw; Alexandre Hudon; Sanjeev Sockalingam; Yanbo Zhang; Adam Dubrowski; Venkat Bhat. Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study. JMIR Medical Education (2026-07-21). https://doi.org/10.2196/88580

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.