Scalable AI-based clinical communication training and automated assessment

Synthesis: SOPHIE 2.0 turns the authors' earlier single-scenario AI-patient trial into a browser-based, self-service platform: 24 clinical scenarios, an embodied talking avatar, immediate personalized feedback, and an LLM judge scoring the 3E communication skills — Empower, Be Explicit, and Empathize. Validated against archived ratings from standardized patients and third-party raters, the selected judge (GPT-5.4 with low reasoning) reached Pearson 0.759 and ICC(A,1) 0.746 against the human consensus, inside the range of individual raters rather than outside it, from transcripts alone. In a remote study of 59 clinicians and students, 92% found it engaging, 86% easy to use, and 83% clinically relevant, and automated scores rose from the first to the second encounter. With no control condition, that rise cannot be attributed to training; what the study establishes is the feasibility of Automated Assessment for clinical communication training.

Key Findings

1. The LLM judge matched individual human raters. It reached Pearson 0.759, Spearman 0.720, MAE 0.106, and ICC(A,1) 0.746 (95% CI, 0.640–0.830) against the consensus of all human rating sources.
2. Transcripts carried most of the signal. The judge read only transcripts, while standardized patients participated live and third-party raters watched recordings; excluding SP ratings still left Pearson 0.719 and ICC 0.702.
3. Explicit communication was the hard dimension. Be Explicit correlations ranged from 0.414 to 0.598 across candidate judges, against 0.799 for Empower and 0.698 for Empathy; the cause is unknown.
4. Scores rose across two encounters. Overall 3E went from 0.403 to 0.471 (Δ = 0.068; 95% CI, 0.038–0.098; p < 0.001); Be Explicit changed little (Δ = 0.014; p = 0.1204).
5. Prior training tracked with higher scores. The 16 participants with previous communication training scored 0.468 against 0.426 for the 43 without (one-sided p = 0.013); 43 of 59, or 72.88%, had no prior training.
6. Participants judged it usable and relevant. 92% felt engaged, 86% found it easy to use, 85% believed it could be learned quickly, and 83% said the virtual patients elicited clinically used skills.
7. Human-likeness did not track effectiveness. 54% judged GPT-4o-mini more natural than the custom HAL model and 46% preferred it later, yet mean 3E performance was 0.437 versus 0.438 (p = 0.481).

An automated assessor that lands inside human rater variability

Self-service Simulation scales only if assessment scales with it. The authors transcribed recordings from their earlier randomized trial and gave candidate LLM judges the human raters' rubric, instructions, and score aggregation. GPT-5.4 with low reasoning was chosen before these analyses on combined agreement and low error, then compared with each human source in a leave-one-rater-out analysis: third of six sources on Pearson and Spearman, lowest MAE at 0.106, and second-highest ICC behind TP0 (0.780). The pattern matters more than the ranking — a judge reading text alone sat within the spread of humans, some of whom watched the encounter. The exception is Be Explicit, where every candidate judge was weaker.

What a self-service platform changes about access

SOPHIE 2.0 exchanges the personnel that constrain standardized-patient programs — actors, facilitators, raters, schedules — for a browser session that runs on laptops, tablets, or phones, at any hour, as often as a learner wants. Its 24 scenarios include 12 adapted from standardized-patient teaching materials and 12 written by clinical experts, spanning cardiology, primary care, oncology, critical care, pediatrics, neonatology, surgery, dental, and oral health, with virtual patients of varied ages, genders, and ethnicities. The backends were GPT-4o-mini and HAL, a Qwen2.5 14B model preference-optimized on roughly 7,500 medical conversation pairs, rendered through a Tavus-speaking avatar. Reported experience supports access without proving learning: 71% anticipated applying the skills they practiced, 73% would recommend it, and 66% found talking with an AI less stressful than talking with a person — though 47% still found it hard to feel real empathy toward an AI patient.

Human-likeness is a property of the interaction, not the model

The paper's most transferable design finding is a negative one. In the authors' earlier text-based work, HAL was judged more human-like than a commercial model; once both were embodied behind an avatar, that advantage disappeared. Participants rated avatar audio and visual quality at 3.95 out of 5 each and conversation content lowest at 3.73, and more often called GPT-4o-mini natural (54% versus 22%, 24% seeing no difference). Measured communication performance was effectively identical between backends. Voice, latency, avatar behavior, prompting, and turn-taking reshaped how the same dialogue quality was perceived: human-likeness is a property of the interaction, not the model.

What this means for practice

• Medical educators. Use AI-patient practice to remove scheduling and cost constraints, not as a substitute for standardized patients: what is demonstrated here is feasibility, usability, and perceived relevance, not transfer to real encounters.
• Assessment designers. Validate an automated rater against the humans it will sit beside and report per-dimension agreement: this judge's weakest dimension was also the only one with no measurable change across encounters.
• Administrators. Budget for infrastructure rather than actors: the reported deployment bottleneck is avatar-platform concurrency tied to a payment plan, and the training needed no live facilitator or human rater.
• Researchers. Prior communication training, not the two encounters, was the variable significantly associated with performance, so an AI-patient study needs a no-practice or no-feedback arm.

Limitations

• 59 of the 91 people who completed the demographic survey finished both encounters and the exit survey, and recruitment was online, so selection and attrition bias cannot be excluded.
• The two-session design had no no-practice or no-feedback control and participants could choose different scenarios, so the score increase cannot be attributed to feedback or learning.
• The LLM judge was internally validated on archived data from the earlier study, showed weaker agreement for Be Explicit, and read transcripts only, without vocal, visual, or affective cues.
• Human-likeness ratings cannot be attributed to the language models alone, since both were embedded in an audiovisual system; the Tavus platform also caps concurrent users by payment plan.

Connected Concepts

• Simulation
• Automated Assessment
• Workplace Learning
• Medical and Health Professions Education
• Conversational AI
• Virtual and Augmented Reality
• Feedback
• Self-Report Measures
• Assessment Validity
• Self-Efficacy
• Experiential Learning
• Usability Research

Connected Articles

• Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training — Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training
• MedEasy: Designing AI Standardized Patients for Clinical Consultation Training — MedEasy: Designing AI Standardized Patients for Clinical Consultation Training
• The Empirically Grounded Adaptive Virtual Patient for Psychotherapy Training: Disclosure That Responds to Therapist — The Empirically Grounded Adaptive Virtual Patient for Psychotherapy Training: Disclosure That Responds to Therapist
• Toward Accessible Psychotherapy Training Using AI-Driven Interactive Patient Avatars — Toward Accessible Psychotherapy Training Using AI-Driven Interactive Patient Avatars
• Using Generative AI to Simulate Patient History-Taking in a Problem-Based Learning Tutorial: A Mixed-Methods Study — Using Generative AI to Simulate Patient History-Taking in a Problem-Based Learning Tutorial: A Mixed-Methods Study
• ResidencyRL: Reinforcement Learning in Simulated Clinical Environments — ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
• AI-Powered Simulation for Nursing Education: Mixed Methods Systematic Review — AI-Powered Simulation for Nursing Education: Mixed Methods Systematic Review
• AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice — AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice
• Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation — Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
• AI as Teammate: Rethinking Task Distribution in Medical Training — AI as Teammate: Rethinking Task Distribution in Medical Training

Citation

Hasan, M., Epstein, R., Carroll, T., & Hoque, E. (2026). Scalable AI-based clinical communication training and automated assessment. arXiv:2609.22517.