Abstract
Background: Safety, triage, clinically important omissions, and communication are important when evaluating patient-facing large language models. Automated LLM-based evaluation may help scale this work, but its interpretation depends in part on how reproducibly human experts apply the reference standard.
Objective: To evaluate the clinical safety and performance of LLM responses to simulated Iraqi Arabic breast cancer scenarios, and assess agreement between GPT-5.1 and blinded human experts.
Methods: Fourteen LLM configurations each answered 50 scenarios, resulting in 700 responses across four clinical categories. GPT-5.1 evaluated each response using four 5-point domains and three binary safety-related flags. Human validation included 80 responses, independently rated by a blinded oncologist and a clinical pharmacist, and major disagreements were adjudicated by a third oncologist. Results: Weighted Cohen's kappa was 0.08–0.14, ICC(2,1) 0.09–0.14, and weighted Gwet's AC2 0.58–0.60. For binary safety-related flags, PABAK was higher (0.70–0.82). Agreement between GPT-5.1 and the final human reference for the composite score remained limited (ICC[2,1]=0.366). Estimated marginal mean final weighted scores were lowest for oncologic emergencies (48.9/100). GPT-5.1 detected all eight responses classified as unsafe by the final human reference, but also labeled 23 of 72 human-safe responses as unsafe.
Conclusions: LLM performance varied significantly, with oncologic emergencies posing the greatest difficulty. Human rating reliability also depended on both outcome assessed and the agreement metric used. GPT-5.1 showed potential as a conservative preliminary safety screener, but the false positives and the small number of unsafe responses do not support its use in autonomous clinical evaluation. Keywords: Large language models; Breast cancer; Iraqi Arabic; Clinical safety. LLM as a judge. Human evaluation.
| Intend to be published in the IJPS ? | NO |
|---|---|
| Has the manuscript been published | not published |
