Benchmarking large language models against practicing clinicians on psychopathological assessment
Bunnell, B. E. et al. Automated detection and prediction of suicidal behavior from clinical notes using deep learning. PLOS ONE 20, e0331459 (2025).
Corcoran, C. M. & Cecchi, G. A. Using Language Processing and Speech Analysis for the Identification of Psychosis and Other Disorders. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 5, 770–779 (2020).
Landauer, T. K., Foltz, P. W. & Laham, D. An introduction to latent semantic analysis. Discourse Process 25, 259–284 (1998).
Häfner, H. Descriptive psychopathology, phenomenology, and the legacy of Karl Jaspers. Dialogues Clin. Neurosci. 17, 19–29 (2015).
Obuchowicz, R. et al. Artificial intelligence-empowered radiology—current status and critical review. Diagnostics 15, 282 (2025).
Lakhan, S. E., Vieira, K. & Hamlat, E. Biomarkers in psychiatry: drawbacks and potential for misuse. Int. Arch. Med. 3, 1 (2010).
Armstrong, C. APA Updates Guidelines on Psychiatric Evaluation in Adults. Am. Fam. Phys. 94, 62–64 (2016).
Crema, C., Attardi, G., Sartiano, D. & Redolfi, A. Natural language processing in clinical neuroscience and psychiatry: A review. Front. Psychiatry 13, 946387 (2022).
De Choudhury, M., Gamon, M., Counts, S. & Horvitz, E. Predicting depression via social media. Proc. Int. AAAI Conf. Web Soc. Media 7, 128–137 (2013).
Levkovich, I. & Omar, M. Evaluating of BERT-based and large language models for suicide detection, prevention, and risk assessment: a systematic review. J. Med. Syst. 48, 113 (2024).
Teferra, B. G. et al. Screening for depression using natural language processing: literature review. Interact. J. Med. Res. 13, e55067 (2024).
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171–4186 (2019).
Ji, S. et al. MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare. In Proceedings of the 13th Language Resources and Evaluation Conference (eds Calzolari, N. et al.) 7184–7190 (2022).
Humayun, A. et al. Artificial intelligence as a predictive tool for mental health status: Insights from a systematic review and meta-analysis. PLOS ONE 20, e0332207 (2025).
Omar, M. et al. Applications of large language models in psychiatry: a systematic review. Front. Psychiatry 15, 1422807 (2024).
Rosenman, G., Hendler, T. & Wolf, L. LLM Questionnaire Completion for Automatic Psychiatric Assessment. In Findings of the Association for Computational Linguistics: EMNLP 2024 (eds Al-Onaizan, Y., Bansal, M. & Chen, Y.-N.) 403–415 (2024).
Chancellor, S. & De Choudhury, M. Methods in predictive techniques for mental health status on social media: a critical review. npj Digit. Med. 3, 43 (2020).
Belcastro, L., Cantini, R., Marozzo, F., Talia, D. & Trunfio, P. Detecting mental disorder on social media: A ChatGPT-augmented explainable approach. Online Soc. Netw. Media 48, 100321 (2025).
Kim, J. et al. Large language models outperform mental and medical health care professionals in identifying obsessive-compulsive disorder. npj Digit. Med. 7, 193 (2024).
Volkmer, S., Glück, A., Meyer-Lindenberg, A., Schwarz, E. & Hirjak, D. Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study. Schizophrenia 11, 47 (2025).
Perfalk, E. et al. Predicting involuntary admission following inpatient psychiatric treatment using machine learning trained on electronic health record data. Psychol. Med. 54, 1–14 (2024).
Hansen, L. et al. The PSYchiatric clinical outcome prediction (PSYCOP) cohort: leveraging the potential of electronic health records in the treatment of mental disorders. Acta Neuropsychiatr. 33, 323–330 (2021).
Li, L. et al. A scoping review of using Large Language Models (LLMs) to investigate Electronic Health Records (EHRs). Preprint at (2024).
Madan, S. et al. Deep Learning-based detection of psychiatric attributes from German mental health records. Int. J. Med. Inform. 161, 104724 (2022).
Schultze-Lutter, F., Schmidt, S. J. & Theodoridou, A. Psychopathology—a precision tool in need of re-sharpening. Front. Psychiatry 9, 446 (2018).
Raballo, A., Ravenda, F. & Mira, A. Diagnosing schizophrenia spectrum disorders: Large language models (LLMs) vs. leading international psychiatrists (LIPs). Psychiatry Clin. Neurosci. 79, 599–600 (2025).
Levkovich, I. Evaluating diagnostic accuracy and treatment efficacy in mental health: a comparative analysis of large language model tools and mental health professionals. Eur. J. Investig. Health Psychol. Educ. 15, 9 (2025).
Buckley, T. A., Crowe, B., Abdulnour, R.-E. E., Rodman, A. & Manrai, A. K. Comparison of frontier open-source and proprietary large language models for complex diagnoses. JAMA Health Forum 6, e250040 (2025).
Takita, H. et al. A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians. npj Digit. Med. 8, 175 (2025).
El Arab, R. A. et al. Bridging the gap: from AI success in clinical trials to real-world healthcare implementation—a narrative review. Healthcare 13, 701 (2025).
Gargari, O. K. et al. Diagnostic accuracy of large language models in psychiatry. Asian J. Psychiatry 100, 104168 (2024).
Egli, S. B., Arpagaus, A., Amacher, S. A., Hunziker, S. & Bassetti, S. Use, knowledge and perception of large language models in clinical practice: a cross-sectional mixed-methods survey among clinicians in Switzerland. BMJ Health Care Inf. 32, e101470 (2025).
Dennstädt, F., Hastings, J., Putora, P. M., Schmerder, M. & Cihoric, N. Implementing large language models in healthcare while balancing control, collaboration, costs and security. npj Digit. Med. 8, 143 (2025).
Keltner, N. L., Grant, J. S. & McLernon, D. Use of actors as standardized psychiatric patients. J. Psychosoc. Nurs. Ment. Health Serv. 49, 34–40 (2011).
Hodges, B., Regehr, G., Hanson, M. & McNaughton, N. Validation of an objective structured clinical examination in psychiatry. Acad. Med. 73, 910–912 (1998).
Piot, M.-A. et al. Simulation in psychiatry for medical doctors: A systematic review and meta-analysis. Med. Educ. 54, 696–708 (2020).
Das AMDP-System: Manual zur Dokumentation des psychischen Befundes in Psychiatrie, Psychotherapie und Psychosomatik. (Hogrefe, Göttingen, 2023). https://doi.org/10.1026/03157-000.
The AMDP System: Manual for Assessment and Documentation of Psychopathology in Psychiatry. (Hogrefe, Boston, 2018).
Stieglitz, R.-D., Haug, A., Fähndrich, E., Rösler, M. & Trabert, W. Comprehensive psychopathological assessment based on the Association for Methodology and Documentation in Psychiatry (AMDP) System: development, methodological foundation, application in clinical routine, and research. Front. Psychiatry 8, 45 (2017).
Bertens, L. C. M. et al. Use of expert panels to define the reference standard in diagnostic research: a systematic review of published methods and reporting. PLOS Med. 10, e1001531 (2013).
Patel, R., Tsalik, E. L., Evans, S., Fowler, V. G. & Doernberg, S. B. Clinically adjudicated reference standards for evaluation of infectious diseases diagnostics. Clin. Infect. Dis. 76, 938–943 (2022).
Müller, M. J. & Szegedi, A. Effects of interrater reliability of psychopathologic assessment on power and sample size calculations in clinical trials. J. Clin. Psychopharmacol. 22, 318–325 (2002).
Rocha Neto, H. et al. Inter-rater reliability between structured and non-structured interviews is fair in schizophrenia and bipolar disorders—a systematic review and meta-analysis. Diagnostics 13, 526 (2023).
Imam, M. F., Lyu, C. & Aji, A. F. Can Multimodal LLMs do visual temporal understanding and reasoning? The answer is No! Preprint at (2025).
Zöller, N. et al. Human–AI collectives most accurately diagnose clinical vignettes. Proc. Natl. Acad. Sci. USA. 122, e2426153122 (2025).
Steyvers, M., Tejeda, H., Kerrigan, G. & Smyth, P. Bayesian modeling of human–AI complementarity. Proc. Natl. Acad. Sci. USA. 119, e2111547119 (2022).
Vaccaro, M., Almaatouq, A. & Malone, T. When combinations of humans and AI are useful: A systematic review and meta-analysis. Nat. Hum. Behav. 8, 2293–2303 (2024).
Reverberi, C. et al. Experimental evidence of effective human-AI collaboration in medical decision-making. Sci. Rep. 12, 14952 (2022).
Bedi, S. et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA 333, 319–328 (2025).
Kasagga, A. et al. Performance of ChatGPT and Large Language Models on Medical Licensing Exams Worldwide: A Systematic Review and Network Meta-Analysis With Meta-Regression. Cureus 17, e94300, (2025).
Liu, N. F. et al. Lost in the middle: how language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2024).
Xie, J., Zhang, K., Chen, J., Lou, R. & Su, Y. Adaptive Chameleon or Stubborn Sloth: revealing the behavior of large language models in knowledge conflicts. In the 12th International Conference on Learning Representations (2024).
Du, Y. et al. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025 23281–23298 (Association for Computational Linguistics, 2025). https://doi.org/10.18653/v1/2025.findings-emnlp.1264.
Escolà-Gascón, Á, Dagnall, N. & Drinkwater, K. Bias analysis in forensic and non-forensic psychiatric assessments. Int. J. Law Psychiatry 88, 101891 (2023).
Mendel, R. et al. Confirmation bias: why psychiatrists stick to wrong preliminary diagnoses. Psychol. Med. 41, 2651–2659 (2011).
Nelson, B. W. et al. Evaluating the performance of general purpose large language models in identifying human facial emotions. npj Digit. Med. 8, 615 (2025).
Elyoseph, Z. et al. Capacity of Generative AI to interpret human emotions from visual and textual data: pilot evaluation study. JMIR Ment. Health 11, e54369 (2024).
Xu, C., Guan, S., Greene, D. & Kechadi, M.-T. Benchmark Data Contamination of Large Language Models: A Survey. Preprint at (2024).
Ott, S., Barbosa-Silva, A., Blagec, K., Brauner, J. & Samwald, M. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nat. Commun. 13, 6793 (2022).
Läge, D., Egli, S., Riedel, M. & Möller, H.-J. Exploring the structure of psychopathological symptoms: a re-analysis of AMDP data by robust nonmetric multidimensional scaling. Eur. Arch. Psychiatry Clin. Neurosci. 262, 227–238 (2012).
Gebhardt, R. & Pietzcker, A. Zur Validierung der AMDP-Syndromskalen. Arch. Psychiatr. Nervenkr. 233, 509–523 (1983).
Fähndrich, E. & Renfordt, E. The AMDP system for the documentation of psychiatric symptoms: course and effectivity of a training seminar. Pharmacopsychiatry 18, 278–281 (1985).
Chiang, W.-L. et al. Chatbot Arena.: Open Platform for Evaluating LLMs by Human Preference (2024).
Dannhoferová, J. & Jedlička, P. Optimizing prompts through AI frameworks: a path to more relevant responses. In Proc. 27th Annual Int. Conference Economic Competitiveness and Sustainability 2025 29–38. (2025).
Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. In Proc. 36th Int. Conference on Neural Information Processing Systems 24824–24837 (Curran Associates, 2022).
Mohamed, A., Rashid, M. E. & Shaalan, K. In-Context Learning in Large Language Models (LLMs): mechanisms, capabilities, and implications for advanced knowledge representation and reasoning. IEEE Access 13, 95574–95593 (2025).
Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. (2023).
Martin, A., Krause, R., Jacobs, A., Chilton, J. & Amsalem, D. The mental status exam through video-clips of simulated psychiatric patients: an online educational resource. Acad. Psychiatry 44, 179–183 (2020).
Blaabjerg, E. S. et al. Variability between psychiatrists on domains of the mental status examination. Nord. J. Psychiatry 74, 287–292 (2020).
Bhaskara, S. M. Setting Benchmarks and Determining Psychiatric Workloads in Community Mental Health Programs. Psychiatr. Serv. 50, 695–696 (1999).
Kumar, D., Jaipurkar, R., Shekhar, A., Sikri, G. & Srinivas, V. Item analysis of multiple choice questions: A quality assurance test for an assessment tool. Med. J. Armed Forces India 77, S85–S89 (2021).
