SAFETYLIT WEEKLY UPDATE

We compile citations and summaries of about 400 new articles every week.
RSS Feed

HELP: Tutorials | FAQ
CONTACT US: Contact info

Search Results

Journal Article

Citation

Imel ZE, Pace B, Pendergraft B, Pruett J, Tanana M, Soma CS, Comtois KA, Atkins DC. Psychiatr. Serv. 2024; ePub(ePub): ePub.

Copyright

(Copyright © 2024, American Psychiatric Association)

DOI

10.1176/appi.ps.20230648

PMID

39026467

Abstract

OBJECTIVE: Counselor assessment of suicide risk is one key component of crisis counseling, and standards require risk assessment in every crisis counseling conversation. Efforts to increase risk assessment frequency are limited by quality improvement tools that rely on human evaluation of conversations, which is labor intensive, slow, and impossible to scale. Advances in machine learning (ML) have made possible the development of tools that can automatically and immediately detect the presence of risk assessment in crisis counseling conversations.

METHODS: To train models, a coding team labeled every statement in 476 crisis counseling calls (193,257 statements) for a core element of risk assessment. The authors then fine-tuned a transformer-based ML model with the labeled data, utilizing separate training, validation, and test data sets.

RESULTS: Generally, the evaluated ML model was highly consistent with human raters. For detecting any risk assessment, ML model agreement with human ratings was 98% of human interrater agreement. Across specific labels, average F1 (the harmonic mean of precision and recall) was 0.86 at the call level and 0.66 at the statement level and often varied as a result of a low base rate for some risk labels.

CONCLUSIONS: ML models can reliably detect the presence of suicide risk assessment in crisis counseling conversations, presenting an opportunity to scale quality improvement efforts.


Language: en

Keywords

suicide; machine learning; AI; crisis counseling; quality assurance

NEW SEARCH


All SafetyLit records are available for automatic download to Zotero & Mendeley
Print