Citation

Cohen J, Wright-Berryman J, Rohlfs L, Trocinski D, Daniel L, Klatt TW. Front. Digit. Health 2022; 4.

Copyright

(Copyright © 2022, Frontiers Media)

DOI

10.3389/fdgth.2022.818705

PMID

unavailable

Abstract

BACKGROUND: Emergency departments (ED) are an important intercept point for identifying suicide risk and connecting patients to care, however, more innovative, person-centered screening tools are needed. Natural language processing (NLP) -based machine learning (ML) techniques have shown promise to assess suicide risk, although whether NLP models perform well in differing geographic regions, at different time periods, or after large-scale events such as the COVID-19 pandemic is unknown.

OBJECTIVE: To evaluate the performance of an NLP/ML suicide risk prediction model on newly collected language from the Southeastern United States using models previously tested on language collected in the Midwestern US.

METHOD: 37 Suicidal and 33 non-suicidal patients from two EDs were interviewed to test a previously developed suicide risk prediction NLP/ML model. Model performance was evaluated with the area under the receiver operating characteristic curve (AUC) and Brier scores.

RESULTS: NLP/ML models performed with an AUC of 0.81 (95% CI: 0.71-0.91) and Brier score of 0.23.

CONCLUSION: The language-based suicide risk model performed with good discrimination when identifying the language of suicidal patients from a different part of the US and at a later time period than when the model was originally developed and trained. Copyright © 2022 Cohen, Wright-Berryman, Rohlfs, Trocinski, Daniel and Klatt.

Language: en

Keywords

United States; adult; human; mental health; suicide; female; male; pilot study; pandemic; interview; validation study; risk assessment; natural language processing; major clinical study; controlled study; emergency ward; confidence interval; Article; receiver operating characteristic; integrated health care system; evaluation study; external validity; internal validity; program acceptability; validation; machine learning; program feasibility; predictive model; coronavirus disease 2019; emergency department (ED); feasibility & acceptability; risk model