A Feasibility Study Using a Machine Learning Suicide Risk Prediction Model Based on Open-Ended Interview Language in Adolescent Therapy Sessions

Overview

Journal Int J Environ Res Public Health

Publisher MDPI

Specialties Environmental Health
Public Health

Date 2020 Nov 10

PMID 33167554

Citations 12

Authors

Joshua Cohen

Jennifer Wright-Berryman

Lesley Rohlfs

Donald Wright

Marci Campbell

Debbie Gingrich

Daniel Santel

John Pestian

Affiliations

Soon will be listed here.

Abstract

Background: As adolescent suicide rates continue to rise, innovation in risk identification is warranted. Machine learning can identify suicidal individuals based on their language samples. This feasibility pilot was conducted to explore this technology's use in adolescent therapy sessions and assess machine learning model performance.

Method: Natural language processing machine learning models to identify level of suicide risk using a smartphone app were tested in outpatient therapy sessions. Data collection included language samples, depression and suicidality standardized scale scores, and therapist impression of the client's mental state. Previously developed models were used to predict suicidal risk.

Results: 267 interviews were collected from 60 students in eight schools by ten therapists, with 29 students indicating suicide or self-harm risk. During external validation, models were trained on suicidal speech samples collected from two separate studies. We found that support vector machines (AUC: 0.75; 95% CI: 0.69-0.81) and logistic regression (AUC: 0.76; 95% CI: 0.70-0.82) lead to good discriminative ability, with an extreme gradient boosting model performing the best (AUC: 0.78; 95% CI: 0.72-0.84).

Conclusion: Voice collection technology and associated procedures can be integrated into mental health therapists' workflow. Collected language samples could be classified with good discrimination using machine learning methods.

Citing Articles

Harnessing digital health data for suicide prevention and care: A rapid review.

Bennett-Poynter L, Kundurthi S, Besa R, Joyce D, Kormilitzin A, Shen N Digit Health. 2025; 11:20552076241308615.

PMID: 39996066 PMC: 11848906. DOI: 10.1177/20552076241308615.

Analysis and evaluation of explainable artificial intelligence on suicide risk assessment.

Tang H, Miri Rekavandi A, Rooprai D, Dwivedi G, Sanfilippo F, Boussaid F Sci Rep. 2024; 14(1):6163.

PMID: 38485985 PMC: 10940617. DOI: 10.1038/s41598-024-53426-0.

A multimodal dialog approach to mental state characterization in clinically depressed, anxious, and suicidal populations.

Cohen J, Richter V, Neumann M, Black D, Haq A, Wright-Berryman J Front Psychol. 2023; 14:1135469.

PMID: 37767217 PMC: 10520716. DOI: 10.3389/fpsyg.2023.1135469.

Virtually screening adults for depression, anxiety, and suicide risk using machine learning and language from an open-ended interview.

Wright-Berryman J, Cohen J, Haq A, Black D, Pease J Front Psychiatry. 2023; 14:1143175.

PMID: 37377466 PMC: 10291825. DOI: 10.3389/fpsyt.2023.1143175.

An efficient landmark model for prediction of suicide attempts in multiple clinical settings.

Sheu Y, Sun J, Lee H, Castro V, Barak-Corren Y, Song E Psychiatry Res. 2023; 323:115175.

PMID: 37003169 PMC: 10267893. DOI: 10.1016/j.psychres.2023.115175.

References

Barak-Corren Y, Castro V, Nock M, Mandl K, Madsen E, Seiger A . Validation of an Electronic Health Record-Based Suicide Risk Prediction Modeling Approach Across Multiple Health Care Systems. JAMA Netw Open. 2020; 3(3):e201262. PMC: 11136522. DOI: 10.1001/jamanetworkopen.2020.1262. View

Simon G, Rutter C, Peterson D, Oliver M, Whiteside U, Operskalski B . Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death?. Psychiatr Serv. 2013; 64(12):1195-202. PMC: 4086215. DOI: 10.1176/appi.ps.201200587. View

Lee Y, Kim S, Gwak A, Kim S, Kang S, Na K . Decreased regional gray matter volume in suicide attempters compared to suicide non-attempters with major depressive disorders. Compr Psychiatry. 2016; 67:59-65. DOI: 10.1016/j.comppsych.2016.02.013. View

Simonyan K, Ackermann H, Chang E, Greenlee J . New Developments in Understanding the Complexity of Human Speech Production. J Neurosci. 2016; 36(45):11440-11448. PMC: 5125211. DOI: 10.1523/JNEUROSCI.2424-16.2016. View

Harris P, Taylor R, Minor B, Elliott V, Fernandez M, ONeal L . The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019; 95:103208. PMC: 7254481. DOI: 10.1016/j.jbi.2019.103208. View

Pestian J . A conversation with Edwin Shneidman. Suicide Life Threat Behav. 2010; 40(5):516-23. DOI: 10.1521/suli.2010.40.5.516. View

Rice M, Harris G . Comparing effect sizes in follow-up studies: ROC Area, Cohen's d, and r. Law Hum Behav. 2005; 29(5):615-20. DOI: 10.1007/s10979-005-6832-7. View

Harris P, Taylor R, Thielke R, Payne J, Gonzalez N, Conde J . Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2008; 42(2):377-81. PMC: 2700030. DOI: 10.1016/j.jbi.2008.08.010. View

Youngstrom E . A primer on receiver operating characteristic analysis and diagnostic efficiency statistics for pediatric psychology: we are ready to ROC. J Pediatr Psychol. 2013; 39(2):204-21. PMC: 3936258. DOI: 10.1093/jpepsy/jst062. View

10.

Posner K, Brown G, Stanley B, Brent D, Yershova K, Oquendo M . The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011; 168(12):1266-77. PMC: 3893686. DOI: 10.1176/appi.ajp.2011.10111704. View

11.

Wolff R, Moons K, Riley R, Whiting P, Westwood M, Collins G . PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med. 2019; 170(1):51-58. DOI: 10.7326/M18-1376. View

12.

Torous J, Larsen M, Depp C, Cosco T, Barnett I, Nock M . Smartphones, Sensors, and Machine Learning to Advance Real-Time Prediction and Interventions for Suicide Prevention: a Review of Current Progress and Next Steps. Curr Psychiatry Rep. 2018; 20(7):51. DOI: 10.1007/s11920-018-0914-y. View

13.

Na P, Yaramala S, Kim J, Kim H, Goes F, Zandi P . The PHQ-9 Item 9 based screening for suicide risk: a validation study of the Patient Health Questionnaire (PHQ)-9 Item 9 with the Columbia Suicide Severity Rating Scale (C-SSRS). J Affect Disord. 2018; 232:34-40. DOI: 10.1016/j.jad.2018.02.045. View

14.

Louzon S, Bossarte R, McCarthy J, Katz I . Does Suicidal Ideation as Measured by the PHQ-9 Predict Suicide Among VA Patients?. Psychiatr Serv. 2016; 67(5):517-22. DOI: 10.1176/appi.ps.201500149. View

15.

Viguera A, Milano N, Laurel R, Thompson N, Griffith S, Baldessarini R . Comparison of Electronic Screening for Suicidal Risk With the Patient Health Questionnaire Item 9 and the Columbia Suicide Severity Rating Scale in an Outpatient Psychiatric Clinic. Psychosomatics. 2015; 56(5):460-9. DOI: 10.1016/j.psym.2015.04.005. View

16.

Pestian J, Grupp-Phelan J, Cohen K, Meyers G, Richey L, Matykiewicz P . A Controlled Trial Using Natural Language Processing to Examine the Language of Suicidal Adolescents in the Emergency Department. Suicide Life Threat Behav. 2015; 46(2):154-9. DOI: 10.1111/sltb.12180. View

17.

Virtanen P, Gommers R, Oliphant T, Haberland M, Reddy T, Cournapeau D . SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods. 2020; 17(3):261-272. PMC: 7056644. DOI: 10.1038/s41592-019-0686-2. View

18.

Cummins N, Baird A, Schuller B . Speech analysis for health: Current state-of-the-art and the increasing impact of deep learning. Methods. 2018; 151:41-54. DOI: 10.1016/j.ymeth.2018.07.007. View

19.

Rossom R, Coleman K, Ahmedani B, Beck A, Johnson E, Oliver M . Suicidal ideation reported on the PHQ9 and risk of suicidal behavior across age groups. J Affect Disord. 2017; 215:77-84. PMC: 5412508. DOI: 10.1016/j.jad.2017.03.037. View

20.

Franklin J, Ribeiro J, Fox K, Bentley K, Kleiman E, Huang X . Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychol Bull. 2016; 143(2):187-232. DOI: 10.1037/bul0000084. View