Automating the Analysis of Facial Emotion Expression Dynamics: A Computational Framework and Application in Psychotic Disorders

Overview

Journal Proc Natl Acad Sci U S A

Specialty Science

Date 2024 Mar 26

PMID 38530896

Authors

Nathan T Hall

Michael N Hallquist

Elizabeth A Martin

Wenxuan Lian

Katherine G Jonas

Roman Kotov

Affiliations

Soon will be listed here.

Abstract

Facial emotion expressions play a central role in interpersonal interactions; these displays are used to predict and influence the behavior of others. Despite their importance, quantifying and analyzing the dynamics of brief facial emotion expressions remains an understudied methodological challenge. Here, we present a method that leverages machine learning and network modeling to assess the dynamics of facial expressions. Using video recordings of clinical interviews, we demonstrate the utility of this approach in a sample of 96 people diagnosed with psychotic disorders and 116 never-psychotic adults. Participants diagnosed with schizophrenia tended to move from neutral expressions to uncommon expressions (e.g., fear, surprise), whereas participants diagnosed with other psychoses (e.g., mood disorders with psychosis) moved toward expressions of sadness. This method has broad applications to the study of normal and altered expressions of emotion and can be integrated with telemedicine to improve psychiatric assessment and treatment.

Citing Articles

Automating the analysis of facial emotion expression dynamics: A computational framework and application in psychotic disorders.

Hall N, Hallquist M, Martin E, Lian W, Jonas K, Kotov R Proc Natl Acad Sci U S A. 2024; 121(14):e2313665121.

PMID: 38530896 PMC: 10998559. DOI: 10.1073/pnas.2313665121.

References

Gupta T, Haase C, Strauss G, Cohen A, Ricard J, Mittal V . Alterations in facial expressions of emotion: Determining the promise of ultrathin slicing approaches and comparing human and automated coding methods in psychosis risk. Emotion. 2020; 22(4):714-724. PMC: 7759595. DOI: 10.1037/emo0000819. View

Bringmann L, Lemmens L, Huibers M, Borsboom D, Tuerlinckx F . Revealing the dynamic network structure of the Beck Depression Inventory-II. Psychol Med. 2014; 45(4):747-57. DOI: 10.1017/S0033291714001809. View

Bromet E, Kotov R, Fochtmann L, Carlson G, Tanenberg-Karant M, Ruggero C . Diagnostic shifts during the decade following first admission for psychosis. Am J Psychiatry. 2011; 168(11):1186-94. PMC: 3589618. DOI: 10.1176/appi.ajp.2011.11010048. View

Wolf K . Measuring facial expression of emotion. Dialogues Clin Neurosci. 2016; 17(4):457-62. PMC: 4734883. View

Cortez C, Mansour O, Qato D, Stafford R, Alexander G . Changes in Short-term, Long-term, and Preventive Care Delivery in US Office-Based and Telemedicine Visits During the COVID-19 Pandemic. JAMA Health Forum. 2022; 2(7):e211529. PMC: 8796900. DOI: 10.1001/jamahealthforum.2021.1529. View

Gruenbaum B, Sandhu M, Bertasi R, Bertasi T, Schonwald A, Kurup A . Absence seizures and their relationship to depression and anxiety: Evidence for bidirectionality. Epilepsia. 2021; 62(5):1041-1056. PMC: 8443164. DOI: 10.1111/epi.16862. View

Wolf K, Mass R, Kiefer F, Wiedemann K, Naber D . Characterization of the facial expression of emotions in schizophrenia patients: preliminary findings with a new electromyography method. Can J Psychiatry. 2006; 51(6):335-41. DOI: 10.1177/070674370605100601. View

Bologna M, Fabbrini G, Marsili L, Defazio G, Thompson P, Berardelli A . Facial bradykinesia. J Neurol Neurosurg Psychiatry. 2012; 84(6):681-5. DOI: 10.1136/jnnp-2012-303993. View

Anusa A, Thavarajah R, Nayak D, Joshua E, Rao U, Ranganathan K . A Study on Drug-Induced Tardive Dyskinesia: Orofacial Musculature Involvement and Patient's Awareness. J Orofac Sci. 2019; 10(2):86-95. PMC: 6333421. View

10.

Roche E, Creed L, MacMahon D, Brennan D, Clarke M . The Epidemiology and Associated Phenomenology of Formal Thought Disorder: A Systematic Review. Schizophr Bull. 2014; 41(4):951-62. PMC: 4466171. DOI: 10.1093/schbul/sbu129. View

11.

Ekman P, Friesen W . Nonverbal leakage and clues to deception. Psychiatry. 1969; 32(1):88-106. DOI: 10.1080/00332747.1969.11023575. View

12.

Abbas A, Hansen B, Koesmahargyo V, Yadav V, Rosenfield P, Patil O . Facial and Vocal Markers of Schizophrenia Measured Using Remote Smartphone Assessments: Observational Study. JMIR Form Res. 2022; 6(1):e26276. PMC: 8817208. DOI: 10.2196/26276. View

13.

Riehle M, Lincoln T . Social consequences of subclinical negative symptoms: An EMG study of facial expressions within a social interaction. J Behav Ther Exp Psychiatry. 2017; 55:90-98. DOI: 10.1016/j.jbtep.2017.01.003. View

14.

Velthorst E, Fett A, Reichenberg A, Perlman G, van Os J, Bromet E . The 20-Year Longitudinal Trajectories of Social Functioning in Individuals With Psychotic Disorders. Am J Psychiatry. 2016; 174(11):1075-1085. PMC: 5474222. DOI: 10.1176/appi.ajp.2016.15111419. View

15.

Cowan T, Strauss G, Raugh I, Le T, Cohen A . How do social factors relate to blunted facial affect in schizophrenia? A digital phenotyping study using ambulatory video recordings. J Psychiatr Res. 2022; 150:96-104. PMC: 10036138. DOI: 10.1016/j.jpsychires.2022.03.024. View

16.

Kotov R, Foti D, Li K, Bromet E, Hajcak G, Ruggero C . Validating dimensions of psychosis symptomatology: Neural correlates and 20-year outcomes. J Abnorm Psychol. 2016; 125(8):1103-1119. PMC: 5119925. DOI: 10.1037/abn0000188. View

17.

Sanchez A, Lavaysse L, Starr J, Gard D . Daily life evidence of environment-incongruent emotion in schizophrenia. Psychiatry Res. 2014; 220(1-2):89-95. PMC: 4252781. DOI: 10.1016/j.psychres.2014.07.041. View

18.

Kuppens P, Verduyn P . Emotion dynamics. Curr Opin Psychol. 2017; 17:22-26. DOI: 10.1016/j.copsyc.2017.06.004. View

19.

Davies H, Wolz I, Leppanen J, Fernandez-Aranda F, Schmidt U, Tchanturia K . Facial expression to emotional stimuli in non-psychotic disorders: A systematic review and meta-analysis. Neurosci Biobehav Rev. 2016; 64:252-71. DOI: 10.1016/j.neubiorev.2016.02.015. View

20.

Frumkin M, Rodebaugh T . The role of affect in chronic pain: A systematic review of within-person symptom dynamics. J Psychosom Res. 2021; 147:110527. PMC: 9009535. DOI: 10.1016/j.jpsychores.2021.110527. View