Understanding the Role of Emotion in Decision Making Process: Using Machine Learning to Analyze Physiological Responses to Visual, Auditory, and Combined Stimulation

Overview

Journal Front Hum Neurosci

Specialty Neurology

Date 2024 Jan 23

PMID 38259333

Authors

Edoardo Maria Polo

Andrea Farabbi

Maximiliano Mollura

Luca Mainardi

Riccardo Barbieri

Affiliations

Soon will be listed here.

Abstract

Emotions significantly shape decision-making, and targeted emotional elicitations represent an important factor in neuromarketing, where they impact advertising effectiveness by capturing potential customers' attention intricately associated with emotional triggers. Analyzing biometric parameters after stimulus exposure may help in understanding emotional states. This study investigates autonomic and central nervous system responses to emotional stimuli, including images, auditory cues, and their combination while recording physiological signals, namely the electrocardiogram, blood volume pulse, galvanic skin response, pupillometry, respiration, and the electroencephalogram. The primary goal of the proposed analysis is to compare emotional stimulation methods and to identify the most effective approach for distinct physiological patterns. A novel feature selection technique is applied to further optimize the separation of four emotional states. Basic machine learning approaches are used in order to discern emotions as elicited by different kinds of stimulation. Electroencephalographic signals, Galvanic skin response and cardio-respiratory coupling-derived features provided the most significant features in distinguishing the four emotional states. Further findings highlight how auditory stimuli play a crucial role in creating distinct physiological patterns that enhance classification within a four-class problem. When combining all three types of stimulation, a validation accuracy of 49% was achieved. The sound-only and the image-only phases resulted in 52% and 44% accuracy respectively, whereas the combined stimulation of images and sounds led to 51% accuracy. Isolated visual stimuli yield less distinct patterns, necessitating more signals for relatively inferior performance compared to other types of stimuli. This surprising significance arises from limited auditory exploration in emotional recognition literature, particularly contrasted with the pleathora of studies performed using visual stimulation. In marketing, auditory components might hold a more relevant potential to significantly influence consumer choices.

References

Kolodyazhniy V, Kreibig S, Gross J, Roth W, Wilhelm F . An affective computing approach to physiological emotion specificity: toward subject-independent and stimulus-independent classification of film-induced emotions. Psychophysiology. 2011; 48(7):908-22. DOI: 10.1111/j.1469-8986.2010.01170.x. View

Gomez C, Vazquez M, Vaquero E, Lopez-Mendoza D, Cardoso M . Frequency analysis of the EEG during spatial selective attention. Int J Neurosci. 1998; 95(1-2):17-32. DOI: 10.3109/00207459809000646. View

Farabbi A, Polo E, Barbieri R, Mainardi L . Comparison of different emotion stimulation modalities: an EEG signal analysis. Annu Int Conf IEEE Eng Med Biol Soc. 2022; 2022:3710-3713. DOI: 10.1109/EMBC48229.2022.9871725. View

Laeng B, Endestad T . Bright illusions reduce the eye's pupil. Proc Natl Acad Sci U S A. 2012; 109(6):2162-7. PMC: 3277565. DOI: 10.1073/pnas.1118298109. View

Lee T, Girolami M, Sejnowski T . Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural Comput. 1999; 11(2):417-41. DOI: 10.1162/089976699300016719. View

Wang C, Baird T, Huang J, Coutinho J, Brien D, Munoz D . Arousal Effects on Pupil Size, Heart Rate, and Skin Conductance in an Emotional Face Task. Front Neurol. 2018; 9:1029. PMC: 6287044. DOI: 10.3389/fneur.2018.01029. View

Vences N, Diaz-Campo J, Rosales D . Neuromarketing as an Emotional Connection Tool Between Organizations and Audiences in Social Networks. A Theoretical Review. Front Psychol. 2020; 11:1787. PMC: 7396554. DOI: 10.3389/fpsyg.2020.01787. View

Stanley G, Poolla K, Siegel R . Threshold modeling of autonomic control of heart rate variability. IEEE Trans Biomed Eng. 2000; 47(9):1147-53. DOI: 10.1109/10.867918. View

Sassi R, Cerutti S, Lombardi F, Malik M, Huikuri H, Peng C . Advances in heart rate variability signal analysis: joint position statement by the e-Cardiology ESC Working Group and the European Heart Rhythm Association co-endorsed by the Asia Pacific Heart Rhythm Society. Europace. 2015; 17(9):1341-53. DOI: 10.1093/europace/euv015. View

10.

Frantzidis C, Bratsas C, Klados M, Konstantinidis E, Lithari C, Vivas A . On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications. IEEE Trans Inf Technol Biomed. 2010; 14(2):309-18. DOI: 10.1109/TITB.2009.2038481. View

11.

Zhuang N, Zeng Y, Tong L, Zhang C, Zhang H, Yan B . Emotion Recognition from EEG Signals Using Multidimensional Information in EMD Domain. Biomed Res Int. 2017; 2017:8317357. PMC: 5576397. DOI: 10.1155/2017/8317357. View

12.

Polo E, Farabbi A, Mollura M, Barbieri R, Paglialonga A, Mainardi L . Analysis of the skin conductance and pupil signals for evaluation of emotional elicitation by images and sounds. Annu Int Conf IEEE Eng Med Biol Soc. 2022; 2022:1968-1971. DOI: 10.1109/EMBC48229.2022.9871493. View

13.

Rahman M, Sarkar A, Hossain M, Hossain M, Islam M, Hossain M . Recognition of human emotions using EEG signals: A review. Comput Biol Med. 2021; 136:104696. DOI: 10.1016/j.compbiomed.2021.104696. View

14.

Polo E, Mollura M, Lenatti M, Zanet M, Paglialonga A, Barbieri R . Emotion recognition from multimodal physiological measurements based on an interpretable feature selection method. Annu Int Conf IEEE Eng Med Biol Soc. 2021; 2021:989-992. DOI: 10.1109/EMBC46164.2021.9631019. View

15.

Kim J, Andre E . Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell. 2008; 30(12):2067-83. DOI: 10.1109/TPAMI.2008.26. View

16.

Pong M, Fuchs A . Characteristics of the pupillary light reflex in the macaque monkey: discharge patterns of pretectal neurons. J Neurophysiol. 2000; 84(2):964-74. DOI: 10.1152/jn.2000.84.2.964. View

17.

Posner J, Russell J, Peterson B . The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev Psychopathol. 2005; 17(3):715-34. PMC: 2367156. DOI: 10.1017/S0954579405050340. View

18.

Pion-Tonachini L, Kreutz-Delgado K, Makeig S . ICLabel: An automated electroencephalographic independent component classifier, dataset, and website. Neuroimage. 2019; 198:181-197. PMC: 6592775. DOI: 10.1016/j.neuroimage.2019.05.026. View

19.

Coelli S, Barbieri R, Reni G, Zucca C, Bianchi A . EEG indices correlate with sustained attention performance in patients affected by diffuse axonal injury. Med Biol Eng Comput. 2017; 56(6):991-1001. DOI: 10.1007/s11517-017-1744-5. View

20.

Gerdes A, Wieser M, Alpers G . Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains. Front Psychol. 2014; 5:1351. PMC: 4248815. DOI: 10.3389/fpsyg.2014.01351. View