Synergy, Redundancy, and Multivariate Information Measures: an Experimentalist's Perspective

Overview

Journal J Comput Neurosci

Specialties Biology
Neurology

Date 2013 Jul 4

PMID 23820856

Citations 63

Authors

Nicholas Timme

Wesley Alford

Benjamin Flecker

John M Beggs

Affiliations

Soon will be listed here.

Abstract

Information theory has long been used to quantify interactions between two variables. With the rise of complex systems research, multivariate information measures have been increasingly used to investigate interactions between groups of three or more variables, often with an emphasis on so called synergistic and redundant interactions. While bivariate information measures are commonly agreed upon, the multivariate information measures in use today have been developed by many different groups, and differ in subtle, yet significant ways. Here, we will review these multivariate information measures with special emphasis paid to their relationship to synergy and redundancy, as well as examine the differences between these measures by applying them to several simple model systems. In addition to these systems, we will illustrate the usefulness of the information measures by analyzing neural spiking data from a dissociated culture through early stages of its development. Our aim is that this work will aid other researchers as they seek the best multivariate information measure for their specific research goals and system. Finally, we have made software available online which allows the user to calculate all of the information measures discussed within this paper.

Citing Articles

CT-based intratumoral and peritumoral radiomics nomogram to predict spread through air spaces in lung adenocarcinoma with diameter ≤ 3 cm: A multicenter study.

Su Y, Tao J, Lan X, Liang C, Huang X, Zhang J Eur J Radiol Open. 2025; 14():100630.

PMID: 39850145 PMC: 11754163. DOI: 10.1016/j.ejro.2024.100630.

Quantifying the diverse contributions of hierarchical muscle interactions to motor function.

OReilly D, Shaw W, Hilt P, de Castro Aguiar R, Astill S, Delis I iScience. 2025; 28(1):111613.

PMID: 39834869 PMC: 11742840. DOI: 10.1016/j.isci.2024.111613.

Transcranial ultrasound stimulation effect in the redundant and synergistic networks consistent across macaques.

Gatica M, Atkinson-Clement C, Mediano P, Alkhawashki M, Ross J, Sallet J Netw Neurosci. 2024; 8(4):1032-1050.

PMID: 39735508 PMC: 11674579. DOI: 10.1162/netn_a_00388.

Informeasure: an R/bioconductor package for quantifying nonlinear dependence between variables in biological networks from an information theory perspective.

Pan C, Chen Y BMC Bioinformatics. 2024; 25(1):382.

PMID: 39695935 PMC: 11658358. DOI: 10.1186/s12859-024-05996-z.

Competitive interactions shape brain dynamics and computation across species.

Luppi A, Sanz Perl Y, Vohryzek J, Mediano P, Rosas F, Milisav F bioRxiv. 2024; .

PMID: 39484469 PMC: 11526968. DOI: 10.1101/2024.10.19.619194.

References

Bettencourt L, Gintautas V, Ham M . Identification of functional information subgraphs in complex networks. Phys Rev Lett. 2008; 100(23):238701. DOI: 10.1103/PhysRevLett.100.238701. View

Kamioka H, Maeda E, Jimbo Y, ROBINSON H, Kawana A . Spontaneous periodic synchronized bursting during formation of mature patterns of connections in cortical cultures. Neurosci Lett. 1996; 206(2-3):109-12. DOI: 10.1016/s0304-3940(96)12448-4. View

Vicente R, Wibral M, Lindner M, Pipa G . Transfer entropy--a model-free measure of effective connectivity for the neurosciences. J Comput Neurosci. 2010; 30(1):45-67. PMC: 3040354. DOI: 10.1007/s10827-010-0262-3. View

SCHREIBER . Measuring information transfer. Phys Rev Lett. 2000; 85(2):461-4. DOI: 10.1103/PhysRevLett.85.461. View

Schneidman E, Bialek W, Berry 2nd M . Synergy, redundancy, and independence in population codes. J Neurosci. 2003; 23(37):11539-53. PMC: 6740962. View

Fujisawa S, Amarasingham A, Harrison M, Buzsaki G . Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex. Nat Neurosci. 2008; 11(7):823-33. PMC: 2562676. DOI: 10.1038/nn.2134. View

Tetzlaff C, Okujeni S, Egert U, Worgotter F, Butz M . Self-organized criticality in developing neuronal networks. PLoS Comput Biol. 2010; 6(12):e1001013. PMC: 2996321. DOI: 10.1371/journal.pcbi.1001013. View

Quian Quiroga R, Panzeri S . Extracting information from neuronal populations: information theory and decoding approaches. Nat Rev Neurosci. 2009; 10(3):173-85. DOI: 10.1038/nrn2578. View

Bialek W, Rieke F, de Ruyter van Steveninck R, Warland D . Reading a neural code. Science. 1991; 252(5014):1854-7. DOI: 10.1126/science.2063199. View

10.

Latham P, Nirenberg S . Synergy, redundancy, and independence in population codes, revisited. J Neurosci. 2005; 25(21):5195-206. PMC: 6724819. DOI: 10.1523/JNEUROSCI.5319-04.2005. View

11.

Ohiorhenuan I, Mechler F, Purpura K, Schmid A, Hu Q, Victor J . Sparse coding and high-order correlations in fine-scale cortical networks. Nature. 2010; 466(7306):617-21. PMC: 2912961. DOI: 10.1038/nature09178. View

12.

Borst A, Theunissen F . Information theory and neural coding. Nat Neurosci. 1999; 2(11):947-57. DOI: 10.1038/14731. View

13.

Tang A, Jackson D, Hobbs J, Chen W, Smith J, Patel H . A maximum entropy model applied to spatial and temporal correlations from cortical networks in vitro. J Neurosci. 2008; 28(2):505-18. PMC: 6670549. DOI: 10.1523/JNEUROSCI.3359-07.2008. View

14.

Lungarella M, Sporns O . Mapping information flow in sensorimotor networks. PLoS Comput Biol. 2006; 2(10):e144. PMC: 1626158. DOI: 10.1371/journal.pcbi.0020144. View

15.

Brenner N, Strong S, Koberle R, Bialek W, de Ruyter van Steveninck R . Synergy in a neural code. Neural Comput. 2000; 12(7):1531-52. DOI: 10.1162/089976600300015259. View

16.

Panzeri S, Senatore R, Montemurro M, Petersen R . Correcting for the sampling bias problem in spike train information measures. J Neurophysiol. 2007; 98(3):1064-72. DOI: 10.1152/jn.00559.2007. View

17.

Wang L, Narayan R, Grana G, Shamir M, Sen K . Cortical discrimination of complex natural stimuli: can single neurons match behavior?. J Neurosci. 2007; 27(3):582-9. PMC: 6672806. DOI: 10.1523/JNEUROSCI.3699-06.2007. View

18.

Schneidman E, Still S, Berry 2nd M, Bialek W . Network information and connected correlations. Phys Rev Lett. 2003; 91(23):238701. DOI: 10.1103/PhysRevLett.91.238701. View

19.

Nemenman I, Bialek W, de Ruyter van Steveninck R . Entropy and information in neural spike trains: progress on the sampling problem. Phys Rev E Stat Nonlin Soft Matter Phys. 2004; 69(5 Pt 2):056111. DOI: 10.1103/PhysRevE.69.056111. View

20.

Averbeck B, Latham P, Pouget A . Neural correlations, population coding and computation. Nat Rev Neurosci. 2006; 7(5):358-66. DOI: 10.1038/nrn1888. View