Extreme Image Transformations Affect Humans and Machines Differently

Overview

Journal Biol Cybern

Specialties Neurology
Physiology

Date 2023 Jun 13

PMID 37310489

Authors

Girik Malik

Dakarai Crowder

Ennio Mingolla

Affiliations

Soon will be listed here.

Abstract

Some recent artificial neural networks (ANNs) claim to model aspects of primate neural and human performance data. Their success in object recognition is, however, dependent on exploiting low-level features for solving visual tasks in a way that humans do not. As a result, out-of-distribution or adversarial input is often challenging for ANNs. Humans instead learn abstract patterns and are mostly unaffected by many extreme image distortions. We introduce a set of novel image transforms inspired by neurophysiological findings and evaluate humans and ANNs on an object recognition task. We show that machines perform better than humans for certain transforms and struggle to perform at par with humans on others that are easy for humans. We quantify the differences in accuracy for humans and machines and find a ranking of difficulty for our transforms for human data. We also suggest how certain characteristics of human visual processing can be adapted to improve the performance of ANNs for our difficult-for-machines transforms.

Citing Articles

What can computer vision learn from visual neuroscience? Introduction to the special issue.

Chen K, Kashyap H, Krichmar J, Li X Biol Cybern. 2023; 117(4-5):297-298.

PMID: 37812267 DOI: 10.1007/s00422-023-00977-6.

References

Tanaka K . Mechanisms of visual object recognition: monkey and human studies. Curr Opin Neurobiol. 1997; 7(4):523-9. DOI: 10.1016/s0959-4388(97)80032-3. View

Tarr M, Bulthoff H . Image-based object recognition in man, monkey and machine. Cognition. 1998; 67(1-2):1-20. DOI: 10.1016/s0010-0277(98)00026-2. View

Hubel D, Wiesel T . Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J Physiol. 1962; 160:106-54. PMC: 1359523. DOI: 10.1113/jphysiol.1962.sp006837. View

Ekstrom A, Isham E . Human spatial navigation: Representations across dimensions and scales. Curr Opin Behav Sci. 2017; 17:84-89. PMC: 5678987. DOI: 10.1016/j.cobeha.2017.06.005. View

Koenderink J . The structure of images. Biol Cybern. 1984; 50(5):363-70. DOI: 10.1007/BF00336961. View

Hochstein S, Ahissar M . View from the top: hierarchies and reverse hierarchies in the visual system. Neuron. 2002; 36(5):791-804. DOI: 10.1016/s0896-6273(02)01091-7. View

Hubel D, Wiesel T . Shape and arrangement of columns in cat's striate cortex. J Physiol. 1963; 165:559-68. PMC: 1359325. DOI: 10.1113/jphysiol.1963.sp007079. View

Frank M, Cebrian M, Pickard G, Rahwan I . Validating Bayesian truth serum in large-scale online human experiments. PLoS One. 2017; 12(5):e0177385. PMC: 5426759. DOI: 10.1371/journal.pone.0177385. View

Martin A . GRAPES-Grounding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain. Psychon Bull Rev. 2015; 23(4):979-90. PMC: 5111803. DOI: 10.3758/s13423-015-0842-3. View

10.

Grill-Spector K, Kourtzi Z, Kanwisher N . The lateral occipital complex and its role in object recognition. Vision Res. 2001; 41(10-11):1409-22. DOI: 10.1016/s0042-6989(01)00073-6. View

11.

Dong Q, Wang H, Hu Z . Commentary: Using goal-driven deep learning models to understand sensory cortex. Front Comput Neurosci. 2018; 12:4. PMC: 5780340. DOI: 10.3389/fncom.2018.00004. View

12.

Zmigrod S, Hommel B . Feature integration across multimodal perception and action: a review. Multisens Res. 2013; 26(1-2):143-57. DOI: 10.1163/22134808-00002390. View

13.

Allison T, McCarthy G, Nobre A, Puce A, Belger A . Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cereb Cortex. 1994; 4(5):544-54. DOI: 10.1093/cercor/4.5.544. View

14.

Yamins D, DiCarlo J . Using goal-driven deep learning models to understand sensory cortex. Nat Neurosci. 2016; 19(3):356-65. DOI: 10.1038/nn.4244. View

15.

Ferrari V, Fevrier L, Jurie F, Schmid C . Groups of adjacent contour segments for object detection. IEEE Trans Pattern Anal Mach Intell. 2007; 30(1):36-51. DOI: 10.1109/TPAMI.2007.1144. View

16.

Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Susstrunk S . SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell. 2012; 34(11):2274-82. DOI: 10.1109/TPAMI.2012.120. View

17.

Koenderink J . The structure of images: 1984-2021. Biol Cybern. 2021; 115(2):117-120. DOI: 10.1007/s00422-021-00870-0. View

18.

Zhang M, Tseng C, Kreiman G . Putting visual object recognition in context. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2021; 2020:12982-12991. PMC: 8459751. DOI: 10.1109/CVPR42600.2020.01300. View

19.

Elsik C, Tayal A, Diesh C, Unni D, Emery M, Nguyen H . Hymenoptera Genome Database: integrating genome annotations in HymenopteraMine. Nucleic Acids Res. 2015; 44(D1):D793-800. PMC: 4702858. DOI: 10.1093/nar/gkv1208. View

20.

Wilson M, Bower J . A Computer Simulation of Oscillatory Behavior in Primary Visual Cortex. Neural Comput. 2019; 3(4):498-509. DOI: 10.1162/neco.1991.3.4.498. View