Augmented Reality Powers a Cognitive Assistant for the Blind

Overview

Journal Elife

Specialty Biology

Date 2018 Nov 28

PMID 30479270

Citations 12

Authors

Yang Liu

Noelle Rb Stiles

Markus Meister

Affiliations

Soon will be listed here.

Abstract

To restore vision for the blind, several prosthetic approaches have been explored that convey raw images to the brain. So far, these schemes all suffer from a lack of bandwidth. An alternate approach would restore vision at the cognitive level, bypassing the need to convey sensory data. A wearable computer captures video and other data, extracts important scene knowledge, and conveys that to the user in compact form. Here, we implement an intuitive user interface for such a device using augmented reality: each object in the environment has a voice and communicates with the user on command. With minimal training, this system supports many aspects of visual cognition: obstacle avoidance, scene understanding, formation and recall of spatial memories, navigation. Blind subjects can traverse an unfamiliar multi-story building on their first attempt. To spur further development in this domain, we developed an open-source environment for standardized benchmarking of visual assistive devices.

Citing Articles

Human-AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era.

Yu R, Lee S, Xie J, Billah S, Carroll J Future Internet. 2025; 16(7).

PMID: 40051468 PMC: 11884418. DOI: 10.3390/fi16070254.

Opportunities for Human-AI Collaboration in Remote Sighted Assistance.

Lee S, Yu R, Xie J, Billah S, Carroll J IUI. 2025; 2022():63-78.

PMID: 39850496 PMC: 11755352. DOI: 10.1145/3490099.3511113.

Helping Helpers: Supporting Volunteers in Remote Sighted Assistance with Augmented Reality Maps.

Xie J, Yu R, Lee S, Lyu Y, Billah S, Carroll J DIS (Des Interact Syst Conf). 2025; 2022:881-897.

PMID: 39807162 PMC: 11727196. DOI: 10.1145/3532106.3533560.

The unbearable slowness of being: Why do we live at 10 bits/s?.

Zheng J, Meister M Neuron. 2024; 113(2):192-204.

PMID: 39694032 PMC: 11758279. DOI: 10.1016/j.neuron.2024.11.008.

A brief reference to AI-driven audible reality (AuRa) in open world: potential, applications, and evaluation.

Ates O, Pandey G, Gousiopoulos A, Soldatos T Front Artif Intell. 2024; 7:1424371.

PMID: 39525498 PMC: 11543578. DOI: 10.3389/frai.2024.1424371.

References

Proulx M, Gwinnutt J, DellErba S, Levy-Tzedek S, de Sousa A, Brown D . Other ways of seeing: From behavior to neural mechanisms in the online "visual" control of action with sensory substitution. Restor Neurol Neurosci. 2015; 34(1):29-44. PMC: 4927905. DOI: 10.3233/RNN-150541. View

Capelle C, Trullemans C, Arno P, Veraart C . A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Trans Biomed Eng. 1998; 45(10):1279-93. DOI: 10.1109/10.720206. View

Striem-Amit E, Guendelman M, Amedi A . 'Visual' acuity of the congenitally blind using visual-to-auditory sensory substitution. PLoS One. 2012; 7(3):e33136. PMC: 3306374. DOI: 10.1371/journal.pone.0033136. View

Berens P, Freeman J, Deneux T, Chenkov N, McColgan T, Speiser A . Community-based benchmarking improves spike rate inference from two-photon calcium imaging data. PLoS Comput Biol. 2018; 14(5):e1006157. PMC: 5997358. DOI: 10.1371/journal.pcbi.1006157. View

Loomis J, Marston J, Golledge R, Klatzky R . Personal Guidance System for People with Visual Impairment: A Comparison of Spatial Displays for Route Guidance. J Vis Impair Blind. 2010; 99(4):219-232. PMC: 2801896. View

Stingl K, Schippert R, Bartz-Schmidt K, Besch D, Cottriall C, Edwards T . Interim Results of a Multicenter Trial with the New Electronic Subretinal Implant Alpha AMS in 15 Patients Blind from Inherited Retinal Degenerations. Front Neurosci. 2017; 11:445. PMC: 5572402. DOI: 10.3389/fnins.2017.00445. View

Stronks H, Mitchell E, Nau A, Barnes N . Visual task performance in the blind with the BrainPort V100 Vision Aid. Expert Rev Med Devices. 2016; 13(10):919-931. DOI: 10.1080/17434440.2016.1237287. View

Auvray M, Hanneton S, ORegan J . Learning to perceive with a visuo-auditory substitution system: localisation and object recognition with 'the vOICe'. Perception. 2007; 36(3):416-30. DOI: 10.1068/p5631. View

Stingl K, Zrenner E . Electronic approaches to restitute vision in patients with neurodegenerative diseases of the retina. Ophthalmic Res. 2013; 50(4):215-20. DOI: 10.1159/000354424. View

10.

Meijer P . An experimental system for auditory image representations. IEEE Trans Biomed Eng. 1992; 39(2):112-21. DOI: 10.1109/10.121642. View

11.

Bourne R, Flaxman S, Braithwaite T, Cicinelli M, Das A, Jonas J . Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. Lancet Glob Health. 2017; 5(9):e888-e897. DOI: 10.1016/S2214-109X(17)30293-0. View

12.

Luo Y, da Cruz L . The Argus(®) II Retinal Prosthesis System. Prog Retin Eye Res. 2015; 50:89-107. DOI: 10.1016/j.preteyeres.2015.09.003. View

13.

Liu Y, Stiles N, Meister M . Augmented reality powers a cognitive assistant for the blind. Elife. 2018; 7. PMC: 6257813. DOI: 10.7554/eLife.37841. View

14.

Weiland J, Humayun M . Retinal prosthesis. IEEE Trans Biomed Eng. 2014; 61(5):1412-24. PMC: 4356127. DOI: 10.1109/TBME.2014.2314733. View

15.

McAnally K, Martin R . Sound localization with head movement: implications for 3-d audio displays. Front Neurosci. 2014; 8:210. PMC: 4130110. DOI: 10.3389/fnins.2014.00210. View

16.

Haigh A, Brown D, Meijer P, Proulx M . How well do you see what you hear? The acuity of visual-to-auditory sensory substitution. Front Psychol. 2013; 4:330. PMC: 3684791. DOI: 10.3389/fpsyg.2013.00330. View

17.

Scholl H, Strauss R, Singh M, Dalkara D, Roska B, Picaud S . Emerging therapies for inherited retinal degeneration. Sci Transl Med. 2016; 8(368):368rv6. DOI: 10.1126/scitranslmed.aaf2838. View

18.

Adebiyi A, Sorrentino P, Bohlool S, Zhang C, Arditti M, Goodrich G . Assessment of feedback modalities for wearable visual aids in blind mobility. PLoS One. 2017; 12(2):e0170531. PMC: 5300186. DOI: 10.1371/journal.pone.0170531. View

19.

Wenzel E, Arruda M, Kistler D, Wightman F . Localization using nonindividualized head-related transfer functions. J Acoust Soc Am. 1993; 94(1):111-23. DOI: 10.1121/1.407089. View

20.

Dobelle W, Mladejovsky M, Girvin J . Artifical vision for the blind: electrical stimulation of visual cortex offers hope for a functional prosthesis. Science. 1974; 183(4123):440-4. DOI: 10.1126/science.183.4123.440. View