Decoding Lip Language Using Triboelectric Sensors with Deep Learning

Overview

Journal Nat Commun

Specialty Biology

Date 2022 Mar 18

PMID 35301313

Authors

Yijia Lu

Han Tian

Jia Cheng

Fei Zhu

Bin Liu

Shanshan Wei

Linhong Ji

Zhong Lin Wang

Affiliations

Soon will be listed here.

Abstract

Lip language is an effective method of voice-off communication in daily life for people with vocal cord lesions and laryngeal and lingual injuries without occupying the hands. Collection and interpretation of lip language is challenging. Here, we propose the concept of a novel lip-language decoding system with self-powered, low-cost, contact and flexible triboelectric sensors and a well-trained dilated recurrent neural network model based on prototype learning. The structural principle and electrical properties of the flexible sensors are measured and analysed. Lip motions for selected vowels, words, phrases, silent speech and voice speech are collected and compared. The prototype learning model reaches a test accuracy of 94.5% in training 20 classes with 100 samples each. The applications, such as identity recognition to unlock a gate, directional control of a toy car and lip-motion to speech conversion, work well and demonstrate great feasibility and potential. Our work presents a promising way to help people lacking a voice live a convenient life with barrier-free communication and boost their happiness, enriches the diversity of lip-language translation systems and will have potential value in many applications.

Citing Articles

Machine learning-assisted wearable sensing systems for speech recognition and interaction.

Liu T, Zhang M, Li Z, Dou H, Zhang W, Yang J Nat Commun. 2025; 16(1):2363.

PMID: 40064879 PMC: 11894117. DOI: 10.1038/s41467-025-57629-5.

Triboelectric sensor gloves for real-time behavior identification and takeover time adjustment in conditionally automated vehicles.

Lu X, Tan H, Zhang H, Wang W, Xie S, Yue T Nat Commun. 2025; 16(1):1080.

PMID: 39870631 PMC: 11772886. DOI: 10.1038/s41467-025-56169-2.

An intelligent hybrid-fabric wristband system enabled by thermal encapsulation for ergonomic human-machine interaction.

Cheng A, Li X, Li D, Chen Z, Cui T, Tao L Nat Commun. 2025; 16(1):591.

PMID: 39799116 PMC: 11724971. DOI: 10.1038/s41467-024-55649-1.

Triboelectric Bending Sensors for AI-Enabled Sign Language Recognition.

Wang W, Bo X, Li W, Eldaly A, Wang L, Li W Adv Sci (Weinh). 2025; 12(8):e2408384.

PMID: 39778014 PMC: 11848593. DOI: 10.1002/advs.202408384.

Emerging Wearable Acoustic Sensing Technologies.

Liu T, Mao Y, Dou H, Zhang W, Yang J, Wu P Adv Sci (Weinh). 2025; 12(6):e2408653.

PMID: 39749384 PMC: 11809411. DOI: 10.1002/advs.202408653.

References

Shi Q, Zhang Z, He T, Sun Z, Wang B, Feng Y . Deep learning enabled smart mats as a scalable floor monitoring system. Nat Commun. 2020; 11(1):4609. PMC: 7490371. DOI: 10.1038/s41467-020-18471-z. View

LeCun Y, Bengio Y, Hinton G . Deep learning. Nature. 2015; 521(7553):436-44. DOI: 10.1038/nature14539. View

Sundaram S, Kellnhofer P, Li Y, Zhu J, Torralba A, Matusik W . Learning the signatures of the human grasp using a scalable tactile glove. Nature. 2019; 569(7758):698-702. DOI: 10.1038/s41586-019-1234-z. View

Zhu G, Pan C, Guo W, Chen C, Zhou Y, Yu R . Triboelectric-generator-driven pulse electrodeposition for micropatterning. Nano Lett. 2012; 12(9):4960-5. DOI: 10.1021/nl302560k. View

Dong K, Peng X, Wang Z . Fiber/Fabric-Based Piezoelectric and Triboelectric Nanogenerators for Flexible/Stretchable and Wearable Electronics and Artificial Intelligence. Adv Mater. 2019; 32(5):e1902549. DOI: 10.1002/adma.201902549. View

Wang S, He M, Weng B, Gan L, Zhao Y, Li N . Stretchable and Wearable Triboelectric Nanogenerator Based on Kinesio Tape for Self-Powered Human Motion Sensing. Nanomaterials (Basel). 2018; 8(9). PMC: 6165338. DOI: 10.3390/nano8090657. View

Hinton G, Salakhutdinov R . Reducing the dimensionality of data with neural networks. Science. 2006; 313(5786):504-7. DOI: 10.1126/science.1127647. View

Lee Y, Cha S, Kim Y, Choi D, Sun J . Transparent and attachable ionic communicators based on self-cleanable triboelectric nanogenerators. Nat Commun. 2018; 9(1):1804. PMC: 5935721. DOI: 10.1038/s41467-018-03954-x. View

Meltzner G, Colby G, Deng Y, Heaton J . Signal acquisition and processing techniques for sEMG based silent speech recognition. Annu Int Conf IEEE Eng Med Biol Soc. 2012; 2011:4848-51. DOI: 10.1109/IEMBS.2011.6091201. View

10.

Kim K, Lee M, Soon B, Ryu M, Kim J . Recognition of sign language with an inertial sensor-based data glove. Technol Health Care. 2015; 24 Suppl 1:S223-30. DOI: 10.3233/THC-151078. View

11.

Pu X, Guo H, Chen J, Wang X, Xi Y, Hu C . Eye motion triggered self-powered mechnosensational communication system using triboelectric nanogenerator. Sci Adv. 2017; 3(7):e1700694. PMC: 5533541. DOI: 10.1126/sciadv.1700694. View

12.

Liu H, Dong W, Li Y, Li F, Geng J, Zhu M . An epidermal sEMG tattoo-like patch as a new human-machine interface for patients with loss of voice. Microsyst Nanoeng. 2021; 6:16. PMC: 8433406. DOI: 10.1038/s41378-019-0127-5. View

13.

Makin J, Moses D, Chang E . Machine translation of cortical activity to text with an encoder-decoder framework. Nat Neurosci. 2020; 23(4):575-582. PMC: 10560395. DOI: 10.1038/s41593-020-0608-8. View

14.

Zhao Z, Yan C, Liu Z, Fu X, Peng L, Hu Y . Machine-Washable Textile Triboelectric Nanogenerators for Effective Human Respiratory Monitoring through Loom Weaving of Metallic Yarns. Adv Mater. 2016; 28(46):10267-10274. DOI: 10.1002/adma.201603679. View

15.

Qin K, Chen C, Pu X, Tang Q, He W, Liu Y . Magnetic Array Assisted Triboelectric Nanogenerator Sensor for Real-Time Gesture Interaction. Nanomicro Lett. 2021; 13(1):51. PMC: 8187499. DOI: 10.1007/s40820-020-00575-2. View

16.

Fagan M, Ell S, Gilbert J, Sarrazin E, Chapman P . Development of a (silent) speech recognition system for patients following laryngectomy. Med Eng Phys. 2007; 30(4):419-25. DOI: 10.1016/j.medengphy.2007.05.003. View

17.

Kudrinko K, Flavin E, Zhu X, Li Q . Wearable Sensor-Based Sign Language Recognition: A Comprehensive Review. IEEE Rev Biomed Eng. 2020; 14:82-97. DOI: 10.1109/RBME.2020.3019769. View

18.

Zou Y, Tan P, Shi B, Ouyang H, Jiang D, Liu Z . A bionic stretchable nanogenerator for underwater sensing and energy harvesting. Nat Commun. 2019; 10(1):2695. PMC: 6584498. DOI: 10.1038/s41467-019-10433-4. View

19.

Pu X, An S, Tang Q, Guo H, Hu C . Wearable triboelectric sensors for biomedical monitoring and human-machine interface. iScience. 2021; 24(1):102027. PMC: 7820136. DOI: 10.1016/j.isci.2020.102027. View

20.

Hua Q, Sun J, Liu H, Bao R, Yu R, Zhai J . Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing. Nat Commun. 2018; 9(1):244. PMC: 5770430. DOI: 10.1038/s41467-017-02685-9. View