US 12,340,019 B2
Multi-modal communication performance improvement system and method designed using similarities between voice and brain signal
Seong Whan Lee, Seoul (KR); Ji Won Lee, Seoul (KR); Seo Hyun Lee, Seoul (KR); Soowon Kim, Daegu (KR); and Jung Sun Lee, Changwon-si (KR)
Assigned to KOREA UNIVERSITY RESEARCH AND BUSINESS FOUNDATION, Seoul (KR)
Filed by Korea University Research and Business Foundation, Seoul (KR)
Filed on Nov. 30, 2023, as Appl. No. 18/524,096.
Claims priority of application No. 10-2022-0163783 (KR), filed on Nov. 30, 2022; and application No. 10-2023-0152350 (KR), filed on Nov. 7, 2023.
Prior Publication US 2024/0176421 A1, May 30, 2024
Int. Cl. G06F 3/01 (2006.01); G10L 15/02 (2006.01); G10L 15/18 (2013.01); G10L 21/0208 (2013.01)
CPC G06F 3/015 (2013.01) [G10L 15/02 (2013.01); G10L 15/18 (2013.01); G10L 21/0208 (2013.01); G10L 2015/027 (2013.01)] 10 Claims
OG exemplary drawing
 
1. A system for performance improvement in multimodal communication comprising:
a display unit configured to output syllables of a specific language;
a brain wave measurement unit configured to measure brain waves according to a user's imagined speech (i.e., imagined speech brain waves) corresponding to the syllables output by the display unit;
a voice signal measurement unit configured to measure voice signals according to the user's speech (i.e., spoken voice signals) corresponding to the syllables output by the display unit;
a storage unit configured to store the imagined speech brain waves measured by the brain wave measurement unit and the spoken voice signals measured by the voice signal measurement unit;
a signal pre-processing unit configured to remove noise from the spoken voice signals;
a syllable decoding model learning unit configured to learn the similarity between the imagined speech brain waves and the spoken voice signals; and
a result classification unit configured to classify the user's intended syllables based on the learning result of the syllable decoding model learning unit.