US 12,008,991 B2
Enabling natural language interactions with user interfaces for users of a software application
Utku Yabas, Toronto (CA); Philipp Hubert, Toronto (CA); and Karl Stahl, Palo Alto, CA (US)
Assigned to SoundHound AI IP, LLC, Santa Clara, CA (US)
Filed by SoundHound, Inc., Santa Clara, CA (US)
Filed on May 27, 2021, as Appl. No. 17/332,927.
Prior Publication US 2022/0383869 A1, Dec. 1, 2022
Int. Cl. G10L 15/22 (2006.01); G06F 40/211 (2020.01); G06F 40/284 (2020.01); G10L 15/183 (2013.01); G10L 15/26 (2006.01)
CPC G10L 15/22 (2013.01) [G06F 40/211 (2020.01); G06F 40/284 (2020.01); G10L 15/183 (2013.01); G10L 15/26 (2013.01); G10L 2015/223 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A computer-implemented method for identifying a user interface element referenced by a natural language command, the computer-implemented method comprising:
receiving, from a remote device over a network, a natural language command and user interface metadata describing a set of user interface elements visible within a user interface of the device;
selecting a grammar corresponding to the device, the grammar being a context-free grammar having a slot and containing two numerical non-terminals indicating a row and a column of a selected one of the visible user interface elements;
parsing the natural language command according to the selected grammar, the slot being filled by text;
matching the text with a user interface element in the user interface metadata; and
providing a descriptor of the matched user interface element to the device.