US 11,727,677 B2
Personalized gesture recognition for user interaction with assistant systems
Xiaohu Liu, Bellevue, WA (US); Paul Anthony Crook, Newcastle, WA (US); Francislav P Penov, Kirkland, WA (US); and Rajen Subba, San Carlos, CA (US)
Assigned to Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed by Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed on Dec. 30, 2021, as Appl. No. 17/566,308.
Application 17/566,308 is a continuation of application No. 17/010,619, filed on Sep. 2, 2020, granted, now 11,231,946.
Application 17/010,619 is a continuation of application No. 16/388,130, filed on Apr. 18, 2019, granted, now 10,802,848, issued on Oct. 13, 2020.
Claims priority of provisional application 62/660,876, filed on Apr. 20, 2018.
Prior Publication US 2022/0179670 A1, Jun. 9, 2022
Int. Cl. G06V 10/82 (2022.01); G06F 9/451 (2018.01); G10L 15/18 (2013.01); G10L 15/183 (2013.01); G10L 15/22 (2006.01); G06F 16/338 (2019.01); G06F 16/332 (2019.01); G06F 16/33 (2019.01); G06N 20/00 (2019.01); G06F 16/9535 (2019.01); G06Q 50/00 (2012.01); H04L 67/306 (2022.01); G06F 16/176 (2019.01); G10L 15/06 (2013.01); G10L 15/16 (2006.01); G06F 3/01 (2006.01); G06F 16/9032 (2019.01); G06F 16/2457 (2019.01); H04L 51/02 (2022.01); G06F 3/16 (2006.01); G06N 3/08 (2023.01); G10L 15/26 (2006.01); G06F 16/9038 (2019.01); G06F 16/904 (2019.01); G06F 40/30 (2020.01); G06F 40/40 (2020.01); G06F 16/22 (2019.01); G06F 16/23 (2019.01); G06F 7/14 (2006.01); H04L 43/0882 (2022.01); H04L 43/0894 (2022.01); H04L 12/28 (2006.01); H04L 41/00 (2022.01); H04L 41/22 (2022.01); H04W 12/08 (2021.01); G10L 15/07 (2013.01); G10L 17/22 (2013.01); G06N 3/006 (2023.01); G10L 17/06 (2013.01); G06F 16/248 (2019.01); G06F 16/951 (2019.01); G06F 16/242 (2019.01); G06F 16/2455 (2019.01); G10L 15/02 (2006.01); G10L 15/187 (2013.01); G06V 20/10 (2022.01); G06V 40/20 (2022.01); G06F 40/295 (2020.01); H04L 51/52 (2022.01); H04L 51/216 (2022.01); H04L 67/50 (2022.01); H04L 67/5651 (2022.01); H04L 51/18 (2022.01); H04L 67/75 (2022.01); G06F 16/903 (2019.01); G06F 40/205 (2020.01); G06F 18/2411 (2023.01); G06N 7/01 (2023.01); G06V 10/764 (2022.01); H04L 67/10 (2022.01); H04L 51/046 (2022.01); G10L 13/00 (2006.01); G10L 13/04 (2013.01); H04L 67/53 (2022.01)
CPC G06V 10/82 (2022.01) [G06F 3/011 (2013.01); G06F 3/013 (2013.01); G06F 3/017 (2013.01); G06F 3/167 (2013.01); G06F 7/14 (2013.01); G06F 9/453 (2018.02); G06F 16/176 (2019.01); G06F 16/2255 (2019.01); G06F 16/2365 (2019.01); G06F 16/243 (2019.01); G06F 16/248 (2019.01); G06F 16/24552 (2019.01); G06F 16/24575 (2019.01); G06F 16/24578 (2019.01); G06F 16/338 (2019.01); G06F 16/3323 (2019.01); G06F 16/3329 (2019.01); G06F 16/3344 (2019.01); G06F 16/904 (2019.01); G06F 16/9038 (2019.01); G06F 16/90332 (2019.01); G06F 16/90335 (2019.01); G06F 16/951 (2019.01); G06F 16/9535 (2019.01); G06F 18/2411 (2023.01); G06F 40/205 (2020.01); G06F 40/295 (2020.01); G06F 40/30 (2020.01); G06F 40/40 (2020.01); G06N 3/006 (2013.01); G06N 3/08 (2013.01); G06N 7/01 (2023.01); G06N 20/00 (2019.01); G06Q 50/01 (2013.01); G06V 10/764 (2022.01); G06V 20/10 (2022.01); G06V 40/28 (2022.01); G10L 15/02 (2013.01); G10L 15/063 (2013.01); G10L 15/07 (2013.01); G10L 15/16 (2013.01); G10L 15/183 (2013.01); G10L 15/187 (2013.01); G10L 15/1815 (2013.01); G10L 15/1822 (2013.01); G10L 15/22 (2013.01); G10L 15/26 (2013.01); G10L 17/06 (2013.01); G10L 17/22 (2013.01); H04L 12/2816 (2013.01); H04L 41/20 (2013.01); H04L 41/22 (2013.01); H04L 43/0882 (2013.01); H04L 43/0894 (2013.01); H04L 51/02 (2013.01); H04L 51/18 (2013.01); H04L 51/216 (2022.05); H04L 51/52 (2022.05); H04L 67/306 (2013.01); H04L 67/535 (2022.05); H04L 67/5651 (2022.05); H04L 67/75 (2022.05); H04W 12/08 (2013.01); G06F 2216/13 (2013.01); G10L 13/00 (2013.01); G10L 13/04 (2013.01); G10L 2015/223 (2013.01); G10L 2015/225 (2013.01); H04L 51/046 (2013.01); H04L 67/10 (2013.01); H04L 67/53 (2022.05)] 20 Claims
OG exemplary drawing
 
1. A system comprising:
a microphone configured to receive a speech-input;
a camera configured to receive a gesture-input; and
circuitry configured to:
communicate a user input comprising at least one of the speech-input and the gesture-input to an external assistant system to cause the external assistant system to determine an output user intent from the user input;
in response to communicating the user input, receive information from the external assistant system for the output user intent; and
execute a task based at least in part on the received information for the output user intent determined from at least one of the speech-input and the gesture-input.