US 12,450,837 B2
Contextual visual and voice search from electronic eyewear device
David Meisenholder, Los Angeles, CA (US); Kameron Sheffield, South Jordan, UT (US); Joseph Timothy Fortier, Aspen, CO (US); Raymond Zeng, Long Island City, NY (US); Andrei Rybin, Lehi, UT (US); and Jonathan Geddes, Saratoga Springs, UT (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on May 12, 2022, as Appl. No. 17/742,900.
Claims priority of provisional application 63/190,613, filed on May 19, 2021.
Prior Publication US 2022/0375172 A1, Nov. 24, 2022
Int. Cl. G06T 19/00 (2011.01); G06F 3/04817 (2022.01); G06F 3/0482 (2013.01); G06F 3/16 (2006.01); G06V 10/40 (2022.01); G06V 10/70 (2022.01); G06V 10/74 (2022.01); G06V 20/20 (2022.01); G06V 20/60 (2022.01); G10L 15/08 (2006.01); G10L 15/22 (2006.01)
CPC G06T 19/006 (2013.01) [G06F 3/04817 (2013.01); G06F 3/0482 (2013.01); G06F 3/167 (2013.01); G06V 10/40 (2022.01); G06V 10/74 (2022.01); G06V 10/768 (2022.01); G06V 20/20 (2022.01); G06V 20/60 (2022.01); G10L 15/08 (2013.01); G10L 15/22 (2013.01); G06T 2200/24 (2013.01); G10L 2015/088 (2013.01); G10L 2015/223 (2013.01)] 17 Claims
OG exemplary drawing
 
1. An electronic eyewear device adapted to be worn on a head of a user, comprising:
a display;
at least one camera adapted to scan a scene in a viewing area around the user and
to capture a scan image;
a microphone adapted to capture voice data from the user;
a memory that stores instructions; and
a processor that executes the instructions to perform operations including:
initiating a scan by the at least one camera to capture the scan image;
processing the scan image or sending the scan image to an image processing device to extract at least one contextual signal from the scan image, wherein the at least one extracted contextual signal includes contextual data from the user and the viewing area around the user;
capturing, via the microphone, voice data from the user;
performing voice-to-text conversion of the captured voice data or sending the captured voice data to a voice data processing device to identify at least one keyword in the voice data;
processing the at least one extracted contextual signal and the at least one identified keyword or forwarding the at least one extracted contextual signal and the at least one identified keyword to an augmented reality feature storage to guide a search for an augmented reality feature having metadata that matches the user's search intent as determined from the at least one identified keyword and the at least one contextual signal and selecting at least one augmented reality feature from the augmented reality feature storage having metadata that matches the at least one extracted contextual signal and the at least one identified keyword;
presenting the selected at least one selected augmented reality feature to the display for user selection; and
applying an augmented reality feature selected by the user to the scene for display on the electronic eyewear device.