US 12,730,496 B2
Virtual interfaces for controlling IoT devices
Youjean Cho, New York, NY (US); Lei Zhang, Ann Arbor, MI (US); Daekun Kim, Santa Monica, CA (US); Ava Robinson, Solana Beach, CA (US); Yu Jiang Tham, Seattle, WA (US); Rajan Vaish, Beverly Hills, CA (US); and Andrés Monroy-Hernández, Seattle, WA (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Youjean Cho, New York, NY (US); Lei Zhang, Ann Arbor, MI (US); Daekun Kim, Santa Monica, CA (US); Ava Robinson, Solana Beach, CA (US); Yu Jiang Tham, Seattle, WA (US); Rajan Vaish, Beverly Hills, CA (US); and Andrés Monroy-Hernández, Seattle, WA (US)
Filed on Aug. 9, 2024, as Appl. No. 18/799,959.
Application 18/799,959 is a continuation of application No. 17/901,630, filed on Sep. 1, 2022, granted, now 12,073,011.
Prior Publication US 2024/0402796 A1, Dec. 5, 2024
Int. Cl. G06F 3/01 (2006.01); G06F 3/033 (2013.01); G06V 10/20 (2022.01); G06V 40/20 (2022.01); G16Y 20/40 (2020.01); H04N 13/111 (2018.01)
CPC G06F 3/011 (2013.01) [G06F 3/017 (2013.01); G06F 3/0334 (2013.01); G06V 10/255 (2022.01); G06V 40/20 (2022.01); G16Y 20/40 (2020.01); H04N 13/111 (2018.05)] 20 Claims
OG exemplary drawing
 
1. A method of controlling an Internet of Things (IoT) product using an electronic eyewear device comprising a camera and a display, the method comprising:
capturing frames of video data using the camera;
detecting the IoT product at an IoT product location in a physical environment based on the frames of video data;
estimating a current eyewear device location relative to the IoT product location;
detecting a foot shape at a foot location relative to the physical environment based on the frames of video data;
establishing a virtual plane relative to the display based on the foot location and the current eyewear device location;
presenting on the display a virtual control interface along the virtual plane according to the IoT product location;
detecting the foot shape performing a foot action relative to the virtual control interface; and
sending a control signal to the IoT product in accordance with the foot action, wherein the control signal comprises instructions associated with an IoT action.