US 12,073,011 B2
Virtual interfaces for controlling IoT devices
Youjean Cho, New York, NY (US); Lei Zhang, Ann Arbor, MI (US); Daekun Kim, Santa Monica, CA (US); Ava Robinson, Solana Beach, CA (US); Yu Jiang Tham, Seattle, WA (US); Rajan Vaish, Beverly Hills, CA (US); and Andrés Monroy-Hernández, Seattle, WA (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Youjean Cho, New York, NY (US); Lei Zhang, Ann Arbor, MI (US); Daekun Kim, Santa Monica, CA (US); Ava Robinson, Solana Beach, CA (US); Yu Jiang Tham, Seattle, WA (US); Rajan Vaish, Beverly Hills, CA (US); and Andrés Monroy-Hernández, Seattle, WA (US)
Filed on Sep. 1, 2022, as Appl. No. 17/901,630.
Prior Publication US 2024/0077935 A1, Mar. 7, 2024
Int. Cl. G06F 3/01 (2006.01); G06F 3/033 (2013.01); G06V 10/20 (2022.01); G06V 40/20 (2022.01); G16Y 20/40 (2020.01); H04N 13/111 (2018.01)
CPC G06F 3/011 (2013.01) [G06F 3/017 (2013.01); G06F 3/0334 (2013.01); G06V 10/255 (2022.01); G06V 40/20 (2022.01); G16Y 20/40 (2020.01); H04N 13/111 (2018.05)] 20 Claims
OG exemplary drawing
 
1. A method of controlling an Internet of Things (IoT) product using a portable electronic device comprising a camera and a display, the method comprising steps of:
capturing frames of video data with the camera;
detecting the IoT product at an IoT product location in a physical environment using the captured frames of video data;
determining a portable electronic device location relative to the IoT product location using the captured frames of video data;
presenting on the display a virtual control interface according to the IoT product location and the portable electronic device location, wherein the virtual control interface comprises a virtual surface virtually presented on the display at a surface position relative to the physical environment;
detecting a foot shape performing a foot action at a foot location relative to the physical environment using the captured frames of video data;
detecting an intersection between the foot location and the surface position using the captured frames of video data, the intersection and the foot action corresponding to an IoT action; and
sending a control signal to the IoT product in accordance with the detected intersection and the foot action, the control signal comprising instructions for use by the IoT product to perform the IoT action.