US 12,405,658 B2
Virtual AR interfaces for controlling IoT devices using mobile device orientation sensors
Lei Zhang, Ann Arbor, MI (US); Youjean Cho, New York, NY (US); Daekun Kim, Santa Monica, CA (US); Ava Robinson, Solana Beach, CA (US); Yu Jiang Tham, Seattle, WA (US); Rajan Vaish, Beverly Hills, CA (US); and Andrés Monroy-Hernández, Seattle, WA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on Jun. 20, 2024, as Appl. No. 18/748,987.
Application 18/748,987 is a continuation of application No. 17/901,568, filed on Sep. 1, 2022, granted, now 12,045,383, issued on Jul. 23, 2024.
Prior Publication US 2024/0338071 A1, Oct. 10, 2024
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 3/01 (2006.01); G06F 1/16 (2006.01); G06T 19/00 (2011.01)
CPC G06F 3/011 (2013.01) [G06F 1/163 (2013.01); G06T 19/006 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A method of controlling an Internet of Things (IoT) product using a portable electronic device comprising an inertial measurement unit (IMU), a camera, and a display, wherein the IoT product is associated with a product API comprising a product type and a controllable feature associated with a range of variable values, the method comprising:
capturing frames of video data with the camera;
detecting the IoT product at an IoT product location relative to a physical environment using the captured frames of video data;
determining a portable electronic device location relative to the IoT product location using the captured frames of video data;
detecting a device orientation relative to the physical environment using the IMU;
presenting on the display, when the IoT product location is within a threshold proximity of the portable electronic device location, a virtual rotational interface at a viewport position according to at least one of the IoT product location or the portable electronic device location, persistently and without regard to the device orientation;
associating the device orientation with the controllable feature;
presenting relative to the virtual rotational interface a slider at a slider position according to the device orientation; and
sending a control signal to the IoT product in accordance with the device orientation, the control signal comprising instructions for use by the IoT product to perform an IoT action relative to the controllable feature.