US 12,135,840 B2
Bimanual gestures for controlling virtual and graphical elements
Karen Stolzenberg, Venice, CA (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on Dec. 15, 2022, as Appl. No. 18/081,736.
Application 18/081,736 is a continuation of application No. 17/589,065, filed on Jan. 31, 2022, granted, now 11,531,402, issued on Dec. 20, 2022.
Claims priority of provisional application 63/153,781, filed on Feb. 25, 2021.
Prior Publication US 2023/0117197 A1, Apr. 20, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 3/04847 (2022.01); G02B 27/01 (2006.01); G06F 3/01 (2006.01); G06F 3/04815 (2022.01); G06T 19/00 (2011.01); G06V 20/40 (2022.01); G06V 40/10 (2022.01); G06V 40/20 (2022.01)
CPC G06F 3/017 (2013.01) [G02B 27/0101 (2013.01); G02B 27/017 (2013.01); G06F 3/04815 (2013.01); G06F 3/04847 (2013.01); G06T 19/006 (2013.01); G06V 20/46 (2022.01); G06V 40/10 (2022.01); G06V 40/28 (2022.01); G02B 2027/0138 (2013.01); G02B 2027/014 (2013.01); G02B 2027/0178 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of controlling movable elements using an eyewear device comprising a camera and a display, the method comprising:
defining a display plane comprising a region smaller than the entire display, wherein the display plane is persistently associated with a current eyewear location relative to a physical environment;
capturing frames of video data with the camera;
detecting a first hand and a second hand in the captured frames of video data;
in response to detecting the first and second hands, presenting on the display a movable element at a first position relative to the display plane;
defining an input plane along a surface of the first hand;
estimating the current eyewear location relative to the input plane;
calculating a correlation between the input plane and the display plane based on the current eyewear location;
detecting a fingertip of the second hand at a current fingertip location over time including a current fingertip location, wherein each of the series is characterized by a set of three-dimensional coordinates relative to the input plane, and wherein at least one of the series of fingertip locations comprises the fingertip touching the surface of the first hand the input plane;
presenting on the display the movable element at a current element position within the display plane in accordance with the correlation, such that the movable element relative to the display plane is persistently correlated with the current fingertip location relative to the input plane; and
executing a selecting action based on the current element position relative to one or more graphical elements presented within the display plane and in accordance with the set of three-dimensional coordinates associated with the current fingertip location.