US 11,875,604 B1
Systems and methods for determining hand poses in artificial reality environments
John-Fred Gilbert Abad, Redwood City, CA (US); Sohail Sayed Shafii, Fremont, CA (US); Andrew Mitchell Welch, Richardson, TX (US); Ryan Daniel Rutherford, Castro Valley, CA (US); Travis Creighton Hoffstetter, San Mateo, CA (US); Paul Nelson, Tracy, CA (US); and Oneil Howell, Clayton, CA (US)
Assigned to Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed by Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed on Feb. 22, 2021, as Appl. No. 17/181,723.
Int. Cl. G06V 40/20 (2022.01); G06T 7/73 (2017.01); G06F 3/01 (2006.01); G06T 19/00 (2011.01); G06V 40/10 (2022.01)
CPC G06V 40/28 (2022.01) [G06F 3/011 (2013.01); G06T 7/73 (2017.01); G06T 19/006 (2013.01); G06V 40/107 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising, by a computing system:
receiving an image of a real-world environment, the image including at least a portion of a hand of a user of an artificial reality device, the hand comprising a palm and a plurality of fingers;
determining, using the image, a hand pose of the hand while the hand is interacting with a virtual object displayed by the artificial reality device;
determining a grab state indicating whether the hand of the user is grabbing the virtual object, wherein the grab state is determined by:
defining, based on the hand pose, a three-dimensional surface positioned in the palm of the hand, wherein the three-dimensional surface is (a) different than the virtual object the user is interacting with, (b) invisible to the user, and (c) is configured for determining the grab state of the hand;
determining distances between predetermined portions of the plurality of fingers and the three-dimensional surface; and
assigning, based on the distances, a pose value for each of the plurality of fingers of the hand, wherein the grab state is determined based on the pose value;
accessing, based on the virtual object and the grab state, one or more display poses representing an appearance of the hand interacting with the virtual object; and
displaying the one or more display poses to the user of the artificial reality device.