US 11,914,836 B2
Hand presence over keyboard inclusiveness
Adrian Brian Ratter, Redwood City, CA (US); Alessia Marra, Zurich (CH); Yugeng He, San Francisco, CA (US); and Panya Inversin, Zurich (CH)
Assigned to Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed by Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed on Dec. 20, 2022, as Appl. No. 18/069,164.
Application 18/069,164 is a continuation of application No. 17/073,200, filed on Oct. 16, 2020, granted, now 11,537,258.
Prior Publication US 2023/0131667 A1, Apr. 27, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 3/04815 (2022.01); G02B 27/00 (2006.01); G02B 27/01 (2006.01); G06T 7/10 (2017.01); G06T 7/70 (2017.01)
CPC G06F 3/04815 (2013.01) [G02B 27/0093 (2013.01); G02B 27/017 (2013.01); G06T 7/10 (2017.01); G06T 7/70 (2017.01); G06T 2207/20132 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising, by a computing system:
accessing an image of a physical environment of a user, the image being associated with a perspective of the user and depicting a physical input device and a physical hand of the user;
determining that a contrast between the physical input device and the physical hand depicted in the image is lower than a predetermined threshold;
modifying the image to increase the contrast between the physical input device and the physical hand depicted in the image, thereby improving a visual distinction between a first set of pixels associated with the physical hand and a second set of pixels associated with the physical input device in the image;
determining a pose of the physical input device;
generating a three-dimensional model representing the physical hand of the user;
generating an image mask by projecting the three-dimensional model onto an image plane associated with the perspective of the user;
generating, by applying the image mask to the modified image, a cropped image depicting at least the physical hand of the user in the image;
rendering, based on the perspective of the user and the pose of the physical input device, a virtual input device to represent the physical input device; and
displaying the cropped image depicting at least the physical hand of the user over the rendered virtual input device.