US 12,494,078 B2
Hand-tracking with IR camera for XR systems
Piotr Gurgul, Hergiswil (CH); Sharon Moll, Lachen (CH); and Pawel Wawruch, Yverdon-les-Bains (CH)
Assigned to SNAP INC., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on Dec. 8, 2022, as Appl. No. 18/063,489.
Prior Publication US 2024/0193982 A1, Jun. 13, 2024
Int. Cl. G06V 20/20 (2022.01); G02B 27/01 (2006.01); G06T 7/246 (2017.01); G06T 7/73 (2017.01); G06V 20/40 (2022.01); G06V 40/10 (2022.01); H04N 23/20 (2023.01)
CPC G06V 40/11 (2022.01) [G02B 27/0101 (2013.01); G02B 27/017 (2013.01); G06T 7/251 (2017.01); G06T 7/75 (2017.01); G06V 20/20 (2022.01); G06V 20/46 (2022.01); H04N 23/20 (2023.01); G02B 2027/0138 (2013.01); G02B 2027/014 (2013.01); G06T 2207/10048 (2013.01); G06T 2207/30196 (2013.01)] 20 Claims
OG exemplary drawing
 
1. An eXtended Reality (XR) system comprising:
one or more visible light cameras;
one or more wide-spectrum cameras;
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the XR system to perform first operations comprising:
capturing, using the one or more visible light cameras, tracking video frame data of a pose of a hand of a user of the XR system;
generating hand-tracking data based on the tracking video frame data, the hand-tracking data comprising a skeletal model and a hand-tracking confidence level indicating a probability that the skeletal model matches the pose of the hand of the user;
comparing the hand-tracking confidence level to a threshold confidence value; and
based on determining the hand-tracking confidence level is below the threshold confidence value, performing second operations comprising:
activating one or more light emitters to illuminate the hand of the user;
deactivating the one or more visible light cameras;
activating the one or more wide-spectrum cameras; and
continuing to capture, using the one or more wide-spectrum cameras, the tracking video frame data of the pose of the hand of the user.