US 11,868,536 B2
Methods and systems for touchless control with a mobile device
Wang Fai Ng, Hong Kong (HK); Wing Hung Chan, Hong Kong (HK); Long Mak, Hong Kong (HK); Keng Fai Lee, Cupertino, CA (US); and On Loy Sung, Hong Kong (HK)
Assigned to NEX Team Inc., San Jose, CA (US)
Filed by NEX Team Inc., San Jose, CA (US)
Filed on Mar. 20, 2020, as Appl. No. 16/824,950.
Application 16/824,950 is a continuation of application No. 16/432,575, filed on Jun. 5, 2019, granted, now 10,620,713.
Prior Publication US 2020/0387232 A1, Dec. 10, 2020
Int. Cl. G06F 3/01 (2006.01); G06F 3/04812 (2022.01); G06T 7/73 (2017.01); G06F 3/04815 (2022.01); G06N 3/08 (2023.01); G06V 40/20 (2022.01)
CPC G06F 3/017 (2013.01) [G06F 3/04812 (2013.01); G06F 3/04815 (2013.01); G06N 3/08 (2013.01); G06T 7/73 (2017.01); G06V 40/28 (2022.01)] 17 Claims
OG exemplary drawing
 
1. A device for performing touchless control, comprising:
at least one memory device that stores computer-executable instructions; and
at least one processor configured to access the memory device, wherein the processor is configured to execute the computer-executable instructions to:
receive at least one image from at least one camera;
apply a machine learning technique to the image to determine a first map, the first map comprising a function that generates a correspondence between data associated with the image to a feature space;
determine a second map comprising a representation of a user in the image based on the first map;
determine a third map comprising a vector field that encodes key-point to key-point relationships in the image;
determine a pose of the user based on the second map and the third map by predicting, via the machine learning technique, a body part confidence associated with a portion of a body of the user;
identify a gesture of the user in the image based on the pose of the user;
track the pose of the user and the portion of the body of the user for performing touchless control of the device to generate a historical data of the pose and the portion of the body of the user;
determine, based on the tracked pose, a mapping between the portion of the user's body and an input feature associated with the device;
modify the input feature to a first location based on the mapping to enable an interaction between the user and the device; and
modify the input feature to a second location different from the first location based on the historical data in addition to the mapping.