US 11,941,824 B2
Video-based hand and ground reaction force determination
SangHyun Lee, Ann Arbor, MI (US); and Meiyin Liu, New Brunswick, NJ (US)
Assigned to VelocityEHS Holdings, Inc., Chicago, IL (US)
Filed by Velocity EHS Inc., Chicago, IL (US)
Filed on Apr. 12, 2022, as Appl. No. 17/718,818.
Claims priority of provisional application 63/173,851, filed on Apr. 12, 2021.
Prior Publication US 2022/0327775 A1, Oct. 13, 2022
Int. Cl. G06T 7/246 (2017.01); G01L 1/00 (2006.01); G06T 17/00 (2006.01); G06T 17/20 (2006.01); G06T 19/00 (2011.01); G06V 40/20 (2022.01)
CPC G06T 7/251 (2017.01) [G01L 1/005 (2013.01); G06T 17/005 (2013.01); G06T 17/20 (2013.01); G06T 19/00 (2013.01); G06V 40/28 (2022.01); G06T 2200/04 (2013.01); G06T 2200/08 (2013.01); G06T 2207/10016 (2013.01); G06T 2207/20084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for determining a hand force and a ground reaction force for a musculoskeletal body of a subject, the method comprising:
obtaining, with a processor, video data for a musculoskeletal body during an action taken by the subject, the video data comprising a plurality of frames;
generating, with the processor, for each frame of the plurality of frames, three-dimensional pose data for the subject based on a three-dimensional skeletal model; and
determining, with the processor, the hand force and the ground reaction force based on the three-dimensional pose data,
wherein determining the hand force and the ground reaction force comprises:
implementing, with the processor, a reconstruction of the hand force and the ground reaction force exerted on the musculoskeletal body during the action based on the three-dimensional pose data, the reconstruction being configured to determine an estimate for the ground reaction force in conjunction with an estimate for the hand force such that the estimate for the hand force is based on the estimate for the ground reaction force; and
applying, with the processor, the three-dimensional pose data, the estimate for the ground reaction force, and the estimate of the hand force, to a recurrent neural network or a computer vision deep neural network to optimize the estimate of the hand force and the estimate of the ground reaction force.