US 12,243,170 B2
Live in-camera overlays
Stefan Johannes Josef Holzer, San Mateo, CA (US); Matteo Munaro, San Francisco, CA (US); Alexander Jay Bruen Trevor, San Francisco, CA (US); Aidas Liaudanskas, San Francisco, CA (US); and Radu Bogdan Rusu, San Francisco, CA (US)
Assigned to Fyusion, Inc., San Francisco, CA (US)
Filed by Fyusion, Inc., San Francisco, CA (US)
Filed on Apr. 28, 2020, as Appl. No. 16/861,100.
Application 16/861,100 is a continuation in part of application No. 16/518,501, filed on Jul. 22, 2019.
Claims priority of provisional application 62/843,122, filed on May 3, 2019.
Claims priority of provisional application 62/795,427, filed on Jan. 22, 2019.
Prior Publication US 2020/0258309 A1, Aug. 13, 2020
Int. Cl. G06T 19/00 (2011.01); G06T 7/70 (2017.01)
CPC G06T 19/006 (2013.01) [G06T 7/70 (2017.01); G06T 2207/30244 (2013.01)] 6 Claims
OG exemplary drawing
 
1. A computing device comprising:
a camera configured to capture a live camera feed of an object;
a processor configured to:
determine an object identity for an object represented the live camera feed during an initialization phase, the initialization phase including projecting one or more triangulated points into a designated frame based on camera pose information determined based on data collected from an inertial measurement unit at the computing device, wherein projecting the one or more triangulated points includes triangulating 2D skeleton joints into 3D skeleton joints,
create an object model of the object by performing skeleton detection on the live camera feed;
determine augmented reality overlay data based on the object identity, the augmented reality overlay data including one or more tags, each of the tags characterizing a feature of the object, each of the tags being associated with a respective location on the object, each of the respective locations being represented in a reference view of the object, wherein a temporal filter is applied to the overlay data locations by weighting or averaging displayed overlay data locations for a current frame based on overlay data locations determined in previous frames, wherein the temporal filter is used to generate a smooth visualization of a result, and
determine, for each of a plurality of frames in the live camera feed, a respective frame location for one or more of the tags, each of the respective frame locations determined based on a correspondence between the reference view of the object and the respective frame; and
a display screen configured to present the live camera feed including the plurality of frames during a presentation phase after the initialization phase, each of the plurality of frames including a respective one of the tags, each of the tags being positioned at the respective frame location, wherein the initialization phase is performed again when the computing device determines that the live camera feed has changed more than a designated amount from the image present when initialization was last performed, wherein initialization comprises estimating a reference-to-live image correspondence using a surface mesh.