US 11,989,843 B2
Robotic learning of assembly tasks using augmented reality
Kai Zhou, Wiener Neudorf (AT); and Adrian Schoisengeier, Vienna (AT)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on Jun. 22, 2022, as Appl. No. 17/846,930.
Prior Publication US 2023/0419615 A1, Dec. 28, 2023
Int. Cl. G06T 19/00 (2011.01); B25J 13/08 (2006.01); G06F 3/01 (2006.01); G06T 17/00 (2006.01); G06T 19/20 (2011.01); G06V 20/64 (2022.01); G06V 40/10 (2022.01)
CPC G06T 19/006 (2013.01) [B25J 13/08 (2013.01); G06F 3/012 (2013.01); G06F 3/017 (2013.01); G06T 17/00 (2013.01); G06T 19/20 (2013.01); G06V 20/64 (2022.01); G06T 2200/08 (2013.01); G06T 2210/56 (2013.01); G06T 2219/2016 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
displaying a first virtual object in a display of an augmented reality (AR) device, the first virtual object corresponding to a first physical object in a physical environment of the AR device;
tracking, using the AR device, a manipulation of the first virtual object by a user of the AR device, the manipulation of the first virtual object being relative to a second physical object in the physical environment or a second virtual object corresponding to the second physical object;
identifying an initial state, a plurality of intermediate states, and a final state of the first virtual object based on the tracking, the initial state corresponding to an initial pose of the first virtual object relative to the second physical object or the second virtual object, the final state corresponding to a final pose of the first virtual object relative to the second physical object or the second virtual object, the plurality of intermediate states being between the initial state and the final state; and
programming by demonstration a robotic system using the tracking of the manipulation of the first virtual object relative to the second physical object or the second virtual object, the initial state of the first virtual object, and the final state of the first virtual object,
wherein tracking the manipulation of the first virtual object comprises: identifying hand gestures of the user, tracking hand gestures of the user over a period of time, tracking, using a 6 degrees-of-freedom tracking system at the AR device, a pose of the AR device over the period of time, and identifying a trajectory of the AR device based on the pose of the AR device over the period of time; and
wherein the initial pose of the first virtual object, the plurality of intermediate states, and the final pose of the first virtual object are based on the tracked hand gestures of the user and the trajectory of the AR device.