US 12,260,476 B2
Augmented reality systems and methods utilizing reflections
Christopher M. Harrises, Nashua, NH (US); Nicole Elizabeth Samec, Fort Lauderdale, FL (US); Nastasja U. Robaina, Coconut Grove, FL (US); Mark Baerenrodt, Millbrae, CA (US); Adam Carl Wright, Fort Lauderdale, FL (US); and Adrian Kaehler, Los Angeles, CA (US)
Assigned to MAGIC LEAP, INC., Plantation, FL (US)
Filed by Magic Leap, Inc., Plantation, FL (US)
Filed on Feb. 1, 2022, as Appl. No. 17/590,724.
Application 17/590,724 is a continuation of application No. 16/543,280, filed on Aug. 16, 2019, granted, now 11,244,485.
Application 16/543,280 is a continuation of application No. 15/409,430, filed on Jan. 18, 2017, granted, now 10,430,985, issued on Oct. 1, 2019.
Claims priority of provisional application 62/445,630, filed on Jan. 12, 2017.
Claims priority of provisional application 62/440,336, filed on Dec. 29, 2016.
Claims priority of provisional application 62/366,533, filed on Jul. 25, 2016.
Claims priority of provisional application 62/343,583, filed on May 31, 2016.
Claims priority of provisional application 62/343,636, filed on May 31, 2016.
Claims priority of provisional application 62/315,456, filed on Mar. 30, 2016.
Claims priority of provisional application 62/294,147, filed on Feb. 11, 2016.
Claims priority of provisional application 62/280,519, filed on Jan. 19, 2016.
Prior Publication US 2022/0156995 A1, May 19, 2022
Int. Cl. G06T 11/60 (2006.01); G02B 27/01 (2006.01); G06T 19/00 (2011.01)
CPC G06T 11/60 (2013.01) [G02B 27/0172 (2013.01); G06T 19/006 (2013.01); G02B 2027/014 (2013.01); G02B 2027/0178 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for sharing an image in an augmented reality telepresence session, the method comprising:
establishing a bidirectional communication between a first augmented reality device associated with a first caller and a second augmented reality device associated with a second caller, the first augmented reality device comprising a first outward facing camera configured to image a region in front of the first augmented reality device;
detecting presence of a reflective surface in a region in front of the first augmented reality device, wherein the detecting the presence of the reflective surface includes keypoints recognition of characteristics of the region in front of the first augmented reality device and one or more depth discontinuities within the reflective surface, wherein detecting the presence of the reflective surface comprises:
identifying a first set of keypoints based on the keypoints recognition of characteristics of the region in front of the first augmented reality device, the first set of keypoints including a first triplet of neighboring keypoints;
maintaining a map of an environment around the first augmented reality device, the map comprising a second set of keypoints corresponding to objects in the environment, the second set of keypoints including a second triplet of neighboring keypoints;
determining geometric relationships between the first triplet of neighboring keypoints and between the second triplet of neighboring keypoints; and
matching the geometric relationships associated with the first triplet of neighboring keypoints and the second triplet of neighboring keypoints;
confirming the presence of the reflective surface based on the matching of the geometric relationships;
capturing, with the first outward facing camera, a first image of the first caller reflected from the reflective surface;
identifying a region of the first image of the first caller that includes an image of the first augmented reality device occluding a portion of a face of the first caller;
generating an unoccluded image of the first caller by substituting the identified region with a corresponding region of an image of the unoccluded face of the first caller without the image of the first augmented reality device that previously occluded the portion of the face of the first caller;
transmitting at least a portion of the unoccluded image of the first caller to the second augmented reality device; and
the determining the geometric relationships between the first triplet of neighboring keypoints and the second triplet of neighboring keypoints includes at least one of a) determining open angles among the first triplet of neighboring keypoints and open angles among the second triplet of neighboring keypoints, b) axially transforming a first geometry represented by the first triplet of neighboring keypoints or axially transforming a second geometry represented by the first triplet of neighboring keypoints, or c) determining an amount of magnification between a first size of the first geometry and a second size of the second geometry.