US 12,406,432 B2
Mixed-reality occlusion using 3D reconstruction
Ari Antti Peuhkurinen, Helsinki (FI); and Mikko Strandborg, Helsinki (FI)
Assigned to Varjo Technologies Oy, Helsinki (FI)
Filed by Varjo Technologies Oy, Helsinki (FI)
Filed on May 18, 2023, as Appl. No. 18/319,953.
Prior Publication US 2024/0386655 A1, Nov. 21, 2024
Int. Cl. G06T 15/40 (2011.01); G06F 3/01 (2006.01); G06T 7/50 (2017.01); G06T 19/00 (2011.01)
CPC G06T 15/40 (2013.01) [G06F 3/013 (2013.01); G06T 7/50 (2017.01); G06T 19/006 (2013.01); G06T 2207/10028 (2013.01); G06T 2207/30196 (2013.01)] 12 Claims
OG exemplary drawing
 
1. A system comprising at least one server that is communicably coupled to at least one client device, wherein the at least one server is configured to:
obtain a three-dimensional (3D) model of a given real-world environment;
receive, from the at least one client device, viewpoint information indicative of a given viewpoint from a perspective of which a given mixed-reality (MR) image is to be generated;
for at least one virtual object that is to be embedded in the given MR image, determine at least one portion of the at least one virtual object that is being occluded by at least one real object present in the given real-world environment, based on optical depths determined from the 3D model corresponding to the given viewpoint, a given position at which the at least one virtual object is to be embedded with respect to the given viewpoint, and at least one of: a size of the at least one virtual object, a shape of the at least one virtual object, a given orientation of the at least one virtual object with respect to the given viewpoint; and
send a remaining portion of the at least one virtual object that is not being occluded to the at least one client device,
wherein the at least one server is configured to:
receive, from the at least one client device, a depth image captured from a perspective of the given viewpoint;
identify at least one additional portion of the at least one virtual object that is likely to be occluded, based on a comparison between the optical depths determined from the 3D model and optical depths in the received depth image;
determine optical depths of pixels that represent the at least one additional portion of the at least one virtual object, based on the given position at which the at least one virtual object is to be embedded with respect to the given viewpoint, and at least one of: the size of the at least one virtual object, the shape of the at least one virtual object, the given orientation of the at least one virtual object with respect to the given viewpoint; and
send, to the at least one client device, meta information indicative of: the at least one additional portion of the at least one virtual object that is likely to be occluded and the optical depths of the pixels that represent the at least one additional portion of the at least one virtual object.