US 11,727,658 B2
Using camera feed to improve quality of reconstructed images
Mikko Strandborg, Hangonkylä (FI); and Petteri Timonen, Helsinki (FI)
Assigned to Varjo Technologies Oy, Helsinki (FI)
Filed by Varjo Technologies Oy, Helsinki (FI)
Filed on Oct. 1, 2021, as Appl. No. 17/492,213.
Prior Publication US 2023/0108922 A1, Apr. 6, 2023
Int. Cl. G06T 19/20 (2011.01); G06T 17/00 (2006.01); G06T 7/50 (2017.01); G06T 7/70 (2017.01); G06F 3/0346 (2013.01)
CPC G06T 19/20 (2013.01) [G06F 3/0346 (2013.01); G06T 7/50 (2017.01); G06T 7/70 (2017.01); G06T 17/00 (2013.01); G06T 2219/2008 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A system comprising at least one server configured to:
receive, from a host device, a plurality of visible-light images of a real-world environment captured by at least one visible-light camera of the host device;
process the plurality of visible-light images to generate a three-dimensional environment model of the real-world environment, wherein the three-dimensional environment model is generated in a global coordinate space;
receive, from a client device, information indicative of a given pose of the client device in the global coordinate space;
utilise the three-dimensional environment model to generate at least one reconstructed image and at least one reconstructed depth map from a perspective of the given pose of the client device;
determine, based on the at least one reconstructed depth map and the given pose of the client device, a position of each pixel of the at least one reconstructed image in the global coordinate space;
receive, from the host device, at least one current visible-light image of the real-world environment captured by the at least one visible-light camera or at least one other visible-light camera of the host device;
receive, from the host device, information indicative of a current pose of the host device from which the at least one current visible-light image is captured, or determine, from the at least one current visible-light image and the three-dimensional environment model, the current pose of the host device from which the at least one current visible-light image is captured, wherein the current pose of the host device is in the global coordinate space;
for a given pixel of the at least one reconstructed image, determine based on the current pose of the host device whether or not a corresponding pixel exists in the at least one current visible-light image whose position in the global coordinate space matches with the position of the given pixel of the at least one reconstructed image;
replace initial pixel values of the given pixel in the at least one reconstructed image with pixel values of the corresponding pixel in the at least one current visible-light image, when the corresponding pixel exists in the at least one current visible-light image; and
send the at least one reconstructed image to the client device for display thereat.