US 12,079,969 B2
Colorization to show contribution of different camera modalities
Raymond Kirk Price, Redmond, WA (US); Michael Bleyer, Seattle, WA (US); and Christopher Douglas Edmonds, Carnation, WA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Apr. 14, 2021, as Appl. No. 17/230,518.
Prior Publication US 2022/0335578 A1, Oct. 20, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G06T 7/174 (2017.01); G02B 27/01 (2006.01); G06T 3/40 (2006.01); G06T 5/20 (2006.01); G06T 5/50 (2006.01); G06T 5/77 (2024.01); G06T 7/13 (2017.01); G06T 7/40 (2017.01)
CPC G06T 5/77 (2024.01) [G02B 27/0172 (2013.01); G06T 3/40 (2013.01); G06T 5/20 (2013.01); G06T 5/50 (2013.01); G06T 7/13 (2017.01); G06T 7/174 (2017.01); G06T 7/40 (2013.01); G06T 2207/20024 (2013.01); G06T 2207/20084 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A method for generating an enhanced image of an environment, where the enhanced image reflects pixel content obtained from cameras of different modalities as well as contextual content representative of a particular property, said method comprising:
generating a first image of an environment using a first camera of a first modality;
generating a second image of the environment using a second camera of a second modality;
identifying pixels that are common between the first image and the second image;
generating an alpha map that reflects edge detection weights that have been computed for each one of the common pixels based on a first saliency generated for the first image and a second saliency generated for the second image, wherein generating the alpha map includes generating a low frequency alpha map, and wherein generating the low frequency alpha map includes:
downscaling the first image twice to generate a downscaled image;
downscaling the second image twice;
after the downscaled image is generated, applying a set of filtering operations to the downscaled image to generate a filtered image, the filtering operations include applying a Sobel filter on the downscaled image and then applying a Gaussian filter to generate the filtered image;
after the second image is downscaled twice, applying the Sobel filter on the second image;
after the Sobel filter has been applied to the second image, applying the Gaussian filter on the second image;
generating a low frequency saliency map by upscaling the filtered image twice after the Gaussian filter has been applied;
generating a second low frequency saliency map by upscaling the second image twice after the Gaussian filter has been applied; and
generating the low frequency alpha map by dividing the second low frequency saliency map by a sum of the first low frequency saliency map and the second low frequency saliency map;
determining how much texture from the first image and/or from the second image to use to generate an enhanced image, said determining being based on the edge detection weights included within the alpha map;
based on the edge detection weights, merging textures from the common pixels included in the first image and the second image to generate the enhanced image; and
adding color to the enhanced image to reflect an additional property that is associated with one or both of the first image or the second image.