US 11,948,245 B2
Relighting images and video using learned lighting and geometry
Shaona Ghosh, Campbell, CA (US); and Mona Zehni, Urbana-Champaign, IL (US)
Assigned to Apple Inc., Cupertino, CA (US)
Filed by Apple Inc., Cupertino, CA (US)
Filed on Nov. 16, 2021, as Appl. No. 17/455,167.
Claims priority of provisional application 63/114,264, filed on Nov. 16, 2020.
Prior Publication US 2022/0157012 A1, May 19, 2022
Int. Cl. G06T 15/50 (2011.01); G06N 3/08 (2023.01); G06T 15/80 (2011.01)
CPC G06T 15/506 (2013.01) [G06N 3/08 (2013.01); G06T 15/80 (2013.01)] 21 Claims
OG exemplary drawing
 
1. A non-transitory program storage device (NPSD) comprising computer readable instructions executable by one or more processors to:
obtain a neural network configured to decompose images into a plurality of lighting-related components, wherein the neural network is trained on real and synthetic images using one or more self-supervision terms, wherein at least one of the one or more self-supervision terms comprises a cross-relighting loss term, and wherein the cross-relighting loss term comprises computing a difference between: (1) a first image that is relit with estimated lighting for a second image; and (2) the second image;
obtain an input image;
use the neural network to decompose the input image into the plurality of lighting-related components;
estimate lighting information for the input image based, at least in part, on the plurality of lighting-related components; and
render a reconstructed version of the input image based, at least in part, on the input image, the estimated lighting information, and the plurality of lighting-related components.