US 11,790,610 B2
Systems and methods for selective image compositing
Matthew Thomas, San Francisco, CA (US); Francisco Avila-Beltran, San Francisco, CA (US); David Royston Cutts, San Francisco, CA (US); William Castillo, San Francisco, CA (US); Giridhar Murali, San Francisco, CA (US); Brandon Scott, San Francisco, CA (US); and Jeffrey Sommers, San Francisco, CA (US)
Assigned to Hover Inc., San Francisco, CA (US)
Filed by HOVER INC., San Francisco, CA (US)
Filed on Nov. 10, 2020, as Appl. No. 17/94,311.
Claims priority of provisional application 63/070,816, filed on Aug. 26, 2020.
Claims priority of provisional application 62/935,630, filed on Nov. 14, 2019.
Claims priority of provisional application 62/933,939, filed on Nov. 11, 2019.
Prior Publication US 2021/0142577 A1, May 13, 2021
Int. Cl. G06T 19/00 (2011.01); G06T 3/00 (2006.01); G06T 17/20 (2006.01); G06T 19/20 (2011.01); G06V 20/64 (2022.01); G06V 20/10 (2022.01); G06F 18/214 (2023.01)
CPC G06T 19/006 (2013.01) [G06F 18/214 (2023.01); G06T 3/0093 (2013.01); G06T 17/20 (2013.01); G06T 19/20 (2013.01); G06V 20/176 (2022.01); G06V 20/647 (2022.01); G06T 2219/2024 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
receiving a two-dimensional (2D) image and metadata, the 2D image including a set of pixels depicting a physical structure captured by an image capturing device, and the metadata representing one or more characteristics of the image capturing device;
identifying a portion of the 2D image to augment with synthetic image data;
segmenting the set of pixels of the 2D image into one or more subsets of pixels;
identifying, from amongst the one or more subsets of pixels, a subset of pixels corresponding to the identified portion of the 2D image;
generating a reference three-dimensional (3D) model of the physical structure from the 2D image, the reference 3D model representing the identified portion of the 2D image in a virtual space, and the generation including determining a 3D orientation of a 3D planar surface of the reference 3D model, wherein determining the 3D orientation comprises:
associating the 3D planar surface with the identified portion of the 2D image to augment with the synthetic image data;
extracting a surface normal value of the identified portion of the 2D image; and
orienting the reference 3D model to align the 3D planar surface according to the extracted surface normal;
applying the synthetic image data onto the reference 3D model;
reprojecting a select 3D geometry of the reference 3D model with the synthetic image data over the portion of the 2D image according to the identified subset of pixels; and
rendering a photorealistic image using the 2D image, the metadata, and the reprojected selected 3D geometry with the synthetic image data.