US 12,333,431 B2
Multi-dimensional generative framework for video generation
Song Bai, Singapore (SG); Zhongcong Xu, Singapore (SG); Jiashi Feng, Singapore (SG); Jun Hao Liew, Singapore (SG); and Wenqing Zhang, Singapore (SG)
Assigned to Lemon Inc., Grand Cayman (KY)
Filed by Lemon Inc., Grand Cayman (KY)
Filed on Dec. 9, 2022, as Appl. No. 18/063,843.
Prior Publication US 2024/0193412 A1, Jun. 13, 2024
Int. Cl. G06K 9/00 (2022.01); G06N 3/08 (2023.01)
CPC G06N 3/08 (2013.01) [G06T 2207/20081 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for generating a multi-dimensional video, the method comprising:
providing an input data to a multi-dimensionally aware generator of a multi-dimensional video generative model;
generating the multi-dimensional video from the input data by the multi-dimensionally aware generator by:
inverting the input data into a latent space of the multi-dimensionally aware generator;
synthesizing content of the multi-dimensional video using an appearance component of the multi-dimensionally aware generator and corresponding camera pose and formulating an intermediate appearance code;
developing a synthesis layer for encoding a motion component of the multi-dimensionally aware generator at a plurality of timesteps and formulating an intermediate motion code;
introducing temporal dynamics into the intermediate appearance code and the intermediate motion code; and
generating multi-dimensionally aware spatio-temporal representations of the input data.