US 12,316,820 B2
Image processing apparatus, method, system, and storage medium to generate a virtual viewpoint image based on a captured image
Shogo Mizuno, Tokyo (JP)
Assigned to Canon Kabushiki Kaisha, Tokyo (JP)
Filed by CANON KABUSHIKI KAISHA, Tokyo (JP)
Filed on May 3, 2023, as Appl. No. 18/311,747.
Claims priority of application No. 2022-078716 (JP), filed on May 12, 2022.
Prior Publication US 2023/0370575 A1, Nov. 16, 2023
Int. Cl. H04N 13/117 (2018.01); G06T 5/50 (2006.01); G06T 7/70 (2017.01); G06T 17/00 (2006.01); H04N 13/279 (2018.01)
CPC H04N 13/117 (2018.05) [G06T 5/50 (2013.01); G06T 7/70 (2017.01); G06T 17/00 (2013.01); H04N 13/279 (2018.05); G06T 2207/20221 (2013.01)] 19 Claims
OG exemplary drawing
 
1. An image processing apparatus comprising:
one or more memories storing instructions; and
one or more processors executing the instructions to:
obtain virtual viewpoint information indicating a position and a direction of a virtual viewpoint;
obtain a three-dimensional subject model to be generated based on captured images obtained by capturing with a plurality of image capturing apparatuses;
predict a virtual viewpoint in a second frame subsequent to a first frame in a virtual viewpoint image based on a virtual viewpoint in a frame prior to the first frame;
predict a position of the three-dimensional subject model in the second frame based on the position of the three-dimensional subject model corresponding to the frame prior to the first frame;
determine an image capturing apparatus from which a captured image to be used for generating the second frame is obtained from among the plurality of image capturing apparatuses based on a predicted virtual viewpoint, a predicted position of the three-dimensional subject model, and image capturing parameters of the plurality of image capturing apparatuses; and
generate the virtual viewpoint image based on a captured image corresponding to the second frame obtained from the determined image capturing apparatus, the three-dimensional subject model corresponding to the obtained second frame, and the virtual viewpoint information corresponding to the obtained second frame.