US 12,256,050 B2
Image processing apparatus, image processing method, storage medium, manufacturing method of learned model, and image processing system
Yoshinori Kimura, Tochigi (JP)
Assigned to CANON KABUSHIKI KAISHA, Tokyo (JP)
Filed by CANON KABUSHIKI KAISHA, Tokyo (JP)
Filed on Jun. 9, 2021, as Appl. No. 17/342,715.
Claims priority of application No. 2020-103809 (JP), filed on Jun. 16, 2020.
Prior Publication US 2021/0392313 A1, Dec. 16, 2021
Int. Cl. H04N 13/111 (2018.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01); G06T 7/00 (2017.01); H04N 13/282 (2018.01)
CPC H04N 13/111 (2018.05) [G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06T 7/97 (2017.01); H04N 13/282 (2018.05); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 7 Claims
OG exemplary drawing
 
1. An image processing apparatus comprising:
an image sensor including a plurality of pixels, with two light-receiving units only in each of the plurality of pixels, and an optical system that forms images of different viewpoints on the only two light-receiving units in each of the plurality of pixels; and
at least one processor or circuit configured to execute a plurality of tasks including:
an acquisition task configured to acquire (i) two first images made by capturing the same object at two viewpoints separated from each other in a first direction using the image sensor including the only two light-receiving units in each of the plurality of pixels and the optical system that forms the images of the different viewpoints on the only two light-receiving units in each of the plurality of pixels and (ii) an imaging condition associated with the two first images and including both of a focal length of the optical system and an ISO speed of the image sensor;
a feature amount generating task configured to generate two first feature amounts by inputting the two first images, the focal length of the optical system, and the ISO speed of the image sensor into a first neural network in a machine learning model;
a comparing task configured to generate a second feature amount by calculating corresponding points of the two first feature amounts through processing based on a matrix product of the two first feature amounts; and
an image processing task configured to estimate a second image at a viewpoint different from the two viewpoints by inputting the second feature amount into a second neural network in the machine learning model,
wherein the viewpoint different from the two viewpoints is shifted in a second direction orthogonal to the first direction, and
wherein, in the image processing task, an image at a viewpoint other than the two viewpoints is not input into the machine learning model in estimating the second image.