US 12,340,540 B2
Imaging sensor, an image processing device and an image processing method
Piergiorgio Sartor, Stuttgart (DE); Alexander Gatto, Stuttgart (DE); Vincent Parret, Stuttgart (DE); and Adriano Simonetto, Stuttgart (DE)
Assigned to SONY GROUP CORPORATION, Tokyo (JP)
Appl. No. 17/774,868
Filed by Sony Group Corporation, Tokyo (JP)
PCT Filed Nov. 12, 2020, PCT No. PCT/EP2020/081926
§ 371(c)(1), (2) Date May 6, 2022,
PCT Pub. No. WO2021/094463, PCT Pub. Date May 20, 2021.
Claims priority of application No. 19209390 (EP), filed on Nov. 15, 2019.
Prior Publication US 2022/0405972 A1, Dec. 22, 2022
Int. Cl. G06T 7/90 (2017.01); G06T 3/4053 (2024.01)
CPC G06T 7/90 (2017.01) [G06T 3/4053 (2013.01); G06T 2207/10024 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 5 Claims
OG exemplary drawing
 
1. An image processing device comprising circuitry configured to:
obtain global image data including image data and spectral data, the global image data representing a global image area, wherein the spectral data represents spectral information acquired from a plurality of spectral imaging portions distributed in a photosensitive area of an imaging sensor, and wherein the image data represents imaging information acquired from an imaging part of the photosensitive area;
input the global image data to a convolutional neural network for generating output spectral data, wherein the convolutional neural network is configured to transform the obtained image data into the output spectral data based on the obtained spectral data, wherein the global image area includes more pixels represented by the obtained image data than pixels represented by the obtained spectral data,
wherein the convolutional neural network is trained using:
a back transformation that converts predicted spectral data to back-transformed image data using a physical model that integrates the spectrum according to color sensitivities;
a first loss function computed between the obtained image data and the back-transformed image data;
a second loss function computed between the obtained spectral data and the predicted spectral data; and
a smoothening constraint applied to the predicted spectral data.