US 12,462,142 B2
Imaging device utilizing a neural network
Yusuke Koumura, Kanagawa (JP); Koki Inoue, Kanagawa (JP); Ayana Kimotsuki, Kanagawa (JP); and Fumiya Nagashima, Kanagawa (JP)
Assigned to Semiconductor Energy Laboratory Co., Ltd., Kanagawa-ken (JP)
Appl. No. 17/629,499
Filed by SEMICONDUCTOR ENERGY LABORATORY CO., LTD., Atsugi (JP)
PCT Filed Jul. 30, 2020, PCT No. PCT/IB2020/057181
§ 371(c)(1), (2) Date Jan. 24, 2022,
PCT Pub. No. WO2021/028762, PCT Pub. Date Feb. 18, 2021.
Claims priority of application No. 2019-146985 (JP), filed on Aug. 9, 2019.
Prior Publication US 2022/0292332 A1, Sep. 15, 2022
Int. Cl. G06N 3/045 (2023.01); G06N 3/063 (2023.01); G06T 1/60 (2006.01); G06T 7/50 (2017.01)
CPC G06N 3/045 (2023.01) [G06N 3/063 (2013.01); G06T 1/60 (2013.01); G06T 7/50 (2017.01)] 5 Claims
OG exemplary drawing
 
1. A system comprising:
an imaging device and an arithmetic circuit,
wherein the imaging device comprises an imaging portion, a first memory portion, and an arithmetic portion,
wherein the arithmetic circuit comprises a second memory portion,
wherein the imaging portion is configured to convert light reflected by an external object into image data,
wherein the first memory portion is configured to store the image data, a first filter for performing first convolutional processing in a first layer of a first neural network, and a second filter for performing second convolutional processing in a first layer of a second neural network,
wherein the arithmetic portion is configured to perform the first convolutional processing on the image data using the first filter as multiplier data and a partial region of the image data as multiplicand data to generate first data,
wherein the arithmetic portion is configured to perform the second convolutional processing on the image data using the second filter to generate second data,
wherein the second memory portion is configured to store the first data, the second data, and a plurality of filters for performing convolutional processing in a second layer and subsequent layers of the first neural network and in a fourth layer and subsequent layers of the second neural network,
wherein the arithmetic circuit is configured to perform processing in the second layer and the subsequent layers of the first neural network using the first data to output third data from an output layer of the first neural network,
wherein the arithmetic circuit is configured to perform pooling processing on the second data in the second layer of the second neural network to generate fourth data,
wherein the arithmetic circuit is configured to combine the third data and the fourth data in a third layer of the second neural network to generate fifth data, and
wherein the arithmetic circuit is configured to perform processing in the fourth layer and subsequent layers of the second neural network using the fifth data to output a depth map of the image data from an output layer of the second neural network;
wherein the arithmetic circuit is configured to determine an image using the image data obtained through the convolutional processing and the pooling processing in a fully connected layer of the second neural network,
wherein the fully connected layer has a structure in which all nodes in one layer are connected to all nodes in a subsequent layer, and
wherein the arithmetic circuit is configured to generate a three-dimensional image using the image data and the depth map.