US 12,273,623 B2
Hybrid dynamic vision sensor and CMOS device for motion compensation
Yan Ye, Singapore (SG); Zhongyang Huang, Singapore (SG); and Guansong Liu, Cupertino, CA (US)
Assigned to OMNIVISION TECHNOLOGIES, INC., Santa Clara, CA (US)
Filed by Omnivision Technologies, Inc., Santa Clara, CA (US)
Filed on Aug. 5, 2021, as Appl. No. 17/394,944.
Prior Publication US 2023/0042364 A1, Feb. 9, 2023
Int. Cl. G06K 9/00 (2022.01); G06T 7/246 (2017.01); H04N 23/45 (2023.01); H04N 23/68 (2023.01)
CPC H04N 23/683 (2023.01) [G06T 7/248 (2017.01); H04N 23/45 (2023.01); H04N 23/6811 (2023.01); G06T 2207/20016 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A method of image processing, comprising:
simultaneously obtaining a first component and a second component of an optical input by a hybrid imaging device comprising a first type sensor and a second type sensor, wherein the first component of the optical input corresponds to the first type sensor and the second component of the optical input corresponds to the second type sensor;
performing feature extraction and matching based on the first component of the optical input, and identifying steady image data and shaky image data from the optical input based on intensity change at pixel-levels of the first component of the optical input;
performing motion compensation on the first component of the optical input;
reconstructing a plurality of image frames based on both the motion compensated first component of the optical input and the second component of the optical input;
constructing a plurality of additional image frames based on both the first and second components of the optical input without motion compensation;
inserting the additional image frames between the reconstructed image frames according to a local frame rate to generate a local output frame rate greater than a predetermined global frame rate;
keeping the local frame rate of the additional frames around the reconstructed image frame corresponding to the steady image data with low intensity change at the pixel-levels;
increasing the local frame rate of the additional image frames around the reconstructed image frame corresponding to the shaky image data with the high intensity change at the pixel-levels; and
outputting the reconstructed image frames at the predetermined global frame rate and the inserted additional image frames at the local output frame rate to a display device,
wherein a plurality of first type sensor cells of the first type sensor and a plurality of second type sensor cells of the second type sensor are arranged in rows and columns of an array, and the second type sensor cells are divided into red sensors, green sensors and blue sensors, and
wherein in each row comprising the first type sensor cells, the second type sensor cells are free of the blue sensors, and in each column comprising the first type sensor cells, the second type sensor cells are free of the red sensors.