US 12,041,360 B2
Capturing and processing of images including occlusions focused on an image sensor by a lens stack array
Kartik Venkataraman, San Jose, CA (US); Amandeep S. Jabbi, San Francisco, CA (US); Robert H. Mullis, Santa Cruz, CA (US); Jacques Duparre, Jena (DE); and Shane Ching-Feng Hu, Fremont, CA (US)
Assigned to Adeia Imaging LLC, San Jose, CA (US)
Filed by Adeia Imaging LLC, San Jose, CA (US)
Filed on Sep. 5, 2023, as Appl. No. 18/461,318.
Application 18/461,318 is a continuation of application No. 17/817,829, filed on Aug. 5, 2022, granted, now 11,792,538.
Application 17/817,829 is a continuation of application No. 16/907,016, filed on Jun. 19, 2020, granted, now 11,412,158, issued on Aug. 9, 2022.
Application 16/907,016 is a continuation of application No. 16/529,522, filed on Aug. 1, 2019, granted, now 10,694,114, issued on Jun. 23, 2020.
Application 16/529,522 is a continuation of application No. 16/199,566, filed on Nov. 26, 2018, granted, now 10,375,319, issued on Aug. 6, 2019.
Application 16/199,566 is a continuation of application No. 15/687,882, filed on Aug. 28, 2017, granted, now 10,142,560, issued on Nov. 27, 2018.
Application 15/687,882 is a continuation of application No. 14/943,009, filed on Nov. 16, 2015, granted, now 9,749,547, issued on Aug. 29, 2017.
Application 14/943,009 is a continuation of application No. 14/704,920, filed on May 5, 2015, granted, now 9,188,765, issued on Nov. 17, 2015.
Application 14/704,920 is a continuation of application No. 14/459,288, filed on Aug. 13, 2014, granted, now 9,049,381, issued on Jun. 2, 2015.
Application 14/459,288 is a continuation of application No. 12/952,134, filed on Nov. 22, 2010, granted, now 8,866,920, issued on Oct. 21, 2014.
Application 12/952,134 is a continuation in part of application No. 12/935,504, granted, now 8,902,321, issued on Dec. 2, 2014, previously published as PCT/US2009/044687, filed on May 20, 2009.
Claims priority of provisional application 61/281,662, filed on Nov. 20, 2009.
Claims priority of provisional application 61/263,339, filed on Nov. 20, 2009.
Claims priority of provisional application 61/054,694, filed on May 20, 2008.
Prior Publication US 2024/0031690 A1, Jan. 25, 2024
Int. Cl. H04N 23/90 (2023.01); G02B 3/00 (2006.01); G02B 5/20 (2006.01); G02B 13/00 (2006.01); G06T 7/50 (2017.01); G06T 7/557 (2017.01); G06T 11/60 (2006.01); G06T 19/20 (2011.01); H04N 5/262 (2006.01); H04N 5/265 (2006.01); H04N 13/128 (2018.01); H04N 13/239 (2018.01); H04N 23/11 (2023.01); H04N 23/13 (2023.01); H04N 23/16 (2023.01); H04N 23/45 (2023.01); H04N 23/54 (2023.01); H04N 23/55 (2023.01); H04N 23/60 (2023.01); H04N 23/69 (2023.01); H04N 23/698 (2023.01); H04N 23/88 (2023.01); H04N 23/951 (2023.01); H04N 25/40 (2023.01); H04N 25/48 (2023.01); H04N 25/581 (2023.01); H04N 25/60 (2023.01); H04N 25/67 (2023.01); H04N 25/705 (2023.01); H04N 25/79 (2023.01); H04N 13/00 (2018.01); H04N 13/257 (2018.01); H04N 25/13 (2023.01); H04N 25/131 (2023.01); H04N 25/133 (2023.01)
CPC H04N 23/90 (2023.01) [G02B 3/0056 (2013.01); G02B 3/0062 (2013.01); G02B 5/20 (2013.01); G02B 5/201 (2013.01); G02B 13/0015 (2013.01); G06T 7/50 (2017.01); G06T 7/557 (2017.01); G06T 11/60 (2013.01); G06T 19/20 (2013.01); H04N 5/262 (2013.01); H04N 5/265 (2013.01); H04N 13/128 (2018.05); H04N 13/239 (2018.05); H04N 23/11 (2023.01); H04N 23/13 (2023.01); H04N 23/16 (2023.01); H04N 23/45 (2023.01); H04N 23/54 (2023.01); H04N 23/55 (2023.01); H04N 23/60 (2023.01); H04N 23/69 (2023.01); H04N 23/698 (2023.01); H04N 23/88 (2023.01); H04N 23/951 (2023.01); H04N 25/41 (2023.01); H04N 25/48 (2023.01); H04N 25/581 (2023.01); H04N 25/60 (2023.01); H04N 25/67 (2023.01); H04N 25/705 (2023.01); H04N 25/79 (2023.01); G06T 2200/04 (2013.01); G06T 2207/10028 (2013.01); G06T 2207/10052 (2013.01); G06T 2207/20221 (2013.01); H04N 2013/0081 (2013.01); H04N 13/257 (2018.05); H04N 25/131 (2023.01); H04N 25/133 (2023.01); H04N 25/135 (2023.01)] 30 Claims
OG exemplary drawing
 
1. A depth-based image processing method that utilizes distance information determined using a plurality of images captured from different viewpoints, comprising:
normalizing, using at least one processor, a set of images captured by an imaging device based upon calibration data stored in a storage device, where the set of images comprises a plurality of images captured from different viewpoints;
measuring, using the at least one processor, parallax between the normalized set of images by:
performing pair-wise measurements between pixels from different images within the set of images to determine similarity of neighborhoods of pixels at different parallax shifts;
keeping a record of the pair-wise measurements and determining parallax shifts that yield the highest similarity between pixels within the set of images based upon the record of the pair-wise measurements; and
disregarding at least one determined parallax shift due to the impact of pixel occlusion;
generating, using the at least one processor, distance information based upon the measured parallax accounting for the viewpoints from which images within the set of images were captured;
processing, using the at least one processor, at least one image from the set of images using the distance information; and
displaying the at least one processed image on a display device.