	US 12,143,731 B1
	Highlight recovery for image sensors
Bartlomiej Pawlik, Singapore (SG)
Assigned to AMAZON TECHNOLOGIES, INC., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Jun. 30, 2022, as Appl. No. 17/854,656.
Int. Cl. H04N 23/88 (2023.01); G06T 7/90 (2017.01); H04N 9/64 (2023.01)

CPC H04N 23/88 (2023.01) [G06T 7/90 (2017.01); H04N 9/646 (2013.01); G06T 2207/10024 (2013.01)]

22 Claims

1. A method comprising:

receiving a first frame of image data representing a plurality of pixels arranged in a grid, wherein a first pixel of the plurality of pixels is associated with an input red pixel value C_R, an input green pixel value C_G, an input blue pixel value C_B, and an input infrared pixel value C_IR;

determining a white-balanced red pixel value C_R^wbfor the first pixel by multiplying the input red pixel value C_Rby a white balance gain corresponding to a red channel WB_R;

determining a clipped, white-balanced red pixel value C_R^wb,clippedby replacing white-balanced pixel values of C_R^wbgreater than 1 with a value of 1 and replacing white-balanced pixel values of C_R^wbless than 0 with a value of 0;

determining a color-corrected red pixel value C_R^ccm, a color-corrected green pixel value C_G^ccm, and a color-corrected blue pixel value C_B^ccmby multiplying a 1×4 matrix by a 4×3 color correction matrix, wherein the 1×4 matrix comprises the clipped, white-balanced red pixel value C_R^WB,clipped, a clipped, white-balanced green pixel value C_G^wb,clipped, a clipped, white-balanced blue pixel value C_B^wb,clipped, and a clipped, white-balanced infrared pixel value C_IR^wb,clipped;

determining a clipped, color-corrected red pixel value C_R^ccm,clippedby replacing color corrected red pixel values C_R^ccmgreater than 1 with the value of 1 and replacing color-corrected red pixel values C_R^ccmless than 0 with the value of 0;

determining an interpolation coefficient α for the first pixel based on the clipped, color-corrected red pixel value C_R^ccm,clipped, a clipped, color-corrected green pixel value C_G^ccm,clipped, and a clipped, color-corrected blue pixel value C_B^ccm,clipped;

determining a first luminance value for the first pixel C^lumausing a weighted combination of the input red pixel value, the input green pixel value, and the input blue pixel value; and

determining a modified red pixel value by interpolating between C^lumaand an unclipped, color-corrected red pixel value C_R^{ccm,unclipped}using the interpolation coefficient α.