US 11,949,845 B2
Dynamic visual overlay for enhanced terrain perception on remote control construction equipment
Craig Allan Lehocky, Pittsburgh, PA (US); Michael William Bridenbaugh, Sewickley, PA (US); Brad Robert Van de Veer, Washington, IL (US); Jeffrey K. Berry, Yorkville, IL (US); and Jeffrey J. Baker, Pittsburgh, PA (US)
Assigned to Caterpillar Inc., Peoria, IL (US)
Filed by Caterpillar Inc., Peoria, IL (US)
Filed on Aug. 31, 2022, as Appl. No. 17/900,586.
Prior Publication US 2024/0073393 A1, Feb. 29, 2024
Int. Cl. H04N 13/156 (2018.01); H04N 13/128 (2018.01); H04N 13/15 (2018.01)
CPC H04N 13/156 (2018.05) [H04N 13/128 (2018.05); H04N 13/15 (2018.05)] 18 Claims
OG exemplary drawing
 
1. A method of augmenting image data during operation of a machine, the method comprising using at least one hardware processor to:
for each of one or more image frames in input image data captured by a camera mounted on the machine,
acquire three-dimensional (3D) data for the image frame based on an output of a 3D sensor mounted on the machine, wherein the 3D data comprise a depth value for each of a plurality of points;
project the 3D data onto a coordinate system of the image frame, wherein projecting the 3D data onto the coordinate system of the image frame comprises mapping at least a subset of pixels in the image frame to the depth values for the plurality of points;
determine a position of an overlay within the image frame;
generate the overlay at the determined position within the image frame to produce an augmented image frame, wherein the overlay comprises at least one semi-transparent band overlaid on pixels whose depth values are within a depth range defined by the determined position; and
output the augmented image frame to output image data,
wherein the position of the overlay within the augmented one or more image frames is moved along a dimension of the output image data in each of a plurality of cycles.