US 12,243,162 B2
Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system
Thomas Ivan Nonn, Kenmore, WA (US); David Julio Colmenares, Seattle, WA (US); James Andrew Youngquist, Seattle, WA (US); and Adam Gabriel Jones, Seattle, WA (US)
Assigned to PROPRIO INC., Seattle, WA (US)
Filed by Proprio, Inc., Seattle, WA (US)
Filed on May 4, 2023, as Appl. No. 18/312,430.
Application 18/312,430 is a continuation of application No. 17/154,670, filed on Jan. 21, 2021, granted, now 11,682,165.
Claims priority of provisional application 62/963,717, filed on Jan. 21, 2020.
Prior Publication US 2024/0005596 A1, Jan. 4, 2024
Int. Cl. G06K 9/00 (2022.01); G06T 7/00 (2017.01); G06T 7/557 (2017.01); G06T 17/00 (2006.01); G06T 17/20 (2006.01); G06V 10/80 (2022.01); G06V 20/20 (2022.01); G06V 20/64 (2022.01); H04N 13/282 (2018.01); H04N 13/00 (2018.01)
CPC G06T 17/00 (2013.01) [G06T 7/557 (2017.01); G06T 17/20 (2013.01); G06V 10/806 (2022.01); G06V 20/20 (2022.01); G06V 20/647 (2022.01); H04N 13/282 (2018.05); G06T 2207/10052 (2013.01); G06T 2207/20104 (2013.01); G06T 2210/41 (2013.01); H04N 2013/0081 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of determining a depth within a surgical scene, the method comprising:
capturing depth data of anatomy of a patient within the surgical scene undergoing a surgical procedure with a depth sensor;
capturing image data of the anatomy with a plurality of cameras;
generating a point cloud representative of the anatomy based on the depth data from the depth sensor;
identifying a missing region of the point cloud in which the point cloud includes no data or sparse data, wherein the missing region corresponds to a portion of the anatomy that is occluded from the depth sensor;
determining at least one depth value from the point cloud for an area adjacent to the missing region;
processing the image data with a depth processing algorithm to generate depth data for the missing region, wherein processing the image data with the depth processing algorithm includes limiting the depth processing algorithm to determine the depth data for the missing region in a depth range surrounding the at least one depth value; and
merging the depth data for the missing region with the depth data from the depth sensor to generate a merged point cloud representative of the anatomy.