US 11,961,193 B2
Method for controlling a display, computer program and mixed reality display device
Sirko Pelzl, Hamburg (DE); and Michael Wozniak, Poznan (PL)
Assigned to APOQLAR GMBH, Hamburg (DE)
Appl. No. 17/634,982
Filed by APOQLAR GMBH, Hamburg (DE)
PCT Filed Sep. 9, 2020, PCT No. PCT/EP2020/075127
§ 371(c)(1), (2) Date Feb. 13, 2022,
PCT Pub. No. WO2021/048158, PCT Pub. Date Mar. 18, 2021.
Claims priority of application No. 19196193 (EP), filed on Sep. 9, 2019.
Prior Publication US 2022/0292786 A1, Sep. 15, 2022
Int. Cl. G06T 19/00 (2011.01); G06T 7/10 (2017.01); G06T 7/194 (2017.01); H04N 13/111 (2018.01); H04N 13/156 (2018.01)
CPC G06T 19/006 (2013.01) [G06T 7/194 (2017.01); H04N 13/111 (2018.05); H04N 13/156 (2018.05); G06T 2200/04 (2013.01); G06T 2207/30041 (2013.01); G06T 2210/41 (2013.01); G06T 2210/56 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for controlling a display of a mixed reality display device, comprising:
a) providing an image data set comprising a plurality of images of a treatment object, wherein the treatment object is a patient's body or a part thereof and images of the plurality of images depict the treatment object from different perspectives,
b) generating a 3D target point cloud from the image data set, wherein the target point cloud comprises a first multitude of points defined in a three-dimensional coordinate system and the first multitude of points represent a surface of the treatment object,
c) determining a number of semantic segmentation masks in the target point cloud by applying semantic segmentation,
d) providing a medical imaging data set that comprises medical imaging data of the treatment object,
e) generating a 3D source point cloud from the medical imaging data set, wherein the source point cloud comprises a second multitude of points defined in a three-dimensional coordinate system and the second multitude of points also represent the surface of the treatment object,
f) determining a number of semantic segmentation masks in the source point cloud by applying semantic segmentation,
g) determining a transformation between the source point cloud and the target point cloud using the segmentation masks of the source point cloud and the segmentation masks of the target point cloud, and
h) visibly displaying at least a part of the medical imaging data on the display while permitting a user to view the treatment object, wherein the medical imaging data is displayed superimposed on the treatment object and aligned with the treatment object using the transformation between the source point cloud and the target point cloud,
wherein step c) comprises
determining a number of semantic segmentation masks in the images of the image data set by applying semantic segmentation to the images of the image data set, and
determining the semantic segmentation masks in the target point cloud using the semantic segmentation masks in the images of the image data set
and/or step f) comprises
determining a number of semantic segmentation masks in the medical imaging data of the medical imaging data set by applying semantic segmentation to the medical imaging data of the medical imaging data set, and
determining the semantic segmentation masks in the source point cloud using the semantic segmentation masks in the medical imaging data of the medical imaging data set,
wherein each semantic segmentation mask refers to a segment of the image that has been determined, using semantic segmentation, as belonging to the same object class.