US 12,230,399 B2
Multimodal fusion for diagnosis, prognosis, and therapeutic response prediction
Faisal Mahmood, Brookline, MA (US); and Richard Chen, Gaithersburg, MD (US)
Assigned to THE BRIGHAM AND WOMEN'S HOSPITAL, INC., Boston, MA (US)
Appl. No. 17/762,125
Filed by THE BRIGHAM AND WOMEN'S HOSPITAL, INC., Boston, MA (US)
PCT Filed Sep. 28, 2020, PCT No. PCT/US2020/053079
§ 371(c)(1), (2) Date Mar. 21, 2022,
PCT Pub. No. WO2021/062366, PCT Pub. Date Apr. 1, 2021.
Claims priority of provisional application 62/907,096, filed on Sep. 27, 2019.
Prior Publication US 2022/0367053 A1, Nov. 17, 2022
Int. Cl. G16H 50/20 (2018.01); G06T 7/00 (2017.01); G06V 10/80 (2022.01); G06V 20/69 (2022.01); G16B 30/00 (2019.01); G16B 40/20 (2019.01)
CPC G16H 50/20 (2018.01) [G06T 7/0012 (2013.01); G06V 10/806 (2022.01); G06V 20/695 (2022.01); G16B 30/00 (2019.02); G16B 40/20 (2019.02); G06T 2207/10056 (2013.01); G06T 2207/20036 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20221 (2013.01); G06T 2207/30096 (2013.01); G06V 2201/03 (2022.01)] 17 Claims
OG exemplary drawing
 
1. A method comprising:
receiving, by a system comprising a processor, histopathology information comprising at least one whole slide image of at least part of a diseased portion of a subject from a first data source;
determining, by the system, a first matrix of histology features, wherein the histology features are of a first data type and comprise image-based histology features and graph-based histology features, wherein the first matrix is of a first size, comprising:
extracting, with a convoluted neural network, the image-based histology features from a region of interest of the histopathology information to reveal any inherent phenotypic intratumoral heterogeneity of the region of interest of the histopathology information; and
extracting, with a graph convolution network, the graph-based histology features from the region of interest of the histopathology information to reveal spatial heterogeneity of cells in the region of interest;
receiving, by the system, omics data related to the diseased portion of the subject from a second data source that is different from the first data source;
determining, by the system, a second matrix of molecular features from the omics data, wherein the molecular features are of a second data type, wherein the second matrix is of a second size;
fusing, by the system, the first matrix of histology features and the second matrix of molecular features by a tensor fusion process to form a third matrix, wherein the tensor fusion process calculates an outer product space of the histology features and the molecular features and captures the space of all possible interactions between the histology features and the molecular features;
determining, by the system, a diagnosis, a prognosis, and/or a therapeutic response profile for the diseased portion of the subject based on at least a portion of the third matrix; and
outputting, by the system, the diagnosis, the prognosis, and/or the therapeutic response profile for the diseased portion of the subject.