US 11,954,902 B2
Generalizable medical image analysis using segmentation and classification neural networks
Jeffrey De Fauw, London (GB); Joseph R. Ledsam, Tokyo (JP); Bernardino Romera-Paredes, London (GB); Stanislav Nikolov, London (GB); Nenad Tomasev, London (GB); Samuel Blackwell, London (GB); Harry Askham, London (GB); Xavier Glorot, Montréal (CA); Balaji Lakshminarayanan, Sunnyvale, CA (US); Trevor Back, Saffron Walden (GB); Mustafa Suleyman, London (GB); Pearse A. Keane, London (GB); Olaf Ronneberger, London (GB); and Julien Robert Michel Cornebise, London (GB)
Assigned to Google LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Dec. 8, 2020, as Appl. No. 17/114,586.
Application 17/114,586 is a continuation of application No. 16/236,045, filed on Dec. 28, 2018, granted, now 10,878,601.
Application 16/236,045 is a continuation of application No. 16/022,170, filed on Jun. 28, 2018, granted, now 10,198,832, issued on Feb. 5, 2019.
Claims priority of provisional application 62/586,822, filed on Nov. 15, 2017.
Claims priority of provisional application 62/526,293, filed on Jun. 28, 2017.
Prior Publication US 2021/0118198 A1, Apr. 22, 2021
Int. Cl. G06V 10/82 (2022.01); G06F 18/21 (2023.01); G06F 18/2413 (2023.01); G06F 18/25 (2023.01); G06T 7/00 (2017.01); G06T 11/00 (2006.01); G06V 10/44 (2022.01); G06V 10/764 (2022.01); G06V 10/80 (2022.01)
CPC G06V 10/82 (2022.01) [G06F 18/217 (2023.01); G06F 18/24133 (2023.01); G06F 18/254 (2023.01); G06T 7/0012 (2013.01); G06T 11/003 (2013.01); G06V 10/454 (2022.01); G06V 10/764 (2022.01); G06V 10/809 (2022.01); G06T 2207/10101 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30041 (2013.01); G06V 2201/03 (2022.01)] 19 Claims
OG exemplary drawing
 
1. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement:
a first set of one or more segmentation neural networks, wherein each segmentation neural network in the first set is configured to:
receive an input image of eye tissue captured using a first imaging modality; and
process the input image to generate a segmentation map that segments the eye tissue in the input image into a plurality of tissue types;
a set of one or more classification neural networks, wherein each classification neural network is configured to:
receive a classification input comprising a segmentation map of eye tissue; and
process the classification input to generate a classification output that characterizes the eye tissue; and
a subsystem configured to:
receive a first image of eye tissue captured using the first imaging modality, wherein the first image of eye tissue captured using the first modality is a three-dimensional image comprising a plurality of voxels;
provide the first image as input to each of the segmentation neural networks in the first set to obtain one or more three-dimensional segmentation maps of the eye tissue in the first image, wherein each of the one or more three-dimensional segmentation maps assigns, to each of the voxels: (i) a respective tissue type from a predetermined set of tissue types, or (ii) a respective tissue type probability distribution over the predetermined set of tissue types;
generate, from each of the three-dimensional segmentation maps, a respective classification input that comprises the three-dimensional segmentation map; and
provide, for each of the three-dimensional segmentation maps, the classification input for the three-dimensional segmentation map as input to each of the classification neural networks to obtain, for each three-dimensional segmentation map, a respective classification output from each classification neural network, wherein the classification outputs of the classification neural network characterize predicted global properties of the first image of the eye tissue; and
generate, from the respective classification outputs for each of the three-dimensional segmentation maps, a final classification output for the first image.