US 11,735,311 B2
Universal image representation based on a multimodal graph
Peer-Timo Bremer, Livermore, CA (US); Rushil Anirudh, Dublin, CA (US); and Jayaraman Jayaraman Thiagarajan, Milpitas, CA (US)
Assigned to LAWRENCE LIVERMORE NATIONAL SECURITY, LLC, Livermore, CA (US)
Filed by Lawrence Livermore National Security, LLC, Livermore, CA (US)
Filed on Sep. 9, 2021, as Appl. No. 17/470,331.
Application 17/470,331 is a division of application No. 16/684,388, filed on Nov. 14, 2019, granted, now 11,145,403.
Prior Publication US 2022/0181006 A1, Jun. 9, 2022
Int. Cl. G06K 9/00 (2022.01); G16H 30/20 (2018.01); G06N 3/08 (2023.01); G06N 20/10 (2019.01); G06T 7/00 (2017.01)
CPC G16H 30/20 (2018.01) [G06N 3/08 (2013.01); G06N 20/10 (2019.01); G06T 7/0014 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30024 (2013.01); G06T 2207/30236 (2013.01)] 21 Claims
OG exemplary drawing
 
1. A method performed by one or more computing systems for classifying a target image with segments having attributes, the method comprising:
identifying segments of the target image;
generating a graph for the target image that includes vertices and edges, each vertex representing a segment and each edge satisfying an edge criterion based on the vertices that the edge connects, each vertex being assigned an attribute of the segment represented by the vertex;
for each vertex, generating a subgraph that includes the vertex as a home vertex and neighboring vertices representing segments of the target image within a neighborhood of the segment represented by the home vertex, the vertices of the subgraph satisfying a neighborhood criterion;
for each subgraph, applying an autoencoder to the subgraph to generate latent variables to represent the subgraph, the autoencoder being trained using subgraphs generated from images and attributes of segments of images of autoencoder training data; and
applying a machine learning algorithm to a feature vector derived from the generated latent variables of the subgraphs to generate a classification for the target image, the machine learning algorithm being trained using feature vectors representing latent variables and labels representing classifications, the latent variables generated by applying the autoencoder to subgraphs for images and attributes of classification machine learning algorithm training data.