US 12,406,142 B2
Situational awareness by fusing multi-modal data with semantic model
Anuradha Bhamidipaty, Yorktown Heights, NY (US); Elham Khabiri, Briarcliff Manor, NY (US); Shuxin Lin, Yorktown Heights, NY (US); Jeffrey Owen Kephart, Cortlandt Manor, NY (US); Yingjie Li, Chappaqua, NY (US); and Bhavna Agrawal, Armonk, NY (US)
Assigned to International Business Machines Corporation, Armonk, NY (US)
Filed by INTERNATIONAL BUSINESS MACHINES CORPORATION, Armonk, NY (US)
Filed on Jul. 20, 2020, as Appl. No. 16/933,964.
Prior Publication US 2022/0019742 A1, Jan. 20, 2022
Int. Cl. G06F 40/30 (2020.01); G06F 40/295 (2020.01); G06N 5/02 (2023.01)
CPC G06F 40/30 (2020.01) [G06F 40/295 (2020.01); G06N 5/02 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A computer implemented method, comprising:
receiving data from one or more input sources in an entity and relationship capture service of a situational awareness engine;
extracting entities and relationships between the entities in two or more different extraction services, the two or more different extraction services including at least two of a table-to-graph service, an event-to-graph service, a sensor-to-graph service, a text-to-graph service, and an image-to-graph service;
upon identifying missing or uncertain entities and/or relationships to generate a semantic model by way of a knowledge graph, eliciting an input to correct the missing or uncertain entities and/or relationships;
generating the semantic model based on fusion and labeling of extracted data provided by the at least two extraction services;
reconciling entities, of the entities and relationships extracted by the two or more different extraction services, into a single node in the knowledge graph, upon determining that the entities should be regarded as a same entity, by assigning a same unique node identifier, merging their properties, and merging their relationships;
receiving a search query; and
responding to the search query based on the generated semantic model.