US 11,734,352 B2
Cross-modal search systems and methods
Diane Larlus, La Tronche (FR); Jon Almazan, London (GB); Cesar De Souza, Grenoble (FR); Naila Murray, Grenoble (FR); and Rafael Sampaio De Rezende, Grenoble (FR)
Assigned to NAVER CORPORATION, Gyeonggi-do (KR)
Filed by NAVER CORPORATION, Gyeonggi-do (KR)
Filed on Feb. 14, 2020, as Appl. No. 16/791,368.
Prior Publication US 2021/0256068 A1, Aug. 19, 2021
Int. Cl. G06F 16/9032 (2019.01); G06F 16/9038 (2019.01); G06F 17/16 (2006.01); G06F 18/22 (2023.01); G06F 18/214 (2023.01); G06V 10/74 (2022.01); G06V 10/764 (2022.01); G06V 10/82 (2022.01)
CPC G06F 16/9032 (2019.01) [G06F 16/9038 (2019.01); G06F 16/90332 (2019.01); G06F 17/16 (2013.01); G06F 18/2148 (2023.01); G06F 18/22 (2023.01); G06V 10/761 (2022.01); G06V 10/764 (2022.01); G06V 10/82 (2022.01)] 15 Claims
OG exemplary drawing
 
1. A system for training a cross-modal search system, comprising:
a training dataset including first objects of a first modality and second objects of a second modality that are associated with the first objects, respectively,
wherein the first modality is different than the second modality, and
wherein the second objects include text that is descriptive of the first objects;
a first matrix including first relevance values indicative of relevance between the first objects and the second objects, respectively;
a second matrix including second relevance values indicative of relevance between the second objects and the first objects, respectively; and
a training module configured to:
based on similarities between ones of the second objects, generate a third matrix by selectively adding first additional relevance values to the first matrix;
based on the similarities between the ones of the second objects, generate a fourth matrix by selectively adding second additional relevance values to the second matrix; and
store the third and fourth matrices in memory of a search module for cross-modal retrieval in response to receipt of search queries.