US 11,755,638 B2
Method of personalized image and video searching based on a natural language query, and an apparatus for the same
Haotian Zhang, Toronto (CA); Allan Jepson, Toronto (CA); Iqbal Ismail Mohomed, Toronto (CA); Konstantinos Derpanis, Toronto (CA); Ran Zhang, Toronto (CA); and Afsaneh Fazly, Toronto (CA)
Assigned to SAMSUNG ELECTRONICS CO., LTD., Suwon-si (KR)
Filed by SAMSUNG ELECTRONICS CO., LTD., Suwon-si (KR)
Filed on Sep. 2, 2021, as Appl. No. 17/465,408.
Claims priority of provisional application 63/151,293, filed on Feb. 19, 2021.
Prior Publication US 2022/0269719 A1, Aug. 25, 2022
Int. Cl. G06F 16/00 (2019.01); G06F 16/535 (2019.01); G06N 20/00 (2019.01)
CPC G06F 16/535 (2019.01) [G06N 20/00 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method of personalized image retrieval comprising:
obtaining a natural language query including a name;
replacing the name in the natural language query with a generic term to provide an anonymized query and named entity information;
obtaining a plurality of initial ranking scores and a plurality of attention weights corresponding to a plurality of images using a trained scoring model that inputs the anonymized query and the plurality of images;
obtaining a plurality of delta scores corresponding to the plurality of images using a re-scoring model that inputs the plurality of attention weights and the named entity information; and
obtaining a plurality of final ranking scores by modifying the plurality of initial ranking scores based on the plurality of delta scores,
wherein the trained scoring model performs semantic based searching; and
wherein the re-scoring model determines a probability that one or more faces detected in the plurality of images correspond to the name.