US 11,782,998 B2
Embedding based retrieval for image search
Suddha Kalyan Basu, San Jose, CA (US); Wei Fan, Sunnyvale, CA (US); Daniel Glasner, New York, NY (US); Sushrut Suresh Karanjkar, Fremont, CA (US); Thomas Richard Strohmann, Cupertino, CA (US); Shubhang Verma, Mountain View, CA (US); Manas Ashok Pathak, Mountain View, CA (US); Wenyuan Yin, Cupertino, CA (US); and Sundeep Tirumalareddy, Mountain View, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Appl. No. 17/277,820
Filed by Google LLC, Mountain View, CA (US)
PCT Filed Feb. 28, 2020, PCT No. PCT/US2020/020459
§ 371(c)(1), (2) Date Mar. 19, 2021,
PCT Pub. No. WO2021/173158, PCT Pub. Date Sep. 2, 2021.
Prior Publication US 2022/0012297 A1, Jan. 13, 2022
Int. Cl. G06F 16/50 (2019.01); G06N 3/02 (2006.01); G06F 16/9538 (2019.01); G06F 16/583 (2019.01); G06F 16/538 (2019.01); G06F 16/587 (2019.01); G06F 16/90 (2019.01); G06N 3/04 (2023.01); G06N 3/08 (2023.01)
CPC G06F 16/9538 (2019.01) [G06F 16/538 (2019.01); G06F 16/583 (2019.01); G06F 16/587 (2019.01); G06N 3/02 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A method comprising:
receiving an image search query;
determining a respective pair numeric embedding for each of a plurality of image-landing page pairs, each image-landing page pair including a respective image and a respective landing page for the respective image, wherein each pair numeric embedding is a numeric representation in an embedding space, wherein determining a respective pair numeric embedding for each of a plurality of image-landing page pairs comprises:
accessing an index database that associates image-landing page pairs with corresponding pair numeric embeddings that have been generated for the image-landing page pairs using a pair embedding neural network, wherein the pair embedding neural network and the image search query embedding neural network have been trained jointly to minimize a loss function that depends on a dot product between (i) a query numeric embedding for a training image search query and (ii) a pair numeric embedding for a training image-landing page pair;
processing features of the image search query using an image search query embedding neural network to generate a query numeric embedding of the image search query, and wherein the query numeric embedding is a numeric representation in the same embedding space;
identifying, as first candidate image search results for the image search query, image search results that identify a subset of the image-landing page pairs having pair numeric embeddings that are closest to the query numeric embedding of the image search query in the embedding space;
ranking a plurality of second candidate image search results that comprises at least some of the first candidate image search results;
generating an image search results presentation that displays the second candidate image search results ordered according to the ranking; and
providing the image search results presentation for presentation by a user device.