US 12,086,198 B2
Embedding based retrieval for image search
Suddha Kalyan Basu, San Jose, CA (US); Wei Fan, Sunnyvale, CA (US); Daniel Glasner, New York, NY (US); Sushrut Suresh Karanjkar, Fremont, CA (US); Thomas Richard Strohmann, Cupertino, CA (US); Shubhang Verma, Mountain View, CA (US); Manas Ashok Pathak, Mountain View, CA (US); Wenyuan Yin, Cupertino, CA (US); and Sundeep Tirumalareddy, Mountain View, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Sep. 5, 2023, as Appl. No. 18/461,049.
Application 18/461,049 is a continuation of application No. 17/277,820, granted, now 11,782,998, previously published as PCT/US2020/020459, filed on Feb. 28, 2020.
Prior Publication US 2023/0409653 A1, Dec. 21, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 16/50 (2019.01); G06F 16/538 (2019.01); G06F 16/583 (2019.01); G06F 16/587 (2019.01); G06F 16/9538 (2019.01); G06N 3/02 (2006.01); G06F 16/90 (2019.01)
CPC G06F 16/9538 (2019.01) [G06F 16/538 (2019.01); G06F 16/583 (2019.01); G06F 16/587 (2019.01); G06N 3/02 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A computer-implemented method, the method comprising:
receiving, by a computing system comprising one or more processors, an image search query;
processing, by the computing system, the image search query with a query embedding model to generate a search query embedding, wherein the search query embedding is descriptive of one or more features of the image search query, wherein the query embedding model was trained by:
obtaining, by the computing system, a training dataset, wherein the training dataset comprises a plurality of training image search queries and a plurality of training image search results, wherein each of the plurality of training image search results is associated with one or more respective training images search queries of the plurality of training image search queries;
processing, by the computing system, a first training image search query of the plurality of training image search queries with the query embedding model to generate a first query embedding, wherein the first query embedding is descriptive of one or more features of the first training image search query;
processing, by the computing system, a first image and a first landing page with a pair embedding model to generate a first pair numeric embedding, wherein the first pair numeric embedding is descriptive of one or more features of the first image and one or more features of the first landing page, wherein the first image and the first landing page are associated with a first training image search result of the plurality of training image search results, wherein the first training image search result is associated with the first training image search query, wherein the first training image search result is a positive training example;
evaluating, by the computing system, a loss function that evaluates a difference between the first query embedding and the first pair numeric embedding, wherein evaluating the loss function comprises generating a gradient that when propagated to the query embedding model adjusts the one or more parameters to increase embedding similarities for embeddings generated based on the first training image search query and the first training image search result; and
adjusting, by the computing system, one or more parameters of the query embedding model based at least in part on the loss function;
determining, by the computing system, a plurality of pair numeric embeddings are associated with the search query embedding, wherein the plurality of pair numeric embeddings are descriptive of a plurality of image-landing page pairs, wherein each of the plurality of pair numeric embeddings are associated with a respective image-landing page pair; and
providing, by the computing system, the plurality of image-landing page pairs as search results.