US 12,346,370 B2
Visual intent triggering for visual search
Xi Chen, Bellevue, WA (US); Houdong Hu, Redmond, WA (US); Li Huang, Sammamish, WA (US); Jiapei Huang, Seattle, WA (US); Arun Sacheti, Sammamish, WA (US); Linjun Yang, Sammamish, WA (US); Rui Xia, Vancouver (CA); Kuang-Huei Lee, Redmond, WA (US); Meenaz Merchant, Kirkland, WA (US); and Sean Chang Culatana, Seattle, WA (US)
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Jul. 16, 2018, as Appl. No. 16/036,224.
Prior Publication US 2020/0019628 A1, Jan. 16, 2020
Int. Cl. G06F 16/50 (2019.01); G06F 16/353 (2025.01); G06F 16/583 (2019.01); G06F 18/24 (2023.01); G06N 3/08 (2023.01); G06N 20/00 (2019.01)
CPC G06F 16/50 (2019.01) [G06F 16/353 (2019.01); G06F 16/583 (2019.01); G06F 18/24 (2023.01); G06N 3/08 (2013.01); G06N 20/00 (2019.01); G06T 2210/12 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A computer implemented method, comprising:
receiving an image as a query at a computer-implemented search engine, wherein the image includes an object;
in response to receiving the image as the query, submitting the image to a multilabel classifier of the computer-implemented search engine, where the multilabel classifier is configured to:
identify a plurality of objects in the image;
place bounding boxes in the image, where each of the bounding boxes substantially bounds a corresponding object; and
assign at least one classification label to each bounding box to identify the corresponding objects in the images;
passing at least one classification label and an associated bounding box to a trained suppression model of the computer-implemented search engine, the trained suppression model computing scores for the bounding box and suppressing at least one classification label along with its associated bounding box based upon the scores;
based on an unsuppressed classification label, selecting, by the computer-implemented search engine, a user intent scenario from amongst a predefined set of user intent scenarios, wherein the user intent scenario in the predefined set of user intent scenarios is selectable due to the classification label being assigned to the user intent scenario;
generating, by the computer-implemented search engine, a query suggestion for review by a user who issued the image as the query, wherein the query suggestion is generated based upon the selected user intent scenario;
subsequent to generating the query suggestion, receiving, by the computer-implemented search engine, an indication that the query suggestion has been selected by the user; and
providing, by the computer-implemented search engine, output that is based upon the user intent scenario.