US 12,118,052 B2
Automatic navigation of interactive web documents
Aleksandra Faust, Palo Alto, CA (US); Dilek Hakkani-Tur, Los Altos, CA (US); Izzeddin Gur, Goleta, CA (US); and Ulrich Rueckert, San Francisco, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by GOOGLE LLC, Mountain View, CA (US)
Filed on Aug. 16, 2023, as Appl. No. 18/234,766.
Application 18/234,766 is a continuation of application No. 17/280,027, granted, now 11,734,375, previously published as PCT/US2019/053569, filed on Sep. 27, 2019.
Claims priority of provisional application 62/737,843, filed on Sep. 27, 2018.
Prior Publication US 2023/0394102 A1, Dec. 7, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 16/954 (2019.01); G06F 16/953 (2019.01); G06N 3/04 (2023.01)
CPC G06F 16/954 (2019.01) [G06F 16/953 (2019.01); G06N 3/04 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A method implemented using one or more processors, comprising:
determining a natural language input, wherein the natural language input comprises a command to perform a task;
analyzing the natural language input to identify one or more key-value pairs;
retrieving an interactive web document that is operable via a graphical user interface (“GUI”) to perform the task, wherein the interactive web document includes one or more constituent interactive document object model (DOM) elements that are operable to input one or more values of the one or more key-value pairs;
encoding the one or more key-value pairs into one or more instruction feature vectors;
encoding one or more words of the one or more key-value pairs that overlap with one or more attributes of one or more of the interactive DOM elements into one or more overlap feature vectors;
encoding the one or more interactive DOM elements of the interactive web document into one or more interactive DOM element feature vectors;
conditioning the one or more interactive DOM element feature vectors based on the one or more overlap feature vectors to generate one or more conditioned interactive DOM element feature vectors, wherein the conditioning includes performing an operation using the one or more interactive DOM element feature vectors and the one or more overlap feature vectors as operands;
generating, based on the one or more conditioned interactive element feature vectors and the one or more instruction feature vectors, one or more probability distributions over the one or more interactive elements; and
facilitating automated navigation through the interactive web document in response to the natural language input based at least in part on one or more of the probability distributions.