US 11,971,936 B2
Analyzing web pages to facilitate automatic navigation
Gökhan Bakir, Zurich (CH); Andre Elisseeff, Basel (CH); Torsten Marek, Zurich (CH); João Paulo Pagaime da Silva, Adliswil (CH); Mathias Carlen, Zurich (CH); Dana Ritter, Horgen (CH); Lukasz Suder, Zurich (CH); Ernest Galbrun, Mulhouse Haut-Rhin (FR); Matthew Stokes, Langnau am Albis (CH); Marcin Nowak-Przygodzki, Bäch (CH); Mugurel-Ionut Andreica, Adliswil (CH); and Marius Dumitran, Buchare (RO)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by GOOGLE LLC, Mountain View, CA (US)
Filed on Oct. 26, 2022, as Appl. No. 17/974,086.
Application 17/974,086 is a continuation of application No. 16/619,777, granted, now 11,487,832, previously published as PCT/US2019/031529, filed on May 9, 2019.
Claims priority of provisional application 62/737,822, filed on Sep. 27, 2018.
Prior Publication US 2023/0050054 A1, Feb. 16, 2023
Int. Cl. G06F 16/00 (2019.01); G06F 16/9032 (2019.01); G06F 16/9535 (2019.01)
CPC G06F 16/9535 (2019.01) [G06F 16/90332 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method implemented using one or more processors, comprising:
determining an identifier of a web page, wherein the identifier was previously validated to indicate that the web page was last known to be operable to carry out an intent of a user, and wherein the web page was last known to include a first set of one or more interactive elements that were known to be operable to input one or more parameters associated with the intent;
retrieving a latest version of the web page using the identifier;
analyzing the updated version of the web page to identify a second set of one or more interactive elements;
generating a first set of universal identifier matrices, wherein the generating includes, for each interactive element of the identified second set, generating a respective universal identifier matrix comprising a two-dimensional grid of cells that semantically captures the interactive element and its surrounding visual context, wherein each cell of the two-dimensional grid has a predetermined width and height, and the generating includes:
positioning the two-dimensional grid over the interactive element to spatially align the underlying interactive element with a central cell of the two-dimensional grid such that other pieces of content that surround the interactive element are overlaid by one or more other cells of the two-dimensional grid that are spatially adjacent the central cell, and
including content that is captured spatially within each cell of the two-dimensional grid of cells in the universal identifier matrix;
determining a reference set of universal identifier matrices generated based on the first set of one or more interactive elements previously known to be included with a prior version of the web page associated with the identifier;
calculating similarity measures between the universal identifier matrices of the first set and each of the universal identifier matrices of the known set;
based on the similarity measures, matching one or more universal identifier matrices of the first set to one or more universal identifier matrices of the known set;
based on the matching, performing actions that were previously performed on the first set of one or more interactive elements on one or more corresponding interactive elements of the second set; and
based on one or more outcomes of the performing, selectively classifying the identifier of the web page as validated or invalidated;
wherein validated web page identifiers are flagged as available to facilitate natural language-based automated navigation through websites that include web pages associated with the validated web page identifiers, and invalidated web page identifiers are flagged as unavailable for natural language-based automated navigation or discarded.