US 12,332,960 B2
System and method for URL fetching retry mechanism
Derry Shribman, Tel Aviv (IL); and Ofer Vilenski, Moshav Hadar Am (IL)
Assigned to BRIGHT DATA LTD., Netanya (IL)
Filed by BRIGHT DATA LTD., Netanya (IL)
Filed on Jan. 10, 2022, as Appl. No. 17/572,090.
Application 17/572,090 is a continuation of application No. 16/938,991, filed on Jul. 26, 2020, granted, now 11,657,110.
Application 16/938,991 is a continuation of application No. PCT/IL2020/050194, filed on Feb. 21, 2020.
Claims priority of provisional application 62/948,265, filed on Dec. 15, 2019.
Claims priority of provisional application 62/855,036, filed on May 31, 2019.
Claims priority of provisional application 62/809,847, filed on Feb. 25, 2019.
Prior Publication US 2022/0129514 A1, Apr. 28, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 1/00 (2006.01); G06F 9/54 (2006.01); G06F 16/909 (2019.01); G06F 16/955 (2019.01); H04L 1/00 (2006.01); H04L 67/01 (2022.01); H04L 67/02 (2022.01); H04L 67/1021 (2022.01); H04L 67/146 (2022.01); H04L 67/568 (2022.01); H04L 69/40 (2022.01); H04L 61/59 (2022.01); H04L 67/56 (2022.01)
CPC G06F 16/955 (2019.01) [G06F 9/547 (2013.01); G06F 16/909 (2019.01); G06F 16/9558 (2019.01); G06F 16/9566 (2019.01); H04L 1/00 (2013.01); H04L 67/01 (2022.05); H04L 67/02 (2013.01); H04L 67/1021 (2013.01); H04L 67/146 (2013.01); H04L 67/568 (2022.05); H04L 69/40 (2013.01); H04L 61/59 (2022.05); H04L 67/56 (2022.05)] 14 Claims
OG exemplary drawing
 
1. A method for data extraction and gathering from a web server, comprising:
receiving, by a first device from a user device, a request;
requesting, by the first device from a second device, an IP address from a database that stores a list of IP addresses;
accessing, by the second device, the database;
choosing, by the second device, an IP address from the list;
providing, by the second device to the first device, the chosen IP address;
sending, by the first device, the request to a proxy device that is addressed by the chosen IP address;
obtaining, by the proxy device, a reply from the web server;
forwarding, by the proxy device to first device, the reply;
analyzing, by the first device, the reply; and
determining, by the first device, pursuant to the analysis, if the reply is proper,
wherein, if the reply is proper, the reply is sent to the user device,
wherein, if the reply is not proper, the method repeats the data extraction and gathering from the web server,
wherein the repeating comprises deciding, pursuant to the analysis of the reply, if an adjustment of the request is needed,
wherein the reply is a Hypertext Transfer Protocol (HTTP) reply that comprises a response body, and the analyzing comprises analyzing of a HTTP reply status code in the HTTP header of the reply or a response body in the reply, and
wherein if the reply is not proper, the repeated data extraction and gathering from the web server is using different parameters that are deemed relevant for obtaining a proper reply.