US 12,141,218 B1
System and method for collecting content from network servers
Uri Knorovich, Tel Aviv (IL); Alon Bar Tzlil, Tel Aviv (IL); and Yuval Shalev, Herzliya (IL)
Filed by THE DATA COMPANY TECHNOLOGIES INC., Wilmington, DE (US)
Filed on May 2, 2023, as Appl. No. 18/142,039.
Int. Cl. G06F 16/00 (2019.01); G06F 16/951 (2019.01); G06F 16/955 (2019.01)
CPC G06F 16/9566 (2019.01) [G06F 16/951 (2019.01)] 10 Claims
OG exemplary drawing
 
1. A method of providing content from network servers, the method comprising:
receiving a data request from a client;
performing an internet search on a search engine based on the data request;
accessing URLs relevant to the request resulting from the search;
inputting parsed text from the accessed URLs into a software model configured to identify terms of use in the URLs;
accessing a robots.txt of the URLs to receive collection rules that define terms for collecting data in the URLs;
sending a data request to the URL, said request includes the details of the requesting entity and an opt-out option for the URLs; and
collecting the data in the URLs.