US 12,332,870 B2
Devices, methods, and systems for generating a highly-scalable, efficient composite record index
Tucker Leavitt, Durango, CO (US); Adam Najman, Jersey City, NY (US); Alfredo Gimenez, Daly City, CA (US); Tyler Flach, New York, NY (US); Abdul Khan, New York, NY (US); and Jonathan Greenblatt, New York, NY (US)
Assigned to BlueVoyant LLC, New York, NY (US)
Appl. No. 18/860,966
Filed by Blue Voyant LLC, New York, NY (US)
PCT Filed Jun. 4, 2023, PCT No. PCT/US2023/024386
§ 371(c)(1), (2) Date Oct. 28, 2024,
PCT Pub. No. WO2023/235619, PCT Pub. Date Dec. 7, 2023.
Claims priority of provisional application 63/365,819, filed on Jun. 3, 2022.
Prior Publication US 2025/0110939 A1, Apr. 3, 2025
Int. Cl. G06F 16/22 (2019.01); G06F 16/2455 (2019.01); G06F 16/25 (2019.01); G06F 16/27 (2019.01)
CPC G06F 16/2282 (2019.01) [G06F 16/24568 (2019.01); G06F 16/258 (2019.01); G06F 16/27 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method for indexing protective Domain Name System (pDNS) records in scalable partitioned data tables that enable constant-time queries, the method comprising:
receiving, by a processor, a data stream from one or more data sources, where the data stream comprises pDNS records;
aggregating, by the processor, the pDNS records to a distributed database;
performing, by the processor, a first structured streaming job on the distributed database;
performing, by the processor, an indexing job on a plurality of group files, wherein the indexing job reads a first set of group files of the plurality of group files and generates a first group index for all queryable fields of the pDNS records, and wherein a subsequent indexing job is performed according to a predetermined indexing job interval;
writing, by the processor, an index group to a first row in a rowkey table, wherein the index group is grouped according to an event timestamp of indexed pDNS records that are indexed during the predetermined indexing job interval;
querying, by the processor, the rowkey table according to a set of queryable fields of the pDNS records; and
returning, by the processor, a query result at a constant-time regardless of a total number of rows in the rowkey table, wherein the first structured streaming job comprises:
processing, by the processor, a first micro-batch of the pDNS records;
reformatting, by the processor, the pDNS records into an extensible format pDNS records;
grouping, by the processor, the extensible format pDNS records according to a first byte of a requesting IP address; and
writing, by the processor, the extensible format pDNS records to the first set of group file of the plurality of group files, and wherein a plurality of subsequent structure streaming jobs are continuously triggered and initiated within a predetermined time period from a prior structured streaming job.