CPC H04L 47/22 (2013.01) [G06F 9/5083 (2013.01); G06F 16/2477 (2019.01); G06F 16/951 (2019.01); H04L 43/16 (2013.01); H04L 67/10 (2013.01); H04L 67/60 (2022.05)] | 20 Claims |
1. A method for data processing a set of records by a distributed processing system, the method comprising:
determining, by one or more processors, a first agent progress in processing the set of records, wherein the first agent progress is based on a first time difference between a first timestamp of an oldest record of the set of records processed by a first agent process and entered into a queue for a second agent process and a current system time, wherein the second agent process is a dependent agent process of the first agent process and processes records that have first been processed by the first agent process;
determining, by the one or more processors, a second agent progress in processing the set of records, wherein the second agent progress is based on a second time difference between a second timestamp of an oldest record of the set of records processed by the second agent process and the current system time, wherein the first timestamp of the oldest record of the set of records processed by the first agent process and the second timestamp of the oldest record of the set of records processed by the second agent process each correspond a time the respective oldest record was received by the distributed processing system;
determining, by the one or more processors, a difference between the first agent progress and the second agent progress; and
throttling, by the one or more processors, performance of the first agent process, based on the difference between the first agent progress and the second agent progress meeting a threshold measure.
|