| CPC G06F 9/4881 (2013.01) [G06F 16/285 (2019.01)] | 12 Claims |
|
1. A method comprising:
performing an execution process, independently and in parallel to a monitoring process, wherein the execution process includes, for a current period of time:
for a first process executing on a first processing node of a plurality of nodes of a data storage system, determining that a first operation to be executed for the first process of the first processing node requires accessing a second processing node of the plurality of nodes of the data storage system over an internal fabric of the data storage system, wherein the first processing node includes a first memory and the second processing node includes a second memory, wherein the first memory and the second memory are included in a global memory accessible by the plurality of processing nodes over the internal fabric, wherein the first operation for the first process of the first processing node includes accessing the second memory of the second processing node over the internal fabric according to an InfiniBand protocol to perform any of: i) read content from the second memory of the second processing node and ii) write content to the second memory of the second processing node, wherein the first process is included in a plurality of processes that perform operations to access the second memory of the second node;
determining that the first operation is of a first category of operation, wherein the first category of operation is high priority;
updating an average rate at which the first process of the first processing node initiates first category operations that require accessing the second processing node over the internal fabric, wherein the average rate is calculated over the current period of time;
determining whether the updated average rate, for the first process of the first processing node, exceeds an individual average rate threshold for individual processes; and
responsive to determining that the updated average rate for the first process of the first processing node exceeds the individual average rate threshold, taking a first corrective action for the first process of the first processing node, wherein the first corrective action is directed to only the first process of the first processing node and includes not allowing high priority operations of the first process to take precedence over other queued non-high priority operations of the first process awaiting servicing; and
when the current period of time has elapsed, performing processing by the monitoring process including:
determining that a second updated average rate, at which the first process initiates first category operations that require accessing the second processing node over the internal fabric, does not exceed the individual average rate threshold, wherein the second updated average rate is calculated as an exponential moving average over a second period of time including at least a period of time preceding the current period of time; and
responsive to determining that the second updated average rate does not exceed the individual average rate threshold, enforcing a reduced corrective action rather than the first corrective action, wherein the reduced corrective action allows each high priority operation of the first process to have precedence over only a specified number of other queued non-high priority operations of the first process.
|