| CPC G06F 9/5016 (2013.01) | 20 Claims |
|
1. A method comprising:
allocating a first memory space to a function, the memory space comprising an initial amount of memory for the function and the function being one of a plurality of functions processing a big data job;
setting the first memory space as a current memory space;
processing data using the function, the processing comprising writing data to the current memory space;
determining that the function requires additional memory space by monitoring that the function has exhausted its current memory space while processing;
determining that memory usage across the plurality of functions exhibits a power-law distribution wherein a small subset of the functions consume significantly more memory than a majority of the functions;
in response to determining that the memory usage exhibits the power-law distribution:
allocating a new memory space based on the current memory space and a geometric growth factor, the new memory space being larger than the current memory space by a factor determined by the geometric growth factor;
copying data in the current memory space to the new memory space;
setting the current memory space as an old memory space;
setting the new memory space as the current memory space; and
not deallocating the old memory space.
|