| CPC G06F 3/0635 (2013.01) [G06F 3/0604 (2013.01); G06F 3/0679 (2013.01)] | 20 Claims |
|
1. A method performed by one or more computers, the method comprising:
accessing a data set comprising data to be processed into a data cube;
generating a memory-mapped data cube based on the data set, the memory-mapped data cube comprising a plurality of files including different segments of the data cube, wherein generating the memory-mapped data cube comprises:
allocating memory-mapped buffers in non-volatile data storage in response to first memory allocation requests, wherein the memory-mapped buffers each allocate extra capacity such that the buffer exceeds an allocation size indicated in the corresponding first memory allocation request;
after allocating the memory-mapped buffers and writing first components of the data cube in the memory-mapped buffers, responding to second memory allocation requests by providing addresses for locations in the extra capacity of the buffers;
based on the provided addresses, writing second components of the data cube to the memory-mapped buffers; and
storing contents of the respective buffers as files of the data cube; and
loading the memory-mapped data cube by storing the files of the data cube in disk-based storage, mapping the stored files of the data cube to virtual memory addresses, and caching portions of the data cube in random-access memory.
|