| CPC G06F 3/0619 (2013.01) [G06F 3/0635 (2013.01); G06F 3/064 (2013.01); G06F 3/065 (2013.01); G06F 3/0653 (2013.01); G06F 3/0659 (2013.01); G06F 3/067 (2013.01); G06F 11/1446 (2013.01); G06F 11/1471 (2013.01); G06F 11/3034 (2013.01)] | 18 Claims |
|
1. A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
receiving, at a data block replicator, a stream of data blocks for storage at a first storage location of a distributed storage system and at a second storage location of the distributed storage system, the first storage location associated with a first geographical region and the second storage location associated with a second geographical region different than the first geographical region;
synchronously writing, from the data block replicator, the stream of data blocks to the first storage location and to the second storage location;
while synchronously writing the stream of data blocks to the first storage location and to the second storage location, determining an unrecoverable failure at the second storage location that prohibits further writing of the stream of data blocks to the second storage location;
determining a failure point in the writing of the stream of data blocks, the failure point demarcating data blocks that were successfully written to the second storage location and data blocks that were not successfully written to the second storage location;
synchronously writing, from the data block replicator and starting at the failure point, the stream of data blocks to the first storage location and to a third storage location of the distributed storage system simultaneously, the third storage location associated with a third geographical region different than the first geographical region and the second geographical region; and
based on determining the unrecoverable failure at the second storage location and after synchronously writing the stream of data blocks to the third storage location, asynchronously writing, from a beginning of the stream of data blocks to the failure point of the stream of data blocks, the data blocks to the third storage location from the data block replicator.
|