obtaining input frames of a video sequence to be compressed using a dyadic hierarchy structure, wherein the dyadic hierarchy structure comprises base layer frames separated by a predetermined number of non-base layer frames, and the base layer frames comprises an I-frame that does not have a reference frame and P-frames that have linear dependency to another base layer frame;

receiving acknowledgement messages from a receiver that acknowledge the receiver has sufficient data to decode an individual base layer frame of the dyadic hierarchy structure;

holding one or more acknowledged base layer frames in an encoder decoded picture buffer;

determining that an acknowledgement for a first base layer frame has not been received at the receiver within a maximum time, the maximum time corresponding to a time for the receiver to display the predetermined number of non-base layer frames separating two base layer frames;

in response to determining that the acknowledgement for the first base layer frame has not been received at the receiver within the maximum time and determining that a last acknowledged base layer frame is present in the encoder decoded picture buffer, updating a base reference frame in an updated reference list for a next base layer frame to the last acknowledged base layer frame; and

in response to determining that the acknowledgement for the first base layer frame has not been received at the receiver within the maximum time and determining that no acknowledged base layer frame is present in the encoder decoded picture buffer, re-transmit one or more packets for a replacement base layer frame to the receiver.