receiving sequential frames of an input video, wherein each of the sequential frames includes a first region and a second region, wherein the first region comprises a hole to be removed and wherein the second region excludes the first region;

removing a rotation between the sequential frames by applying an inverse matrix of a rotation transformation matrix corresponding to feature points extracted from a frame pair among the sequential frames to a feature point of a frame in the frame pair;

estimating first optical flow information of pixels in the second region of the sequential frames with the rotation removed;

estimating a polynomial model corresponding to the first optical flow information;

estimating second optical flow information of pixels in the first region of the sequential frames by interpolating the first optical flow information based on the polynomial model, wherein the second optical flow information is different from the first optical flow information;

searching optical flow information toward a first pixel in the first region of a first frame in the sequential frames and optical flow information having the first pixel as a starting point, based on the first optical flow information and the second optical flow information; and

inpainting the input video at the first region by estimating color information of the first pixel based on color information of a second pixel of a second frame in the sequential frames according to the optical flow information toward the first pixel and color information of a third pixel of a third frame in the sequential frames according to the optical flow information having the first pixel as the starting point.