US 12,148,175 B2
Multi-frame optical flow network with lossless pyramid micro-architecture
Yingmao Li, Allen, TX (US); Chenchi Luo, Plano, TX (US); Gyeongmin Choe, San Jose, CA (US); and John Seokjun Lee, Allen, TX (US)
Assigned to Samsung Electronics Co., Ltd., Suwon-si (KR)
Filed by Samsung Electronics Co., Ltd., Suwon-si (KR)
Filed on Feb. 2, 2022, as Appl. No. 17/590,998.
Prior Publication US 2023/0245328 A1, Aug. 3, 2023
Int. Cl. G06T 7/269 (2017.01)
CPC G06T 7/269 (2017.01) [G06T 2207/10016 (2013.01); G06T 2207/20081 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
obtaining a first optical flow vector representing motion between consecutive video frames during a previous time step;
generating a first predicted optical flow vector from the first optical flow vector using a trained prediction model, the first predicted optical flow vector representing predicted motion during a current time step; and
refining the first predicted optical flow vector using a trained update model to generate a second optical flow vector representing motion during the current time step, wherein the trained update model uses the first predicted optical flow vector, a video frame of the previous time step, and a video frame of the current time step to generate the second optical flow vector;
wherein the trained update model comprises:
multiple shuffle layers configured to (i) receive the video frame of the previous time step and the video frame of the current time step and (ii) generate a lossless pyramid of additional images at decreasing resolutions and increasing numbers of channels; and
multiple subnetworks configured to be performed sequentially, each of the multiple subnetworks configured to receive a portion of the additional images at a different one of the decreasing resolutions.