US 11,057,567 B2
Anti-shake method and apparatus for panoramic video, and portable terminal
Cong Chen, Shenzhen (CN); and Jinlin Cai, Shenzhen (CN)
Assigned to ARASHI VISION INC., Shenzhen (CN)
Appl. No. 16/759,315
Filed by ARASHI VISION INC., Shenzhen (CN)
PCT Filed Oct. 16, 2018, PCT No. PCT/CN2018/110467
§ 371(c)(1), (2) Date Apr. 24, 2020,
PCT Pub. No. WO2019/080748, PCT Pub. Date May 2, 2019.
Claims priority of application No. 201711015180.0 (CN), filed on Oct. 25, 2017.
Prior Publication US 2020/0314340 A1, Oct. 1, 2020
This patent is subject to a terminal disclaimer.
Int. Cl. H04N 5/232 (2006.01); G01P 3/22 (2006.01); G01P 15/18 (2013.01)
CPC H04N 5/23258 (2013.01) [G01P 3/22 (2013.01); G01P 15/18 (2013.01); H04N 5/23238 (2013.01)] 18 Claims
OG exemplary drawing
 
1. An anti-shake method for a panoramic video, comprising steps of:
acquiring, in real time, a current state timestamp, an accelerometer data and an angular velocity of a gyroscope in a portable terminal;
estimating a rotation from the portable terminal to a world coordinate system using Extended Kalman Filtering combined with both the accelerometer data and the angular velocity;
synchronizing the timestamp of the gyroscope with a timestamp of a panoramic video frame;
performing quaternion interpolation in a rotation from the portable terminal to a world coordinate system to obtain a rotation matrix corresponding to the panoramic video frame; and
rotating a panoramic image according to the current rotation matrix to generate a stable video frame;
wherein the step of estimating a rotation from the portable terminal to a world coordinate system using Extended Kalman Filtering combined with both the accelerometer data and the angular velocity, specifically comprises steps of:
S1021, calculating an initial-state rotation q0+ and an initial process covariance matrix

OG Complex Work Unit Math
where, d0 is a measured initial accelerometer data, g is the gravity vector of the world coordinate system; an initial process covariance matrix

OG Complex Work Unit Math
S1022, calculating a State Transition Matrix Φ(ωk) at time K using the angular velocity ωk;
Φ(ωk)=exp(−[ωk·Δt]x), where, ωk is an angular velocity at time K, Δt represents a sampling time interval of the gyroscope;
S1023, calculating a State Noise Covariance Matrix Qk, updating an a priori estimate for a state rotation qk and an a priori estimate for a process covariance matrix pk;

OG Complex Work Unit Math
where Qk is a State Noise Covariance Matrix;
qk=Φ(ωk)*qk−1+, where, qk−1+, is an a posteriori estimation for a state rotation at time K−1;
Pk=qk·Pk−1+·qk+Qk, where, Pk−1+ is an a posteriori estimation for a process covariance matrix at time K−1;
S1024, updating an Observation Noise Covariance Matrix Vk from the accelerometer data dk, calculating an Observation Transformation Jacobian Matrix Hk, and calculating an Observation Error ek between the current observation and estimated observation;

OG Complex Work Unit Math
where, δkkvarkmean, δkvar=α∥dk|−|dk−1∥+(1−α)δk−1var,

OG Complex Work Unit Math
α is a smoothing factor for an acceleration change, and β is an impact factor for an acceleration module;

OG Complex Work Unit Math
where, h is an observation function, h(q,v)=q·g+Ik, g is the gravity vector of the world coordinate system, q is a state rotation, that is, a rotation from the world coordinate system to the gyroscope coordinate system; Ik is a measurement noise;
ek=dk×h(qk,0);
S1025, updating an optimal Kalman Gain Matrix Kk at time k;
Kk=Pk·HkT(Hk·Pk·HkT+Vk)−1; and
S1026, updating the a posteriori estimation for a rotation qk+ from the portable terminal to the world coordinate system, and the a posteriori estimation for the process covariance matrix pk+ according to the optimal Kalman Gain Matrix Kk and the Observation Error ek;

OG Complex Work Unit Math