US 11,887,403 B1
Mouth shape correction model, and model training and application method
Huapeng Sima, Nanjing (CN); and Guo Yang, Jiangsu (CN)
Assigned to NANJING SILICON INTELLIGENCE TECHNOLOGY CO., LTD., Jiangsu (CN)
Filed by NANJING SILICON INTELLIGENCE TECHNOLOGY CO., LTD., Jiangsu (CN)
Filed on Jun. 21, 2023, as Appl. No. 18/338,460.
Claims priority of application No. 202210971051.3 (CN), filed on Aug. 15, 2022.
Int. Cl. G06V 20/40 (2022.01); G06V 40/16 (2022.01)
CPC G06V 40/171 (2022.01) [G06V 20/46 (2022.01)] 9 Claims
OG exemplary drawing
 
1. A mouth shape correction model, comprising a mouth feature extraction module, a key point extraction module, a first video module, a second video module, and a discriminator, wherein
the mouth feature extraction module is configured to obtain a mouth image corresponding to a face in a to-be-corrected mouth shape video, and extract a mouth shape change feature;
the key point extraction module is configured to obtain a face image containing the mouth image in the to-be-corrected mouth shape video, and extract a mouth key point feature;
the first video module is configured to obtain a video-to-frame split image corresponding to the face image; mask a mouth in the video-to-frame split image to obtain a mouth covered image; extract a cover feature; and splice the cover feature, the mouth key point feature, and the mouth shape change feature and input the same to a decoder in the first video module, to obtain a predicted face image;
the second video module is configured to extract a mouth feature from the mouth image; mask a mouth area in the predicted face image to obtain a predicted-face cover feature; and splice the predicted-face cover feature, the mouth feature, and the mouth shape change feature and input the same to a decoder in the second video module, to obtain a mouth-shape-corrected face image; and
the discriminator is configured to calculate discriminator losses in the first video module and the second video module; determine whether the predicted face image and the mouth-shape-corrected face image are real face images based on the discriminator losses; and if a determining result is yes, output the predicted face image and the mouth-shape-corrected face image.