US 12,229,852 B2
Audio reactive augmented reality
David Robert Kasper, Pacifica, CA (US); Catherine Yoo, San Francisco, CA (US); and Matthew Alexander Nichols, San Francisco, CA (US)
Assigned to Meta Platforms, Inc., Menlo Park, CA (US)
Filed by Meta Platforms, Inc., Menlo Park, CA (US)
Filed on Apr. 30, 2021, as Appl. No. 17/245,514.
Prior Publication US 2022/0351424 A1, Nov. 3, 2022
Int. Cl. G06T 11/00 (2006.01); G10L 21/013 (2013.01); G10L 25/57 (2013.01); G10L 25/90 (2013.01)
CPC G06T 11/00 (2013.01) [G10L 21/013 (2013.01); G10L 25/57 (2013.01); G10L 25/90 (2013.01); G10L 2021/0135 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method for augmenting a video, comprising:
receiving a selection of an effect;
receiving user-generated content comprising video data and audio data;
detecting a characteristic of the audio data comprising at least a volume and/or a pitch of the audio data during a period of time;
determining a series of numeric values based on the characteristic of the audio data during the period of time, individual numeric values of the series of numeric values being correlated with an amplitude of the volume and/or pitch at a discrete point within the period of time;
determining a plurality of meshes to augment the video data and/or the audio data, wherein the plurality of meshes comprise at least: a primary mesh and an auxiliary mesh, wherein the auxiliary mesh and the primary mesh have different shapes, and a shape associated with the auxiliary mesh is defined by an energy meter configured to determine a transformation coefficient representing a scaled magnitude of distortion between a minimum level of distortion and maximum level of distortion; and
augmenting at least one of the video data and/or the audio data to include the effect of transitioning between the plurality of meshes based on the individual numeric value of the series of numeric values at discrete points in time within the period of time.