US 11,783,864 B2
Integration of audio into a multi-view interactive digital media representation
Stefan Johannes Josef Holzer, San Mateo, CA (US); Radu Bogdan Rusu, San Francisco, CA (US); Vladimir Roumenov Glavtchev, Mountain View, CA (US); and Alexander Jay Bruen Trevor, San Francisco, CA (US)
Assigned to Fyusion, Inc., San Francisco, CA (US)
Filed by Fyusion, Inc., San Francisco, CA (US)
Filed on Sep. 22, 2015, as Appl. No. 14/861,019.
Prior Publication US 2017/0084293 A1, Mar. 23, 2017
Int. Cl. G11B 27/00 (2006.01); G11B 27/32 (2006.01); G06F 16/68 (2019.01); G11B 27/10 (2006.01); H04N 13/349 (2018.01)
CPC G11B 27/32 (2013.01) [G06F 16/686 (2019.01); G11B 27/10 (2013.01); G11B 27/322 (2013.01); H04N 13/349 (2018.05)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
retrieving a multi-view interactive digital media representation of image data, wherein the multi-view interactive digital media representation includes a plurality of two-dimensional (2D) images captured from a mobile device, wherein the 2D images are separated into content and context models which are fused together to render the multi-view interactive digital media representation navigable in one or more dimensions by selecting a viewpoint from a plurality of different viewpoints from which to view the image data, each viewpoint corresponding to a different frame in a plurality of frames associated with the multi-view interactive digital media representation, wherein the models are fused based in part on location information captured from an inertial measurement unit at the mobile device, wherein the content model includes an object and the context model includes scenery surrounding the object;
retrieving audio data to be integrated into the multi-view interactive digital media representation, wherein the audio data and the plurality of images are captured separately, and wherein the audio data can include a separate audio file recorded at a different time from the plurality of images;
attaching the audio data to specific frames in the multi-view interactive digital media representation, the specific frames corresponding to specific viewpoints of the multi-view interactive digital media representation such that certain portions of the audio are played when the specific frames in the multi-view interactive digital media representation are reached during navigation;
associating a first segment with a first position in the multi-view interactive digital media representation; and
playing the audio data in coordination with the multi-view interactive digital media representation based on a user's navigation through the multi-view interactive digital media representation, wherein the first position in the multi-view interactive digital media representation triggers playback of the first segment, wherein the first position is a position of an object of a position of a capture device, wherein navigating the multi-view interactive digital media representation in one direction plays the audio data forward, navigating the multi-view interactive digital media representation in the opposite direction plays the audio data backwards, and the speed of playing the audio data corresponds to navigation speed.