US 12,437,783 B2
System and method for AI/XI based automatic song finding for videos in a mobile app or web environment to achieve high-performance
Dieter Rein, Berlin (DE); and Jurgen Jaron, Berlin (DE)
Assigned to BELLEVUE INVESTMENTS GmbH & Co. KGaA, (DE)
Filed by BELLEVUE INVESTMENTS GmbH & Co. KGaA, Berlin (DE)
Filed on Mar. 28, 2024, as Appl. No. 18/620,738.
Application 18/620,738 is a continuation in part of application No. 18/078,329, filed on Dec. 9, 2022, granted, now 12,009,013.
Claims priority of provisional application 63/454,981, filed on Mar. 28, 2023.
Prior Publication US 2024/0242737 A1, Jul. 18, 2024
Int. Cl. G11B 27/031 (2006.01); G06F 16/435 (2019.01)
CPC G11B 27/031 (2013.01) [G06F 16/435 (2019.01)] 15 Claims
OG exemplary drawing
 
1. A method of automatically matching one or more songs to a video work selected by a user, said video having a length associated therewith, comprising:
(a) computer software executing in a computer, said computer software programmed to perform at least the steps of:
(i) accessing said video work;
(ii) dividing said video into a plurality of equal length frames;
(iii) uploading said plurality of frames to an AI server;
(iv) within each of said plurality of frames, identifying one or more objects;
(v) assigning an object label to each of said identified one or more objects;
(vi) determining an accuracy value for each of said assigned object labels;
(vii) selecting from said assigned object labels only those object labels identified with an accuracy value greater than a predetermined accuracy value, thereby obtaining a reduced set of object labels;
(viii) using an emotion matrix to assign an energy level and an emotion type to each of said object labels in said reduced set of object labels;
(ix) using said assigned energy level and said assigned emotion type associated with each of said object labels in said reduced set of object labels, to determine a frame energy level and a frame emotion type for each of said plurality of frames;
(x) using said frame energy level and said frame emotion type associated with each of said plurality of frames, to determine one or more characteristic energy levels and one or more characteristic emotion types for said video;
(xi) using said assigned energy level and said assigned emotion type associated with each of said object labels in said reduced set of object labels, to determine an assigned genre associated with each of said object labels in said reduced set of object labels;
(xii) using said assigned genre associated with each of said object labels in said reduced set of object labels to determine one or more characteristic genres;
(xiii) accessing a song database containing a plurality of songs, each of said songs having at least a song emotion type, a song genre, a song energy, and a song quality associated therewith;
(xiv) selecting a minimum song quality level;
(xv) using said minimum song quality level, one or more characteristic emotions, said one or more characteristic genres, and said one or more characteristic energies to select one or more candidate songs from said song database;
(xvi) allowing the user to select from said matching one or more candidate songs a soundtrack song;
(xvii) combining said soundtrack song and said video work to produce a combined video work; and
(xviii) performing at least a portion of said combined video work for the user.