US 11,670,267 B2
Computer vision and mapping for audio applications
Ilteris Canberk, Marina Del Rey, CA (US); Donald Giovannini, Venice, CA (US); and Sana Park, Los Angeles, CA (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on Aug. 9, 2021, as Appl. No. 17/397,145.
Application 17/397,145 is a continuation of application No. 16/724,209, filed on Dec. 21, 2019, granted, now 11,087,728, issued on Aug. 10, 2021.
Prior Publication US 2021/0366449 A1, Nov. 25, 2021
This patent is subject to a terminal disclaimer.
Int. Cl. G10H 1/00 (2006.01); G06T 7/90 (2017.01)
CPC G10H 1/0025 (2013.01) [G06T 7/90 (2017.01); G10H 1/0041 (2013.01); G06T 2207/10024 (2013.01); G06T 2207/30204 (2013.01); G10H 2220/101 (2013.01); G10H 2220/445 (2013.01); G10H 2220/455 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of playing a musical composition based on a color image, the method comprising:
initiating a computer vision and mapping application on a mobile device, said mobile device comprising a processor, a memory, a camera, a loudspeaker, and a display screen;
storing in the memory a plurality of preset token color values, each associated with a set of sound samples;
capturing with the camera a color image of a note matrix comprising one or more fields defined by rows and columns, wherein each row is associated with a musical note and each column is associated with a beat duration;
determining, with a hand detection algorithm, whether the captured color image includes at least a portion of a hand;
in response to determining that the captured color image does not include at least a portion of a hand, detecting in the captured color image one or more token color values, each characterized by a position relative to a field of the note matrix;
correlating the detected one or more token color values with the stored plurality of preset token color values; and
playing through the loudspeaker one or more sound samples retrieved from the memory, according to the correlated token color values and the detected positions, and in a sequence according to the note matrix.