US 12,148,448 B2
Authoring tools for creating interactive AR experiences
Lei Zhang, Ann Arbor, MI (US); Daekun Kim, Santa Monica, CA (US); Youjean Cho, New York, NY (US); Ava Robinson, Solana Beach, CA (US); Yu Jiang Tham, Seattle, WA (US); Rajan Vaish, Beverly Hills, CA (US); and Andrés Monroy-Hernández, Seattle, WA (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Lei Zhang, Ann Arbor, MI (US); Daekun Kim, Santa Monica, CA (US); Youjean Cho, New York, NY (US); Ava Robinson, Solana Beach, CA (US); Yu Jiang Tham, Seattle, WA (US); Rajan Vaish, Beverly Hills, CA (US); and Andrés Monroy-Hernández, Seattle, WA (US)
Filed on Sep. 1, 2022, as Appl. No. 17/901,540.
Prior Publication US 2024/0079031 A1, Mar. 7, 2024
Int. Cl. G11B 27/031 (2006.01); G06F 3/04817 (2022.01); G06F 3/04842 (2022.01); G06F 3/16 (2006.01); G06T 19/00 (2011.01); G06V 20/20 (2022.01); G10L 15/08 (2006.01); G10L 15/22 (2006.01); G11B 27/34 (2006.01)
CPC G11B 27/031 (2013.01) [G06F 3/04817 (2013.01); G06F 3/04842 (2013.01); G06F 3/167 (2013.01); G06T 19/006 (2013.01); G06V 20/20 (2022.01); G10L 15/08 (2013.01); G10L 15/22 (2013.01); G11B 27/34 (2013.01); G06T 2200/24 (2013.01); G10L 2015/088 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A method of generating an interactive AR experience using a portable electronic device comprising a microphone, a camera, a display, and a memory, the method comprising:
presenting on the display a story creation interface for creating a recording associated with a story comprising one or more scenes, wherein the story creation interface comprises a keyword module, a virtual object module, and an IoT product module, wherein the IoT product module comprises:
capturing frames of video data with the camera,
detecting an IoT product at an IoT product location in a physical environment using the captured frames of video data;
determining a portable electronic device location relative to the IoT product location using the captured frames of video data;
presenting an IoT interface when the device location is detected within a threshold proximity of the IoT product location;
detecting an interaction between the device and the IoT interface; and
storing in the memory the IoT product, the IoT product location, and the detected interaction in association with the recording;
and wherein the keyword module comprises:
presenting on the display a question and an answer button;
receiving through the microphone a spoken word;
identifying a keyword based on the spoken word;
presenting on the display a transcription of the identified keyword;
storing in the memory the keyword associated with the recording, wherein detecting the sound of the keyword during playback triggers a start of a subsequent scene;
starting a subsequent recording associated with the subsequent scene; and
storing in the memory the recording and the subsequent recording, wherein the recordings together comprise instructions associated with the story.