| CPC G06Q 30/0643 (2013.01) [G06F 18/214 (2023.01); G06F 18/217 (2023.01); G06T 19/006 (2013.01); G06V 10/751 (2022.01); G06V 20/41 (2022.01)] | 20 Claims |
|
1. A method comprising:
receiving, by one or more processors of a user device, a video that includes a depiction of one or more objects in a room within a home;
processing by a machine learning model the video to determine a room classification for the room by processing the one or more objects depicted in the video, the machine learning model trained based on training data to establish a relationship between a plurality of training images and ground truth room classifications for each of the training images, the machine learning model providing an output that includes an estimated age of a person associated with the room, the machine learning model comprising a neural network and is trained by performed training operations comprising:
receiving the training data comprising the plurality of training images and the ground truth room classifications, each of the plurality of training images depicting a different room in a home;
extracting one or more features from a first training image of the plurality of training images corresponding to real-word objects;
applying the neural network to the extracted one or more features of the first training image of the plurality of training images to estimate an individual room classification of an individual room depicted in the first training image;
computing a deviation between the estimated room classification and the ground truth room classification associated with the first training image;
updating parameters of the neural network based on the computed deviation;
determining that the room classification of the room corresponds to a bedroom; and
repeating the training operations for each of the plurality of training images;
determining an age range associated with the bedroom based on the one or more objects depicted in the video based on the estimated age of the person included in the output of the machine learning model;
selecting, by the one or more processors of the user device, one or more augmented reality items available for purchase based on the room classification and the one or more objects depicted in the video, the selecting of the one or more augmented reality items comprising searching for one or more toys associated with the age range;
generating, for display within the video on a display of the user device, the one or more augmented reality items that have been selected at a display position within the video corresponding to the one or more objects depicted in the video; and
replacing, in the display of the user device and by the one or more processors of the user device, a depiction of an individual real-world object in the video with an individual augmented reality item of the selected one or more augmented reality items, the replacing comprising:
identifying, by the one or more processors, a display position of the individual real-world object;
calculating characteristic points for a set of elements of the individual real-world object to generate a mesh based on the calculated characteristic points;
generating one or more areas on the mesh of the individual real-world object;
aligning the one or more areas of the individual real-world object with one or more elements of a first augmented reality representation of a given one of the one or more toys with a position;
modifying one or more visual properties of the one or more areas to cause the user device to display the first augmented reality representation within the video at an individual display position relative to the display position of the individual real-world object that has been identified by the one or more processors;
receiving input associated with the first augmented reality representation; and
completing a purchase transaction of the given one of the one or more toys in response to receiving the input.
|