US 11,712,633 B2
Crowd-sourced data collection and labelling using gaming mechanics for machine learning model training
Kevin Yu, Hong Kong (CN); Simone Rebaudengo, Hong Kong (CN); Nick Pittom, Hong Kong (CN); Kang Zhang, Hong Kong (CN); and Matt Au, Hong Kong (CN)
Assigned to SideChef Group Limited, Hong Kong (CN)
Filed by SideChef Group Limited, Hong Kong (CN)
Filed on Mar. 12, 2020, as Appl. No. 16/817,358.
Application 16/817,358 is a continuation of application No. PCT/CN2019/122166, filed on Nov. 29, 2019.
Prior Publication US 2021/0166077 A1, Jun. 3, 2021
Int. Cl. A63F 13/825 (2014.01); A63F 13/58 (2014.01); G06N 20/00 (2019.01); G06T 19/00 (2011.01); G06F 18/214 (2023.01); G06V 10/778 (2022.01); G06V 20/70 (2022.01); G06V 20/10 (2022.01); G06V 20/20 (2022.01); G06V 20/68 (2022.01)
CPC A63F 13/825 (2014.09) [A63F 13/58 (2014.09); G06F 18/2155 (2023.01); G06N 20/00 (2019.01); G06T 19/006 (2013.01); G06V 10/7784 (2022.01); G06V 10/7788 (2022.01); G06V 20/10 (2022.01); G06V 20/20 (2022.01); G06V 20/70 (2022.01); A63F 2300/65 (2013.01); A63F 2300/695 (2013.01); G06V 20/68 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method of data collection and labeling, the method comprising:
receiving, at a server in a cloud environment from an augmented reality environment in a gamified application executing on a mobile device, a photo of a food item captured by a user of the mobile device using a camera installed thereon and information indicating that the food item has been fed to a virtual character displayed in the augmented reality environment, wherein the cloud environment includes a data store that maps the virtual character to an evolution path customized based on one or more attributes of the user of the mobile device, wherein the customized evolution path defines an ordered sequence of food items, wherein the received information indicating that the food item has been fed to the virtual character causes the server to associate the food item and the virtual character in the data store and evolve the virtual character along the customized evolution path to a particular stage for an award;
in response to receiving the photo, generating, by a machine learning model executing in the cloud environment, classification information for the food item represented by the photo, wherein the classification information includes a classification label for the photo, and a set of attributes for the food item;
sending the classification information to the mobile device for display in the augmented reality environment thereon;
receiving, from the mobile device, verified information, wherein the verified information includes an indication for associating the photo with the virtual character, a verified classification label, a set of verified attributes, wherein at least one of the set of verified attributes is input by the user via either voice or text via an interface in response to a confirmation prompt for the user to verify the classified information displayed in the augmented reality environment;
storing the photo and the verified information into the data store in the cloud environment; and
periodically retraining the machine learning model using data stored in the data store, wherein at least a portion of the data is labeled by verified information from a plurality of mobile devices.