US 12,136,282 B2
Image-based kitchen tracking system with dynamic labeling management
Evan DeSantola, Pittsburgh, PA (US); Alex Litzenberger, Pittsburgh, PA (US); Rassoul Mesbah, Aukland (NZ); and Prashasti Sar, Pittsburgh, PA (US)
Assigned to HME Hospitality & Specialty Communications, Inc., Carlsbad, CA (US)
Filed by HME Hospitality & Specialty Communications, Inc., Carlsbad, CA (US)
Filed on Feb. 17, 2023, as Appl. No. 18/171,237.
Application 18/171,237 is a continuation of application No. 17/690,537, filed on Mar. 9, 2022, granted, now 11,594,050.
Claims priority of provisional application 63/160,668, filed on Mar. 12, 2021.
Prior Publication US 2023/0360412 A1, Nov. 9, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06V 20/68 (2022.01); G01B 11/22 (2006.01); G01S 17/89 (2020.01); G06F 3/0482 (2013.01); G06F 18/214 (2023.01); G06N 20/00 (2019.01); G06Q 10/0631 (2023.01); G06Q 20/20 (2012.01); G06Q 50/12 (2012.01); G06T 7/11 (2017.01); G06T 7/20 (2017.01); G06T 7/70 (2017.01); G06V 10/70 (2022.01); G06V 10/764 (2022.01); G06V 20/00 (2022.01); G06V 20/20 (2022.01); G06V 20/52 (2022.01); G06V 20/64 (2022.01); H04N 7/18 (2006.01); H04N 23/90 (2023.01)
CPC G06V 20/68 (2022.01) [G01B 11/22 (2013.01); G01S 17/89 (2013.01); G06F 3/0482 (2013.01); G06F 18/214 (2023.01); G06N 20/00 (2019.01); G06Q 10/063112 (2013.01); G06Q 20/20 (2013.01); G06Q 50/12 (2013.01); G06T 7/11 (2017.01); G06T 7/20 (2013.01); G06T 7/70 (2017.01); G06V 10/70 (2022.01); G06V 10/765 (2022.01); G06V 20/20 (2022.01); G06V 20/36 (2022.01); G06V 20/52 (2022.01); G06V 20/64 (2022.01); H04N 7/18 (2013.01); H04N 23/90 (2023.01); G06T 2200/24 (2013.01); G06T 2207/10028 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/30232 (2013.01); G06V 2201/08 (2022.01)] 15 Claims
OG exemplary drawing
 
1. A method, comprising:
obtaining image data comprising one or more image frames indicative of a state of an area in a restaurant, wherein the one or more image frames have been captured by a camera, wherein the camera has a field of view that includes the area in the restaurant;
providing the image data as input into a machine learning model that has been trained to detect objects in the one or more image frames, wherein the machine learning model includes an artificial neural network;
obtaining one or more outputs from the machine learning model, wherein the one or more outputs are indicative of a first feature characterization;
obtaining, by a processing device, the first feature characterization that is representative of a first object or action, wherein the first feature characterization is based on the one or more outputs of the machine learning model and includes a first feature vector of values;
determining, based on output by the machine learning model, that the first feature characterization does not meet one or more classification criteria for a set of object or action classifications;
determining, by the processing device, a proximity between the first feature characterization and one of a set of previous feature characterizations corresponding to unclassified objects or actions, wherein the set of previous feature characterizations includes one or more feature vectors, and wherein the proximity is determined by using a distancing algorithm between the first feature vector and the one or more feature vectors;
generating, by the processing device, a new object classification based on the first feature characterization and the one of the set of previous feature characterizations, wherein the new object classification is added to the one or more classification criteria for the set of object or action classifications; and
providing, by the processing device, a notification related to the new object classification.