US 12,254,713 B2
In-store food and beverage transfer and collection system using image recognition and method of transferring and collecting food and beverage in store using the same
Sung Jae Hwang, Seongnam-si (KR)
Assigned to XYZ, Inc., Seoul (KR)
Filed by XYZ, Inc., Seoul (KR)
Filed on Jul. 8, 2021, as Appl. No. 17/370,112.
Application 17/370,112 is a continuation of application No. PCT/KR2020/011022, filed on Aug. 19, 2020.
Claims priority of application No. 10-2019-0114672 (KR), filed on Sep. 18, 2019.
Prior Publication US 2021/0342573 A1, Nov. 4, 2021
Int. Cl. G06V 40/16 (2022.01); G05D 1/00 (2024.01); G06V 10/74 (2022.01); G06V 20/68 (2022.01); G06V 40/20 (2022.01)
CPC G06V 40/168 (2022.01) [G05D 1/0088 (2013.01); G05D 1/0094 (2013.01); G06V 10/74 (2022.01); G06V 20/68 (2022.01); G06V 40/172 (2022.01); G06V 40/20 (2022.01)] 16 Claims
OG exemplary drawing
 
1. An in-store food and beverage transfer and collection system using image analysis, the in-store food and beverage transfer and collection system comprising:
an image obtaining unit comprising a plurality of RGB-depth cameras configured to obtain images of a predetermined area and construct three-dimensional (3D) spatial information of a structure of a store and tables based on the images;
a processing unit configured to process a control command for processing an order of a customer by using the image obtained by the image obtaining unit;
a customer recognition unit configured to analyze a face image of the customer obtained by the image obtaining unit and identify the customer based on the face image; and
a transfer robot configured to obtain order information of the customer according to the control command of the processing unit, and transfer food and beverage that are ordered,
wherein the processing unit comprises:
an object tracking unit configured to:
obtain information of a table occupied by the customer, the information of the table including a location or an identification code of the table;
track a real-time location of the customer identified by the customer recognition unit based on the 3D spatial information of the structure of the store and tables by using a shape of clothes of the customer, a color of clothes of the customer, a hair style of the customer, or a skin color of the customer in the image obtained by the image obtaining unit;
determine whether the customer moves toward the table or moves away from the table based on the 3D spatial information of the structure of the store and tables and the real-time location of the customer;
determine whether the customer is continuously out of a predetermined space around the table for a predetermined time; and
release an occupancy state of the table by the customer in response to determining that the customer is continuously out of the predetermined space around the table for the predetermined time; and
an order information matching unit configured to match the order information with the information of the table occupied by the customer and the face image of the customer.