US 12,243,004 B2
System and method for determining out-of-stock products
Sarjoun Skaff, Pittsburgh, PA (US); Marios Savvides, Wexford, PA (US); Uzair Ahmed, Pittsburgh, PA (US); Nikhil Mohan, New York, NY (US); and Sreena Nallamothu, Pittsburgh, PA (US)
Assigned to Carnegie Mellon University, Pittsburgh, PA (US)
Appl. No. 17/425,089
Filed by CARNEGIE MELLON UNIVERSITY, Pittsburgh, PA (US); and BOSSA NOVA ROBOTICS, San Francisco, CA (US)
PCT Filed Apr. 13, 2020, PCT No. PCT/US2020/027977
§ 371(c)(1), (2) Date Jul. 22, 2021,
PCT Pub. No. WO2020/210820, PCT Pub. Date Oct. 15, 2020.
Claims priority of provisional application 62/832,755, filed on Apr. 11, 2019.
Prior Publication US 2022/0108264 A1, Apr. 7, 2022
Int. Cl. G06Q 10/087 (2023.01); B65G 1/137 (2006.01); G06K 7/14 (2006.01); G06T 3/4038 (2024.01); G06T 7/60 (2017.01); G06T 7/70 (2017.01); G06V 10/10 (2022.01); G06V 10/22 (2022.01); G06V 10/764 (2022.01); G06V 20/52 (2022.01); G09F 3/00 (2006.01); G09F 3/20 (2006.01); H04N 7/18 (2006.01); H04N 23/698 (2023.01); H04N 23/90 (2023.01)
CPC G06Q 10/087 (2013.01) [B65G 1/137 (2013.01); G06K 7/1413 (2013.01); G06K 7/1417 (2013.01); G06T 3/4038 (2013.01); G06T 7/60 (2013.01); G06T 7/70 (2017.01); G06V 10/16 (2022.01); G06V 10/22 (2022.01); G06V 10/225 (2022.01); G06V 10/764 (2022.01); G06V 20/52 (2022.01); G09F 3/0297 (2013.01); G09F 3/204 (2013.01); H04N 7/18 (2013.01); H04N 23/698 (2023.01); G06T 2207/20081 (2013.01); G06T 2207/30232 (2013.01); G06V 2201/07 (2022.01); H04N 23/90 (2023.01)] 21 Claims
OG exemplary drawing
 
1. A system, comprising:
one or more processors;
one or more sensors coupled to the one or more processors; and software implementing:
a product detection neural network for detecting products in images captured by the one or more sensors;
a label detection neural network for detecting labels in images captured by the one or more sensors;
a product type classifier for identifying products as any one of a shelf product, a peg product, a grill product, or a shelf-ready package;
a shelf ready classifier for determining whether a shelf-ready package is empty or not empty; and
a label type classifier for identifying label types;
wherein the software causes the system to perform the functions of:
obtaining a representation of an aisle containing stocked products from the one or more sensors;
identifying product labels in the representation;
classifying the product labels as specific types of labels using the label type classifier;
identifying products in the representation;
classifying the products as a specific type of product using the product type classifier;
identifying an area associated with each product label, based on the type of the label;
associating products with product labels, based on a placement of the product with respect to a label of the same type as the product; and
identifying, as out-of-stock products, products identified by product labels having no associated products.