US 12,390,931 B2
Autonomous robot packaging of arbitrary objects
Fan Wang, Woburn, MA (US); and Kristoffer Karl Hauser, Champaign, IL (US)
Assigned to Duke University, Durham, NC (US); and The Board of Trustees of the University of Illinois, Urbana, IL (US)
Filed by The Board of Trustees of the University of Illinois, Urbana, IL (US)
Filed on Jul. 15, 2021, as Appl. No. 17/377,232.
Claims priority of provisional application 63/051,956, filed on Jul. 15, 2020.
Prior Publication US 2022/0016779 A1, Jan. 20, 2022
Int. Cl. B25J 9/16 (2006.01); B25J 13/08 (2006.01); G06N 3/08 (2023.01); G06T 7/11 (2017.01); G06T 7/50 (2017.01); G06T 7/73 (2017.01); G06T 17/20 (2006.01)
CPC B25J 9/1669 (2013.01) [B25J 9/163 (2013.01); B25J 9/1633 (2013.01); B25J 13/085 (2013.01); G06N 3/08 (2013.01); G06T 7/11 (2017.01); G06T 7/50 (2017.01); G06T 7/75 (2017.01); G06T 17/20 (2013.01); G06T 2207/10024 (2013.01); G06T 2207/10028 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A system for packing objects into a container comprising:
a robotic device comprising:
a robotic arm;
a plurality of object detection sensors; and
a controller comprising at least one processor and a non-transitory computer-readable medium wherein the non-transitory computer-readable medium stores a set of program instructions, wherein the at least one processor executes the program instructions so as to carry out operations, the operations comprising:
sensing a measurement of each object among a plurality of objects with at least one object detection sensor of the plurality of object detection sensors;
determining, using the sensed measurement for each object in the plurality of objects, and based on a trained machine learning model, whether each object among the plurality of objects matches a predetermined three-dimensional model, wherein the predetermined three-dimensional model is retrieved from a database of reference three-dimensional models;
obtaining a three-dimensional model for each object in the plurality of objects, wherein obtaining the three-dimensional model comprises:
based on determining a match between a matched object in the plurality of object and the predetermined three-dimensional model, obtaining the predetermined three-dimensional model;
based on determining no match between a no-match object in the plurality of objects and the predetermined three-dimensional model, obtaining a dynamically generated three-dimensional model by: (i) determining a depth reading for each no-match object assuming that the no-match object is lying flat and fully touching a table top, and (ii) determining at least one hidden side of the no-match object in the plurality of objects using linear interpolation from a top periphery of the no-match object and a bottom periphery of the no-match object:
determining a packing plan for the plurality of objects based on the three-dimensional model for each object; and
loading, by the robotic arm, at least a portion of the plurality of objects into a container according to the packing plan for the plurality of objects.