US 11,966,901 B2
Automated shopping experience using cashier-less systems
Amir Hossein Khalili, Santa Clara, CA (US); Bhooshan Supe, Santa Clara, CA (US); Jung Ick Guack, Santa Clara, CA (US); Shantanu Patel, Santa Clara, CA (US); Gaurav Saraf, Santa Clara, CA (US); Baisub Lee, Santa Clara, CA (US); Helder Silva, Santa Clara, CA (US); Julie Huynh, Santa Clara, CA (US); and Jaigak Song, Santa Clara, CA (US)
Assigned to LG ELECTRONICS INC., Seoul (KR)
Filed by LG ELECTRONICS INC., Seoul (KR)
Filed on Aug. 31, 2021, as Appl. No. 17/463,419.
Claims priority of provisional application 63/073,319, filed on Sep. 1, 2020.
Prior Publication US 2022/0067390 A1, Mar. 3, 2022
Int. Cl. G06Q 20/20 (2012.01); G01G 19/414 (2006.01); G06F 18/25 (2023.01); G06N 3/02 (2006.01); G06N 20/00 (2019.01); G06Q 10/087 (2023.01); G06T 7/246 (2017.01); G06T 7/73 (2017.01); G06V 20/52 (2022.01); G06V 40/10 (2022.01); G06V 40/16 (2022.01); G06V 40/20 (2022.01); H04N 23/61 (2023.01)
CPC G06Q 20/208 (2013.01) [G01G 19/4144 (2013.01); G06F 18/251 (2023.01); G06N 3/02 (2013.01); G06N 20/00 (2019.01); G06Q 10/087 (2013.01); G06Q 20/203 (2013.01); G06T 7/251 (2017.01); G06T 7/73 (2017.01); G06V 20/52 (2022.01); G06V 40/107 (2022.01); G06V 40/161 (2022.01); G06V 40/168 (2022.01); G06V 40/172 (2022.01); G06V 40/28 (2022.01); H04N 23/61 (2023.01); G06T 2207/10028 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30201 (2013.01); G06T 2207/30232 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
capturing images from a camera mounted on a shelf unit;
identifying a user from the captured images, wherein the captured images comprise a representation of a body of the user;
identifying joints of the identified user by performing a deep neural network body joint detection on the captured images;
detecting and tracking actions of the identified user over a first time period by:
detecting an affinity of joints for performing a deep learning identification on a first bounding box to identify a particular user and for associating results from the deep neural network body joint detection with the particular user to re-identify joints of the identified user, wherein the first bounding box comprises identified joints of the identified user,
transforming a 2D position information of the identified joints into a 3D position information of the identified joints by mapping 2D locations of the detected joints in the captured images to a corresponding point cloud, and
tracking the 3D position information of the detected joints to determine when the detected joints have entered bins within the shelf unit, wherein the detected joints comprise at least a wrist;
tracking an object from the bins over a second time period by associating the object with one or more joints among the identified joints that have entered the bins within the shelf unit, wherein the object is associated with the one or more joints based at least in part on a determination that the 3D position of the identified joints has entered a particular bin within the shelf unit, wherein the object was predetermined to be located in the particular bin using a virtual map of the location of the bins on the shelf unit and corresponding objects located in the bins, wherein the virtual map is stored on a memory storage on the shelf unit; and
determining an action inference of the identified user based at least in part on the associated object with the one or more joints and results from the deep learning identification on the first bounding box.