US 12,001,216 B2
Carpet detection method, movement control method, and mobile machine using the same
Chuqiao Dong, Pasadena, CA (US); Dan Shao, San Gabriel, CA (US); Zhen Xiu, Chino Hills, CA (US); Dejun Guo, San Gabriel, CA (US); and Huan Tan, Pasadena, CA (US)
Assigned to UBKANG (QINGDAO) TECHNOLOGY CO., LTD., Qingdao (CN)
Filed by UBTECH NORTH AMERICA RESEARCH AND DEVELOPMENT CENTER CORP, Pasadena, CA (US); and UBTECH ROBOTICS CORP LTD, Shenzhen (CN)
Filed on May 31, 2021, as Appl. No. 17/334,834.
Prior Publication US 2022/0382293 A1, Dec. 1, 2022
Int. Cl. G05D 1/02 (2020.01); G05D 1/00 (2006.01); G06N 3/04 (2023.01); G06T 7/55 (2017.01)
CPC G05D 1/0238 (2013.01) [G05D 1/0223 (2013.01); G05D 1/0246 (2013.01); G06N 3/04 (2013.01); G06T 7/55 (2017.01); G06T 2207/10024 (2013.01); G06T 2207/10028 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30261 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A carpet detection method, comprising:
obtaining, through an RGB-D camera, one or more RGB-D image pairs, wherein each RGB-D image pair comprises and RGB image and a depth image, and the depth image comprises data for representing a distance to scene objects in the depth image;
detecting one or more carpet areas in the RGB image of the one or more RGB-D image pairs and generating a first 2D bounding box to mark each of the one or more carpet areas using a first deep learning model based on a first data set containing a carpet class, wherein the first data set comprises a plurality of images of carpets in various scenes;
detecting one or more carpet-curl areas in the RGB image of the one or more RGB-D image pairs and generating a second 2D bounding box to mark each of the one or more carpet-curl areas using a second deep learning model based on a second data set containing a carpet-curl class, wherein the second data set comprises a plurality of images of carpet-curls in various scenes;
generating a group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the first 2D bounding box corresponding to the carpet area to each pixel in the depth image of the RGB-D image pair;
generating a group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the second 2D bounding box corresponding to the carpet-curl area to each pixel in the depth image of the RGB-D image pair; and
controlling a mobile machine to move and avoid one or more carpet-curls, according to the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs, and the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs.