US 12,243,159 B2
Digital twin modeling method and system for assembling a robotic teleoperation environment
Chengjun Chen, Shandong (CN); Zhengxu Zhao, Shandong (CN); Tianliang Hu, Shandong (CN); Jianhua Zhang, Shandong (CN); Yang Guo, Shandong (CN); Dongnian Li, Shandong (CN); Qinghai Zhang, Shandong (CN); and Yuanlin Guan, Shandong (CN)
Assigned to QINGDAO UNIVERSITY OF TECHNOLOGY, Shandong (CN); SHANDONG UNIVERSITY, Jinan Shandong (CN); and EBEI UNIVERSITY OF TECHNOLOGY, Tianjin (CN)
Filed by Qingdao University of Technology, Shandong (CN); Shandong University, Jinan Shandong (CN); and Hebei University of Technology, Tianjin (CN)
Filed on Aug. 23, 2022, as Appl. No. 17/893,368.
Prior Publication US 2023/0230312 A1, Jul. 20, 2023
Int. Cl. G06T 17/00 (2006.01); B25J 9/16 (2006.01); B25J 13/08 (2006.01); G06T 7/00 (2017.01); G06T 7/194 (2017.01); G06T 7/50 (2017.01); G06T 7/70 (2017.01); G06T 19/20 (2011.01); H04N 13/20 (2018.01); H04N 23/10 (2023.01)
CPC G06T 17/00 (2013.01) [B25J 9/1605 (2013.01); B25J 9/1687 (2013.01); B25J 9/1689 (2013.01); B25J 9/1697 (2013.01); B25J 13/08 (2013.01); G06T 7/0004 (2013.01); G06T 7/194 (2017.01); G06T 7/50 (2017.01); G06T 7/70 (2017.01); G06T 19/20 (2013.01); H04N 13/20 (2018.05); H04N 23/10 (2023.01); G06T 2207/10024 (2013.01); G06T 2207/10028 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/30164 (2013.01); G06T 2207/30204 (2013.01); G06T 2219/2004 (2013.01)] 10 Claims
OG exemplary drawing
 
1. A digital twin modeling method of assembling a robotic teleoperation environment, comprising the steps of:
capturing, by an image acquisition device, an image of a teleoperation environment;
identifying a part currently being assembled in the image;
querying the assembly assembling order to obtain a list of assembled parts according to the part currently being assembled;
generating a three-dimensional model of the current assembly from the list of assembled parts and calculating position pose information of the current assembly in an image acquisition device coordinate system;
loading a three-dimensional model of the robot, determining a coordinate transformation relationship between a robot coordinate system and an image acquisition device coordinate system;
position calibrating the robot in the image according to the coordinate transformation relationship, and determining position pose information of the robot in an image acquisition device coordinate frame;
determining a relative positional relationship between the current assembly and the robot from the position pose information of the current assembly in the image acquisition device coordinate system and the position pose information of the robot in the image acquisition device coordinate system;
establishing a digital twin model of the teleoperation environment based on a three-dimensional model of the current assembly, a three-dimensional model of the robot, and a relative positional relationship between the current assembly and the robot.