US 11,926,064 B2
Remote control manipulator system and remote control assistance system
Shota Narasaki, Tokyo (JP); Noboru Kawaguchi, Tokyo (JP); Masaki Hirano, Tokyo (JP); Muneharu Kuwata, Tokyo (JP); Masashige Suwa, Tokyo (JP); Toshiyuki Ando, Tokyo (JP); Hitomi Ono, Tokyo (JP); Akihiro Fujie, Tokyo (JP); Yoshitaka Koda, Tokyo (JP); and Toru Sato, Tokyo (JP)
Assigned to MITSUBISHI ELECTRIC CORPORATION, Tokyo (JP)
Appl. No. 18/265,269
Filed by Mitsubishi Electric Corporation, Tokyo (JP)
PCT Filed Dec. 10, 2021, PCT No. PCT/JP2021/045527
§ 371(c)(1), (2) Date Jun. 5, 2023,
PCT Pub. No. WO2022/124398, PCT Pub. Date Jun. 16, 2022.
Claims priority of application No. 2020-205183 (JP), filed on Dec. 10, 2020.
Prior Publication US 2024/0033930 A1, Feb. 1, 2024
Int. Cl. G06F 17/00 (2019.01); B25J 9/16 (2006.01); G06T 17/00 (2006.01); H04N 13/117 (2018.01); H04N 13/239 (2018.01); H04N 13/332 (2018.01); H04N 13/38 (2018.01); H04N 13/398 (2018.01)
CPC B25J 9/1689 (2013.01) [G06T 17/00 (2013.01); H04N 13/117 (2018.05); H04N 13/239 (2018.05); H04N 13/332 (2018.05); H04N 13/38 (2018.05); H04N 13/398 (2018.05)] 27 Claims
OG exemplary drawing
 
1. A remote control manipulator system comprising:
a manipulator controlled by an operator remotely to handle an object;
an operation instruction inputter through which the operator inputs an operation instruction instructing an operation of the manipulator to move or to be still;
a control signal generator to generate a control signal controlling the manipulator from the operation instruction;
a right camera and a left camera mounted on the manipulator, the right camera and the left camera having a predetermined positional relationship;
a stereoscopic display device to display a right-eye image captured by the right camera and a left-eye image captured by the left camera to be viewed stereoscopically;
a position and posture sensor to detect position and posture information representing a position and a posture of the manipulator;
a structure data storage to store manipulator structure data representing a structure of the manipulator and camera structure data representing structures of the right camera and the left camera;
a manipulator model generator to generate a manipulator model being a three-dimensional model of the manipulator including the right camera and the left camera by referring to the manipulator structure data, the camera structure data, and the position and posture information;
an image processing model generator to perform image processing on the right-eye image and the left-eye image based on positions and capturing directions of the right camera and the left camera included in the manipulator model to generate a surrounding environment model being a three-dimensional model of the object and a surrounding environment being a thing being present around the object;
a model integrator to generate an integrated three-dimensional model being a three-dimensional model obtained by integrating the surrounding environment model and the manipulator model;
a model image generator to generate a model image being an image of the integrated three-dimensional model viewed from a viewpoint designated by the operator; and
a model image display device to display the model image and configure to be able to be viewed by the operator switching between the stereoscopic display device and the model image display device through moving the head of the operator.