US 12,456,266 B2
Operation assistance system
Wen-Yuh Jywe, Taipei (TW); Tung-Hsing Hsieh, Taipei (TW); Shang-Kai Liao, Taipei (TW); Yung-Chuan Huang, Taipei (TW); and Ruo-Heng Wang, Taipei (TW)
Assigned to National Taiwan University, Taipei (TW)
Filed by National Taiwan University, Taipei (TW)
Filed on Apr. 21, 2023, as Appl. No. 18/137,452.
Claims priority of application No. 112108159 (TW), filed on Mar. 6, 2023.
Prior Publication US 2024/0303936 A1, Sep. 12, 2024
Int. Cl. G06T 19/00 (2011.01); G02B 27/01 (2006.01); G06V 30/224 (2022.01)
CPC G06T 19/006 (2013.01) [G02B 27/0172 (2013.01); G06V 30/2247 (2022.01); G06T 2219/024 (2013.01)] 15 Claims
OG exemplary drawing
 
1. An operation assistance system, adaptable for providing operational guidance to a subject device, comprising:
a user device, comprising a wearable display, capable of observing the subject device, capturing on-site video, and simultaneously displaying visual aids, the visual aids being interactive with the subject device for providing a positional reference for a wearer in relation to the subject device to perform interactive operations;
an intelligent monitor device, coupled to the subject device, arranged to monitor statuses of multiple sensors related to the subject device to determine a fault state and cause of the fault state;
an object database, configured to store a plurality of component models, and comprising a component model of the subject device;
a document database, configured to store text files of predetermined formats, and including operation manuals for the subject device;
a workflow database, configured to store a plurality of procedure scripts, wherein the procedure scripts describe operational steps for the subject device in various operational scenarios including operational steps for solving a particular fault scenario, along with relevant component models and document files required in the operational steps; and
a server, coupled to the user device and the intelligent monitor device, the server comprising a processor module coupled to object database, the document database, and the workflow database, executing a procedure script in the workflow database based on the fault state, and inferencing the object database and the document database to generate the visual aids and provide the visual aids to the user device based on the fault state;
wherein the user device is further configured to allow the wearer to proprioceptively perceive the visual aids displayed in a specific relative position in an on-site space where the subject device under test is located using an on-site coordinate system and depth information.