US 12,340,709 B2
Adaptive tutoring system for machine tasks in augmented reality
Karthik Ramani, West Lafayette, IN (US); Gaoping Huang, West Lafayette, IN (US); Alexander J Quinn, West Lafayette, IN (US); Yuanzhi Cao, Redmond, WA (US); Tianyi Wang, West Lafayette, IN (US); and Xun Qian, West Lafayette, IN (US)
Assigned to Purdue Research Foundation, West Lafayette, IN (US)
Filed by Purdue Research Foundation, West Lafayette, IN (US)
Filed on Nov. 3, 2021, as Appl. No. 17/517,949.
Claims priority of provisional application 63/162,108, filed on Mar. 17, 2021.
Claims priority of provisional application 63/109,154, filed on Nov. 3, 2020.
Prior Publication US 2022/0139254 A1, May 5, 2022
Int. Cl. G09B 5/02 (2006.01); G06N 20/00 (2019.01); G06T 13/40 (2011.01); G06T 19/00 (2011.01); G06V 10/25 (2022.01); G06V 40/20 (2022.01); G09B 19/00 (2006.01); G09B 19/24 (2006.01)
CPC G09B 19/0069 (2013.01) [G06N 20/00 (2019.01); G06T 13/40 (2013.01); G06T 19/006 (2013.01); G06V 10/25 (2022.01); G06V 40/23 (2022.01); G09B 5/02 (2013.01); G09B 19/24 (2013.01); G06T 2210/36 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A method for providing tutorial guidance for performing a machine task, the method comprising:
storing, in a memory, tutorial data defining a plurality of steps of a machine task, the plurality of steps including interactions with a machine in an environment;
displaying, on a display, an augmented reality graphical user interface including graphical tutorial elements that convey information regarding the plurality of steps of the machine task, the graphical tutorial elements being superimposed on at least one of (i) the machine and (ii) the environment;
monitoring, with at least one sensor, motions of a first person and states of the machine during a performance of the machine task by the first person by (i) detecting a position of a respective component of the machine in the environment, (ii) determining a bounding box in an image that encompasses the respective component of the machine within the image based on the position of the respective component of the machine, and (iii) determining a state of the respective component using a machine learning model based on the image cropped by the bounding box;
determining, with a processor, during a performance of a respective step of the plurality of steps involving a respective component of the machine, that the first person is stuck in the performance of the respective step based on the monitored motions of the first person and the monitored states of the machine; and
increasing, with the processor, a level of detail of the graphical tutorial elements in the graphical user interface that are displayed with respect to the respective component in response to determining that the first person is stuck in the performance of the machine task.