| CPC G09B 23/28 (2013.01) [G06F 3/017 (2013.01); G06T 7/20 (2013.01); G06T 19/006 (2013.01); G09B 5/06 (2013.01); G06T 2207/30004 (2013.01); G06T 2207/30196 (2013.01); G06T 2210/41 (2013.01); G06V 40/20 (2022.01)] | 9 Claims |
|
1. A virtual and real integrated medical teaching assistance system based on mixed reality, comprising:
a camera device, configured to obtain and provide a physical environment image of a physical space;
a virtualization computing device, coupled to the camera device, wherein the virtualization computing device comprises:
a first non-transitory computer-readable storage medium, configured to store first computer readable instructions; and
a first hardware processor, electrically connected to the first non-transitory computer-readable storage medium, and configured to execute the first computer readable instructions to make the virtualization computing device execute:
receiving the physical environment image from the camera device, and correcting and positioning a virtual posture model to a predefined position on the physical environment image;
performing gesture recognition on the physical environment image to obtain an operation gesture, and when a medical teaching function on a medical-teaching virtual interface is determined to be triggered, providing a trigger instruction corresponding to the medical teaching function;
receiving medical-teaching guidance information or a step instruction to select one of teaching steps of the medical-teaching guidance information; and
recognizing a hand position and detecting a hand motion on the hand position on the physical environment image based on the posture model positioning points, to generate human body information and hand motion information corresponding to the selected one of the teaching steps of the medical-teaching guidance information;
a teaching device, coupled to the virtualization computing device, wherein the teaching device comprises:
a second non-transitory computer-readable storage medium, configured to store second computer readable instructions; and
a second hardware processor, electrically connected to the second non-transitory computer-readable storage medium, and configured to execute the second computer readable instructions to make the teaching device execute:
generating and providing the medical-teaching virtual interface;
receiving the trigger instruction from the virtualization computing device, and providing the medical-teaching guidance information corresponding to the trigger instruction; and
receiving the human body information and the hand motion information from the virtualization computing device, determining motion correctness and a score of the human body information and the hand motion information corresponding to the selected one of the teaching steps of the medical-teaching guidance information, to generate the step instruction; and
a virtual-and-real display device, coupled to the virtualization computing device and the teaching device, and configured to display virtual-and-real integration of the posture model positioning points, the virtual posture model, the medical-teaching virtual interface, and the physical space, display a change in adjustment of the virtual posture model in real time, display virtual-and-real integration of the physical space and the teaching steps of the medical-teaching guidance information in sequential order, and display virtual-and-real integration of the physical space, each of the teaching steps of the medical-teaching guidance information and the motion correctness and the score thereof.
|