US 11,983,824 B2
System and method for augmenting and synchronizing a virtual model with a physical model
Mordechai Avisar, Highland Heights, OH (US); and Alon Yakob Geri, Orange Village, OH (US)
Assigned to SURGICAL THEATER, INC., Beachwood, OH (US)
Filed by Surgical Theater, Inc., Mayfield Village, OH (US)
Filed on Jan. 31, 2021, as Appl. No. 17/163,471.
Claims priority of provisional application 63/062,921, filed on Aug. 7, 2020.
Claims priority of provisional application 62/968,340, filed on Jan. 31, 2020.
Prior Publication US 2021/0241534 A1, Aug. 5, 2021
Int. Cl. G06T 19/00 (2011.01); G06F 3/01 (2006.01); G06F 16/245 (2019.01); G06T 7/246 (2017.01); G06T 7/33 (2017.01); G06T 19/20 (2011.01); G16H 30/20 (2018.01)
CPC G06T 19/006 (2013.01) [G06F 3/012 (2013.01); G06F 16/245 (2019.01); G06T 7/246 (2017.01); G06T 7/33 (2017.01); G06T 19/003 (2013.01); G06T 19/20 (2013.01); G16H 30/20 (2018.01); G06T 2207/30204 (2013.01); G06T 2210/41 (2013.01); G06T 2219/2004 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A method for synchronizing and augmenting a virtual model with a physical model, comprising the steps of:
an AR synchronization computer receiving synchronization and navigation data from a navigation system and generating a physical frame of reference with respect to a registered physical model based on the synchronization and navigation data;
the AR synchronization computer registering an augmented reality head mounted display with the navigation system using the synchronization and navigation data, thereby enabling tracking movement of the augmented reality head mounted display with respect to the physical model;
the AR synchronization computer receiving data representative of a virtual model from a virtual model database;
the AR synchronization computer anchoring the virtual model to the physical frame of reference;
the AR synchronization computer receiving tracking data indicative of a position and angle of view of the augmented reality head mounted display with respect to the physical frame of reference; and
responsive to the AR synchronization computer determining that the physical model is within a field of view of the augmented reality head mounted display;
the AR synchronization computer rendering a virtual image from the virtual model in real time based on the received tracking data;
the AR synchronization computer streaming the virtual image to the augmented reality head mounted display, thereby generating a synchronized and augmented reality view of the physical model;
the AR synchronization computer registering a physical surgical tool having a plurality of markers with the navigation system using the synchronization and navigation data, thereby enabling tracking movement of the physical surgical tool with respect to the physical model;
the AR synchronization computer generating a virtual surgical tool representative of the physical surgical tool and comprising a plurality of virtual representations of the surgical tool markers;
the AR synchronization computer aligning the virtual surgical tool with the physical surgical tool by aligning the virtual representations of the physical tool markers with the physical tool markers; and
the AR synchronization computer streaming the virtual image to the augmented reality head mounted display further comprises streaming the virtual surgical tool to the augmented reality head mounted display.