US 12,266,440 B2
Automated instrument or component assistance using mixed reality in orthopedic surgical procedures
Sergii Poltaretskyi, Plougonvelin (FR); Damien Cariou, Loperhet (FR); Yannick Morvan, Saint Renan (FR); Vincent Gaborit, Saint Martin D'Hères (FR); Benjamin Dassonville, Saint Hilaire du Touvet (FR); and Jean Chaoui, Locmaria Plouzané (FR)
Assigned to HOWMEDICA OSTEONICS CORP., Mahwah, NJ (US)
Filed by HOWMEDICA OSTEONICS CORP., Mahwah, NJ (US)
Filed on Dec. 10, 2020, as Appl. No. 17/117,730.
Application 17/117,730 is a continuation of application No. PCT/US2019/037007, filed on Jun. 13, 2019.
Claims priority of provisional application 62/804,383, filed on Feb. 12, 2019.
Claims priority of provisional application 62/804,402, filed on Feb. 12, 2019.
Claims priority of provisional application 62/804,392, filed on Feb. 12, 2019.
Claims priority of provisional application 62/778,789, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,774, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,791, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,797, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,772, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,760, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,782, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,788, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,796, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,764, filed on Dec. 12, 2018.
Claims priority of provisional application 62/778,778, filed on Dec. 12, 2018.
Claims priority of provisional application 62/739,406, filed on Oct. 1, 2018.
Claims priority of provisional application 62/687,014, filed on Jun. 19, 2018.
Prior Publication US 2021/0093387 A1, Apr. 1, 2021
Int. Cl. G16H 20/40 (2018.01); A61B 5/00 (2006.01); A61B 5/11 (2006.01); A61B 17/14 (2006.01); A61B 17/16 (2006.01); A61B 17/17 (2006.01); A61B 34/00 (2016.01); A61B 34/10 (2016.01); A61B 90/00 (2016.01); A61B 90/92 (2016.01); A61F 2/40 (2006.01); G02B 27/00 (2006.01); G02B 27/01 (2006.01); G06F 3/01 (2006.01); G06F 3/04815 (2022.01); G06F 3/0482 (2013.01); G06F 18/21 (2023.01); G06F 30/10 (2020.01); G06F 30/20 (2020.01); G06N 3/08 (2023.01); G06N 20/00 (2019.01); G06T 7/00 (2017.01); G06T 7/11 (2017.01); G06T 7/55 (2017.01); G06T 11/00 (2006.01); G06T 19/00 (2011.01); G06T 19/20 (2011.01); G09B 5/06 (2006.01); G09B 9/00 (2006.01); G09B 19/00 (2006.01); G09B 23/28 (2006.01); G16H 30/20 (2018.01); G16H 30/40 (2018.01); G16H 40/20 (2018.01); G16H 40/60 (2018.01); G16H 40/63 (2018.01); G16H 40/67 (2018.01); G16H 50/30 (2018.01); G16H 50/50 (2018.01); G16H 50/70 (2018.01); G16H 70/20 (2018.01); G16H 70/60 (2018.01); G16H 80/00 (2018.01); H04N 13/122 (2018.01); H04N 13/332 (2018.01); A61B 17/00 (2006.01); A61B 17/15 (2006.01); A61B 34/20 (2016.01); A61B 90/50 (2016.01); A61F 2/46 (2006.01); G06F 3/0483 (2013.01); G06N 3/04 (2023.01); G16H 50/20 (2018.01)
CPC G16H 20/40 (2018.01) [A61B 5/1114 (2013.01); A61B 5/1121 (2013.01); A61B 5/1127 (2013.01); A61B 5/681 (2013.01); A61B 17/142 (2016.11); A61B 17/1604 (2013.01); A61B 17/1626 (2013.01); A61B 17/1659 (2013.01); A61B 17/1684 (2013.01); A61B 17/1703 (2013.01); A61B 34/10 (2016.02); A61B 34/25 (2016.02); A61B 34/76 (2016.02); A61B 90/08 (2016.02); A61B 90/36 (2016.02); A61B 90/361 (2016.02); A61B 90/37 (2016.02); A61B 90/39 (2016.02); A61B 90/92 (2016.02); A61F 2/40 (2013.01); A61F 2/4081 (2013.01); G02B 27/0075 (2013.01); G02B 27/017 (2013.01); G02B 27/0172 (2013.01); G06F 3/011 (2013.01); G06F 3/04815 (2013.01); G06F 3/0482 (2013.01); G06F 18/2163 (2023.01); G06F 30/10 (2020.01); G06F 30/20 (2020.01); G06N 3/08 (2013.01); G06N 20/00 (2019.01); G06T 7/0012 (2013.01); G06T 7/11 (2017.01); G06T 7/55 (2017.01); G06T 11/00 (2013.01); G06T 19/006 (2013.01); G06T 19/20 (2013.01); G09B 5/06 (2013.01); G09B 9/00 (2013.01); G09B 19/003 (2013.01); G09B 23/28 (2013.01); G16H 30/20 (2018.01); G16H 30/40 (2018.01); G16H 40/20 (2018.01); G16H 40/60 (2018.01); G16H 40/63 (2018.01); G16H 40/67 (2018.01); G16H 50/30 (2018.01); G16H 50/50 (2018.01); G16H 50/70 (2018.01); G16H 70/20 (2018.01); G16H 70/60 (2018.01); G16H 80/00 (2018.01); H04N 13/122 (2018.05); H04N 13/332 (2018.05); A61B 5/744 (2013.01); A61B 2017/00115 (2013.01); A61B 2017/00119 (2013.01); A61B 2017/00123 (2013.01); A61B 17/151 (2013.01); A61B 17/1775 (2016.11); A61B 17/1778 (2016.11); A61B 2034/102 (2016.02); A61B 2034/104 (2016.02); A61B 2034/105 (2016.02); A61B 2034/107 (2016.02); A61B 2034/108 (2016.02); A61B 2034/2048 (2016.02); A61B 2034/2051 (2016.02); A61B 2034/2055 (2016.02); A61B 2034/2065 (2016.02); A61B 2034/2068 (2016.02); A61B 2034/252 (2016.02); A61B 2034/254 (2016.02); A61B 2090/062 (2016.02); A61B 2090/067 (2016.02); A61B 2090/0801 (2016.02); A61B 2090/0807 (2016.02); A61B 2090/365 (2016.02); A61B 2090/366 (2016.02); A61B 2090/367 (2016.02); A61B 2090/368 (2016.02); A61B 2090/373 (2016.02); A61B 2090/374 (2016.02); A61B 2090/3762 (2016.02); A61B 2090/378 (2016.02); A61B 2090/3937 (2016.02); A61B 2090/3945 (2016.02); A61B 2090/397 (2016.02); A61B 2090/502 (2016.02); A61B 2505/05 (2013.01); A61B 2562/0219 (2013.01); A61F 2002/4011 (2013.01); A61F 2/4606 (2013.01); A61F 2/4612 (2013.01); A61F 2002/4633 (2013.01); A61F 2002/4658 (2013.01); A61F 2002/4668 (2013.01); G02B 2027/0141 (2013.01); G02B 2027/0174 (2013.01); G06F 3/0483 (2013.01); G06N 3/04 (2013.01); G06T 2200/24 (2013.01); G06T 2207/10016 (2013.01); G06T 2207/20036 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30008 (2013.01); G06T 2207/30052 (2013.01); G06T 2207/30204 (2013.01); G06T 2210/41 (2013.01); G06T 2219/2004 (2013.01); G06V 2201/03 (2022.01); G16H 50/20 (2018.01)] 34 Claims
OG exemplary drawing
 
1. A medical device system comprising:
a plurality of surgical items associated with an orthopedic surgical procedure, wherein the plurality of surgical items is identified in a surgical plan for a patient and wherein the surgical plan for the patient defines a sequence for using the plurality of surgical items over a course of the orthopedic surgical procedure; and
a visualization device configured to present a mixed reality (MR) presentation to a user; and
one or more processors configured to select a surgical item of the plurality of surgical items based on the orthopedic surgical procedure and present, in the MR presentation, virtual information that identifies the selected surgical item among the plurality of surgical items, wherein the virtual information is presented on or adjacent a position of the selected surgical item visible via the visualization device, and wherein the virtual information includes virtual elements that identify all of the surgical items in the plurality of surgical items while distinguishing a particular one of the surgical items corresponding to the selected surgical item, wherein the virtual information includes a color coordination scheme that distinguishes a first subset of the surgical items, a second subset of the surgical items, and a third subset of the surgical items, wherein the first subset of the surgical items corresponds to already used surgical items within the sequence defined in the surgical plan, the second subset of the surgical items corresponds to a currently used surgical item that is the selected surgical item within the sequence defined in the surgical plan, and the third subset of the surgical items corresponds to subsequently needed surgical items within the sequence defined in the surgical plan.