US 12,315,036 B2
Mixed reality feedback systems that cooperate to increase efficient perception of complex data feeds
Frederick E. Shelton, IV, Hillsboro, OH (US); Matthew D. Cowperthwait, Cincinnati, OH (US); Kevin M. Fiebig, Cincinnati, OH (US); Shane R. Adams, Lebanon, OH (US); Cory G. Kimball, Hamilton, OH (US); Monica L. Z. Rivard, Cincinnati, OH (US); Leonardo N. Rossoni, Rahway, NJ (US); Risto Kojcev, Santa Clara, CA (US); and Felix J. Bork, Schnürpflingen (DE)
Assigned to Cilag GmbH International, Zug (CH)
Filed by Cilag GmbH International, Zug (CH)
Filed on Mar. 7, 2022, as Appl. No. 17/688,671.
Claims priority of provisional application 63/284,326, filed on Nov. 30, 2021.
Claims priority of provisional application 63/174,674, filed on Apr. 14, 2021.
Prior Publication US 2022/0335696 A1, Oct. 20, 2022
Int. Cl. G06T 11/00 (2006.01); A61B 34/00 (2016.01); A61B 34/20 (2016.01); A61B 34/30 (2016.01); A61B 34/32 (2016.01); A61B 90/00 (2016.01); G06F 3/14 (2006.01); G06F 3/147 (2006.01); G06T 7/00 (2017.01); G06T 7/20 (2017.01); G06T 19/00 (2011.01); G06V 20/20 (2022.01); G08B 21/18 (2006.01); G16H 20/40 (2018.01); G16H 40/67 (2018.01); H04L 9/40 (2022.01); H04L 67/12 (2022.01); H04W 24/10 (2009.01); H04W 76/14 (2018.01); A61B 34/10 (2016.01)
CPC G06T 11/00 (2013.01) [A61B 34/20 (2016.02); A61B 34/25 (2016.02); A61B 34/30 (2016.02); A61B 34/32 (2016.02); A61B 34/76 (2016.02); A61B 90/36 (2016.02); A61B 90/361 (2016.02); A61B 90/37 (2016.02); A61B 90/39 (2016.02); G06F 3/14 (2013.01); G06F 3/1454 (2013.01); G06F 3/147 (2013.01); G06T 7/0012 (2013.01); G06T 7/20 (2013.01); G06T 19/006 (2013.01); G06V 20/20 (2022.01); G08B 21/182 (2013.01); G16H 20/40 (2018.01); G16H 40/67 (2018.01); H04L 63/105 (2013.01); H04L 67/12 (2013.01); H04W 24/10 (2013.01); H04W 76/14 (2018.02); A61B 2034/102 (2016.02); A61B 2034/2055 (2016.02); A61B 2034/2072 (2016.02); A61B 2090/365 (2016.02); A61B 2090/368 (2016.02); A61B 2090/371 (2016.02); A61B 2090/372 (2016.02); A61B 2090/373 (2016.02); A61B 2090/3937 (2016.02); A61B 2090/3975 (2016.02); G06T 2207/10028 (2013.01); G06T 2207/30024 (2013.01); G06T 2210/41 (2013.01)] 20 Claims
OG exemplary drawing
 
1. An augmented reality (AR) device for use during a surgical procedure, the AR device comprising:
a processor configured to:
receive, from an imaging device, a first data feed comprising a real image of a surgical area during the surgical procedure;
receive, from a sensor device, a second data feed comprising sensed data, wherein the sensed data comprises functional data associated with functionality of a surgical instrument;
generate AR content based on the first and second data feeds, wherein the processor being configured to generate the AR content based on the first and second data feeds comprises the processor being configured to generate the AR content based on the functional data associated with functionality of the surgical instrument, and the AR content comprises an alert associated with at least one of: performance of the surgical instrument, or incompatible usage of the surgical instrument; and
output the AR content, wherein the AR content comprises a visual portion and a non-visual portion.