US 12,340,310 B2
Energy tool activation detection in surgical videos using deep learning
Meysam Torabi, Union City, CA (US); Varun Goel, Santa Clara, CA (US); Jocelyn Elaine Barker, San Jose, CA (US); Rami Abukhalil, Santa Clara, CA (US); Richard W. Timm, Cincinnati, OH (US); and Pablo E. Garcia Kilroy, Menlo Park, CA (US)
Assigned to Verb Surgical Inc., Santa Clara, CA (US)
Filed by Verb Surgical Inc., Santa Clara, CA (US)
Filed on Jul. 27, 2022, as Appl. No. 17/875,122.
Prior Publication US 2024/0037385 A1, Feb. 1, 2024
Int. Cl. G06N 3/08 (2023.01); G06N 3/045 (2023.01)
CPC G06N 3/08 (2013.01) [G06N 3/045 (2023.01)] 17 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
receiving a surgical video of a surgical procedure involving energy tool activations;
applying a sequence of sampling windows to the surgical video to generate a sequence of windowed samples of the surgical video;
for each windowed sample in the sequence of windowed samples, applying a deep-learning model to a sequence of video frames within the windowed sample to generate an activation/non-activation inference and a confidence level associated with the activation/non-activation inference, thereby generating a sequence of activation/non-activation inferences and a sequence of associated confidence levels; and
identifying a sequence of activation events based on the sequence of activation/non-activation inferences and the sequence of associated confidence levels, wherein identifying an activation event in the sequence of activation events comprises:
identifying multiple consecutive activation inferences located between two non-activation inferences, in the sequence of activation/non-activation inferences, as a single activation;
identifying the first and the last inferences in the multiple consecutive activation inferences corresponding to two partial-activation windowed samples that partially overlap with the activation event;
determining first and second amounts of partial-overlap between the two partial-activation windowed samples, respectively, and the activation event, based on the confidence levels associated with the first and the last inferences; and
computing a duration of the activation event, wherein the duration comprises a sum of the first and second amounts of partial-overlap.