| CPC G06V 10/7788 (2022.01) [G06F 3/0482 (2013.01); G06F 3/0488 (2013.01); G06V 10/774 (2022.01); G06V 10/945 (2022.01); G06V 20/70 (2022.01); G06V 2201/03 (2022.01)] | 20 Claims |
|
1. A method for facilitating collection of annotations for images or video associated with a surgical application for training of a machine learning model, the method comprising:
obtaining over a computer network at an annotation server, inputs from an administrative user interface of an administrator application executing on an administrative client device, the inputs corresponding to a set of predefined input fields, and the inputs specifying at least an assignment of an annotation job to a set of annotators selectable from a predefined list of available annotators, inclusion criteria for identifying content items for annotating in the annotation job, a predefined set of selectable labels for the annotation job, and a target number of judgments for the annotation job;
configuring, by a job management engine of an annotation server, an annotation job associated with the machine learning model based on the inputs;
obtaining, from a content database based on the inputs, a set of unannotated content items meeting the inclusion criteria;
for each of the set of unannotated content items:
applying the machine learning model to generate a prediction associated with the content item and a confidence metric associated with the prediction;
determining if the confidence metric meets a predefined confidence threshold; and
responsive to the confidence metric failing to meet the confidence threshold, adding the content item to an annotation set associated with the annotation job;
facilitating availability of an on-demand annotation sessions for the set of annotators via respective annotation application executing on respective annotator client devices, wherein during an on-demand annotation session, an annotation application sequentially presents content items from the annotation set, obtains selections of respective labels via interactions captured in a user interface of the annotation application, and transmits the selections to the annotation server;
at the annotation server, obtaining and aggregating the respective labels for the presented content items obtained from the on-demand annotation sessions to generate aggregated label data;
determining, for each of the content items in the annotation set based on the aggregated label data, if the target number of judgments is met; and
responsive to the target number of judgments being met for a given content item in the annotation set, adding the given content item and the aggregated label data to a training set;
updating parameters of the machine learning model by training the machine learning model using the training set; and
storing the machine learning model.
|