| CPC G06N 20/00 (2019.01) | 20 Claims |
|
4. A computer-implemented method comprising:
receiving, at a model selector of a prognostics and health management service of a multi-tenant provider network, a request to perform model selection, wherein the multi-tenant provider network receives sensor data from one or more managed devices;
evaluating, by the model selector executing model selection code using one or more processors, a plurality of trained models to select a trained model by:
applying, as input, a common set of testing data to the plurality of trained models to generate, as output from the plurality of trained models, a plurality of metrics for each of the plurality of trained models, the plurality of metrics comprising all of:
a forewarning time metric of how much in advance of a failure an alert can be raised,
an event recall metric of how many failure events were alerted to in advance of failure,
an event precision metric of a ratio of true and false positives, and
an area under a receiver operating characteristic (ROC) curve,
calculating, for each of the plurality of trained models, a weighted harmonic mean of all of the plurality of metrics generated for the trained model, and
selecting one of the plurality of trained models based on the calculated weighted harmonic means;
causing a prediction to be made, using the selected model, of a remaining useful life of a managed device of the one or more managed devices;
causing, in response to the prediction, maintenance to be performed on the managed device;
generating and providing a report regarding the selected model, wherein the report comprises the forewarning time metric, the event recall metric, the event precision metric, and the area under the ROC curve;
receiving implicit user feedback including a number of discarded alerts; and
reevaluating the plurality of trained models, based on the implicit user feedback, to select a different trained model, the different trained model is different from the selected model.
|