US 12,314,385 B1
Automated generation of anomaly scenarios for testing machine learned anomaly detection models
Jocelyn Beauchesne, Saint-Lormel (FR); John Lim Oh, Mukilteo, WA (US); Vasudha Shivamoggi, Cambridge, MA (US); and Roy Donald Hodgman, Cambridge, MA (US)
Assigned to Rapid7, Inc., Boston, MA (US)
Filed by Rapid7, Inc., Boston, MA (US)
Filed on Dec. 31, 2020, as Appl. No. 17/139,809.
Application 17/139,809 is a continuation in part of application No. 17/024,481, filed on Sep. 17, 2020, granted, now 12,088,600.
Application 17/139,809 is a continuation in part of application No. 17/024,506, filed on Sep. 17, 2020, granted, now 11,509,674.
Claims priority of provisional application 62/901,991, filed on Sep. 18, 2019.
Int. Cl. H04L 9/00 (2022.01); G06F 11/30 (2006.01); G06F 11/34 (2006.01); G06F 21/52 (2013.01); G06F 21/55 (2013.01); G06F 21/57 (2013.01); G06N 20/00 (2019.01)
CPC G06F 21/552 (2013.01) [G06F 11/3051 (2013.01); G06F 11/3457 (2013.01); G06F 21/52 (2013.01); G06F 21/577 (2013.01); G06N 20/00 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A system comprising:
one or more computing devices that implement an anomaly detection system, configured to:
train, using a machine learning technique, an anomaly detection model to detect anomalies in observation records of a plurality of machines, wherein individual ones of the observation records indicate presence or absence of different types of processes on individual ones of the machines;
perform a test of the anomaly detection model, including to:
select a scenario that includes a type of anomaly on at least one machine;
generate a test dataset that includes synthetic test data, wherein the synthetic test data includes (a) at least one anomalous record that contains an anomalous value indicative of the type of anomaly and (b) a plurality of other observation records that do not contain the anomalous value, wherein the anomalous record is generated according to a formula that randomly generates values of the anomalous record based on a selected distance between the anomaly record and a randomly selected one of the other observation records in the test dataset, and wherein the selected distance is checked to verify that the selected distance places the anomalous record at a location in a dimension space of the test dataset that falls below a specified density threshold; and
determine a model performance result of the anomaly detection model using the test dataset; and
based on a determination that the model performance result satisfies a performance criterion, promote the anomaly detection model to be used on real observation records collected from the machines.