US 12,244,485 B1
Methods and apparatus for monitoring a sensor validator
Volodymyr Ivanchenko, Woodside, CA (US); and Volker Grabe, Redwood City, CA (US)
Assigned to Waymo LLC, Mountain View, CA (US)
Filed by Waymo LLC, Mountain View, CA (US)
Filed on Aug. 16, 2023, as Appl. No. 18/450,743.
Application 18/450,743 is a continuation of application No. 17/131,388, filed on Dec. 22, 2020, granted, now 11,765,067.
Claims priority of provisional application 62/954,445, filed on Dec. 28, 2019.
Int. Cl. H04L 43/50 (2022.01); B60W 50/02 (2012.01); B60W 50/04 (2006.01); B60W 50/14 (2020.01); B60W 60/00 (2020.01); H04L 43/065 (2022.01); H04L 67/12 (2022.01)
CPC H04L 43/50 (2013.01) [B60W 50/0205 (2013.01); B60W 50/045 (2013.01); B60W 50/14 (2013.01); B60W 60/001 (2020.02); B60W 60/005 (2020.02); H04L 43/065 (2013.01); H04L 67/12 (2013.01); B60W 2050/146 (2013.01); B60W 2420/403 (2013.01); B60W 2420/408 (2024.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
receiving sensor data from a sensor coupled to an autonomous vehicle, wherein the sensor data is generated by the sensor based on an environment surrounding the autonomous vehicle;
determining, by a computing device, a true positive detection rate of a sensor validator on an autonomous vehicle, wherein the sensor validator is configured to detect problems with the sensor based on the sensor data, wherein determining the true positive detection rate comprises:
mutating at least one sample of the sensor data to generate at least one sample of mutated sensor data while leaving at least one sample of non-mutated sensor data, wherein the at least one sample of mutated sensor data simulates a particular problem with the sensor;
receiving, from the sensor validator, a first output for the at least one sample of mutated sensor data and a second output for the at least one sample of non-mutated sensor data, wherein the first output indicates whether the sensor validator has detected any problem with the sensor based on the at least one sample of mutated sensor data, and wherein the second output indicates whether the sensor validator has detected any problem with the sensor based on the at least one sample of non-mutated sensor data; and
determining whether the first output indicates that the sensor validator has detected the particular problem with the sensor simulated by the at least one sample of mutated sensor data;
determining, by the computing device, a false positive detection rate of the sensor validator;
determining, by the computing device, a performance metric associated with the sensor validator based on at least the true positive detection rate, the false positive detection rate, and a latency of the sensor validator; and
generating, by the computing device, an alert in response to a determination that the performance metric fails to satisfy a threshold.