CPC G05D 1/0221 (2013.01) [B60W 50/0098 (2013.01); B60W 50/0205 (2013.01); B60W 50/045 (2013.01); B60W 50/06 (2013.01); B60W 60/001 (2020.02); B60W 60/0015 (2020.02); G05D 1/0088 (2013.01); G05D 1/0214 (2013.01); G06F 30/27 (2020.01); G06N 3/02 (2013.01); G06N 3/08 (2013.01); G06N 5/025 (2013.01); G06N 7/01 (2023.01); G06N 7/023 (2013.01); G06N 20/00 (2019.01); B60W 2050/0052 (2013.01); B60W 2050/0215 (2013.01); B60W 2555/20 (2020.02)] | 17 Claims |
1. A computer system for testing and/or training a runtime stack for a robotic system, the computer system comprising:
at least one processor; and
at least one storage medium having encoded thereon executable instructions that, when executed by the at least one processor, cause the at least one processor to carry out a method comprising:
running a simulated scenario in a simulator configured to run simulated scenarios, in which a simulated agent interacts with one or more external objects, wherein the runtime stack is configured to make autonomous decisions for each simulated scenario in dependence on a time series of perception outputs computed for the simulated scenario and configured to generate a series of control signals for causing the simulated agent to execute the autonomous decisions as the simulated scenario progresses;
wherein running the simulated scenario comprises computing each perception output at least by:
computing a perception ground truth based on a current state of the simulated scenario,
applying a perception statistical performance model (PSPM) to the perception ground truth to determine a probabilistic perception uncertainty distribution, and
sampling the perception output from the probabilistic perception uncertainty distribution;
wherein the PSPM is for modelling a perception slice of the runtime stack and is configured to determine the probabilistic perception uncertainty distribution based on a set of parameters learned from a set of actual perception outputs generated using the perception slice to be modelled; and
re-running the simulated scenario based on a time series of perception ground truths determined for the re-run scenario, without applying the PSPM to those perception ground truths and therefore without perception errors.
|