US 12,292,741 B2
Performance testing for robotic systems
John Redford, Cambridge (GB); Benedict Peters, Cambridge (GB); and Simon Walker, Bristol (GB)
Assigned to Five AI Limited, Cambridge (GB)
Appl. No. 17/636,530
Filed by Five AI Limited, Bristol (GB)
PCT Filed Aug. 21, 2020, PCT No. PCT/EP2020/073563
§ 371(c)(1), (2) Date Feb. 18, 2022,
PCT Pub. No. WO2021/037761, PCT Pub. Date Mar. 4, 2021.
Claims priority of application No. 1912145 (GB), filed on Aug. 23, 2019; and application No. 20168311 (EP), filed on Apr. 6, 2020.
Prior Publication US 2022/0269279 A1, Aug. 25, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G05D 1/00 (2024.01); B60W 50/00 (2006.01); B60W 50/02 (2012.01); B60W 50/04 (2006.01); B60W 50/06 (2006.01); B60W 60/00 (2020.01); G06F 30/27 (2020.01); G06N 3/02 (2006.01); G06N 3/08 (2023.01); G06N 5/025 (2023.01); G06N 7/01 (2023.01); G06N 7/02 (2006.01); G06N 20/00 (2019.01)
CPC G05D 1/0221 (2013.01) [B60W 50/0098 (2013.01); B60W 50/0205 (2013.01); B60W 50/045 (2013.01); B60W 50/06 (2013.01); B60W 60/001 (2020.02); B60W 60/0015 (2020.02); G05D 1/0088 (2013.01); G05D 1/0214 (2013.01); G06F 30/27 (2020.01); G06N 3/02 (2013.01); G06N 3/08 (2013.01); G06N 5/025 (2013.01); G06N 7/01 (2023.01); G06N 7/023 (2013.01); G06N 20/00 (2019.01); B60W 2050/0052 (2013.01); B60W 2050/0215 (2013.01); B60W 2555/20 (2020.02)] 17 Claims
OG exemplary drawing
 
1. A computer system for testing and/or training a runtime stack for a robotic system, the computer system comprising:
at least one processor; and
at least one storage medium having encoded thereon executable instructions that, when executed by the at least one processor, cause the at least one processor to carry out a method comprising:
running a simulated scenario in a simulator configured to run simulated scenarios, in which a simulated agent interacts with one or more external objects, wherein the runtime stack is configured to make autonomous decisions for each simulated scenario in dependence on a time series of perception outputs computed for the simulated scenario and configured to generate a series of control signals for causing the simulated agent to execute the autonomous decisions as the simulated scenario progresses;
wherein running the simulated scenario comprises computing each perception output at least by:
computing a perception ground truth based on a current state of the simulated scenario,
applying a perception statistical performance model (PSPM) to the perception ground truth to determine a probabilistic perception uncertainty distribution, and
sampling the perception output from the probabilistic perception uncertainty distribution;
wherein the PSPM is for modelling a perception slice of the runtime stack and is configured to determine the probabilistic perception uncertainty distribution based on a set of parameters learned from a set of actual perception outputs generated using the perception slice to be modelled; and
re-running the simulated scenario based on a time series of perception ground truths determined for the re-run scenario, without applying the PSPM to those perception ground truths and therefore without perception errors.