US 12,271,201 B2
Performance testing for robotic systems
John Redford, Cambridge (GB); Sebastian Kaltwang, Cambridge (GB); Blaine Rogers, Cambridge (GB); Jonathan Sadeghi, Bristol (GB); James Gunn, Cambridge (GB); Torran Elson, Bristol (GB); and Adam Charytoniuk, Cambridge (GB)
Assigned to Five AI Limited, Cambridge (GB)
Appl. No. 17/636,521
Filed by Five AI Limited, Bristol (GB)
PCT Filed Aug. 21, 2020, PCT No. PCT/EP2020/073562
§ 371(c)(1), (2) Date Feb. 18, 2022,
PCT Pub. No. WO2021/037760, PCT Pub. Date Mar. 4, 2021.
Claims priority of application No. 1912145 (GB), filed on Aug. 23, 2019; and application No. 20168311 (EP), filed on Apr. 6, 2020.
Prior Publication US 2022/0297707 A1, Sep. 22, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G05D 1/00 (2024.01); B60W 50/00 (2006.01); B60W 50/02 (2012.01); B60W 50/04 (2006.01); B60W 50/06 (2006.01); B60W 60/00 (2020.01); G06F 30/27 (2020.01); G06N 3/02 (2006.01); G06N 3/08 (2023.01); G06N 5/025 (2023.01); G06N 7/01 (2023.01); G06N 7/02 (2006.01); G06N 20/00 (2019.01)
CPC G05D 1/0221 (2013.01) [B60W 50/0098 (2013.01); B60W 50/0205 (2013.01); B60W 50/045 (2013.01); B60W 50/06 (2013.01); B60W 60/001 (2020.02); B60W 60/0015 (2020.02); G05D 1/0088 (2013.01); G05D 1/0214 (2013.01); G06F 30/27 (2020.01); G06N 3/02 (2013.01); G06N 3/08 (2013.01); G06N 5/025 (2013.01); G06N 7/01 (2023.01); G06N 7/023 (2013.01); G06N 20/00 (2019.01); B60W 2050/0052 (2013.01); B60W 2050/0215 (2013.01); B60W 2555/20 (2020.02)] 19 Claims
OG exemplary drawing
 
1. A computer system for testing and/or training a runtime stack for a robotic system, the computer system comprising:
at least one processor; and
at least one storage medium having encoded thereon executable instructions that, when executed by the at least one processor, cause the at least one processor to carry out a method comprising:
running a simulated scenario in a simulator configured to run simulated scenarios, in which a simulated agent interacts with one or more external objects, wherein the runtime stack is configured to make autonomous decisions for each simulated scenario in dependence on a time series of perception outputs computed for the simulated scenario and configured to generate a series of control signals for causing the simulated agent to execute the autonomous decisions as the simulated scenario progresses;
wherein running the simulated scenario comprises computing each perception output at least in part by:
computing a perception ground truth for a current time instant based on a current state of the simulated scenario,
applying a perception statistical performance model (PSPM) to the perception ground truth to determine a probabilistic perception uncertainty distribution for the current time instant, and
sampling the perception output from the probabilistic perception uncertainty distribution;
wherein the PSPM is for modelling a perception slice of the runtime stack and is configured to determine the probabilistic perception uncertainty distribution based on a set of parameters learned from a set of actual perception outputs generated using the perception slice to be modelled; and
wherein the PSPM comprises a time-dependent model such that the perception output sampled at the current time instant depends on at least one of: an earlier one of the perception outputs sampled at a previous time instant, or an earlier one of the perception ground truths computed for a previous time instant.