US 12,071,030 B2
Camera-sensor fusion module for surface detection and fleet vehicle control systems and methods
Garrett Korda Drayna, San Carlos, CA (US); and Jens Paul Windau, San Francisco, CA (US)
Assigned to Lyft, Inc., San Francisco, CA (US)
Filed by Lyft, Inc., San Francisco, CA (US)
Filed on Dec. 23, 2019, as Appl. No. 16/726,156.
Prior Publication US 2021/0191424 A1, Jun. 24, 2021
Int. Cl. G05D 1/02 (2020.01); B60L 50/20 (2019.01); G05B 13/02 (2006.01); G05D 1/00 (2006.01); G06N 3/04 (2023.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01); G06V 10/80 (2022.01); G06V 10/82 (2022.01); G06V 20/56 (2022.01); G07C 5/08 (2006.01); B60L 15/20 (2006.01); G06Q 30/04 (2012.01)
CPC B60L 50/20 (2019.02) [G05B 13/027 (2013.01); G05D 1/0231 (2013.01); G05D 1/027 (2013.01); G05D 1/0291 (2013.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06V 10/803 (2022.01); G06V 10/82 (2022.01); G06V 20/56 (2022.01); G07C 5/0808 (2013.01); G07C 5/0825 (2013.01); G07C 5/0833 (2013.01); G07C 5/085 (2013.01); B60L 15/20 (2013.01); G05D 1/0278 (2013.01); G06Q 30/04 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A fleet vehicle control system for a fleet vehicle of a plurality of fleet vehicles, the fleet vehicle control system comprising:
a camera-sensor fusion package configured to detect sensor data of an operational surface on which the fleet vehicle operates, wherein the camera-sensor fusion package includes at least (1) a first sensor of a first type and (2) a second sensor of a second type that is different from the first type;
one or more non-transitory computer-readable storage media embodying instructions; and
one or more processors coupled to the non-transitory computer-readable storage media and operable to execute the instructions to cause the fleet vehicle control system to perform operations comprising:
receiving, using the camera-sensor fusion package of the fleet vehicle, (1) first data associated with the operational surface, wherein the first data is captured by the first sensor and comprises at least image data of the operational surface in a field of view of the first sensor, and (2) second data associated with the operational surface, wherein the second data is captured by the second sensor which is different from the first sensor;
determining, by the fleet vehicle using a fusion algorithm based on a neural network model, a surface classification of the operational surface, wherein the fusion algorithm switches between a first detection model configured to perform surface classification using the first data and a second detection model configured to perform surface classification using the second data, wherein the switching is determined based on one or more criteria associated with the first and second data, wherein the fusion algorithm is trained for surface classifications of operational surfaces associated with the plurality of fleet vehicles;
determining an operational parameter comprising a restricted movement requirement for the fleet vehicle based, at least in part, on the surface classification; and
causing the fleet vehicle to come to a stop based, at least in part, on the restricted movement requirement determined based on the surface classification.