US 12,079,954 B2
Modifying sensor data using generative adversarial models
Victor Carbune, Winterthur (CH); Daniel M. Keysers, Stallikon (CH); and Thomas Deselaers, Zurich (CH)
Assigned to Google LLC, Mountain View, CA (US)
Appl. No. 17/603,362
Filed by Google LLC, Mountain View, CA (US)
PCT Filed Jun. 10, 2019, PCT No. PCT/US2019/036263
§ 371(c)(1), (2) Date Oct. 13, 2021,
PCT Pub. No. WO2020/251523, PCT Pub. Date Dec. 17, 2020.
Prior Publication US 2022/0198609 A1, Jun. 23, 2022
Int. Cl. G06K 9/40 (2006.01); G06T 3/4046 (2024.01); G06T 5/00 (2024.01); G06T 5/50 (2006.01)
CPC G06T 3/4046 (2013.01) [G06T 5/50 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
receiving a set of first training data generated by a first environmental sensor having a first quality, the first set of training data being of the first quality and comprising one or more defects;
receiving a set of second training data generated by a target environmental sensor having a target quality higher than the first quality, the set of second training data being of the target quality and not having the one or more defects in the first set of training data, wherein the target environmental sensor generates data of a same type as the first environmental sensor;
training, using the set of first training data and the set of second training data, a generative adversarial model to modify sensor data from the first environmental sensor by reducing a difference in quality associated with the one or more defects of the first set of training data between the sensor data generated by the first environmental sensor and sensor data generated by the target environmental sensor, wherein the training includes:
obtaining, from a generator model of the generative adversarial model and using one or more data items in the set of first training data, a set of modified sensor data having a quality different from the first quality;
inputting a set of data items comprising one or more data items in the set of second training data and the set of modified first sensor data into a discriminator model of the generative adversarial model;
determining, by the discriminator model and using the set of data items, a classification for each of the data items, the classification indicative of whether a data item originates from the set of modified sensor data or the set of second training data;
determining a classification error based on the classifications for each data item; and
adjusting, based on the classification error, the discriminator model and the generator model;
determining one or more known defects associated with an input environmental sensor;
providing an input set of sensor data generated by the input environmental sensor to the trained generative adversarial model, the input set of sensor data having the first quality and comprising the one or more known defects, wherein the input environmental sensor has a first resolution; and
generating, by the trained generative adversarial model, modified input sensor data having the target quality and not having the one or more known defects, wherein the target quality is a higher resolution than the first resolution of the input environmental sensor.