US 11,881,016 B2
Method and system for processing an image and performing instance segmentation using affinity graphs
Hiroaki Shimizu, Brussels (BE); Bert De Brabandere, Leuven (BE); Davy Neven, Leuven (BE); Marc Proesmans, Leuven (BE); and Luc Van Gool, Leuven (BE)
Assigned to TOYOTA MOTOR EUROPE, Brussels (BE); and KATHOLIEKE UNIVERSITEIT LEUVEN, K.U. LEUVEN R&D, Leuven (BE)
Appl. No. 17/264,164
Filed by TOYOTA MOTOR EUROPE, Brussels (BE); and KATHOLIEKE UNIVERSITEIT LEUVEN, Leuven (BE)
PCT Filed Sep. 21, 2018, PCT No. PCT/EP2018/075710
§ 371(c)(1), (2) Date Jan. 28, 2021,
PCT Pub. No. WO2020/057758, PCT Pub. Date Mar. 26, 2020.
Prior Publication US 2021/0287049 A1, Sep. 16, 2021
Int. Cl. G06N 3/08 (2023.01); G06F 18/21 (2023.01); G06N 3/045 (2023.01); G06V 10/82 (2022.01); G06V 10/44 (2022.01); G06V 10/26 (2022.01); G06V 20/00 (2022.01); G06F 18/20 (2023.01); G06V 10/764 (2022.01); G06V 20/56 (2022.01)
CPC G06V 20/00 (2022.01) [G06F 18/2163 (2023.01); G06F 18/29 (2023.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06V 10/26 (2022.01); G06V 10/454 (2022.01); G06V 10/764 (2022.01); G06V 10/82 (2022.01); G06V 20/56 (2022.01)] 8 Claims
OG exemplary drawing
 
1. A method for processing an image so as to perform instance segmentation, comprising:
a—inputting the image to a first neural network configured to output, for each pixel of the image, an affinity vector wherein the components of the vector are each associated with other pixels of the image at positions relative to the pixel predefined in an affinity pattern, the value of each component being set to a first value if the neural network determines that the other pixel associated with the component belongs to the same instance as the pixel of the image and set to a second value which differs from the first value if the neural network determines that the other pixel associated with the component does not belong to the same instance as the pixel of the image, the affinity vectors of all the pixels of the image forming an affinity graph,
b—inputting, to a second neural network, the affinity graph and a predefined seed-map having the resolution of the image and at least one pixel having a value set to the first value, so as to:
determine whether other pixels belong to the same instance as the at least one pixel of the seed-map having a value set to the first value, and
set at the first value the value of the other pixels determined as belonging to the same instance as the at least one pixel of the seed-map having a value set to the first value.
 
7. A system for processing an image so as to perform instance segmentation, comprising:
a module for inputting the image to a first neural network configured to output, for each pixel of the image, an affinity vector wherein the components of the vector are each associated with other pixels of the image at positions relative to the pixel predefined in an affinity pattern, the value of each component being set to a first value if the neural network determines that the other pixel associated with the component belongs to the same instance as the pixel of the image and set to a second value which differs from the first value if the neural network determines that the other pixel associated with the component does not belong to the same instance as the pixel of the image, the affinity vectors of all the pixels of the image forming an affinity graph,
a module for inputting, to a second neural network, the affinity graph and a predefined seed-map having the resolution of the image and at least one pixel having a value set to the first value, so as to:
determine whether other pixels belong to the same instance as the at least one pixel of the seed-map having a value set to the first value, and
set at the first value the value of the other pixels determined as belonging to the same instance as the at least one pixel of the seed-map having a value set to the first value.