	US 11,854,119 B2
	Automatic object re-colorization
Siavash Khodadadeh, Orlando, FL (US); Zhe Lin, Fremont, CA (US); Shabnam Ghadar, Menlo Park, CA (US); Saeid Motiian, San Francisco, CA (US); Richard Zhang, San Francisco, CA (US); Ratheesh Kalarot, San Jose, CA (US); and Baldo Faieta, San Francisco, CA (US)
Assigned to Adobe Inc., San Jose, CA (US)
Filed by Adobe Inc., San Jose, CA (US)
Filed on Jan. 22, 2021, as Appl. No. 17/155,570.
Prior Publication US 2022/0237830 A1, Jul. 28, 2022
Int. Cl. G06N 3/08 (2023.01); G06T 7/90 (2017.01); G06N 3/045 (2023.01); G06T 11/00 (2006.01)

CPC G06T 11/001 (2013.01) [G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06T 7/90 (2017.01)]

20 Claims

1. A computer-implemented method comprising:

receiving a request to recolor an object in an image, the request including an object identifier and a color identifier;

identifying an object in the image associated with the object identifier;

generating a mask corresponding to the object in the image;

providing the image, the mask, and the color identifier to a color transformer network, the color transformer network trained to recolor objects in input images, wherein the color transformer network comprises a network architecture with concatenating layers from an encoder to a decoder, and wherein the mask is concatenated with features of layers of the encoder and a color mask is concatenated with features of layers of the decoder; and

generating, by the color transformer network, a recolored image, wherein the object in the recolored image has been recolored to a color corresponding to the color identifier.

9. A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to:

receive a request to recolor an object in an image, the request including an object identifier and a color identifier;

identify an object in the image associated with the object identifier;

generate a mask corresponding to the object in the image;

provide the image, the mask, and the color identifier to a color transformer network, the color transformer network trained to recolor objects in input images, wherein the color transformer network comprises a network architecture with concatenating layers from an encoder to a decoder, and wherein the mask is concatenated with features of layers of the encoder and a color mask is concatenated with features of layers of the decoder; and

generate, by the color transformer network, a recolored image, wherein the object in the recolored image has been recolored to a color corresponding to the color identifier.

17. A computer-implemented method comprising:

generating, by a color transformer network, a first training image based on an image and a corresponding a mask from a training image set and a target color;

generating, by the color transformer network, a second training image based on the first training image, a color tag associated with the image, and the mask;

providing, by the color transformer network, the first training image, a first object tag, the target color, the second training image, a second object tag, and the color tag to a paired adversarial discrimination network to determine if the first and second training images are a positive pair or a negative pair; and

training the color transformer network to learn updated transformations using an adversarial loss based on the determination of the paired adversarial discrimination network, which outputs a positive result if the color tag and the target color match object colors of their respective images and if the first object tag and the second object tag match.