US 12,229,179 B1
Mitigating bias in multimodal models via query transformation
Matthaeus Kleindessner, Tuebingen (DE); Christopher Michael Russell, Tuebingen (DE); Kailash Budhathoki, Tuebingen (DE); Ali Caner Turkmen, Berlin (DE); Siqi Deng, Bellevue, WA (US); Varad Gunjal, Brooklyn, NY (US); Ashwin Swaminathan, Dublin, CA (US); Raghavan Manmatha, San Francisco, CA (US); and Hao Yang, Seattle, WA (US)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Nov. 20, 2023, as Appl. No. 18/515,105.
Int. Cl. G06F 16/40 (2019.01); G06F 16/432 (2019.01); G06F 16/53 (2019.01)
CPC G06F 16/432 (2019.01) [G06F 16/53 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A system comprising:
computer-readable memory storing a multimodal (MM) embedding model; and
one or more processors configured by executable instructions to:
receive an input query;
determine that the input query expresses an intent to perform an image search and that at least one criterion of the image search comprises a first sensitive attribute;
generate a query embedding of the input query using the multimodal (MM) embedding model;
obtain a bias mitigation transformation associated with a second sensitive attribute different from the first sensitive attribute;
generate, based on the query embedding and the bias mitigation transformation, a transformed query embedding that suppresses at least a portion of the query embedding related to the second sensitive attribute without suppressing any portion of the query embedding related to the first sensitive attribute;
execute, using the transformed query embedding, a similarity search in a media embedding model to identify one or more image embeddings that are similar to the transformed query embedding, wherein the one or more image embeddings are generated by the multimodal (MM) embedding model; and
transmit the one or more image embeddings.