extracting, from a source sentence, a first phrase and a second phrase;

executing a first encoder neural network that generates, based on the first phrase, a first encoding of the first phrase and, based on the second phrase, a second encoding of the second phrase;

executing one or more neural networks to convert a plurality of bounding boxes for a plurality of objects included in a source image into a plurality of box states corresponding to the plurality of bounding boxes, wherein each box state included in the plurality of box states encodes one or more interrelationships between a corresponding bounding box included in the plurality of bounding boxes and one or more additional bounding boxes included in the plurality of bounding boxes;

executing a decision neural network to generate a first plurality of grounding decisions based on input that includes (i) a first box state that is included in the plurality of box states and corresponds to a first bounding box included in the plurality of bounding boxes, (ii) the first encoding, and (iii) the second encoding;

performing one or more comparisons using the first plurality of grounding decisions and a decision threshold to determine that a first grounding decision included in the first plurality of grounding decisions indicates that the first phrase matches the first bounding box;

generating a first matched pair that specifies the first phrase and the first bounding box; and

causing one or more annotation operations to be performed on the source image based on the first matched pair.