a semantic keypoint module including instructions that when executed by the processor cause the processor to: 1) receive, from an image capture device, a sequence of images of a scene that includes a view of a vehicle with lights, and 2) determine, based on inputting the images into a first neural network, both semantic keypoints, in the images and associated with the lights, and classifications for the semantic keypoints wherein a semantic keypoint, of the semantic keypoints, corresponds to a single pixel near a center and away from an edge of an object in an image of the images and the first neural network is a deep neural network; and

a classification module including instructions that when executed by the processor cause the processor to: 1) obtain difference images that are each a difference between successive images from among the sequence of images, the successive images being aligned based on their respective semantic keypoints, and 2) determine a classification of the lights based at least in part on the difference images by inputting the difference images into a second neural network that generates a feature vector.