receiving, by one or more processors of a vehicle having an autonomous driving mode, first sensor data simulating a virtual object generated by one or more testing computing devices such that the vehicle is able to perceive and respond to the virtual object as a first real object in an actual environment of the vehicle, and wherein the virtual object does not exist in the actual environment of the vehicle;

receiving, by the one or more processors, second sensor data for a second real object in an actual environment of the vehicle, wherein the second sensor data is generated by one or more sensors of the vehicle;

controlling, by the one or more processors, the vehicle in response to the virtual object and the second real object in the actual environment of the vehicle in order to cause the vehicle to take an action; and

storing, by the one or more testing computing devices, the action for later retrieval in order to determine whether an error other than a collision occurred at the vehicle.