receiving, at a computing system, sensor data captured by a camera-based sensor system associated with a vehicle in an environment;

processing, by the computing system, the received sensor data and thereby determining a sensor-based track for an agent that is identified within the sensor data, wherein the sensor-based track comprises a time-sequence of sensor-based positions of the agent;

generating, by the computing system, a first updated time-sequence of sensor-based positions of the agent by performing a first optimization operation on the time-sequence of sensor-based positions, wherein the first optimization operation includes processing at least two or more sensor-based positions of the time-sequence of sensor-based positions by beginning with a sensor-based position having a highest confidence in the time-sequence of sensor-based position observations and proceeding in a first direction;

after generating the first updated time-sequence of sensor-based positions, generating, by the computing system, a second updated time-sequence of sensor-based positions of the agent by performing a second optimization operation on the first updated time-sequence of sensor-based positions, wherein the second optimization operation includes processing at least two or more sensor-based positions of the first updated time-sequence of sensor-based positions in a second direction opposite the first direction;

deriving, by the computing system, a trajectory for the agent in the environment based on the second updated time-sequence of sensor-based positions of the agent; and

storing the derived trajectory, wherein the derived trajectory is thereafter utilized to perform a task that benefits from increased-accuracy trajectories.