collecting sensor data from one or more sensors mounted on an autonomous driving vehicle (ADV) while the ADV is moving within a region of interest (ROI) that includes a plurality of obstacles, the sensor data including obstacle information of the obstacles and vehicle data of the ADV, each of the vehicle data including driving commands issued and responses of the ADV at a specific point in time, wherein the driving commands issued include a throttle, brake, or steering command of the ADV, and the responses of the ADV include a speed, acceleration, deceleration, or direction of the ADV;

timestamping each of the vehicle data with a current time at which the vehicle data is captured to generate a plurality of timestamps that correspond to the vehicle data, the plurality of timestamps being mapped to the specific point in time of the driving commands issued and the responses of the ADV; and

aggregating the obstacle information, the vehicle data, and the corresponding timestamps into training data to align timing of the obstacle information with the plurality of timestamps of the throttle, the brake, or the steering command of the driving commands issued and the speed, the acceleration, the deceleration, or the direction of the responses of the ADV, wherein the training data is used to train a set of parameters that is subsequently utilized to predict at least in part future obstacle behaviors and vehicle movement of the ADV.