US 11,987,264 B2
Method and system for recognizing activities in surrounding environment for controlling navigation of autonomous vehicle
Chetan Singh Thakur, Bangalore (IN); Anirban Chakraborty, Bengaluru (IN); Sathyaprakash Narayanan, Chennai (IN); and Bibrat Ranjan Pradhan, Bhubaneswar (IN)
Assigned to Wipro Limited, Bangalore (IN); and Indian Institute of Science, Bangalore (IN)
Filed by Wipro Limited, Bangalore (IN); and Indian Institute of Science, Bangalore (IN)
Filed on Jul. 16, 2021, as Appl. No. 17/377,761.
Claims priority of application No. 202141014742 (IN), filed on Mar. 31, 2021.
Prior Publication US 2022/0324477 A1, Oct. 13, 2022
Int. Cl. G06T 7/246 (2017.01); B60W 60/00 (2020.01); G06N 3/045 (2023.01); G06V 20/40 (2022.01); G06V 20/56 (2022.01)
CPC B60W 60/001 (2020.02) [G06N 3/045 (2023.01); G06T 7/246 (2017.01); G06V 20/40 (2022.01); G06V 20/56 (2022.01); G06T 2207/20084 (2013.01); G06T 2207/30252 (2013.01)] 15 Claims
OG exemplary drawing
 
1. A method of recognising activities in surrounding environment for controlling navigation of an autonomous vehicle, the method comprising:
receiving, by an activity recognition system of the autonomous vehicle, a first data feed from a neuromorphic event-based camera and a second data feed from a frame-based RGB video camera, configured in the autonomous vehicle, wherein the first data feed comprises high-speed temporal information encoding motion associated with a change in the surrounding environment at each spatial location, and the second data feed comprises spatio-temporal data providing scene-level contextual information associated with the surrounding environment;
performing, by the activity recognition system, adaptive sampling of the second data feed with respect to a foreground activity rate based on an amount of foreground motion encoded in the first data feed;
recognizing, by the activity recognition system, activities associated with at least one object in the surrounding environment by identifying a correlation between the first data feed and the second data feed by using a two-stream neural network model, wherein a first neural network of the two-stream neural network model analyzes motion associated with the at least one object, as encoded in the first data feed, and a second neural network of the two-stream neural network model analyzes the scene-level contextual information based on the adaptive-sampled data; and
controlling, by the activity recognition system, the navigation of the autonomous vehicle based on the determined activities.