US 12,094,035 B2
Methods and systems for localized smoke removal and color restoration of real-time video
Jayavardhana Rama Gubbi Lakshminarasimha, Bangalore (IN); Karthik Seemakurthy, Bangalore (IN); Vartika Sengar, Bangalore (IN); Aparna Kanakatte Gurumurthy, Bangalore (IN); Avik Ghose, Kolkata (IN); Balamuralidhar Purushothaman, Bangalore (IN); Murali Poduval, Mumbai (IN); Jayeeta Saha, Mumbai (IN); Srinivasan Jayaraman, Lake Oswego, OR (US); and Vivek Bangalore Sampathkumar, Bangalore (IN)
Assigned to TATA CONSULTANCY SERVICES LIMITED, Mumbai (IN)
Filed by Tata Consultancy Services Limited, Mumbai (IN)
Filed on Dec. 20, 2021, as Appl. No. 17/645,116.
Claims priority of application No. 202121016802 (IN), filed on Apr. 9, 2021.
Prior Publication US 2022/0366618 A1, Nov. 17, 2022
Int. Cl. G06T 11/00 (2006.01); G06T 7/90 (2017.01); G06V 10/56 (2022.01)
CPC G06T 11/001 (2013.01) [G06T 7/90 (2017.01); G06V 10/56 (2022.01); G06T 2207/10016 (2013.01)] 13 Claims
OG exemplary drawing
 
1. A processor-implemented method for localized smoke removal and color restoration of a real-time video, the method comprising the steps of:
receiving, via one or more hardware processors, the real-time video, wherein the real-time video comprises a plurality of video frames;
identifying, via the one or more hardware processors, one or more smoky video frames and one or more smoke-free video frames, for the real-time video, from the plurality of video frames, wherein each smoky video frame of the one or more smoky video frames comprises one or more precise smoke regions;
generating, via the one or more hardware processors, a smoky feature map for each smoky video frame of the one or more smoky video frames, using smoke relevant features of the corresponding smoky video frame, wherein the smoky feature map for each smoky video frame comprises features of the one or more precise smoke regions;
automatically estimating, via the one or more hardware processors, an air-light for each smoky video frame of the one or more smoky video frames, using the corresponding smoky feature map, wherein the air-light (Ac) for each smoky video frame (Ic(x)) is estimated according to a relation:

OG Complex Work Unit Math
wherein Cbmap(x) represent a binary map obtained from the smoky feature map of the corresponding smoky video frame, and x represents a pixel location having the smoke;
generating, via the one or more hardware processors, an intermediate de-smoked video frame for each smoky video frame of the one or more smoky video frames, based on the corresponding estimated air-light, using a de-smoking algorithm;
identifying, via the one or more hardware processors, a smoke-free reference video frame for each smoky video frame of the one or more smoky video frames, out of the one or more smoke-free video frames, wherein the smoke-free reference video frame for each smoky video frame is a smoke-free video frame out of the one or more smoke-free video frames, present just before the corresponding smoky video frame;
generating, via the one or more hardware processors, a de-smoked video frame for each smoky video frame of the one or more smoky video frames, by compensating color information obtained from the corresponding smoke-free reference video frame, locally in the one or more precise smoke regions of the corresponding intermediate de-smoked video frame; and
stitching, via the one or more hardware processors, the de-smoked video frame for each smoky video frame of the one or more smoky video frames, to obtain a de-smoked video in real-time.