US 11,836,973 B2
Automated direction of capturing in-room information for use in usability assessment of buildings
Lambert E. Wixson, Bellevue, WA (US); and Christopher Buehler, Seattle, WA (US)
Assigned to MFTB Holdco, Inc., Seattle, WA (US)
Filed by MFTB Holdco, Inc., Seattle, WA (US)
Filed on Feb. 25, 2021, as Appl. No. 17/185,816.
Prior Publication US 2022/0269885 A1, Aug. 25, 2022
Int. Cl. G06V 20/00 (2022.01); G06T 7/70 (2017.01); G06N 3/08 (2023.01); G06T 7/00 (2017.01); G06T 11/00 (2006.01); G06F 18/24 (2023.01); H04N 23/60 (2023.01); H04N 23/698 (2023.01)
CPC G06V 20/00 (2022.01) [G06F 18/24 (2023.01); G06N 3/08 (2013.01); G06T 7/0002 (2013.01); G06T 7/70 (2017.01); G06T 11/00 (2013.01); H04N 23/64 (2023.01); H04N 23/698 (2023.01); G06T 2207/10016 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30168 (2013.01)] 30 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
obtaining, by one or more computing systems, one or more panorama images captured in a room of a house and having visual data that in combination provides 360 degrees of horizontal visual data of the room;
analyzing, by the one or more computing systems, the visual data of the one or more panorama images to identify problems with quality of the one or more panorama images and with coverage of the house by the one or more panorama images;
initiating, by the one or more computing systems, recapture of the one or more panorama images to produce new copies of the one or more panorama images that have no problems with quality and no problems with coverage of the house;
determining, by the one or more computing systems after the analyzing and the initiating, and for each of multiple objects installed in the room, one or more target attributes of that object for which to capture additional data, including analyzing the visual data of the recaptured one or more panorama images to identify the multiple objects and to determine a type of each of the multiple objects and to determine locations of the multiple objects in the room and to determine based at least in part on the determined type of each of the multiple objects that the one or more target attributes of that object lack details in the visual data of the one or more panorama images sufficient to satisfy a defined threshold;
providing, by the one or more computing systems, instructions about capturing the additional data about the one or more target attributes of each of the multiple objects and information about the determined locations of the multiple objects, wherein the capturing of the additional data includes obtaining additional perspective images of one or more specified types of the multiple objects;
analyzing, by the one or more computing systems and via use of at least one trained neural network, additional visual data of the additional perspective images to, for each of the multiple objects, verify whether the additional data about the one or more target attributes of that object has been captured, including to determine for one of the additional perspective images that the one additional perspective image lacks any of the additional data by failing to include visual data that satisfies the defined threshold about any target attribute of any of the multiple objects;
providing, by the one or more computing systems, further instructions to recapture the one additional perspective image, and obtaining a new copy of the one additional perspective image that includes at least some of the additional data;
determining, by the one or more computing systems, an assessment of a current usability of the room for an indicated purpose, including assessing current contributions of the multiple objects to the current usability of the room based at least in part on information from the additional data about the one or more target attributes of each of the multiple objects, and including combining information about the assessed current contributions of the multiple objects; and
displaying, by the one or more computing systems, information about the determined assessment of the current usability of the room along with additional visual information about the room.