US 11,900,490 B1
Mobile app, with augmented reality, for checking ordinance compliance for new and existing building structures
Shashi Yadav, Edison, NJ (US); and Rishi Bhatnagar, East Brunswick, NJ (US)
Assigned to Morgan Stanley Services Group Inc., New York, NY (US)
Filed by Morgan Stanley Services Group Inc., New York, NY (US)
Filed on Sep. 9, 2022, as Appl. No. 17/941,561.
Int. Cl. G06Q 50/16 (2012.01); G06T 7/60 (2017.01); G06V 20/10 (2022.01)
CPC G06Q 50/163 (2013.01) [G06T 7/60 (2013.01); G06V 20/176 (2022.01); G06T 2207/30184 (2013.01)] 21 Claims
OG exemplary drawing
 
1. A system comprising:
a back-end computer system that comprises a computer database, wherein the computer database stores building ordinance data for a government agency and stores permit data for real properties in the government agency; and
a mobile device for communicating with the back-end computer system via an electronic data network, wherein the mobile device comprises:
a display;
a camera system that comprises one or more cameras;
a GPS receiver;
one or more inertial sensors;
a processor; and
a memory in communication with the processor, wherein the memory stores a mobile app, the mobile app comprising software that when executed by the processor causes the processor to:
determine a physical address for a subject property based on a location and orientation of the mobile device, wherein the location and orientation of the mobile device is determined based on, at least in part, inputs from the GPS receiver and inputs from the one or more inertial sensors; and
after determining the physical address for the subject property, in real time as a user of the mobile device moves around the subject property:
detect, using computer vision, a first object at the subject property in a field of view of the camera system;
determine, based on visual media captured by the camera system, a compliance status of the first object, wherein the compliance status comprises an assessment of whether the first object is compliant with ordinances and permitting requirements of the government agency, and wherein the assessment comprises, when a permit for the first object is required, a permit compliance assessment that comprises (a) whether a permit for the first object at the subject property exists in the computer database and (b) upon a determination that the permit for the first object at the subject property exists in the computer database, a comparison of (i) dimensions and position of the first object at the subject property, based on an image of the first object captured by the mobile device, to (ii) dimensions and position for the first object according to the permit; and
display, on the display of the mobile device, augmented reality, wherein the augmented reality comprises real time visual media captured by the camera system of the mobile device and a first virtual indicator for the first object within the field of view of the camera system, wherein the first virtual indicator is overlaid on the first object in the display, and wherein the first virtual indicator indicates the compliance status of the first object.