receiving an update to a map used by an autonomous vehicle (AV), the map including raw map data overlaid with semantic map features;

determining an inconsistent portion of the update, the inconsistent portion of the update being inconsistent with a portion of the map;

determining a current limited visibility map defining a first geographical area within a predetermined distance from a body of the AV as the AV is navigating a route, the current limited visibility map being a subset of the map and including a subset of the semantic map features;

identifying a semantic map feature of the subset of semantic map features that extends outside of the first geographical area, the semantic map feature having a boundary area;

expanding the first geographical area defined in the current limited visibility map to include the boundary area of the identified semantic map feature;

determining the inconsistent portion of the update is within the boundary area of the expanded first geographical area defined in the current limited visibility map;

applying a next limited visibility map, defining a second geographical area, as the AV moves out of expanded first geographical area defined in the current limited visibility map;

determining that the inconsistent portion of the update is outside of the second geographical area defined in the next limited visibility map;

in response to determining that the inconsistent portion of the update is outside the second geographical area, applying the inconsistent portion of the update to the map.