Detecting the flooding of Kherson using broadcasted news related to natural disasters
Finding hotspots of natural disasters using broadcasted news is a common intelligence requirement. We expect spatial and temporal patterns of the mentioned locations related to natural disasters. Not only the event location, but the related locations as well. News articles related to a broken dam as a cause of flooding near Kherson also mentioning Moscow creates a specific coincidence between these two entities without being spatially related to each other.
This is the second blog post showing some common geospatial intelligence practices related to natural disaster news. For more details, look at the previous post “Mapping the broadcasted news related to natural disasters”.
Geospatial Intelligence Portal Prototype
We need to extend our geospatial intelligence portal implementation. Publish and insert of geospatial features are required.
Before doing so, we have to authenticate ourselves using some of the available login options. The login method authenticates a dedicated user against ArcGIS Online. Calling the login method of a portal instance using a Jupyter notebook offers a secret input prompt. After successfully authenticating, the portal implementation manages a valid GIS instance. This instance offers ready-to-use spatial capabilities.
# author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
class GeospatialIntelligencePortal(object):
def login(self, username):
"""
Login into ArcGIS Online using a password input.
:param str username: Your ArcGIS Online username.
"""
self._gis = GIS(username=username)geoint_portal = GeospatialIntelligencePortal()
arcgis_user = 'arcgis-online-user-name'
geoint_portal.login(arcgis_user)Publish geospatial features
The features representing mentioned locations related to natural disasters need some validation and pre-processing. They contain a property named count, and this is a SQL restricted word. So that we need to rename the property to count_. The date property represents the ISO formatted string. The easiest way to create a valid date is using the datetime module. But, we have to set the time zone equal to UTC. This is a common mistake when dealing with dates. The default behavior of the strptime function uses the time zone info of the operating system. Being located in Germany with daylight saving time, we create an unwanted -2 hour shift. The hosted feature service would save every date, as the previous one having a time value equal to 10:00 PM.
# author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
def validate_features(featureset):
"""
Validates and returns the specified geospatial features, renames properties and converts field values.
:param FeatureSet featureset: The geospatial features to validate.
"""
# Validate fields
deleted_fields = []
for field in featureset.fields:
if 'count' == field['name']:
field['name'] = 'count_'
if 'date' == field['name']:
field['type'] = 'esriFieldTypeDate'
if 'esriFieldTypeString' == field['type']:
field['length'] = 1024
if 'esriFieldTypeOID' == field['type']:
deleted_fields.append(field)
for deleted_field in deleted_fields:
featureset.fields.remove(deleted_field)
for feature in featureset.features:
if 'count' in feature.attributes:
feature.attributes['count_'] = feature.attributes['count']
del feature.attributes['count']
if 'date' in feature.attributes:
feature.attributes['date'] = datetime.strptime(feature.attributes['date'], '%Y-%m-%d').replace(tzinfo=timezone.utc)
for deleted_field in deleted_fields:
del feature.attributes[deleted_field['name']]
return featuresetThe two publish methods create a feature service and add some tag to the underlying newly created portal item. So that we can search for the corresponding feature layer item using the title and specific tag.
# author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
class GeospatialIntelligencePortal(object):
def publish_hotspots(self, featureset, service_name='Hotspots', folder='Disasters'):
"""
Publish geospatial features as a feature service.
:param FeatureSet featureset: The geospatial features to publish.
:param str service_name: The name of the feature service.
:param str folder: The name of the existing output folder.
"""
featureset.sdf.spatial.to_featurelayer(service_name, folder=folder)
def publish_aggregations(self, featureset, service_name='Aggregations', folder='Disasters'):
"""
Publish geospatial features as a feature service.
:param FeatureSet featureset: The geospatial features to publish.
:param str service_name: The name of the feature service.
:param str folder: The name of the existing output folder.
"""
featureset.sdf.spatial.to_featurelayer(service_name, folder=folder)The hosted feature services show the schema defining the field types, alias names and any other restriction. If we want to enable time-awareness for this feature layer, we can specify the supported date field.
Define geospatial visualization
After publishing the geospatial features, we tweak the underlying feature service. We offer appropriated mapping capabilities. The classified renderer should use the mentions count of every geospatial feature. So that the color theme reaches from high to low.
We specify the renderer using the Visualization tab from our dedicated ArcGIS Online hosted feature service. For more details, see Apply styles.
Furthermore, we design a dedicated web map referencing these two feature layers and define our favorite basemap. We add some label class using the mention count and time-enable our web map. So that we offer day-by-day time stepping through the features. For more details, see Create maps.
Define a ready-to-use dashboard
On top of the time-enabled web map, we create a no-code based ArcGIS dashboard. The donut chart and bar chart widget filter the underlying features of broadcasted news related to natural disasters. We were able to identify a hotspot related to flooding starting from 6th June mentioning Kherson, Kiew and Moscow. The reason was the destruction of the Kakhovka Hydroelectric Power Plant dam. Which flooded the Kherson region in southern Ukraine. It caused a significant change of landscape and human settlements.
Accessing the geospatial ground truth
Our portal implementation should offer access to the published feature services. The search method returns the first feature layer of the matching portal item using the specified title, item type, and the “Natural Disasters” tag.
# author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
class GeospatialIntelligencePortal(object):
def search_featurelayer(self, service_name):
"""
Searches and returns a feature layer.
:param str service_name: The name of the existing feature service.
"""
search_result = self._gis.content.search(f'title: {service_name} AND tags:"Natural Disasters"', item_type='Feature Layer')
if len(search_result) < 1:
print('No feature layer exists!')
return None
if 1 < len(search_result):
print('More than one feature layer exists!')
return None
portal_item = search_result[0]
if len(portal_item.layers) < 1:
print('The layer list is empty!')
return None
if 1 < len(portal_item.layers):
print('The layer list contains more than one layer!')
return None
return portal_item.layers[0]Insert geospatial features
We need to insert new features representing mentioned locations daily. Therefore, our portal implementation should insert features by specifying the service name. The corresponding search method uses the service name as the item title and “Natural Disasters” as the item tag.
# author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
class GeospatialIntelligencePortal(object):
def insert_features(self, featureset, service_name):
"""
Inserts geospatial features into an existing feature service.
:param FeatureSet featureset: The geospatial features to insert.
:param str service_name: The name of the existing feature service.
"""
feature_layer = self.search_featurelayer(service_name)
if None is feature_layer:
return
return feature_layer.edit_features(adds=featureset.features)We want to create web map instances visualizing the hosted feature services. Our portal implementation provides a function creating a web map instance. This instance visualizes the specified hosted feature service using the specified service name.
# author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
class GeospatialIntelligencePortal(object):
def map_layer(self, service_name):
"""
Creates and returns a simple map view with the existing feature service.
:param str service_name: The name of the existing feature service.
"""
map_view = self._gis.map('Europe')
feature_layer = self.search_featurelayer(service_name=service_name)
if not None is feature_layer:
map_view.add_layer(feature_layer)
return map_viewMapping the geospatial ground truth
Executing the following snippet using a Jupyter notebook shows a map widget rendering our hotspots feature layer. We could also directly access the hosted web map as a portal item. So that the map widget would render the ready-to-use feature layer using the time-enabled web map.
geoint_portal.map_layer(service_name='Hotspots')
Querying the geospatial ground truth
We are interested in querying the hotspots of the 6th June 2023. The dashboard showed some unusual hotspot of mentioned locations where flooding occurred. So that we need a method querying the hosted feature layer using a date value. Hosted feature layer support date and time span filtering out-of-the box.
# author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
class GeospatialIntelligencePortal(object):
def query_features(self, service_name, seen_date=None):
"""
Queries and returns the geospatial features.
:param str service_name: The name of the existing feature service.
:param datetime seen_date: The date of interest.
"""
feature_layer = self.search_featurelayer(service_name)
if None is feature_layer:
return
if None is seen_date:
return feature_layer.query()
previous_day = seen_date - timedelta(days=1)
where_clause = f"date BETWEEN DATE '{previous_day.strftime('%Y-%m-%d')}' AND DATE '{seen_date.strftime('%Y-%m-%d')}'"
return feature_layer.query(where=where_clause)The following snippet returns the mentioned locations related to natural disasters of 6th June 2023. The returned feature set offers access to a spatially enabled data frame. This data frame offers well-known pandas utility functions, like sorting using the returned count.
hotspots_featureset = geoint_portal.query_features(service_name='Hotspots', seen_date=datetime(2023, 6, 6))
hotspots_featureset.sdf.sort_values(by='count_', ascending=False)
We can use a box plot identifying the theme outliers. Inspecting the box plot shows that broadcasted news mentioned flood and earthquake are most common. The data frame offers access using the matplot module. The box plot needs the numeric values and the unique values representing the series of interest.
hotspots_featureset.sdf.boxplot(column='count_', by='theme')
Summary
ArcGIS Online offers ready-to-use web and instant apps. We published our collected geospatial features and got insights into natural disasters. Our geospatial intelligence portal implementation using Python offers publishing, inserting and querying of features.
Combining mapping and geospatial intelligence services supports common intelligence workflows. The time-enabled web map offers insights into the flooding of Kherson, Ukraine.
Next time, we need to extend our portal implementation and offer automatic updating the feature services. We prefer a serverless approach for system-to-system communication.
