Analyzing and Visualizing Earthquake Data Received with USGS API in Python Environment
The scars of the devastating earthquake are still in our lives, perhaps the wounds that will never go away have opened. Well, if we were to look at the events through a data-oriented lens, what would he/she see?
Hello everyone from a cloudy Sunday. I hope the weekend is going well for you. I love cloudy and rainy weather, I usually stay at home on weekends when the weather is like this, read my articles that I added to my list during the week and plan for the next week.
Well, when I look at my articles that I have shared, I am sharing my first article in 2023 with you. The year 2023 has not started very productively for me, and it cannot be said that it has started well for my country. The earthquake that occurred in Turkey, Hatay and the surrounding provinces in February affected us all. Thousands of people lost their lives and earthquake victims had to migrate to different cities from their cities where they lived for years.
In order to look at this devastating event we have experienced from a different perspective, I wanted to do some research on the tremors experienced and share the findings I have obtained with you.
Today, we will be using Web Scrapping to obtain the minimum 3 magnitude earthquakes that have occurred in Turkey and its surrounding points in the last 6 months by using an API belonging to the US Geological Survey (USGS).
Afterwards, we will visualize the data we have obtained on the map using the Heatmap. After the visualization, he conducts an exploratory data analysis study about the concussions experienced, but briefly summarizing the severity of the experienced situations.
Today, we will be using Web Scrapping to obtain the minimum 3 magnitude tremors that have occurred in Turkey and its surrounding points in the last 6 months by using an API belonging to the US Geological Survey (USGS).
Afterwards, we will visualize the data we have obtained on the map using the Heatmap. After the visualization, I will be conducting an exploratory data analysis study about the concussions experienced, but briefly summarizing the severity of the experienced situations.
Let’s dive in our work right away, without further ado:
- First, we import the Python Libraries that we will use in the study.
import requests
import pandas as pd
import folium
from folium.plugins import HeatMap
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.set_option('display.float_format', lambda x: '%.3f' % x)
pd.set_option('display.expand_frame_repr', False)
pd.set_option('display.width', 500)We define the data we need for the scrapping process:
start_time = 'now-180days'
min_magnitude = 3
latitude = 39.1458
longitude = 34.1614
max_radius_km = 1500- Since we will be scrapping the data from the last six months, set the “start-time” parameter as “now-180days”;
- Since we want to receive vibrations with a minimum magnitude of 3.0, set the “magnitude” parameter to 3;
- Since we want to receive the tremors that occur in Turkey and its surrounding locations, the latitude parameter “latitude” and the longitude parameter “longitude” parameters are 39.1458 and 34.1614, respectively;
- As the last parameter, we define the value of 1500 for the parameter named “min_radius_km” to get the vibrations that occur in an area of 1500 square kilometers.
Now, let’s enter a url for the scrapp process and place the parameters we have defined into this url using the fstring method.
With this code, we have sent an HTTP GET request using the requests library. We have received the data we need from the USGS Earthquake API, which is available at earthquake.usgs.gov, in JSON format as a result of the request.
url = requests.get(f'https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime={start_time}&minmagnitude={min_magnitude}&latitude={latitude}&longitude={longitude}&maxradiuskm={max_radius_km}')
dataset = url.json()The requests.get() method sends a GET request to a URL and receives a response. In this code, the URL parameters are created using the f-string formatting method.(e.g. Uraz Akgul | USGS API ile Alınan Deprem Verilerinin Python Kullanılarak Görselleştirilmesi)
The url.json() part seen in this block converts the HTTP Response, response in json format to Python data format. The data type obtained as a result of url.json() is a Python dictionary containing data in JSON format from the USGS Earthquake API.
When we run the response, if 200 comes as below, it means that the response was successful.
url
<Output:> <Response [200]>
We get the “features” in this json from data.
features = dataset['features']To make the dataset more readable, we create several empty lists. Afterwards, we fill these lists with the appropriate information with a for loop. After that, our data will be suitable for visualization and data manipulation processes.
places = []
mags = []
times = []
lats = []
lons = []
weights = []
for feature in features:
places.append(feature['properties']['place'])
mags.append(feature['properties']['mag'])
times.append(pd.to_datetime(feature['properties']['time'], unit='ms').strftime('%y/%m/%d %H:%M:%S'))
lats.append(feature['geometry']['coordinates'][1])
lons.append(feature['geometry']['coordinates'][0])
weights.append(feature['properties']['mag'])The for loop above adds the place, magnitude, time, latitude, longitude, and weight values of each feature to the places, mags, times, lats, lons, and weights lists, respectively.
In order to convert the data into a data frame format, we define a pd.DataFrame and send the lists we created as empty at the beginning into this data frame, each with a dataframe column. Now our data has been turned into a dataframe.
There is no longer any obstacle to visualize the data that we have converted to dataframe format. Let’s move on to the visualization part.
Data Visualization :
In this section, we will visualize the data we hold as data frame on a map image with HeatMap in the Folium library.
turkey_coord = [39, 35]
turkey_map_normal = folium.Map(location=turkey_coord, zoom_start=5.5)
HeatMap(data=df[['Latitude', 'Longitude', 'Weight']], radius=15).add_to(turkey_map_normal)
for index, row in df.iterrows():
folium.CircleMarker(
location=[row['Latitude'], row['Longitude']],
radius=row['Magnitude']/2,
color='red',
fill_color='red').add_to(turkey_map_normal)
turkey_map_normalA map was created in this code block and visualization was made using heatmap colors as a depiction tool according to the shaking magnitudes on the map. Let’s examine this code blog written in more detail:
- As a first step, we defined a list named “turkey_coord” to specify the center coordinates of the map. The coordinates in this list were used as the center coordinates of the map. Afterwards, the map was created using the folium library and a variable named “turkey_map_normal” was assigned. The zoom level was set to 5.5 to make it easier to view while creating the map.
- Then, with HeatMap, a mask was created on the map to describe the magnitudes of the tremors. While creating this mask, in the data set; “Latitude”, “Longitude” and “Weight” columns were used. Finally, this heat map, which was created by setting the Radius degree to 15, was applied on the map named “turkey_map_normal”.
- In the last step, a circle was drawn with the help of the CircleMarker function for each observation line in the data set with a for loop. As input to the CircleMarker function, the variable (Place) holding the location information of the earthquake and the variable (Magnitude) holding the tremor magnitude information were given. Filling was done with red in the created circle. The created circles were added to the turkey_map_normal map.
The map image we created as the last step is as seen below:
To make the map in bold format, add a simple parameter:
turkey_map_dark = folium.Map(location=turkey_coord, zoom_start=5.5, tiles='cartodbdark_matter')
We have done the data visualization part for now, now I want to analyze the data we have by manipulating it a little bit.
Data Manipulation :
First, there are empty values in the Place variable in the dataset. I want to fill these empty values with pd.Fillna.
df['Place'].fillna("NoInformation", inplace= True)Now we will be extracting the Country and City information from the “Place” feature, using some string methods. You could get faster results by using regex here, but I wanted to do it in a more challenging way because I am not good at regex(e.g Reguler Expression) yet.
for place in list(df['Place'].unique()):
if "," in place and "border" not in place:
if "Kahramanmaras" in place:
df.loc[(df['Place'] == place), 'Country'] = 'Turkey'
else:
df.loc[(df['Place'] == place), 'Country'] = df['Place'].apply(lambda x : x.split(',')[-1])
elif "," not in place and "border" not in place and " " in place and len(list(place.split(' ')))>1 and len(list(place.split(' ')))<3:
if "Iran" in place:
df.loc[(df['Place'] == place), 'Country'] = "Iran"
elif "Turkey" in place:
df.loc[(df['Place'] == place), 'Country'] = "Turkey"
elif "Greece" in place:
df.loc[(df['Place'] == place), 'Country'] = "Greece"
elif "," not in place and "border" not in place and " " in place :
if "Kahramanmaras" in place:
df.loc[(df['Place'] == place), 'Country'] = 'Turkey'
elif "region" in place:
df.loc[(df['Place'] == place), 'Country'] = df['Place'].apply(lambda x : x.split(' ')[0])
elif "Sea" in place:
df.loc[(df['Place'] == place), 'Country'] = place
elif "coast" in place:
df.loc[(df['Place'] == place), 'Country'] = df['Place'].apply(lambda x : x.split(' ')[-1])
else:
df.loc[(df['Place'] == place), 'Country'] = df['Place'].apply(lambda x : x.split(' ')[1])
elif "border" in place:
df.loc[(df['Place'] == place), 'Country'] = place
else:
df.loc[(df['Place'] == place), 'Country'] = place
df.loc[((df['Country'].isnull()) & (df['Place'].str.contains('Cyprus'))), 'Country' ] = 'Cyprus'
df.loc[((df['Country'].isnull()) & (df['Place'].str.contains('Sea'))), 'Country' ] = df['Place']
for place in list(df['Place'].unique()):
if "," in place:
df.loc[(df['Place'] == place), 'City'] = df['Place'].apply(lambda x: x.split(',')[0].split(' ')[-1])
df['City'].fillna('NO CITY INFORMATION' ,inplace = True)The Time variable we have is in the format “yyyy/mm/dd HH:MM:ss”. Let’s keep this Time variable by separating the date and time parts into two.
df['Date'] = df['Time'].apply(lambda x: x.split(" ")[0])
df['TimeZone'] = df['Time'].apply(lambda x: x.split(" ")[1])In the last case, our data set took a form as seen below. I think it has turned into a format that can be considered much more suitable for analysis than the previous one. Well, we are nearing the end…
I wanted to create a new variable in order to group the tremors according to months and years. Our variable named “Year/Month” keeps the year and month information of the earthquake.
df['Year\Month'] = df['Date'].apply(lambda x : str(x.split("/")[0]) + "/" + str(x.split("/")[1]))Exploratory Data Analysis :
In the exploratory data analysis part, we ask some questions to our dataset, and answer these questions using aggregation functions and groupby method.
1 . How many earthquakes have occurred in which country in the last 6 months?
df.groupby('Country').agg({"Weight": "count"}).sort_values(by = "Weight", ascending = False)[:10]
- When we look at the tremors that have occurred within the countries in the last six-month period, we can see that quite a number of tremors have occurred in Anatolia.
2. What is the mean and standart deviation of the Magnitudes of the earthquakes that will occur?
mean_std = df.groupby('Country').agg({"Magnitude": "mean",
"Weight" : "std"}).sort_values(by = "Magnitude", ascending = False)[:10]
mean_std.columns = ['Magnitude_mean', 'Magnitude_std']
mean_std
- It is an interesting result that although there are many earthquakes in Turkey, its average is 4.44. Although the 7.2 and 7.6 earthquakes, which are the big earthquakes, were very big tremors, the average magnitude of the earthquake in the main mass was seen as 4.44. As a separate result, the average of earthquakes in Iraq is seen as the highest. However, considering the number of tremors that occurred, very few tremors had occurred in Iraq. (e.g First question answer)
3. We know that earthquakes have occurred in the last 6 months. So, how many earthquakes were there in which month and what are their magnitudes?
df.groupby(['Year\Month','Country']).agg({"Magnitude": "count"}
).reset_index().sort_values(by = "Magnitude", ascending = False)[:15]
- There is no surprise in this result, in the last six months, Turkey is at the top in the number of earthquakes. It is seen that an earthquake over 3.0 is experienced almost every day. It’s a worrying situation…
4. I made the distribution according to the countries. How many earthquakes occurred in which city of which country?
df.groupby(['Year\Month','Country', 'City']).agg({"Magnitude": "count"}
).reset_index().sort_values(by = "Magnitude", ascending = False)[:15]
4.1 What is the average magnitude of these earthquakes?
table_ea = df[df['Country'] != "NoInformation"].groupby(['Year\Month','Country', 'City']).agg({"Magnitude": "count",
"Weight" : "mean"}).sort_values(by = "Magnitude", ascending = False)
table_ea.columns = ['Count', 'Magnitude']
table_ea
Well, we got the data with Web Scrapping, then we visualized the data we obtained on a map. We made the data we visualized ready for analysis processes with data manipulation methods. And finally, we tried to find answers to our questions by asking questions to the data set. We have come to the end of our study today. I hope it has been a useful reading session for you and opening the door to new ideas.
In the meantime, don’t forget to follow me on my social media accounts and check out my other articles. See you again :)
