Creating a Live World Weather Map using Shiny
Combining OpenWeatherMap API, Python/Shell scripts, Dropbox, Scheduled Jobs to create a live world weather map in Shiny that updates every hour
In the previous tutorial, we introduced sending requests to and receiving information from the OpenWeatherMap API and parsing that information in a way to create a dataset containing various data points related to the weather for the world’s 200 most populous cities. At the end of that tutorial, we compiled the weather conditions and other metrics of interest in a Pandas dataframe and downloaded it to our local computer as a CSV.
In this tutorial, we’ll use that data to plot the weather conditions on a world map, deploy this map online so it can be accessed by anyone, and automate the whole process such that it updates the weather regularly without you having to edit run the scripts manually.
To accomplish this, we will perform the following steps:
- Transforming OpenWeatherMap API calls into a Python script
- Visualizing the static data on a world map using Leaflet package in R
- Creating a Shiny app and deploying the world weather map online
- Updating the weather data regularly using cronjobs and Dropbox
1. OpenWeatherMap Python script (Quick recap)
In the previous post, the environment we used to call the OpenWeatherMap API and generate the CSV was a Jupyter notebook. This was due to its overall simplicity and our ability to iterate quickly with code chunks.
However, if we‘re looking to get the most recent weather data on a more regular basis, writing a Python script is the way to go. The script will essentially contain the same code as in the notebook except it won’t be chunked up into blocks and instead will just have one continuous flow.
The flow of the script will go as follows:
A) Top Cities Loading/Extraction
To send requests to the OpenWeatherMap API, we would need the name or names of the cities as a string or a list of strings as the input to the request. In the last post, we used web-scraping to get the names of the cities and stored them in a list that we loop through later.
For our exercise, however, it doesn’t seem necessary to get an updated list of cities every time we look to update the weather info. This is because the populations of the world’s 200 most populous are unlikely to change drastically in a short period of time. On top of that, web-scraping code can be a tricky affair (unusable, obsolete) if the code is not maintained.
Instead, what we’ll do is run it once manually, and store the list of cities and their populations as a CSV file in our data/working directory.
Then, as we go through our weather update script, we will check if there is a most populous cities file present in the directory.
If the file is present, then it will just assign local list variables with the values from the file and we’ll use those to send requests to the Weather API.
If for whatever reason the file is not present (ex: due to corruption of the file), then it will go through the most populous cities extraction web-scraping code, tally the city names and their populations, and then store them in the appropriate lists.
Going about it this way will both save us time when executing the script and give us extra safety from running obsolete scraping code that could damage the pipeline.
B) Weather Update
After this, the script will loop through the top cities, calling the OpenWeatherMap API and getting a weather report as a JSON for each city in the list.
From each JSON response, we‘ll extract the information we need and then store them in a dictionary.
Amongst other values, each city’s dictionary will contain:
- Main Weather condition (ex: Clear, Clouds, Rain…)
- Temperature in Celsius
- Wind Speed
- Latitude
- Longitude
- Time of Last Update
Then, each dictionary will be stored in an overall list that will be converted into a dataframe which we’ll output and save in our directory as a CSV.
2. (Static) World Weather Map in R using Leaflet
Now onto the good stuff, plotting the cities on a world map with each city displaying its weather conditions.
There are many tools/packages that allow us to plot geo-data on a map. For this project, we’ll use R and specifically the Leaflet R package that is a wrapper for the main Leaflet library in JavaScript. The Leaflet library makes generating interactive maps easy and allows us to display anything of interest on a map with about 10 lines of code.
The first step is to get the map working and displaying the information in the format that we want before deploying it as an application to Shiny.
For this, the environment we’ll work in the R-Studio IDE. Inside R-Studio, I prefer working in an R-Markdown because it is easy to separate different steps in the code into chunks (similar to a Jupyter Notebook). Code chunks will make it easier to quickly iterate and debug the map during development.
Quick intro to Leaflet
Adding the Base Maps
To create a Leaflet map, we start by calling leaflet() which creates the map widget. Beyond that, we add layers on top that include the type of map (addTiles) and markers on a map (addMarkers) to modify the widget. There are a variety of maps that one can choose as the base depending on the use-case, even combining more than one on top of each other if need be. For a full list of leaflet map providers, see here.
For our map, we’ll use the Esri World Terrain basemap because of its aesthetic appeal and simplicity. After all we just need a simple map without street-level information or other extra details.
library(dplyr)
library(leaflet)leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain)
Beyond this, we’ll need to set the view to be zoomed out and centred to capture all of the cities in one view. To do this, we’ll use setView using a zoom level of 2 and coordinates of 30, 30 (which appears to be the centre of the civilized world).
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2)Lastly, because leaflet allows unlimited dragging/scrolling, we want to make sure that the user viewing the map cannot drag the map outside of the main view. For this, we’ll use setMaxBounds and set them a little beyond the maximum coordinates of our cities in all four directions.
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70 )Put together, our map now looks like this:
Plotting the cities using addMarkers
Next, we’ll use the coordinate data to pinpoint the cities on the map.
First, we’ll load our data in the R session
city_info <- read.csv("data/weather.csv")
We’ve got the dataframe with all the necessary columns to create our map for 196 cities.
To plot the cities on the map, we’ll add them as Markers using addMarkers and specify the dataset that we’ll use in the data attribute and the respective latitude and longitude columns for the lat and lng attributes.
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude)
Weather Icons
Instead of displaying the weather conditions as a number (temperature) or text (conditions) on the Markers, we can make things a little more appealing visually by giving an icon representing each main weather condition, and have that display as the main button for each city.
To do this, there’s a useful integration within Leaflet called IconList that allows us to specify a path to a custom image and have that map to a particular value in the conditions column.
We’ll simplify the weather conditions into seven main conditions (clear, cloudy, overcast, rainy, snowy, thunderstorms, haze) and create an icon for each. For the icons, I found the following stock image on the internet that we’ll crop and use for each individual condition.
To create the iconList, it’s as simple as calling makeIcon for each condition in our conditions column with the attributes: iconUrl (location of the png file, here in the img folder), the iconWidth, and the iconHeight. And doing this for all seven of our weather conditions.
weatherIcons <- iconList(
Clear= makeIcon(iconUrl="img/Clear.png",iconWidth=20,iconHeight=20),
Rainy= makeIcon(iconUrl="img/Rain.png", iconWidth=20,iconHeight=20),
etc...
)On the data side, we’ll need make sure our mapping column contains only one of these seven conditions. The following is a code snippet that I used that is admittedly messy and far from perfect but does the job. Essentially what’s happening here is we’re creating a new column called ‘conditions’ that will be used to map the icons, and using many if-else statements to ensure that there are only the main seven conditions in the column.
city_info <- city_info %>%
mutate(conditions = factor(
ifelse(Main_Weather=="overcast clouds", "Overcast",
ifelse(Weather == "Drizzle", "Rain",
ifelse(Weather == "Mist", "Rain,
etc...
as.character(Weather))..))And we add them in addMarkers in the icon argument:
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions])
Now, our map has taken shape and looks pretty appealing. A couple of tweaks to the design left and we’ll start working on the Shiny app with this template.
Information Pop-up
The last cool feature in Leaflet that we’ll cover in this walkthrough is the popup. After specifying the latitude, longitude, and icon corresponding to the weather for each city, one thing that sticks out is that we’re not sure which city is which, especially in areas that have multiple large cities adjacent to one another (looking at you, China).
To solve this, we can add a popup on each icon such that the user can click on the icon and get presented with a display showing the temperature, population, and even the time the weather was last updated; all the data that we’ve already extracted and is available in the dataframe for each city.
The popup also comes in the form of an attribute to addMarkers and will contain the following information:
- City, Country
- Time of Last Update
- Population
- Detailed Weather Condition
- Temperature (in Celsius)
- Wind Speed (in km/h)
A collection of most of the details that a user might look for when checking the weather for a particular city.
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions],
popup = paste(
"<b>",city_info$City,", ", city_info$Country,"</b>","<br>",
"<b>Updated: </b>",city_info$DateTime,"<br>",
"<b>Population: </b>",city_info$Population,"<br>",
"<b>Weather: </b>",city_info$weather_main,"<br>",
"<b>Temperature: </b>",city_info$temp, " C","<br>",
"<b>Wind Speed: </b>",city_info$Wind_Speed, " km/h",
sep=""))
And now we’ve got a good-looking template to work with.
3. Shiny Application
Our map looks good in our local environment. The next step is turning it into a live Shiny application and deploying it.
Shiny is a platform built for R that allows developers to build interactive web applications that can be used to visualize data in various forms and makes it easily accessible. Shiny takes care of the hosting part of the application on their servers so that developers don’t have to worry too much about the ‘web development’ part of the process and can focus more on what goes into their visualizations.
We’ll sign up for the freemium model which allows up to 5 applications and a monthly limit on visitor viewing time, but nonetheless serves us well here where the project is a simple application.
Signing Up
First things first, go to shinyapps.io and Sign Up. In the shinyapps.io console, you’ll find a 3 point instruction manual on how to set up.
In summary, install the rsconnect package, copy the secret token issued to you by shinyapps.io, and in your R Console connect your Shiny account to your R-Studio environment using rsconnect::setAccountInfo() where you’ll specify your token.
Now, you’re ready to go and are just a few clicks from deploying the app.
Building the Shiny App
Turning our map into an R-Shiny app requires a few tweaks to our Leaflet code to fit the form that the Shiny engine expects. Namely, we have to separate the server logic from the UI (User Interface) logic and call the shinyApp function which combines both and renders the interactive map.
We begin by creating a folder in our directory called app.R. Inside app.R, we’ll load the necessary libraries that we’ll use followed by the data that we’ll need to process for the visualization.
# Loading packageslibrary(shiny)
library(leaflet)
library(dplyr)# Data Processingcity_info <- read.csv("data/weather.csv")city_info <- city_info %>%
mutate(conditions = factor(ifelse(
Main_Weather=="overcast clouds", "Overcast",
ifelse(Weather == "Drizzle", "Rain",
ifelse(Weather == "Mist", "Rain",
etc...
, as.character(Weather)))))
Server Logic
The server logic will contain the brains of our weather map, which is almost all of the logic that we used to render the map earlier in the RMarkdown. We’ll wrap our code in the renderLeaflet function and assign that to the $map variable in output. The output will be used later in the UI section to dictate what gets displayed.
server <- function(input, output) {
output$map <- renderLeaflet({
weatherIcons <- iconList(
Clear=makeIcon(iconUrl="img/Clear.png",iconWidth=20,iconHeight=20),
Clouds=makeIcon(iconUrl="img/Clouds.png",iconWidth=20,iconHeight=20)
etc...
)
leaflet(options = leafletOptions(minZoom = 2)) %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions],
popup = paste("<b>",city_info$City,", ", etc...))
})
}UI Logic
The UI logic will take care of how the entire page is laid out and how the server logic gets displayed. As a design, we’re going to want the map to take over the entire page. For starters, however, let’s keep it simple and just render the map to see how it holds up when we try to run the application.
ui <- fluidPage(
leafletOutput(outputId = "map", width = 1000, height = 1000)
)The function fluidPage creates a fluid layout that scales the components to fit the available browser width so we don’t have to worry about sizes or measurements. Its output is passed to the UI variable.
Inside the fluidPage function, we call leafletOutput, a function that specifically deals with leaflet maps that are created in the server chunk. Inside, we assign the outputId argument with map variable that we specified in the server logic.
This is how the two sections communicate with each other. In addition, we’ve arbitrarily specified the width and the height of our map to be 1000px.
shinyApp function
Finally, at the bottom of our application, we combine the server and UI logic and call the shinyApp function:
shinyApp(ui = ui, server = server)And press Run App in the top right corner of the R-Studio IDE.
Looks great. But one thing that remains unappealing is that the map doesn’t take up the entirety of the window.
To solve this, we can create a CSS style sheet file (called styles.css) and add it in our working directory. Inside the CSS file, we’ll have:
# styles.cssdiv.outer {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
overflow: hidden;
padding: 0;
}
Not a CSS expert so don’t take my word here for this. But essentially for the outer class that we’ll use in the UI, we’ve specified that the padding and all of the sides consume 0 pixels to fill up the page. There’s an abundance of CSS resources online that can help you further understand what’s going on here and potentially guide you to add more stylistic functionality to the app, but for this tutorial, we’ll stick to this.
Now we can modify the UI by specifying the class as outer and including the CSS file in our fluidPage call and using “100%” for the width and height in the leafletOutput. This should give us the full-page view.
ui <- fluidPage(
class = "outer",
tags$head(includeCSS("styles.css")),
leafletOutput(outputId = "map",width = "100%",height = "100%")
)Running the application again:
We’ve got a full-page map with all the information that we need with the code written in the Shiny format.
Deploying the Shiny App
We’re now ready to deploy the application to the Shiny servers.
Just as a reminder, since we’ve added a few files to our working directory that may have been difficult to keep track of, the working directory should contain the following files and folders:
WorldWeatherMap
|
|- app.R
|- styles.css
|- img/ -- Clear.png, Clouds.png ...
|- data/ -- weather.csvThis is important to note as when deploying the app, Shiny packages all these files and folders in a container of sorts and runs them every time the web-page is opened. Which means you’ll want everything ordered and organized in the folder and make sure its all-encompassing (i.e. containing all the resources your app needs including data and images).
Furthermore, the name of your directory (here: WorldWeatherMap) will become the name of the web app and be included in the domain name for the application, so make sure you name it accordingly.
To deploy the app, ensure your credentials are loaded (see: Signing Up section), and make sure you’re in the correct working directory in your R Console. Then type the following:
library(rsconnect)
rsconnect::deployApp()It is as simple as that. This might take a few minutes while it uploads the container and gets the instances running. But after that, you should be able to find your live web application at the following domain:
{username}.shinyapps.io/{name_of_app}/
If your application is live, congratulations on making it this far. If not, double check the steps you took and don’t hesitate to ask in the comments if anything was unclear.
4. Updating the Weather Regularly
We’re almost home. But our application is missing one crucial element that makes any weather map a weather map… recency. The application wouldn’t be of much use if it displayed weather conditions from years ago or even a day ago.
So for the final order of business, we’ll configure the pipeline to update our weather map on a regular basis.
As a reminder, our data is currently static. As in, it comes packaged with the entire application container that gets deployed. So to update it would require re-deploying the app with every update - a tedious affair.
What we’ll do instead is take advantage of a Dropbox-R interface called rdrop2 that allows us to feed the data that we use as input from a file stored on Dropbox instead of from a file saved locally. This will save us from having to re-deploy the application every time we want to update the weather data.
Once this is hooked up, we’ll automate the update of the file on Dropbox that will feed the map data using a scheduled job, and this will save us from having to update the file manually.
Two main steps to accomplish this:
- Convert the input source to read a data file from a dropbox folder instead of a file stored locally
- Run a cronjob to run the script that we developed in Part 1 that updates this data file on a regular basis.
Changing the Data Source to Dropbox
R-Studio allows a few ways to load data from outside the application environment that includes databases and cloud based storages. You can read about them here. One of these ways is Dropbox, a familiar easy to use cloud-based storage system.
Authenticating with rdrop2
To connect our R app with Dropbox, we need to install the rdrop2 package and set up a token or a key that allows our R application to access the files in dropbox.
The following is taken directly from the rdrop2 docs, which is a great resource on how to set up. Highly recommend you read through it to familiarize yourself with the process and check out some examples on how it is used.
# In the consoledrop_auth()# This will launch your browser and request access to your Dropbox account. You will be prompted to log in if you aren't already logged in.
# Once completed, close your browser window and return to R to complete authentication.
# The credentials are automatically cached (you can prevent this) for future use.
# If you wish to save the tokens, for local/remote use
token <- drop_auth()
saveRDS(token, file = "token.rds")
Connecting Dropbox with Shiny
Once authenticated, in app.R, we’ll add the following chunk before we load the data
library(rdrop2)# Data Download (from Dropbox)token <- drop_auth(rdstoken = 'token.rds')drop_download(path = '/Apps/weather_app/weather.csv',
local_path = 'data/weather.csv',
overwrite=TRUE)# Data Processingcity_info <- read.csv("data/weather.csv")
In summary, we use the drop_download function:
- Specify the path to get the data from (path)
- Specify the location to download the data to (local_path)
- Set the overwrite parameter to true so it updates the existing file every time it is called.
Now, instead of having little control over the data file that the application is reading from after it is deployed, we have the ability to download the data file from a dropbox folder every time the app starts.
If we are able to update the file in the dropbox folder, we are able to update the data in the application without re-deploying.
We’re only halfway done, because we’d still have to update the file in the dropbox folder manually.
Uploading an updated data file to Dropbox
To upload a file to Dropbox from a Python script on a local computer, we would need a separate Dropbox interface for Python and separate authentication.
Connecting Python with Dropbox
To get an authentication token, go to dropbox.com/developers. Log in and go to the App Console. Inside the console, click on ‘Create a New App’, fill in all the necessary information and specify a path in the App folder where you’ll upload the data file.
Once that’s done, go to ‘Generate Access Token’ and copy the token into the config.py file that you have in your directory (the one that also has the API key to OpenWeatherMap).
Upload.py Script
Download the dropbox Python package: pip install dropbox
Then, create a new script in your directory called upload.py. This script will simply take the data found in a particular directory (in our case data/weather.csv) and upload it to the Dropbox folder that you specified above.
# upload.pyimport dropbox
import configaccess_token = config.drop_keyfile_from = 'data/weather.csv'file_to = '/weather_csv/weather.csv'dbx = dropbox.Dropbox(access_token)with open(file_from, 'rb') as f:
dbx.files_upload(f.read(),file_to, mode=dropbox.files.WriteMode.overwrite)print("Upload complete")
The script is fairly self-explanatory. Do note however: the mode = dropbox.files.WriteMode.overwrite is very important otherwise you’ll get an error any time it tries to write a file to Dropbox and it finds another file there.
As a quick recap, there’s now:
updateWeather.py— Script that generates an up-to-date weather CSV and stores it in the data/ folderupload.py— Script that uploads weather.csv from data/ to a Dropbox folderrdrop2integration in app.R that downloads fresh data from Dropbox on application startup
Scheduling Upload Job using Cron
Since all the pieces are in place now, we need to automate the pipeline.
First, let’s simplify the two Python scripts by combining them into a shell script upload.sh that will just execute one after the other with one command.
#!/usr/bin/env bashpython3.6 ~/WorldWeatherMap/updateWeather.py
&&
python3.6 ~/WorldWeatherMap/csv_upload.py
Note: You might have to specify full paths for both Python and the two scripts, depending on the location of your Python path and working directory on your local PC.
Now that the upload process is simplified into one command, we want to automate the execution of this script on our computer. Enter cronjobs.
A cronjob is a job that executes a script or command every given period of time on a scheduled basis. We’ll use this tool to run the script our upload.sh script every hour.
An hour is a reasonable period of time to update the weather. A shorter period and we might violate the limit of API calls that we can make to the OpenWeatherMap API and a longer period and we risk showing obsolete weather data.
There are different ‘flavors’ of cron jobs that have different specs. The one we’ll use is cron is native to Unix environments. The one downside to cron is that it won’t run if the computer is not on at the time it is scheduled to run and won’t run the missed job again in the future.
There are other versions that run the backlog of missed jobs as soon as the computer turns back on. Try to keep that in mind when choosing the version of cron to meet your needs. There are also more advanced solutions using public servers to run the job but these are out of scope here.
To set up a job with cron, in the terminal:
crontab -eAn empty file should appear. Inside this file, we specify the time period that the script should run as well as the path to the script that we want to execute. Using your preferred text editor (vim, nano, etc…):
0 * * * * cd ~/WorldWeatherMap/ && ~/WorldWeatherMap/upload.sh > ~/WorldWeatherMap/logs/back.log 2>&1This specifies to run the job at the 0th minute of every hour. In addition, we’ve piped in a log file that can store any potential errors that occurred while attempting to run the script. This can be useful while debugging.
Checking if the job is working
Wait until the hour break, and check the Dropbox directory, if the data has been updated, then the cron job works. If not, check your back.log file in the logs directory where you can find the error that occurred.
Next check your Shiny app, you should be able to see the Updated At: in your popup reflecting the most recent weather report.
Conclusion
And voila, you’ve got a regularly updating weather map deployed online.
Every once in a while, check the deployed application to make sure things are running smoothly. Often times there might be a small detail that was overlooked or a weather condition that we forgot to specify in our data processing section that are causing the map to malfunction and we would need to add or fix something to correct it.
This was a good project to work on the various parts that go into creating a useful visualization online, regularly updating the flow of data, and sharing it with others. Shiny is an extremely handy tool and can be used for a variety of use-cases, your imagination being the only limit.
Check out the link to the live weather map here. To see the full code that went into this, check it out on the Github page for the project. There may be a few things that are different there but all of the core parts are the same.
Finally, if there was anything that was unclear or could use further clarification from my end, please don’t hesitate to comment below.
Happy coding!
