Asynchronous Loading of Leaflet Layer Groups

This is another post in a series of working with Leaflet and maps. The previous posts can be found here, here, and here.

Photo by Curioso Photography on Unsplash

If you have quite a few layer groups in your Leaflet map, and you are annoyed by the super long loading time of the map that makes you feel you’d better go get some coffee, this blog might be able to mitigate the issue for you by improving the map loading time.

Completely solve it? No, no. How can you expect someone good in R who happens to be creating a Shiny dashboard, most likely to share insights with colleagues or collaborators rather than inviting every Internet user to come and visit, to have a background in front-end engineering and know clearly what is being transferred between the server and the client, and when and how, based on which the optimization could be eventually done? I know there are such people, but I also know you who opened this blog is not likely to be one of them.

Fortunately or unfortunately, I’m not one of them either, so rest assured. The trick you will see has nothing to do with this unfamiliar world. Even if you don’t have any knowledge in website development, there is still something you can do that will more or less improve the performance.

Like the previous blogs in the series, we will use the same data to create this example. The Shiny application can be found here, and code here.

In this example application, we will create a tabBox with 4 tabs:

• initialization: set to be the default tab, it is used to isolate the initialization phase of the application from the rendering of Leaflet maps that we would like to time.
• the complete map with multiple layers
• the map with only one layer
• the complete map with multiple layers using the lazy loading trick described later in this blog

TL;DR

1. pass only the first layer group to your output$map object
2. in observeEvent, use leafletProxy() to modify the map in output$map by adding the remaining layer groups

Regular Leaflet Maps

First of all, a baseline map would be one with multiple layers that you regularly do, and in the shiny app we will have it in tab “Default Loading”.

This data set comes with two variables, ALAND (area of land) and AWATER (area of water) of each county. We can add each of them as a separate layer group in Leaflet.

Two layers might not show a big difference as compared to one layer. Let’s create more variables.

shapes <- rgdal::readOGR("cb_2018_us_county_5m","cb_2018_us_county_5m")
df_geo <- read_excel('all-geocodes-v2019.xlsx',skip=4)  %>% # the table starts from row 5
  filter(`Summary Level`=='040') %>%
  select(`State Code (FIPS)`, `Area Name (including legal/statistical area description)`)
colnames(df_geo) <- c('STATEFP','STATENAME')
shapes@data <- shapes@data %>% 
  left_join(df_geo) %>%
  mutate(ALAND = as.numeric(as.character(ALAND)),
         AWATER = as.numeric(as.character(AWATER)),
         ALAND_RANK = round(percent_rank(ALAND),3),
         AWATER_RANK = round(percent_rank(AWATER),3),
         WATER_LAND_RATIO = AWATER / ALAND,
         WATER_LAND_DIFF = AWATER - ALAND,
         NAME = as.character(NAME))

Using the two variables, we generate ALAND_RANK (percentile of ALAND), AWATER_RANK (percentile of AWATER), WATER_LAND_RATIO (water-to-land ratio), and WATER_LAND_DIFF (difference between area of water and area of land). This gives 6 variables that we can use for map layers.

To simplify the map, let’s use the same color palette OrRd for all our layers (though sometimes it might not look good or even correct). The colors in OrRd ranges from deep orange to white then to deep red. That being said, the only difference in the code of each layer (basically what we specify in the addPolygons function) is the layer group name passed to parameter group and the variable name passed to the palette in parameter fillColor.

Repeating copy-and-paste and making modifications will always work. It’s just that the lazy part of me is screaming whenever I think about doing so. Alright, alright. Maybe we should do a loop.

layer_input <- c('Area of Land'='ALAND',
                 'Area of Water'='AWATER',
                 'Area of Land, Percentile'='ALAND_RANK',
                 'Area of Water, Percentile'='AWATER_RANK',
                 'Water-to-Land Ratio'='WATER_LAND_RATIO',
                 'Difference between Water and Land'='WATER_LAND_DIFF')
group_display <- 'Area of Land'
output$map_default <- renderLeaflet({
  map <- shapes %>%
    leaflet() %>%
    addTiles('http://{s}.tile.openstreetmap.de/tiles/osmde/{z}/{x}/{y}.png')
  
  for (group_name in names(layer_input)){
    variable_name <- layer_input[group_name]
    map <- map %>%
      addPolygons(data = shapes,
                  group = group_name,
                  weight=1, opacity = 1.0,color = 'white',
                  fillOpacity = 0.9, smoothFactor = 0.5,
                  fillColor = ~colorBin('OrRd',get(variable_name))(get(variable_name)),
                  label = lapply(shapes$content,HTML))
  }
  
  map %>%
    addLayersControl(
      position = "bottomright",
      baseGroups = names(layer_input),
      options = layersControlOptions(collapsed = TRUE)) %>%
    hideGroup(names(layer_input)) %>%
    showGroup(group_display)
})

We create a named vector where the name is the layer group name and the value is the corresponding variable name. A named vector allows us to use the name to retrieve the value, with which once we know which layer group we are looking at, we can easily find out the variable name to use.

When creating the map in output (the id of theleafletOutput is set to be map_default), we build the base map first and save it to a variable map. Then we loop over our pre-configured layer group names, add polygon layers using the pair of layer group name and variable name. You may be used to passing the variable name as is (e.g., ALAND) directly to the color palette rather than the variable name as characters (e.g., “ALAND”). To pass a dynamic string as variable to the color palette, get() helps to do the conversion.

Loading this 6-layer map on my laptop takes about 30 seconds.

the default loading approach that takes ~30 seconds (it really is playing :p)

What if we have a map of only the first layer?

output$map_single_layer <- renderLeaflet({
  shapes %>%
    leaflet() %>%
    addTiles('http://{s}.tile.openstreetmap.de/tiles/osmde/{z}/{x}/{y}.png') %>%
    addPolygons(data = shapes,
                group = 'Area of Land',
                weight=1, opacity = 1.0,color = 'white',
                fillOpacity = 0.9, smoothFactor = 0.5,
                fillColor = ~colorBin('OrRd',ALAND)(ALAND),
                label = lapply(shapes$content,HTML)) %>%
    addLayersControl(
      position = "bottomright",
      baseGroups = names(layer_input),
      options = layersControlOptions(collapsed = TRUE)) %>%
    hideGroup(names(layer_input)) %>%
    showGroup(group_display)
})

This leafletOutput has an id of map_single_layer. To make a fair comparison, we add the layer controller as well even though we know that there is only one layer available.

loading a single layer

This single-layer map takes 5 seconds. That’s a huge difference, isn’t it?

You may think, “but this is just 1 layer and in the end we need 6, so this doesn’t matter”.

Well, yes and no. The total loading time may be hard to get shortened, but does a user have to wait till all the components are loaded to start the exploration?

Asynchronous Loading

If you play online RPG games, you might have noticed that when you try a new game for the very first time, some of them finish downloading very quickly and let you start in the beginner village. While you are playing, in the background it downloads a much bigger package of data for the rest of the world in the game as well as for the features that won’t be unlocked until you reach the requirement to leave the beginner village.

Although the total downloading time could be several hours, you are able to start the game in, for example, half an hour, once the data needed for the beginner village is ready. The waiting time you need to spend is actually only that half an hour.

This is the idea of the asynchronous loading trick here.

The reason why we experience 30 seconds for the Leaflet map to render is that a synchronous loading pattern is adopted, so that the user won’t be able to see the map until it is fully ready.

We can apply the same idea of asynchronous loading to the Leaflet map. In this case, the beginner village is the default layer to show on the map, and the rest of the world is the other layers that the user won’t see until they click to switch.

Ideally, the user only needs to wait for 5 seconds to see the map, and when he explores the default layer, the other layers are loaded behind the scene one by one for another 25 seconds in total.

Without much understanding about the underlying interactions between the UI components and the Shiny server, we can implement this idea by isolating the remaining 5 layers from R shiny’s output. We will have:

• the map with a single layer in the output
• an observeEvent that adds missing layers

The key here is being able to know when this observeEvent needs to be executed. Like the action button in Shiny, we set a reactive value rvs$to_load and add 1 every time when the map output is re-calculated (although in this example the map will be calculated only once), which the observeEvent monitors.

rvs <- reactiveValues(to_load=0,map=NULL)
output$map_async <- renderLeaflet({
  rvs$to_load <- isolate(rvs$to_load) + 1 # change the value to trigger observeEvent
  
  rvs$map <- shapes %>%
    leaflet() %>%
    addTiles('http://{s}.tile.openstreetmap.de/tiles/osmde/{z}/{x}/{y}.png') %>%
    addPolygons(data = shapes,
                group = 'Area of Land',
                weight=1, opacity = 1.0,color = 'white',
                fillOpacity = 0.9, smoothFactor = 0.5,
                fillColor = ~colorBin('OrRd',ALAND)(ALAND),
                label = lapply(shapes$content,HTML)) %>%
    addLayersControl(
      position = "bottomright",
      baseGroups = names(layer_input),
      options = layersControlOptions(collapsed = TRUE)) %>%
    hideGroup(names(layer_input)) %>%
    showGroup(group_display)
  
  rvs$map
})

In observeEvent, we modify the existing map by calling leafletProxy() followed by addPolygons() for each of the rest of the layers groups, using a for loop like what we did for the complete Leaflet map.

observeEvent(rvs$to_load,{
  req(rvs$map) # if it's not null or false
  
  group_names_to_load <- names(layer_input)
  group_names_to_load <- group_names_to_load[group_names_to_load != group_display] # take out the default layer already added
  
  for (group_name in group_names_to_load){
    variable_name <- layer_input[group_name]
    leafletProxy("map_async") %>%
      addPolygons(data = shapes,
                  group = group_name,
                  weight=1, opacity = 1.0,color = 'white',
                  fillOpacity = 0.9, smoothFactor = 0.5,
                  fillColor = ~colorBin('OrRd',get(variable_name))(get(variable_name)),
                  label = lapply(shapes$content,HTML))
    }
})

On my laptop, this asynchronous version takes about 10 seconds to render. Not as ideal as 5 seconds, but it’s already 2x faster than the original 30-second loading time!

The only trouble is, if the user is too eager to view a layer that hasn’t be loaded yet, he will not be able to see any polygon on it until after a short time, hopefully not long enough for him to lose the hope:

loading layers asynchronously

In addition to polygon layers as shown above, the same trick can be easily applied to marker layers (or other types of layer), which could take a long time to load when a good amount of location points are added.

Like I mentioned at the very beginning, it is not an actual acceleration of the loading time, but a deception to the perception by showing the Leaflet map when it is partially ready. Although your problem might not be completely resolved, the user experience after this trick is applied could be good enough for you to call it a day.

Other blogs in this series:

1. Center Diverging Colors on Leaflet Map
2. Capture and Leverage Mouse Locations and Clicks on Leaflet Map
3. Layer-Specific Legend in Leaflet