How to Use Google Maps With Ecto and Phoenix LiveView
Today I’ve been asked to create a dashboard to monitor the download of different generic apps. These were the specifics:
Create a web dashboard that can be used internally to show a real time overview of our mobile apps download.
The system should provide data about the downloads on our mobile application in real-time.
Each download is an object that has this properties:● longitude
● latitude
● app_id (an internal identifier for the app)
● downloaded_atThe data should be shown in an interactive map that shows the downloads of the applications. The user should be able to also have some statistics about app download, like: downloads by country, downloads by time of the day etc..
In the past months I wanted to try the new Phoenix LiveView release and this 👆 seems the perfect occasion to do that.
Technologies 💻
Before going into the solution of the problem let’s see the technologies used.
Phoenix LiveView
Stated in the documentation:
LiveView provides rich, real-time user experiences with server-rendered HTML. The LiveView programming model is declarative: instead of saying “once event X happens, change Y on the page”, events in LiveView are regular messages which may cause changes to its state. Once the state changes, LiveView will re-render the relevant parts of its HTML template and push it to the browser.
It seems the perfect fit for the problem, doesn’t it ?
Google Maps for javascript
From the documentation:
The Maps JavaScript API lets you customise maps with your own content and imagery for display on web pages and mobile devices.
Ecto
The documentation:
The database wrapper and query generator for Elixir. Ecto provides a standardised API and a set of abstractions for talking to all the different kinds of databases, so that Elixir developers can query whatever database they’re using by employing similar constructs.
We will use a Postgres instance running in the background to store our data
The solution 💡
I decided to divide this article into phases but if you want to jump directly into the finished code you can go here.
Initial setup
First of all, let’s install Elixir (1.11.0), Erlang (23.1.1.) and Phoenix (you can find the guide here) and create the automatic generate project with mix phx.new liveMapApp --live . This command will generate the necessary code to have a running elixir server.
Done that, we want to generate a new page called Dashboard . It should be linked with the different application’s downloads. To do so we also create an app component and its related database table. The new Phoenix version can do all these actions in one command:
mix phx.gen.live Dashboard App apps longitude:decimal latitude:deciaml app_id download_at:utc_date_timeAfter running it, we should just copy the new router generated into the router page as it is stated in the command’s output:
Add the live routes to your browser scope in lib/liveMapApp_web/router.ex:
live “/apps”, AppLive.Index, :index
live “/apps/new”, AppLive.Index, :new
live “/apps/:id/edit”, AppLive.Index, :editlive “/apps/:id”, AppLive.Show, :show
live “/apps/:id/show/edit”, AppLive.Show, :edit
Our basic application is setup, we can test it by creating the database and applying migration. This is done by:
mix ecto.create
mix ecto.migrate
mix phx.serverActually, if we forget to run the database creation and the migration, Phoenix remembers it for us and shows a nice error message with the option to launch it though a button.
Google Maps integration 🗺
At the route page of our application we now have a list of downloaded applications stored into a table and some buttons to add/show/delete them. Let’s integrate with Google Maps.
The once below is the suggested from the documentation:
We can change part of this code to be generated dynamically by our liveView:
<%= for download <- @downloaded_apps do %>
new google.maps.Marker({
position:
{
lat: <%= download.latitude %>,
lng: <%= download.longitude %>
},
map,
title: "<%= download.app_id %>",
});
<% end %>Moreover, in order to see the map in the page we need to tweak Phoenix a little bit by adding the following piece to our map container: phx-update="ignore". This command will allow us to avoid the map deletion from Phoenix. Be also sure to have an id in the container as the map will keep refreshing every milliseconds. To understand better this phenomena you can read how LiveView lifecycle works here.
We can now add a new app using the button and we can see the changes in the table but not on the map. That is because our Google Maps object is not yet linked with any update actions from LiveView. We will take care of it in a minute.
Dashboards 📊
Let’s first create some dashboards. We need at least to show the downloads by time of the day and by country. To do so we need to know from which country the download comes from. We could use GoogleAPI to know the country by looking at the coordinates and then group them by country. This would imply to do a GoogleAPI call for every row in the database every time we refresh the page. This is not scalable at all 🙉!! I decided to go with a cheaper and scalable option which is to store the country when the new download is registered. This implies only one API call for each download in the database.
To do so we have to add various things:
- A new migration file to add the new column. We can do so by using
mix ecto.gen.migration add_countryand inserting this code into the generated file:
defmodule LiveMapApp.Repo.Migrations.AddCountry do
use Ecto.Migration def change do
alter table(:apps) do
add :country, :string, default: "Unknown"
end
end
end
- Change the Ecto schema to expect the new field into
lib/liveMapApp/dashboard/app.ex - run
mix ecto.migrateto apply the new migration - add an endpoint to insert new
downloaded_app. We can do that by adding a new controller into the application which will be in charge of validate the parameters, make the API call to gather the country, store the information into the db
Now we can setup our simple dashboards by adding some HTML and SCSS:
@for $i from 1 through 100 {
.percentage-#{$i}:after {
$value: ($i * 1%);
width: $value;
}
}<dl>
<dt>Download by country</dt>
<%= for {countryName, count} <-
Enum.reduce(@downloaded_apps,
%{},
fn x, acc ->
Map.update(acc, x.country, 1, &(&1 + 1))
end)
do %>
<dd class="percentage
percentage-<%= trunc((count/length(@downloaded_apps))*100) %>">
<span class="text">
<%= countryName %>:
<%= Float.round((count/length(@downloaded_apps))*100,2) %>
</span>
</dd>
<% end %>
</dl>
This will generate 100 css classes that can be used by the HTML code. In the HTML we use Elixir to group the downloaded app list by country and calculate the percentage.
We can now add many histogram dashboards we want by slightly changing the HTML code.
Note: If you want to see the whole SCSS used have a look at the repository.
Transformation into a Real-Time application 🔌
We now have a working application which needs to be reloaded every time we have a change. This means every time we insert a new row into the database.
LiveView takes care of most of the real time part, we just need to broadcast the changes into our GenServer wrapper which is called PubSub . To do so we just need to add these lines of code where we want to generate the update request. In our case it will be in the module Dashboard which handles the insertion into the database:
defp broadcast({:error, _reason}=error, _event), do: error
defp broadcast({:ok, app}, event) do
Phoenix.PubSub.broadcast(LiveMapApp.PubSub, "apps",
{event, app})
{:ok, app}
endAs you can see above, we need to do pattern matching on the parameters as we will call this function just after the insert.
In our index page we then need to subscribe to the channel created and handle the event sent.
if connected?(socket), do: Dashboard.subscribe()Before subscribing we need to check if the socket on the frontend exists and if it is connected to the backend. In the end we need to handle the broadcast event in order to update the view:
def handle_info({:download_added, app}, socket) do
{:no_replay, update(socket, :downloaded_apps,
fn apps -> [ apps]end)}Now we are able to hit the app endpoint using PostMan or Curl and we can see immediately the changes in the page, the dashboard and the table change but Google Maps doesn’t. That is because we blocked Phoenix to update the component in order to be able to use it.
To solve this we can use a new feature in LiveView called Hook which works in a pretty similar way as the React hook. Some more details here . We need to add this code into our app.js
let Hooks = {}Hooks.MapMarkerHandler = {
mounted() {this.handleEvent("new_marker", ({ marker }) => {var markerPosition = { lat: parseFloat(marker.latitude), lng: parseFloat(marker.longitude) }const mapMarker = new google.maps.Marker({
position: markerPosition,
animation: google.maps.Animation.DROP,
title: marker.app_id
})
mapMarker.setMap(window.map)
});
}
}let csrfToken = document.querySelector("meta[name='csrf-token']").getAttribute("content")
let liveSocket = new LiveSocket("/live", Socket, {hooks: Hooks, params: {_csrf_token: csrfToken}})
This will create an eventHandler on the event coming from the server. Let’s link our container by specifying phx-hook="MapMarkerHandler" into it and firing a new event from the backend every time we need to add a new marker. To do so we need to change our broadcast function to react to the creation event in a different way, leaving the general one so that a new event will be still handled:
defp broadcast({:error, _reason}=error, _event), do: error
defp broadcast({:ok, app}, :download_added = event) do
Phoenix.PubSub.broadcast(LiveMapApp.PubSub, "apps",
{event, app})
Phoenix.PubSub.broadcast(LiveMapApp.PubSub, "apps",
{:new_marker, app})
{:ok, app}
end
defp broadcast({:ok, app}, event) do
Phoenix.PubSub.broadcast(LiveMapApp.PubSub, "apps",
{event, app})
{:ok, app}
endThis is needed as we cannot push more than one event to the client in a single handle_info method in our index.ex file. We will then need also
def handle_info({:new_marker, app}, socket) do
{:noreply, push_event(socket, "new_marker",
%{marker: %{
latitude: app.latitude,
longitude: app.longitude, app_id: app.app_id}
}
)}
endTo handle the new event in the page.
We now have a function application respecting all the initial requirements. Let’s move to the most important part of it: TESTS
Let’s test 🛠
I usually start from the tests and then write the code to have them working. In this case, as it was my first time using Phoenix LiveView, I decided to go with a code first methodology. That is because I didn’t know what the app would have looked like at the end. Now the app is functional and we can actually see which tests are already there and which ones we want to add.
If you don’t write unit tests first the best practise is to write them along the way because while writing the code you know better in which way your code could fail. If you do it at the end, especially on big projects you will most probably forget one method that contains a tiny typo that will wake you up in the middle of the night on a Saturday😅.
Phoenix and LiveView help you in the test creation by adding some standard tests that you could start from. In particular for this application we want to test:
- The controller, we only have one, with only one function but we have to test the data validation that the user is sending to it. An important thing to remember here is that we are calling GoogleApi in the controller code. Of course we want to test this but we also don’t want waste API requests every time we run unit tests, it could be expensive. To do so we can use Mox i.e
a library for defining concurret mocks in Elixirfrom the documentation - Database, this is almost completely covered by the automatic generated tests from Phoenix. We can just add some more invalid tests
- Main page, Phoenix creates some automatic tests on the basic feature. We deleted most of them and completely change the page so they do not make sense anymore. In this case we tested the existence of specific elements as the Dashboard labels , the Goole Maps frame with the markers, the title of the page. Unfortunately we cannot tests the real time feature of the app which can be done in the service level tests.
- Service level tests, described here as:
Service Tests consist of the scaffolding, dependencies, and actual tests necessary to isolate and assert a service can function.
Service level tests are really useful to test the whole application without worrying about the dependencies. In the context of the article I didn’t wrote them but is certainly something to add as future improvement.
Note: to run the tests we just need to run MIX_ENV=test mix test
Last but not least: Docker 🐳
Even though I’ve done all this article on my local machine I prefer it to be dockerized as it’s easier to run from other environments and also on kubernetes 😉. Let’s do it.
In order to run this into docker we need to create some folders and files:
- docker-compose.yml
version: '2.2'
services:
live-map-app:
environment:
- UID
build:
context: .
dockerfile: Dockerfile.local
args:
hex_repo_key: ${HEX_REPO_KEY}
uid: ${UID}
command: bash scripts/dev.sh
env_file: ./env/dev.env
ports:
- 4005:4000
volumes:
- .:/home/app/service/
- elixir-artifacts:/home/app/elixir-artifacts
depends_on:
live-map-app-db:
condition: service_healthy
networks:
- shared
- defaultlive-map-app-db:
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: "postgres"
PSQL_TRUST_LOCALNET: 'true'
ENCODING: UTF8
image: postgres:9.6
healthcheck:
test: ["CMD", "pg_isready", "-d", "postgres", "-U", "postgres"]
interval: 10s
timeout: 3s
retries: 10networks:
shared:
external: truevolumes:
elixir-artifacts: {}
This is the starting point from where we create a database and a container for our application. Then we can add the Dockerfile.local which contains:
# Elixir 1.10 with Erlang/OTP 22
FROM elixir@sha256:ba981350b63eb016427d12d90dad643eea8e2bfed37e0f2e4f2bce5aa5303eaeLABEL maintainer="francesco.zanoli"ARG run_deps="inotify-tools"ARG mix_env="dev"
ARG http_port="4000"
ARG app_path="/home/app/service"
ARG uid="1000"USER rootENV TERM xtermENV HOME /home/app
ENV APP_PATH ${app_path}
ENV HTTP_PORT ${http_port}
ENV MIX_ENV ${mix_env}
ENV ERL_AFLAGS="-kernel shell_history enabled"ENV REFRESHED_AT 2020-10-08RUN apt-get -q update && apt-get -qy --no-install-recommends install ${run_deps}
RUN curl -sL https://deb.nodesource.com/setup_14.x
RUN apt-get install -y nodejs npmRUN adduser --disabled-password --gecos '' app --uid ${uid}RUN mkdir -p /home/app/elixir-artifacts
RUN chown -R app:app /home/app/elixir-artifactsUSER app:appRUN /usr/local/bin/mix local.hex --force && \
/usr/local/bin/mix local.rebar --force && \
/usr/local/bin/mix hex.infoRUN echo "PS1=\"\[$(tput setaf 3)$(tput bold)[\]\\u@\\h$:\\w]$ \"" >> /home/app/.bashrcCOPY --chown=app:app . ${APP_PATH}WORKDIR ${APP_PATH}EXPOSE ${HTTP_PORT}CMD ["sh", "script/start.sh"]
The only thing to note here it’s that we pointed to a specific docker image, using the SHA256. This will avoid strange situations in which a future deploy will change the Elixir or OTP version and causing the crash of the application or even worst, some strange bugs.
Now we can create the .env file, we will use it in the next section. Finally we have to add the `dev.sh` file which contains all the code to be run once the container is up:
#!/bin/bashset -excd assets
npm install
cd ..mix deps.get
mix compile
mix do ecto.create, ecto.migrateecho "run docker exec -it ${HOSTNAME} sh in another console to jump into the container!"tail -f /dev/null
#mix phx.server
I usually don’t start directly the service because this allows to run the test first, as I don’t usually run them directly in my local machine. To start it up we just need to follow the instructions to enter the container and launch mix phx.server
Nipsticks 💅
We have a functional, working, tested, dockerized application, now let’s make it pretty 🎉. We have different aspects to cover:
The project generated from CLI doesn’t take care of the environment variable. However there are some information that we don’t want to store in the code which should be kept secret. In this project the most important ones are the GoogleApi Key and the Phoenix encryption key. It’s also best practise to include the database information as this could be a sensitive data in a production scenario. Elixir has a folder to manage the configuration, config , so we just need to change the files for the different environments. For the database we just need to change LiveMapApp.Repo config in dev and test
config :live_map_app, LiveMapApp.Repo,
username: System.get_env("POSTGRES_USER"),
password: System.get_env("POSTGRES_PASSWORD"),
database: System.get_env("POSTGRES_DB"),
hostname: System.get_env("POSTGRES_HOSTNAME")For the keys we need to update the main configuration as they won’t change depending on the environment
config :live_map_app,
ecto_repos: [LiveMapApp.Repo],
api_token: System.get_env("API_TOKEN")# Configures the endpoint
config :live_map_app, LiveMapAppWeb.Endpoint,
url: [host: "localhost"],
secret_key_base: System.get_env("PHOENIX_SECRET")
To access to them we could then do: Application.get_env(:live_map_app, :api_token)
Note: This usage will not work for the production environment as the configuration we changed are loaded at compile time. In order to postpone this loading for run time we need to use an Elixir feature called mix release which is not discussed in this article.
After the environment variables we want to remove all the warning ⚠️ from the test and from the application, cleaning them up will help us for future development and to discover problems.
If you have followed this article until here you would have notice that the months in the dashboard as well as the days are shown by using numbers. To change this we can add a private function into our controller as follow:
defp get_day_name(day) do
case day do
1 -> "Monday"
2 -> "Tuesday"
3 -> "Wednesday"
4 -> "Thursday"
5 -> "Friday"
6 -> "Saturday"
7 -> "Sunday"
_ -> "Unknown"
end
enddefp get_month_name(month) do
case month do
1 -> "Jan"
2 -> "Feb"
3 -> "Mar"
4 -> "Apr"
5 -> "May"
6 -> "Jun"
7 -> "Jul"
8 -> "Aug"
9 -> "Sep"
10 -> "Oct"
11 -> "Nov"
12 -> "Dec"
_ -> "Unknown"
end
end
We will just need to update the code to call this function before passing the values to the dashboard component.
If you look at the app you will now notice that we are missing a dashboard from the requirements. I left this at the end in order to show how LiveView allows you to go fast in future improvements. As we have our component we just need to add a new line into the view in order to have the dashboard we are looking for:
<%= live_component(@socket,
Dashboard,
id: "ByDayTime",
title: "Dashboard by time of the day",
list: Enum.reduce(@downloaded_apps, %{},
fn x, acc ->
Map.update(acc,
get_day_time(x.download_at), 1,
&(&1 + 1))
end),
total: length(@downloaded_apps),
percentage: true)%>The reduce function we are using this time calls a new function from the controller to get in which interval the download is in.
So the app is working but it’s really ugly, let’s add some filters in the dashboard in order to hide some of them. To do so we can use Css and a radio button using this HTML:
<input type="radio" id="filter" name="categories" value="filter">
<label class="filters" for="filter">Country</label>an this CSS:
[value="filter"]:checked ~ .dashboard:not([data-category*="filter"]) {
display: none;
}Then we just need to add data-category="filter" and class="dashboard"to the HTML component we want to show when the filter is clicked. In particular the css code will hide all the components which are part of the same parents and have the dashboard class.
⚠️ ⚠️ Phoenix will not care about the state of the radio button during the update. We need to add phx-update="ignore" to all the filters that we want to.
In this phase I also always double check the documentation in the code and remove unnecessary comments, as the one generated by Phoenix. This time I decided to leave all of them in order for you to understand better what is what and what was automatically generated.
Lastly we need to update the README.md to describe how to use, test, run the project.
Thoughts , Future improvement 📚
I didn’t really discuss the choice of the technologies used in this article, the main reasons for it are the speed real-time performance and the pattern matching properties of Elixir which seemed a good match with problem’s requirements and moreover I really wanted to test it our 😃
It was really fun and fast starting up a functioning application with LiveView even if it was the first time I ever used it.
However the application could use some improvements which were skipped for timely matters:
- More unit tests, especially in the View part and hooks, using this
- More explicit error message in the endpoint
- A better UI
- Service level tests
- Heatmap layer, using this
You can find the full code here. I hope you liked this article 👋 🌊
