Barking at daemons

Monitoring a Shiny Server with Monit

My team uses a Shiny Server to share some data analysis and dashboards with other teams.

The server has automatic processes to get data from external sources, and any team member can publish new code to production at any time.

Most of the time things go right, but I want to make sure everything keeps working, regardless of the changes that happen. To monitor the server, we decided to use monit.

Installing Monit

The installation depends on the operating system you’re using. For CentOS, here are the steps to install:

sudo yum install monit
sudo systemctl enable monit
sudo systemctl start monit

You can check if everything is working by running curl http://localhost:2812, although the result is not very interesting... A 401 error.

We can configure monit to allow remote access to the web interface, so that we can see the results in a remote browser.

Edit /etc/monitrc, and search for the embedded HTTP interface configuration (just look for 2812). Replace what's there by:

set httpd port 2812
    allow admin:monit

And reload monit: sudo monit reload. If you now launch a browser a navigate to http://<yourhost>/:2812, you should see something like this:

Monit, up and running

Note: You should later get back to this configuration and make it more secure!

Monitoring the Shiny process

To monitor the shiny process, the 1st thing you need is to launch Shiny with a configuration that uses a PID file.

Run systemctl status shiny-server and this will give you the name of the service. In my machine it's /etc/systemd/system/shiny-server.service

You can edit this file and change the ExecLaunch section to add a PID file (--pidfile=/var/run/shiny-server.pid)

(...)
[Service]
Type=simple
ExecStart=/usr/bin/env bash -c 'exec /opt/shiny-server/bin/shiny-server --pidfile=/var/run/shiny-server.pid >> /var/log/shiny-server.log 2>&1'
(...)

Run sudo systemctl daemon-reload and sudo systemctl restart shiny-server. You should now have a PID file.

Next, we need to configure monit. Create a file called /etc/monit.d/shinymonitor and add the following content:

check process shiny-server with pidfile /var/run/shiny-server.pid
start program "/usr/bin/systemctl start shiny-server.service"
stop program "/usr/bin/systemctl stop shiny-server.service"
if failed port 80 protocol http then restart

You can check that everything is ok by running sudo monit -t, and then restart with monit reload.

You are now monitoring your Shiny Server! Go back to the web console and check it out. Click the link, and you can see additional information about the process.

Monitoring Applications

In addition to monitoring if the service is running, you can make sure specific applications are running. Just to be sure a move to production or a data fetch hasn’t caused any problems.

You can do this by creating a file called /etc/monit.d/shinyappsmonitor with the following content:

check host myhost-myimportantapp address myhost.com
        if failed port 80 protocol http request /myapp/ then alert
check host myhost-otherimportantapp address myhost.com
        if failed port 80 protocol http request /otherapp/ then alert
check host myhost-reports address myhost.com
        if failed port 80 protocol http request /report/ then alert
        if failed port 80 protocol http request /anotherreport/ then alert

The name after check host is what you'll see in the report. You could put all applications under the same host, but I find it easier to look at the report like this.

What’s next?

If you have Rmd files running on your Shiny Server, these take a bit more work to monitor. Take a look at this article to learn how to do it.

Other things I’d like to do with monit:

1. Send alerts via slack. You can only configure monit to send alerts through email
2. Monitor cron jobs. We have some data processing jobs running through cron, would like to know if everything succeeded.

I’ll write about these things as soon as I build them! :)