I’m designing a network for MRI automatic labeling (more on this in a future post) with a relatively large training set. Each epoch takes about 45 minutes to run (on my brand new Nvidia 1080 TI card) and convergence takes about ten epochs. I wanted to be notified of the progress as each epoch finishes and get the precision metric for that epoch so that if the precision number does not converge as I expect it, I can change some hyper-parameters or model structure. I did not want to sit in front of the computer while training progresses.
This post is about a super-trivial way of achieving that goal. We will create an IFTTT applet which sends a push notification to your phone when it receives a trigger in the form of an http GET request.
IFTTT Set Up
Login to your IFTTT account (or signup for it, if you haven’t already — it’s free at ifttt.com). Then click on “My Applets” and start a new applet by tapping on, you guessed it, “New Applet” button as shown in the image below:
Search for the “webhook” service (this used to be called the “Maker channel”). IMO this is the most useful service in IFTTT. This service triggers an action (the “then” part of IFTTT) when it receives a specific http GET request.
Click on the “Webhook” to go to the next page and configure this service.
Click on “+ this” to get to the next screen shown below where we assign a name to our event. We’ll call it “epoch_ended”
Click on “Create Trigger” to get to the screen shown below:
Click on “+ that” and in the search-box enter notification. This is the service that we will use to generate a push notification to our phone.
Tap on notification. We can generate a “simple” or “rich” notification. The rich notification allows you to specify links to images and other stuff. The simple notificatin suffices for what we are trying to do, but select “rich notification” just to see what kind of stuff you can include in the notification.
In this screen, we can specify the content of the push notification. In the “Message” box, tap on “Add Ingredient” and select “value1”. In the python code that we will include in our program, we can pass in a dictionary as part of the http request with three keys: ‘value1’, ‘value2’, ‘value3’. We can use these keys to pass parameters to IFTTT and display them in the notification as we desire. Here, I only make use of ‘value1’ because I can do the formatting in python and set ‘value1’ to whatever string that I want. As you will see in this example, I will include two parameters (precision and recall) in ‘value1’.
Click on “create action” and then on “Finish”:
One more step in IFTTT: Before we move to our python code, we need our secret key for our webhook service. Tap on the “webhook” as shown in the screen-grab below:
Here you see your webhook secret key. Copy it.
The Python Code
This python function takes message and sends a properly formatted http request to IFTTT which triggers the push notification from IFTTT to your phone. Don’t forget to replace the ‘<your_ifttt_key_here>’ with your IFTTT secret key that you copied in the previous step.
Now you are ready to call the notification function at the end of each epoch with metric which you’d like to monitor:
In addition to sending push notifications, you can create IFTTT applets that send emails or SMS. I found push notification to be more useful. All you need to do now is to download the IFTTT app for your iOS/Android device and login to your account on the device.
Here are examples of the notifications on my phone and Apple Watch:
You can now start your training and go to the gym or go biking or do something fun. Time passes faster when you are not behind the computer watching progress of your machine learning training :)