Automated monitoring of subdomains for fun and profit — Release of Sublert

Bug bounty has become a fast-growing industry with programs launching almost daily bringing along with it a fierce competition among hackers. It’s a sort of a monetized race which revolves around the first one to report a bug: first come, first served. Therefore, it’s essential to step up your game and try to stay ahead of the game. Learning and expanding your skill set is one way to do it but also improving your testing methodology is quite important, and polishing you reconnaissance for instance surely going to help a bunch.

In this regard, I have built a security and reconnaissance tool that I’ve been using for quite some time. It should allow you to monitor subdomains of specific organizations automatically and get notified each time something is found.

The tool was announced at OWASP Seasides during NullCon 2019 which took place in the beautiful city of Goa, India.

What’s Sublert?

Certificate Transparency (CT) is a new internet standard that came into the light to make TLS/SSL certificate issuance process open and publicly available. The aim is to allow domain owners to be aware of TLS/SSL certificates issued to the domain by public certificate authorities (CA).

Sublert is a security and reconnaissance tool that was written in Python to leverage certificate transparency for the sole purpose of monitoring new subdomains deployed by specific organizations and issued a TLS/SSL certificate. The tool is supposed to be scheduled to run periodically at fixed times, dates, or intervals (Ideally each day). New identified subdomains would be sent to Slack workspace with a notification push. Furthermore, the tool performs DNS resolution to determine working subdomains.

Sublert is mainly built for bug bounty hunters hacking on broad scopes to step up their reconnaissance game and beat the competition by gaining early access to new subdomains. However, Sublert could also be used internally as part of an AppSec program to monitor various subdomains deployed by the engineering team.

What’s up?

There are already open source tools, the likes of Cert Spotter or Facebook monitoring tool but these tools are mainly built for organizations to monitor phishing attempts by getting notified each time there is a certificate fraudulently or mistakenly being issued to a domain name. That means that you will be getting spammed by irrelevant results consisting of phishing domain names and uninteresting out of scope assets along with hundreds of emails on a daily basis in case you’re monitoring an organization with a considerable number of assets.

How to set up?

Requirements:

• Virtual Private Server (VPS) running on Unix. (I use digitalOcean)
• Python 2.x or 3.x.
• Free Slack workplace.

First of all, you will have to clone the tool from Github to your web server:

$ git clone https://github.com/yassineaboukir/sublert.git && cd sublert

This step is optional but highly recommended. It’s about setting up a python envrionment for Sublert.

$ sudo pip install virtualenv setuptools

If you’re using Python3:

$ sudo pip3 install virtualenv setuptools

Then:

$ virtualenv sublert
$ source sublert/bin/activate

Now, we can install the required modules as follows:

for Python 2:

$ sudo pip install -r requirements.txt

alternatively, Python 3:

$ sudo pip3 install -r requirements.txt 

Next step is to create a Slack workplace to where new subdomains will be sent to. Just browse to https://slack.com/ and create a free account.

Then, create two channels for the new subdomains or another of error logging.

Slack channels

Next, we need to generate Webhook URLs for both channels so we can leverage Slack API.

Browse to https://api.slack.com/apps and create a new application.

Browse to Incoming Webhooks and create two webhooks and link both to the previous channels:

Incoming webhooks

And link each webhook with the corresponding Slack channel.

Linking Webhook with Slack workplace

You will obtain a link in the following format:

https://hooks.slack.com/services/XXXXXXX/BF0XXXXME/XXXXXXXXXXXXX

Copy both two webhook links and edit config.py

You’re almost ready! Interesting part now is adding domains to monitor for new subdomains. For now, it’s preferable to add each domain separately. Let’s add PayPal for instance:

$ python sublert.py -u paypal.com

You can append `-q true` to the command to skip the question at the end.

Add PayPal to the monitored list.

You can proceed the same way with all the other domain names you would like to monitor. Once you’re finished adding the targets, the next step would be to configure Cron in order to schedule execution of Sublert periodically at fixed times.

Let’s make Sublert.py executable:

$ chmod u+x sublert.py

Now, we need to add a new Cron job to schedule execution of Sublert at given time. To do it, type:

$ Crontab -e

Add the following line at the end of the Cron file:

0 */12 * * * cd /root/sublert/ && /usr/bin/python sublert.py -r -l >> /root/sublert/sublert.log 2>&1

You will need to change /root/sublert/ to the exact directory where you have Sublert.

If you’re using python3 make sure to change /usr/bin/python to /usr/bin/python3

• 0 */12 * * * → instructs Cron to run Sublert every 12th hour. If you wish to change it to any fixed time you want, feel free to use https://crontab.guru/ to customize your timing.
• -r → instructs Sublert to perform DNS resolution. It’s optional but recommended.
• - l → instructs Sublert to log errors and push them to Slack channel. It’s optional but recommended.
• /var/log/sublert.log 2>&1 → Instructs Cron to save Sublert output to /var/log/sublert.log which you can inspect at any given time if you have any doubts the tool is running smoothly.

YOU ARE ALL SET, CONGRATULATIONS!

YOU GOAT IT!

You can make sure it’s properly configured and running smoothly by executing below command. You should receive a notification push on Slack:

$ python sublert.py -l -r

Sublert will be executed each 12th hour, and if by any chance, your monitored list of domains deploys a new subdomain, you’ll get notified on Slack like the following:

Sublert notifications on Slack

If Sublert is executed but nothing found, you should still get a new notification push. See:

Sublert couldn’t find any new subdomains

It’s recommended to install a mobile version of Slack on your mobile phone and disable Do Not Disturb mode to get notified whenever and wherever you’re.

You use Sublert at any given time either to add a new domain name to monitor or to remove something. Below are the various arguments supported:

Add yahoo.com to the monitored list.

$ python sublert.py -u yahoo.com

Delete yahoo.com from the list and it will no longer be monitored.

$ python sublert.py -d yahoo.com

Display all the domain names currently being monitored.

$ python sublert.py -a

Specify the number concurrent threads to use (Default: 20). You should add the argument to the Cron job as well if needed.

$ python sublert.py -t 30

Instruct Sublert to perform DNS resolution.

$ python sublert.py -r

Enable Slack-based error logging. You should add the argument to the Cron job as well if needed.

$ python sublert.py -l

Instructs Sublert to reset everything: will clean the monitored list and remove all locally stored files. Should ONLY be used if something is going wrong.

$ python sublert.py -m

You MUST only monitor bug bounty programs which has a wide and open scope to avoid unauthorized monitoring and violation of the program’s security policy.

Sublert isn’t perfect, right?

Sublert leverages certificate transparency which means it will only monitors subdomains for which SSL/TLS certificate is issued, however subdomains running on HTTP/80 will not be identified unfortunately. You can nevertheless use other open source tools (Massdns, Aquatone, Sublist3r, etc.).

Is there a roadmap?

YES! The tool is now open sourced to be used by the community but contributions are valuable and highly appreciated. I have a number of items that I will be working on to polish the tool, among of which are:

• Use of a relational database instead of text files for storage.
• Extracting as much information as possible including: title, status code, screenshot and checking for potential subdomain takeovers.
• Integrate Telegram too for notification pushes.

Feedback and issues?

If you have any feedback, anything that you want to see implemented or running into issues using Sublert, please feel free to file an issue on https://github.com/yassineaboukir/sublert/issues

Huge shout out to Ayoub Fathi for proofreading the present article. And thank you everyone for reading it. Feel free to follow me on Twitter: @yassineaboukir