How to run subfinder with Axiom

Or running subdomain finder in a distributed manner.

This is a example how to use Axiom with the module (tool) subfinder.

For the propose of the following example, we will use Verizon’s subdomains, downloaded from here (collected by Project Chaos)

Let’s download the file, create a folder and put all the files together:

mkdir Verizon
cd Verizon
wget https://chaos-data.projectdiscovery.io/verizon_media.zip
unzip verizon_media.zip

Subfinder

This tool is used to find more subdomains starting with a list of domains. In the Verizon file we already have subdomains for every domain in the program. But let’s do the subdomain discovery using Axiom.

We need a list with only the root domains, so let’s run:

ls -I '*verizon*'   | sed 's/\.txt//g' > verizon_domains.txt

The content of the verizon_domains.txt is

└─$ cat verizon_domains.txt                                                                                                       
buildseries.com
builtbygirls.com
engadget.com
huffingtonpost.ca
huffingtonpost.com.au
huffingtonpost.com.mx
huffingtonpost.com
huffingtonpost.co.uk
huffingtonpost.co.za
huffingtonpost.de
huffingtonpost.es
huffingtonpost.fr
huffingtonpost.gr
huffingtonpost.in
huffingtonpost.it
huffingtonpost.jp
huffingtonpost.kr
huffpo.net
huffpostarabi.com
huffpostbrasil.com
huffpost.ca
huffpost.com.au
huffpost.com.mx
huffpost.com
huffpost.co.uk
huffpost.co.za
huffpost.de
huffpost.es
huffpost.fr
huffpost.gr
huffpost.in
huffpost.it
huffpost.jp
huffpost.kr
huffpostmaghreb.com
huffpost.net
makers.com
protrade.com
rivals.com
techcrunch.com
yahooapis.com
yahoo.com

Now, we need to create the fleet, I will create a fleet of 20 droplets:

axiom-fleet -i=20

It will take a while to create the instances we need.

Fleet created correctly!

Now, when we run the scan the input file will be split in 20 parts.

axiom-scan verizon_domains.txt -m subfinder -o subfinder_verizon.txt

Axion-scan starting with module subfinder

The results are impressive, 1 minute and 3 seconds to run subfinder using 42 domains. Great!

Let’s count the results:

└─$ wc -l subfinder_verizon.txt
128484 subfinder_verizon.txt

128,484 subdomains found, but the amount of subdomains we downloaded from Chaos Project was 427,496!

Well, the answer is quite simple, I didn’t use a config file with resolvers and API keys from different services. This is my config file (you need to get the API keys for every service you want to use):

Now we need to use this config file in all the droplets running subfinder. Here’s how to do it:

First, create the config folder on every droplet:

axiom-exec 'mkdir /home/op/.config/subfinder' 'dirac*'

And then upload the config file:

└─$ axiom-scp subfinder.config.yaml 'dirac*':/home/op/.config/subfinder/config.yaml

Notice that dirac is the prefix of my fleet, you need to change this value accordingly.

After running subfinder again, this is the result:

└─$ wc -l subfinder_verizon.txt                                                                                                 
430986 subfinder_verizon.txt

Now we have 430,986 results, 3,490 more than what we downloaded from Chaos’ website. Imagine if between those extra results we find a bug!

Note: Axiom-scan auto sorts and discard repeated values.

I hope you liked this example. Give me some claps if you find it useful!