Building a High Performance, Asynchronous, Port Scanner with… Ruby?
Asynchronous, high performance, and Ruby — you might not think that makes a lot of sense. However, the async Ruby future is here; and it’s fast.
To see just how fast, let’s try building an asynchronous port scanner; meaning a port scanner that can look for many open ports at a time. To qualify as “fast”, at least for the sake of this blog post, means it should be able to scan hundreds of ports a second.
Just how do we do that?
🔧 Single-Threaded Design
The first option you might jump to in order to increase performance in languages like Ruby or Python is usually multi-threading. This isn’t the absolute worst choice but quickly becomes a bottle-neck in and of itself due to a global lock these languages enforce on their runtime ensuring only one thread runs at a time, even if you have many.
Effectively, this means your “multi-threaded” programs are really more like a confusing single-threaded program — sometimes actually performing worse than their truly single-threaded variants. If this is the first time you’ve heard this, it can be kind’of tricky to really understand.
The main advantage of using multi-threading is providing a means to deal with real world situations like network or disk I/O (input/output) operational latency. So, programmers have come up with abstractions to deal with this. With multi-threading APIs in Ruby or Python, we have an abstraction as a programmer to splice up our work into time segments. On I/O blocking operations, the language will decide that another thread can run, usually providing a speed boost for things like varying network connection speeds. This doesn’t come cheap though.
Threads will often end up “fighting” each-other for control of the runtime causing CPU context-switching which eats up your performance. This can be a really annoying thing to wrap your head around and provide a rather erratic runtime behavior that is a pain to debug or code complex interactions.
To avoid the headache of orchestrating multiple threads, we will be building a single-threaded port scanner. But, like in a multi-threaded program, we still want to be able to weave together tasks asynchronously.
If a program can be broken up into a series of tasks that can be executed independently, it can be considered asynchronous. It is a property of how a program can be executed, not what it does. — Samuel Williams
⚡️ Modern Async Support for Ruby
For sometime now, Samuel Williams and friends have been working away on bringing modern, asynchronous and timeout-capable I/O for Ruby to the masses under the Socketry project.
Side Note: You can read more about it here.
Using async-await
will provide us with a clean API to describe asynchronous tasks. To ensure our network connections are asynchronous, we will use async-io
. Using async
Gems within the Socketry project provides a plethora of options to build non-blocking I/O programs with great performance. The underlying design of async
is based on a Reactor Pattern for concurrency backed by Fibers.
Side Note: Another project you might want to check out is Falcon, a high performance asynchronous multi-process web-server built on
async
libraries.
👩🏽💻 Implementing the Port Scanner with Async
To get started, let’s write the first part of the script to require
the async
libraries we want to use.
require "async/await"
require "async/io"
require "async/semaphore"# more code ...
Now, let’s create a PortScanner
class, including the libraries within it to provide clean async
semantics and create our #initialize
method:
# previous code ...class PortScanner
include Async::Await
include Async::IO def initialize(host: "127.0.0.1", ports: (1..1024))
@host = host
@ports = ports
@semaphore = Async::Semaphore.new(`ulimit -n`.to_i)
end # more code...
end
By default, if no host:
argument is given when creating a new instance of our scanner, our localhost will be the target to scan. If not given any Range/Array/Enumerable for ports:
, then ports 1...1024
will be used, very similar to nmap
.
The @semaphore
will be used to lock down the number of scans going on at a time to whatever the max number of available file descriptors are on the system. For every open connection, there’s an open file on the system, and we want to limit this to what ulimit
tells us that is, at least specifically on unix-like operating systems including macOS or Linux.
Note:
ulimit
is a builtin shell command used to manage various resource restrictions. The-n
flag will reveal the maximum number of open files allowed;#to_i
on the output from that command in the code will turn it into a Integer. We cache this result and use it to set the maximum number of open connections we’ll try at a time, since there’s no reason to push past that threshold.
Now, let’s implement a method which can scan a single port in our PortScanner
class:
⚠️ This is the code to scan a single port. Moreover, this is will be the most complicated part of the code because of all the things we need to juggle, which I’ll explain in a bit more detail in just a moment.
# inside of PortScanner, under initialize ...def scan_port(port, timeout: 0.5)
timeout(timeout) do
Async::IO::Endpoint.tcp(@host, port).connect do |peer|
peer.close
puts "#{port} open"
end
end
rescue Errno::ECONNREFUSED, Async::TimeoutError
puts "#{port} closed"
rescue Errno::EMFILE
sleep timeout
retry
end# more code...
To break down what’s happening, the method #scan_port
will attempt to connect to the target on the given port
— I know, that’s the obvious part.
If any error occurs — and that’s going to happen— we can rescue
that method and do something based on the cause of the error, right inside of the method itself.
🤔 Why do we do this again?
During a port scan, especially for very large ranges, say thousands, or tens of thousands, our scanner may try to open up a connection, but — due the the limitations of our operating system — we may not be able to. We could be browsing Twitter or GitHub while our scanner runs super large batches after-all. Remember ulimit
and the @semaphore
from earlier inside of #initialize
?
🤹🏻 Three Essential Errors to Juggle:
- 🛑
Errno::ECONNREFUSED
is an error that will occur when a port is not open to connect to. This is is a pretty typical response for most ports, and this response that is enforced with mostREJECT
firewall rules. - ⏱
Async::TimeoutError
is an error produced bytimeout(timeout)
when an operation didn’t complete within the allottedtimeout:
value ( default of0.5
). This may happen for various reasons including aDROP
firewall rule where the target doesn’t explicitly tell you a port isn’t open from the scanner’ perspective. We really don’t want to wait around forever to get nothing. - 📑
Errno::EMFILE
is an error that will occur when we try to connect to a port, but we have way too many connections (files, really) already open and need to schedule the execution to happen again, but just a little later, continuously delaying its execution until it’s able to be processed.
Keeping track of the appropriate errors helps prevent missing open ports on larger scan ranges and allows us to handle things according to the error. In certain situations, multi-threaded port scanners can end up missing open ports simply because the scanner is counting any and all errors as a “closed port” when it probably should be retrying the ports that error’d out because of the operating system’s limitations. This goes for basically every port scanning implementation I’ve created and/or used.
In order to scan all the ports the PortScanner
class is initialized with, let’s implement a #start
method which will iterate over each port and calls the #scan_port
method we just created. But, this time we’ll use the async
keyword in-front of the method definition.
# inside of PortScanner, under scan_port ...async def start(timeout: 0.5)
@ports.map do |port|
@semaphore.async do
scan_port(port, timeout: timeout)
end
end.collect(&:result)
end
Within the #start
method, each port in the range that the PortScanner
is initialized with will be scanned, running asynchronously — and resource bound to the operating system’s limitations with the @semaphore
. The #collect(&:result)
near the very end of the method will allow us to collect the result of each port scan attempt.
⚙️ Using the Scanner
To use the port scanning API we just created could look something just like the following:
# previous code ...PortScanner.new(host: "127.0.0.1", ports: (1..1024)).start
There aren’t any open ports to test with? Using nc
, it’s pretty easy to create some open ports. The following will setup a TCP listener on your localhost on port 1024
:
$ nc -v -l -p 1024
Now, let’s run our port scanner, using grep
to filter for the open ports:
$ ruby async_port_scanner.rb | grep "open"
1024 open
💥Bam! Just like that, we have a port scanner! How fast does it go? Using time
on Unix-like operating systems can help us get an idea:
$ time ruby async_port_scanner.rb | grep "open"
1024 openreal 0m2.345s
user 0m0.531s
sys 0m0.130s
That’s roughly 2–3 seconds to scan all those ports (1024
of them ) on a single thread! What if we wanted to scan all the ports (1…65535
), how long would that take?
real 1m12.693s
user 1m5.933s
sys 0m5.157s
So, if I can do the calculation right: 65535 ports/(1min + 12secs = 72secs)
would be roughly ~910
ports a second. That’s pretty fast! 🚀
📃 Source Code
Below is the full source code for the scanner (copy+paste friendly for testing):
40 lines of asynchronous port scanning awesomeness!
👋 Conclusion
Building a high performance, asynchronous port scanner with Ruby is actually pretty awesome, and very possible! By embracing single threaded performance backed by Fibers provides really interesting opportunities, and great performance for I/O-bound operations ( like a port scanner ) that should only get better over time.
If you like what you see, be sure to give the Async/Socketry gem(s) a star 🌟on GitHub — or, make a pull request to make them better! If you see where I did something wrong, want to improve the implementation, or have any questions, feel to chat on Gitter!
Until next time, that’s all folks!