Building a Simple Bot Protection With NGINX JavaScript Module (NJS) and TypeScript
I love Lua. I also love NGINX. The three of us get along just great. Like every relationship, we’ve had our highs and lows (yes, I’m looking at you Lua patterns), but overall life was perfect. Then, NGINX JavaScript Module (NJS for short) came along.
NGINX JavaScript module was first introduced in 2015 but recently received a big boost in functionality with the 0.5.x update. Since I'm a sucker for anything JS, I decided to test it out by building a simple (read naive and not production ready) bot protection module 🤖.
Configuring NGINX
Before diving into bot fight, we have to set up NGINX to support the JavaScript module. The instructions below are for my setup (Ubuntu 20.4/Nginx 1.18), so YMMV, but the general idea should be the same for most setups.
- Start by adding the NGINX PPA key by running:
curl -s https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
2. Setup the repository key by running:
sudo sh -c 'echo "deb http://nginx.org/packages/ubuntu/ focal nginx" >> /etc/apt/sources.list.d/nginx.list'
3. Update the repository list by runningsudo apt update
.
4. Install NJS by running sudo apt install nginx-module-njs
.
If all went well, at this point, you should get this lovely message on your terminal:
5. Enable NJS by adding the following to the top of your main nginx.conf file:
load_module modules/ngx_http_js_module.so;
6. Restart NGINX to load NJS into the running instance:
sudo nginx -s reload
Now your NGINX is ready for some JS love, so let’s move on and create our first line of defense — IP filtering!
Opening Act — Creating the Project
Our bot protection project is going to be written in TypeScript. For that, we need to create a project that will transpile TypeScript to ES5 JavaScript, which NJS can understand. As you may have guessed, NodeJS is a must here, so make sure you are all set up before continuing.
- Create the new project folder and initialize it:
mkdir njs-bot-protection && cd njs-bot-protection
npm init -y
2. Install the required packages:
npm i -D @rollup/plugin-typescript @types/node njs-types rollup typescript
3. Add the build script to the package.json’s scripts section:
{
...
"scripts": {
"build": "rollup -c"
},
...
}
4. To compile the project, you’ll need to tell the TypeScript compiler how to do that with the tsconfig.json file. Create a new tsconfig.json file in the root of the project, and add the following content to it:
5. Lastly, let’s add the rollup config, which will wrap everything up and produce the endgame js file that NJS will read.
Create a new rollup.config.js file in the root of the project, and add the following content to it:
And with that, our boilerplate is all loaded and ready to go. That means it’s time to kick some bots!
Round 1 — IP Filtering
Our first line of bot defense is IP blocking; we compare the IP of an incoming request with a list of known IPs with bad reputations, and if we find a match, we redirect the request to a “block” page.
We’ll begin with creating the JavaScript module:
- In the project root folder, create a new folder called src, and then inside of it create a new bot.ts file.
- Add the following code snippet to bot.ts:
💡 So what do we have here?
- Line 1: Imports the built-in module for the file system (i.e., fs). This module deals with the file system, allowing us to read and write files, among other activities.
- Line 2: Calls the
loadFile
function, passing it the name of the file we wish to load. - Lines 4–12: The implementation of
loadFile
. First, we initialize thedata
variable to an empty string array (line 5), then we try to read and parse a text file containing a list of bad IP addresses into thedata
object (line 7), and finally we return thedata
object (line 11). - Lines 14–21: The implementation of
verifyIP
— the heart of our module (for now). This is the function we will expose to NGINX to verify the IP. We first check if the array of bad reputation IPs contains the current request client IP (line 15). If yes, redirect the request to the block page and end processing (lines 16 and 17). If not , redirect internally to thepages
location (line 20). - Line 23: Exports (read exposes)
verifyIP
externally.
3. Build the module by running npm run build
in your terminal. If all goes well, you should find the compiled bot.js file in the dist folder 🎉
With the file in hand, let’s configure NGINX to be able to use it:
- In your NGINX folder (/etc/nginx in my case) create a folder named njs and copy bot.js from the previous section inside it.
- Create a new folder called njs under /var/lib, create a file called ips.txt inside it, and populate it with a list of bad reputation IPs (one IP per line). You can either add your own list of IPs or use something like https://github.com/stamparm/ipsum.
- In your nginx.conf, under the
http
section, add the following:
js_path "/etc/nginx/njs/";
js_import bot.js;
💡 So what do we have here?
- js_path — Sets the path for the NJS modules folder.
- js_import — Imports a module from the NJS modules folder. If not specified, the imported module namespace will be determined by the file name (in our case,
bot
)
4. Under the server
section (mine is on /etc/nginx/conf.d/default.conf) modify the /
location as follows:
location / {
js_content bot.verifyIP;
}
By calling verifyIP
using the js_content
directive we set it as the content handler, which means verifyIP
can control the content we send back to the caller (in our case, either show a block page or pass the request to the origin)
5. Still under the server
section, add the block.html
location and the pages
named location:
location @pages {
root /usr/share/nginx/html;
proxy_pass http://localhost:8080;
}location /block.html {
root /usr/share/nginx/html;
}
(The namedpages
location will be used by our NJS module to internally redirect the request if it shouldn't be blocked. You likely have your own logic for this redirection so change this to fit your needs)
6. At the bottom of the file, add the server block for port 8080:
server {
listen 8080;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
}
7. Under the /usr/share/nginx/html folder, add the block.html file as follows:
And with that, our IP protection is ready! Add your own IP to the ips.txt file and restart NGINX (sudo nginx -s reload
). Browse to your instance and you should be greeted with the following:
Round 2 — JavaScript Detection
Our second protection layer is JavaScript detection. We use this detection to determine if the visitor coming to our site is running JavaScript (which every normal browser should do) or not (a warning sign that this visitor might not be a legitimate user). We begin with injecting a JavaScript snippet to the pages that will bake a cookie on the root path:
- Add the following code snippets to bot.ts:
💡 So what do we have here?
- Line 1: Imports the built-in Crypto module. This module deals with cryptography, and we will soon use it for creating an HMAC.
- Lines 5–18: The implementation of
getCookiePayload
. The function sets adate
object to one hour ahead of the current time (lines 6–8), then uses thedate
object to HMAC (using thecrypto
module) the signature we passed to the function (thevalue
object) with thedate
object (lines 10–14). Lastly, the function returns the cookie information in a string format (name, value, expiration, etc.). You may notice the cookie value contains not only the hashed signature but also thedate
object we used to HMAC the signature with. You’ll see why we do that soon. - Lines 20–30: The implementation of
addSnippet
. The function buffers the request data, and once it finishes (line 23) it:
- Creates a signature based on the client IP and the User-Agent header (line 24).
- Replaces the closinghead
tag with ascript
section that inserts a cookie (from thegetCookiePayload
function) on the browser side using JavaScript’sdocument.cookie
property. (lines 25–28).
- Sends the modified response back to the client (line 29).
2. Export the new addSnippet
function by updating the export statement at the bottom of the file:
export default { verifyIP, addSnippet };
3. Under the @pages
location block, modify the /
location as follows:
location @pages {
js_body_filter bot.addSnippet;
proxy_pass http://localhost:8080;
}
Unlike verifyIP
, we don't want addSnippet
to manage the content of the response, we want it to inject content (a script
tag in our case) to whatever response comes back from the origin. This is where js_body_filter
comes into play. Using the js_body_filter
directive we tell NJS that the function we provide will modify the original response from the origin and return it once finished.
4. Restart NGINX and browse to a page on your instance. You should see our new script added just before the closing head
tag:
If the client is running JavaScript, a new cookie called njs will be baked. Next, let’s create the validation for this cookie/lack of cookie:
- Add the
verifyCookie
function (and its supportive functions/variables), to bot.ts:
💡 So what do we have here?
- Lines 5–11: The implementation of the
updateFile
function, which uses thefs
module to save an array of strings to a file. - Lines 13–52: The motherload implementation. When validating the njs cookie, we have a flow of verification and consequences we have to follow:
a. We begin with extracting the njs cookie from the request’s Cookie header (lines 14–20).
b. If we don’t have a cookie (or we do and it’s malformed), we compare the client IP against our list of client IPs that have reached us without a cookie. If we find a match from within the last hour, we fail the request (returning false, lines 26–27). If we don’t, we delete the IP (if it's on the list but past one hour) and pass the request (lines 29–34).
c. If we do have a cookie, we split it into a timestamp and a payload and use the timestamp to create our own HMAC hash based on the request’s User-Agent header and client IP. If our own HMAC matches the HMAC of the njs cookie, we pass the request. Otherwise, we fail it (lines 38–45).
d. If anything goes wrong during the validation, we fail open (meaning pass) the request (lines 48–51).
2. Add the new verify
function, which calls the new verifyCookie
function, and act according to its result:
🔥 At the point you might be thinking to yourself at this point that this verify
function looks eerily similar to the verifyIP
function from the earlier — you are absolutely right, and I will touch on that in a minute!
3. To test our new cookie validation functionality, open up your configuration file (mine is at /etc/nginx/conf.d/default.conf) and change the js_content directive from verifyIP
to verify
:
location / {
js_content bot.verify;
}
4. Restart NGINX and try to visit the site twice without the njs cookie — ✋ 🎤- you are blocked!
Final Round — Bringing It All Together
So now we have the cookie verification, but we took off our IP verification because we can only have one js_content
directive, how do we go around fixing that?
You may remember that a few minutes ago we created the verify
function (which eagle-eyed readers may have noticed is VERY similar to the verifyIP
function we used before). If we update our verifyIP
function so that it returns a boolean response as verification, and add that verification to verify
, we get the best of both worlds with one big function that verifies requests for both IPs and cookies!
- Refactor the
verifyIP
function as follows:
2. Update the verify
function to call verifyIP
as follows:
3. Update the export
statement, as we no longer need to expose verifyIP
:
export default { addSnippet, verify };
4. Restart NGINX and enjoy your home-made bot protection using NJS and TypeScript 🎉
🍾 The module source code is available on GitHub!