Thinking like a Hacker: Automated security regressions testing in CI/CD with Nuclei

Published in

Version 1

7 min readMar 27, 2023

Nuclei is an open-source, template-based, vulnerability scanner maintained by the team at Project Discovery, if you have used pen testing tools, you might know them from sub finder, httpx or dnsx.

Baby steps

The cool thing about Nuclei is that it is so extensible, it takes advantage of Go’s goroutines and it clusters http requests to be pretty blazingly fast and efficient.

To install Nuclei, you can use Go:

go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest

You can also use Brew or Docker.

Ensure you do this in a secure sandboxed environment using a VM and/or tip off your security team first as some templates contain strings that seem very similar to malware and may set off your malware detection software.

Upon installing, you then must download and install the official templates (if you don’t do this, it will be done on the first scan):

nuclei --update-templates

__     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.9

                projectdiscovery.io[INF] nuclei-templates are not installed, installing...
[INF] Successfully downloaded nuclei-templates (v9.3.9) to /home/ben/nuclei-templates. GoodLuck!

A bunch of templates will be installed to ~/nuclei-templates .

You can now run a scan on a locally running service (we’ll use the classic DVWA or Damn Vulnerable Web App) and see what happens. Just pass in a url, ip and port or CIDR range via the u flag (or a list via the list flag).

WARNING: If you are running the DVWA, do so in a safe, sandboxed environment as it is, as the name says, vulnerable to a LOT of exploits

docker run --rm -it -p 80:80 vulnerables/web-dvwa

nuclei -u localhost:80

__     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.9

                projectdiscovery.io[INF] Using Nuclei Engine 2.8.9 (latest)
[INF] Using Nuclei Templates 9.3.9 (latest)
[INF] Templates added in last update: 61
[INF] Templates loaded for scan: 5638
[INF] Targets loaded for scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[INF] Templates clustered: 995 (Reduced 917 Requests)
[tech-detect:php] [http] [info] http://localhost:80
[exposed-gitignore] [http] [info] http://localhost:80/.gitignore
[waf-detect:apachegeneric] [http] [info] http://localhost:80/
[robots-txt-endpoint] [http] [info] http://localhost:80/robots.txt
[readme-md] [http] [info] http://localhost:80/README.md

We didn’t find anything too crazy as we have no credentials to log in with and we aren’t going to brute force them (though nuclei can be used for this). But hopefully, this does serve as a simple example of how to use the tool.

Be sure to check if you need to rate limit/use particular headers to help your fellow engineers parse logs/not affect services negatively.

Template Magic

Let’s take a look at one of those templates in ~/nuclei-templates that checks for the existence of an exposed phpmyadmin panel:

$ cat ~/nuclei-templates/exposed-panels/phpmyadmin-panel.yaml

id: phpmyadmin-panelinfo:
  name: phpMyAdmin Panel - Detect
  author: pdteam
  severity: info
  description: phpMyAdmin panel was detected.
  classification:
    cvss-metrics: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:N
    cvss-score: 0.0
    cwe-id: CWE-200
  metadata:
    shodan-query: http.title:phpMyAdmin
  tags: panel,phpmyadminrequests:
  - method: GET
    path:
      - "{{BaseURL}}/phpmyadmin/"
      - "{{BaseURL}}/admin/phpmyadmin/"
      - "{{BaseURL}}/_phpmyadmin/"
      - "{{BaseURL}}/administrator/components/com_joommyadmin/phpmyadmin/"
      - "{{BaseURL}}/apache-default/phpmyadmin/"
      - "{{BaseURL}}/blog/phpmyadmin/"
      - "{{BaseURL}}/forum/phpmyadmin/"
      - "{{BaseURL}}/php/phpmyadmin/"
      - "{{BaseURL}}/typo3/phpmyadmin/"
      - "{{BaseURL}}/web/phpmyadmin/"
      - "{{BaseURL}}/xampp/phpmyadmin/"
      - "{{BaseURL}}/phpMyAdmin/"    stop-at-first-match: true
    matchers:
      - type: word
        words:
          - "<title>phpMyAdmin"
          - "pmahomme"    extractors:
      - type: regex
        part: body
        group: 1
        regex:
          - 'v=([a-z0-9-._]+)'# Enhanced by md on 2022/12/06

As you can see, a Nuclei template is a YAML file made up of and id (a unique string which must contain no spaces), info (containing template metadata) and requests.

info has a few fields that are definitely worth filling in for when you write your own template (particularly the tags and severity fields) as these can be used for filtering which templates are used in the scans. This block provides name, author, severity, description, reference, tags and metadata but you can define any field you want and it will be valid if you want to have custom fields for your internal processes.

requests is where the logic of the template lives. In here:

method defines the http method to be used (this is a simple GET request)
path is a list of paths to send requests to (using the BaseURL variable, i.e. the URL passed in when the scan is run),
matchers is a list of matcher objects (the word matcher type is a very simple type that matches direct strings) that should be used to indicate whether the template has succeeded (i.e. in this case if the title tag of that page starts with “phpMyAdmin” or the string “pmahomme” exists)
extractors is a list of extractor objects which grab information to display when the matcher field passes

The main matcher types are as follows:

status: The HTTP status of the response
size: The Content-Length of the response
word: Whether the response contains one or more strings
regex: Whether the response matches a regex
binary: Matching file types (e.g. 504B0304 for zip archive)
dsl: A simple language for querying anything that’s too complex for the other types

The main extractor types are as follows:

regex: Extract using regex and capture groups
kval: Extract key-value pairs from the response headers
json: Extract from response json (like using curl and jq )
xpath: Extract xpath data from response
dsl: Similar to dsl above, but for extraction

It should be noted that this is a very simple example and you can use Nuclei to send and receive:

Other HTTP request methods (POST, PUT etc.)
Raw HTTP requests
Headless requests (this opens the given path in a headless browser and allows you to script user actions, not a Selenium replacement or anything, but very cool)
Network requests (like an automated Netcat, sending and receiving just bytes)
DNS requests (e.g. look for a CNAME/A Record at the specified URL)
File requests (e.g. search a file system for just py files containing the string crypto )

For more templating help, see the very helpful Project Discovery docs.

I’m reading for the CI/CD, get on with it!

So, this is cool for hackers, pentesters and bug bounty hunters… But how can I, a humble DevOps Engineer, utilize this technology?

What if we put Nuclei in the CI/CD pipeline? What if we could deploy to any of our environments and test them all for a multitude of CVEs and other vulnerabilities? What if we could discover a vulnerability on one project, write a template to detect it, fix it, and add the template to a shared repo that will be used when other projects in our org are deployed?

A cycle of six boxes connected with arrows that read counter clockwise from 12: Fix vulnerability, Write nuclei template, Add to template repo, Deploy new version, CI/CD pipeline uses nuclei to test new version, Find vulnerability during QA/Security testing — A very simplified regressions cycle

It becomes more accessible and easier to use and you have to write templates less and less frequently as time goes on and it can stop these important bugs from being reintroduced.

Example time

I like using Gitlab CI and GitHub Actions for CI/CD however you can make Nuclei work with pretty much anywhere you can install go or docker.

If you are using GitHub, then you can just use the official action (which can auto-create GitHub issues when it finds a vulnerability). However, I’ll be showing a Gitlab CI job example:

Bear in mind these are not best practices, I have tried to make them as brief as possible while still having a somewhat complete example.

image: docker:19.03.13
variables:
  DOCKER_TLS_CERTDIR: "/certs"
services:
  - docker:19.03.13-dind

stages:
- test
- build
- deploy
- smoke-test
- nuclei-scan
test:
  stage: test
  image: node:18
  script:
    - ./ci-scripts/test.sh
  only:
    - main
build:
  stage: build
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$COMMIT_SHA .
    - docker tag $CI_REGISTRY_IMAGE:$COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:$COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
  only:
    - main
deploy:
  stage: deploy
  image: registry.gitlab.com/benhowarth/runner-images/gcp-ci:latest
  script:
    - ./ci-scripts/deploy.sh $DEPLOY_HOST
  only:
    - main
smoke-test:
  stage: smoke-test
  image: registry.gitlab.com/benhowarth/runner-images/gcp-ci:latest
  script:
    - ./ci-scripts/smoke-test.sh $DEPLOY_HOST
  only:
    - main
nuclei:
  stage: nuclei
  image: registry.gitlab.com/benhowarth/runner-images/my-nuclei-image-with-custom-templates:latest
  script:
    - nuclei -t /custom-templates-location -u $DEPLOY_HOST
only:
    - main

As you can see in the example above, we build a docker image, deploy it to the given environment, smoke test the service (just a ping to check it’s alive) and finally run all our nuclei templates on the given host to see if we find a vulnerability.

nuclei -t /custom-templates-location -u $DEPLOY_HOST

It should be noted that this example uses a custom-built image for running this nuclei command. This would come from a repository with a number of custom nuclei templates that have a CI/CD pipeline set up to rebuild the docker image registry.gitlab.com/benhowarth/runner-images/my-nuclei-image-with-custom-templates:latest when any new templates are added. Meaning no one has to worry about adding the new templates to their current project repo or stress out debugging this step.

Is it right for you?

This has been a small exploration into how nuclei can be used for automated security regressions to avoid security flaws being introduced or reintroduced and how a more secure infrastructure can become relatively easy to keep on top of.

This approach may not be right for you if:

You are already paying for a security auditing service that will cover these things
You are limited by CI/CD pipeline spend/time already
You have a different security solution in mind
You despise YAML with a passion

Regardless, I hope this got some cogs turning for you about nuclei and security testing in general.

If this piece interests you, feel free to dig into fuzzing with nuclei (there’s another official template repo just for this), workflows (the ability to make templates more efficient by using one to detect if others should be run), or using the nuclei tool as a go library for a larger security tool or utility you might want to build.

About the author:
Ben Howarth is a Senior AWS DevOps Engineer here at Version 1.