How I used HAProxy to overcome the AWS Web Application Firewall’s hard limit of 10 Rules per Web ACL

I love Amazon Web Services. Most of the time, I find it intuitive and a good match for my needs. From experience, I’ve found that when you hit an AWS hard limit you are generally working outside of Amazon’s expected use of the feature and you should probably pivot. I hit one such hard limit recently.

AWS Application Load Balancers (ALBs) are great at distributing HTTP traffic to different targets. A single load balancer can support up to 1000 targets and 100 rules. A single AWS account plus region combination can support 50 load balancers. It’s both cost effective and easy to combine many sites into a single load balancer, so long as you stay away from these limits. This was a good match for our needs.

ALBs use rules to direct traffic to the appropriate target and 100 rules is plenty, and you can add more ALBs if needed to reach up to 5000 rules in total (though you will likely hit other limits first). Each rule can execute up to 5 evaluations and this is where I first ran into problems. One additional note that isn’t listed in the Limits page:

You can specify up to three match evaluations per condition

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html#rule-condition-types

That’s great if you can combine all of your rules within these limits, but what if you can’t?

Enter AWS WAF

AWS Web Application Firewall allows you to add additional rules for the processing of content. I like this because it allows me to separate security processing from normal behaviour. I use the ALB rules to define how requests should be handled and I use WAF rules to act as a security gate on all content with a variety of conditions.

WAF is much more powerful than the ALB rule conditions but beware those pesky hard limits:

• Rules per web ACL: 10
• Conditions per rule: 10

For me, the limit of 10 rules per web ACL is a massive disappointment. The way I interpret this is that you are expected to use a separate web ACL per site. However, the ALB only allows you to attach one web ACL and so this limits how much consolidation you can achieve. Indeed, if you publish your sites via CloudFront and limit access to the ALB to CloudFront sources, you may never see this as an issue.

Once I hit this limit, I needed to find another solution.

Enter HAProxy

I was looking for a rule system that could apply a combination of host and source IP filters, only now the source IP filter would need to be on the X-Forwarded-For header populated by the ALB.

There are plenty of options to choose from, but I found HAProxy to be the easiest to configure by far.

My HAProxy configuration file looks a bit like the following:

global
  log stdout len 32000 user info
resolvers all
  parse-resolv-conf
  hold valid 10s
frontend ingress
  bind *:8000
  mode http
  option httplog
  log global
  default_backend egress
  timeout client 900000ms
  capture request header X-Forwarded-For len 15
  capture request header Host len 128
  option forwardfor if-none
# Allow ALB health check pass-through
  acl health_check_path               path      /healthz
  http-request allow if               health_check_path
# 1. DEFINE HOSTS
  # .... Example Host
  acl host_example    hdr(host) -i example.mycompany.com
  # .... Another Host
  acl host_another    hdr(host) -i sensitive.mycompany.com
# 2. DEFINE COMMON ACLs
  acl client_xff_my_company hdr_ip(x-forwarded-for) 10.12.0.0/16
  acl client_xff_vpc        hdr_ip(x-forwarded-for) 192.168.18.0/24
# 3. DEFINE BEHAVIOUR PER HOST
  # .... Example Host
  http-request allow if   host_example client_xff_my_company
  http-request allow if   host_example client_xff_vpc
  # .... Another Host
  http-request allow if   host_another client_xff_vpc
# 4. BLOCK ALL ELSE
  http-request deny if    TRUE
frontend stats
  bind *:9000
  mode http
  timeout client 5000ms
  stats enable
  stats show-node
  stats refresh 30s
  stats uri /haproxy
  monitor-uri /healthz
  acl site_dead nbsrv(egress) lt 1
  monitor fail if site_dead
backend egress
  mode http
  option redispatch
  server-template tunnel 1-5 my-targets.me.local:32323
  timeout tunnel 28800000ms # 8 hours, websockets
  timeout connect 5000ms
  timeout server 900000ms

I first define an ACL for each host header that I expect. I then define an ACL for each IP match that I can support. I then list all of the allowed combinations as http-request allow if statements with the combination of the previously defined ACLs. Finally, anything that hasn’t already matched is blocked using http-request deny if TRUE. TRUE is a predefined ACL that matches everything.

I’m also making use of HAProxy’s ability to update the IP addresses it associates with a DNS entry with the server-template clause in the backend block. Watch out for this if you are using nginx or an earlier version of HAProxy.

Hosting HAProxy

In my case, I hosted HAProxy inside my Kubernetes cluster next to the ingress controller it was protecting. In your case, you might host it somewhere else. In either case, here’s my configuration outline:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: firewall-haproxy
  namespace: ingress
spec:
  replicas: 3
  selector:
    matchLabels:
      app: firewall-haproxy
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: firewall-haproxy
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - firewall-haproxy
              topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - "-f"
        - "/injected/haproxy.cfg"
        image: haproxy:2.0.1
        imagePullPolicy: IfNotPresent
        name: haproxy
        tty: false
        ports:
        - containerPort: 8000
          name: http
          protocol: TCP
        - containerPort: 9000
          name: monitor
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          timeoutSeconds: 2
          successThreshold: 1
          initialDelaySeconds: 3
          periodSeconds: 2
          httpGet:
            port: monitor
            path: /healthz
            scheme: HTTP
        livenessProbe:
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 2
          timeoutSeconds: 2
          exec:
            command:
              - bash
              - -c
              - 'tmpfile=/tmp/haproxy-config.md5 ; current_md5=`cat /injected/haproxy.cfg | md5sum` ; if [[ ! -f "$tmpfile" ]] ; then echo "$current_md5" > $tmpfile ; fi ; stored_md5=`cat $tmpfile` ; if [[ ! "$stored_md5" == "$current_md5" ]] ; then echo "config changed, liveness check failing" ; exit 1; fi ; exit 0'
        resources:
          requests:
            cpu: "100m"
            memory: "256Mi"
          limits:
            memory: "256Mi"
        volumeMounts:
          - mountPath: /injected
            name: config-haproxy
        securityContext:
          privileged: true
      volumes:
      - name: config-haproxy
        configMap:
          name: firewall-haproxy-config

The liveness check here is worth a short mention. I’m hacking the liveness probe to act as a regular configuration check. If the configuration has changed it forces the liveness check to report an error and this causes the pod to be killed and replaced. This may be too brutal for your environment but it worked perfectly for mine.

References

• https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html
• https://docs.aws.amazon.com/waf/latest/developerguide/limits.html
• https://www.haproxy.org

This post is also published on my blog at https://stevehorsfield.wordpress.com/2019/10/10/how-i-used-haproxy-to-overcome-the-aws-web-application-firewall's-hard-limit-of-10-rules-per-web-acl/.