Secure and handy VPC access for remote workers — WireGuard, Unbound, Bastillion

The first question required to answers when bootstrapping a cloud infrastructure for the team: How to enable access to the VMs/servers with proper authentication and authorization? The team may consist of developers, DevOps, SRE, or even consultants. Some security measures are required when designing how team members accessing the infrastructure. Some high-level objective is required.

• Minimize the public access point to avoid brute-force attacks and centralize the make it easier for abnormality detection and auditing
• Ease to grant or revoke access for team members’ onboarding and departure.
• Enable SSH-based access when team members using their notebook and work remotely

Architecture

Setup

[DNS] Unbound DNS Setup

As to enable private domain name for all VMs within VPC, Unbound DNS is used to host all the internal domains for all VMs and VPN Clients.

yum install unbound
> change /etc/unbound.conf
## Listen on VPC network only
interface: 10.124.0.2
## Accept all DNS query from VPC network
access-control: 0.0.0.0/0 allow
service unbound restart

And all VMs within VPC needed to use the unbound DNS as the DNS resolvers. Register some internal domain for use

unbound-control local_zone bootcamp.ynerv.io. static
unbound-control local_data "vm3.bootcamp.ynerv.io IN A 10.124.0.3"
unbound-control local_data "vm4.bootcamp.ynerv.io IN A 10.124.0.4"
unbound-control local_data "vpn01.bootcamp.ynerv.io IN A 10.124.0.2

• Remark 1: The reason for using Unbound is on its remove unbound-ctl which can make it easier to automate the domain modification.
• Remark 2: unbound-control setup cannot survive a reboot. Any advice to make the change persistence?

[VPN] WireGuard VPN Server Setup

https://www.wireguard.com/install/

eth1 is the network device of the private IP of the VPN server (10.128.0.2). So the VPN server forwards the IP to 10.128.0.0/24 subnet.

[Interface]
Address = 10.10.0.1/24
ListenPort = 51820
PrivateKey = <VPN Server Private Key>
PreUp = iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o eth1 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -s 10.10.0.0/24 -o eth1 -j MASQUERADE
[Peer]
PublicKey = <Client 1 Public Key>
AllowedIPs = 10.10.0.2/32

https://www.wireguardconfig.com/ is handy on generating all these configs.

[VPN] WireGuard VPN Client Setup

[Interface]
Address = 10.10.0.2/24
PrivateKey = <Client 1 Private Key>
# As for split traffic for only VPC subnet, need to use OSX resolver to only resolve the internal domain
Postup = echo ‘nameserver 10.124.0.2’ > /etc/resolver/bootcamp.ynerv.io
PostDown = rm /etc/resolver/bootcamp.ynerv.io
[Peer]
PublicKey = <VPN Server Public Key>
AllowedIPs = 10.10.0.0/24, 10.124.0.0/20
Endpoint = 125.125.125.125:51820

Brew on OS X is used instead of the one from the App Store. Because the WireGuard App does not support Postup and PostDown directives to enable OSX resolver only for the internal domains.

brew install wireguard-tools
wg-quick up ./client1.conf

[VPN] Verify the connection

On VPN Server

# wg show
interface: wg0
  public key: (***masked***)
  private key: (hidden)
  listening port: 51820
peer: (***masked***)
  endpoint: (***masked***)
  allowed ips: 10.10.0.2/32
  latest handshake: 31 minutes, 45 seconds ago
  transfer: 3.54 KiB received, 1.25 KiB sent

On Client

# sudo wg show
interface: utun4
  public key: (***masked***) 
  private key: (hidden)
  listening port: (***masked***)
peer: (***masked***)
  endpoint: (*** VPN Server IP***)
  allowed ips: 10.10.0.0/24, 10.124.0.0/20
  latest handshake: 33 minutes, 41 seconds ago
  transfer: 1.25 KiB received, 5.60 KiB sent
# ping -c 2 10.124.0.2
PING 10.124.0.2 (10.124.0.2): 56 data bytes
64 bytes from 10.124.0.2: icmp_seq=0 ttl=64 time=163.541 ms
64 bytes from 10.124.0.2: icmp_seq=1 ttl=64 time=160.561 ms
# ping -c 2 10.124.0.3
PING 10.124.0.3 (10.124.0.3): 56 data bytes
64 bytes from 10.124.0.3: icmp_seq=0 ttl=63 time=163.306 ms
64 bytes from 10.124.0.3: icmp_seq=1 ttl=63 time=161.447 ms
#ping -c 2 10.124.0.4
PING 10.124.0.4 (10.124.0.4): 56 data bytes
64 bytes from 10.124.0.4: icmp_seq=0 ttl=63 time=163.128 ms
64 bytes from 10.124.0.4: icmp_seq=1 ttl=63 time=160.986 ms
# traceroute -n 10.124.0.3
traceroute to 10.124.0.3 (10.124.0.3), 64 hops max, 52 byte packets
 1  10.10.0.1  161.936 ms  160.558 ms  160.922 ms
 2  10.124.0.3  162.992 ms  160.247 ms  161.033 m
# ping vm3.bootcamp.ynerv.io
PING vm3.bootcamp.ynerv.io (10.124.0.3): 56 data bytes
64 bytes from 10.124.0.3: icmp_seq=0 ttl=63 time=163.268 ms
64 bytes from 10.124.0.3: icmp_seq=1 ttl=63 time=163.201 ms

[SSH] Bastillion — Key Management

To manage the SSH access the VMs within VPC for team members, we ssh key management, and a handy web-based shell terminal is required. I picked Bastillion for the case.

Bastillion — Documentation

I put the bastillion web inside the VPC network and expose the port on private IP, to restrict access from the public and only reachable from the VPN network.

Please refer to another post for more details on the setup.

Discussion

WireGuard is super fast and pretty easy to config while the hardest part is on the network design. VPN User additional currently is a bit clumsy and looks forward to better tools for the whole config management.

Firewall policy has not been strictly enforced in this example. Suggest reusing the cloud infrastructure to control the network access instead of setting it on the OS level.

Some future works want to be explored, and feel free to share your advice.

• Automation Auto Register VM with Bastillion
• Unbound Persistent Storage
• Work Station Setup (SSH Config, Ansible, VSCode Integration)

Next Coming Up

I want to try Chef on this VPC setup instead of Ansible. Stay tuned for the next update.

Reference

WireGuard VPN Road Warrior Setup

BetterWayElectronics/secure-wireguard-implementation

Wireguard VPN: Typical Setup - The poetry of (in)security