Proxmox — Framework, Setup
pfSense — Interface Management/Assignment, DHCP, OpenVPN
Security Onion — Proxmox Bridge Traffic Mirroring, Fullscreen on Proxmox
Quick Note upfront
This is not a super detailed step-by-step guide. This is more of an explanation of my proxmox/homelab setup. Where ever I gloss over the details of an install or process I have (hopefully) added a link to a more detailed guide in the references.
Howdy all. Too often I see (and have used) guides that focus a lot on installing and configuring a basic Proxmox setup, with no guidance on how to organize it all. Having used Proxmox for nearly 3 years now, as a homelaber, and for an organization of 20–30 active users, it can become a mess really quickly. Which bridges are for what network, what VMs do what (no “TestVM” does not tell me what its for), or what VMs in which network. Some Proxmox admins lean heavily on standardizing VMIDs, and embedding the purpose, network, lifespan, and ownership right in the number. This works great until you have multiple people, or you just cant be bothered and you want a ubuntu container to watch Starwars via telnet.
telnet towel.blinkenlights.nlSo, I wanted to make something for myself that I would be happy to use as definitely not a poweruser proxmox administrator.
Goal/Features of this setup
- Keep things organized. Have simple scaling. (Some other stuff too).
- One Pool = One Bridge = One Network = One Purpose = One Pod.
Pools keep things nice and organized (and allow for easy user/group permissions when my friends want a VM and network). One network per bridge allows easier VM deployment in the GUI. One purpose per network lets me remember good vs bad vs neutral traffic.
For my internal lexicon, I call these pods.
This is in no way revolutionary, but I haven't seen a lot of VM/Proxmox organization discussion on /r/homelab.
Specifically for me, I am interested in Incident response and Active Directory/corporate infrastructure. The 3 pods I decided to start simple with an AD, Red Team, and IR. The Deployment pod is a story for a later time.
You may by now have noticed the floating computer — that's a Security Onion instance I have running pretty much off the shelf. More on that later.
Deploying a VM/container with this setup —
This isn’t some Proxmox revolution, but I think its a simple, modular, replicable workflow that keeps things pretty organized.
I won't cover the install process as mine is not optimal nor will it add to this article. Link in the reference to do this.
Settings up Proxmox Pools and Bridges
Some definitions up from basically straight from the proxmox documentation:
Pools — Pools can be used to group a set of virtual machines and data stores. Think of them like an Organizational Unit (OU). For our purposes, we can think of them as just folders to organize vms.
Bridges — Bridges are like physical network switches implemented in software. The installation program creates a single bridge named vmbr0, which is connected to the first Ethernet card.
Easier part first. I find it is really important to add the subnet into the pool comment. Its just a little bit of extra work, but you are working for future you. Be nice to future you.
Each network has its own pool and bridge. Lets make the bridge.
Now the semi-annoying thing — Restart Proxmox. Yes, restart your node.
Now you have a bunch of virtual networks and bridges and pools. You’re gonna need something to route all that traffic. pfSense is great because it is easy and simple to get a lot of basic functions working really quickly. In the future, I might switch to a more industry-standard firewall, but for my purposes this is perfect.
Install and Initial Setup
I won't go over the install, (check if you want references) it's just clicking through. Before you set up your pfsense, only start with two interfaces, your WAN and one pod. Assigning interfaces can be a bit of a guessing game, just start with two, then add one at a time.
Adding new interfaces
pfSense loves to be pure NAT, and deal with just a WAN and LAN. So we need to be a little more delicate.
The comments make that pretty easy to remember right?
Access the pfSense WebGUI. If this is your first time go through the setup wizard.
pfSenese will have a green “add” button at the bottom to add the new interface. (I didn't get a picture of this I'm sorry). Select the interface you just added by clicking on its name. It’ll probably be OPT1. This is the special part. I needed to restart pfsense, and even proxmox one time for it to register the new interface. If you don't see it, confirm the bridge is active, and that it is attached. If you don't see it… no joke try rebooting.
It should look something like this when all is said and done.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
Update — 5/15/2020
Howdy all, quick update, you need to make a quick change to pfsense if using this model. It’ll run unbearably slow if you do not.
- Log in to pfSense.
- Go to System > Advanced > Networking.
- Enable Disable hardware checksum offload.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
I run DHCP on a section of the network pods with pfSense. It makes deploying random vms less of a pain and lets me do DHCP with AD/Something else on pods where I want to practice that.
Again, when I initially went to set this up the VMBR2 option wasn't there. For a while, I thought it might not be supported. Nope, the service needed to be restarted to recognize the new interface.
A comfort feature needed to make working with the homelab environment easier is a VPN to access the pods. In the future, I want to set up Direct Access, but that's another time. See the references for a link to a setup guide I used. The only thing I can add is make sure you really look at your firewall rules, and check your logs (Status > System Logs > OpenVPN). I had a problem where I was blocking local addresses on the WAN, which makes sense for a border device, but not this. Oh yeah, and make sure you goto Interfaces > Assignment… and add the OpenVPN interface.
The last core part of this infra is a security onion deployment. SecOnion is a great tool that I really want to get better at. For those who don't know, SecOnion is a distro that comes packed with a whole range of Network Security Monitoring tools configured right out of the box! I wanted to add this since I’ve had some trouble in the past with implementing a span port or even changing the bridge to act as a hub. And its position in the network map means that it can pick up all of the malicious packets from the RedTeam network aimed at the AD/Corp pod.
Install SecOnion as Usual
SecurityOnion Full Screen
So… you want the good stuff right? You don't want to be limited to a 900x800 (or whatever it was) crappy screen right? You want to be part of the cool 1920x1080 gang. Well, you are kinda in the right place. After a lot of attempts, I followed this guide. And it didn't work until I changed the display type to VGA. (Don't forget to reboot the box, the display is not hot-swappable).
Configure the Bridges to act as Hubs
To do this, the Proxmox forms have two competing approaches, modifying the interface config or using
tc I personally found success in editing the config directly.
nano /etc/network/interfaces Make sure you are root. Add:
as shown below. Yes, you do need to restart the proxmox node again for this to work. Edited 5/29/20 formatting and added
bridge-ageing 0 .
I hope this article can serve as a jumping-off point for those interested in making a homelab, or improving their own. This was by no means a step-by-step guide, but hopefully, it was detailed enough to illustrate how/if you could do this in your own environment. This homelab is what works for me, but is also a pretty solid framework/foundation that I bet can fill most use cases.
Quick shoutout to Tyler Blanco for being awesome.
Basics Proxmox Install
pfsense Install in Promxox