Functional Proxmox Homelab Framework

Topics Covered

Proxmox — Framework, Setup

pfSense — Interface Management/Assignment, DHCP, OpenVPN

Security Onion — Proxmox Bridge Traffic Mirroring, Fullscreen on Proxmox

Quick Note upfront

This is not a super detailed step-by-step guide. This is more of an explanation of my proxmox/homelab setup. Where ever I gloss over the details of an install or process I have (hopefully) added a link to a more detailed guide in the references.

Proxmox Framework

Howdy all. Too often I see (and have used) guides that focus a lot on installing and configuring a basic Proxmox setup, with no guidance on how to organize it all. Having used Proxmox for nearly 3 years now, as a homelaber, and for an organization of 20–30 active users, it can become a mess really quickly. Which bridges are for what network, what VMs do what (no “TestVM” does not tell me what its for), or what VMs in which network. Some Proxmox admins lean heavily on standardizing VMIDs, and embedding the purpose, network, lifespan, and ownership right in the number. This works great until you have multiple people, or you just cant be bothered and you want a ubuntu container to watch Starwars via telnet. So, I wanted to make something for myself that I would be happy to use as definitely not a poweruser proxmox administrator.

Goal/Features of this setup

  • Keep things organized. Have simple scaling. (Some other stuff too).

Solution

  • One Pool = One Bridge = One Network = One Purpose = One Pod.

Pools keep things nice and organized (and allow for easy user/group permissions when my friends want a VM and network). One network per bridge allows easier VM deployment in the GUI. One purpose per network lets me remember good vs bad vs neutral traffic.

Image for post
Image for post
Quick Network Map of this setup.

For my internal lexicon, I call these pods.

This is in no way revolutionary, but I haven't seen a lot of VM/Proxmox organization discussion on /r/homelab.

Specifically for me, I am interested in Incident response and Active Directory/corporate infrastructure. The 3 pods I decided to start simple with an AD, Red Team, and IR. The Deployment pod is a story for a later time.

You may by now have noticed the floating computer — that's a Security Onion instance I have running pretty much off the shelf. More on that later.

Deploying a VM/container with this setup —

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Example Deployment of Container

This isn’t some Proxmox revolution, but I think its a simple, modular, replicable workflow that keeps things pretty organized.

Proxmox

I won't cover the install process as mine is not optimal nor will it add to this article. Link in the reference to do this.

Image for post
Image for post
Options before installation. I put my personal email, I don't know if that is best practice.
Image for post
Image for post
Clean Install.

Settings up Proxmox Pools and Bridges

Some definitions up from basically straight from the proxmox documentation:

Pools — Pools can be used to group a set of virtual machines and data stores. Think of them like an Organizational Unit (OU). For our purposes, we can think of them as just folders to organize vms.

Bridges — Bridges are like physical network switches implemented in software. The installation program creates a single bridge named vmbr0, which is connected to the first Ethernet card.

Pool Setup

Easier part first. I find it is really important to add the subnet into the pool comment. Its just a little bit of extra work, but you are working for future you. Be nice to future you.

Image for post
Image for post
Create a new pool — Datacenter > Permissions > Pools -> Create
Image for post
Image for post
An example Pool.

Bridge Setup

Each network has its own pool and bridge. Lets make the bridge.

Image for post
Image for post
Datacenter > Node > System > Network > Create -> Linux Bridge
Image for post
Image for post
Setup for example. Keep the name default, and add the CIDR and Add it to the comment.
Image for post
Image for post
What my pretty simple setup looks like.

Now the semi-annoying thing — Restart Proxmox. Yes, restart your node.

pfSense Setup

Now you have a bunch of virtual networks and bridges and pools. You’re gonna need something to route all that traffic. pfSense is great because it is easy and simple to get a lot of basic functions working really quickly. In the future, I might switch to a more industry-standard firewall, but for my purposes this is perfect.

Install and Initial Setup

I won't go over the install, (check if you want references) it's just clicking through. Before you set up your pfsense, only start with two interfaces, your WAN and one pod. Assigning interfaces can be a bit of a guessing game, just start with two, then add one at a time.

Image for post
Image for post
After install use the “assign interfaces” interactive menu to assign your WAN and your first pod as your LAN.

Adding new interfaces

pfSense loves to be pure NAT, and deal with just a WAN and LAN. So we need to be a little more delicate.

Image for post
Image for post
Image for post
Image for post
Add your bridge to pfsense. Datacenter > pfsense VM > Hardware > Add -> Network Device -> vmbrX. (The example vmbr is not active since I didn't reboot yet. Do as I say not as I do).

The comments make that pretty easy to remember right?

Access the pfSense WebGUI. If this is your first time go through the setup wizard.

Image for post
Image for post
Image for post
Image for post
Access the interface assignments. Interfaces > Assignments.

pfSenese will have a green “add” button at the bottom to add the new interface. (I didn't get a picture of this I'm sorry). Select the interface you just added by clicking on its name. It’ll probably be OPT1. This is the special part. I needed to restart pfsense, and even proxmox one time for it to register the new interface. If you don't see it, confirm the bridge is active, and that it is attached. If you don't see it… no joke try rebooting.

Image for post
Image for post
Image for post
Image for post
Edit the interface information. Change the name to reflect the bridge name, and add the gateway IP and the subnet. This is for my pod2.
Image for post
Image for post

It should look something like this when all is said and done.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

Update — 5/15/2020

Howdy all, quick update, you need to make a quick change to pfsense if using this model. It’ll run unbearably slow if you do not.

  1. Log in to pfSense.
  2. Go to System > Advanced > Networking.
  3. Enable Disable hardware checksum offload.
Image for post
Image for post
Image for post
Image for post
Which do you want?

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

DHCP

I run DHCP on a section of the network pods with pfSense. It makes deploying random vms less of a pain and lets me do DHCP with AD/Something else on pods where I want to practice that.

Image for post
Image for post
Image for post
Image for post
pfsense Services > DHCP Server > <Your interface> (VMBR2 for this example).
Image for post
Image for post
Configuration for the DHCP service on this bridge.

Again, when I initially went to set this up the VMBR2 option wasn't there. For a while, I thought it might not be supported. Nope, the service needed to be restarted to recognize the new interface.

OpenVPN

A comfort feature needed to make working with the homelab environment easier is a VPN to access the pods. In the future, I want to set up Direct Access, but that's another time. See the references for a link to a setup guide I used. The only thing I can add is make sure you really look at your firewall rules, and check your logs (Status > System Logs > OpenVPN). I had a problem where I was blocking local addresses on the WAN, which makes sense for a border device, but not this. Oh yeah, and make sure you goto Interfaces > Assignment… and add the OpenVPN interface.

Image for post
Image for post
Setup OpenVPN to access the pods. VPN > OpenVPN -> Wizard. Setting it up can be a bit finicky, but its worth it.

Security Onion

The last core part of this infra is a security onion deployment. SecOnion is a great tool that I really want to get better at. For those who don't know, SecOnion is a distro that comes packed with a whole range of Network Security Monitoring tools configured right out of the box! I wanted to add this since I’ve had some trouble in the past with implementing a span port or even changing the bridge to act as a hub. And its position in the network map means that it can pick up all of the malicious packets from the RedTeam network aimed at the AD/Corp pod.

Install SecOnion as Usual

Image for post
Image for post
Install and run the setup script like normal. See References on how to do this.

SecurityOnion Full Screen

So… you want the good stuff right? You don't want to be limited to a 900x800 (or whatever it was) crappy screen right? You want to be part of the cool 1920x1080 gang. Well, you are kinda in the right place. After a lot of attempts, I followed this guide. And it didn't work until I changed the display type to VGA. (Don't forget to reboot the box, the display is not hot-swappable).

Image for post
Image for post
Changing the Display to VGA plus the custom mileage may vary
Image for post
Image for post
Its annoying but well worth it.

Configure the Bridges to act as Hubs

To do this, the Proxmox forms have two competing approaches, modifying the interface config or using I personally found success in editing the config directly.

Image for post
Image for post
Goto Datacenter > node > Shell.

Make sure you are root. Add:

as shown below. Yes, you do need to restart the proxmox node again for this to work. Edited 5/29/20 formatting and added .

Image for post
Image for post
Image for post
Image for post
Working Config that makes the bridges act like hubs.
Image for post
Image for post
Image for post
Image for post
A quick test showing traffic being sent out.

Conclusion

I hope this article can serve as a jumping-off point for those interested in making a homelab, or improving their own. This was by no means a step-by-step guide, but hopefully, it was detailed enough to illustrate how/if you could do this in your own environment. This homelab is what works for me, but is also a pretty solid framework/foundation that I bet can fill most use cases.

Quick shoutout to Tyler Blanco for being awesome.

References

Basics Proxmox Install

Virtualize Everything! — Proxmox Install Tutorial

pfsense Install in Promxox

pfsense virtualized in proxmox

pfSense OpenVPN

How To Setup OpenVPN For Remote Access On pfsense

SecOnion Install

Security Onion Lab: How to Install/Configure/Troubleshoot *NEW*\

SecOnion Resolution

xrandr — How to set a custom resolution?

[SOLVED] Qemu-kvm guest resolution

Setting the screen resolution in Linux to something higher than 1024x768?

Broadcast Traffic

How to configure a Linux Bridge to act as a Hub instead of a Switch

[SOLVED] — Linux bridge port mirror using tc (only receiving broadcast traffic)

Deploying Security Onion / Proxmox Port mirroring | Proxmox Support Forum

Pool and Bridge Information

User Management

Network Configuration

Windows and IR are pretty cool.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store