Intel NUC 10/Proxmox VE HomeLab
I’ve always wanted to have a personal HomeLab for studying purposes and for running services in my home network and at the end of 2022 I’ve been able to finally build one and I want to share with you some details about it.
Introduction
Before building my HomeLab, I was running Virtual Machines with VMWare Fusion on my MacBook Pro (late-2013) and even if it was a powerful machine with 16GB of ram when I’ve bought it, I was starting to ask too much when I wanted to run 5–6 Kubernetes nodes and other stuff such as Visual Studio Code, so it was time to find a replacement. Some of my home network services were also running on a Raspberry Pi3b and on my Synology DS218+ NAS (2x6TB RAID1 Disks + 10GB ram) through Docker containers and I wanted to move some of them on a more performing box, thus making me desire even more a new small, low-power but powerful home server.
The Hardware
In the last months of 2022 I was studying for CKA (My Journey to the CKA has some details about that adventure) and the need for a more performing server become higher and higher and so I’ve started looking for an Intel NUC 10th Gen, as some of my tech-friends were happily using it as HomeLab.
After a bit of search on Ebay and Subito.it I’ve managed to get the following hardware (RAM has been recently bumped from 32GB to 64GB):
- Intel NUC Performance NUC10i7FNH with 6-cores (12 threads) Intel i7 Gen10 CPU
- 64GB (2x32GB) DDR4 3200 (running at 2666Mhz) of DDR4 RAM by Crucial
- 2TB NVMe Sabrent Rocket 4 PCI Gen4 SSD for OS and lvm-thin pool (Virtual Machines and Linux Containers)
- 2TB Crucial MX500 SATA SSD for automatic backups via Proxmox Backup Server
Virtualization Software
The hardware was ready, it was time to choose the virtualization software. ESXi is widely used on Intel NUCs with a lot of online tutorials about installing and configuring it… but I wanted something more customizable without paying a subscription to have all the features I wanted, so I choose Proxmox VE, “a complete, open-source server management platform for enterprise virtualization” as they say on their site.
Installing Proxmox VE on an Intel NUC is an easy task and the only issue I’ve found was a problem related to e1000 Intel NIC drivers that were causing an interface reset every few minutes. I’ve solved the problem by disabling some kind of offload in the interface configuration as you can see in the following code snippet, where I also enforce Wake-On-LAN settings to be able to power-on the NUC with a magic packet:
Some references if you’re facing similar issues with e1000 on Proxmox VE:
- Disable TSO and GSO: https://forum.proxmox.com/threads/e1000-driver-hang.58284/page-8#post-390709
- Disable GRO: https://serverfault.com/questions/616485/e1000e-reset-adapter-unexpectedly-detected-hardware-unit-hang/616623#616623
Backup & Disaster Recovery Strategies
One of the first things I like to do after configuring a new device is setting up automatic backup jobs to avoid loosing important data in case of disasters (broken disks or human errors).
These are the items that need a backup on my Proxmox VE HomeLab:
- Proxmox VE (aka PVE) configuration and custom scripts.
- Proxmox Backup Server (aka PBS) running as LXC (LinuX Container, or CT from now on).
- LXCs and VMs running on PVE.
Backup Strategy
This is how I’m actually backing up the whole HomeLab:
- Daily cronjob script that saves /etc, /opt and /root/scripts on an NFS share on my Synology NAS (which in turn saves the home-devices backup on Synology C2 Cloud along with other important data).
- Daily backup scheduled on PVE that saves PBS CT (container) to an NFS share on my Synology NAS.
- Daily backup scheduled on PVE that saves all the important VMs and CTs (the ones running services for the Home-LAN) on the 2TB SATA disk via PBS.
- Weekly/monthly backups of all the other VMs and CTs on the 2TB SATA disk via PBS.
- Monthly backup of all the VMs and CTs (except PBS) on an NFS share on my Synology NAS via PBS.
- Last but not least: document all the important configs and troubleshooting you do.
Note about PBS: why am I using it as backup target instead of backing up everything directly via PVE on the SATA disk? The answer is simple: PBS is another great piece of open-source software that allows for deduplication, compression and incremental backups (some interesting technical info here). Running it as an LXC container was the best option for my homelab, no additional hardware required and no fixed resources allocated to it when it is not performing backups. And furthermore, easy backups.
Disaster Recovery Strategy
You always hope that everything will work smoothly forever, but you know that sooner or later something bad can happen, due to an hardware failure, a software misconfiguration, an human error. So, it is mandatory to think about backing up data but also about what to do in case of disaster.
In case of problems with one of the CTs or VMs (already experienced) recovering them from a PBS backup is simply a matter of seconds via GUI or CLI.
Let’s consider now the worst case scenario, where my NVMe disk breaks and PVE host environment with all the VMs and CTs are lost:
- Buy a new NVMe disk (quite obvious :) )
- Install PVE from scratch by looking at my personal docs and using the PVE backup on the NFS share.
- Restore PBS CT from backup on NFS share and configure PBS in PVE
- Restore all the VMs and CTs via PBS from the most recent backup on the SATA disk, if available, otherwise from the NFS share
- Enjoy the HomeLab again
PVE-Bulk tool
Proxmox VE has a very good GUI and CLI interface that allows you to to everything in a quite easy way, and also the Administration PDF book is very detailed and informative, a must read if you want to use PVE at best. BTW, I’ve found also a good tool that makes it easier to perform startup/shutdown/snapshot/restore of a set of VMs and CTs from CLI.
View ponceto/pve-bulk on GitHub.
Give it a try, I’ve made some small contributions to the code too and I think it is a great tool.
Monitoring PVE/PBS
I think that it is always good to monitor your home devices, to spot issues and malfunctioning hardware/software in a timely fashion. PVE and PBS allow you to specify an InfluxDB server to which they can send metrics and I’ve configured them to send data to the InfluxDB2 server running on a docker container on my Synology NAS.
Since I’m never satisfied with the default stuff, I wanted to have more data about CPU/Disks’ temperatures and health of my NVMe and SATA disks, and so I’ve built two Python scripts that upload data to InfluxDB2 and run every minute via cronjob schedule. You can find them here, but remember to customize them for your own environment since sensors may vary and the scripts are shown to help you and are not meant to work out-of-the-box on every hardware.
View MightySlaytanic/pve-monitoring on GitHub.
You can find the full-size image here.
Home Network Design
You can see the overall design of my home network in the following image.
You can find the full-size picture here.
Some notes:
- GW VM is used as an entrypoint to reach K8S and CEPHs VMs without exposing them directly on my Home-LAN. I use SSH tunnels to reach those machines, defined as in the snipped below:
- GW VM is also masquerading connections from the VMs toward the Home-LAN and the Internet.
- JumpServer VM is used to reach TOR VM for management and monitoring purposes and allows some specific traffic via Source/Destination NAT from TOR VM to some services like the syslog collector running on the Synology NAS.
Conclusions
I hope you have found something interesting and maybe some new ideas for your own HomeLab… It took a very long time for me to buy an Intel NUC, because I didn’t want to spend too much money for my HomeLab, but now that I have it, I couldn’t live without… it’s so useful to experiment new things, have multiple running VMs and CTs, study for exams and have fun without melting down your own laptop and without spending too much in electricity, since Intel NUC is a powerful but also low-power device.
Enjoy your HomeLab!
