Above 4G Decoding + GRUB
I recently decided to build a server with 6 1080 GPUs. I opted to use a GA-H110-D3A motherboard with PCI X16 to X1 reisers. The server is using Ubuntu server 16.04 LTS.
The 6th GPU never worked. I could see it in lspci but nvidia-smi listed only 5 cards. When I looked at memory mappings, I noticed the 6th card had undefined addresses. I assumed my motherboard had an issue mapping addresses for that card.
So, I tried turning on “Above 4G Decoding” in my motherboard, but instantly my computer stopped booting. After the BIOS post, I’d get a black screen.
I had to do a few things to finally get things working:
- Disable all unused SATA ports
- Disable on-board video
- Disable on-board audio
- Disable on-board USB which were unused
- Disable on-board NIC (I am using USB wifi)
- Set PCIe to “Gen 1”
After doing this, GRUB still never appears but after the timeout, the system finally boots. This is a case where I am not going to ask questions and be happy everything works(ish.)