Above 4G Decoding + GRUB

Background

I recently decided to build a server with 6 1080 GPUs. I opted to use a GA-H110-D3A motherboard with PCI X16 to X1 reisers. The server is using Ubuntu server 16.04 LTS.

The problem

The 6th GPU never worked. I could see it in lspci but nvidia-smi listed only 5 cards. When I looked at memory mappings, I noticed the 6th card had undefined addresses. I assumed my motherboard had an issue mapping addresses for that card.

So, I tried turning on “Above 4G Decoding” in my motherboard, but instantly my computer stopped booting. After the BIOS post, I’d get a black screen.

The fix

I had to do a few things to finally get things working:

  • Disable all unused SATA ports
  • Disable on-board video
  • Disable on-board audio
  • Disable on-board USB which were unused
  • Disable on-board NIC (I am using USB wifi)
  • Set PCIe to “Gen 1”

After doing this, GRUB still never appears but after the timeout, the system finally boots. This is a case where I am not going to ask questions and be happy everything works(ish.)