Above 4G Decoding + GRUB

Background

I recently decided to build a server with 6 1080 GPUs. I opted to use a GA-H110-D3A motherboard with PCI X16 to X1 reisers. The server is using Ubuntu server 16.04 LTS.

The problem

The 6th GPU never worked. I could see it in lspci but nvidia-smi listed only 5 cards. When I looked at memory mappings, I noticed the 6th card had undefined addresses. I assumed my motherboard had an issue mapping addresses for that card.

So, I tried turning on “Above 4G Decoding” in my motherboard, but instantly my computer stopped booting. After the BIOS post, I’d get a black screen.

The fix

I had to do a few things to finally get things working:

  • Disable all unused SATA ports
  • Disable on-board video
  • Disable on-board audio
  • Disable on-board USB which were unused
  • Disable on-board NIC (I am using USB wifi)
  • Set PCIe to “Gen 1”

After doing this, GRUB still never appears but after the timeout, the system finally boots. This is a case where I am not going to ask questions and be happy everything works(ish.)

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.