Building a deep learning machine

I recently decided to stop using AWS and build my own machine for deep learning work (edit: I actually did this several months ago but only just got round to writing this up. There are more guides available now, but when I built it I ended up sending various parts back eg motherboards, GPUs because I didn’t know about various things eg PCIe lanes, GPU cooling. So hopefully this is still useful!)

I set myself a budget of $3k and wanted to get as close as possible to the NVIDIA DIGITS devbox for this. I read several useful posts about other people who had built DL machines and it mostly went fine, except for some issues about cooling and air circulation that I had to address.

Here are the parts I used in the end: https://pcpartpicker.com/user/andytwigg/saved/zcqrHx

Here are some articles I found useful:

https://medium.com/@vivek.yadav/deep-learning-machine-first-build-experience-d04abf198831#.t649py9a2
https://medium.com/@acrosson/building-a-deep-learning-box-d17d97e2905c#.u63bfo6c7
http://graphific.github.io/posts/building-a-deep-learning-dream-machine/
https://www.chillblast.com/chillblast-fusion-deep-learning-pc.html
https://linustechtips.com/main/topic/441555-cheatsheet-3-way-sli-with-5820k-and-6800k/

I decided to build a system with 3xGTX1080Ti GPUs, i7 5820k CPU, 64GB DDR4 RAM, 1x525GB SSD (for working files), and 3x 3 TB HDD in RAID-5 (for data). For another $820, I could have a 4xGPU system but this is a good enough setup to handle most things now.

Here are some useful things I learned that weren’t obvious at the start:

• CPU PCIe lanes — each GPU needs x8 PCIe lanes. You need a CPU with sufficient lanes. This leaves the i7 5820k (6 cores, 28 lanes) or 5930k (6 cores, 40 lanes). Since I will only be using 3 GPUs to stay within budget, I got the 5820k which is about $200 cheaper.
• Motherboard PCIe lane switching/allocation — you need to check how the motherboard allocates sufficient (x8) PCIe lanes depending on which processor is installed. For 4 GPUs, you need a 40 lane CPU and a motherboard that allocates the lanes x8/x8/x8/x8. For a 28 lane CPU, the MSI X99A SLI PLUS allocates lanes as follows: x16 (1 way), x16/x8 (2 way), x8/x8/x8 (3 way). You should also check that all 4 slots are double width, as some only have 3 double and 1 single width. Another good choice is the Gigabyte GA-X99P-SLI which supports 4 double-width slots.
• GPU cooling — I originally bought the EVGA GTX 1080 SC card, as I read that it had better cooling than the reference ‘founders edition’ card. Things are different when multiple cards are installed. These cards take up the entire width of the double slot so the fans cannot work effectively. You want positive air pressure (PAP) in the case, driven by the front intake fans, and then passing through the GPU and exhausted through the backplates. This means you want longitudinal cooling fins and a ‘blower’ cooler type on the GPU, which is what the founders edition/reference cards have. So lesson learned, stick with the reference board. Update: I replaced them with 3x GTX1080 TI — this is now the card to get.

Modifying the case

Here is NVIDIA’s documentation [http://www.azken.com/download/DIGITS_DEVBOX_DESIGN_GUIDE.pdf] for the details of how they constructed the devbox. It describes two things different from the stock Corsair 540 case:

1. custom GPU mounting bracket to stop the cards sagging
2. Better airflow management

“Our sample build used a Corsair Carbide Series Air 540 chassis with upgraded front fans. We also chose to close off unneeded vents in order to increase air pressure within the chassis.”

The mounting points for the 120mm fans allow cooling to the bottom GPU, which the stock 140mm fans cannot reach. I installed 3x120mm fans each providing 83CFM (Enermax UCMAA12A 83.0 CFM 120mm Fan). I also connected them directly to the PSU since their speed is usually linked to either CPU or MB temp, and I wanted to make sure the GPUs always had enough cooling.

I also sealed the top vents for higher positive air pressure. This makes the GPU blowers more effective. I ended up using these, which worked well: http://www.caselabs-store.com/140-2-280-cover-plate/

Part list

https://pcpartpicker.com/user/andytwigg/saved/zcqrHx
CPU: Intel Core i7–5820K 3.3GHz 6-Core Processor $378.50
CPU Cooler: Cooler Master Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler $29.93
Motherboard: MSI X99A SLI PLUS ATX LGA2011–3 Motherboard $179.99
Memory: Corsair Vengeance LPX 64GB (8 x 8GB) DDR4–2666 Memory $300
Storage: Crucial MX300 525GB 2.5" Solid State Drive $138.29
Storage: 3x Toshiba P300 3TB 3.5" 7200RPM Internal Hard Drive $89.99 (you want 3 of these in a RAID5 configuration)
GPU: 3x EVGA GeForce GTX 1080 TI: $699.00
Case: Corsair Air 540 ATX Mid Tower Case $126.29
Power Supply: Rosewill 1200W 80+ Gold Certified Fully-Modular ATX Power Supply $149.99
Case Fan: 3x Enermax UCMAA12A 83.0 CFM 120mm Fan $8.00

Total cost: $ 3693

Benchmarking and setup

Setup and benchmarking are covered pretty well in this post: https://medium.com/@acrosson/installing-nvidia-cuda-cudnn-tensorflow-and-keras-69bbf33dce8a