How to Build a Silent, Multi-GPU Water-Cooled Deep-Learning Rig for under $10k
So you kick off training of your latest and greatest deep-learning model, and the next thing you know it’s impossible to hear anyone talking because the various fans attached to your box start spinning at very high RPM. You start to get stares and snark from your co-workers. If you’re at home your significant other starts to wonder how much he/she really needs to tolerate before bringing up couples therapy. You think maybe it’s time to move all your training to the cloud, or better yet, you contemplate building a personal water-cooled machine that won’t completely break the bank or your relationships.
I’m going to focus on the latter, and take you through a recent build of a very powerful deep-learning rig that’s completely silent, and also reasonably compact in a “mid-tower case” (according to PC Part Picker). It’s perfect for the home office or under the desk at work. It can save you thousands of dollars on your deep-learning training bill, and your ear won’t be able to tell if it’s running under full load or sitting idle.
The first thing you should ask yourself is whether building your own water-cooled rig makes sense at all:
Should you just use the cloud?
You would think the cloud is the quietest way to train — and you’d be right — until your accounting department gets the bill and starts screaming and looking for you. For a full discussion on this topic, see my other post: Deep-Learning in the Cloud vs. On-Premises Machines. This will tell you why and when you want your own machines in your workflow, and how to augment it with the cloud to get the best of both worlds. The TLDR is that the case for managing your own on-premises machines is now better and easier than ever, and with containers & Horovod you can also scale to the cloud with little or no extra work.
Now there is something remarkable about letting a little, tiny $1000 GPU under your desk run non-stop for weeks, and having it spit out a (semi) state-of-the-art model. Somehow it’s very satisfying knowing you can replicate the same results from scratch that previously only a Google/Facebook/etc. would be able to produce.
But there is one big problem with the solution under your desk: the noise. A couple years ago I fell in love with the tiny design of Falcon Northwest’s Tiki system. It’s probably the smallest physical machine you can put the latest Nvidia 2080 Ti blower GPU card in. Unfortunately, Falcon NW machines are super expensive — so I bought a used one that was 3 years old and replaced the Power Supply (PSU), memory, and put in a 2080 Ti which I used to train a variety of CNN models from scratch. But the noise under load was terrible.
Partly this is the small form-factor design, but to keep temps below redline, I had to manually set the GPU fan to be about 75–80% power. At this RPM it generated a ton of noise, and the temps were pretty poor (low-to-mid 70s degrees C). The card starts to throttle itself at 83 degrees C — and heat is also a hardware killer. Running a card for weeks at a time in the 70s is not ideal. I’ve tried larger cases with better cooling and the situation is not much improved. GPU fans and/or case fans need to be on high RPM to get adequate airflow, and even then temps are still higher than desired, and the noise is annoying.
Now multiply this noise by the number of GPUs. With each card having nearly 250 watts of thermal output — and a single high RPM fan destroying the ambient quiet, it’s easy to see how putting an aircooled 4-GPU system nearby is like living next to a jet runway.
Now if you have a home office or machine room at your work with sound protection and decent cooling, then by all means get an aircooled machine and be done with it. If you have some dollars to burn, you can get a system integrator to do a water-cooled build for you. For example, Microway has a beautiful 4-GPU Titan watercooled rig that you can buy for $25k. Or they’ll do 2080 Ti’s for $17k. I saw these at the GTC conference and they were stunning. But damn they’re expensive. Either way, that’s way out of reach for individuals. Below I’ll show you how to build the same thing for under $10k. And because you’re building yourself, you can start smaller with 1–2 GPUs for about $4k-$5k and add as necessary.
Now on with the build….
A silent but deadly, multi-GPU, water-cooled deep-learning rig
Before I begin with the design and build log, I should pay tribute to a few folks who’s posts have been incredibly valuable in learning the art of system building. I should caveat that I’m not an expert builder by any means — in fact, prior to this project it was probably over 20 years since I last put a PC together from scratch. Many things have changed.
First off, I should mention two fantastic blog posts by Tim Dettmers: A Full Hardware Guide to Deep Learning, and also Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning. These posts have an incredible amount of useful information, and along with the Puget HPC Blogposts from Donald Kinghorn, are all you’ll need to know about where to invest your dollars to get the most bang-for-the-buck.
When it comes to water-cooling, you shouldn’t even think about building anything until you’ve read James Sunderland’s An ultimate beginners’ guide to PC water cooling, and also watched a bunch of videos from JayzTwoCents. Seriously, his videos are probably the single most valuable resource for learning the art and science of water-cooling, and allowed me to pull off this ambitious project with only a few tiny glitches. In particular, the video of Jay on Geforce Garage installing the waterblock on the 2080 Ti gave me the confidence to rip open nearly $5k worth of Nvidia Founder Edition 2080 Ti cards.
Phase 1: Air Cooling
Since I hadn’t built a PC in over 20 years, I decided to split this project into two main phases. In phase 1, I’d build the machine from its basic parts and air-cool it, and also do some basic benchmarking and burn-in tests before I potentially destroyed it with water. In phase 2, I’d water-cool it.
The Parts
Below I’ll do a review of each of the main system components, and the logic/thinking behind the choices I made. The bulk of the time I spent on this build was just doing the research on all the pieces/parts — so if you just follow the below you’ll know that it all works together and fits, and you can save yourself the research required for this design.
Motherboard: ASUS WS X299 SAGE — I wanted a motherboard from a mainstream manufacturer that could support 4-GPU cards with ease. What tipped me off to this board was that I kept seeing it over and over again from professional system integrators when I would troll the expo floors at conferences like GTC. Now one of the main benefits of this board is that it has something called a PLX switch, which allows the GPU cards to think they are using all x16 PCIe lanes, despite the processor only having 44 PCIe lanes available. This is what “SAGE” means in the product title, and beware there are similar WS boards that don’t have this feature. I’ve seen a few posts that claim that GPU cards can perform fine in x8 mode, but with 4 cards that’s still a lot more than the limited 16 PCI lanes available in the lower end Intel chips. So you need either an X-series chip (the 9800 series has 44 lanes) or a Xeon chip (some have 54+ lanes) to really handle 4 GPUs and other devices that also need PCI lanes like disk drives, network, etc. The X-series is sort of in between the lower-end Intel 1159-socket consumer/gamer chips, and the higher-end workstation Xeon chips. It’s a good compromise for this kind of build, and the motherboard can be had for about $500. There are also Xeon versions of this board, but getting the Xeon chip adds to the cost, and I think the benefits of Xeon are unnecessary for this kind of build (see memory discussion below). Note that Xeon and the X-Series both use the LGA 2066 socket — but they are not compatible chipsets. If the board is X299 you need an X-series like the 9820X. If it’s the Xeon chipset you need a Xeon processor. You get the idea.
Now this motherboard physical format is “CEB-SSI”. What the heck is that? PC Parts Picker reports an ‘incompatibility’ between this and nearly every case, so what’s the deal. Well don’t worry, CEB-SSI uses the same ATX holes, and is a little bigger than ATX, but smaller than E-ATX. It’s a quirk of Parts Picker. In my S2 case below it fits fine, although it does cut off about half the rubber pass-throughs but I can still get all the cables through. If you get an E-ATX case, you should be fine.
Lastly, I should mention there are two X299 SAGE versions — one has regular 1 Gbit ethernet, and the other has 10 Gbit ethernet. It’s about $100 cost difference — but I wasn’t planning to network multiple machines together to do multi-node training. If this is a consideration, maybe get the faster ethernet. Or you could get the cheaper version and put in a Mellanox card for crazy network speeds.
Processor: Intel 9820X X-Series Processor — this is a 10-core, 20-thread processor. Now as Tim Dettmers notes in his blog post, you typically need 1–2 cores per GPU to keep it fed, especially if you’re doing models than need some runtime data augmentations like CNNs/vision models. I can confirm training a Resnet benchmark model on single GPU used only 11% of this CPU. And 2 GPUs this was closer to 20% and 3 GPUs closer to 30%. So this processor has a few more cores than you really need to support 4 GPUs, but I found it’s a sweet spot in terms of price/cores/PCIs lanes/etc. You can buy it now (as of Nov 2019) for under $700.
Now some folks will point to AMD’s new Threadripper processors and note the large number (64) of PCI lanes. That might work fine, but I saw enough people complaining online of taking advantage of these, and having weird bottleneck issues on multi-GPU training on various Linux kernel versions, that I decided to avoid this and stick with a motherboard I saw the pro system integrators using over and over. And again, who needs all those cores anyway for deep-learning?
Memory: Corsair Vengeance LPX (DDR4 2666 Mhz). I put in 2 packs of 2x16 for 64 GB total. Now memory is one of the easiest places to overpay, and there are a million choices especially when considering gamer style memory with extra large heat sinks, RGB lighting, and crazy overclock speeds. And what I can tell you is none of this really matters. When you bottleneck on the GPU, it doesn’t really matter if your memory runs at 2666 mhz or uses a 4000 mhz overclock. So the baseline is go with a reputable manufacturer and check your motherboard manual/memory list to ensure compatibility. What will matter for water-cooling is having memory that is lower profile, to avoid hitting radiators/fans/etc. The Vengeance memory listed here is low-profile and was a non-issue making everything fit. It’s about $160 per 32 GB as of Nov 2019. Specifically, part number CMK32GX4M2A2666C16.
For curious folks, I’ll also say you don’t need ECC memory as supported by the higher end Xeon chipsets. Some folks who run mission critical infrastructure swear by ECC — and can’t tolerate even a few minutes of downtime. For our case, this is a training rig, and you’re check-pointing your model anyway right? So in case of an extremely rare memory error, you crash and can reboot. I’ve never seen this actually happen. So save some dough and get non-ECC memory required by the X299 chipset. That said, I did have one pack of memory that ended up being bad, so run the Windows Memory Diagnostic or MemTest86.
Now if you really care about memory benchmarks and think this is an over-simplification, I’ll leave you with two pieces of memory advice. JayzTwoCents has a bunch of videos on memory benchmarking — and there are two things you can do if you’re really concerned on memory performance (if you have a CPU bound application): 1) make sure you put memory in the right slots so you get multi-channel. So check your motherboard manual. The ASUS board above supports 4 channels. Most boards support at least two. But if you don’t use the right slots, you might only use 1 channel and can take a performance hit. Rule of thumb — fill every memory slot on your motherboard. 2) when it comes to memory speed, 3000–3100 mhz seems to be a sweet spot. In Jayz benchmark videos, he doesn’t show any benefit going faster, and this speed is now the same price as most slower speed DDR4 anyway.
Last note — although I put only 64 GB in initially, I’ll probably max out at 128 GB since memory keeps getting cheaper. I have seen that the Linux kernel does a good job caching training data from disk. This won’t matter for your own machine with fast local disk, but if you use AWS cloud instances which have shitty default EBS bandwidth for small EBS volumes, the extra memory will come in handy on an instance and you won’t have to over provision io ops (and pay a ton to do so) to get reasonable bandwidth to feed multi-gpus.
SSD Drive: Samsung 970 EVO Plus NVMe M.2 — this drive is nothing short of amazing. My first PC didn’t have a hard-drive. My 2nd PC had a 30 Megabyte HD (that’s right “MB”). So to put in a drive that 1) fits in a special M.2 gumstick sized slot on the motherboard 2) Reads and writes at over 3 GigaBytes (not bits) per second 3) Costs under $200 for a 1 TB drive. It’s the only product I’ve ever seen on Amazon that had a true 5 star rating with over 1000 reviews. It’s a complete no-brainer.
Power Supply (PSU): Corsair AX1600i (Titanium) — With potentially 4 GPUs at 250 Watts TDP each and a CPU at 150 Watts TDP, it was pretty clear I was going to need a large power supply. Now here’s the thing with power supplies: they are most efficient around 50% of max load, the fan noise ramps considerably above 70% load. So in theory, I might have gotten away with something slightly smaller power-wise, but I didn’t want to debug insufficient power issues, and wanted to maximize efficiency while minimizing noise. So this basically meant getting the biggest power supply I could. Most PSUs in this ballpark were in the ~$350 range — and this didn’t seem like the place to save $50–$100 going with a cheap-o solution. I had previously purchased a great PSU from EVGA — but a lot of the Amazon reviews complained about the noise in their 1600 Watt unit. I didn’t see that problem with the Corsair product reviews, so ended up going down that route. I’m very happy with it so far — it’s totally silent under load, and it comes with pretty much every single cable you need (see tips below though about a motherboard power extension).
CPU Air Cooler (for Phase 1): Noctua NH-U9S — Since phase 1 of this project was to get everything running under air cooling, I needed a temporary air cooler that I could easily install and easily remove later when time to do the water-block. I wanted something cheap, that wasn’t going to cause any issues on size and removal. This Noctua cooler got great reviews for both install and thermals, and was about $59. Speaking of coolers, one additional benefit of using a X-series processor is that it uses the LGA 2066 Intel CPU socket. One nice surprise of this socket is that coolers/waterblocks don’t need a backplate like the LGA 1159 socket does. This means the install can be quite a bit easier depending on your case and available access to the back of the motherboard.
If you haven’t built a PC before or installed a CPU and cooler this is probably the most intimidating step. Watch a few Youtube videos on installing an X299/2066 socket based CPU. It’s pretty hard to screw up. But anytime you’re handling a nearly ~$1000 part, it makes you sweat. One tip here is that the pressure you need to apply to close the bars/levers back up once installed is a lot more than you’d expect. It feels almost like you’re doing something wrong. Just double check your alignment arrows and notches and you’ll be fine.
The other intimidating thing here is applying thermal paste. Again — if you haven’t done this before it feels weird to apply random goop to a ~$1000 dollar part. Don’t worry — you’ll get used to thermal paste by the time you finish mounting the water-blocks on 4 GPUs, and it won’t feel weird anymore.
Now you’ll find how to apply paste is almost a religious debate amongst builders. The consensus I’ve seen is to use the ‘X’ method, and/or spread smoothly with a credit card. Other places just recommend the ‘dot’ method below. In this case, since it was only temporary and I wanted to make later removal and cleanup easy, I just used the dot method and the minimal amount.
As far as removing paste later — you need two things: isopropyl alcohol, and coffee filters. Coffee filters are remarkably useful for building machines (more on that later). Also — keep in mind this paste is non-conducting, so don’t have a freak out if god-forbid it falls somewhere it shouldn’t.
Wireless PCIe card: Asus PCE-AC55BT — you may not need a wireless card if you have hardwire ethernet near your machine. You may also run Windows, and pretty much any pluggable USB Wifi will work. Now if you’re doing this build for deep-learning, you probably want to run Linux. And if you plan to install Linux somewhere where you have Wifi only and no hardwire — then you need a card that’s going work out of the box. And most pluggable USBs and many PCIe cards don’t work out of the box — you need to download and compile the kernel drivers. Good luck with that if you have no internet. So let my pain be your gain — get the Asus PCE-AC55BT PCIe card for $35 and it will just work out of box with a stock Ubuntu 18.04.3 LTS install (this only works because my final build had 3 GPUs btw and not 4, so I still had a slot available). Note that this card also had Bluetooth if you connect an internal USB cable to the motherboard. I just ignored that and didn’t use the cable since I don’t need Bluetooth. One other good option is to get something like the IOGear Wifi-to-Ethernet adapter — this will fool your machine into thinking it’s really on Ethernet, and you’ll save a slot. In hindsight, I probably should have ordered this and saved a slot.
GPUs: 3x Nvidia Founder’s Edition 2080 Ti — the main computational engine for this build is 3 Nvidia Founder’s Edition 2080 Ti cards. I was originally planning on 4 GPUs and every other design choice supports using 4 GPUs except the case/radiators (see below). These were $1200 each.
Read all the blog posts mentioned above from Tim Dettmers and Donald Kinghorn, and you’ll learn that the 2080 Ti is a great card for machine learning work. It’s a sweet-spot in terms of price/performance. It’s not the most efficient from a cost per training/inference standpoint — but it’s basically the most powerful card you can put it other than the Titan series which is more than double the cost of the 2080 Ti — for somewhere between 0% and 30% boost for the slightly faster Titan cards. Most importantly, the 2080 Ti has Tensorcores, which means basically a 2x speedup on neural network training using mixed-precision/FP16 mode. Note the 2080 Ti does not do 64-bit floating point (FP64) well — so if you’re a HPC person and really need double precision, you need to think about the Titan V card.
Now you might ask, “Why get the Nvidia Founder’s Edition? Why not EVGA, Strix, FTW3, etc. etc. Why not the MSI Seahawk or Gigabyte card with the waterblocks already mounted?” There are a million variants of the 2080 Ti so let me give you a few thoughts.
When I first started this project, the thought of ripping open $5k worth of GPUs to manually install water-blocks was very intimidating. Once you do it one or twice you’ll no longer think this is a thing. That said, I thought very hard of getting the MSI Seahawk card with the pre-mounted block. But the reviews were so poor with all sorts of complaints I couldn’t pull the trigger. The other important factor was that I wanted to be able to mount 4 cards, with 2-slot spacing (i.e. in slots 1, 3,5,7 which is what the Asus motherboard required). To do this both easily and compactly, you want to use a multi-GPU block. Later I’ll talk about why I chose EKWB for all my water-cooling parts, but the baseline is that in the Spring of 2019 they introduced a new multi-GPU block called “EK-Scalar” that fixed a lot of issues with previous multi-GPU blocks. These work with their new ‘Vector’ GPU water-blocks, so that meant mounting my own EKWB water-blocks on the cards directly.
Now EKWB makes water-blocks for a variety of different cards. So why the Founder’s Edition cards? There were a few big reasons: 0) the reviews were great 1) on the EKWB compatibility list, this was the only card (at the time) that they themselves had physically tested/mounted. The other cards were marked compatible only by visual inspection. Minimizing risk to me meant going with the thing they had actually tested. 2) They were on the cheaper end of these cards — paying for the fancy gaming overclocking stuff doesn’t make sense when doing deep-learning under the Nvidia Linux driver which I’ll explain below 3) I talked to Donald Kinghorn at Puget systems who had great luck and spoke to the quality and reliability of the Founder’s Edition cards for really long-running workloads. 4) Nvidia GeForce garage released the above referenced video of JayzTwoCents doing a EK waterblock install on the Founder’s Edition 2080 Ti — that gave me a lot more confidence that I could do that too. 5) I wanted to maximize the chance of having minimal problems with the Linux Nvidia driver — so why not use the actual Nvidia card? So there you have it.
Lastly, I’ll mention as part of the exchange with Donald we discussed how the the Nvidia Linux driver (v. 430) tends to drive the clock speed up as long as the thermals are under roughly 80 degrees C. Using ‘nvidia-smi dmon’ command, you can monitor the clocks and temps. I’ll steal some thunder from below, but because the temps are so low with water-cooling, the card basically runs at full power (250 watts) at 1995 mhz (and occasionally over 2000 mhz). This is despite being marketed as 1635 mhz with a slight overclock. I’m not sure this is true under Windows. So baseline, I don’t think there is any reason to pay for a fancy gamer overclocked card when you’re water-cooling and using Linux for deep-learning workloads. But I’d be curious if anyone has actually benchmarked this. (JayzTwoCents — if you’re reading this, seems like it would be a great video!).
Now one last caveat is that in hindsight I should have considered the Asus Strix cards. They also have EKWB water-blocks, and I’ve heard rumors that if you have an Asus motherboard you can get GPU temps reported into the BIOS so you can set fan curves based on GPU temps and not CPU temps. I haven’t done this personally, so please comment if you have any experience with this. (And we’ll discuss later in the ‘fans’ section why this is nice to have).
Case: Fractal Design S2 — ah the case. Probably the most stressful decision point of this build and maybe any build. First off, there are just hundreds if not thousands of options. Choose wisely, and everything may just go smoothly. Choose poorly, and well everything else might just go to shit.
Case choice when water-cooling becomes even more critical, because you need to add radiators, and fans, and pumps, and hoses, and fittings, etc. etc. A good chunk, if not all your pain doing a water-cooled build will likely come from choosing the wrong case, where you just can’t squeeze all the things in that you need to.
Buying the largest case you can might seem like a reasonable strategy, but that doesn’t mean you won’t have to drill holes, remove drive plates, and do other annoying things, especially if it’s not designed for water-cooling.
I did a ton a research on cases, and I’ll get to the punchline:
The Fractal Design Define S2 case is a fabulous case for water-cooling with up to 3 GPUs.
But 3 is the max you’ll be able to get in there. I originally thought I might be able to get 4 using the vertical slot too, but that was just a total noob mistake.
I choose this case for a few reasons: 1) it’s part of the Fractal Design ‘Define’ series, which is really focused on quiet operation and has damping rubber, quiet fans, etc. Since the whole point here was to be totally silent, this was a good selling point 2) it’s basically the popular and well-reviewed R6 case, but revised with water cooling in mind: most of the disk drive slots were removed and there are extra mounting holes for pumps, reservoirs, etc. The documentation showed lots of options for mounting fans, radiators, etc. There is a convenient water-fill port and removable top. 3) It’s actually on the smaller end of tower cases, and PC Part Picker calls it a “Mid-Tower”. So it’s probably one of the smaller form-factors you could have that still allows water-cooling 3 GPUs. It’s about $200.
If there was any part of this build that involved luck, it was getting all the various pieces to fit in the S2. It wasn’t always obvious how to get everything to fit, but it actually did, I’ll cover that below so you can save a lot of time by following the design.
This case is nearly perfect except for two things: I wish it were 2 inches taller with 8 PCI slots and true E-ATX support, and the PSU shroud should be removable without drilling rivets. Fractal Design — if you’re reading this, please make a XL version of the S2 with these in mind.
And now that we’ve covered all the parts, here is the end of Act I: Air Cooling.
Not a lot a style points, but it’s all working. You’ll note that there is quite a bit of space left for the water cooling parts. I’m not an expert at cable management, but the case makes it easy to do a reasonable job. Now I ordered my 3rd GPU well into phase 2, so I initially only air-cooled with 2 GPUs.
The parts total for all the pieces described above as of Nov 2019: with 3 GPUs: $5,953, (and if you did 4 GPUs it would be $7,153 but you’d need to pay a little more for a larger case). For comparison, running a p3x8xlarge in AWS (4-GPUs) for 4-weeks: $12/hr * 4 weeks * 7 days * 24 hours = $8,064 (not including data storage). Now the AWS machines are about 20–30% faster than the 2080 Ti — so you need to take that into account. But still, you can see how quickly your cloud bill adds up.
Here is the basic part list on PCPartPicker.com assuming 3-GPUs. Note they flag that the motherboard is not compatible with the Define S2 case, but that is a quirk and not correct.
Benchmarks:
There were a few benchmarking/burn-in tools I used that were really useful to test all the pieces and make sure everything worked fine: Prime95 (CPU), OCCT (CPU/GPU), Aida64 (mostly for monitoring), and Furmark (GPU).
Here’s what I found: The Noctua CPU cooler did a great job, and CPU temps were fine except under the ‘small FFT’ Prime95 test. That test redlined the CPU at 95 degrees C immediately as it’s the most extreme load you can put on a CPU, but all other were tests were fine with temps in 60s or 70s. So for most of my normal CPU tasks, there wouldn’t be any issue.
The GPUs, however, struggled to keep temps in the 70s under Furmark even with the GPU fans and case fans spun to 80+% (which was extremely loud). Within a few minutes they throttled themselves. Of course, there are better cases than the S2 if you’re planning to air-cool, but it was a good sanity check that a big case, with big fast, loud fans, still couldn’t keep up with the 250 watts output by each GPU. This would be concerning if the ultimate plan wasn’t water-cooling, but really the goal here was just to make sure all the parts were working and put together before we added the complexity of liquid.
Quick snag: there was really only one snag that I hit during phase 1. Turns out that the motherboard wants two auxillary extra 8-pin power cables plugged into the PSU in addition to the main 24-pin large cable. These are marked ‘CPU’ on the Corsair cables. Well they didn’t fit because the Corsair fitting was too big and hit the nearby large heatsink on the Asus X299 Motherboard. I had to order a couple 8-pin extension power cables with thinner fittings to solve the problem. Note these are different 8-pin cables than the PCIe cables.
Phase 2: Water-Cooling
Now that everything was working, it was time to take the GPUs out and remove the CPU cooler to prepare for water-cooling. I got out the big box of parts that I had ordered from EKWB a few weeks earlier, and started to think about where I was going to put everything.
Now I chose EKWB for a few reasons. 1) they are well known in the water-cooling community as a top manufacturer 2) they have excellent guides and product documentation to help the complete newbie 3) Alex Frangis told me about them — and for anyone that knows Alex, you just do what he says because he pretty much knows everything.
I’ve heard of other manufacturers too like Bitspower and AlphaCool. In theory all these folks use a lot of the same sizes for everything (e.g. G 1/4" thread), and should be relatively intercompatible. But everything has a tolerance, and I made a conscious choice to get all my parts from a single manufacturer to minimize risk of leaks and potentially other things not working or fitting together. Maybe this was overly conservative and I overpaid a bit as EKWB isn’t cheap— but since this was my first water-cooled build I wanted to optimize for success and not save every possible dollar.
The primary order I placed directly through the EKWB website and online store. It took a little longer to arrive since it came from Slovenia, but most parts were cheaper than the reseller equivalents on Amazon (e.g. TitanRig, Performance PC, etc.).
One of the places you can waste a lot of money is paying for fittings, connectors, etc. that you don’t need, and it’s hard to know in advance what pieces exactly you will need until you really start placing things. But if you follow the below design, I’ll just tell you the pieces I used that worked, and you can avoid buying pieces you don’t need.
Here was my parts list:
EK-CoolStream SE 360 (Slim Triple) — for the front radiator: $64.74
EK-CoolStream PE 360 (Triple) — for the top radiator: $74.79
EK-XTOP Revo Dual D5 PWM Serial (double pump in single unit) — $233.74
3 of EK-Vector RTX 2080 Ti RGB — Nickel + Plexi — $136 x 3 = $408
3 of EK-Vector RTX Backplate — Nickel — $42.49 x 3= $127.47
EK-Velocity RGB (Full Nickel) — $93.49
EK-ACF Fitting 12/16 — Nickel — 16 pieces x $5.94 = $95.04
EK-Vardar EVO 120ER White BB Fans — 6 fans x $16.99 = $101.94
EK-Scalar Triple 2-slot — Plexi = $49.99
EK-RES X3 150 Lite (Reservoir that screws into pump) — $45.04
EK-Tube ZMT Matte Black 16.1/11.1 mm (3m) — $16.14
EK-AF FillPort G1/4 — Nickel — $8.49
EK-AF Angled 45 degrees G 1/4– 8 pieces @ $6.79= $54.32
EK-AF Extender 30mm M-F G1/4 — Nickel — 3 pieces @ $5.09 = $15.27
EK-AF Angled 90 degree G1/4 Nickel — 8 @ $5.94 = $47.52
EK-AF T-Splitter 3F G1/4 — Nickel — $8.99
EK-AF Ball Valve (10mm) G1/4 — Nickel — $16.99
EK-AF Extender Rotary M-M G1/4 — Nickel — 4 @ $5.99 = $23.96
EK-AF Extender Rotary M-F G1/4 — Nickel — 2 @ $5.99 = $11.98
EK-ATX Bridging Plug — $2.12
Filling Bottle — $6.80
EK-CryoFuel Navy Blue (Premix 1000 mL) — $14.44
EK-Cable Y-Splitter 2-Fan PWM — $2.96
(I also ordered a couple of 3-way PWM splitters for the fans, and a bunch of PWM extension cables from a local store for about $20. I didn’t like the EK 4-way splitters because they required separate power which was overkill and added an unnecessary cable). I also left off about $100 worth of fittings that I didn’t end up needing. Note I didn’t use all of these pieces, but the rotary extenders, 30 mm extenders, 45s, and 90s are worth their weight in gold so there are a couple more than you really need above but I’d get extra to be safe, especially if you’re trying this in a different case. Same for compression fittings.
Total: $1,474 — (water-cooling is not a cheap endeavor)
Remember the system parts total including the GPUs: $5,953 for 3-GPUs and 4 GPUs all-in would be $7,153. Adding about $1500 for the water-cooling pieces:
3 GPU water-cool total: $5,953 + $1,474 = $7,427
4 GPU water-cool total: $7,153 +$1,474 + ($200 for extra block/back) = $8,827
There you have it. And most of these items were ordered on an EKWB holiday sale with about 10% off the total. Beware of shipping and foreign transaction fees on your credit card as they were shipped from Slovenia. Compare against the resellers on Amazon if you have Prime. Also, as I mentioned above, I got a quote from the system integrator Microway for a 4-GPU 2080 Ti build and they quoted me about $17k without tax. So my rough estimate is that by doing this yourself you’re saving about $8k. For a new builder, I’d budget about 100 hours of time for this project. If you are an experienced builder or have done water-cooling before I think it could be done in 40–50 hrs. Plan on 1–3 days of work for the phase 1 build, benchmark, and OS/driver install. And then 3–4 days all-in for the phase 2 water-cooling part. If you’re a first timer, I think it’s about 2 weeks of full-time work including research time reading blogs and watching how-to videos.
A couple thoughts on the parts:
Radiators — as noted on many water-cooling blogs, it’s actually the length that matters here, not width/thickness. You always want to maximize the length if you can (e.g. 360 mm rads) before worrying about thickness. The rule of thumb is you need about 240 mm length of radiator to cool a GPU. This design has two 360 mm radiators, one for the top (the PE) and one for the front (the slim). When I get to temps later, you’ll see how this is fine for 3 GPUs. Remember that in deep-learning, you may only run 1–2 cores for each GPU, so your CPU on a typical load will likely add <100 watts, so is negligible compared to the thermal output of the GPUs (750 watts for 3, and 1000 watts for 4 GPUs). We’ll talk more about this below in the fans section. But I think it’s worth noting that because of the limited role of the CPU in deep-learning, it makes a lot more sense to run a single loop on a dual D5 pump so you get all radiators on the same loop than two separate loops (i.e. one for CPU and one for GPUs) as a lot of the gamers do. You don’t want to waste precious radiator on a CPU that could likely be handled by a single 120 mm rad anyway.
I thought about ordering 2 PE radiators, but ended up getting one PE and one Slim edition as a spacing risk mitigator, and I’m glad I did. I definitely needed the extra space up front to fit the pump and fans in. The EK Slim 360 isn’t the most efficient of the slim radiators in some thermal benchmarks I’ve seen here, and in the excellent Radiator Roundup here. In hindsight, I probably should have used the HardwareLabs GTS slim rad up front, but more on that below.
Pump — after chatting with EK technical support, I decided to get the Dual D5 pump (yes it’s two pumps attached together). The D5 pumps are 1) much quieter than the higher head pressure DDC pump near full RPM 2) have 30–50% higher flow-rate, but less pressure than the DDC pumps. With many different parts to flow through though, I needed high pressure and good flow, especially since the GPU block from EK is a parallel block (instead of serial, so the flow is reduced). And of course I wanted total silence: the solution was a dual D5 at the expense of space. Now when this came in I was worried it wouldn’t fit — but I got it in with a clever hack that I’ll show you below. The other benefit of a dual pump, is that if one fails there will still be some flow from the other pump, which will hopefully allow enough time to take note and shutdown before anything burns out.
Reservoir — you don’t need a big one. It takes slightly longer to fill a loop with a smaller reservoir, but the space savings is worth it. My reservoir screws right into the pump to save even more space and eliminate an unnecessary tube.
Tube — this was my first water build and I wanted to minimize complexity, so I went with the black ZMT tube (“Zero-Maintenance-Tube” — sounds good to me!). Of course, now that I’m done I want to think about the beautiful custom hard tubing, but I’m convinced this was the right call as a newbie. Aesthetically, I actually think the black soft tube looks better than transparent soft tube. This is because with black your eye is drawn more towards the liquid in the reservoir, the multi-GPU block, and the nickel fittings. I think the transparent soft tube would highlight the fact that it’s soft but without the sexy angles you get with the hard tubing— but that’s just my newbie opinion.
Attaching the Water-Blocks
Attaching the water-blocks is arguably the most intimidating step as you’re ripping apart thousands of dollars worth of GPUs. But it’s a well-trodden path and turns out that it’s tedious, but not that hard. The first block took me 4 hrs, but the 2nd and 3rd blocks took about 1.5–2 hrs each. The EK documentation is ok — but the GeForce Garage channel video of JayzTwoCents doing this is much better instruction.
A couple tricky bits to call out: 1) don’t forget the 3 larger screws on the side of the block PCIe mount 2) when separating the fan/cooler from the PCB, realize that the black PCIe mount is not part of the cooler, so holding that and wiggling the cooler can help detach everything 3) don’t forget to detach the small rainbow ribbon cable connecting the PCB and fan/cooler 4) Jay’s tip to place the PCB on the water-block as opposed to EK documention is really helpful 5) remember your plastic washers and to use a ‘star pattern’ to tighten the 4 inner screws after placement. 6) really pay attention to his ‘chilling’ comment about getting the RGB cable out of the way — easy to get this in an awkward position if you’re not careful.
Pump Fitting Hack
Probably the trickiest thing for this water cooling build was figuring out what to do with the pump. Of course, it all goes back to the case choice. When the Dual D5 came in, I was originally worried it wouldn’t fit and that I screwed up. But here is a little hack that made it all work.
There were a couple problems initially — one is that the annoying PSU shroud is riveted in, and would need to be drilled out to completely remove it. There is a small piece in the front that’s removable so you can get radiators and fans in there — but if you put the pump on the bottom of the case the outlet port is hard to reach with the PSU shroud in place. Because of the huge PSU, the cables start to bunch up in there too, so there is not much room to work. Similarly, if you put the pump parallel to the depth axis of the case there really isn’t a lot of room to get the front fans/radiator in. Hmm.
So here was the trick: the S2 case comes with 3 140 mm 3-pin fans out of the box. You have to remove two of these in the front anyway to put your radiator in there with new quiet 120mm EK Vardar fans. Luckily the pump comes with a mount to attach it directly to fans. So I just repurposed the removed S2 case fans, doubled them up and put the pump right on top perpendicular to the depth axis of the case as shown in the picture below.
Well it barely fit, but it did fit — and conveniently, because of the height of the fans, the outflow port of the pump was now above the PSU shroud, so nothing needed to be drilled out, and it was easier to get a drain value in there too. The fans also were probably a good idea anyway, to get additional airflow and maintain positive case pressure (more on fans below). There is another YouTuber called Optimum Tech who has some great benchmarks on adding intake fans on the bottom of cases, and this is consistent with all the fan videos JayzTwoCents has done as well. Overall, everything fit and was clean/compact and almost looked like it belonged there.
One last bit on the pump: none of the instructions had any notes of what to do with the cables. This dual D5 came with 2 four-pin fan-header like connectors, along with two other old-school power-looking connectors (turns out these are accessory or ‘Molex’ connectors). Maybe this was totally obvious to everyone else, but I had to scratch my head a bit. So here’s the deal: The two motors are powered off the Molex connectors, and my Corsair PSU had an accessory cable that accepted these. The four pin connectors are PWM and attach to a motherboard fan header, but they don’t draw power from that. You can look at the wires and see they only accept the PWM control and report the RPM back.
I read online somewhere if you don’t use the PWM connectors then the pumps run at 65% power, and someone else reported 100% power. So I don’t really know. EKWB? In my case, I used a 2-way, Y-splitter PWM cable, and attached both motors 4-pin connectors to the W_PUMP header on the motherboard. If you look closely, the RPM value is being reported back only from one of the pumps, but that’s fine since they’re driven by the same power setting. Now by default, the W_PUMP header on the motherboard is not adjustable like fans are in the BIOS/UEFI — but there was another setting in another menu that made it adjustable like a fan. So I just set it to about 70% power always regardless of CPU temp, unless CPU temp got into the mid-60s, where I ramp to 100%. Again, this is a weird thing to have to set fans/pumps based on CPU temps, when you really want GPU temps (but the BIOS doesn’t know about that).
I picked this value for a few reasons. I’ve seen JayzTwoCents videos that shows very little temperature difference (~5 degrees C) depending on flow rates. But because of all the pieces and parallel GPU block, I wanted to make sure I had decent flow. I did a quick sanity test and noticed about 4 degree C difference on the GPU between 100% and 50% pump power settings. At 70%, I couldn’t hear it at all in the case and the temps were great with 2-GPUs. On a lot of mechanical systems I’ve worked on before, 65–70% of max power often ends up being a ‘cruise’ setting on motors/engines where you can just leave things running for a long time. So it seemed like a reasonable place to start. (Update: when I added a 3rd GPU using the parallel EKWB multi-block, I up’d this to 85% to ensure adequate flow. You can still can barely hear it. )
One final tip is to make sure you keep the pump Molex power connectors easily accessible. You’ll want to use a 2nd cheap-o power supply with a loopback bridging plug to fill and leak test the system without plugging in or getting your main PSU involved.
Radiator Fitting Hack
The second hardest part of this build is figuring out how to connect the radiators. Read the manual for the S2, and it just looks so easy, the top can take a 420 mm and the front can get a 360mm — so it should be easy to just get two 360 mm rads in there right? Well getting the radiators mounted in there with fans is one thing, actually attaching the tubes and fittings is another.
So the trick here was using a 90 degree, a 30mm extender, and a 45 degree fitting to route out of the tight spot. This output of the radiator then goes back into the reservoir. These are the three fittings to consider getting more than you actually need, because they will save your ass when you get in a tight spot (literally). For connection in the back of the radiator, I just used a 45 degree fitting out of the CPU outlet, to route directly into the back port of the radiator. Place the top radiator last so you can get everything else hooked up first. You can also see below the tube connecting the case fill port into one the 3 openings of the reservoir. Don’t forget the vortex foam in the bottom of the reservoir as specified in the EK manual.
Loop Config:
If you’ve read anything about water-cooling or watched the benchmark videos above you’ll realize the loop order doesn’t matter, other than having the reservoir feed the pump directly. In our case, the reservoir actually screws right into the pump, so we don’t need to worry about that. The order is mostly driven by convenience of hooking up the tubes.
The Order: Reservoir screwed into pump -> T-fitting+drain -> GPU block ->top EK-PE radiator (the back connector)-> (front connector of top radiator) to CPU block (intake) -> output port of CPU block to Slim Front radiator (back connector)-> (front connector of radiator) back to reservoir. There is also a separate fill tube from the top fill-port into the reservoir.
Leak Test
Yes, you really need to do this. Some folks recommend at least 24 hours. JayzTwoCents says if you have a leak you’ll probably find it in the first hour, so you don’t need that much time. Regardless, I’d spend at least a couple hours sitting next to it after your fill to leak test and take corrective action immediately if something isn’t right. In my case, I found my first and only leak after 30 minutes. A few drips appeared because the fitting between the reservoir and tube connecting the front radiator had gotten loose. Replacing that solved the issue and luckily have not seen any other drips.
Get the small filling bottle, a funnel, and extra PSU with loopback plug — these items will make this step significantly easier. Remember to flush your radiators before installing fans and placing in the case to get any gunk out. I also did a fill, leak test, and flush/drain with distilled water before doing the final fill with EK CryoFuel. Leak testing with distilled water is nice because there is no dye to stain anything. FWIW, the fill took about 600–700 ml total (almost 3/4th of the CryoFuel bottle).
Fans
The fans were not something I thought much about until it came time to hook everything up. Figuring out exactly how and where to connect the various fans to the motherboard, direction of flow on the radiators, and how to tune the curves added a bit of unexpected complexity.
I keep referring back to JayzTwoCents videos, because the level of detail he covers is really incredible. There were videos for most of my fan questions.
At a high level, you want to maintain “positive” case pressure, meaning that the inflow to the case is higher than the outflow. This means that air will be forced out of various cracks and openings, taking both heat and dust with it. Now you can’t just go by the number of inflow fans vs. outflow fans, each fan is different and RPM of each matters a ton, as well as if it’s a flow optimized or static pressure optimized fan.
When attaching to a radiator, you have to decide whether you are going to ‘pull’ air through the radiator, or ‘push’ air out across the radiator depending on how the fans are mounted to it. Some fans are easy to tell the direction of airflow as it’s marked on the fan casing. For others you need to spin the fan to make sure (like the EK Vardars. Don’t rely on the fan center brand label as different manufacturers do different things).
I had three, 120mm fans attached to each radiator and I decided to make the front radiators a ‘pull’ config, pulling fresh outside air into the case across the slim radiator. I made the the top PE radiator a push config, pushing air through the top radiator and out the top of the case. The later made more sense to me, as we all know heat rises, so that seemed reasonable. It was a bit counter-intuitive to me that the front fans/radiator should be a pull config — pulling air across the radiator heats the air up, and now that heat is inside your case, making the job of top fans/radiator harder. Of course, there is a JayzTwoCents video on this exact thing, and the data don’t lie. Intuitively, I think you need to realize you’re moving a lot of air, and you’re not heating it all up all at once, so just making sure you get nice cool outside air in is the most important thing.
The bottom two repurposed S2 case fans also help in that regard, and I have them ‘pull’ additional outside cool air from the bottom into the case. The pump does generate heat too, so this is a side benefit to get some extra flow there as well.
The last confusing thing was thinking about 3-pin analog connectors (i.e. the three built-in S2 case fans that come pre-installed with the S2 case) vs. the 4-pin digital PWM connectors. My motherboard uses 4-pin PWM as do the 6 Vardar radiator fans I bought. Now the S2 case also comes with a fan hub, that accepts both 3-pin and 4-pin fans (there is a picture of this much earlier in article when discussing the case choice).
So here’s what I did: I used three PWM extender cables to attach the 3 front radiator fans to the S2 case fan hub. I also connected the two stacked 3-pin fans under the pump also to this hub as well. I then connected the fan hub to the CPU_FAN header on the motherboard. At bare minimum, something needs to connect to the CPU_FAN header to tell the motherboard there is something cooling the CPU, otherwise your motherboard will shutdown as a safety measure. I could have split these out, but the case fan hub solved the 3 & 4 pin problem, and also I found it easier to grok having all my intake fans under a single power setting. Remember each fan header on the motherboard usually only supports 1 amp total — so this only works with this many fans because the fan hub has external power from the PSU (in general you can get 3–4 fans on one header — but check yours to confirm).
For the 3 top radiator fans, I used a 3-way PWM splitter, and connected this to the CPU_OPT header. This header is not configurable/tunable in the BIOS, it’s designed to run at the same RPM as CPU_FAN in case your CPU air cooler has both a front and back fan. The benefit here is that since the radiator fans were the same, and there were two more intake fans on the bottom, and the slim radiator up front, I had very high probability of maintaining positive pressure in the case based on a single CPU_FAN header setting (ignoring for a minute the back exhaust case fan).
For the back fan, I ended up removing the 140 mm 3-pin S2 fan, and just put in a quiet Corsair 140mm 4-pin PWM fan. This allowed me to plug it right into another motherboard PWM header. The obvious choice here were CHA_FAN2 or AIO_PUMP as they are right next to the rear fan. (If you choose AIO_PUMP, you need to manually enable that as configurable in the BIOS, similar to W_PUMP as above to make it a controllable/tunable fan header).
As for setting the fan curves, I experimented with a few settings. The whole point here was noise elimination, and in the S2 case I found I couldn’t hear anything if the fans were set at around 1000 RPM or below. I didn’t want to get fancy, so as an initial setting just left the intake CPU_FAN curve at 1000 RPM all the time until CPU temps were in the 60s, then just ramped them up to 100% linearly. For the rear outflow fan, I just followed pretty much the same curve, but kept them about 5–10% less power than the intake fans to ensure that case pressure remained positive. It wasn’t the most scientific, but it seems to work fine. You can sanity check pressure using some candle smoke or burning incense sticks if you want to. In the below discussion on fans, you’ll be surprised (as I was) on the RPM impact on temps.
The final thing I’ll say about fans is that if you have a gonzo large case with extra room, you can use a “push-pull” config, and put fans on both sides of the radiator. The net result is you can increase your thermal reduction 20–30% without increasing RPM (And since RPM affects noise super-linearly, (e.g. going from 1000 to 2000 RPM has much more than doubling effect on the perception of ‘loudness’), you can trade space to get more thermal benefits whiling keeping noise and flow restriction to a minimum. In this setup, it doesn’t matter, as we’ll see below.
Benchmark and Temperatures
Well the moment of truth is here — what were the temperatures with all these GPUs running under load? Were the fans and pumps set correctly? What about under a real deep-learning load? Well there were a bunch of surprises here.
At first I booted into Windows (I had installed Windows and Ubuntu 18.04.3 side-by-side) and ran the same benchmark apps I ran before. On Prime95, I saw CPU temps in the 50s, and even with the most extreme “Small FFT” test, I saw temps only climb into the 60s. So much better than the air-cooler.
More importantly, using FurMark the GPU temperature maxed out after 20 minutes in the mid 40s, nearly 35 degrees C cooler than when I tried this with air cooling! Better yet, it was totally silent with all the fans running unnoticeably at 1000 RPM and the pump at 70%. Now with FurMark and a single monitor, I could only test one GPU at a time, so I did the multi-GPU test in Linux under a real deep-learning load.
But before I switched to Linux, I did one last test in Windows, with both Prime95 running the blend test, with the GPU running FurMark at the same time. Surprisingly, the respective temps were exactly the same — I didn’t see any impact of running CPU and GPU full steam at the same time.
Getting over to Linux, I used Horovod and NGC containers to spin up everyone’s favorite ResNet50 deep-learning CNN benchmark. I used “nvidia-smi dmon” command to monitor power and temperature in real-time. So here was the cool thing (pun intended) — running with 2 GPUs, each was running at 250 watts, the clocks were automatically driven up to 1995 mhz automatically (that’s just how the Nvidia Linux driver works), the memory clock was maxed out at 6800 mhz, and all the temps were stable in the high 40s! With only a single GPU, the temperature was in mid-40s, so I was surprised that with two going it barely raised the temperature of each.
Now here’s the weird thing — as an experiment I raised the fan speeds up to a very loud 2000 RPM to see the difference on the temperatures — there weren’t any! The temps were still in the high 40s with both GPUs going and there was no noticeable temperature difference between 1000 and 2000 RPM other than the noise. (Head scratching). I wish I knew more about thermodynamics to explain this, but my only theory is that there must be still considerable thermal headroom, and the fans even at the slower 1000 RPM are able to remove heat from the 720 mm across both radiators faster than the liquid can move it into the radiator. If you read this very detailed radiator benchmark, this does seem to make sense. Looking up my specific radiators, it appears I have about 500 watts of cooling around 1300 RPM, with only a 10 degree difference between coolant temperature and ambient temperature outside the radiators. So for 2-GPUs, 720mm of radiator is sufficient to keep temps in the 40s.
But what about 3- GPUs? Given these benchmarks, it seems like 500 watts might not be enough for 3 GPUs at 250 watts each and also the rest of the system components (PC Part Picker estimates the total system need at 913 watts). We’re probably a little less than that given the 10 core CPU only needs about 30% for 3-GPUs, but still… The thing to keep in mind is that these thermal estimations are based on “10-delta-T” measurement method, meaning a 10 degree difference in ambient temp and coolant temp. Given the extra heat we’re putting into the system, you would expect at least another 5–10 degree temperature rise in the coolant to dissipate this heat, and therefore would also expect a rise in GPU temps.
And that’s exactly what I saw. After running the resnet benchmark for an hour with 3-GPUs maxed out, temperatures stabilized in the high-50s, with the hottest card (the 3rd GPU) reaching a max at one point of 63 degrees C. This is about 10 degrees C hotter than many water-cooling folks might like, but we’re still 20–25 degrees C below the redline, and the Nvidia driver still reports the cards running at 1950+ mhz, which is a much higher clock than advertised for these cards. I did tweak the fans and pump a bit — which seemed like it was worth a few degrees. Given the parallel multi-GPU block from EKWB, you do need to think about flow — so I raised the pump to 85% and the fans to a barely noticable 1300 RPM (and rear to 1000 RPM). Overall, I’m happy with the temperatures. It sounds ‘off’ with 2-GPUs at 70% pump and 1000 RPM fans, and barely noticable with 3-GPUs at 85% pump and 1300 RPM fans.
And speaking of silence, that really is all the reward here. It was extremely fun to kick off a training benchmark, see the clocks, temps, and watts all ramp up — and hear absolutely no change and barely any noise whatsoever from the machine that is now generating nearly 900 watts of heat under full load! For anyone sitting nearby, it’s as good as off.
Tips and Key Learnings:
Here are a few final tips and key learnings:
- The 45 and 90 degree fittings are the most valuable. Get extra, as well as some 30 mm Male-Female extenders and a couple of the rotary fittings M-M/M-F.
- If you’re planning on using this design to do 3-GPUs, just do all three at once and get the 3 GPU multi-block from the start. I originally did 2 as a risk mitigator — but this ended up creating a lot more work and risk to have to drain and cut and re-do tubes and the multi-block, and refill to get the third GPU in.
- Don’t forget to flush radiators with distilled water after you take them out of the box (and before you put the fans on). But definitely install the fans on the radiators before you install the radiators in the case. Double check directions for push vs. pull.
- Coffee filters are great to clean up old thermal paste with isopropyl alcohol. They are also great to help tighten down compression fittings when your hands get sweaty. Rubber lid openers can help too. Fitting tubes still felt very tedious and I don’t think I found a better way than just installing the fittings first, and then cutting/adding the tube, and then tightening the compression fittings after it was installed. The rotary pieces can help a little bit in some places — but I still feel like I need advice in this area. Removing tubes meant cutting and replacing — I don’t know how to reuse tube here with compression fittings already installed.
- Get yourself a $10 tube/hose cutter like this one from XSPC. Also a funnel that will fit in the fill port in the top of the case (remember to open the 3rd port opening in the reservoir before using the funnel, or it will all spill out — air needs to go somewhere!).
- Get yourself a cheap-o power supply that supports Molex connectors so you can leak-test without your main PSU. A loopback connector is necessary and can be bought for $2.
- Since this build is using the 2066 socket for the CPU, don’t forget to open up your Velocity CPU water-block and switch the “jet” metal flow thing to the 2066 version. Only takes 5 minutes and the EKWB instructions are good.
Finale
Future Work
Although this build came out successful, it fell short of my original goal of getting 4 water-cooled GPUs in there. I’d love to try again with a larger case. That said, I do think the S2 case is fantastic for 3 GPUs, and I think it would be very difficult to do in a more compact form factor, since everything barely fit. Given what I know now, the only case I know about that seems great for 4 GPUs is the recently released Phanteks Enthoo 719 (a.k.a Luxe 2). Some other older options might be the Obsidian 750D or Cosmos from Corsair — although while larger they both lack some of the nice features of the Fractal Design S2 that made it such a pleasure to work with. I hope Fractal makes a water-cooled optimized XL version of the S2 — basically just a little bigger with 8-slots. Like their Define XL — but with the same water-cooling changes they made for the S2. If you’ve successfully water-cooled 4-GPUs, what case did you use? Given the recent release of the Enthoo 719 — that would be my next choice if I were starting from scratch.
The other thing I would dig more into is using the ASUS Strix version of 2080 Ti. I don’t know if the rumors are true — but the possibility of the Strix cards getting GPU temps into the ASUS motherboard BIOSs— so fans curves can be set in BIOS off GPU and not CPU (without needing a Windows application like SpeedFan or Fan XPert), that would be worth figuring out. There are EKWB water-blocks available for the Strix, so the process should be the same.
I also don’t know if I could have fit a 420 mm radiator on top, with a 280 mm in the front. That may not have worked at all — but I’d be curious if it would be any easier without all the creative plumbing/routing of fittings, and only a touch less overall surface area. The other obvious radiator change would be to replace the front SE radiator with the best-in-slim-class Hardware Labs GTS 360 to get another 50–100 watts of cooling for the same form-factor, as shown in these XtremeRigs Radiator benchmarks. The EKWB SE slim is small for a 360mm, but you can see from the charts is pretty much at the bottom in terms of thermal dissipation.
Given the jump in thermal performance between 2 and 3 GPUs, I don’t think these two radiators will suffice for a 4-GPU system unless you keep the unit in a temperature controlled machine room environment ~60 degrees F, or run the fans much higher near 1800+ RPM (which defeats the purpose of the silent water-cooling). Some folks have suggested setting the front radiator as an exhaust and using the back fan as an intake — maybe that’s worth a few degrees, but I doubt it would be enough. To do 4-GPUs, you’ll need a bigger can anyway, so going with the Enthoo 719 or similar, you’ll easily be able to use 420/480 rads and even a 3rd radiator which should fix the issue.
Last but not least, I want to try making it all sexier with some rigid transparent tubing and custom power cables like Jay’s beautiful Skunkworks build.
Good luck — and remember to use any knowledge here at your own risk! Mark Palatucci
