Setting up GTX-1070 Passthrough with ESXi

David Ramsay
Mar 18, 2018 · 5 min read

I’ve had to do this a sufficient number of times at this point that I figured I’d write a post about it.

Getting ESXi working isn’t too hard *if* you have the right hardware. For me (and for the LSI controller I bought to allow me to virtualize FreeNAS) it took a lot of trial and error. If I had it to do over again, I’d probably just buy a separate case/motherboard for myFreeNAS machine instead of virtualizing it.

V-Front.de is the best website to figure out all of these things; for me, I ended up setting up my BIOS (and the BIOS of my LSI card) to boot off of the LSI card, and ended up passing the onboard AHCI controller of my motherboard to my FreeNAS instance.

Setting up the Image

Image for post
Image for post
GEForce showing up as additional devices, ISO file in CD/DVD drive pointing to Xubuntu.

I called my machine ‘machine-learn-box’ and saved it in ‘datastore1’. After successfully creating the VM, we need to edit the *.vmx file for the instance. You can do this from the command line if you’ve enabled the shell/ssh on your server:

ssh root@192.168.1.252
cd /vmfs/volumes/datastore1/machine-learn-box
vim machine-learn-box.vmx

and add the line

hypervisor.cpuid.v0 = "FALSE"

But it’s easier to accomplish the same thing from the Edit Settings > VM Options > Advanced > Edit Configuration… menu.

Image for post
Image for post
setting the cpuid parameter.

This setting hides the virtualization from the guest OS, so it thinks it is running on bare metal (and it allows the Nvidia drivers to actually work unencumbered). This setting will only take effect with the VM powered down/on reset.

Installing Xubuntu

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-384

After running these commands, you should see a table when you run the following command after restart.

nvidia-smi

this should be enough for something like an ethereum miner script to utilize the GPU.

Bonus: Install Tensorflow

##install pip, vim, and virtualenvsudo apt install python-pip
sudo apt install vim
pip install virtualenv virtualenvwrapper
echo 'export WORKON_HOME=$HOME/.virtualenvs' >> ~/.bashrc
echo 'export PROJECT_HOME=$HOME/Devel' >> ~/.bashrc
echo 'source $HOME/.local/bin/virtualenvwrapper.sh' >> ~/.bashrc
source ~/.bashrc

Let the fun begin…

##install cuda 9.0wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run# CNTRL+ALT+(FN)+F1 to close X-server, then at promptsudo service lightdm stop
sudo apt purge nvidia*
sudo sh cuda_9.0.176_384.81_linux-run
sudo service lightdm start
echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"' >> ~/.bashrcecho 'export CUDA_HOME=/usr/local/cuda' >> ~/.bashrc

Of course, to get *all* of the dependencies you need from Nvidia, you must sign up as a developer; aka, give them your email address, fill out a short survey, and “opt in” to receiving their emails (required). I’m excited for the coming ASICs that will hopefully give us a realistic option to take our business elsewhere. Until then…

Once you’ve logged in, clicked around, and forfeited your information to the Nvidia gods, you’ll eventually be bestowed with a download link. Be sure to choose cuDNN 7.0.* for CUDA 9.0.

##install cudnn 7.0#check Ubuntu version
lsb_release -a
# Download cuDNN 7.0 with CUDA 9.0 support for our Ubuntu versioncd ~/Downloads
sudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb
Image for post
Image for post
That’s right, Nvidia actually makes you log in and click links to download this.

Once that’s installed, we should be ready to go!

##make a virtualenvmkvirtualenv machine-learning
workon machine-learning
##install tensorflow and keraspip install --upgrade tensorflow-gpu
pip install keras h5py

Now we should be able to run tensorflow. I’ve been using this program to test the GPU access:

#test_gpu.pyimport tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

Unfortunately for me, I get a ‘Illegal Instruction (core dumped)’ error when I try to import tensorflow (version=1.6). This is because I’m using an older CPU, so to fix it I needed to:

#fix Illegal Instruction (core dumped) errorpip uninstall tensorflow-gpu
pip install tensorflow-gpu==1.5

and it works!

Image for post
Image for post
tensorflow recognizing a GPU successfully!

A Note for Other Machines

This is an example for when you find a root user has installed cuda-8.0, but not 9.0, which should work out-of-the-box:

echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/extras/CUPTI/lib64"' >> ~/.bashrcecho 'export CUDA_HOME=/usr/local/cuda-8.0' >> ~/.bashrc
source ~/.bashrc
virtualenv ml_venv
source ml_venv/bin/activate
pip install tensorflow-gpu==1.4
pip install keras h5py

Good luck with your new machine learning setup!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store