Getting Started with Conducto on Windows Subsystem for Linux
If you want a unix environment on Windows, you have probably reached for Windows Subsystem for Linux (WSL) on Windows 10 Pro (or better). It works well and achieves the goal of running the vast majority of text based Linux tools. Conducto also is well supported on WSL. This post takes you through the details to get you started quickly if you wish to use Conducto from WSL.
First, consider your use case and whether you may be just as well supported by using Conducto from Windows directly. See the companion post Getting Started with Conducto on Windows to review that option. Configuring Docker on Windows is a prerequisite whether you run Conducto on Windows from PowerShell or from the WSL prompt.
When using WSL attached to Docker Desktop on Windows, one generally must remember that docker can only bind to directories in your Windows file system. This means that your pipelines with shared context must have that context on the Windows file system. You will see also below that we require you to expose your Docker daemon port 2375 with-out TLS. For these reasons, we generally suggest you use Windows directly. However we enjoy starting our pipelines from Linux for some applications and we believe that you might too.
With that said, let’s hop right in for some unix with Windows fun. If you are in a hurry, head down to our condensed version.
Prerequisites
This post will guide you through the following prerequisites:
- On Windows you need to install and configure Docker — https://docs.docker.com/docker-for-windows/
- Ubuntu WSL distribution (Debian works equally well).
- Docker configuration in WSL
- Python package installation in WSL
Installing Docker
Get the docker installation from here and install the download. You can test your install by running the following command from a PowerShell prompt. You should see “Hello from Docker!” and some explanatory details.
docker run --rm hello-worldWe require a few settings which I’ll guide you through here. This part of the configuration is entirely in Windows. Start “Docker Desktop” from the start menu. You may need to right click on the Docker whale in the lower right system tray and select “Dashboard”. There are two requirements for your Docker configuration:
- Enable “Expose daemon on tcp://localhost:2375 without TLS” (It is our current understanding that it is not possible to use TLS for Linux on Windows Docker with-out using experimental features.)
- C drive must be shared with your containers (This is true for using Conducto on PowerShell or with-in WSL)
While you are there you can also check the Advanced settings. This allows you to limit the CPUs and Memory assigned to Docker containers. I found the Docker default of 2 CPUs to be quite conservative for my 12 CPU machine. Conducto will follow these resource limits when assigning parallel workers so this is an important limit to know about to maximize your pipeline through-put.
Installing WSL
Install (or continue in your existing installation) a WSL distribution from the Microsoft Store. Open the Microsoft Store app from the start menu and search for Ubuntu. Click through the install. After install, start the distribution from your start menu and enter your username and password.
Finally, Conducto!
Now you can get started with Conducto. There are only 3 steps:
- Create a Conducto account.
- Install our python package.
- Run the demo.
Create an account
Creating an account is easy. Just register here, click the link in the verification email we send you, login, and you are good to go.
Install our python package
Important: Depending on your python installation, you may need to run python3 and pip3 in the commands below.
We recommend that you create a virtual environment. Virtual environments are a common way in Python development to isolate the Python level dependencies from your system. Launch the WSL distribution and the following commands will prepare a virtual environment with the Conducto package installed. It is always a good idea to keep package managers up-to-date and the Python pip package installer is no exception.
python -m venv venv
. ./venv/bin/activate
# update the pip package manager
python -m pip install --upgrade pip
# install our package
pip install conductoYou can confirm successful installation by running this.
conducto --versionNote that you can exit the virtual environment when you are done like this. But don’t do that quite yet because you’ll need to be in the virtual environment with conducto installed to run our demo.
deactivateRun the demo
Finally, grab our demo and run it. The demo will print a URL and attempt to automatically open a web browser that shows your live pipeline.
git clone https://github.com/conducto/demo.git
cd demo
python demo.py hello --local
If you’d rather, you can fully interact with your pipeline from a notebook or a phone. Just navigate to the URL shown in the command line output.
A good next step is to read our Introduction to Conducto Pipelines.
Condensed Set-up Instructions for WSL
Install Docker from the Docker Desktop for Windows page. In Docker Desktop settings, review the following:
- (required) Check “Expose daemon on tcp://localhost:2375 without TLS”
- (required) Share your C drive with your containers under resources.
- (optional) Tweak your CPU/memory settings under resources.
Install WSL. Do the following to install Docker and point it to the Docker in Windows.
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker <user>
echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc# check for success with --
docker info
Create a Conducto account at our sign-up page.
Install our Python package
python -m venv venv
. ./venv/bin/activate
# update the pip package manager
python -m pip install --upgrade pip
# install our package
pip install conductoIn this same prompt, you can run our demo.
git clone https://github.com/conducto/demo.git
python demo/hello.py --local
