Tutorial: Install Diffgram In Google Compute Engine
In this post we will learn how to install Diffgram on a compute engine instance. This is the fastest method to get Diffgram up and running if you want to test it out on a cloud environment in GCP without any additional managed services, making it the most cost effective way to test the tool on the cloud.
Quick Install
If you already have a GCP instance up and running, and also have docker compose on your instance your can just follow our Getting Started Guide.
Full Guide
First let’s start by creating our Compute Engine instance and configuring it.
On your GCP Console Go to the compute engine dashboard and click the create button
After that write the name of your compute instance, and choose and select the machine type, we will choose a 4CPU 16GB ram, but feel free to change it.
Make sure to check the boxes that say allow HTTP and HTTPS traffic:
Now hit the create button and you’ll have your compute instance!
Add Firewall Rules To Allow Traffic
In order to access the instance from port 8085 (The diffgram port). Go to the compute engine instance list, select diffgram instance and then click view network details:
Now go to the firewall section on the left panel and click “Create Firewall Rule”. On targets section, set the option “all instances in the network”, and make sure to allow traffic from port 8080,8081,8082,8085 as in the image below.
Create the rule and now you should be able to access it
Connecting to the compute instance
Connecting is very simple, just select your compute instance and click the “SSH” button on the screen:
Once the instance connects you will have a shell available to install Diffgram:
Installing Diffgram and Docker/Docker-compose Single Command:
We will need to install git and docker before continuing with Diffgram installation. You can copy and paste the following commands to install everything and get started right away with diffgram. Or scroll to the next section for a more in depth explanation
Install Everything: Copy/Paste
sudo apt-get update ;\
sudo apt-get install -y \
git \
apt-transport-https \
ca-certificates \
curl \
gnupg \
lsb-release ; \
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg ; \
echo \
"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null ; \
sudo apt-get update ; \
sudo apt-get install -y docker-ce docker-ce-cli containerd.io ; \
ps aux | sudo groupadd docker > /tmp/test.txt && echo 1 || echo 0
sudo usermod -aG docker $USER ; \
newgrp docker ; \
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose ; \
sudo chmod +x /usr/local/bin/docker-compose ; \
git clone https://github.com/diffgram/diffgram.git ; \
cd diffgram ; \
python3 install.py
Installing Diffgram and Docker/Docker-compose Step by Step:
Explaining Each Command from the above section:
Install dependencies for docker and git:
sudo apt-get updatesudo apt-get install \
git \
apt-transport-https \
ca-certificates \
curl \
gnupg \
lsb-release
Add GPG Key for docker installation
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
Setup Stable Repository
echo \
"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Install Docker Engine
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
Use Docker without sudo
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
Test docker install by running
docker run hello-world
Install docker-compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
Installing Diffgram
To install Diffgram run the following commands
Get the install script and docker compose from the repo:
git clone https://github.com/diffgram/diffgram.git
cd diffgram
Run install Script:
python3 install.py
You will get a prompt like this:
Depending on your cloud provider, select the appropriate option. We will choose AWS for this example.
Follow the steps on the prompt and After installation has finished you will see something like this:
Accessing Diffgram
Now that you have Diffgram running, you can go to the external IP of your compute instance on port 8085 and you should see your Diffgram instance ready to go!
Final Comments
Although Diffgram is super easy to install there are some changes that are suggested for production payloads. This includes moving the database out of the docker compose container and into a managed service like Cloud SQL or RDS. Also we suggest separating the containers into independently scalable machines/pods/services, so that the API service (default) and the File Processing (walrus) can be scaled independently. You can also consider our kubernetes helm chart to have a granular control over each of the services, or our fully hosted version of Diffgram