Tutorial: Install Diffgram In Google Compute Engine

In this post we will learn how to install Diffgram on a compute engine instance. This is the fastest method to get Diffgram up and running if you want to test it out on a cloud environment in GCP without any additional managed services, making it the most cost effective way to test the tool on the cloud.

Quick Install

If you already have a GCP instance up and running, and also have docker compose on your instance your can just follow our Getting Started Guide.

Full Guide

First let’s start by creating our Compute Engine instance and configuring it.

On your GCP Console Go to the compute engine dashboard and click the create button

Click the create button

After that write the name of your compute instance, and choose and select the machine type, we will choose a 4CPU 16GB ram, but feel free to change it.

Make sure to check the boxes that say allow HTTP and HTTPS traffic:

Now hit the create button and you’ll have your compute instance!

Add Firewall Rules To Allow Traffic

In order to access the instance from port 8085 (The diffgram port). Go to the compute engine instance list, select diffgram instance and then click view network details:

Go to network details

Now go to the firewall section on the left panel and click “Create Firewall Rule”. On targets section, set the option “all instances in the network”, and make sure to allow traffic from port 8080,8081,8082,8085 as in the image below.

Firewall rule to access diffgram ports

Create the rule and now you should be able to access it

Connecting to the compute instance

Connecting is very simple, just select your compute instance and click the “SSH” button on the screen:

Once the instance connects you will have a shell available to install Diffgram:

Here you can start installing diffgram!

Installing Diffgram and Docker/Docker-compose Single Command:

We will need to install git and docker before continuing with Diffgram installation. You can copy and paste the following commands to install everything and get started right away with diffgram. Or scroll to the next section for a more in depth explanation

Install Everything: Copy/Paste

sudo apt-get update ;\
sudo apt-get install -y \
    git \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release ; \
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg ; \
echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null ; \
sudo apt-get update ; \
sudo apt-get install -y docker-ce docker-ce-cli containerd.io ; \
ps aux | sudo groupadd docker > /tmp/test.txt && echo 1 || echo 0
sudo usermod -aG docker $USER ; \
newgrp docker ; \
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose ; \
sudo chmod +x /usr/local/bin/docker-compose ; \
git clone https://github.com/diffgram/diffgram.git ; \
cd diffgram ; \
python3 install.py

Installing Diffgram and Docker/Docker-compose Step by Step:

Explaining Each Command from the above section:

Install dependencies for docker and git:

sudo apt-get update
sudo apt-get install \
    git \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
 

Add GPG Key for docker installation

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Setup Stable Repository

echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Use Docker without sudo

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Test docker install by running

docker run hello-world 

Install docker-compose

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Installing Diffgram

To install Diffgram run the following commands

Get the install script and docker compose from the repo:

git clone https://github.com/diffgram/diffgram.git
cd diffgram

Run install Script:

python3 install.py

You will get a prompt like this:

Depending on your cloud provider, select the appropriate option. We will choose AWS for this example.

Follow the steps on the prompt and After installation has finished you will see something like this:

Accessing Diffgram

Now that you have Diffgram running, you can go to the external IP of your compute instance on port 8085 and you should see your Diffgram instance ready to go!

Visit the External IP on your browser to Access Diffgram on port 8085 (For This example: 34.122.203.93:8085)

Diffgram Instance Ready to go!

Final Comments

Although Diffgram is super easy to install there are some changes that are suggested for production payloads. This includes moving the database out of the docker compose container and into a managed service like Cloud SQL or RDS. Also we suggest separating the containers into independently scalable machines/pods/services, so that the API service (default) and the File Processing (walrus) can be scaled independently. You can also consider our kubernetes helm chart to have a granular control over each of the services, or our fully hosted version of Diffgram