It took me a while to get used to, but I can confidently say that using gcloud is much easier than going in and graphically playing around with the Google Cloud Platform. Below we will see how to use gcloud, and after setting that up, we will set up a VM instance in the Compute engine section, and finally an App engine instance.
To begin, you need to open a local terminal and make sure you have everything you need installed. Below is what I normally add when using Debian or Ubuntu. This will automatically install everything that one would normally need for basic programming and using gcloud.
I use Ubuntu (or Debian) for its universality and ease of use. I don’t doubt that there might be better distros, but given that I don’t use a gui most of the time, and I can use it pretty much anywhere (like on a Raspberry Pi), I usually stick with Ubuntu. We will see how to install Ubuntu on a Raspberry Pi in another story.
If you are using Mac, Windows, or another Linux Distro, you will probably need to modify what I have below so that the same apps can be installed. Keep that in mind before just doing a blind “copy-paste” movement in your terminal.
First, we’ll just make sure all the basics are installed.
sudo apt update -y && sudo apt-get update -y && sudo apt-get upgrade -y && sudo apt dist-upgrade -y && sudo apt-get autoremove -y && sudo apt-get clean -y && sudo apt-get autoclean -y && sudo apt-get install software-properties-common build-essential cmake git wget curl mosh vim mlocate postgresql rclone nginx gunicorn python3-pip ca-certificates gnupg apt-transport-https -y && curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash - && sudo apt-get install -y nodejs
I’ve added postgresql and Vim to the “basics” because for me, they are things I use constantly, but if you use Nano or Emacs for editing, you obviously don’t need Vim and if you don’t work with databases, you don’t need postgresql either and can delete them from the list above. Just make sure that the spaces in between each app that will be installed are respected when you finish deleting them.
After that, we can set up everything we need for gcloud.
echo "deb http://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && sudo apt-get update && sudo apt-get install google-cloud-sdk && sudo apt-get install google-cloud-sdk-app-engine-python && sudo apt-get install google-cloud-sdk-app-engine-python-extras -y
Google really should make it easier to install gcloud, but after trying to add it the normal way and having so many errors come out, I just stuck with doing it the “complex” way, as above. You can see other instructions here. If this ends up not working for you, I would first suggest updating Ubuntu or Debian to the latest version, then copying the error, and pasting it as a response to this story so that others can see what the problem was. Careful as well, because gcloud does not work with certain CPUs and that may also cause a problem. (I had a lot of problems trying to install it in Termux on android).
Then, I finish off by adding all my pip3 modules. You can skip this if you normally don’t program in Python, but I use these all the time and will be using them in other tutorial stories.
pip3 install cryptography psycopg2-binary asyncio asyncpg joblib scrapy selenium scrapy-selenium unicodedata2 requests-html beautifulsoup4 multiprocess httplib2 numpy pipenv virtualenv django flask pandas aiogram python-telegram-bot google-api-python-client google_auth_oauthlib google_spreadsheet
If you want the three previous steps in a nice “copy-past” bundle, I’ve combined everything into one below which should take around 10 to 20 minutes to install in normal circumstances with a normal WIFI speed.
sudo apt update -y && sudo apt-get update -y && sudo apt-get upgrade -y && sudo apt dist-upgrade -y && sudo apt-get autoremove -y && sudo apt-get clean -y && sudo apt-get autoclean -y && sudo apt-get install software-properties-common build-essential cmake git wget curl mosh vim mlocate postgresql rclone nginx gunicorn python3-pip ca-certificates gnupg apt-transport-https -y && curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash - && sudo apt-get install -y nodejs && pip3 install cryptography psycopg2-binary asyncio asyncpg joblib scrapy selenium scrapy-selenium unicodedata2 requests-html beautifulsoup4 multiprocess httplib2 numpy pipenv virtualenv django flask pandas aiogram python-telegram-bot google-api-python-client google_auth_oauthlib google_spreadsheet && echo "deb http://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && sudo apt-get update && sudo apt-get install google-cloud-sdk && sudo apt-get install google-cloud-sdk-app-engine-python && sudo apt-get install google-cloud-sdk-app-engine-python-extras -y
After finishing that up, I would restart your device (or remote server) just in case, so that everything “settles” so to speak, and Ubuntu registers all the newly installed apps. Normally it’s not necessary, but it doesn’t hurt to do it. (Remember, when something doesn’t work, it’s always good to restart your device to begin diagnostics anyway…)
Before we begin, I would like to clarify that anything that should be changed for your particular account will be in upper-case and in bold. So, anytime you need to put something specific for YOU, we will put like that, and that will indicate to you that you should modify it.
Now that we have everything installed, we can start gcloud with:
This will ask you first if you want to log in:
You must log in to continue. Would you like to log in (Y/n)?
After typing y to accept, it will offer you link like this (the … means that there is more text in the link that I have omitted, but you need to copy the whole link and past it in your browser:
Go to the following link in your browser:
It will then ask you to select an account, and then allow access.
Google Cloud SDK wants to access your Google Account
Just click Allow .
It will then have you copy a verification code which you should past into the terminal with ctrl + shift + v where it says the following:
Enter verification code:
After that it will offer you this:
Pick cloud project to use:
 project 1
 Create a new project
Please enter numeric choice or text value (must exactly match list
But, we don’t need to do this right now, so we can press ctrl + c to cancel this, and now we are ready to begin using gcloud. If you don’t have anything in the Google Cloud Platform, it may only offer you one option, either way, just cancel it with ctrl + c so we can do other things.
To start off, if you want to see who is logged in just type the following:
gcloud auth list
If you want to change the account (completely optional), you can do the following:
To set the active account, run:
$ gcloud config set account `ACCOUNT`
You can also check the configurations of gcloud with
gcloud config configurations list and then use the command below to change the configuration:
gcloud config configurations activate ACCOUNT
Remember, the upper case is for you, so where ACCOUNT is, you would put your email of the account you want active in gcloud at a certain time and then press enter.
If you have not logged into another account yet (again, completely optional), you can do that by typing:
gcloud auth login
If you want to log out of all accounts, you can type:
gcloud auth revoke --all
Now that we are in, we can start a project, which is the first thing you will do when you start using Google Cloud Platform (GCP). Everything in GCP is always under a project, so we will start with that first.
Side note: Remember that the && means that you are connecting multiple command into one so that you don’t have to paste them in, one by one.
As well make sure that all spaces are correct. If you have one space out of place, it will mark an error.
gcloud projects create PROJECTNAME && gcloud config set project PROJECTNAME && gcloud beta billing projects link PROJECTNAME --billing-account=BILLINGID
Project names should be at least 6 characters long, otherwise, GCP will add more characters to the name. Here we created a project and set it as the project we want to work with. Finally we connected the billing account…
Now, if you are just starting off, you will not incur costs, and there are many “free tier” plans in GCP. Either way, they want you to add a credit card just in case you do use it seriously. In this case, you will have to go to https://console.cloud.google.com/ and set up billing first before beginning to use GCP.
The billing ID will be in the form of 18 characters with two hyphens, like this, under Overview (the one below is just an example):
My Billing Account, 0044CC-244890–22D34C
Once you have a billing account, you can find it by typing:
gcloud alpha billing accounts list
To list your projects just put
gcloud projects list in your terminal.
IMPORTANT! After creating the project, your “PROJECTNAME” is really your PROJECT_ID! When you list the projects, the “name” you gave your project will end up being the ID. Please be careful of this if you create your project graphically in the Google Cloud Platform pages and then use gcloud after, because the ID and the NAME may not coincide…
If you want to delete your project, just put
gcloud projects delete PROJECTNAME
Now that our project is created and selected as the active project, let’s start adding things to it, like certain services… We will need theses when we want to set up stuff that will use Google APIs for our apps in the future. We can only activate up to 20 at a time, but the ones we will use in other stories are the following, so we will activate those in the project now.
gcloud services enable admin.googleapis.com appengine.googleapis.com caldav.googleapis.com cloudapis.googleapis.com calendar-json.googleapis.com chat.googleapis.com classroom.googleapis.com compute.googleapis.com contacts.googleapis.com docs.googleapis.com drive.googleapis.com gmail.googleapis.com groupssettings.googleapis.com iam.googleapis.com iamcredentials.googleapis.com people.googleapis.com sheets.googleapis.com slides.googleapis.com
After all of these are activated we can now add our Compute Engine VM Instance, which is where you can play around in your own little remote server instead of doing all of this on you local computer. For the free tier, we will do the following:
gcloud compute instances create "VMINSTANCENAME" --boot-disk-device-name "VMINSTANCENAME" --zone us-central1-f --machine-type f1-micro --image-project ubuntu-os-cloud --image-family ubuntu-2004-lts --boot-disk-size 30 --boot-disk-type "pd-standard" --maintenance-policy "MIGRATE" --tags http-server,https-server --scopes cloud-platform
So, this creates our little remote server with Ubuntu 20.04 with 0.6 GB of RAM and 30 GB of standard hard drive. With this setup, you have your free-tier vm instance.
Now, in this world, nothing is perfect, and neither is CCP. There seems to be a bug (or it applies to another option, I have no idea), and even though we put
--tags http-server,https-server it does not seem to activate them, so we will sadly have to go back to the graphic part in GCP on to do this. This article on Stack Overflow also insinuates this problem.
You’ll have to go to your CCP, > click on the 3 horizontal bars in the upper right > select Compute Engine > then VM Instances > then click right on top of the name of your instance > then click EDIT > then go to the following and check them:
Now that we have our little server set up, let’s add some firewall rules to let some traffic in, like an SSH connection, and maybe let some bots communicate with it as well. (We’ll see bots in another story.)
gcloud compute firewall-rules create FIREWALLNAME1 --allow tcp:5000 --direction=INGRESS && gcloud compute firewall-rules create FIREWALLNAME2 --allow tcp:5001 --direction=INGRESS && gcloud compute firewall-rules create FIREWALLNAME3 --allow udp:60000-61000 --direction=INGRESS
Now, the general http and https traffic should already be active, but if not, you can use the ones below. Just be careful, if they are already activated, you’ll get an error in the terminal indicating the same.
gcloud compute firewall-rules create default-allow-http --allow tcp:80 --direction=INGRESS && gcloud compute firewall-rules create default-allow-https --allow tcp:443 --direction=INGRESS
If you want to list your firewall rules just type:
gcloud compute firewall-rules list
If you forgot to add some tags in the previous step, you can do so like this:
gcloud compute instances add-tags jjirmeapp1 --zone us-central1-f --tags=http-server,https-server
In order to SSH into your new instance, you will need to add a public SSH key. and you should be able to do that with the following (but sadly almost never works — another bug on the loose…). Keep in mind that the “USERNAME” here would be the first part of your email you are using in gcloud.
ssh-keygen -t rsa && cat ~/.ssh/id_rsa.pub > NAME.txt && sed -i '1s/^/USERNAME:/' NAME.txt && sed -i 's/\=.*/= USERNAME/' NAME.txt && gcloud compute instances add-metadata VMINSTANCENAME --zone=us-central1-f --metadata-from-file ssh-keys=NAME.txt && ssh USERNAME@IPINSTANCE
To get your SSH keys, you would just put this in the terminal:
ssh-keygen -t rsa, hitting enter three times when it asks you for a file name and if you want to add a password (none of that is necessary). If you already have your SSH keys, then just omit the
ssh-keygen -t rsa && at the beginning.
Again, the problem is that there is a bug, and I never got it to work, so… for now, you will have to go back to the gui in GCP > click on the 3 horizontal bars in the upper right > select Compute Engine > then VM Instances > then click right on top of the name of your instance > then click EDIT > then go to the following:
And there you will add your public SSH key along with the username at the end of the key (after the = sign).
Update: I just tried the code again and it seemed to work, so let’s hope you don’t have to go to the gui for this in the future.
Finally, to list your Compute Instances we type:
gcloud compute instances list --project PROJECTNAME
And, to delete an instance, we type the following:
gcloud compute instances delete VMINSTANCENAME --zone=us-central1-f
Now that we finished the Compute Engine part, let’s add an App Engine Instance as well. Keep in mind, unlike the Compute Engine instance, the command below only activates the App Engine section, but does not activate any apps in and of themselves. Actual apps are called “services” .
gcloud app create --project=PROJECTNAME --region=us-central
To see a list of your apps (or services) you can type the following:
gcloud app services list --project=PROJECTNAME
To deploy an app, you would use the following, keeping in mind that the app.yaml is a file that we will see in the story on how to create a bot.
gcloud app deploy APP.yaml --project=PROJECTNAME --stop-previous-version
To delete a service (or app) you would type this:
gcloud app services delete SERVICENAME --project=PROJECTNAME
If you want to list the versions of a service (app) then type:
gcloud app versions list — project=PROJECTNAME
Or, if you just want to to stop or delete a version of an app (service) you would use the following, changing the version number when needed:
gcloud app versions stop --project=PROJECTNAME --service SERVICENAME v1gcloud app versions delete --project=PROJECTNAME --service SERVICENAME v1
Let’s finish off by adding some service accounts. Now, besides creating the service account, we are relating the admin email to the service account, and this will be VERY important for when we start using Google APIs in future stories.
gcloud iam service-accounts create NAME --description "DESCRIPTION" --display-name “DISPLAYNAME” && gcloud iam service-accounts add-iam-policy-binding NAME@PROJECTNAME.iam.gserviceaccount.com --member 'user:USERNAME@EMAIL.com' --role 'roles/owner' && gcloud iam service-accounts add-iam-policy-binding NAME@PROJECTNAME.iam.gserviceaccount.com --member 'user:USERNAME@EMAIL.com' --role 'roles/iam.serviceAccountUser'
Finally, the last thing we should do is to Enable G Suite Domain-wide Delegation. But again sadly, we cannot do this with gcloud. It is not even a bug, there simply isn’t an option to do it, so… back to the CCP gui:
Click on the 3 horizontal bars in the upper right > select IAM & Admin> then Service Accounts> then click right on top of the name of your Service account> then click on SHOW DOMAIN-WIDE DELEGATION > then activate the following, putting any Product name you want in that box:
In order to use this Service account with Google APIs, we will need to create a key for the service account, but we can do that with gcloud:
gcloud iam service-accounts keys create --iam-account NAME@PROJECTNAME.iam.gserviceaccount.com keys.json
In the keys.json file created, we can find the “client_id” which is an ID that we will put in the Google Admin Console under Security > Advanced settings > Authentication > Manage API client access > Client Name (this is where the client ID will go), and beside that, we will add the scopes that we need. I have put the scopes below for those who want to add them, but either way, we will see those in another article about bots.
To finish off, you can list your service accounts by typing
gcloud iam service-accounts list
To delete a service account just type the following (the word delete has a space after it, not an enter as it appears here).
gcloud iam service-accounts delete NAME@PROJECTNAME.iam.gserviceaccount.com
There are obviously other options given in GCP as well as in gcloud, but I wanted to give a summary of the most commonly used ones to give you a head start in your work.
If you want to see a video on all of this, you can check it out here.
We will see many other things in upcoming stories about python and bots, but it was important to get these basic elements out of the way first.
If there was anything I missed, please let me know. Cheers.
Code all together as an example (careful with the spacing!):
(to get billing id)
gcloud alpha billing accounts listgcloud projects create youtubet1597 && gcloud config set project youtubet1597 && gcloud beta billing projects link youtubet1597 --billing-account=0044CC-244890–22D34C && gcloud services enable admin.googleapis.com appengine.googleapis.com caldav.googleapis.com cloudapis.googleapis.com calendar-json.googleapis.com chat.googleapis.com classroom.googleapis.com compute.googleapis.com contacts.googleapis.com docs.googleapis.com drive.googleapis.com gmail.googleapis.com groupssettings.googleapis.com iam.googleapis.com iamcredentials.googleapis.com people.googleapis.com sheets.googleapis.com slides.googleapis.com && gcloud compute instances create "vmyoutubet1" --boot-disk-device-name "vmyoutubet1" --zone us-central1-f --machine-type f1-micro --image-project ubuntu-os-cloud --image-family ubuntu-2004-lts --boot-disk-size 30 --boot-disk-type "pd-standard" --maintenance-policy "MIGRATE" --tags http-server,https-server --scopes cloud-platform && gcloud compute firewall-rules create bot1 --allow tcp:5000 --direction=INGRESS && gcloud compute firewall-rules create bot2 --allow tcp:5001 --direction=INGRESS && gcloud compute firewall-rules create mosh --allow udp:60000-61000 --direction=INGRESS && gcloud app create --project=youtubet1597 --region=us-central && gcloud iam service-accounts create iamyoutubet1 --description "iamyoutubet1" --display-name “iamyoutubet1” && gcloud iam service-accounts add-iam-policy-binding email@example.com --member 'user:firstname.lastname@example.org' --role 'roles/owner' && gcloud iam service-accounts add-iam-policy-binding email@example.com --member 'user:firstname.lastname@example.org' --role 'roles/iam.serviceAccountUser' && gcloud iam service-accounts keys create --iam-account email@example.com keys.json(to get public IP of vim instance)
gcloud compute instances list --project youtubet1597ssh-keygen -t rsa && cat ~/.ssh/id_rsa.pub > sshs.txt && sed -i '1s/^/user:/' sshs.txt && sed -i 's/\=.*/= user/' sshs.txt && gcloud compute instances add-metadata vmyoutubet1 --zone=us-central1-f --metadata-from-file ssh-keys=sshs.txt && ssh user@IPINSTANCE
SCOPES FOR ADMIN CONSOLE