Continuous Integration with Gitlab, Selenium and Google Cloud SDK

Kim T
Creative Technology Concepts & Code
4 min readJun 29, 2017

As development tools have improved, we have started to see Continuous Integration workflows which allow code to be stored, tested and deployed easily. This post is going to look at one of the most popular open-source platforms for developers: Gitlab.

Gitlab combines code storage with, issue tracking, testing and deployment tools. It’s designed for the complete development workflow from start to finish. We are going to play with the Gitlab pipelines and setup a complete automated workflow based on branches.

  1. Setup Gitlab. Either sign-in to the free hosted version, or download and install your own version.
  2. Create a new Project and download the code repository to your local machine.
  3. Create a backend codebase. In my example i’ve decided to create a simple Python/Flask application which renders a single page:

backend/requirements.txt

Flask==0.12

backend/__init__.py

import os, sys

lib_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'lib')
sys.path.insert(0, lib_path)

from .server import app

if __name__ == "__main__":
app.run()

backend/server.py

import os
from flask import Flask, render_template, send_from_directory

static_dir = os.path.join(os.path.dirname(__file__), 'static')
template_dir = os.path.join(os.path.dirname(__file__), 'templates')
app = Flask(__name__, static_folder=static_dir, template_folder=template_dir)

@app.route('/')
def index():
return render_template('index.html')

@app.route('/api')
def test():
return 'Hello people!'

if __name__ == '__main__':
app.run(host='0.0.0.0', port=3000, debug=True)

backend/templates/index.html

<p>it works</p>

You can run the application using the commands:

pip install -r requirements.txt
python server.py

4) In the root of the project we can now add support for Google AppEngine using:

app.yml

runtime: python27
threadsafe: true

handlers:
- url: /
script: backend.app

- url: /static
static_dir: backend/static

skip_files:
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?..*$
- ^(.*/)?.*_test.(html|js|py)$
- ^(.*/)?.*.DS_Store$
- ^.*bower_components(/.*)?
- ^.*node_modules(/.*)?
- ^.*jspm_packages(/.*)?

5) Make sure you have installed the Google Cloud SDK. We can now deploy this manually to AppEngine using the commands:

gcloud init
gcloud app deploy

But we want to automate this process, how?

6) Create a .gitlab-ci.yml file in the root of your project containing the steps to build the backend, and deploy the code:

.gitlab-ci.yml

build_backend:
image: python
stage: build
script:
- pip install -t backend/lib -r backend/requirements.txt
- export PYTHONPATH=$PWD/backend/lib:$PYTHONPATH
artifacts:
paths:
- backend/

deploy:
image: google/cloud-sdk
stage: deploy
environment:
name: $CI_BUILD_REF_NAME
url: https://$CI_BUILD_REF_SLUG-dot-$GAE_PROJECT.appspot.com
script:
- echo $GAE_KEY > /tmp/gae_key.json
- gcloud config set project $GAE_PROJECT
- gcloud auth activate-service-account --key-file /tmp/gae_key.json
- gcloud --quiet app deploy --version $CI_BUILD_REF_SLUG --no-promote
after_script:
- rm /tmp/gae_key.json

This will use the current branch name as the version, so if you commit code to master, your appengine url will be https://master-dot-projectname.appspot.com

7) Last step before committing your changes to Gitlab (and running the pipeline), you need to set the variables. Go to:

Gitlab Project > Settings > CI/CD Pipelines > Secret Variables

And fill out the following variables:

* GAE_KEY: Secret key for a user with the required permissions.

* GAE_PROJECT: Name of the target AppEngine application.

You will need to get these from Google Cloud at:
Google Cloud Project > IAM & Admin > Service Accounts > Create Service Acccount

8) If you push the code to Gitlab, you should now see the steps running one-by-one:

Click on a failed job to see the output of the errors:

9) Last step is to add some unit/functional tests. I’ve decided to go use PyUnit and Selenium tools to make the process easier. First we update our project code:

backend/requirements.txt

Flask==0.12
chromedriver-installer==0.0.6
selenium==3.4.2

backend/test_server.py

import unittest
from backend import app

class Test(unittest.TestCase):
def test(self):
result = app.test_client().get('/api')

self.assertEqual(result.data.decode('utf-8'), 'Hello people!')

qa/browser.py

import os
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

DRIVER = os.getenv('DRIVER', 'headless_chrome')
BASE_URL = os.getenv('BASE_URL', 'http://backend:3000')
SELENIUM = os.getenv('SELENIUM', 'http://localhost:4444/wd/hub')


def get_chrome_driver():
desired_capabilities = webdriver.DesiredCapabilities.CHROME
desired_capabilities['loggingPrefs'] = {'browser': 'ALL'}

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(
"--user-data-dir=/tmp/browserdata/chrome
--disable-plugins --disable-instant-extended-api")

desired_capabilities.update(chrome_options.to_capabilities())

browser = webdriver.Chrome(
executable_path='chromedriver',
desired_capabilities=desired_capabilities)

# Desktop size
browser.set_window_position(0, 0)
browser.set_window_size(1366, 768)

return browser


def get_headless_chrome():
desired_capabilities = webdriver.DesiredCapabilities.CHROME
desired_capabilities['loggingPrefs'] = {'browser': 'ALL'}

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(
"--user-data-dir=/tmp/browserdata/chrome
--disable-plugins --disable-instant-extended-api
--headless")

desired_capabilities.update(chrome_options.to_capabilities())

browser = webdriver.Remote(
command_executor=SELENIUM,
desired_capabilities=desired_capabilities)

# Desktop size
browser.set_window_position(0, 0)
browser.set_window_size(1366, 768)

return browser


DRIVERS = {
'chrome': get_chrome_driver,
'headless_chrome': get_headless_chrome
}

def get_browser_driver():
return DRIVERS.get(DRIVER)()

qa/test_example.py

import unittest
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from browser import get_browser_driver, BASE_URL


class ExampleTestClass(unittest.TestCase):

def setUp(self):
print("BASE_URL", BASE_URL)
self.driver = get_browser_driver()

def tearDown(self):
self.driver.quit()

def test_page_title(self):
self.driver.get(BASE_URL)
print("driver.title", self.driver.title)
self.assertIn("Gitlab AppEngine CI", self.driver.title)
elem = self.driver.find_element(By.NAME, "search")
elem.send_keys("Selenium")
# elem.send_keys(Keys.RETURN)
# assert "No results found." not in driver.page_source

def test_javascript_text(self):
self.driver.get(BASE_URL)
wait = WebDriverWait(self.driver, 10)
wait.until(EC.visibility_of_element_located(
(By.CSS_SELECTOR, 'div#output')))
elem = self.driver.find_element(By.ID, 'output')
self.assertIn("JavaScript + gulp too!", elem.text)

if __name__ == "__main__":
unittest.main()

You should be able to run the tests manually using the commands:

python -m unittest discover -s backend
python -m unittest discover -s qa

10) Let’s automate the unit/functional tests by adding the following lines to the .gitlab-ci.yml file:

.gitlab-ci.yml

test_unit:
image: python
stage: test
script:
- export PYTHONPATH=$PWD/backend/lib:$PYTHONPATH
- python -m unittest discover -s backend

test_functional:
image: python
stage: test
services:
- selenium/standalone-chrome
script:
- export PYTHONPATH=$PWD/backend/lib:$PYTHONPATH
- SELENIUM="http://selenium__standalone-chrome:4444/wd/hub" BASE_URL="https://$CI_BUILD_REF_SLUG-dot-$GAE_PROJECT.appspot.com" DRIVER="headless_chrome" python -m unittest discover -s qa

Hope that helps you get set up!

View the full working code here:
https://gitlab.com/kim3/gitlab-appengine-ci

--

--

Kim T
Creative Technology Concepts & Code

Creative Technologist, coder, music producer, and bike fanatic. I find creative uses for technology.