Vue Storefront on Heroku — Elasticsearch, Redis, GitHub, and Varnish (Fastly)

Fetching a cached page

Fetching a page with Fastly in front on Heroku.

This guide will describe how you can deploy Vue Storefront on Heroku. To follow this guide it requires a basic understanding of Vue Storefront. The cache concept is fairly simple, we will use Vue Storefront SSR tags “X-VS-Cache-Tags” as surrogate keys in Fastly. (Some of the code is not the most graceful, feel free to help me if you want to get it more pretty)

Before we start, here is a short overview and cost-breakdown of the services we will use:

• Elastic.co around $60/month
• Heroku: Vue Storefront API Standard-1X dynos $25/month
• Heroku: Vue Storefront frontend Standard-2X dynos $50/month
• Heroku Addon: Heroku Redis Premium 0 — $15/month
• Heroku Addon: Fastly Quick TLS $55/month
• Total per Domain on Heroku $145/month
• Github private unlimited private repos $7/month
• + Make sure that you have a version of Magento 2 up and running with some data in.

Let’s start with deploying a new app in Heroku for the Vue Storefront-API

And one Heroku app for Vue Storefront

I use Github to host all my repositories so I will create a new private repository that will host the API and one for Vue Storefront:

Now, when I have my new empty API repository, we will fill it up with a copy of Vue Storefront-API, open a terminal and:

$ mkdir tmp-api
$ cd tmp-api/
$ git clone --bare https://github.com/DivanteLtd/vue-storefront-api.git
Cloning into bare repository 'vue-storefront-api.git'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 2810 (delta 3), reused 2 (delta 0), pack-reused 2796
Receiving objects: 100% (2810/2810), 3.90 MiB | 1.10 MiB/s, done.
Resolving deltas: 100% (1717/1717), done.
$ cd vue-storefront-api.git/
$ git push --mirror https://github.com/YOURUSER/YOUR-api.git
Counting objects: 2810, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1041/1041), done.
Writing objects: 100% (2810/2810), 3.90 MiB | 1.02 MiB/s, done.
Total 2810 (delta 1717), reused 2810 (delta 1717)
remote: Resolving deltas: 100% (1717/1717), done.
To https://github.com/YOURUSER/YOUR-api.git
* [new branch]      develop -> develop
* [new branch]      feature/docker -> feature/docker
* [new branch]      master -> master
* [new branch]      revert-107-commander -> revert-107-commander
* [new tag]         v1.0.1 -> v1.0.1
* [new tag]         v1.0.4 -> v1.0.4
* [new tag]         v1.0.5 -> v1.0.5
* [new tag]         v1.1.0 -> v1.1.0
* [new tag]         v1.2.0 -> v1.2.0
* [new tag]         v1.3.0 -> v1.3.0
* [new tag]         v1.3.1 -> v1.3.1
* [new tag]         v1.3.2 -> v1.3.2
* [new tag]         v1.4.0 -> v1.4.0
* [new tag]         v1.4.2 -> v1.4.2
* [new tag]         v1.4.3 -> v1.4.3
$ cd ..
$ rm -rf vue-storefront-api.git

Then just make the same procedure for the Vue Storefront repository to an own private repository.

Go ahead and launch an Elasticsearch in the cloud, you can launch it from Heroku. But you will get a 14 days free trail if you launch it from https://www.elastic.co/cloud/elasticsearch-service . Note down your passwords, ports and end points. Make sure you choose a compatible version of Elastic Stack (5.6.12):

(otherwise you will run into “Rejecting mapping update to [vue_storefront_catalog] as the final mapping would have more than 1 type: [product, category]”. When this task is ready it should be more interesting to move over to v6 of Elasticsearch)

Now go to Heroku and the API app and the tab Resources and add the Add-On for Heroku Redis. I choose the “Premium 0” plan.

Provision Heroku Redis

Go to Settings tab and note down your Redis setting

Now let’s change some code and endpoints in the API repo. First we need to make sure that we add support for Elastic over https. In file: src/platform/magento2/tax.js add support for https on row 67:
protocol: ‘https’,

In config/default.json configure your details:
- Update the server settings to the Heroku url (you fill find it in Heroku) and port to 80.
- Configure the remote Elasticsearch endpoints, users, port and password and add protocol: ‘https’.
- Configure Redis url and port.

In migrations/.common.js on row 4, where KUE is handling the Redis, I needed to hardcode the settings for Redis:

let queue = kue.createQueue(Object.assign(config.kue, { redis: {
      host: "CHANGEME",
      port: "CHANGEME",
      auth: "CHANGEME",
      db: 0
    } }))

And in the migrations/.common.js also added a support for https in the elastic connection:

const esConfig = {  
  host: {
    host: config.elasticsearch.host,
    protocol: 'https',
    port: config.elasticsearch.port
  },

In scripts/elastic.js add support for https. Change row 17 and 32 from http to https.

In src/graphql/elasticsearch/client.js add support for https. Change row 6 from http to https.

In src/api/catalog.js add support for https. Change row 36 from http to https.

In package.json update start and add a few other dependencies:

Also add .babelrc to your root folder

Also update your yarn file

gist.github.com/jorkvist/2fd2a4bf9dadca309ff64f51834bbb64

In src/index.js add:

import 'babel-polyfill';

And remove host on row 44:

app.server.listen(port, () => {    console.log(`Vue Storefront API started at http://${host}:${port}`);  });

Now lets connect the repository and then do the first deploy of API, it will throw some errors but we will just need it to create the index for now. After the deployment is done.

Open the console for the API app

Run the command for adding a new empty index

Make sure there are no errors in the creation of the index

Download mage2vuestorefront on your local machine or on your server and configure it:

In src/config.js update to your magento2 settings and change Elasticsearch url

url: 'https://USER:PASS@ELASTIC-SERVER:PORT',

And the Redis section configure the remote Redis

    host: 'SERVER',
    password: 'XXXX',
    port: PORT,
    db: 0

In src/cli.js we need to change the Redis connection to remote on row 15:

let queue = kue.createQueue(Object.assign(config.kue, { redis: {
      host: "XXXX",
      port: "XXXX",
      auth: "XXXX",
      db: 0
    } }));

After that, now you should be able to sync to the remote Redis and remote elastic. When your sync is done go to:

https://YOUR-APP.herokuapp.com/api/catalog/vue_storefront_catalog/_search and you should se a json response with data in it.

Now we only have the front-end left! Update default.json and enable useOutputCacheTagging and useOutputCache. Configure the remote redis, api etc.

And update package.json

And update /core/scripts/server.js

Surrogate Keys

Update core/scripts/server.js on row 185 update if you didn’t copy that change from the server.js file over here.

res.setHeader('X-VS-Cache-Tags', cacheTags)

To

res.setHeader('Surrogate-Key', cacheTags)

Now deploy again and check so the Surrogate-Key headers on a category page look the way I present it here:

To purge the SSR cache just go to: YOUR-APP.herokuapp.com/invalidate?key=YOURKEYFROMCONFIG&tag=product,category,home

Now lets change your dyno type on both deployed apps in Heroku to at least standard 1X (check your stats for memory later when its running, you might need to upgrade to a 2X).

In Heroku, go to resource page and Provision Fastly Quick TLS (you need TLS for https).

After that’s done you can click Fastly and start to look around. First validate that your Fastly url is working. Login to Fastly and go to the configuration and the active version:

Go to the active version

And then find your current Fastly URL:

Find your Fastly domain

And just for fun, test it! You should get a hit from the Fastly cache on second load (and it strips the surrogate keys). And now you should have some awesome loading time on the page. For me both TTFB and is under 40ms. Now let’s configure our domain to work with Fastly. First in Fastly clone your current config (next to the active version button). And add your own domain, and then activate the new version.

Now open your terminal and make sure that you have Heroku cli installed, and run:

$ heroku fastly:tls DOMAIN --app APP

Now you should get a TXT record in response, however i did not get that. But just email the Fastly support and ask for it, they usually answer in a few minutes. The txt record you need to setup at your DNS provider. When that’s done, go back to the Heroku CLI and start the verification process with

heroku fastly:verify start DOMAIN —app APP

When that is started you should be able to run and get the current status with:

heroku fastly:verify status DOMAIN --app APP

When its done you should get the CNAME in return that you need to setup for your domain:

When your DNS changes are ready you should be able to surf to your domain and see the cached site. Now we just need to do a few more settings before it is all done.

In Fastly clone your config and change “Override host” to your own domain and enable “Force TLS and enable HSTS”. Under Gzip enable default gzip policy. Under VCL snippets we need to add a fix for the force to https (it will force to your Heroku domain if you don’t add this it):

The host should be your Vue Storefront herokuapp domain

When you are done, activate your new version and purge the cache. Now you should be good to go! Verify that you getting hits from the X-Cache: HIT when you load the page.

Not covered:

• Purge the cache in Fastly. Find Purge API documentation here a suggested strategy is when you purge SSR cache from mage2vuestorefront
• Setup a proxy to the API in Fastly like the Vue Storefront production setup is suggesting. Setup another VCL snippet in vcl_recv

if (req.url ~ "^/api") {    set req.http.host = "API-HOST"; }