Serving Dynamic Meta Tags in a Serverless Architecture
How to use Fastly and Prerender.io to serve dynamic meta tags in a single page application architecture
When we launched Cactus, we chose to build it as a Javascript (Vue) Single Page App (SPA) and adopted a serverless architecture using Firebase Hosting to serve static files from the cloud. This gave us instant scalability with a relatively simple technology stack for iterating towards product-market fit, as fast as possible.
The premise of Cactus is simple — every day it sends you a question designed to help you focus on what really matters to you and write down your answer. From the beginning, our team wanted Cactus to be something that people wanted to share with other people.
Why dynamic meta tags?
It quickly became important that we could optimize the sharing experience for our users. This meant when you shared a question or answer with someone else through a text message, Slack, or on a social network, we wanted the “preview” to better communicate specifically the content you were sharing with your friend.
In a more traditional app with server-side rendering of HTML, customizing the meta tags (title, description, and image) is trivial and typically essential for SEO. But in our app, the customization code executed inside each visitor’s browser. This meant that the HTML returned for every URL of our app was exactly the same.
How could we improve this sharing experience without significantly changing our technology infrastructure?
Common dynamic rendering solutions include Puppeteer, Rendertron, and prerender.io. All of these solutions require installing server-side middleware and some require adding serious complexity to your overall infrastructure.
These solutions work by examining the User Agent of each request to see if it’s coming from a known crawler / bot. If yes, the request is redirected to an API that returns server-side rendered HTML as the response. Since our front-end architecture was serverless and essentially static files served from a CDN, we did not have access to the details of each request or a server to parse them. Using one of these solutions was not an option for us without significant effort.
To work around these constraints we came up with a novel solution that’s been working remarkably well for nearly a year.
Fastly + Prerender.io
Instead of using installed server middleware, we use Fastly to do the following for each request to cactus.app:
- Sniff the User Agent of each request to Cactus, detecting whether a request is from a crawler bot.
- If the request User Agent matches a known bot, we rewrite & redirect it to the Prerender.io API, which in-turn makes a request back to us, downloads our Javascript app, renders the page, then returns the rendered HTML with dynamic meta tag logic applied.
- If the request is not from a bot, it gets passed through directly to our Firebase Hosting serverless app.
This process happens in an entirely transparent way to users. It just works.
Tradeoffs
The downside to this solution is that it costs money. At least $60 / month (Fastly has a $50 minimum charge per month). But this cost has far outweighed the cognitive overhead of managing our own servers, middleware, cloud functions, or APIs for server-side rendering.
There have been additional unexpected benefits of using Fastly in our infrastructure tool belt, including controlling our static file caching with more precision which allows us to serve static files longer than Firebase Hosting normally would. And Fastly is fast — really fast.
Implementation Details
Below is the VCL syntax for our Fastly Request Condition. First this sniffs the request User Agent for crawler bots. In a secondary statement, we exclude requests for static asset files (favicon, JS, CSS).
req.http.User-Agent ~ “(?i)googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|showyoubot|outbrain|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp|ImgProxy” && req.url ~ “^(?!.*?(assets|favicon|\.js|\.css))”
When this condition is met, we do the following in Fastly:
- Add our Prerender.io API Token to a new X-Prerender-Token header.
- Rewrite the url header to Prerender.io’s API syntax of
/https%3A%2F%2Fyour.app.domain.com” urlencode(req.url.path) - Send the request to a separate Origin Host for Prerender.io’s API server:
service.prerender.io
Conclusion
Setting up an isomorphic Javascript app can be cumbersome. Using the approach described above, we were able to keep the Cactus front-end single page application architecture but still enjoy the necessary benefits of server-side rendering without the server.
