Creating websites for search engines with web components ( StencilJS )

“A bright neon on a brick wall in a store” by Austin Chan on Unsplash

Hello. It’s me again. In the previous post I just dropped a note about how StencilJS and native web components improved our developers work flow for applications. This time I want to tell you more about how StencilJS changed the work flow for the website development within our teams (customer side or in — house).

Creating templates for various content management systems was always a pain from a developers perspective. The results were often unstructured or not as clean as the structure could be. There were a lot of includes and preprocessors and so on. The result was often a mix of HTML and ( often PHP ) code.

On the other side there is the option to create a single page application in JavaScript which results in a very nice experience for the user, except if it comes to the initial load time. After the application has been loaded the website typically behaves like a local app and the user experience is very good. The two downsides are typically the initial load time and possible problems for search engine crawlers. Even if Google is nowadays able to crawl the website, you get in trouble as soon as a JavaScript error occurs and the Google bot also wants a lot of polyfills to work with a SPA.

So what do I want from a developers perspective?

· I want to have a smooth development work flow

· I want to have clearly structured code

· I want to have well documented code

· I want to be able to reuse parts of my code and templates

What does want the SEO guy want?

· He wants to be able to do a right — click in the browser and see the complete rendered output when he clicks on the view — source button

· He wants to be able to modify the header information

· He wants the hreflang tags to be set correctly

· He wants a working internal link structure

· He wants a fast initial load time

· ….

In the first collection of the developers needs you see that this does not fit for the most common content management systems and in the SEO guy’s world it will not fit for a SPA.

There are several technical solutions to solve this problem if you want to use a component based approach. Typically this can be done with server side rendering or a prerenderer like http://prerender.io.

From my personal experience both create complexity and you will always need server side solutions, like redirecting the search engine bots to prerender or create a server application.

You often develop two projects and are never able to share components between each other, because the application has been developed in a JS framework and the static pages are rendered by a CMS, especially if you have a website which is a mixture of static pages and application behavior like a booking engine or shop. With StencilJS this has changed for us. We are now able to create the application’s UI elements in StencilJS ( even if the application itself is developed in another framework like angular, react, or vue) and the static website can also be created with web components.

A very nice example in my oppinion on how a SPA/MPA mix can work is https://www.wired.com. If you visit the website without enabling JS in your browser you are able to see all the contents. If you click an internal link you have a full request and reload of the page. If you enable JavaScript after the initial HTML has been distributed to your browser the SPA starts and if there is a click to an internal link it just loads the data from the server and renders it instead of doing a full reload.

This can be done with StencilJS in a very simple way. The Stencil — Starter — Project already includes the prerender functionality needed to create a website with the above SPA/MPA feature. When I tried the starter project I thought I would have to add every page as an entry in the router and I failed. The reason was clear, as the website I tried to render had around 9000 sub — pages and as soon as I visited one of the pages there were all the url’s in the HTML. This was a massive overhead and slowed the page down. My solution was to have only one single route and create a loader which tries to fetch the contents via HTTP request from an API or local JSON files. If a file does not exists it’s just redirecting to the home.

So how does this look like? In the app-root I added the following code:

<stencil-router>
 <stencil-route component=’site-loader’></stencil-route>
</stencil-router>

This means that on every URL the site-loader will be active. If the site-loader imports the MatchResults and adds a property, the component’s update functions will be triggered on every route change.

/**
 This is just a example. Please integrate you own error handling  (404) and document parsing if needed
 */
import { Component, Element } from '@stencil/core';
import { MatchResults } from '@stencil/router';
 @Component({
   tag: 'site-loader',
   styleUrl: 'site-loader.scss'
 })
 export class SiteLoader {
 
   @Prop() match: MatchResults;
   @Prop() data: any;
   @Element() el: HTMLElement;
   componentWillLoad() {
         return this.fetchContents();
   }
   
   componentWillUpdate() {
       return this.fetchContents();
   }
   
   fetchContents() {
     const documenturl = document.location.pathname;
     const splitres = documenturl.split('/');
     let lang = 'en';
     if(splitres[1].length === 2) {
       lang = splitres[1];
     }
     return fetch('/contents/'+ documenturl + '/index.json')
     .then(data => {
          return res.json()
             .then(json => {
                 this.data = json;
             })
     })
   }
   
   /**
   here the render function will just append a component for the pagetype (template) and add the data as attributes
   */
   render() {
       const element: any = document.createElement('pagetype-' + this.data.template);
         element.data = this.data;
         // clearn content
         this.el.innerHTML = '';
         this.el.appendChild(element);
         return;
   }
 }

After the site-loader is triggered on every URL change, the content will be loaded and the respective pagetype-TYPENAME component will be added to the content. Typically we add the rendering of the HTML header information here in the render function with the Helmet Stencil component.

Build and prerender

The basic setup of the page has been done, so if we just add one page and create a build with

npm run build

the Stencil compiler will tell us that its prerendering for / has been done and we will have a index.html in www with the contents from the template and hopefully the contents we have in /contents/index.json and those used in our template.

But this is not enough to enable the prerendering of all sites, the link structure needs to be set up. This means we need a menu in our template or just links in our contents.

The links need to be wrapped in a stencil-route-link component.

<stencil-route-link url="/about">
     <button>about</button>
 </stencil-route-link>

Conclusion

In a StencilJS application it’s possible to prerender whole websites. I did that for sites > 9000 pages and it works pretty well. The only downside I see at the moment is the time for the prerendering, but I think this problem will soon be solved. As @lilpwa told me, the guys from ionic are working on a solution to improve the performance of the rendering. The resulting HTML loads very, very fast and it’s easily crawlable by all search engine bots.

INSPIRATIONlabs is a studio for digital solutions and consulting located in Heidelberg, Germany. If you are interested in our work, come visit us, be it physically or digitally at http://www.inspirationlabs.com