Progressive Web App — Search Engine Optimization in Shuvayatra App
Most resistance from most start-ups and developers when they’re asked why not build their web apps with single page app is because it’s not search engine friendly. They think that the page is not able to read by search engine crawler especially Googlebot due on client side rendering isn’t search engine friendly.
But most of them doesn’t know that Googlebot Crawler already able to parse Javascript and render the page just like your browser render it. Yes, there is still limitation since the browser and Javascript specs are iterate fast. But it’s not something that holds the developers to build their app with latest web technique which mostly now using client side rendering with libraries and frameworks like React, Polymer, Angular, Vue, Preact, Next, Skate, etc.
I’ve been helping a team at Nepal to build a Progressive Web App for Nepali Migrant Workers. The app called Shuvayatra, or safe travel in English. The app available with Android Native and Progressive Web App. Because the main feature of this app is about delivering information to Nepali migrant workers, so search engine optimization is a key to help migrant workers find the information they need when they are traveling abroad through a search engine. And this article will describe what experiments we’ve been doing with Shuvayatra to make the contents inside Shuvayatra PWA indexable by search engine and can be found by the migrant workers.
Facebook Sharing
We build the Shuvayatra PWA with Polymer. Previously we build it with Polymer 1.x but later we trying to convert it to Polymer 2.0 that just released early this year. The first issue that we encounter is Facebook sharing. When a user shares our article on Facebook, it will not return an overview of our article, like the cover image, and content overview. It’s only able to show the app title that not relevant to the content page.
Then we find out that Facebook crawler is work just like previous Google bot. It’s not able to render javascript, so it’s not able to read our content. And for this issue, the only solution is using server side rendering content. But instead using service like pre-render to render our PWA, or Chrome Headless and Phantom JS, we decided to build Accelerated Mobile Page and render it with server rendered NodeJS. The reason, we also think that having an AMP page for our content will help our users to load it faster on mobile and will help our search engine result too.
But our users not only use that AMP, sometimes our users also share the PWA version to Facebook. To make sure the Facebook Crawler able to read and return our overview content that user share on Facebook, we use NGINX redirect to redirect all bot request on our PWA to our AMP page. This is works perfectly, and the Facebook Share can read our content even the user post our PWA content link.
Make it indexable by Google
As usual, I always submit the website I build to Google Webmaster Tool to make sure Google Search aware and index the website. But after I submit it, I didn’t see good result on typing `site:shuvayatra.org` on Google Search. It does not return the results as expected. it’s only able to show one page result with the static pages like home, contact, and about. But all the articles do not show up on the result. This means Google didn’t index all the article.
Then I check our PWA through Google Search Console to see what happen. When I test it with Fetch as Google tool, it returns with a grey blank image without any content on it. Then I know that Google bot can’t see our PWA correctly. After trying to adjust different things, I found out that Google bot can’t parse our PWA when it’s built with ES6. After switch the build config on `polymer.json` to compile the script with ES5 compatible, Google finally able to see our PWA as expected.
Then I found a documentation how Google Bot works which is basically based on Chrome 41 rendering engine. So to see what API and features that could be read by Google Bot, you can check on Chrome Status but some limitation applied.
Make Googlebot Discover Shuvayatra’s Content
So after it’s readable and indexable by Google, we need to tell Google that we have content to index. Even you already submit the website, but it doesn’t mean that Googlebot know all our content that need to index. So we need generate a sitemap file to tell Google what content we have and where it exists.
To able generate a sitemap file, I wrote a script to parse our API and iterate through all pages and put all contents in to one sitemap file. Then we submit it to Google Search Console and wait Googlebot to come.
Structure Data with Rich Search Result
I’m thinking to have rich results when people search content with Shuvayatra. Like semantic web, we need to make Google understand what type of content that we’re trying to provide in the page. Is it an article, video, audio, or news. To deal with this, you need to put more structured data in your page. And if you can make it right, you will have a rich search result like below.
The way to make rich result like above is by put some structure data inside the page. But after trying to put some structure data based JSON-LD data format dynamically when it load the post page, I found out that it doesn’t work and no data detected even on source code I already put the data like this.
<script id="postMicroData" type="application/ld+json">
{
"@context":"http://schema.org",
"@type":"Article",
"mainEntityOfPage":{
"@type":"WebPage",
"@id":"https://shuvayatra.org"
},
"headline":"हप्तामा एकदिन तलबी बिदा दिनैपर्ने कतार सरकारको घोषणा ",
"image":{
"@type":"ImageObject",
"url":"https://api.shuvayatra.org/v1/uploads/posts/workers-heat-3jpg-771-514.jpg",
"height":800,
"width":800
},
"datePublished":"Thu Mar 30 2017",
"dateModified":"Mon May 01 2017",
"author":{
"@type":"Organization",
"name":"Shuvayatra Team"
},
"publisher":{
"@type":"Organization",
"name":"Shuvayatra",
"logo":{
"@type":"ImageObject",
"url":"https://shuvayatra.org/images/amp_logo.jpg",
"width":600,
"height":60
}
}
}
</script>When I tested it with Google Structure Data Testing Tool. It’s not show any data, means it’s doesn’t work at all. I still trying to find confirmation about this, is the PWA will support structure data that dynamically generated on client side or it only works on server rendered page.
Based on Google Structure Data Testing Tool no data shows up. So I still not sure if this testing tool using the same rendering engine with Fetch As Google tool.
Next Step
So after done all technical requirements that needed to make sure Google can read and index Shuvayatra’s content then the next homework is create more high quality and unique content where it will help to boost the search ranking result. Because make it readable and indexable is not enough to shows it up on search result.
