JavaScript Frameworks & SEO Challenges

Search Engine Optimization (SEO)

Search engine optimization (SEO) is the practice of increasing the quantity and quality of traffic to your website through organic search engine results.— Moz

Google, Bing, Facebook, Twitter, and other social media sites rely on web crawlers to index your application content and make that content searchable on the web.

A Web Crawler is an automated script used to browse the World Wide Web.


How crawling of major Search Engines like Google work?

In early 2000’s pages were simply HTML based and there was little to no interactivity. But gradually JavaScript starting taking place and most of the content was loaded using Asynchronous JavaScript (AJAX etc.), and nowadays we have Single Page Applications (Angular, Vue, React, Ember etc.) that load only once and you can move back and forth between the pages without reloading it.

It sounds amazing RIGHT!? But it comes with its own challenges. SEO is one of the major issues with JavaScript frameworks (terminology is referred as JavaScript SEO).

Let’s see how traditional HTML is rendered by a Search Engine as compared to Single Page Application

Difference between HTML based and JS based website rendering

— Traditional HTML based website

  1. GoogleBot Fetches an HTML page
  2. Extracts links from the code fetched so that the links can be visited simultaneously.
  3. It then downloads CSS files
  4. Downloaded assets and resources are then sent to the Indexer
  5. And lastly the indexer indexes the page

This whole process is pretty straightforward and thus blazing fast

— JavaScript or SPA based website

JavaScript based website is comparatively more complicated to index

  1. GoogleBot Fetches an HTML page
  2. Downloads all the CSS and JS files
  3. GoogleBot then has to use Google’s WRS (Web Rendering Services) to parse, compile and execute JS code.
  4. Then WRS fetches the data from external APIs, from the database, etc.
  5. Finally, the indexer can index the content.
  6. Now Google can discover new links and add it to the GoogleBot's crawling queue.

The whole process is much more complicated than HTML crawling.


— GOOGLE

Since google is the leading Search Engine, we’ll discuss issues in context to Google mostly.

1. Will Google render my JavaScript?

Yes, Google claims that they are able to render the JavaScript based website but with some caveats.

Sometimes things don’t go perfectly during rendering, which may negatively impact search results for your site. — Google

2. JavaScript is Error-Sensitive

Unlike HTML if there is an error in your JavaScript code Google will not be able to render it, thus penalizing your website.

Initially in Angular2 Google used let keyword instead of var which was considered to be erroneous for the version of Chrome that Google uses for rendering i.e. Chrome 41.

So if Google wasn’t able to index their own site just because of a tiny mistake, imagine having those errors in your site and what Google could do to your website’s ranking.

3. Three(3) years old browser

As mentioned above Chrome 41 is being used by Google for indexing and rendering your website. There’s only partial support for ES6 features and some of the keywords might not work like let, const & class etc.

4. Using modern JavaScript features

To be able to use modern JavaScript you either need poly-fills or transpile your code down ES5 for better JS-SEO.

5. Caution! GoogleBot is not a real browser

GoogleBot doesn’t just goes on crawling your content and downloading everything that comes in its way. GoogleBot has to be smart and make decisions on whether to fetch a particular resource or not and whether it’s relevant or not.

A huge JS resource or some JS code that takes too long to execute might be ignored by the Bot completely, which might effect your sites’ ranking if that script contains some critical content of your site.

6. Canonical Tags via JavaScript

Changing canonical tags via JS is considered not reliable by Google. So make sure you have your canonical URLs in the HTML and not JS. Chances are Google may have addressed this issue but it’s just a chance, and when it comes to SEO one should not risk it unless dead sure.

7. Hashes in URL

JavaScript uses hashed URLs in the routes, this hash has its own advantages but in terms of SEO it’s considered extremely BAD!!

  • Bad: google.com/#/search/
  • Good: google.com/search/

8. Slow Websites

If downloading, parsing and rendering takes longer than 5 seconds due to a slow API or scripts, chances are google will either defer it or ignore the page completely and move on.

The rendering of JavaScript powered websites in Google Search is deferred until GoogleBot has resources available to process that content. — Google

— Bing & Other Search Engines

For a moment let’s consider Google is able to index your site perfectly what about the rest of Search Engines. Bing alone shares 33% of the US internet users searches as compared to Google.

So make sure you test your website for other Search Engines as well, to have better reach.


Server Side Rendering vs Client Side Rendering

Server Side Rendering: with SSR or Server Side Rendering the content is already there when the browser receives it. It means it’s not just easier for the bot to crawl but is extremely fast as it’s classical HTML approach.

Client Side Rendering: with CSR or Client Side Rendering the browsers have to struggle a bit as it might receive little or not content at all on initial load and fetch more content later with Asynchronously with JavaScript. With CSR you have to make sure your website is crawlable by the Bot.

Isomorphic Rendering vs Prerendering

Prerendering is a technique where you feed the crawler a snapshot version of your website HTML with no JavaScript when it visits your website and direct user to the normal website on the other hand. For this purpose you use external services like prerender.io or use tools like PhantomJS etc.

Isomorphic Rendering or Server Side Rendering is another approach in which the user and bot both have access to completely rendered content when they visit the site. All the JS features are available which makes it rich for the users and have all things a bot needs to have to index the page. It’s the most recommended option.

Conclusion

Crawlers and Search Engines are made capable enough to parse, render and index JavaScript based website like they crawl HTML based websites. But it’s up to the developers to make their websites obtainable and crawlable, and to understand how modern JavaScript websites and their SEO works. Each website is constructed with it’s own paradigms and it’s the job of the developers on how to best optimize it for Search Engines and Bots.

Thanks for reading! 😎