How to Get Google to Index Your Website
The day finally comes. You get that much awaited email from your developer saying that your big, shiny new website has been launched out into the World Wide Web.
Before breaking out the champagne, you rush over to Google, excited to search for your business and to find your brand new digital home awaiting you.
Only one problem.
You type in your business name and your new website is nowhere to be found.
Before you panic and begin frantically writing an email to your developer, here’s something you need to know.
Before your website can show up in the search engine, Google needs to index it (we’ll get to what this means). This takes time (not what you want to hear). Not even Google can tell you how long it will take but generally speaking, it can take anywhere from 4 days to 4 weeks.
Don’t bail on me now; I come bearing solutions. There are things you can do to speed up the process.
But first, what is Google indexing?
Indexing is the process of adding webpages into Google search. Just like a book with individual chapters, Google has an index to guide your internet browsing.
How does Google do this?
With super smart virtual critters called bots. Think of them as Google’s highly trained virtual spiders that crawl around the internet, creating a web of information that they report back to Google. Why does Google send these bots out after our websites? Because Google is user-focused. They strive to ensure their users have the most useful and up-to-date information. In order for Google to deem you, your business, and your website as useful and relevant, it needs time to crawl your pages.
What do the spiders look for?
They start with the most important information:
URLs — The address. This is pretty important. Otherwise, no one will ever be able to find your website in the first place.
Title tags — The name of your website.
Metadata — The description of your website, as well as any relevant keywords.
This is the main information that the spiders retrieve for Google’s index. And this is what you see in an index.
This explainer video by the former head of the web spam team at Google, Matt Cutts sums it up nicely.
Now on to the solutions I promised you…
How to get Google to index your website
1. Create a sitemap
A sitemap is an XML document on your website’s server that basically lists each page of your website. It lets search engines know when new pages have been added and how often they should check back for updates on specific pages. For example, you might want a search engine to come back and check your homepage daily for new products, news items or other new content.
If your website is built in WordPress, you can install the Google XML Sitemaps plugin and have it automatically create and update your sitemap for you as well as submit it to search engines.
2. Submit your sitemap to Google Search Console (previously, Google Webmaster Tools)
Once you have your sitemap created, take it on over to Google Search Console. If you don’t already have one, all you have to do is create a free Google Account and then sign up for Webmaster Tools. You add your new site to Webmaster Tools.
Then, go to Optimization > Sitemaps and add the link to your website’s sitemap to Webmaster Tools to notify Google about it and the pages you have already published.
3. Create a Robots.txt
The robots.txt is a file on your website that tells search engines what they should and should not index.
If your robots.txt doesn’t say, “you can index me,” giving Google unrestricted permission to crawl the site, then the spider will move along.
What would you not want Google to index? Admin pages, for example. If you update your content in WordPress, that would be the wp-admin URL which you wouldn’t want the public to be able to find when searching for your business.
4. Install Google analytics
You’ll want to do this for tracking purposes regardless, but it definitely helps advise Google that a new website has been added to the mix.
5. Create internal links
The paths that the spiders take are formed by links. When one page links to another page, the spider follows that path.
Within your own site, make sure that you’ve created links to and from all your most important pages.
This should happen naturally if your site is well organized.
It’s helpful to link between as many pages as possible. For example, linking one blog article to another one, or from a blog article to relevant content page on your website.
6. Create a blog
Create a blog and draw in the spiders by producing timely and relevant content for them to crawl and Google to index. The more high-quality content you put out there, the more indexation you get, and the more relevant you will be to Google and its users.
A blog is just one of a few crucial components to your online success. Before you open your virtual doors, make sure your digital home is up to speed with our 9 Website Must-haves.
7. Check for crawl errors
In the event that an error does occur (it happens), having a witty 404 error message like the one above will, at the very least, briefly entertain the user and divert their frustration at not finding the content they were searching for.
A website can have issues with crawling when you make significant changes to your website such as adding, removing, or moving pages.
To access your crawl report, open up Google Search Console and select your website.
Click on Crawl.
Click Crawl Errors.
You’ll see the errors that need to be corrected.
Once your website is indexed, you’ll start to see more traffic from Google search.
However, it’s important to remember that getting your website indexed by Google is only the first step to having a successful website. Once your site has been indexed, you want to prove to the spiders (and Google) that you’re an invaluable member of the internet society by creating valuable content regularly.
Above all, have patience.
Remember that these things take time.
Do what you can on your end and if you don’t want to worry about all this yourself, make sure you count on someone who will.