How to get your website indexed instantly at Google using Sitemaps & Robots.txt

One of my friends met me a few weeks back and showed a website showcasing her newly launched coffee house business. It was clean and crisp website filled with loads of images and adequate information about their shop which also included their menu items and I was impressed by the entire presentation. I was told that the website contained 15 static pages with approx 45 high quality images and its been almost of a month they went online but surprisingly it was NOT indexed by Google.

Being novice towards the digital marketing or SEO, my friend asked me the following questions:

1. How long does it take to get my website indexed at Google?
2. Do I need to contact Google by email or phone in order to get indexed?
3. Does Google charge money in order to get indexed in their search engine?
4. Is Indexing depending upon the total number of pages in the website?
5. Some of their high resolution images were sized more than 1MB. Did that prevent Google from getting indexed?

There are thousands of startup businesses across the world which enters the digital world on a daily basis and they also come across a similar number of queries while getting indexed at search engines like Google or Bing.

How to check if your website is indexed in Google?

Firstly, you DO NOT need to be technically savvy in order to find out whether your website is indexed in Google. Just open http://www.google.com in your web browser i.e., Internet Explorer, Firefox, Google Chrome or anything else and enter the query “site:yourfullwebsitename” like for example “site:creativemoz.com” or “site:http://www.creativemoz.com” Once you search with this query, you will find a list of all the web pages in your website being currently indexed at Google.

In my friend’s case, Google showed this following statement:

“A description for this result is not available because of this site’s robots.txt”

Now I got the answer to my friend’s first question 🙂 It so happened that while developing the website, the web designer had accidentally created a robots.txt file in the web server and had written a small piece of code which had been blocking Google to index the website. This was done because the website had loads of placeholder images and text content and the web designer did not want Google to index the website with those junk or irrelevant content and hamper its first impression about the website.

So! What’s next?

We immediately removed the piece of code which blocked Google from indexing the website and then went ahead to create a sitemap.xml which helps in speeding up the process of getting indexed almost instantly.

What is sitemap.xml? How does it help in getting indexed in Google?

According to Wikipedia, Sitemap.xml is

A Sitemap is an XML file that lists the URLs for a site. It allows webmasters to include additional information about each URL: when it was last updated, how often it changes, and how important it is in relation to other URLs in the site. This allows search engines to crawl the site more intelligently.

A sitemap is like a blue print of a building which provides information about the number of rooms, doors, windows and all the structural details of navigating across the entire building. For search engines, it is a file which is .XML in format like you have your documents saved in Word and is .doc format. This file contains the list of the URLs in the website which obviously includes the homepage of the website and has to be updated whenever you add or remove any web page from the website.

How do i create sitemap.xml?

If your website contains 500 or lesser number of webpages (not counting images or videos) then I would suggest visiting https://www.xml-sitemaps.com/ I’ve been creating free XML sitemaps from this website since a long time and its very easy to create the sitemap here. You just need to execute the following steps in order to get it done:

1. Enter your full website URL and some optional parameters in the form provided in the form provided at their website.
2. Press ‘Start’ button and wait until the site is completely crawled. Takes a few seconds if you have a website containing lesser than 50 webpages. Will take longer time if you got a big website.
3. Once your sitemap file is created, you will be redirected to the generated sitemap details page, including number of pages, broken links list, XML file content and link to a sitemap file.
4. Download the sitemap file using that link and put it into the domain root folder of your site. In most of the cases, its inside the “public_html” folder in your website hosting.
5. Then Go to your Google Search Console account and add your sitemap full URL. For example: http://www.example.com/sitemap.xml

Got a big website with more than 500 web pages? No worries. They do have paid plans which allows hassle-free creation of XML Sitemaps for more number of pages.

Sitemaps.xml in WordPress Platform

Below are some of the plugins that I love and trusted while developing sitemap.xml files in WordPress.org platform and they are FREE to use:

1. XML Sitemaps by Arne Brachhold — https://wordpress.org/plugins/google-sitemap-generator/

1. This plugin has been rated as the best WordPress plugin This plugin was developed to create a XML sitemap file which can be compatible to various search engines like Ask.com, Bing.com, Yahoo.com and of-course Google.com
2. XML Sitemap by Yoast — https://wordpress.org/plugins/wordpress-seo/

1. I like this XML Sitemap plugin because of its advanced features which lets you choose the types of web pages you want to be indexed i.e., Web Pages or Posts. Its easy to use control panels can be handled by anyone without having much technical expertise. The Yoast SEO plugin also goes the extra mile to take care of all the technical optimization and foremost helps you write better content. Yoast SEO forces you to choose a focus keyword when you’re writing your articles, and then makes sure you use that focus keyword everywhere.

RSS, mRSS, and Atom 1.0

If you have a blog with an RSS or Atom feed, you can submit the feed’s URL as a sitemap. Most blog software is able to create a feed for you, but recognize that this feed only provides information on recent URLs.

• Google accepts RSS (Real Simple Syndication) 2.0 and Atom 1.0 feeds.
• You can use an mRSS (media RSS) feed to provide Google details about video content on your site.

Text Sitemaps

If your website’s sitemap includes only web page URLs, you can provide Google with a simple text file that contains one URL per line. For example:

http://www.example.com/file1.html http://www.example.com/file2.html

Guidelines for text file sitemaps:

• Encode your file using UTF-8 encoding.
• Your text file should contain nothing but the list of URLs.
• You can name the text file anything you wish, provided it has a .txt extension (for instance, sitemap.txt).

Sitemaps for Websites powered by Google Sites

If you’ve created and verified a site using Google Sites, Sites will automatically generate a sitemap for you. You cannot modify the sitemap, but you can submit it to Google if you want to read the sitemap report data. Note that your sitemap might not be displayed properly if you have more than 1,000 pages in a single sub-directory.

• If your site is hosted at Google Sites, your sitemap URL is http://sites.google.com/site/yoursitename/system/feeds/sitemap
• If you created your site using Google Apps, your sitemap URL is http://sites.google.com/yourdomain/yoursitename/system/feeds/sitemap

A well structured Sitemap can do even more than just becoming a blue print of your website. A well planned and documented sitemap file can inform the following types of information to search engines which can further help your website get better rankings:

1. Informs the full webpage address (i.e., URL): <loc>http://www.example.com/mypage</loc>
2. When a particular page was last changed: <lastmod>2017–03–02</lastmod>
3. Frequency in which the page is updated: <changefreq>monthly</changefreq>
4. What priority the page has in relationship to other pages on the website: <priority>1</priority>

Google Search Console also informs you about the total number of pages you have submitted and how many are being currently indexed through your Sitemap file.

With that said, I would like to add that Sitemaps are just one of the way in which you as a website owner or webmaster can inform Google about the number of pages in your website. However, Google bots might crawl through your website and automatically fetch a list of web pages in a website based upon its inter linking.

Getting back to my Friend’s Questions

1. How long does it take to get my website indexed at Google?
   A: Earlier it used to take upto 7 days. These days, you can expect your website/ blog to be indexed within 72 hours if you havent blocked it using robots.txt & have created and uploaded the sitemap file properly in your web server.
2. Do I need to contact Google by email or phone in order to get indexed?
   A: Sign up using a google email account i.e., gmail account or incase you have Google Apps email to Google Search Console (previously known as Google Webmasters Tool) and submit your website over there.
3. Does Google charge money in order to get indexed in their search engine?
   A: No, not, never. Google never charged anyone for getting their website or blog indexed in their search engine.
4. Is Indexing depending upon the total number of pages in the website?
   A: Yes, if you have a big website with thousands of pages, then you need to wait for sometime to get all your pages indexed. Interestingly you can see the progress details at Google Search Console. So you can learn if Google bots are facing any issues while accessing the sitemap file or indexing your web pages.
5. Some of their high resolution images were sized more than 1MB. Did that prevent Google from getting indexed?
   A: No. Google and other search engines love high resolution images which are properly tagged with Alt Tags. Having said that, I would personally advise the website images to be in JPG or JPEG format if they dont contain any animated content because JPG or JPEG files generally smaller in file size and will consume lower disk space of your webserver. Also try to accomodate your images within 1 or 2 MB size to make it browser friendly, mobile/ tablet/ desktop device friendly and will take lesser time to load for users having lower internet speed.

Liked or Disliked this article? I would be eager to know your feedback. Do feel free to comment below or email me at rahul[at]creativemoz.com if you have any queries about getting indexed in Google or any search engine and I will be happy to revert back to you.

Originally published at www.creativemoz.com on April 3, 2017.