Edge Delivery Services Sitemap and robots.txt

As part of this blog, we will try to get overall undertaking around sitemap and rotbots.txt implementation in Edge Delivery Services.

Sitemap

In Edge Delivery Services, Sitemaps are based on indexes. For automatic sitemap generation, please ensure that you have already added the query-index document as explained in the indexing document.

If we already followed steps for setting up an initial query index in the document indexing, the system will generate a sitemap at the location:

https://<branch>--<repo>--<owner>.hlx.page/sitemap.xml

And a sitemap configuration at the following location:

https://<branch>--<repo>--<owner>.hlx.page/helix-sitemap.yaml

Simple Sitemap

The following is a simple helix-sitemap.yaml. It assumes a single index containing all the pages that need to appear in the sitemap.

sitemaps:
   example:
     source: /query-index.json
     destination: /sitemap-en.xml

If you want last modification dates to be included in the URLs of your sitemap, add a lastmod property including a format to your configuration.

sitemaps:
   example:
     source: /query-index.json
     destination: /sitemap-en.xml
     lastmod: YYYY-MM-DD

Multiple Sitemaps

It is common to have sitemaps per section of the sites and/or per country or language. AEM supports sitemaps including the corresponding hreflang references. In the following example we assume that there is a one to one mapping between the indexes and the sitemaps XML files.

sitemaps:
   example:
     languages:
       en:
         source: /en/query-index.json
         destination: /sitemap-en.xml
         hreflang: en
       fr:
         source: /fr/query-index.json
         destination: /sitemap-fr.xml
         hreflang: fr
         alternate: /fr/{path}

Note: HREFLang is supported as part of sitemaps (sitemap.xml) and can be configured in a flexible manner to reference different locales where needed.

Robots.txt

We can add robots.txt as part of our code in root folder as shown below:

Edge Delivery Services Overview
Development
Personalized Google Drive
Sidekick Chrome Extension
Page Hierarchy
Local environment Setup
Page HTML Structure
Section Metadata
Metadata
Redirects
Response Headers
Access excel data in JSON format
Fragment
Forms
Indexing
Services Sitemap and robots.txt
Code Structure
Card Block Example

I hope you found out this article interesting and informative. Please share it with your friends to spread the knowledge.

You can follow me for upcoming blogs follow.
Thank you!