How to Configure Magento 2 robots.txt File

A Magento 2 robots.txt file tells search engine crawlers about the URLs that they can access on your Magento 2 site.

Mainly, a robot.txt file is used for managing crawler traffic to your site and to avoid crawling unimportant pages on your site. It also avoids overloading your site with requests. Thus, a Magento 2 robots.txt file can easily control the pages that crawlers can have access to.

In Magento 2, the robot.txt file is a simple Magento SEO configuration that is available by default. It is even used to hide some pages of your website from search crawlers. You may require to configure robots.txt in Magento 2 to hide some pages or URLs from the search engines.

Here, I have provided complete solution to configure Magento 2 robots.txt file that you can use.

How to Configure Magento 2 robots.txt file?

  1. Login to your admin panel.
  2. Go to Content > Design > Configuration.Magento 2 Configuration
  3. Select the store you want to configure the robot.txt file for.Select Store to Configure Magento robots.txt
  4. Find the search engine robots section and select the default robots of your website.
  5. Select the default robots that you want to enable. You can choose from either of the robots:
    • Index, follow – If you want the web crawlers to index a page and follow links on that page.
    • Noindex, follow – If you don’t want the web crawlers to index a page, but want them to follow links on that page.
    • Index, nofollow – If you want the web crawlers to index a page, but don’t want them to follow the links on that page.
    • Noindex, nofollow – If you want crawlers to neither index nor follow the links on that page.
  6. Enter the custom instruction you want to enter in the Edit custom instruction of the robot.txt file.
  7. To delete the custom instructions, click the Reset to Defaults button.
  8. At the end, Save the configuration.
  9. After saving the configuration, you can clear the cache.

Magento 2 robots.txt Location

Example of robots.txt file in Magento 2:

Some pages of your website can be hidden from crawlers by setting custom instructions in the Magento 2 robots txt file.

Magento 2 robots.txt example:

User-agent:*
Disallow:

Thus, this article will help you to learn not only to create but also to configure the robots.txt files in Magento 2.

Having a thorough sitemap is essential for maintaining a successful website. So, improve your website’s SEO with Magento 2 XML Sitemap.

I’d be pleased to solve any doubts you face in creating Magento 2 robots.txt file for your Magento store. Do let me know in the comments section below.

Thank you.

5
(based on 6 Reviews)
How to Configure Magento 2 robots.txt File

Ronak Bhatti

Ronak is a Magento Developer at Meetanshi. He loves playing cricket. Apart from the work, he is an enthusiastic person who loves learning new skills.

Leave a Reply

Your email address will not be published.