{"id":415,"date":"2019-05-15T08:25:34","date_gmt":"2019-05-15T08:25:34","guid":{"rendered":"https:\/\/meetanshi.com\/blog\/2019\/05\/15\/magento-2-robots-txt-meta-robots\/"},"modified":"2025-07-23T09:53:23","modified_gmt":"2025-07-23T04:23:23","slug":"magento-2-robots-txt-meta-robots","status":"publish","type":"post","link":"https:\/\/meetanshi.com\/blog\/magento-2-robots-txt-meta-robots\/","title":{"rendered":"How to Configure Robots.txt and Meta Robots for Magento 2"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">If you are reading the post, it means either you are a Magento 2 store owner or is responsible for the SEO of the store. You must be aware of the importance of Magento robots.txt file. It is a text file that instructs the web robots, i.e., search engines which pages to crawl and which not!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the words of <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener noreferrer\" data-cke-saved-href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" data-cke-saved->Google<\/a>,<br><i>&#8220;A robots.txt file tells search engine crawlers which pages or files the crawler can or can&#8217;t request from your site.&#8221;<\/i><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This tiny file is a part of EVERY website, and Magento 2 stores are no exception. Fortunately, the default Magento 2 supports creating a robots.txt file and I&#8217;ll show how to do so here.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Meta robots are a way for webmasters to offer search engines details about their stores. It is a piece of code in the &nbsp;&lt;head&gt; section of your webpage that tells the search engines what to follow and what not to! Moreover, it tells the crawlers what links to follow and what links to stop with.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Create Magento 2 Robots.txt file and assist your SEO efforts because:<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Controls how the search engine spiders see and interact with the pages of Magento 2 store.<\/li>\n\n\n\n<li>Its improper use can affect the rankings of your store negatively.<\/li>\n\n\n\n<li><a href=\"https:\/\/meetanshi.com\/free-tools\">Robots.txt file<\/a> is the foundation of how the search engine works.<\/li>\n\n\n\n<li>Restrict duplicate content pages from appearing in SERPs.<\/li>\n\n\n\n<li>It helps to avoid overloading your store with requests from Google&#8217;s crawler.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Now that we&#8217;ve talked enough about robots.txt and its importance for Magento 2 store, let&#8217;s create one!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Steps to create Magento 2 Robots.txt:<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">1. Log in to Admin Panel.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">2. Navigate to <strong>Content &gt; Design &gt; Configuration<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2019\/05\/1_Global-edit.png\" alt=\"1_Global edit\" class=\"wp-image-5603\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">3. Expand the <strong>Search Engine Robots<\/strong>.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Default Robots:<\/strong> Set&nbsp;to one from the drop-down.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Edit Custom Instruction of Robots.txt File<\/strong>: Enter any custom details if needed in this field.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Reset to Default: <\/strong>Use this button to set the Magento 2 Robots.txt file to its default configuration.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2019\/05\/2_Global-Search-Engine-Robots.png\" alt=\"set Meta Robots from Magento admin panel\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Save<\/strong> the Configuration<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The robots exclusion protocol or standard, also known as Robots.txt can be configured in default Magento 2 with the above steps.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why Should I Set NoIndex NoFollow Tags to Links in Magento 2:<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For example, you are launching a new product in your Magento 2 store. However, your team is still working on it. For time being, you can set that product page to NoIndex to tell the search engine to not index that page.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this way, you can test the changes in the live store and at the same time restrict the search engine to index it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Also, the NoFollow tags to links can be useful when you want to offer any additional information that is located on a particular web address but not pass the link equity.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Meta robots tags: NOINDEX, NOFOLLOW<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now that you have created robots.txt successfully, pay attention to the meta robots tags. Cover the unnecessary parts of the code from crawlers using these tags.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>No Index&nbsp;is an attribute of tag, that restricts the transfer of the page weight to a non certified source. Moreover, it can be used for pages with a large number of external links.<\/li>\n\n\n\n<li>No Follow hides the page from indexation<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Apply Nofollow or Noindex to your configuration by either updating the robots.txt file or using meta name=\u201crobots\u201d tag.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Possible Combinations:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>INDEX, FOLLOW: <\/strong>Instructs the web crawlers to index the store and check back later for the changes.<\/li>\n\n\n\n<li><strong>NOINDEX, FOLLOW: <\/strong>Instructs the web crawlers not to index the store but check back later.<\/li>\n\n\n\n<li><strong>INDEX, NOFOLLOW: <\/strong>Instructs the web crawlers to index the site once and avoid checking back later.<\/li>\n\n\n\n<li><strong>NOINDEX, NOFOLLOW:&nbsp;<\/strong>Instructs the web crawlers not to index the site and also not check back later<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Add the following code to the robots.txt file in order to hide specific pages:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: *\nDisallow: \/myfile.html<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Alternatively, you can restrict indexation with the following code:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;html &gt;\n&lt;head &gt;\n&lt;meta name=\u201drobots\u201d content=\u201dnoindex, follow\u201d\/ &gt;\n&lt;title&gt;page title&gt;\n&lt;\/head &gt;<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">The recommended default settings for Magento 2:<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">User-agent: *\n\n# Directories\nDisallow: \/app\/\nDisallow: \/bin\/\nDisallow: \/dev\/\nDisallow: \/lib\/\nDisallow: \/phpserver\/\nDisallow: \/pkginfo\/\nDisallow: \/report\/\nDisallow: \/setup\/\nDisallow: \/update\/\nDisallow: \/var\/\nDisallow: \/vendor\/\n\n# Paths (clean URLs)\nDisallow: \/index.php\/\nDisallow: \/catalog\/product_compare\/\nDisallow: \/catalog\/category\/view\/\nDisallow: \/catalog\/product\/view\/\nDisallow: \/catalogsearch\/\nDisallow: \/checkout\/\nDisallow: \/control\/\nDisallow: \/contacts\/\nDisallow: \/customer\/\nDisallow: \/customize\/\nDisallow: \/newsletter\/\nDisallow: \/review\/\nDisallow: \/sendfriend\/\nDisallow: \/wishlist\/\n\n# Files\nDisallow: \/composer.json\nDisallow: \/composer.lock\nDisallow: \/CONTRIBUTING.md\nDisallow: \/CONTRIBUTOR_LICENSE_AGREEMENT.html\nDisallow: \/COPYING.txt\nDisallow: \/Gruntfile.js\nDisallow: \/LICENSE.txt\nDisallow: \/LICENSE_AFL.txt\nDisallow: \/nginx.conf.sample\nDisallow: \/package.json\nDisallow: \/php.ini.sample\nDisallow: \/RELEASE_NOTES.txt\n\n# Do not index pages that are sorted or filtered.\nDisallow: \/*?*product_list_mode=\nDisallow: \/*?*product_list_order=\nDisallow: \/*?*product_list_limit=\nDisallow: \/*?*product_list_dir=\n\n# Do not index session ID\nDisallow: \/*?SID=\nDisallow: \/*?\nDisallow: \/*.php$\n\n# CVS, SVN directory and dump files\nDisallow: \/*.CVS\nDisallow: \/*.Zip$\nDisallow: \/*.Svn$\nDisallow: \/*.Idea$\nDisallow: \/*.Sql$\nDisallow: \/*.Tgz$<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Check these frequently used Magento 2 robots.txt file examples:<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Allow full access to all directories and pages:<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">User-agent:*\nDisallow:<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Don\u2019t allow access for any user-agent to any directory and page:<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">User-agent:*\nDisallow: \/<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Default Instructions:<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">Disallow: \/lib\/\nDisallow: \/*.php$\nDisallow: \/pkginfo\/\nDisallow: \/report\/\nDisallow: \/var\/\nDisallow: \/catalog\/\nDisallow: \/customer\/\nDisallow: \/sendfriend\/\nDisallow: \/review\/\nDisallow: \/*SID=<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Restrict User Accounts and Checkout Pages<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">Disallow: \/checkout\/\nDisallow: \/onestepcheckout\/\nDisallow: \/customer\/\nDisallow: \/customer\/account\/\nDisallow: \/customer\/account\/login\/<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To disallow duplicate content<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">Disallow: \/tag\/\nDisallow: \/review<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To disallow CMS directories<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">Disallow: \/app\/\nDisallow: \/bin\/\nDisallow: \/dev\/\nDisallow: \/lib\/\nDisallow: \/phpserver\/\nDisallow: \/pub\/<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To disallow Catalog and Search Pages<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">Disallow: \/catalogsearch\/\nDisallow: \/catalog\/product_compare\/\nDisallow: \/catalog\/category\/view\/\nDisallow: \/catalog\/product\/view\/<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To disallow URL Filter Searches<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">Disallow: \/*?dir*\nDisallow: \/*?dir=desc\nDisallow: \/*?dir=asc\nDisallow: \/*?limit=all\nDisallow: \/*?mode*<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"common-web-crawlers-bots\">Common Web crawlersL:<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Here are some common bots on the internet:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">User-agent: Googlebot\nUser-agent: Googlebot-Image\/1.0\nUser-agent: Googlebot-Video\/1.0\nUser-agent: Bingbot\nUser-agent: Slurp\t\t# Yahoo\nUser-agent: DuckDuckBot\nUser-agent: Baiduspider\nUser-agent: YandexBot\nUser-agent: facebot\t\t# Facebook\nUser-agent: ia_archiver\t\t# Alexa<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Communicate with search engines the correct way! After the successful&nbsp; Magento robots.txt creation, you can check for the validation of it using <a href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\" target=\"_blank\" rel=\"noopener noreferrer\" data-cke-saved-href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\" data-cke-saved->Google&#8217;s robots.txt Tester<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><i><strong>Create Magento 2 Robots.txt<\/strong><\/i> file and improve your SEO today!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Do rate the post with 5 stars if it was helpful.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thanks  <\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you are reading the post, it means either you are a Magento 2 store owner or is responsible for the SEO of the store&#8230;.<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[34],"tags":[],"class_list":["post-415","post","type-post","status-publish","format-standard","hentry","category-magento"],"acf":[],"_links":{"self":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts\/415","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/comments?post=415"}],"version-history":[{"count":5,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts\/415\/revisions"}],"predecessor-version":[{"id":19104,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts\/415\/revisions\/19104"}],"wp:attachment":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/media?parent=415"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/categories?post=415"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/tags?post=415"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}