{"id":3596,"date":"2024-08-28T12:24:52","date_gmt":"2024-08-28T12:24:52","guid":{"rendered":"https:\/\/meetanshi.com\/blog\/shopify-robots-txt\/"},"modified":"2026-02-12T12:35:57","modified_gmt":"2026-02-12T07:05:57","slug":"shopify-robots-txt","status":"publish","type":"post","link":"https:\/\/meetanshi.com\/blog\/shopify-robots-txt\/","title":{"rendered":"Shopify Robots.txt: What It Is and How To Configure"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">If you are struggling to get organic traffic then chances are the search engine bots have not crawled important pages of your Shopify store yet.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Your Shopify robots.txt file plays a key role to play here.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this blog, learn in detail about robots.txt file in Shopify and how you can edit it for SEO benefits.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What is Shopify Robots.txt?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Robots.txt is a file that tells search engine crawlers\/bots which pages they must access from your website and which they must avoid to manage its visibility on the Search Engine Result Pages (SERPs).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Think of the file as a guiding map, telling the bot where to go and where to not go.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can easily check the robots.txt file of your store or any website online with a simple \/robots.txt at the end.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here is an example.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"972\" height=\"672\" src=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/Robots_txt_file_example_.png\" alt=\"Robots txt file example on live site\" class=\"wp-image-25714\" srcset=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/Robots_txt_file_example_.png 972w, https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/Robots_txt_file_example_-300x207.png 300w, https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/Robots_txt_file_example_-768x531.png 768w\" sizes=\"auto, (max-width: 972px) 100vw, 972px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The good news is Shopify provides a default robots.txt file using the dynamic Liquid template called robots.txt.liquid. The file is a mix of plain text and Liquid code.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The search bots generally check the file before crawling any of your store&#8217;s web pages. While the bots are not forced to adhere to the file, many popular search bots do refer to it before crawling.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Customize Robots.txt in Shopify<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Since Shopify creates a default robots.txt file for you, there is no need to create and add one to your store.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can customize it based on preference.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In some cases, Shopify\u2019s default rules might disallow crawling for pages you actually want to rank. Such rules prevent search engines from seeing your newest content and potentially limiting your organic traffic.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What can you change in the Shopify Robots.txt file?&nbsp;<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">There are three important changes you can make in your file.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Add a new rule to existing rules<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">This implies you would like to add new rules to the default set of rules.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You will add the conditions for which new rules will be applicable. If you want to update different bot settings for other search engines, you can also update them here.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Condition<\/strong><\/td><td><strong>Meaning<\/strong><\/td><td><strong>Example<\/strong><\/td><\/tr><tr><td>Disallow: \/<\/td><td>This tells crawlers to stay away from your entire website.<\/td><td>Used mainly for staging sites or during a major site maintenance.<\/td><\/tr><tr><td>Disallow: [path]<\/td><td>This blocks crawlers from a specific folder or group of pages.<\/td><td>Disallow: \/collections\/ blocks all collection pages.<\/td><\/tr><tr><td>Allow: \/<\/td><td>This explicitly gives crawlers access to your entire site (overriding broad blocks).<\/td><td>Usually used to &#8220;reset&#8221; access if a previous rule was too strict.<\/td><\/tr><tr><td>Allow: [path]<\/td><td>This opens a specific door inside a section that is otherwise blocked.<\/td><td>Allow: \/collections\/bestsellers opens one collection even if others are blocked.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Remove a rule from existing rules<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Although it\u2019s not recommended practice from Shopify if you want to remove any existing rules from the default robots.txt.liquid file, you need to mention the code for the same accurately.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">{% for rule in group.rules -%} {%- unless rule.directive == 'Disallow' and (rule.value == '\/search' or rule.value == '\/blogs') -%} {{ rule }} {%- endunless -%} {% endfor -%}<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Add custom rules<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to add new rules that aren\u2019t in the default group, you can update those changes at the bottom of the file template.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Shopify Robots.txt Example<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When you create your robots.txt.liquid file, Shopify pre-fills it with a Liquid loop. You can make your own rules, you should insert them into that loop.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here is a comprehensive Shopify robots.txt example of a customized file that includes:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The Default Shopify Rules.<\/li>\n\n\n\n<li>Custom Disallows.<\/li>\n\n\n\n<li>A Custom Allow.<\/li>\n\n\n\n<li>AI Bot Blocking.<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">{%- comment -%}\n# This is a customized Shopify robots.txt.liquid template.\n# It includes default rules plus custom SEO tweaks.\n{%- endcomment -%}\n\n{% for group in robots.default_groups %}\n {{- group.user_agent -}}\n\n {% for rule in group.rules %}\n {%- comment -%}\n REMOVE RULE EXAMPLE: Uncomment below to 'unblock' policies\n {%- unless rule.directive == 'Disallow' and rule.value == '\/policies\/' -%}\n {%- endcomment -%}\n\n {{- rule -}}\n\n {%- comment -%}\n {%- endunless -%}\n {%- endcomment -%}\n {% endfor %}\n\n {%- if group.user_agent.value == '*' -%}\n {{ 'Disallow: \/blogs\/*\/tagged' }}\n {{ 'Disallow: \/collections\/vendors*?*q=' }}\n {{ 'Allow: \/collections\/bestsellers' }}\n {%- endif -%}\n\n {%- if group.sitemap != blank -%}\n {{ group.sitemap }}\n {%- endif -%}\n{% endfor %}\n\n# BLOCK AI CRAWLERS (Added at the very bottom)\nUser-agent: GPTBot\nDisallow: \/\nUser-agent: CCBot\nDisallow: \/\n<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">How to Edit Robots.txt in Shopify (Two Methods)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Method 1: Manually Edit<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Since the robots.txt is a generated file, you don&#8217;t find it in your folders. You must create a Shopify robots.txt.liquid template that overrides the default file.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Open your Shopify admin head to the Online Store.<\/li>\n\n\n\n<li>Under the <strong>Themes<\/strong> section, find your <strong>Live theme<\/strong>.<\/li>\n\n\n\n<li>Click the <strong>three dots (&#8230;)<\/strong> or <strong>Actions<\/strong> button and select <strong>Edit code<\/strong>.<\/li>\n\n\n\n<li>In the left sidebar, scroll to the <strong>Templates<\/strong> folder and click <strong>Add a new template<\/strong>.<\/li>\n\n\n\n<li>A pop-up will appear. In the first dropdown, select <strong>robots.txt<\/strong>.<\/li>\n\n\n\n<li>Click <strong>Create Template<\/strong> and add the needed changes.<\/li>\n\n\n\n<li>Save the file.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Once you create a template, you will get a default Shopify robot.txt.liquid template ready. It\u2019s good to go for SEO purposes.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter\"><img decoding=\"async\" src=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2024\/08\/Shopify-robot.txt-liquid-template.png\" alt=\"Shopify robot.txt liquid template \" class=\"wp-image-41553\"\/><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">However, you can update the code here if you wish to make any amendments.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Method 2: Using Shopify Robots.txt App&nbsp;<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Use the <a href=\"https:\/\/apps.shopify.com\/mit-custom-robots-txt-rules\" target=\"_blank\" rel=\"noopener\">MIT Custom Robots.txt Rules<\/a> Shopify app to easily edit the default file, it manages the creation and updating of the file for you, providing a user-friendly interface so you don&#8217;t have to touch any Liquid code.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It directly extracts the default file in the Shopify admin, ready to edit using the structured editor or the raw file.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"400\" src=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/MIT_Custom_Robots_txt_Rules_Shopify_app-1024x400.png\" alt=\"MIT Custom Robotx txt app for Shopify\" class=\"wp-image-25713\" srcset=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/MIT_Custom_Robots_txt_Rules_Shopify_app-1024x400.png 1024w, https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/MIT_Custom_Robots_txt_Rules_Shopify_app-300x117.png 300w, https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/MIT_Custom_Robots_txt_Rules_Shopify_app-768x300.png 768w, https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/MIT_Custom_Robots_txt_Rules_Shopify_app-1536x600.png 1536w, https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2026\/02\/MIT_Custom_Robots_txt_Rules_Shopify_app-2048x800.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The app manages the creation and updating of the file for you, providing a user-friendly interface so you don&#8217;t have to touch any Liquid code.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Pick the format you are comfortable with and add user-agent groups or edit directives (Allow, Disallow, or Crawl-delay).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Additionally, you can easily perform:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Undo &#8211; Reverse the last performed action.<\/li>\n\n\n\n<li>Redo &#8211; Restore the change you just reversed.<\/li>\n\n\n\n<li>Reset to Default &#8211; Revert the file back to the original default file.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The app completely manages the workflow: simply make your Shopify robots.txt edit, save them, and it directly updates the file on Shopify, making it live for the next crawl.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Test &amp; Verify Your Shopify Robots.txt File<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s tricky to look at the file after making the needed changes. Even a small syntax error can impact the functioning of your site and block unnecessary pages.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The best way to verify your file is using the <a href=\"https:\/\/meetanshi.com\/blog\/add-google-search-console-to-shopify\/\">Shopify Google Search Console<\/a> dashboard.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Go to Settings &gt; Crawling &gt; robots.txt. Then, click on Open Report.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">From here, you can verify whether the robots.txt file is correctly implemented for your store or not.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter\"><img decoding=\"async\" src=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2024\/08\/Test-Verify-Your-Shopify-Robots.txt-File.png\" alt=\"Test Verify Your Shopify Robots.txt File\" class=\"wp-image-41553\"\/><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">You can check the issues in this report and fix any errors.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Best Practices for Configuring Robots.Txt File in Shopify<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Understand which website pages should be accessible to search engines to minimize the crawl budget for your online store.<\/li>\n\n\n\n<li>Use \u2018Disallow\u2019 directives to block access to the irrelevant pages of your store, like admin pages, checkout, cart, or any duplicate content. Here\u2019s how you can restrict access or <a href=\"https:\/\/meetanshi.com\/blog\/how-to-fix-shopify-duplicate-content\/\">fix duplicate content<\/a> pages based on filters.<\/li>\n\n\n\n<li>In the robots.txt file, include a link to the sitemap to help search engines find all the pages of your online store.<\/li>\n\n\n\n<li>When you block search engine bots, be specific about managing how different search engine bots will access your site.<\/li>\n\n\n\n<li>Verify the updates you\u2019ve made in robots.txt before implementation. Using the Google Search console tester tool, you can check whether the file\u2019s syntax is correct and functioning as expected.<\/li>\n\n\n\n<li>Avoid blocking too many URLs and use comments to clarify wherever necessary to explain the purpose of directives. This will be helpful for your team and future reference.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently Asked Questions&nbsp;<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. How to fix &#8220;Blocked by robots.txt&#8221; on Shopify?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">If you see this error in Google Search Console, it means Google found a page it thinks is important, but in your robots.txt it is blocked. Edit the robots.txt template file and set the \u201cAllow\u201d rule.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. When should you edit a Shopify robots.txt file?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You don&#8217;t need to edit it just for the sake of it. You should only touch it in these specific scenarios like you want to save your crawl budget, prevent AI bots, or enhance the visibility of quality pages.&nbsp;&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Why is a page &#8220;Indexed though blocked by Shopify robots.txt&#8221;?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The core reason is external backlinks the page is still getting while you have blocked it in your robots.txt file.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want the page gone completely from Google, unblock it in robots.txt and add a noindex tag to the page\u2019s &lt;head&gt; section. Google needs to be allowed to &#8220;crawl&#8221; the page to see the &#8220;No Index&#8221; sign!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Over to You<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Implementing best practices in your robots.txt file can effectively enhance the performance of your Shopify store.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Editing your robots.txt file is easier using the MIT Custom Robot.txt file without any need to copy\/paste from the default .txt file. You can also add comments in the file to keep rules organized for future edits.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There are high chances of making errors while editing manually, which can impact your SEO ultimately your store traffic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The app makes editing and updating organized and easier.&nbsp;<\/p>\n\n\n<div class=\"meetanshi-cta\">\r\n    <div class=\"cta-content-wrapper\">\r\n        <span>MIT Custom Robots.txt Rules<\/span>\r\n        <p>Optimize your robots.txt file easily to ensure search engines find what matters most.<\/p>\r\n        <a href=\"https:\/\/apps.shopify.com\/mit-custom-robots-txt-rules\" target=\"_blank\" class=\"btn-primary\" rel=\"noopener\">Optimize Now<\/a>\r\n    <\/div>\r\n    <div class=\"cta-image-new\">\r\n        <img decoding=\"async\" src=\"https:\/\/meetanshi.com\/blog\/wp-content\/uploads\/2024\/08\/MIT-Custom-Robots.txt-Rules.png\" alt=\"MIT Custom Robots.txt Rules\">\r\n    <\/div>\r\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>If you are struggling to get organic traffic then chances are the search engine bots have not crawled important pages of your Shopify store yet&#8230;.<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[787],"tags":[],"class_list":["post-3596","post","type-post","status-publish","format-standard","hentry","category-shopify"],"acf":[],"_links":{"self":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts\/3596","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/comments?post=3596"}],"version-history":[{"count":6,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts\/3596\/revisions"}],"predecessor-version":[{"id":25763,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/posts\/3596\/revisions\/25763"}],"wp:attachment":[{"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/media?parent=3596"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/categories?post=3596"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/meetanshi.com\/blog\/wp-json\/wp\/v2\/tags?post=3596"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}