The wordpress robots.txt file looks small but plays a very important role in SEO.

If you are running a WordPress site, then this file tells the search engine bots which pages of your site they can see and which they cannot.

If you have ignored this file till now,

Then now the time has come to take it seriously.

In this blog you will learn:

  • What is Robots.txt file
  • How does it work
  • How can you improve the SEO of your website by optimizing it
  • You will also get some advanced tips so that your site crawls faster and smarter

Table of Contents

Where Exactly Is The WordPress robots.txt File?

The robots.txt file is located in the root folder of your website. It is a simple text file that tells search engines like Google which parts of your site they can see and which they cannot.

A normal robots.txt file link looks like this: www.yoursite.com/robots.txt

What does WordPress do? When your WordPress site is live, WordPress automatically creates a virtual robots.txt file. But if you want more control, you can create your own robots.txt file.

Where can I find this file?

  • Are you using a file manager (like cPanel)?
  • Then look in the root folder of the website.
  • If you use a plugin (Yoast SEO or All in One SEO):
  • You can edit robots.txt from within the plugin.

If you like coding:

You can create and upload a robots.txt file using an FTP client (like FileZilla).

The Default WordPress Robots.txt (And Why It’s Not Enough)

When you create a WordPress site, its default robots.txt file comes with some basic rules.

This default setting works, but it is missing some important optimization things.

The default WordPress robots.txt file looks like this:

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php

What does it do?

Disallow: Tells search bots not to crawl the /wp-admin/ folder.

Allow: Tells bots that they can access the admin-ajax.php file, which is needed for some front-end functions.

This setup is okay, but not perfect.

Because some bots keep visiting pages of your site that are not important for SEO. This wastes your crawl budget meaning Google is not able to pay attention to the important pages of your site.

That is why optimization is important.

You can improve both the SEO and crawl speed of your site by making your robots.txt file a little better.

Always Include Your XML Sitemap(s)

There is one important rule in the robots.txt file – to link to your XML sitemap.

An XML sitemap helps search engines find important pages on your website quickly and easily.

How to add it?

Just write this line in the robots.txt file:

Sitemap: https://www.yoursite.com/sitemap.xml

If your site has multiple sitemaps (e.g. different ones for blog posts, products or videos), then it is important to include all sitemap links in the file.

Pro Tip:

You can use Yoast SEO or Rank Math plugin.

These tools automatically create and manage your sitemap.

Some Things Not To Block

Sometimes it seems that you should block all the extra things, but it is important to keep in mind what things are being removed from the search engines.

If you accidentally block important files like CSS or JavaScript, then both the design and ranking of your site can be spoiled.

What things should not be blocked:

Important files like /wp-includes/

CSS and JavaScript folders like:

/wp-content/themes/ and /wp-content/plugins/

Google uses these files so that it can understand how your site looks and how it works. If these files get blocked, your SEO performance and especially mobile usability score can be affected.

Managing Staging Sites

If you are running a staging or development site, it is important that it does not show up in search engines like Google. If such a site gets indexed, it can cause duplicate content problems, which confuses Google.

Block such sites using the Robots.txt file.

For example staging site:

User-agent: * Disallow: /

This setting tells you to block everything from search engines.

Another safe option is to password protect the staging site (using HTTP authentication). This will prevent anyone from accessing the site without permission.

Clean Up Some Non-Essential Core WordPress Paths

There are many such file paths in WordPress which are not beneficial for SEO such as:

/wp-json/
/trackback/

If search engine bots crawl these pages, then the crawl budget is wasted. Crawl budget means how many pages of your site Google can view at a time.

Those important paths should be blocked. Example:

Disallow: /cgi-bin/ 
Disallow: /trackback/ 
Disallow: /wp-json/ 
Disallow: /?attachment_id=*

This way you can make your robots.txt file clean and simple so that search engines focus only on the pages that help you in ranking.

Disallow Specific Query Parameters

Dynamic URLs with query parameters are often generated during internal search or filtering. These URLs can clutter Google’s index with duplicate or thin content, which adds no real value.

For example:

  • /shop/?color=red
  • /shop/?sort=price

How To Block Query Strings:

“`

Disallow: /?

“`

This rule blocks all URLs containing query strings, helping you avoid diluted SEO performance. However, be cautious. If query parameters genuinely add value (e.g., a vital search filter for users), exclude those specific parameters.

Disallowing Low-Value Taxonomies And SERPs

Your WordPress site may generate archive pages, tags, or category indexes. If not managed, these pages can clutter search engine results, compete with valuable pages on the site, and offer little user value.

Example Rules:

“`

Disallow: /category/

Disallow: /tag/

Disallow: /author/

Disallow: /search/

“`

By disallowing low-value, duplicate, and thin pages, you tighten your site’s overall SEO positioning.

Monitor On Crawl Stats

Even if you have created a good robots.txt file, it is still important to check it from time to time.

You have to see how bots like Google are behaving with your site. If bots are viewing too many pages that are not important, or your important pages are not being indexed, then the file should be checked again.

Tools you can use:

Google Search Console: Check the “Crawl Stats” report to see what the bots are viewing.

Screaming Frog: This tool shows how robots.txt is affecting your site.

Ahrefs or BuzzSumo: See if the pages that are being indexed are important to you or not.

If something seems wrong or some pages are being crawled incorrectly, then update the robots.txt file

Fine-Tune Your Crawl Budget with Robots.txt

A well-designed robots.txt file is not just an extra—it is an essential SEO tool for WordPress websites.

With this file, you can tell search engines which parts of your site they should see and which they should not. It helps them crawl your site faster and smarter.

If you add an XML sitemap and block low-value or useless pages, search engines like Google will rank your site better.

And if you need more SEO tips or help, try our SEO tools or hire our experts. Together we can take your WordPress site to the next level!

Want to stay ahead? Subscribe for daily search insights at bizaen and ensure your SEO strategies are always up to date