What is robots.txt


User-agent: * # Mask * means ‘every rather than specially for Google, Bing etc.’.
Disallow: /wp-admin/ # Admin site.
Disallow: *?s=
Disallow: /search
Disallow: */embed$

Sitemap: http://example.com/sitemap.xml



Robots.txt file is to be located at the root of the website.

Reasons why it is very important:

  • Check if your web site is allowed for indexing at all.
  • Disallow everything but your content (duplicates, service pages etc.)
  • Show where your sitemap is situated.


Rate article
Add a comment