What is robots.txt

[txt]

User-agent: * # Mask * means ‘every rather than specially for Google, Bing etc.’.
Disallow: /wp-admin/ # Admin site.
Disallow: *?s=
Disallow: /search
Disallow: */embed$

Sitemap: http://example.com/sitemap.xml
[/txt]

 

https://developers.google.com/search/docs/advanced/robots/create-robots-txt

Robots.txt file is to be located at the root of the website.

Reasons why it is very important:

  • Check if your web site is allowed for indexing at all.
  • Disallow everything but your content (duplicates, service pages etc.)
  • Show where your sitemap is situated.

Contents

Rate article
Add a comment