User-agent: * # Mask * means ‘every rather than specially for Google, Bing etc.’.
Disallow: /wp-admin/ # Admin site.
Robots.txt file is to be located at the root of the website.
Reasons why it is very important:
- Check if your web site is allowed for indexing at all.
- Disallow everything but your content (duplicates, service pages etc.)
- Show where your sitemap is situated.