How To Fix And Write Robots.txt File In Blogger

For bloggers who has little experience about setting up robots.txt this tutorial will guide you to set up your robots.txt and understand all the basics of it.
Many do fear not to touch the robots.txt so as not to deindexed their site as when you get there you will be strictly warned by google not to touch those settings as it can make robots not to index your website.

What is robots.txt?

Custom robots.txt simply tells robot where to touch on your site and where shouldn't be touch, its simple right

Now:

When robots get to your site the first thing they will look upto is your site custom robots.txt if you a'int activating it that means bots are free to index anywhere on the site but activating it and learning this tutorial you can easily command them not to touch some specific places on your site like your search pages e.g www.mydomain.com/search/label/news yes you can easily stop bots not to crawl this as stated in Google Seo Guide 'you should not allow bots to crawl some specific pages'

How does custom robots.txt look like

• User Agent: Mediapartner-google
• User Agent: *
• Disallow: /search
• Allow: /
• Sitemap: http://yourwebsite.com/sitemap.xml

You can easily get any website robots.txt by typing their site name e.g www.bloggertasks.com/robots.txt

Lets simple explain what those robots tag means.
Read : How to set up custom robot header tags

To activate your robots.txt simply navigate to your blogger settings>>search preference then check yes to activate the custom robots.txt
now you can edit it

User agent :

It indicate the name of the bots you are pointing to

User agent: Mediapartner-google :

This means google adsense bots will crawl your site to look for relevants ads to serve for you if you disable this google adsense crawlers will be unable to see any ads on your pages.

User Agent:

Programmers are well aware of '*' it means All this means all bots and crawlers are welcome to crawl my sites

Disallow: /search :

As explained earlier google seo guide explained to disallow some pages on your site as explained earlier this means that crawlers will never be able to index your label pages which starts with /search i.e example.com/search/label/news

Allow: /

This simply tells crawlers to allow bots to index all your website homepage and your contents

Sitemap: http://www.bloggertasks.com/sitemap.xml

This is a map for robots to index your pages all your website urls are stored here and it can be easily seen by crawlers.

You can easily instruct bots not to index some specific urls ,indexed by robots i.e.you can instruct them not to index the url by putting Disallow: /year/mm/post-url . Make sure you suscribe to our newsletters.

Tweet