robots.txt

Updating the robots.txt in google

Sometimes you setup a new website (like I did) and you force search engines to not index your work in progress. You can add a robots.txt. in the hope spiders will honor your robots.txt and skip indexing your website. For instance with this robots.txt:

User-agent: *
Disallow: /

When you are going “live” with your website you obviously want your website to get indexed into google, therefor updating your robots.txt so that your website (or parts of it) can be indexed again.

There is only a downside to this. Once google fetched your robots.txt knowing he has no business here, it might take ages for Google to return to your website to reindex your robots.txt!

Updating the robots.txt in google

There is a page in the webmasters tool at Google: robots-testing-tool. It might be a little bit vague but you can actually push a new robots.txt to google here.

robots.txt updating 1

robots.txt updating 1

So you have this “testing” area, but when you click the submit button you are actually prompted with 3 options:

robots.txt updating 2

robots.txt updating 2

This way you can “submit” a request for google to re-check your robots.txt, hence making it faster to get your website indexed again.

In all fairness, it took me a while to figure this feature out.. thats why I thought to share it!

Leave a Reply

Your email address will not be published. Required fields are marked *