Adobe Community: Preventing Google from indexing your test server?

Currently Being Moderated

1. SnakEyez02,

Mar 8, 2012 11:20 AM in reply to kcrossley

Report

Here's a simple version of what you need to do:

http://www.robotstxt.org/robotstxt.html

It just goes in the web directory of your site (eg: /www or /public_html). I would also make sure that you password it too and use other methods protection. That way if some bot decides to ignore the robots.txt file, which is not supposed to but can happen, the person visiting the site gets forced to your login page.

|

Mark as:

Currently Being Moderated

2. kcrossley,

Mar 8, 2012 11:24 AM in reply to SnakEyez02

Report

Thanks for the reply.

I probably need to put more thought into this as access to my test site section does not require a password. Also, I only want robots to ignore my client directory, which is www.mycompany.com/clients/ Where should I put the robots.txt file to prevent robots for indexing ONLY my clients directory and subdirectories?

|

Mark as:

Currently Being Moderated

3. John Waller,

Mar 8, 2012 11:48 AM in reply to kcrossley

Report

http://en.wikipedia.org/wiki/Robots.txt See Examples

Place robots.txt in the root folder of your website

To exclude indexing one folder named "test" only

User-agent: *

Disallow: /test/

Remember that robots.txt works on an honor basis. It asks bots not to index. They obey your instructions only if they're programmed to do so. Google honors robots.txt according to Google.

But

"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results."

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

So, no guarantees your test sites won't end up on the web somewhere without password protection.

|

Mark as:

Currently Being Moderated

4. kcrossley,

Mar 8, 2012 11:55 AM in reply to John Waller

Report

So I put the file in the root folder, not the "test" subdirectory? What about all of the subdirectories below /test/? Will they be disallowed too?

|

Mark as:

Currently Being Moderated

5. John Waller,

Mar 8, 2012 12:52 PM in reply to kcrossley

Report

Yes, according to my reading of the Google link.

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

To block a directory and everything in it, follow the directory name with a forward slash.

Disallow: /junk-directory/

If you have any further questions on this, feel free to read through the Wikipedia and Google links which should be able to provide most of the answers.

|

Mark as:

Dreamweaver

Preventing Google from indexing your test server?

Mar 8, 2012 11:12 AM

More Like This

Bookmarked By (0)

Answers + Points = Status