4 Replies Latest reply: Jul 28, 2013 6:58 PM by hettyere

Unable to replicate robots.txt through to the dispatcher

hettyere Jul 27, 2013 10:31 PM

I implemented robots.txt on both author and publish instances using the link below:

http://crxdelight.com/2012/02/04/how-to-protect-your-cq-instances-from-google-searches/

robots.txt exist and it is visible on both author and publish instances. When i tried to display the robots.txt on dispatcher link "http://localhost/robots.txt", i got the error below:

Not Found

The requested URL /robots.txt was not found on this server.

Is there any specific configuration settings needed on the dispatcher for it to recognize the robots.txt? Or are there any issues that could cause this problem?

Thank you.

1. Re: Unable to replicate robots.txt through to the dispatcher

Yogesh Upadhyay Jul 28, 2013 7:23 AM (in response to hettyere)

If this file exists in publish then there is a dispatcher rule that is blocking this file.

Yogesh
Like Show 0 Likes (0)

Actions
2. Re: Unable to replicate robots.txt through to the dispatcher

hettyere Jul 28, 2013 7:59 AM (in response to Yogesh Upadhyay)

Thank you Yogesh for your prompt response.

Will you be able to tell which dispatcher rule that may be blocking this file?
Like Show 0 Likes (0)

Actions
3. Re: Unable to replicate robots.txt through to the dispatcher

Sham HC Jul 28, 2013 3:30 PM (in response to hettyere)

Following the article the filter [1] is blocking it as expected.
[1] /0080 { /type "deny" /glob "GET * /robots.txt*" }
Like Show 0 Likes (0)

Actions
4. Re: Unable to replicate robots.txt through to the dispatcher

hettyere Jul 28, 2013 6:58 PM (in response to Sham HC)

Thank you Sham for your response.

I added the line of code below in my dispatcher.any file. It is still not able to see the robots.txt.

# Deny content grabbing
/0080 { /type "allow" /glob "GET * /robots.txt*" }
Like Show 0 Likes (0)

Actions

Go to original post

Unable to replicate robots.txt through to the dispatcher

Not Found

1. Re: Unable to replicate robots.txt through to the dispatcher

2. Re: Unable to replicate robots.txt through to the dispatcher

3. Re: Unable to replicate robots.txt through to the dispatcher

4. Re: Unable to replicate robots.txt through to the dispatcher

Actions

More Like This

Legend