3 Replies Latest reply: Mar 13, 2014 6:41 AM by Smoke25

Robots.txt file is not found on my website, how do I add it?

naturescape_peopletecture May 13, 2013 11:41 AM

I am looking to improve the Search Engine Optimization of my website and in doing so I am using a few websites to review my own site. A major citicism of several website reviews is that my Muse-designed website does not have a robots.txt file. These websites claim that adding a robots.txt file is easy and is important for restricting access of search engine robots.

Anyway, is there a way for my site to have a robots.txt file so these websites reviewing my site can at least recognize that my site has a robots.txt file

Thanks,

1. Re: Robots.txt file is not found on my website, how do I add it?

naturescape_peopletecture May 13, 2013 11:45 AM (in response to naturescape_peopletecture)

Also, I used this robots.txt file website to check my site and it errors because my site has no robots.txt file:

http://tool.motoricerca.info/robots-checker.phtml

My website is www.jlbruce.com so I typed in:

http://www.jlbruce.com/robots.txt
Like Show 0 Likes (0)

Actions
2. Re: Robots.txt file is not found on my website, how do I add it?

Vinayak_Gupta Jun 6, 2013 4:37 PM (in response to naturescape_peopletecture)

You can create a robots.txt file through FTP or Develop tab of the BC site admin. You'll find a number of tools over the web in order to generate a robots.txt file such as this - http://tools.seobook.com/robots-txt/generator/

Thanks,
Vinayak
Like Show 0 Likes (0)

Actions
3. Re: Robots.txt file is not found on my website, how do I add it?

Smoke25 Mar 13, 2014 6:41 AM (in response to naturescape_peopletecture)

The only time that you need a robots.txt file is if you have pages that you do not want Google to crawl. If everything is available, if you have no robots.txt file it will return a 404 error and Google will continue to crawl the site. See: https://support.google.com/webmasters/answer/2409682?hl=en
Like Show 0 Likes (0)

Actions

Go to original post

More Like This

Incoming Links

Re: I have tried to create a search engine in Swiftype for my temporary Muse site but Siftype indicates a Robot txt file preventing web crawling. Could you please assist me with this?

Legend

Correct Answers - 10 points