how do I remove robots.txt file from my site so i can index it for search ?

[catfordboyjohn]

Hi I am trying to find out how to remove the robots.txt file so I can index it with swiftype and use amuse themes widget to get a search function on my site so customers can search a long list of addresses by postcde (zipcode) anyone know how I can do this or is there a better way of building  a search function into my site in muse ?

[Liam Dilley]

Find the file in file manager or via FTP and delete it.

[gretelaCandeo]

I am having this  same issue. I've looked in the file manager and FTP and do not see the robot.txt. However in the search results I get this error:

A description for this result is not available because of this site's robots.txt

I've checked the pages, Site Manager > Pages and all the pages say: Show this page for search engines

Your help is greatly appreciated.

Thank you.

[Liam Dilley]

Can you provide the link to your website and a screenshot of where you are seeing the message about robots.txt

Thanks

[gretelaCandeo]

If you search on two brothers oshkosh in google it is the first listing. The site is twobrothersfamilyrestaurant.com.

Here is the search: Google

Please let me know if you need any other information. Thanks!

[gretelaCandeo]

Not sure if it was Google or re-submitting the site map, but it's working now.

Thanks anyway.

[Liam Dilley]

Sitemap.xml is very different to robots.txt

[gretelaCandeo]

Yes, I know that. Thank you. You have been extremely helpful!

[gretelaCandeo]

Hello,

Ok, it is back again.

Google

Any ideas why it keeps coming back?

The SEO Metadata, Search Results are all marked to Show this page for search engines.

There is not a robot.txt on the site that I can see via ftp or the file manager.

Any help is greatly appreciated. Thanks.

[Gaurav.Sharma]

Hi

There is no robots.txt file on your site. try to open, yoursite.com/robots.txt, you will know what I mean.

You can create your own robots.txt and upload it to BC. To see the detailed instructions, follow this guide.

The Web Robots Pages

Let me know if you still have any question.

[gretelaCandeo]

Thank you. I don't want a robot.txt on my site. I want the message: A description for this result is not available because of this site's robots.txt to be removed from the google search.
Google

Thank you.

[Gaurav.Sharma]

If I search your site in Google, this is what I see,

Looks fine to me. Let me know if I am looking at wrong site.

[gretelaCandeo]

That is not what I see unfortunately:

It was resolved for a while yesterday, but it came back.

Thank you for your help.

[Liam Dilley]

i do not see that either doing the same.

Try doing your search in incognito mode in a browser so you do not have cache, cookies or logged in.

Google search results are no longer the same for anyone by a long way.
Google search results runs all its calculations for lots of factors but if you are logged into google it will include your searching history to rank items based on that. Cookies and cache can also have an effect on your results.

[gretelaCandeo]

Thank you. I was using incognito mode. I even tried on a computer that had never done the search.

I received the same results I posted above in both Google and Bing.

I will try to see if Google has an answer.

Thanks for your time.

[anjaa25778737]

Hi! I'm allowing myself to join the discussion (in 2018 ; ) ). I'm having the same issue, and google webmaster tools is showing me this message:

Is there a way i can change the noindex in the http-header in muse so that google can index my page?

Thank you for your time and help!

[Liam Dilley]

Login to the BC admin itself and just go into the page in question and ensure the SEO options are ticked to allow the page to be searched by google.