When I am checking my robots.txt file in Bing webmaster tool robots.txt Tester it's showing an error
robots.txt is not found.
While my robots.txt is available in the root directory and accessible in the browser as well as accessible in google etc. Then why is bing showing such an error? Here is a screenshot of error saying "robots.txt is not found"
I had checked my robots.txt permission and all but was not able to fetch it in the bing robots.txt tester.
Related
In my site their is a button linking to PDF. Let say current pdf url on button is http://www.abc.come/wp-content/uploads/2016/09/xyz.pdf and this url google bot has crawled. Now later after month from admin, administrator uploads new pdf let say http://www.abc.come/wp-content/uploads/2016/09/xyz-latest.pdf and updates url on button.
Issue is that googlebot is still crawling old url with xyz.pdf and giving 404 in webmaster tools.
How can we make googlebot to stop crawling old url and crawl new ones.
Thanks.
Yup you can.
Under webmaster go to Google Index -> Remove URLS. Remove you url from here and then from your app. Works for me.
I had the same issue, my solution was an entry in the .htaccess with an 410 ('gone') statuscode. After some time google stops crawling.
But I also read that Google will stop crawling when 404. But on my site it keeps crawling 404-sites.
While I am submitting my website robots.txt to Google Search Console it's showing as an error like the below screenshot.
Just upload the robots.txt file to the root. Check it yourself by going to yourdomain.com/robots.txt.
If it works, then it ... works! It can take a while before Google updates the status in Search Console. Sometimes you need to lift your eyes from Search Console ;-)
It's an not an error. its saying upload the robot.txt file in the root directry of your website. And then ask google to update the robot.txt file.
How to check how Google robot sees my local website?
Is there an official Google tool? I've heard of one that check online websites, but what about locally site not avaible on network?
I don't think Google Webmasters Tools allow crawling or fetching local site.
Also, in addition to crawling local site, you'll have to make sure your local site or URLs don't get crawled in Google search as it will devalue your live site in future.
Thanks
You can use fetch as googlebot in webmaster tools to see how google views your site.
On the sitemaps.org it says that it is possible to submit the sitemap.xml via HTTP request to the search engine. However I'm unable to find documentation on how to do this for Google. I'm only finding the documentation on submitting it via Google Webmaster Tools.
Any ideas, is this even possible?
You can ping the sitemap url :
http://www.google.com/webmasters/sitemaps/ping?sitemap=URLOFSITEMAP.xml
Pinging google sitemap after every new article submission?
You don't need to submit and resubmit sitemap.xml to search engines. You can define them in your robots.txt file and web crawlers will find them and crawl them frequently.
I have an app running in Heroku.
I am using sitemap_generator to generate sitemap and save it into s3.
I have added the robots.txt to contain my sitemap location.
My question are.
How can I know my sitemap are successfully find by search engine like google?
How can I monitor my sitemap?
If my sitemap is located in my app server I can add the sitemap manually into google webmaster tools for monitoring. Because when I click on "Test/Add sitemap" in Google webmaster tools, it default to the same server.
Thanks for your help.
I got it to work.
Google has something called cross submission: http://googlewebmastercentral.blogspot.com/2007/10/dealing-with-sitemap-cross-submissions.html
You might want to visit this blog as well:
http://stanicblog.blogspot.sg/2012/02/how-to-add-your-sitemap-file-located-in.html
Thanks for your help, yacc.
Let me answer your two first questions, one at a time (I'm not sure what you mean by 'how can I monitor my sitemap' so I'll skip it):
Manually submit a sitemap to Google
If you can't use Google webmaster form to submit your sitemap, use an HTTP get request to notify Google of your new site map.
If your sitemap is located at https://s3.amazonaws.com/sitemapbucket/sitemap.gz , first URL encode your sitemap URL (you can use this online URL encoder/decoder for that) then using curl or wget to submit your encoded URL to Google:
curl www.google.com/webmasters/tools/ping?sitemap=https%3A%2F%2Fs3.amazonaws.com%2Fsitemapbucket%2Fsitemap.gz
If your request is successful you'll get a 200 answer with a message like this:
... cut ...
<body><h2>Sitemap Notification Received</h2>
<br>
Your Sitemap has been successfully added to our list of Sitemaps to crawl.
... cut ...
Checking that Google knows about your new sitemap
Open Webmaster Tools, navigate to Site sonfiguration->Sitemaps, there you should see the sitemaps that you've submited. It might take sometime for a new sitemap to show up there, so check frequently.