User-agent: *
Disallow:
Disallow: /admin
Disallow: /admin
Sitemap: http://www.myadress.com/ext/sm/Sitemap_114.xml
I've found this robots.txt file in one of my website's root folder. I don't know i made it or who.
I think this file does not allow any robots to admin folder. This is good.
But i wonder if this blocks all robots to all files in my website?
I've changed it with this file:
User-agent: *
Disallow: /admin
Allow: /
Sitemap: http://www.myadress.com/ext/sm/Sitemap_114.xml
ps: this website is not getting any index for a long time. was the old file problem?
The old script, other than the duplicate /Admin entry, was correct. There is no Allow: command, and an empty 'Disallow:' opens the site up to robots.
http://www.free-seo-news.com/all-about-robots-txt.htm
Specifically, check item #7 under 'Things you should avoid' and #1 under 'Tips and Tricks'
Possibly. I read your old file the same way you did - that root was being disallowed. Either way, your new robots file is set up how I expect that you want it.
Related
I am trying to disallow one page in subdirectory
I am using that robots.txt code is it
User-Agent: *
Disallow:
Disallow: /form.aspx
but the form.aspx is in processfolder and my url is showing like
www.yoursite.com/process/form.aspx
so how can I disallow form.aspx in robots.txt.
The format which is given above robots.txt: is it right?
please guide
If you want to block http://example.com/process/form.aspx and allow everything else, you can use:
# robots.txt on <http://example.com/robots.txt>
User-agent: *
Disallow: /process/form.aspx
Note that this would also block URLs like http://example.com/process/form.aspx.foo, http://example.com/process/form.aspx/bar, etc.
I want my site to be indexed in search engines except few sub-directories. Following are my robots.txt settings:
robots.txt in the root directory
User-agent: *
Allow: /
Separate robots.txt in the sub-directory (to be excluded)
User-agent: *
Disallow: /
Is it the correct way or the root directory rule will override the sub-directory rule?
No, this is wrong.
You can’t have a robots.txt in a sub-directory. Your robots.txt must be placed in the document root of your host.
If you want to disallow crawling of URLs whose paths begin with /foo, use this record in your robots.txt (http://example.com/robots.txt):
User-agent: *
Disallow: /foo
This allows crawling everything (so there is no need for Allow) except URLs like
http://example.com/foo
http://example.com/foo/
http://example.com/foo.html
http://example.com/foobar
http://example.com/foo/bar
…
Yes there are
User-agent: *
Disallow: /
The above directive is useful if you are developing a new website and do not want search engines to index your incomplete website.
also,you can get advanced infos right here
You can manage them with robots.txt which sits in the root directory. Make sure to have allow patterns before your disallow patterns.
Im looking to disallow a subdirectory in my root folder but allow a folder within that.
What I have:
User-Agent: *
Disallow: /admin
I want to allow /admin/images
Is this possible?
Try this. Give a try in robots.txt tester to avoid any negative impact.
Allow: /admin/images/
Disallow: /admin/
I'm working on a WordPress site that has a login portal where users can access 'classified' documents in pdf,doc and a few other formats. The files are uploaded via the media manager, so are always stored in /wp-content/uploads
I need to make sure these file types are not shown in search results. I've made some rules in .htaccess and robots.txt that I think will work, but it's very hard to test, so I was hoping someone could glance over them and let me know if they'll do what I'm expecting them to or not. One thing in particular I wasn't sure of: would the disallow: /wp-content/ stop the x-robots-tag from being seen?
.htaccess - under # end Wordpress
# do not index specified file types
<IfModule mod_headers.c>
<FilesMatch "\.(doc|docx|xls|xlsx|pdf|ppt|pptx)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
</IfModule>
robots.txt - complete
User-agent: *
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /growers-portal
Disallow: /growers-portal/
Disallow: /grower_posts
Disallow: /grower_posts/
Sitemap: http://www.pureaussiepineapples.com.au/sitemap_index.xml
Neither of those stop anyone reading your "classified" documents. To do that you really want to restrict access based on logged in users.
The robots tag will keep the files out of the search results.
However, robots.txt does not stop files being in the search results. Google takes that directive to say they can't read the file but they can still include it in the index.
This causes an interesting scenario. Your robots.txt stops Google reading the robots tag so does not know you want it out of the index.
So, if you're not going to physically control access to the files I would use the robots tag but not robots.txt directives.
should i then do
User-agent: *
Disallow: /
is it as simple as that?
or will that not crawl the files in the root either?
basically that is what i am after - crawling all the files/pages in the root, but not any of the folders at all
or am i going to have to specify each folder explicitly.. ie
disallow: /admin
disallow: /this
.. etc
thanks
nat
Your example will block all all the files in root.
There isn't a "standard" way to easily do what you want without specifying each folder explicitly.
Some crawlers however do support extensions that will allow you to do pattern matching. You could disallow all bots that don't support the pattern matching, but allow those that do.
For example
# disallow all robots
User-agent: *
Disallow: /
# let google read html and files
User-agent: Googlebot
Allow: /*.html
Allow: /*.pdf
Disallow: /