I have a secret folder in my website and I don't want search engines to know about it. I didn't put the folder name in the Disallow rule of robots.txt because writing this folder name in robots.txt means telling my visitors about that secret folder.
My question is, will search engines be able to know about this folder / crawl it even if I don't have any links published to this folder?
The only truly reliable way to hide a directory from everyone is to put it behind a password. If you absolutely cannot put it behind a password, one band-aid solution is to name the folder something like:
http://example.com/secret-aic7bsufbi2jbqnduq2g7yf/
and then block just the first part of the name, like this:
Disallow: /secret-
This will effectively block the directory without revealing its full name. It will prevent any crawler that obeys robots.txt from crawling the directory, but it won't make the directory easy for hostile crawlers to find. Just don't mistake this for actual security. This will keep the major search engines out. There are no guarantees beyond that. Again, the only truly reliable way to keep everyone out of a secret directory is to put the directory behind a password.
Yes they can crawl it.
Your folder is not "secret" at all. Do a quick search for a curl command line to download the whole site then try it on your site to convince yourself your security approach is invalid.
Here is a good example: download allfolders subfolders and files using wget
You can you .htaccess to prevent agents being able to request the directory listing, and this will probably protect you fairly well if you don't give your folder an obvious name like "site", but I'd test it.
see deny direct access to a folder and file by htaccess
Related
I will explain, what I want to do.
We have an eshop, which generates htaccess file (server is Unix). Sometimes, it crashes during generating htaccess file, then whole site (frontend, admin, cron scripts => everything) is returning server error 500, due to not finished content in htaccess file.
And here is my question. I created script, which will regenerate htaccess file. But this script cant be anywhere in root (or subdirectories), because it will also return 500. Also subdomains are in root in subdirectory /_sub.
Is there any change to put it somewhere, where our customer will be able to execute it himself?
I cant use another domain; so I am asking, if there is any chance to ignore htaccess file, when it contants wrong content?
Thanks a lot.
EDITED:
I know, maybe one solution is here, but I think it will consume much more time, to code it all, but maybe not:
on another domain code script, which will connect through socket to ftp of eshop domain
it will delete htaccess file and recreate it to basic content
executing that cron, which will regenerate whole htaccess file
This should work, I think.
The question is, how layman your customer are. Thw options: 1) you make this from cron 2) you put this in a cgi out.
But AFAIK the best were, if only a testing script from cron runs, and this reconstructed only the bad htaccess files.
I know, maybe one solution is here, but I think it will consume much more time, to code it all, but maybe not:
on another domain code script, which will connect through socket to ftp of eshop domain
it will delete htaccess file and recreate it to basic content
executing that cron, which will regenerate whole htaccess file
This should work, I think.
I know it's a bit of a beginner question but tried several combinations and none work... here's what i need to achieve using .htaccess and .htpasswd
I have a www.mysite.com/protected folder.
By default, all ip's shouldn't see this folder and any other files inside it exists (basically they should be redirected to www.mysite.com)
A certain IP (my external IP M.M.M.M) should be able to access that protected folder and be requested a username and password
All ip's should be able to see some files inside the folder without being asked for password (www.mysite.com/protected/file1 and www.mysite.com/protected/file2)
I know the latter rule opens an exception on the first but that's how i need it (a general rule denying all and then whitelisting some files)
Thanks in advance for any help, been struggling with this for a bit now :(
I have a directory on my website specifically for javascript files, I want these javascript files to be hidden, so if I type the url to it it says Forbidden or disallows access, but my front-end website files can still access them to execute them when needed. Is there a way to do this through a FTP client?
Cheers,
Dan
You can't do this trough a ftp client. It is the task of your webserver to forbid access to certain files.
If you change permission, the webserver won't have access to them anymore, so this is not the way to go.
You must configure your webserver to restrict the access. If you're using Apache, you can use an .htaccess file. There's different ways of doing this, many depends on the way the webserver is configured.
The easiest is to put an .htaccess file in your Scripts folder which contain only this none line :
deny from all
However, like peeter said, there's a good chance this will break your site, since the browser must access theses files, so you can't restrict access.
Put a htaccess file in your scripts folder containing deny from all, but this will stop your pages from accessing the scripts also (though not if you pass them through the PHP engine first)
You're trying to hide JavaScript files that are executed on the clients side. If a client(browser) cannot access the files means non of your javascript code is executed.
If I understood your question correctly then you cannot achieve what you're trying to achieve.
I want to block access to only one directory say http://www.example.com/pictures
Every file in other directory should be available as normal.
I have gone through a referrer checking solution. That can be easily bypassed.
I Simply want to show images in pictures directory on a page say view.php
if images in this directory accessed from anywhere else. Just redirect to an image that says hotlinking not allowed"
Thanks
I have gone through a referrer checking solution. That can be easily bypassed.
Yes, it can. But so can any hotlink-prevention scheme. It'll deter casual/careless use, but that's about it.
For what it's worth, referrer-checking is pretty much the way to do it.
I currently have css and javascript file calls (amongst other things) like the following:
href="/css/default.css"
src="/js/ui_control.js"
putting the preceding / in to make the files relative to the root.
This works great when my page is in the root of the domain.
However, I'm currently in the middle of transferring my site to a new hosting provider and as such have a temporary URL which is: HOST-IP/~username
As such, all file calls are trying to be called from HOST-IP/css/default.css etc instead of within the ~username sub-folder.
Of course I can wait until the domain name servers propagate but that's beside the point.
How would I go about writing a rule in the .htaccess file that would redirect all file calls that start with a /, from going to HOST-IP/FILE-CALL, and instead to go to HOST-IP/~USERNAME/FILE-CALL. ?
Any ideas?
I'd suggest changing the references in your HTML to the files to be relative, as this will work either in a sub folder or as the root of the domain.
This works great when my page is in the root of the domain. However, I'm currently in the middle of transferring my site to a new hosting provider and as such have a temporary URL which is: HOST-IP/~username
How would I go about writing a rule in the .htaccess file that would redirect all file calls that start with a /, from going to HOST-IP/FILE-CALL, and instead to go to HOST-IP/~USERNAME/FILE-CALL. ?
Unless you can put a .htaccess at HOST-IP/.htaccess on the new server, you can't do this with .htaccess. It sounds like you're on a shared host, so any approach that'd let you do this with .htaccess would allow you to hijack everyone else's site on the server.