I have been searching for quite some time and finally decided to posts this question on how Mod Security locks out a user from a domain.
I have a large site with a lot of legacy URL's with '$' and "%" in them, this was removed but there are legacy links all over that will trip some mod security rules.
The main issue is once a rule is triggered, a 403 error is returned as expected on that page, but going to any other page on the domain now will throw a 403 error as well utill the cookies are cleared on the browser. This of course is not user friendly as many people will not know about the clear cookies fix and If they are locked out I cant obviously let them know easily while not wanting to remove all the rules that cause this.
example of a url
[code]
Request: GET /phpBB2/promotions/9927-1st-deposit-bonus-125%25move-up-sun-palace-casin.html
Action Description: Access denied with code 403 (phase 2).
Justification: Invalid URL Encoding: Non-hexadecimal digits used at REQUEST_URI.
[/code]
this is a
950107: URL Encoding Abuse Attack Attempt
Also many errors in the Mod Security Log I see simply GET / as the trigger and that obviously is not the root cause.
My first thought is if this rule is firing incorrectly because such a use case is (or was) valid then you should probably disable this rule with:
SecRuleRemoveById 950107
If your page no longer exists then this will presumably return a standard 404 message without this rule. Which is probably more correct to the user than 403 and better UX. Will also mean you don't get ModSecurity alerts in your logs for these false positives. OK it could mean you miss a genuine attack this rule is designed to block, so you need to weigh up the risk of that versus the downside of upsetting some of your users. Personally I don't think this rule is protecting you from a major security problem so would disable it.
Also need to understand exactly what the problem is here. Rule 950107 checks the URL, not the cookies so, while I understand it firing for the initial request, it must be a different rule which is firing for the subsequent blocking errors? Not sure how cookies are being set incorrectly and to such a value that causes problems in future so need more details.
It is possible to remove cookies using a combination of ModSecurity and Apache using a method documented by Ivan Ristic in his ModSecurity Handbook, but it's a bit convoluted and involves a couple of steps:
1) Create a rule which detects when you want to remove the cookies and sets an environmental variables. For example to check for a URL containing the word "value_to_check" use a rule like this:
SecRule REQUEST_URI "value_to_check" "id:12345,phase:1,setenv:DISABLE_OUTBOUND_SESSION"
2) Get Apache to request the browser unsets the SESSIONID Cookie where that env variable is set, using mod_header:
Header always set Set-Cookie "SESSIONID=; expires 31-Dec-1999 00:00:00 GMT" env=DISABLE_OUTBOUND_SESSION
Note that this doesn't fix any bad incoming cookies. There's similar fixes for that depending on the exact problem.
I would suggest however you understand the full problem first as there may be better solutions.
Related
So I've been going through my forms recently to check my SQL queries are secure along with sanitizing any input and have just found that entering <? into a text box triggers a 403 before it even hits the processing file, I can only assume it must be related to mod_security??
My question is, is this something to just not worry about if it's controlled by the web host as I'm using shared hosting.
I recently ran into a problem with submitting form data via a GET request to the server after using jQuery's .serialize() function for the submitted variables. These were web apps that had worked flawlessly for years. It turned out that after a recent ModSecurity rule set update, I was triggering the 211700 (HTTP redirect) and 217280 (smuggling attack) rules in Comodo's WAF ruleset, which the server uses with ModSecurity. I wasn't getting a 403. My IP address got blocked by the firewall. :(
The fix was switching my AJAX code to use to POST instead of GET, and not using .serialize(). I still have some web apps that use .serialize() and GET requests via AJAX without triggering ModSecurity, so I believe it is also necessary to pass suspect characters, as you discovered, though in my testing, all I was using was parentheses.
Since you're on a shared server, it's probably not possible--or worth your time--to find out what rule set the host is using, so your best bet is most likely to switch your form submissions to using POST instead of GET, and not use .serialize(). I figure those rules are there for a reason, and a better approach is to avoid having my code look like it's doing something nefarious than to disable the rules.
My Question is this
I redirect the form in
Redirect /formviewer/faces/pages/view/viewform.xhtml
///formviewer/faces/pages/view/viewform.xhtml
in Apache Http Server but the form is not redirecting while doing post request from client Side.
So what is it doing?
I rather suspect that its normalising the thing you want to redirect to ///... and considering it the same.
When looking at redirects, there are three things in my list of 'go-to' tools:
curl -I http://..... and look at the response and the Location header. I use this when installing new redirects (it avoids following chained redirects and gives me a better view of what is happening at each step.
Something like Fiddler or a browsers developer tool. Frankly I'd only use this if curl wasn't doing it for me.
Enable the RewriteLog in Apache... I almost never use this, but for your case it may shed some light.
But let's take a step back: why are you wanting to redirect to a path with and extra // prepended (which as I've previously speculated, likely won't work)?
I am trying to access a secure website using this kind of url: https://securenet.someBank.com. Everything is good and I am shown the login page. Now when I just type:
http://securenet.someBank.com (i.e http instead of https) I expect to get back a page with https in the browser. (e.g when you say:http://mail.yahoo.com, you get back https://mail.yahoo.com).
But in this case https:://securenet.someBank.com just says :Page cannot be displayed.
So what did the website developer do wrong in implementing security? I am just curious. I thought this kind of thing (http --> https redirection) was handled automatically by the web server and the website developer does not even need to do anything. But apparently it is not so.
The redirections from HTTP to HTTPS are merely a convenience for the user.
As I was saying in this answer on Webmasters.SE, only the end user can check whether HTTPS is used at all, and whether it's used correctly. A MITM attacker could otherwise prevent that initial redirection from happening at all.
These automatic redirections are only useful based on the assumption that there's no MITM performing such an attack. They're useful to get the user used to seeing HTTPS on pages that should be secure, but whatever happens, it will always be up to the user to check what they're connecting to. Therefore, I wouldn't necessarily call the absence of such a redirection a developer or sysadmin mistake.
As a user, you should always bookmark and use the https:// address for sites where you expect it should be used.
[...] https://securenet.someBank.com. Everything is good and I am shown the login page.
[...]
But in this case https:://securenet.someBank.com just says :Page cannot be displayed.
Here, assuming the double :: is a typo in your question, you seem to contradict yourself. If https://securenet.someBank.com just says "Page cannot be displayed", this would be a mistake indeed.
besides the recommendation by Bruno above I would recommend you to read the following:
https://www.owasp.org/index.php/HTTP_Strict_Transport_Security
There are two things you could do:
1) Force HTTP Strict Transport Security
2) Do a permanent redirect as described in the example on that page.
Any questions, just let me know.
Fabio
#fcerullo
Probably wrong server configuration. For example in apache one must define a redirect option in httpd.conf file in order to automaticaly redirect to the https URL of the page.
This is mostly a philosophical question about the best way to interpret the HTTP spec. Should a directory with no directory index (e.g. index.html) return 404 or 403? (403 is the default in Apache.)
For example, suppose the following URLs exist and are accessible:
http://example.com/files/file_1/
http://example.com/files/file_2/
But there's nothing at:
http://example.com/files/
(Assume we're using 301s to force trailing slashes for all URLs.)
I think several things should be taken into account:
By default, Apache returns 403 in this scenario. That's significant to me. They've thought about this stuff, and they made the decision to use 403.
According to W3C, 403 means "The server understood the request, but is refusing to fulfill it." I take that to mean you should return 403 if the URL is meaningful but nonetheless forbidden.
403 might result in information disclosure if the client correctly guesses that the URL maps to a real directory on disk.
http://example.com/files/ isn't a resource, and the fact that it internally maps to a directory shouldn't be relevant to the status code.
If you interpret the URL scheme as defining a directory structure from the client's perspective, the internal implementation is still irrelevant, but perhaps the outward appearance should indeed have some bearing on the status codes. Maybe, even if you created the same URL structure without using directories internally, you should still use 403s, because it's about the client's perception of a directory structure.
In the balance, what do you think is the best approach? Should we just say "a resource is a resource, and if it doesn't exist, it's a 404?" Or should we say, "if it has slashes, it looks like a directory to the client, and therefore it's a 403 if there's no index?"
If you're in the 403 camp, do you think you should go out of your way to return 403s even if the internal implementation doesn't use directories? Suppose, for example, that you have a dynamic web app with this URL: http://example.com/users/joe, which maps to some code that generates the profile page for Joe. Assuming you don't write something that lists all users, should http://example.com/users/ return 403? (Many if not all web frameworks return 404 in this case.)
The first step to answering this is to refer to RFC 2616: HTTP/1.1. Specifically the sections talking about 403 Forbidden and 404 Not Found.
10.4.4 403 Forbidden
The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.
10.4.5 404 Not Found
The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.
My interpretation of this is that 404 is the more general error code that just says "there's nothing there". 403 says "there's nothing there, don't try again!".
One reason why Apache might return 403 on directories without explicit index files is that auto-indexing (i.e. listing all files in it) is disabled (a.k.a "forbidden"). In that case saying "listing all files in this directory is forbidden" makes more sense than saying "there is no directory".
Another argument why 404 is preferable: google webmaster tools.
Indeed, for a 404, Google Webmaster Tool displays the referer (allowing you to clean up the bad link to the directory), whereas for a 403, it doesn't display it.
I have been noticing on my trackers that bots are visiting my site ALOT. Should I change or edit my robots.txt or change something? Not sure if thats good, because they are indexing or what?
Should i change or edit my robots.txt or change something?
Depends on the bot. Some bots will dutifully ignore robots.txt.
We had a similar problem 18 months ago with the google AD bot because our customer was purchasing Soooo many ads.
Google AD bots will (as documented) ignore wildcard (*) exclusions, but listen to explicit ignores.
Remember, bots that honor robots.txt will just not crawl your site. This is undesirable if you want them to get access to your data for indexing.
A better solution is to throttle or supply static content to the bots.
Not sure if thats good, because they are indexing or what?
They could be indexing/scraping/stealing. All the same really. What I think you want is to throttle their http request processing based on UserAgents. How to do this depends on your web server and app container.
As suggested in other answers, if the bot is malicious, then you'll need to either find the UserAgent pattern and send them 403 forbiddens. Or, if the malicious bots dynamically change user agent strings you have a two further options:
White-list UserAgents - e.g. create a user agent filter that only accepts certain user agents. This is very imperfect.
IP banning - the http header will contain the source IP. Or, if you're getting DOS'd (denial of service attack), then you have bigger problems
I really don't think changing the robots.txt is going to help, because only GOOD bots abide by it. All other ignore it and parse your content as they please. Personally I use http://www.codeplex.com/urlrewriter to get rid of the undesirable robots by responding with a forbidden message if they are found.
The spam bots don't care about robots.txt. You can block them with something like mod_security (which is a pretty cool Apache plugin in its own right). Or you could just ignore them.
You might have to use .htaccess to deny some bots to screw with your logs.
See here : http://spamhuntress.com/2006/02/13/another-hungry-java-bot/
I had lots of Java bots crawling my site, adding
SetEnvIfNoCase User-Agent ^Java/1. javabot=yes
SetEnvIfNoCase User-Agent ^Java1. javabot=yes
Deny from env=javabot
made them stop. Now they only get 403 one time and that's it :)
I once worked for a customer who had a number of "price comparison" bots hitting the site all of the time. The problem was that our backend resources were scarce and cost money per transaction.
After trying to fight off some of these for some time, but the bots just kept changing their recognizable characteristics. We ended up with the following strategy:
For each session on the server we determined if the user was at any point clicking too fast. After a given number of repeats, we'd set the "isRobot" flag to true and simply throttle down the response speed within that session by adding sleeps. We did not tell the user in any way, since he'd just start a new session in that case.