Background
We have been running an application on JBoss that is exposed to the clients via an Apache Reverse Proxy. We recently introduced "HTTP 429 Too many requests" to slow down high velocity requests.
Problem
However, it seems that apache2 changes the HTTP status code from 429 to 500.
Root cause analysis
Confirmed from JBoss that it sends HTTP 429, by bypassing the proxy, and talking to it directly.
Confirmed from /var/log/apache2/access.log, that apache2 gets HTTP 429
10.0.0.161 - - [16/Jul/2014:07:27:47 +0000] "POST /the/URL/ HTTP/1.1" 429 1018 "-" "curl/7.36.0" |0/466110|
Curl Client gets 500, somehow.
There's also been a bug filed few years back on Bugzilla #900827. I remember reading that it has been fixed in 2.2.18. Yet, I still face the problem -- which leads me to think there's probably a different problem altogether.
Questions
As I have read elsewhere, Apache might not relay the code perfectly for custom HTTP status codes. But isn't HTTP 429 as a part of additional HTTP status codes RFC, a standard code enough to be recognised and relayed?
Is there something crucial that I am missing here?
PS: Since this question is more about HTTP status spec, I asked here. If the community feels its more about apache, please feel free to vote to move the question to Server Fault.
I just stumbled upon your question because i was once again researching a similar problem, where our Apache Reverse Proxy returned a 500 status code on an ActiveSync Response 449.
I also found the Bugzilla entry you mentioned and the statments that it should have been fixed with version 2.2.18, however we use 2.2.22 and still faced the problem.
After further reading into the comments in the Bugzilla entry which then lead to the apache bug entry #44995. Reading these comments, especially the last one lead me to the believe that the issue, especially with custom error codes without status message has not been fixed in any 2.2.x versions but is included in 2.3/2.4
So we moved on and updated our reverse proxy to a 2.4 version, and to our surprise the error code 449 was correctly passed over by the proxy.
As you didnt mentioned your apache2 version used, i can only guess that an update to 2.4 or 2.3 might be a possible solution for you.
Related
I am trying to understand/reproduce Log4shell vulnerability, using this poc and also information from Marshalsec.
To do that, I've downloaded Ghidra v10.0.4, which is said (on Ghidra download page) to be vulnerable to log4shell. Installed it on an ubuntu VM, along with java 1.8 (as stated in POC), and loaded the Poc + marshalsec snapshot.
Tried to start Ghidra, it said java 11 was needed, so although I've installed java 1.8 I still downloaded java 11 and, when you start ghidra, it says the installed version is not good enough and ask for the path to a java11 version; so I just gave him path to the jdk11 directory and it seems happy with it. Ghidra starts alright.
Then set up my listener and launched the poc, got the payload string to copy/paste in ghidra, and got a response in the ldap listener saying it'll send it to HTTP. But nothing more. The end.
Since the HTTP server is set up by the same POC, I thought maybe I just couldn't see the redirection, so I started the http server myself, started the ldap server myself with marshalsec, and retried (see pics below for exact commands/outputs).
Setting http server:
Set listener:
Setting LDAP server:
Send payload string in Ghidra (in the help/search part, as shown in kozmer POC); immediately got an answer:
I still receive a response on the LDAP listener (two, in fact, which seems weird), but nothing on the HTTP. The the Exploit class is never loaded in ghidra (it directly sends me a pop-up saying search not found, I think it is supposed to wait for the server answer to do that?), and I get nothing back in my listener.
Note that I don't really understand this Marshalsec/LDAP thing so I'm not sure what's happening here. If anyone have time to explain it will be nice. I've read lot of stuff about the vuln but it rarely goes deeply into details (most is like: the payload string send a request to LDAP server, which redirect to HTTP server, which will upload the Exploit class on the vulnerable app and gives you a shell).
Note: I've checked, the http server is up and accessible, the Exploit.class file is here and can be downloaded.
Solved it.
Turned out for log4shell to work you need a vulnerable app and a vulnerable version of Java; which I thought I had, but nope. I had Java 11.0.15, and needed Java 11 (Ghidra need Java 11 minimum, only vulnerable version of Java 11 is the first one).
Downloaded and installed Java 11, POC working perfectly.
We have a Tomcat 7 running behind an Apache2 Server, connected via AJP.
Some AJAX-Requests we receive at the Tomcat do not contain any request Parameters.
This applies to GET and to POST requests as well.
Inspecting the Apache access_log the requests were all answered with a HTTP Status Code of 103:
IP - - [SNIP] "POST /redacted/ticket/1234567 HTTP/1.1" 103 655 "https://redactedhost.com/redacted/ticket/1234567" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
I can't figure where this Status Code comes from. Has anybody ever heard of it? As far as I have examined it is not a Custom Response from our Application and it is not defined in our Tomcat or Apache configuration.
HTTP 103 has been approved as status code for indicating hints to client.
More details here IETF
Official IANA Registry [1] says that code 103 is unassigned.
Some googling finds "103 Checkpoint" from "Resumable HTTP Requests Proposal" [2] [3] (I don't know whether that document in [3] is up-to-date, it is just from first page of search results. Usually such proposals should go further as an IETF draft). Note that it is not official, and response code 308 from [3] has already been assigned for different purpose by RFC 7538 [1].
As far as I have examined it is not a Custom Response from our Application
If it is not in plain view, it may be one of filters that you are using. Look through your libraries. You may try running with a debugger.
Do you see that response code at Apache HTTPD side only, or in Tomcat access log as well?
The Status Code 103 was an "unofficial" implementation which ended up in some (very few) products/software source codes, reason why people stumbled upon it in (rare) occasions.
History tells that the Status Code 103 was planned to respond when the request was installation/implementation dependent, which means its should appear when something (required) was missing or faulty in the Server installation or Network.
The message "Access denied while creating Web Service" was originally intended for the Status Code 103, but only few implementations used it. There were also cases it was simply translated as "Checkpoint".
For a while those messages made little to no sense, and the Status Code 103 was kept listed as "unofficial", until when it was deprecated even before becoming official, which happened some point in time around 2012.
However, in December 20, 2017, suggested by K. Oku, the IETF (Internet Engineering Task Force) retrieved and reset the Status Code 103 from "Deprecated" to "Experimental", retrofitting it as "Early Hints", as it has been listed in the RFC 8297.
You can read more its new purpose and utilization here: An HTTP Status Code for Indicating Hints
In 2018 The Status Code 103 was officially listed in the IANA HTTP Registry, even before being set as official by the IETF. Based on that, the odds are that the new meaning of the Status Code 103 should be safe to use and soon enough it shall became "official" by the IETF too. That is how it has been happening for a long time for other Status Codes.
I must express my appreciation and thank Ujjwal Gulecha for taking the time to bring this information forward on an old thread. It helps a lot keeping the information up-to-date in Stack Overflow when something in the industry change and affect the answers provided. Good job, bother!
So I've got a problem where a small percentage of incoming requests are resulting in "400 bad request" errors and I could really use some input. At first I thought they were just caused by malicious spiders, scrapers, etc. but they seem to be legitimate requests.
I'm running Apache 2.2.15 and mod_perl2.
The first thing I did was turn on mod_logio and interestingly enough, for every request where this happens the request headers are between 8000-9000 bytes, whereas with most requests it's under 1000. Hmm.
There are a lot of cookies being set, and it's happening across all browsers and operating systems, so I assumed it had to be related to bad or "corrupted" cookies somehow - but it's not.
I added \"%{Cookie}i\" to my LogFormat directive hoping that would provide some clues, but as it turns out half the time the 400 error is returned the client doesn't even have a cookie. Darn.
Next I fired up mod_log_forensic hoping to be able to see ALL the request headers, but as luck would have it nothing is logged when it happens. I guess Apache is returning the 400 error before the forensic module gets to do its logging?
By the way, when this happens I see this in the error log:
request failed: error reading the headers
To me this says Apache doesn't like something about the raw incoming request, rather than a problem with our rewriting, etc. Or am I misunderstanding the error?
I'm at a loss where to go from here. Is there some other way that I can easily see all the request headers? I feel like that's the only thing that will possibly provide a clue as to what's going on.
We set a lot of cookies and it turns out we just needed to bump up LimitRequestFieldSize which defaults to 8190. Hope this helps someone else some day...
I’m trying to get a mod_perl2 application ported to AWS. As part of the port I thought I’d move from Debian Squeeze to Wheezy with the latest stable mod_perl & Apache2 combination.
The application works right up to the point I try and write JSON responses to the client. At this point, each request is canceled on the client and on the server I get the error
Apache2::RequestIO::print: (103) Software caused connection abort
whenever I write to the client, i.e.:
$self->req->print($output);
I’ve tried tcpdumping the response to the client, and I can see it being written out, but no response is received on the client end and it just barfs chips. I can’t find any information on how to get around this.
I found quite a few people asking about this question on the net without many answers. The solution to my problem was very specific but I thought I’d post what I did anyway, it may help someone.
The client was canceling the request before the response was fully written, which was crapping out Apache::RequestIO (for reasons I still don’t know).
I couldn’t work out why I was seeing this behavior.
By using tcpdump I could see that data was being written out to the client – and it looked fine.
By inspecting the page in Chrome and looking at the network stack, I could see that my request for data was being canceled after no response was received (which was odd because the code worked fine on other servers and I could see the response was being written). Debugging was may harder because with Apache crashing out with an error in print IO I couldn’t check if the bytes written equaled the bytes of data. I wasn’t sure if something was getting stuck on the server side.
So, I changed the Content-Type of the response from application/json to text/html, so that I could query the page and just look at the actual response as text. Once I did that, I could see that the response was fine.
I started to look for other causes, and I found that in the migration to the new server, I’d missed altering some URLs in the DB to point to the new server, which meant my application was trying to get some data from the old DB.
This in turn was causing a load of timing issues, which was causing my problems. Once I fixed the config, the problems went away.
If you have to take a site down for some type of unavoidable maintenance task (and it's not a big enough site that you have a backup server), what HTTP status code should you have your server return to minimize the possibility that search engines will think the site is gone?
I found this list of status codes from W3C, of which the following seem applicable:
503 Service Unavailable
500 Internal Server Error
408 Timeout
404 Not Found
I think 503 is the most appropriate, but I don't know what search engines might prefer.
From the horse's mouth:
If my site is down for maintenance, how can I tell Googlebot to come back later rather than to index the "down for maintenance" page?
You should configure your server to return a status of 503 (network unavailable) rather than 200 (successful). That lets Googlebot know to try the pages again later.
Don't send a 404 -- they may remove you from their index.
I'd probably send a 503 and an appropriate Retry-After, although I don't know if anything actually uses the header.
According to Google the 503 code would be the way to go, since it means "the server is temporarily unavailable."
Also check out the W3C page on the same.