I've got a fairly normal setup in which Apache proxies requests to a servlet running inside Tomcat over the AJP protocol.
We've run this setup on Apache 2.0.46/Tomcat 5.0.28 for ages without problems but have recently updated to Apache 2.2.3/Tomcat 5.5.
The problem is that we've noticed that intermittently (maybe one time in 3) Apache will somehow convert the "Content-Type" HTTP header of a page served by the servlet from "text/html" to "text/plain", which results in the browser displaying the HTML source instead of rendering it.
Has anyone seen this sort of behavior before and know what might be the cause? I suspect we're doing something bad in our servlet code that the old version of Tomcat/Apache was more forgiving of.
Update: I have confirmed that it's Apache changing the headers. If I browse directly to Tomcat the problem doesn't occur.
Some webapps do not properly set mime types of content they serve, but still may work properly when served standalone because client applications like browsers are able to interpret the type of the content. But when served behind Apache, these apps will not behave correctly because Apache will provide a default type of text/plain.
A solution is to add a DefaultType None line to your apache virtual host for these web apps:
DefaultType None
http://httpd.apache.org/docs/2.2/mod/core.html#defaulttype
From my blog post:
http://patternbuffer.wordpress.com/2011/11/30/mime-type-issue-with-apache-mod_jk-and-mod_proxy-serving-plain-text/
If you're seeing this problem intermittently, it's almost certain to be something in the servlet code rather than a misconfiguration of Tomcat or httpd. Do you have logging that you can turn on to print the contents of the HTTP headers ?
To isolate the problem a bit further, you could also try bypassing httpd and going direct to the Tomcat URLs for your pages.
I haven't seen this particular behaviour before myself, so sorry I can't be more specific.
By intermittent, do you mean that some pages exhibit this behaviour and others don't, or that there are pages that sometimes exhibit the behaviour and sometimes not?
Can you attach any logging to the AJP layer to log HTTP headers at that level, so you can verify whether it's Apache or Tomcat adding the bogus header?
Are you proxying back to a cluster? Maybe one of the servers is configured wrong.
Ok. I figured it out, it was a bug in the servlet code:
We were doing something like this to write serialized Java objects as the result of HTTP requests:
DeflaterOutputStream dos = new DeflaterOutputStream(response.getOutputStream());
ObjectOutputStream oos = new ObjectOutputStream(dos);
response.setContentType("application/x-java-serialized-object");
oos.writeObject(someObject);
What seemed to be happening was that the DeflaterOutputStream and ObjectOutputStream would get garbage-collected three or four requests later when they were still attached to the response object's output stream and this would cause something to happen on the stream that confused Apache and caused it to rewrite the headers.
I replaced the above with:
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
DeflaterOutputStream dos = new DeflaterOutputStream(byteStream);
oos = new ObjectOutputStream(dos);
response.setContentType("application/x-java-serialized-object");
oos.writeObject(someObject);
oos.flush();
dos.finish();
byteStream.writeTo(response.getOutputStream());
and the problem has gone away.
The following links seem to describe a similar problem:
AJP Flush Packet causing text/plain
ASF Bugzilla – Bug 43478
I was also facing the same issue it got resolved.
If problem in only one folder then there is some servlet that is blocking the request/response and making a customize request/response to tomcat.
Tomcat 7.0.x
Related
We developed a web application which uses opencmis and a windows client which uses dotcmis. The web application runs behind an apache httpd.
We are facing the following problem:
Small files can be uploaded by the client without problems (< 1,5 gigabytes).
However, if we try to upload larger files, we get a "Proxy Error". The stacktrace does not give any more information.
We also tried to upload via cmis workbench with the same result...
Are there any configuration parameters for apache we maybe overlooked? Or do you think the problem should be searched elsewhere?
EDIT: I should mention, that the file is uploaded completely nevertheless. And also: We tried disable apache, connect via http instead of https and upload a file and it works perfectly.
EDIT2: We found a solution, although it does not seem to be a very good one... We set the following configuration entries in httpd.conf:
Timeout=500 and ProxyTimeout=500. Default value is 60 for these entries.
This solved the problem. However, it would be nice to know, why this problem occures in the first place.
Greets
I'm trying to use Liferay for http and https
if I include in portal-ext.properties:
company.security.auth.requires.https=true
web.server.protocol=https
Will be working ok with https but in http is showing incorrect themes due is trying to load https://domain.com/theme
If I remove this two lines is working ok for http but not for https.
What can I do?
IMHO mixed mode, e.g. offering http as well as https never gives you what you expect: You expect security from https, but you always risk leaking session information, e.g. being vulnerable to session-hijacking attacks (ala Firesheep). My actual advice would be to go https only if you do https for security. Read on if that's not an option for you, but don't complain when you find information leaking (this is not dependent on Liferay, but for any web-based environment)
What is the exact problem that you have with the themes? (images/css through http?) Which version of Liferay are you using?
Before you specify more, you might want to configure your theme's "virtual path", this will rewrite all the URLs referring to your theme. It's typically used to serve static resources through a webserver or cdn, but it works with any kind of URL. Simply using a protocol-relative URL should work (I love this mostly unknown http feature):
Add this to your theme's liferay-look-and-feel.xml:
<look-and-feel>
<theme id="my" name="My Theme">
<virtual-path>//domain.com/myTheme</virtual-path>
</theme>
</look-and-feel>
note that the URL omits the protocol part, http: or https:, thus the browser will use the same protocol that the whole page is loaded with.
Edit: corrected the xml. Will investigate if there's a problem with protocol-relative URLs in themes.
Edit 2: Something is weird. It seems, virtual-path does not work like this, but I recall it did earlier. Do you add domain.com as cdn.host.http or cdn.host.https? (this would be concatenated)
On related stuff, please check if you're running Apache in front of your appserver. In this case you might forward some traffic for the portal (e.g. in the virtual host for http) but not forward the traffic in the https virtual host.
I am trying to setup Nginx + Unicorn + Rails 3. Nginx will also serve some static and php projects. However when I open the site I always see a
400 Bad Request
Request Header Or Cookie Too Large
error page. Nothing in the access nor error logs.
/etc/nginx
nginx.conf https://gist.github.com/1117152
php.conf https://gist.github.com/1117154
drop.conf https://gist.github.com/1117158
/etc/nginx/sites-enabled
https://gist.github.com/1117161
I am pretty stuck here because I don't see anything in logs.
hmm of course it's the users fault. I had a wrong references to the socket in the site-available conf and an endless loop was the result. I fixed it in the gist.
Check your "large_client_header_buffers". You might need to use a larger value if your application require
This came high up on Google so to save time for people in the same situation:
I included the JavaScript FB_Init (Facebook) on a page and had the "cookie=true" which produced an amazingly big cookie. Turning this to false was the solution and resolved the identical message.
Note: Made some updates based on new information. Old ideas have been added as comments below.
Note: Made some updates (again) based on new information. Old ideas have been added as comments below (again).
We are running two instances of CouchDB on separate computers behind Apache reverse proxies. When attempting to replicate between the two instances:
curl -X POST http://user:pass#localhost/couchdb/_replicate -d '{ "source": "db1", "target": "http://user:pass#10.1.100.59/couchdb/db1" }' --header "Content-Type: application/json"
(we started using curl to debug the problem)
we receive an error similar to:
{"error":"case_clause","reason":"{error,\n {{bad_return_value,\n {invalid_json,\n <<\"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /couchdb/db1/_local/01e935dcd2193b87af34c9b449ae2e20 was not found on this server.</p>\\n<hr>\\n<address>Apache/2.2.3 (Red Hat) Server at 10.1.100.59 Port 80</address>\\n</body></html>\\n\">>}},\n {child,undefined,\"01e935dcd2193b87af34c9b449ae2e20\",\n {gen_server,start_link,\n [couch_rep,\n [\"01e935dcd2193b87af34c9b449ae2e20\",\n {[{<<\"source\">>,<<\"db1\">>},\n {<<\"target\">>,\n <<\"http://user:pass#10.1.100.59/couchdb/db1\">>}]},\n {user_ctx,<<\"user\">>,\n [<<\"_admin\">>],\n <<\"{couch_httpd_auth, default_authentication_handler}\">>}],\n []]},\n temporary,1,worker,\n [couch_rep]}}}"}
So after further research it appears that apache returns this error without attempting to access CouchDB (according to the log files). To be clear when fed the following URL
/couchdb/db1/_local/01e935dcd2193b87af34c9b449ae2e20
Apache passes the request to CouchDB and returns CouchDB's 404 error. On the other hand when replication occurs the URL actually being passed is
/couchdb/db1/_local%2F01e935dcd2193b87af34c9b449ae2e20
which apache determines is a missing document and returns its own 404 error for without ever passing the request to CouchDB. This at least gives me some new leads but I could still use help if anyone has an answer offhand.
The source CouchDB (localhost) is telling you that the remote URL was invalid. Instead of a CouchDB response, the source is receiving the Apache httpd proxy's file-not-found response.
Unfortunately, you may have some reverse-proxy troubleshooting to do. My first guess is the Host header the source is sending to the target. Perhaps it's different from when you connect directly from a third location?
Finally, I think you probably know this, but the path
/couchdb/db1/_local%2F01e935dcd2193b87af34c9b449ae2e20
Is not a standard CouchDB path. By the time CouchDB sees a request, it should have the /couchdb stripped, so the query is for a document called _local%2f... in the database called db1.
Incidentally, it is very important not to let the proxy modify the paths before they hit couch. In particular, if you send %2f then CouchDB had better receive %2f and if you send / then CouchDB had better receive /.
From official documentation...
Note that HTTPS proxies are in theory supported but do not work in 1.0.1. This is because 1.0.1 ships with ibrowse version 1.5.5. The CouchDB version in trunk (from where 1.1 will be based) ships with ibrowse version 1.6.2. This later ibrowse contains fixes for HTTPS proxies.
Can you see which version of ibrowse is involved? Maybe update that ver?
Another thought I have is with regard to the SSL certs. If you don't have any, and I know you don't :), then technically you're doing SSL wrong. In java we know there are ways around this, but maybe try putting in proper certs since all SSL stuff basically involves certs.
And for my last contribution (today) I would say have you looked through this document which seems highly relevant?
http://wiki.apache.org/couchdb/Apache_As_a_Reverse_Proxy
I have the following system configured:
Tomcat -> Apache
Now, I have some URLs on which I have Max-Age, LastModified and Etags set.
My expectation is when Client1 makes a call to the server, the page should get served from tomcat, but should get cached in the mod_cache module of Apache. So that when next client makes a call, the page is served from Apache and it doesnt have to hit the Tomcat server, if the page is still fresh. If the page isnt fresh, Apache should make a Conditional Get to validate the content it has.
Can someone tell me if there is any fundamental mistake in this thinking? It doesnt happen to work that way. In my case, when client2 makes a call, it goes straight to the Tomcat server(not even a Conditional Get).
Is my thinking incorrect or my Apache configuration incorrect?! Thanks
The "What can be cached" section of the docs has a good summary of factors - such as response codes, GET request, presence of Authorization header and so on - which permit caching.
Also, set the Apache LogLevel to debug in httpd.conf, and you will get a clear view of whether or not each request got cached. Check the error logs.
You should be able to trace through what is happening based on these two.