I'm currently in the process of configuring Apache httpd to direct all traffic for '/api/foo' to a specific Tomcat instance via AJP (using mod_proxy_ajp). For this purpose I have the following ProxyPass rule in my httpd configuration file:
ProxyPass /api/foo ajp://localhost:9999/api/foo connectiontimeout=300 timeout=300 retry=3
This Tomcat instance has the following connector defined in its server.xml:
<Connector port="9999" protocol="AJP/1.3" redirectPort="9443"/>
With this configuration I reach my Tomcat instance correctly when visiting /api/foo. However, when the URL becomes bigger than 300 characters I can't seem to reach my Tomcat instance, sometimes.
The first hour it could be that I get a timeouts and the other hour it could be that everything comes through just fine.
When I get timeouts, I see the following errors in my httpd error log:
[error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[error] ajp_read_header: ajp_ilink_receive failed
[error] (70007)The timeout specified has expired: proxy: read response failed from 127.0.0.1:9999 (localhost)
And the following result in my httpd access log:
"GET /api/foo/barrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr HTTP/1.1" 503 323 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0"
The strange thing is that this request doesn't seem to reach my Tomcat instance. I don't see anything reaching my application logging, and I also don't see the request in my Tomcat access log.
Does anyone have an idea what could be the cause of this problem? The timeout is set to more than 1 minute, so I'd expect the request to at least reach my Tomcat instance...
As a last note, I've tried setting the AJP connector's packageSize to 65536 and Apache httpd's LimitRequestFieldSize and ProxyIOBufferSize to 65536. This didn't solve the problem.
In the absence of a better answer I though I would share my experience with a similar and possibly the same problem and give you a recommendation to change the connector you are using.
I have also received the same ajp_read_header: ajp_ilink_receive failed errors while using mod_proxy_ajp in slightly different environments using apache 2.2 & tomcat 6.
I've tried many various and recommended fixes including the connector timeout and other connector settings when we encountered our issue.
From what I've read, some of these will work in different scenarios and the error ajp_ilink_receive failed message seems to be quite generic. In your case this occurs when the URL is longer than 300 characters - well sometimes at least..
After trying many of the different connector properties both from within the tomcat server.xml and the apache ProxyPass settings appended after the ajp:// URI I pretty much gave up going down this path. Since then I have tried mod_proxy and mod_jk which both seem fix the issue or at the very least there is no more ajp error. But this wouldn't happen with mod_proxy anyway because it doesnt connect with ajp.
So I suggest to change your tomcat connector to either mod_proxy or mod_jk. There are various write ups on the differences but the pages here and here on tomcat expect are a reasonably good source of information. You;ll want to choose between mod_proxy and mod_jk based on your setup. mod_jk, while different, is actually not too hard to set up (but you may have to build the mod_jk.so for apache).
Summarising the different connectors;
mod_jk: highest recommendation. ajp connector. purpose designed. different configuration setup that resides outside apache. difficult to configure when connecting a webapp with a certian name to a different url name.
mod_proxy: http connector. reportedly more stable than mod_proxy_ajp. problems with passing ssl variables?
mod_proxy_ajp: least recommended but most common. ajp connector. extension of mod_proxy. easy implementation.
If this doesn't work for some reason. Others have had success looking into problems on the tomcat side such as long running queries.
Related
I am able to reach the GeoServer log-in form on a new server instance using a URL like this: https://sub.domain.tld/geoserver.
However, entering the correct username and password causes Cannot POST /geoserver/j_spring_security_check to be displayed in plain text in the browser window and "http://localhost:8080/geoserver/j_spring_security_check" to appear on the address line (Chrome and Firefox). Logging-in works just fine from http://sub.domain.tld:8080/geoserver and, while logged in that way, the SSL URL will skip the log-in form entirely and open the full GUI -- but none of the example layer previews will work (Cannot GET /geoserver/tiger/wms ...or whatever layer).
I have 3 similar servers set up with earlier software versions that all work perfectly:
CentOS Linux release 7.5.1804 (Core)
Apache 2.4.6
Tomcat 9.0.4.0
JVM 1.8.0_171-b10 (Oracle)
GeoServer 2.15.1 (also 2.14.1)
This latest server is:
CentOS Linux release 7.9.2009 (Core)
Apache 2.4.6
Tomcat 9.0.48.0
JVM 1.8.0_292-b10 (Red Hat)
GeoServer 2.19.1
I closely followed the GeoServer install procedure found here. Apache was already installed and working with a virtual host configuration using certbot and a Let's Encrypt certificate. I added the following to the <VirtualHost *:443> section of the /etc/httpd/sites-available/sub.domain.tld.conf file:
ProxyRequests Off
ProxyPass /geoserver http://localhost:8080/geoserver
ProxyPassReverse /geoserver http://localhost:8080/geoserver
<Location "/geoserver">
Order allow,deny
Allow from all
Header set Access-Control-Allow-Origin "*"
</Location>
I also set "https://sub.domain.tld/geoserver/" as the Proxy Base URL in GeoServer. This is exactly how my working instances are set up.
The newest /opt/tomcat/webapps/geoserver/WEB-INF/web.xml file is slightly different in that it has separate CORS sections to be uncommented for Jetty vs. Tomcat. The Tomcat section and the cross-origin filter-mapping are uncommented. Unlike before, I had to enable the SELinux httpd_can_network_connect process in order to get to the GeoServer log-in form via SSL. Disabling SELinux enforcement, however, does not solve the j_spring_security_check problem.
Sensitive WMS and WFS content must be served via SSL. I am now spinning my wheels on research so any help in resolving this would be greatly appreciated!
I was running into a similar problem trying to reverse proxy a Geoserver docker container using Apache.
After bashing my head for a few days, I found I needed a couple of changes on both ends of the setup.
Apache:
<Location "/geoserver">
ProxyPreserveHost On
ProxyPass http://localhost:8080/geoserver
ProxyPassReverse http://localhost:8080/geoserver
</Location>
The ProxyPreserveHost directive should ensure that sub.domain.tld is what's used rather than localhost.
However, at this point I hit another problem, no matter what I did the login & logout URLs were using http:// and not https://
This required adjusting the server config of Tomcat, specifically the scheme used by the connector.
Tomcat:
<Connector
port="8080"
protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
scheme="https"
/>
Note the value of scheme is https, normally it's http because it's the http connector. This connector doesn't SSL/TLS encrypt the traffic so it would, under normal circumstances (appropriately), set the protocol to http.
Since we're using a proxy for SSL/TLS encryption we don't need Tomcat to do that work but we still need to tell Tomcat to describe the server name using https instead of the usual http.
Alternatively, you could also look at using the AJP connector which is an entirely different protocol but gets around some of these issues rather neatly and can be more performant.
The issue with this approach is that it requires more Apache mods to be enabled to work as well as some security concerns given how much more powerful AJP can be VS the http connector.
Also, there are some other proxy settings that can be used in Tomcat to possibly remove the need for ProxyPreserveHost in Apache, but this should get you where you're going.
Additional reading:
Tomcat HTTP Connector docs: https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
Tomcat AJP docs: https://tomcat.apache.org/tomcat-9.0-doc/config/ajp.html
Currently, the server at work is underperforming and the way it's set up is not ideal either. For this reason I'm trying to find a new way to do things that will hopefully help with both, performance and deployment.
The approach I decided for is to have tomcat instances for our webapps (currently there are two, so it'd be an instance per webapp) and use Apache as a "front". I'm not experienced in this, so It's normal I'm having issues here and there, but so far I've manage to get this going.
What I expect is to redirect from mysite.com index page to either mysite.com/service1 or mysite.com/service2. Service1 was setup in out test server at port 8080 and service2 at 8081. I installed Apache2 and mod_jk yesterday and set up apache with the contents of mysite.com. Today I started the configurations, that ended up as follow:
workers.properties
worker.list=s1
worker.s1.type=ajp13
worker.s1.port=8009
#host is localhost by default according to the documentation
jk.load
LoadModule jk_module /usr/lib/apache2/modules/mod_jk.so
JkWorkersFile /etc/apache2/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel debug
JkMount /service1/* s1
Service1's server.xml connector (The rest is all default)
<Connector protocol="AJP/1.3" port="8009" redirectPort="8443" />
I had more, but because of the errors, I took a step back and tried with only one tomcat for now. I will add the second tomcat and a loadbalancer.
Ok, so what's going on?
I can access the server and the index page of our system with no problem. The problem is when I try to redirect to service1. It just loads without response, but if I try to access service1 directly by port 8080, it works properly (I tired commenting out this connector. No luck).
Looking at server-status, I see the request stuck at w/sending reply, and in mod_jk.log I see that the worker properly matches the request. So while my configurations seem to be right, there is something in between happening. I don't really know if it's something with Apache, Tomcat or Mod_jk. I also tried to follow several guides of how to do this, but all of them got me to 404s. Looking around here and ServerFault didn't shed much light unfortunately so I'm the one asking now.
Am I missing something? Should I just use another approach? I'm very new at this and I'm at loss right now. The configuration and the logs show that nothing is really wrong (at first glance, at least...) so I'm entirely sure if my case scneario is even posible with mod_jk... HOnestly to run it back and try with proxy is very tempting at this point, but if I am, I'd rather know where Im wrong.
Additional info: Running on Ubuntu Server 18.04, lastest apache2 and mod_jk avaliable from apt (as of Apr 14), java 1.8 and Tomcat 8.5.64.
There was a change in Tomcat last year (from version 8.5.51 and version 9.0.31), which introduced a secretRequired attribute to the AJP connector with a default of true (cf. documentation). Hence you can either:
add a shared secret between the AJP connector and mod_jk
or add secretRequired="false" to the AJP connector:
<Connector protocol="AJP/1.3" port="8009" secretRequired="false" redirectPort="8443" />
Remark: AJP is a very old protocol and rarely used. Since your installation is pretty new, you might consider using directly HTTP (cf. this talk).
For various reasons I want to shutdown my server after a certain period of idle time. I am running Tomcat 8.5.29 and Apache2 (not sure the version) on Debian 4.9.88. I wrote a script to look at the last time Tomcat had an access. I only have one app on the server and it is at "http://hostname/source/". My problem is that there are number of webserver vulnerabilities out there and I am getting a constant flow of requests to:
"GET / HTTP/1.1"
"POST /GponForm/diag_Form?images/ HTTP/1.1"
"GET /jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.system%3Aservice%3DMainDeployer HTTP/1.1"
"POST /user/register?element_parents=account/mail/%23value&ajax_form=1&_wrapper_format=drupal_ajax HTTP/1.1"
What I would like to do is stop anything that doesn't begin with "/source" from even getting to tomcat. I presume that a rewrite or something in Apache can do it, but I am not sure how the requests get to Tomcat in the first place.
Any ideas?
There are two possible ways for requests to reach Tomcat:
The probes send requests directly to the Tomcat port (typically port 8080). You can fix this by restricting Tomcat's listening address to the loopback address, by adding the attribute address="127.0.0.1" to the corresponding Connector element in conf/server.xml. Or you can just block port 8080 in your firewall.
The requests are forwarded from Apache to Tomcat via a reverse proxy configuration in Apache. This means there is a line such as the following in one of the Apache configuration files:
ProxyPass / http://127.0.0.1:8080/
If you add an explicit path prefix to both arguments, you can restrict which requests are passed to Tomcat:
ProxyPass /source http://127.0.0.1:8080/source
This ensures that only requests that begin with "/source" are forwarded to Tomcat.
Some Apache configurations use the AJP protocol instead of HTTP for proxying, but the same reasoning applies.
We are running Tomcat 7 behind a load balancer that works also as SSL terminator, and an Apache HTTP Server 2.4. The Apache connects to the Tomcat via mod_proxy_ajp.
For the application it is important that Tomcat is aware that the request is coming in via HTTPS and is thus secure. As e.g. this article recommends, it is common to configure this on the Tomcat's Connector using the attributes secure="true" and possibly scheme="https" proxyPort="443". While this works, it is inconvenient since we are using HTTP for some purposes as well, and thus we would need to set up two Tomcat connectors for this purpose. And it has a smell, since this way we basically tell Tomcat to override the wrong information it gets from the Apache HTTP Server that the request is HTTPS instead of HTTP, instead of telling the Apache that it should send the correct information on the protocol and secure status.
So my question: is it somehow possible to configure the Apache HTTP Server itself that it passes the correct information via the AJP protocol: that the request is received via HTTPS and is secure? The problem is that it doesn't know it's HTTPS, since there is a SSL terminator before it and the requests arrives via HTTP, as far as it is concerned. Can I tell the Apache somehow that it's actually HTTPS?
A partial solution seems to be to set the protocol on a ServerName directive in the virtual host in the Apache HTTP server:
ServerName https://whatever
This way any Location: headers in redirects seem to be rewritten to https in the Apache, but the Tomcat is still passed the wrong information via AJP.
I always thought that AJP transfers this information automagically - but I'm not using mod_proxy_ajp, rather mod_jk. It's one of the reasons why I much prefer AJP over HTTP (and proxying).
Might be worth to change the module/connection
I've been dealing with this whole day and still can't figure it out.
I've setup Zabbix on one machine, and I want to monitor the Apache server on another machine.
I've completed all the steps described in the docs: http://www.zabbix.com/wiki/templates/apache
and still I get no data in the Apache Template. When checking the logs on the Apache server, I can see in access.log:
IPADDR - - [16/Jul/2012:13:29:08 +0000] "GET /server-status?auto HTTP/1.0" 404 13826 "-" "Python-urllib/1.17"
I think it might has something to do with Virtual Servers and additional sites I have on that machine, but I can't figure it out, and nothing mentioned in the docs...
The Apache checks are not as clever as you may think.
Can Zabbix communicate with your apache server? Link it to a template with something simple like "uptime" and verify that it indeed gets data.
Next, verify that there aren't any firewall rules prevent the zabbix server from communicating with your web server. Can you curl your homepage from the zabbix host without problems?
Are the apache checks active checks? If so you'll need to make sure active checks are enabled in the /etc/zabbix/zabbix-agentd.conf file and that the "Hostname" within the conf is unique and matching up correctly with what you have in the zabbix server.
If that fails, change the DebugLevel to 4 in /etc/zabbix-agentd.conf and tail the zabbix log. Look and see if it is having trouble with any checks.
This is an apache configuration problem, zabbix can't do anything if /server-status yields a 404 error.
Maybe the <Location /server-status>...</Location> directive is not at the right place in apache's configuration.
Try to move it inside the <VirtualHost> section of the specific virtual server to which the GET /server-status is routed.
Also make sure that mod_status is enabled.