Httpd returning 503 Service Unavailable with mod_proxy for Tomcat 8 - apache

I'm trying to integrate Tomcat with Apache. My aim is to redirect all the requests with
http://localhost/myapp to http://localhost:8080
I followed this guide: http://tomcat.apache.org/tomcat-8.0-doc/proxy-howto.html
My httpd.conf looks like this:
Include conf.modules.d/*.conf
LoadModule proxy_module modules/mod_proxy.so
ProxyPass /myapp http://localhost:8080 retry=0 timeout=5
ProxyPassReverse /myapp http://localhost:8080
My server.xml in apache-tomcat looks like this:
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" proxyPort="80" />
Now when I try the url http://localhost/myapp, it gives 503 Service Unavailable error.
Both Tomcat and Apache are up and running. The URL http://localhost:8080 works fine.
Can there be an issue with file permissions?
For tomcat the user and group are root/root and for httpd, the user and group are apache/apache
Am I missing something or am I doing it wrong?
Httpd version is 2.4.6 and Tomcat's version is 8.0
The httpd error logs:
[proxy:error] [pid 19905] (13)Permission denied: AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[proxy:error] [pid 19905] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[proxy_http:error] [pid 19905] [client ::1:51615] AH01114: HTTP: failed to make connection to backend: localhost
Solved!
The answer is here: http://sysadminsjourney.com/content/2010/02/01/apache-modproxy-error-13permission-denied-error-rhel/

(Answered by the OP in a question edit. Converted to a community wiki answer. See Question with no answers, but issue solved in the comments (or extended in chat) )
The OP wrote:
The answer is here: http://sysadminsjourney.com/content/2010/02/01/apache-modproxy-error-13permission-denied-error-rhel/
Which is a link to a blog that explains:
SELinux on RHEL/CentOS by default ships so that httpd processes cannot initiate outbound connections, which is just what mod_proxy attempts to do.
If this is the problem, it can be solved by running:
/usr/sbin/setsebool -P httpd_can_network_connect 1
And for a more definitive source of information, see https://wiki.apache.org/httpd/13PermissionDenied

Resolve issue Immediate, It's related to internal security
We, SnippetBucket.com working for enterprise linux RedHat, found httpd server don't allow proxy to run, neither localhost or 127.0.0.1, nor any other external domain.
As investigate in server log found
[error] (13)Permission denied: proxy: AJP: attempt to connect to
10.x.x.x:8069 (virtualhost.virtualdomain.com) failed
Audit log found similar port issue
type=AVC msg=audit(1265039669.305:14): avc: denied { name_connect } for pid=4343 comm="httpd" dest=8069
scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket
Due to internal default security of linux, this cause, now to fix (temporary)
/usr/sbin/setsebool httpd_can_network_connect 1
Resolve Permanent Issue
/usr/sbin/setsebool -P httpd_can_network_connect 1

this worked for me by editing my *.conf file in apache:
ProxyRequests Off
ProxyPreserveHost On
RewriteEngine On
<Proxy http://localhost:8123>
Order deny,allow
Allow from all
</Proxy>
ProxyPass /node http://localhost:8123
ProxyPassReverse /node http://localhost:8123

On CentOS Linux release 7.5.1804, we were able to make this work by editing /etc/selinux/config and changing the setting of SELINUX like so:
SELINUX=disabled

Related

Flask app with Apache proxy + Gunicorn not working on HTTPS

Updates in the bottom, I kind of solved it but not sure if the solution is a correct one.
I have Apache running on CentOS with a proxy to localhost port 8080 where I have Flask app running using Gunicorn. This setup works on Apache port 80 (HTTP) and I can connect to it using my domain http://example.com with a browser but now I have tried to setup SSL/HTTPS and it just doesn't work.
Navigating to https://example.com tries to load the page for a while (like 30ish seconds) and then it shows 502 error page:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.
Reason: Error reading from remote server
Apache error log:
[proxy_http:error] [pid 30209] (103)Software caused connection abort: [client xx.xxx.xxx.xxx:60556] AH01102: error reading status line from remote server localhost:8080
[proxy:error] [pid 30209] [client xx.xxx.xxx.xxx:60556] AH00898: Error reading from remote server returned by /
Gunicorn error log (first 7 lines are from Gunicorn startup, no idea why this info is in error log, last 3 lines are when the HTTPS request returns 502 error):
[29478] [INFO] Listening at: http://127.0.0.1:8080 (29478)
[29478] [INFO] Using worker: sync
[29480] [INFO] Booting worker with pid: 29480
[29481] [INFO] Booting worker with pid: 29481
[29482] [INFO] Booting worker with pid: 29482
[29483] [INFO] Booting worker with pid: 29483
[29484] [INFO] Booting worker with pid: 29484
[29478] [CRITICAL] WORKER TIMEOUT (pid:29480)
[29480] [INFO] Worker exiting (pid: 29480)
[29554] [INFO] Booting worker with pid: 29554
Apache config which is working for HTTP (/etc/httpd/conf/httpd.conf):
Listen 80
#other default config values here
<VirtualHost *:80>
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
</VirtualHost>
IncludeOptional conf.d/*.conf
Apache config which is not working for HTTPS (/etc/httpd/conf.d/ssl.conf):
Listen 443 https
#other default config values here
<VirtualHost *:443>
#other default config values here too
SSLCertificateFile /etc/pki/tls/certs/cert.pem
SSLCertificateKeyFile /etc/pki/tls/private/cert.key
SSLProxyEngine on
ProxyPass / https://localhost:8080/
ProxyPassReverse / https://localhost:8080/
</VirtualHost>
If I remove/comment the SSLProxyEngine, ProxyPass amd ProxyPassReverse lines and restart Apache I get the default Apache welcome page and HTTPS works just fine so clearly the problem is in the Proxy somehow?
The flask app is started with Gunicorn by:
gunicorn --config gunicorn_config.py app:app
gunicorn_config.py:
workers = 5
bind = '127.0.0.1:8080'
umask = 0o007
reload = True
accesslog = 'log_gunicorn_access.txt'
errorlog = 'log_gunicorn_error.txt'
app.py:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello world!'
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080)
And once again, this works when navigating to my domain using HTTP but doesn't work when using HTTPS.
Any help?
UPDATE:
I managed to get another error. Now navigating to https://example.com loads instantly and shows 500 error page:
Proxy Error
The proxy server could not handle the request GET /.
Reason: Error during SSL Handshake with remote server
Apache error log:
[proxy:error] [pid 32385] (502)Unknown error 502: [client xx.xxx.xxx.xxx:50932] AH01084: pass request body failed to [::1]:8080 (localhost)
[proxy:error] [pid 32385] [client xx.xxx.xxx.xxx:50932] AH00898: Error during SSL Handshake with remote server returned by /
[proxy_http:error] [pid 32385] [client xx.xxx.xxx.xxx:50932] AH01097: pass request body failed to [::1]:8080 (localhost) from xx.xxx.xxx.xxx ()
No more any errors in Gunicorn error log.
I added these two lines to gunicorn_config.py:
keyfile = '/etc/pki/tls/private/cert.key'
certfile = '/etc/pki/tls/certs/cert.pem'
and made sure both files are accessible by the user running Gunicorn (chmod o+r cert.key/pem).
No idea if I was supposed to change it like this as I thought the traffic should go like: client --https--> Apache and then Apache --http--> Gunicorn.
Also HTTP (http://example.com) no longer works and gives the previous 502 error page but I guess running Gunicorn with the cert configs doesn't allow HTTP anymore and would need to run the app twice with different configs).
UPDATE 2:
I added more Apache logging by adding this line to /etc/httpd/conf.d/ssl.conf inside virtual host:
LogLevel info
And now I got additional info in Apache error log:
[ssl:info] [pid 3808] [remote 127.0.0.1:8080] AH02411: SSL Proxy: Peer certificate does not match for hostname localhost
Then I added new line to /etc/httpd/conf.d/ssl.conf inside virtual host:
SSLProxyCheckPeerName off
And now I got another Apache error:
[ssl:info] [pid 3999] [remote 127.0.0.1:8080] AH02005: SSL Proxy: Peer certificate CN mismatch: Certificate CN: example.com Requested hostname: localhost
Added new line to /etc/httpd/conf.d/ssl.conf inside virtual host:
SSLProxyCheckPeerCN off
Aaaand now navigating to https://example.com correctly works and I get "Hello world" back from the app!
Now I guess my question needs update as well: Is it bad practice, wrong or insecure to use SSLProxyCheckPeerName off and SSLProxyCheckPeerCN off in this context? Or is there a better way as I don't think there's a way to order an official SSL certificate on localhost?
You're using
ProxyPass / http://localhost:8080/
and
ProxyPass / https://localhost:8080/
(note the 1 letter difference).
Your localhost:8080 will serve either http or https. Based on your description (and common expectations) it's serving http. If you proxy even your :443 virtual host to http, it'll work better.
You might run into more issues, as the proxied application doesn't really know that it's actually served through https, but that's a different beast than this question.

Apache reverse proxy help needed

I've got a somewhat unique (maybe not, but unique to me) architecture here for some home apps that variously require PHP 5.6 and PHP 7.1. This is the architecture recommended to me by /r/docker:
One container acting as Apache Reverse Proxy that will ingest a DNS request on port 80 and then pass it to one of the below containers' ports.
One container running Apache with PHP 5.6, listening on port 8081.
One container running Apache with PHP 7.1, listening on port 8080.
I've been able to get domainname.com to work this way, by setting up the following domainname.com.conf in /etc/apache2/sites-available:
<VirtualHost *:80>
ServerName domainname.com
<Proxy *>
Allow from localhost
</Proxy>
ProxyPass "/" "http://public.ip.address:8080/" retry=0
ProxyPreserveHost On
</VirtualHost>
However, one of my subdomains needs to push to an application on the PHP 5.6 side. So I've set up this subdomain.domainname.com.conf:
<VirtualHost *:80>
ServerName subdomain.domainname.com
<Proxy *>
Allow from localhost
</Proxy>
ProxyPass "/" "http://public.ip.address:8081/subdomain/" retry=0
ProxyPreserveHost On
</VirtualHost>
Indeed, going directly to http://public.ip.address:8081/subdomain/ loads the application just fine, but for whatever reason, going to http://subdomain.domain.com gives a 503. Originally it was just showing the Apache on Ubuntu page, which is when I realized I needed to a2ensite and reload Apache, which has since been done.
The logs show this:
[Wed Jul 10 21:36:59.049131 2019] [proxy:error] [pid 2298:tid 139882856969984] (113)No route to host: AH00957: HTTP: attempt to connect to public.ip.address:8081 (3.220.176.240) failed
[Wed Jul 10 21:36:59.049182 2019] [proxy_http:error] [pid 2298:tid 139882856969984] [client 128.229.4.2:58611] AH01114: HTTP: failed to make connection to backend: public.ip.address
We know that the system can access 8080 just fine, so I'm not sure why it's struggling with this traffic. As mentioned above, hitting the ProxyPass URL directly from the outside loads the page fine, so traffic inbound to that container is fine. SELinux is disabled. This is in AWS and Security Groups are configured properly as far as I can tell (see: traffic works from outside).
Using the public IP address is the only way I've been able to get inter-container traffic to work, because using Docker's bridge IPs didn't work previously.
On the old system, when all my apps were just PHP 5.6, I had a standard VirtualHost statement that just took in the subdomain.domain.com and pointed it to a specific DocumentRoot. I can't do that here since the reverse proxy container needs to hand off the traffic to the container with the right PHP version.
Many thanks for any tips or tricks here.

Trouble setting up SSL to work with MAMP 5.3

After adding a self-signed SSL certificate, I am unable to get my Drupal site to work on localhost.
I have attempted various proposed solutions that I have found online but none have gotten me past a 400 error at https://localhost/
I have uncommented the following in httpd.conf:
LoadModule ssl_module modules/mod_ssl.so
I have Listen set to port 80 in this same file. When I set it to the SSL channel, 443, Apache does not load.
In my httpd-ssl.conf file:
I have set the paths for my server.crt and server.key files correctly.
I have enabled SSLEngine
<VirtualHost *:443>
# General setup for the virtual host
DocumentRoot "/Applications/MAMP/Library/htdocs"
ServerName https://127.0.0.1:443
ServerAdmin you#example.com
ErrorLog "/Applications/MAMP/Library/logs/error_log"
TransferLog "/Applications/MAMP/Library/logs/access_log"
I have tried a variety of possibilities for the ServerName. The one above along with localhost:443 both lead to 400 errors.
I have Listen set to 443 here. Setting to 80 so it matches the httpd.conf file leads to the same result described above...not able to connect Apache.
Here is my error log for Apache:
Digest: generating secret for digest authentication ...
Digest: done
FastCGI: process manager initialized (pid 1845)
Apache/2.2.34 (Unix) mod_wsgi/3.5 Python/2.7.13 PHP/7.2.14 mod_ssl/2.2.34 OpenSSL/1.0.2o DAV/2 mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_perl/2.0.9 Perl/v5.24.0 configured -- resuming normal operations
[error] [client 127.0.0.1] client denied by server configuration: /Applications/MAMP/htdocs/.DS_Store, referer: http://localhost/MAMP/?language=English
[error] [client 127.0.0.1] client denied by server configuration: /Applications/MAMP/htdocs/.DS_Store, referer: http://localhost/MAMP/?language=English
[notice] caught SIGTERM, shutting down
I don't know enough about Apache server configuration to figure out, when I start MAMP to be able to navigate with HTTPS to my Drupal project, which is in the HTDOCs file and make it run without the 400 error.
I solved the problem by commenting out the line "Document root" above in the virtual host section.

How to cure 503 Service Unavailable in Apache running in Docker Container?

I am trying to create an Apache virtual host proxy in a Docker container (I am on Docker 1.6), so I did the following:
First set up two Docker containers, each running their own web application:
docker run -it -p 8001:80 -p 4431:443 --name build ubuntu:latest
apt-get update
apt-get install apache2 libapache2-mod-php -y
echo “<?php phpinfo(); ?>” >> /var/www/html/info.php
service apache2 restart
then
docker run -it -p 8002:80 -p 4432:443 --name cicd ubuntu:latest
apt-get update
apt-get install apache2 libapache2-mod-php -y
echo “<?php phpinfo(); ?>” >> /var/www/html/info.php
service apache2 restart
Each of these runs perfectly on their respective ports. So next I create a container running Apache:
docker run -it -p 8000:80 -p 4430:443 --name apache_proxy ubuntu:latest
apt-get update
apt-get install apache2 -y
a2enmod proxy
a2enmod proxy_http
service apache2 restart
This works perfectly on it's own at port 8000.
Then I created a virtual host file for each of the other Docker containers:
<VirtualHost *:80>
ServerName build.example.com
<Proxy *>
Allow from localhost
</Proxy>
ProxyPass / http://localhost:8001/
</VirtualHost>
and
<VirtualHost *:80>
ServerName cicd.example.com
<Proxy *>
Allow from localhost
</Proxy>
ProxyPass / http://localhost:8002/
</VirtualHost>
Then placed both in /etc/apache2/sites-available/ of the apache_proxy container.
Now, I went back into the apache_proxy container and performed the following:
a2ensite build.example.conf
a2ensite cicd.example.conf
service apache2 restart
Running apachectl -S from the command line of the apache_proxy container I can see the following is in effect:
VirtualHost configuration:
*:80 is a NameVirtualHost
default server 172.17.0.17 (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost 172.17.0.17 (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost build.example.com (/etc/apache2/sites-enabled/build.example.conf:1)
port 80 namevhost cicd.example.com (/etc/apache2/sites-enabled/cicd.example.conf:1)
Here is what the setup looks like:
I can reach each individual container via it's respective port and I should be able to go to the following URL's to get to the respective sites:
build.example.com:8000 should proxy to the container/website on port 8001
cicd.example.com:8000 should proxy to the container/website on port 8002
Instead I get the following error:
503 Service Unavailable
Checking the logs I get the following:
[Mon Oct 16 21:17:32.510127 2017] [proxy:error] [pid 165:tid 140552167175936] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8001 (localhost) failed
[Mon Oct 16 21:17:32.510278 2017] [proxy:error] [pid 165:tid 140552167175936] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[Mon Oct 16 21:17:32.510302 2017] [proxy_http:error] [pid 165:tid 140552167175936] [client 172.26.16.120:61391] AH01114: HTTP: failed to make connection to backend: localhost
[Mon Oct 16 21:17:32.799053 2017] [proxy:error] [pid 166:tid 140552217532160] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8001 (localhost) failed
[Mon Oct 16 21:17:32.799232 2017] [proxy:error] [pid 166:tid 140552217532160] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[Mon Oct 16 21:17:32.799256 2017] [proxy_http:error] [pid 166:tid 140552217532160] [client 172.26.16.120:61392] AH01114: HTTP: failed to make connection to backend: localhost, referer: http://build.example.com:8000/info.php
I have been going down rabbit holes for the past several hours trying to get this to work and I am sure now I am missing something quite simple. Can anyone shed light on the error of my ways?
NOTE
I followed a huge rabbit hole concerning SELinux which is not enabled or even really installed in the Ubuntu/Apache proxy container.
I should also state that I am not a network guru or master web-server configurator. I only know enough to be dangerous.
EDIT
Based on suggestions I have tried the following:
ProxyPass / http://cicd:80/ causes a 502 error
ProxyPass / http://ip.address.of.server:8002/ times out
ProxyPass / http://ip.address.of.container:8002/ causes a 503 error
ProxyPass / http://127.0.0.1:8002/ retry=0 causes a 503 error (suggested in other answers)
Your two other containers are not localhost but respectively
build:80
cicd:80
Reflect that in your apache proxy and you should be good to go
What I really needed to remember is that we’re working within Docker’s network once we pass in the request, so using things like ProxyPass / http://localhost:8002/ will not work because ‘localhost’ belongs to the Docker container in which we’ve made the request.
So I started searching outside of the error box, so to speak, and came upon this answer From inside of a Docker container, how do I connect to the localhost of the machine?
What I determined is we need to pass the request to the Docker network. To get that information I ran sudo ip addr show docker0 from the server’s command line and it returns:
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
inet 172.17.42.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::5484:7aff:fefe:9799/64 scope link
valid_lft forever preferred_lft forever
Showing Docker’s internal network to be on 172.17.42.1 Changing the pass to be ProxyPass / http://172.17.42.1:8002/ hands off the request to the Docker network and subsequently succeeds.
<VirtualHost *:80>
ServerName cicd.example.com
<Proxy *>
#Allow from localhost
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://172.17.42.1:8002/ retry=0
</VirtualHost>

Apache Webserver ReverseProxy to serve Apache Solr Admin Panel

I'm trying to run an Apache Solr Service (on its emdedded jetty server) on a remote server. The admin has provided me following information:
DNS: my.server.com
IP: xxx.xxx.xxx
Server OS: 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux
Only Port 80 is accessible. On the server we want to deploy Apache Solr and a microservice which uses Solr as search engine. I want to use Apache Webserver to forward the HTTP-Request to the Solr Admin UI and to the microservice UI, but it doesn't seem to work, I use Apache Server version: Apache/2.4.10 (Debian)
Server built: Sep 15 2016 20:44:43.
I installed Apache and started the server, so far everything works as expected. I can access the admin view from Apache entering the DNS in my browser.
I enabled a few modules following this articel https://www.digitalocean.com/community/tutorials/how-to-use-apache-http-server-as-reverse-proxy-using-mod_proxy-extension:
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_ajp
a2enmod rewrite
a2enmod deflate
a2enmod headers
a2enmod proxy_balancer
a2enmod proxy_connect
a2enmod proxy_html
Then I tried to configure a virtual host under /etc/apache2/sites-available/myconf.conf:
<VirtualHost *:80>
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined
ProxyPass /solr http://my.server.com:8983 retry=0 timeout=5
ProxyPassReverse /solr http://my.server.com:8983
ProxyPass /microservice http://my.server.com:6868 retry=0 timeout=5
ProxyPassReverse /microservice http://my.server.com:6868
LogLevel debug
</VirtualHost>
Solr uses its standard port 8983 and the microservice will be on port 6868. When I try to acces solr with http://my.server.com/solr I get an HTTP 503 Service unavailable.
I first tried this:
/usr/sbin/setsebool -P httpd_can_network_connect 1
But it changed nothing. I also had to install first:
apt-get install policycoreutils
to make this option available. The solr service seems to be ok:
solr status
Found 1 Solr nodes:
Solr process 14082 running on port 8983
{
"solr_home":"/etc/apache-solr/solr-6.2.0/server/solr",
"version":"6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:41:37",
"startTime":"2016-10-07T12:02:05.300Z",
"uptime":"0 days, 1 hours, 29 minutes, 55 seconds",
"memory":"29.7 MB (%6.1) of 490.7 MB"}
The Apache log keeps saying:
The timeout specified has expired: AH00957: HTTP: attempt to connect to xxx.xxx.xxx:8983 (my.server.com) failed
AH00959: ap_proxy_connect_backend disabling worker for (my.server.com) for 0s
AH01114: HTTP: failed to make connection to backend: my.server.com
Without my timeout setting everthing keeps the same but it takes ages before I get the 503 Error.
Any hints? After one day struggeling I'm depressed ... all I want is to finish the task.
Thanks in advance!
It turns out that I needed to append a slash to the urls:
ProxyPass /solr/ http://my.server.com:8983/ retry=0 timeout=5
ProxyPassReverse /solr/ http://my.server.com:8983/
ProxyPass /microservice/ http://my.server.com:6868/ retry=0 timeout=5
ProxyPassReverse /microservice/ http://my.server.com:6868/