Flask app with Apache proxy + Gunicorn not working on HTTPS - apache

Updates in the bottom, I kind of solved it but not sure if the solution is a correct one.
I have Apache running on CentOS with a proxy to localhost port 8080 where I have Flask app running using Gunicorn. This setup works on Apache port 80 (HTTP) and I can connect to it using my domain http://example.com with a browser but now I have tried to setup SSL/HTTPS and it just doesn't work.
Navigating to https://example.com tries to load the page for a while (like 30ish seconds) and then it shows 502 error page:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.
Reason: Error reading from remote server
Apache error log:
[proxy_http:error] [pid 30209] (103)Software caused connection abort: [client xx.xxx.xxx.xxx:60556] AH01102: error reading status line from remote server localhost:8080
[proxy:error] [pid 30209] [client xx.xxx.xxx.xxx:60556] AH00898: Error reading from remote server returned by /
Gunicorn error log (first 7 lines are from Gunicorn startup, no idea why this info is in error log, last 3 lines are when the HTTPS request returns 502 error):
[29478] [INFO] Listening at: http://127.0.0.1:8080 (29478)
[29478] [INFO] Using worker: sync
[29480] [INFO] Booting worker with pid: 29480
[29481] [INFO] Booting worker with pid: 29481
[29482] [INFO] Booting worker with pid: 29482
[29483] [INFO] Booting worker with pid: 29483
[29484] [INFO] Booting worker with pid: 29484
[29478] [CRITICAL] WORKER TIMEOUT (pid:29480)
[29480] [INFO] Worker exiting (pid: 29480)
[29554] [INFO] Booting worker with pid: 29554
Apache config which is working for HTTP (/etc/httpd/conf/httpd.conf):
Listen 80
#other default config values here
<VirtualHost *:80>
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
</VirtualHost>
IncludeOptional conf.d/*.conf
Apache config which is not working for HTTPS (/etc/httpd/conf.d/ssl.conf):
Listen 443 https
#other default config values here
<VirtualHost *:443>
#other default config values here too
SSLCertificateFile /etc/pki/tls/certs/cert.pem
SSLCertificateKeyFile /etc/pki/tls/private/cert.key
SSLProxyEngine on
ProxyPass / https://localhost:8080/
ProxyPassReverse / https://localhost:8080/
</VirtualHost>
If I remove/comment the SSLProxyEngine, ProxyPass amd ProxyPassReverse lines and restart Apache I get the default Apache welcome page and HTTPS works just fine so clearly the problem is in the Proxy somehow?
The flask app is started with Gunicorn by:
gunicorn --config gunicorn_config.py app:app
gunicorn_config.py:
workers = 5
bind = '127.0.0.1:8080'
umask = 0o007
reload = True
accesslog = 'log_gunicorn_access.txt'
errorlog = 'log_gunicorn_error.txt'
app.py:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello world!'
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080)
And once again, this works when navigating to my domain using HTTP but doesn't work when using HTTPS.
Any help?
UPDATE:
I managed to get another error. Now navigating to https://example.com loads instantly and shows 500 error page:
Proxy Error
The proxy server could not handle the request GET /.
Reason: Error during SSL Handshake with remote server
Apache error log:
[proxy:error] [pid 32385] (502)Unknown error 502: [client xx.xxx.xxx.xxx:50932] AH01084: pass request body failed to [::1]:8080 (localhost)
[proxy:error] [pid 32385] [client xx.xxx.xxx.xxx:50932] AH00898: Error during SSL Handshake with remote server returned by /
[proxy_http:error] [pid 32385] [client xx.xxx.xxx.xxx:50932] AH01097: pass request body failed to [::1]:8080 (localhost) from xx.xxx.xxx.xxx ()
No more any errors in Gunicorn error log.
I added these two lines to gunicorn_config.py:
keyfile = '/etc/pki/tls/private/cert.key'
certfile = '/etc/pki/tls/certs/cert.pem'
and made sure both files are accessible by the user running Gunicorn (chmod o+r cert.key/pem).
No idea if I was supposed to change it like this as I thought the traffic should go like: client --https--> Apache and then Apache --http--> Gunicorn.
Also HTTP (http://example.com) no longer works and gives the previous 502 error page but I guess running Gunicorn with the cert configs doesn't allow HTTP anymore and would need to run the app twice with different configs).
UPDATE 2:
I added more Apache logging by adding this line to /etc/httpd/conf.d/ssl.conf inside virtual host:
LogLevel info
And now I got additional info in Apache error log:
[ssl:info] [pid 3808] [remote 127.0.0.1:8080] AH02411: SSL Proxy: Peer certificate does not match for hostname localhost
Then I added new line to /etc/httpd/conf.d/ssl.conf inside virtual host:
SSLProxyCheckPeerName off
And now I got another Apache error:
[ssl:info] [pid 3999] [remote 127.0.0.1:8080] AH02005: SSL Proxy: Peer certificate CN mismatch: Certificate CN: example.com Requested hostname: localhost
Added new line to /etc/httpd/conf.d/ssl.conf inside virtual host:
SSLProxyCheckPeerCN off
Aaaand now navigating to https://example.com correctly works and I get "Hello world" back from the app!
Now I guess my question needs update as well: Is it bad practice, wrong or insecure to use SSLProxyCheckPeerName off and SSLProxyCheckPeerCN off in this context? Or is there a better way as I don't think there's a way to order an official SSL certificate on localhost?

You're using
ProxyPass / http://localhost:8080/
and
ProxyPass / https://localhost:8080/
(note the 1 letter difference).
Your localhost:8080 will serve either http or https. Based on your description (and common expectations) it's serving http. If you proxy even your :443 virtual host to http, it'll work better.
You might run into more issues, as the proxied application doesn't really know that it's actually served through https, but that's a different beast than this question.

Related

Trouble setting up SSL to work with MAMP 5.3

After adding a self-signed SSL certificate, I am unable to get my Drupal site to work on localhost.
I have attempted various proposed solutions that I have found online but none have gotten me past a 400 error at https://localhost/
I have uncommented the following in httpd.conf:
LoadModule ssl_module modules/mod_ssl.so
I have Listen set to port 80 in this same file. When I set it to the SSL channel, 443, Apache does not load.
In my httpd-ssl.conf file:
I have set the paths for my server.crt and server.key files correctly.
I have enabled SSLEngine
<VirtualHost *:443>
# General setup for the virtual host
DocumentRoot "/Applications/MAMP/Library/htdocs"
ServerName https://127.0.0.1:443
ServerAdmin you#example.com
ErrorLog "/Applications/MAMP/Library/logs/error_log"
TransferLog "/Applications/MAMP/Library/logs/access_log"
I have tried a variety of possibilities for the ServerName. The one above along with localhost:443 both lead to 400 errors.
I have Listen set to 443 here. Setting to 80 so it matches the httpd.conf file leads to the same result described above...not able to connect Apache.
Here is my error log for Apache:
Digest: generating secret for digest authentication ...
Digest: done
FastCGI: process manager initialized (pid 1845)
Apache/2.2.34 (Unix) mod_wsgi/3.5 Python/2.7.13 PHP/7.2.14 mod_ssl/2.2.34 OpenSSL/1.0.2o DAV/2 mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_perl/2.0.9 Perl/v5.24.0 configured -- resuming normal operations
[error] [client 127.0.0.1] client denied by server configuration: /Applications/MAMP/htdocs/.DS_Store, referer: http://localhost/MAMP/?language=English
[error] [client 127.0.0.1] client denied by server configuration: /Applications/MAMP/htdocs/.DS_Store, referer: http://localhost/MAMP/?language=English
[notice] caught SIGTERM, shutting down
I don't know enough about Apache server configuration to figure out, when I start MAMP to be able to navigate with HTTPS to my Drupal project, which is in the HTDOCs file and make it run without the 400 error.
I solved the problem by commenting out the line "Document root" above in the virtual host section.

Setting up a secure proxy using Apache and Raspberry Pi

I am trying to set up my own secure proxy to be utilised by a Django Python script I have written that depends on a secure proxy to return an https request correctly. This is because the script is being run from an American host server and I want the response to be from the UK.
To achieve this and learn something new I am trying to configure an Apache server running on my Raspberry Pi. (I am not expecting a vast amount of traffic!)
I have successfully created an SSL certificate and have the correct files .pem, .csr (and .csr)
I believe I have copied these to the correct locations: /etc/ssl/private and /etc/ssl/certs respectively. I think the only issue I am now having is with the apache configuration.
At present I am trying to configure sites-enabled/default-ssl.conf files as follows:
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin webmaster#localhost
DocumentRoot /var/www/html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
# A self-signed (snakeoil) certificate can be created by installing
# the ssl-cert package. See
# /usr/share/doc/apache2/README.Debian.gz for more info.
# If both key and certificate are stored in the same file, only the
# SSLCertificateFile directive is needed.
#SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
#SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
SSLCertificateFile /etc/ssl/certs/aa5758835031e4a3.crt
SSLCertificateKeyFile /etc/ssl/private/<domainname>.key
I'm very new to Server set-ups and have spent a large amount of time web searching and trying many different suggested methods. I am sure it is something very simple I am neglecting, but not being experience enough; I can't intuitively work this out.
Help and pointers appreciated...
Update:
Hey, thanks for replying.
Response to http: is
* Rebuilt URL to: http://*.*.*/
* Trying 184.168.221.1...
* TCP_NODELAY set
* Connected to *.*.* port 80 (#0)
> GET / HTTP/1.1
> Host : *.*.*
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Connection: close
< Pragma: no-cache
< cache-control: no-cache
< Location: /VZaKp/
<
* Curl_http_done: called premature == 0
* Closing connection 0
and https is:
* Rebuilt URL to: https://*.*.*/
* Trying 50.63.202.18...
* TCP_NODELAY set
* connect to 50.63.202.18 port 443 failed: Connection timed out
* Failed to connect to *.*.* port 443: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to *.*.* port 443: Connection timed out
Update 23.04.2018:
Thank you. I believe I have sorted the listening issues, but am now coming across a new issue!
When trying to restart apache2 service I get the following error:
Apr 23 12:31:30 matthew-raspberry systemd[1]: Starting The Apache HTTP Server...
Apr 23 12:31:31 matthew-raspberry apachectl[8341]: Action 'start' failed.
Apr 23 12:31:31 matthew-raspberry apachectl[8341]: The Apache error log may have more information.
Apr 23 12:31:31 matthew-raspberry systemd[1]: apache2.service: Control process exited, code=exited status=1
Apr 23 12:31:31 matthew-raspberry systemd[1]: Failed to start The Apache HTTP Server.
Apr 23 12:31:31 matthew-raspberry systemd[1]: apache2.service: Unit entered failed state.
Apr 23 12:31:31 matthew-raspberry systemd[1]: apache2.service: Failed with result 'exit-code'.
I suspect this may be something to do with my fqdn setup and/or the certificate. I got this error since adding ServerName localhost as a global name.
I added this as the previous error message was about determining my fqdn name and this suggested adding ServerName as a global variable?
The port 443 is not reachable even at a basic TCP level.
Make sure to include somewhere in Apache config:
Listen 443 https
Reload httpd after config changes. Examine your firewall-cmd --list-all-zones or iptables -S to see if you allow connections to your port 443.

Google Cloud Platform Load Balancer with Apache mod_proxy

I am trying to configure httpd mod_proxy to serve as a reverse proxy in Google Cloud Platform. I set up a TCP load balancer and tried something like this in a VM apache config.
<Proxy "*">
</Proxy>
Listen 7001
Listen 7002
<VirtualHost *:7001>
ProxySourceAddress <LB IP 1 here>
</VirtualHost>
<VirtualHost *:7002>
ProxySourceAddress <LB IP 2 here>
</VirtualHost>
Apache logs show my attempts to GET through to yandex.ru via proxy:
AH01114: HTTP: failed to make connection to backend: www.yandex.ru
(70007)The timeout specified has expired: AH00957: HTTP: attempt to connect to 5.255.255.5:80 (*) failed
httpd version 2.4.6. OS Centos 7.
What am I missing here?

Httpd returning 503 Service Unavailable with mod_proxy for Tomcat 8

I'm trying to integrate Tomcat with Apache. My aim is to redirect all the requests with
http://localhost/myapp to http://localhost:8080
I followed this guide: http://tomcat.apache.org/tomcat-8.0-doc/proxy-howto.html
My httpd.conf looks like this:
Include conf.modules.d/*.conf
LoadModule proxy_module modules/mod_proxy.so
ProxyPass /myapp http://localhost:8080 retry=0 timeout=5
ProxyPassReverse /myapp http://localhost:8080
My server.xml in apache-tomcat looks like this:
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" proxyPort="80" />
Now when I try the url http://localhost/myapp, it gives 503 Service Unavailable error.
Both Tomcat and Apache are up and running. The URL http://localhost:8080 works fine.
Can there be an issue with file permissions?
For tomcat the user and group are root/root and for httpd, the user and group are apache/apache
Am I missing something or am I doing it wrong?
Httpd version is 2.4.6 and Tomcat's version is 8.0
The httpd error logs:
[proxy:error] [pid 19905] (13)Permission denied: AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[proxy:error] [pid 19905] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[proxy_http:error] [pid 19905] [client ::1:51615] AH01114: HTTP: failed to make connection to backend: localhost
Solved!
The answer is here: http://sysadminsjourney.com/content/2010/02/01/apache-modproxy-error-13permission-denied-error-rhel/
(Answered by the OP in a question edit. Converted to a community wiki answer. See Question with no answers, but issue solved in the comments (or extended in chat) )
The OP wrote:
The answer is here: http://sysadminsjourney.com/content/2010/02/01/apache-modproxy-error-13permission-denied-error-rhel/
Which is a link to a blog that explains:
SELinux on RHEL/CentOS by default ships so that httpd processes cannot initiate outbound connections, which is just what mod_proxy attempts to do.
If this is the problem, it can be solved by running:
/usr/sbin/setsebool -P httpd_can_network_connect 1
And for a more definitive source of information, see https://wiki.apache.org/httpd/13PermissionDenied
Resolve issue Immediate, It's related to internal security
We, SnippetBucket.com working for enterprise linux RedHat, found httpd server don't allow proxy to run, neither localhost or 127.0.0.1, nor any other external domain.
As investigate in server log found
[error] (13)Permission denied: proxy: AJP: attempt to connect to
10.x.x.x:8069 (virtualhost.virtualdomain.com) failed
Audit log found similar port issue
type=AVC msg=audit(1265039669.305:14): avc: denied { name_connect } for pid=4343 comm="httpd" dest=8069
scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket
Due to internal default security of linux, this cause, now to fix (temporary)
/usr/sbin/setsebool httpd_can_network_connect 1
Resolve Permanent Issue
/usr/sbin/setsebool -P httpd_can_network_connect 1
this worked for me by editing my *.conf file in apache:
ProxyRequests Off
ProxyPreserveHost On
RewriteEngine On
<Proxy http://localhost:8123>
Order deny,allow
Allow from all
</Proxy>
ProxyPass /node http://localhost:8123
ProxyPassReverse /node http://localhost:8123
On CentOS Linux release 7.5.1804, we were able to make this work by editing /etc/selinux/config and changing the setting of SELINUX like so:
SELINUX=disabled

Why am I getting an Apache Proxy 503 error?

My server was doing just fine up until yesterday. It was running Redmine, and it was the happiest little server until my "friend" imported a SQL table that my little guy couldn't take. Unfortunately, after an hour of trying to get the lil guy to respond, we had to power cycle him.
Now after restart, we get a 503 error when trying to visit the domain connected to Redmine. It's hooked up to a Mongrel daemon, and we use Apache Proxy to direct all connections to the port Redmine is running on.
Using Lynx on the server (http://localhost:8000) you can see the Ruby application working fine. But this bit is not working in my Apache configuration file:
<VirtualHost *:80>
ServerName sub.example.com
ProxyPass / http://localhost:8000
ProxyPassReverse / http://localhost:8000
ProxyPreserveHost on
LogLevel debug
</VirtualHost>
Here's the error log output for Apache:
[debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //localhost:8000
[debug] proxy_util.c(1335): [client 216.27.137.51] proxy: http: found worker http://localhost:8000 for http://localhost:8000/
[debug] mod_proxy.c(756): Running scheme http handler (attempt 0)
[debug] mod_proxy_http.c(1687): proxy: HTTP: serving URL http://localhost:8000/
[debug] proxy_util.c(1755): proxy: HTTP: has acquired connection for (localhost)
[debug] proxy_util.c(1815): proxy: connecting http://localhost:8000/ to localhost:8000
[debug] proxy_util.c(1908): proxy: connected / to localhost:8000
[debug] proxy_util.c(2002): proxy: HTTP: fam 2 socket created to connect to localhost
[error] (13)Permission denied: proxy: HTTP: attempt to connect to 127.0.0.1:8000 (localhost) failed
[error] ap_proxy_connect_backend disabling worker for (localhost)
[debug] proxy_util.c(1773): proxy: HTTP: has released connection for (localhost)
Apache will respond with 503's for at least 60 seconds any time it detects that the backend server is down. This is the default behavior. As in your example, if you restart your backend server (Rails in this example) and someone tries to access it through the Apache proxy before Rails is ready then Apache will return 503's for the next 60 seconds regardless if your backend is now 'up'. Please see the apache docs on ProxyPass where it states:
retry 60
Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance, and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.
So if you set your Proxy Pass to include retry=0 you won't see the 503's when you restart your backend service. This is also useful when using Apache as reverse proxy during development! For example:
ProxyPass / http://localhost:8000 retry=0
Run following command
# /usr/sbin/setsebool httpd_can_network_connect 1
OR
# /usr/sbin/setsebool httpd_can_network_connect true
and after that restart httpd
# service httpd restart
Are you sure they're restarting in the correct order? I've had weird issues where Apache starts, then Mongrel starts and although Mongrel is running, Apache still throws the proxy error.
I've solved this in the past with various incantations and restarts of Apache and eventually the gods are happy. It seems that sometimes the Mongrel processes don't properly shut down so you have to manually kill them. Here's a link with some [possible] help.
I ended up adding a "kill" option to my /etc/init.d/ mongrel script because it happened so much. It stop Mongrel, killed all Mongrel sessions, started Mongrel and restarted Apache.
<snip>
kill)
echo "Stopping, killing, starting, and restarting Apache..."
mongrel_cluster_ctl stop -c $CONF_DIR --clean
killall -u mongrel
mongrel_cluster_ctl start -c $CONF_DIR --clean
/etc/init.d/httpd restart
RETVAL=$?
;;
</snip>
Probably not a very good solution but the evil went away.
Try running monit to monitor your mongrels behind Apache, and that way it can restart mongrels for you if they die or get too hungry for memory. If for any reason Apache still gets confused you may just have to gracefully restart apache and it should resolve itself, but for 99% of cases having monit watch over your mongrels should avoid this happening again. The other option is look into Phusion Passenger.
First check whether the port 8080 is listening or not by the following command
netstat -tlpn
If not than restart the jenkins server by the following command
sudo /etc/init.d/jenkins start
It should work now. Hope it helps.
Fist, you must install selinux: (SELinux stands for Security-Enhanced Linux.)
apt-get install selinux
After that, you can enable Security Policy of SElinux by follow command:
sed -i 's/SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
Notice:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
Final,restart apache!