Why am I getting an Apache Proxy 503 error? - apache

Question

My server was doing just fine up until yesterday. It was running Redmine, and it was the happiest little server until my "friend" imported a SQL table that my little guy couldn't take. Unfortunately, after an hour of trying to get the lil guy to respond, we had to power cycle him.
Now after restart, we get a 503 error when trying to visit the domain connected to Redmine. It's hooked up to a Mongrel daemon, and we use Apache Proxy to direct all connections to the port Redmine is running on.
Using Lynx on the server (http://localhost:8000) you can see the Ruby application working fine. But this bit is not working in my Apache configuration file:
<VirtualHost *:80>
ServerName sub.example.com
ProxyPass / http://localhost:8000
ProxyPassReverse / http://localhost:8000
ProxyPreserveHost on
LogLevel debug
</VirtualHost>
Here's the error log output for Apache:
[debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //localhost:8000
[debug] proxy_util.c(1335): [client 216.27.137.51] proxy: http: found worker http://localhost:8000 for http://localhost:8000/
[debug] mod_proxy.c(756): Running scheme http handler (attempt 0)
[debug] mod_proxy_http.c(1687): proxy: HTTP: serving URL http://localhost:8000/
[debug] proxy_util.c(1755): proxy: HTTP: has acquired connection for (localhost)
[debug] proxy_util.c(1815): proxy: connecting http://localhost:8000/ to localhost:8000
[debug] proxy_util.c(1908): proxy: connected / to localhost:8000
[debug] proxy_util.c(2002): proxy: HTTP: fam 2 socket created to connect to localhost
[error] (13)Permission denied: proxy: HTTP: attempt to connect to 127.0.0.1:8000 (localhost) failed
[error] ap_proxy_connect_backend disabling worker for (localhost)
[debug] proxy_util.c(1773): proxy: HTTP: has released connection for (localhost)

Answer 1

Apache will respond with 503's for at least 60 seconds any time it detects that the backend server is down. This is the default behavior. As in your example, if you restart your backend server (Rails in this example) and someone tries to access it through the Apache proxy before Rails is ready then Apache will return 503's for the next 60 seconds regardless if your backend is now 'up'. Please see the apache docs on ProxyPass where it states:
retry 60
Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance, and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.
So if you set your Proxy Pass to include retry=0 you won't see the 503's when you restart your backend service. This is also useful when using Apache as reverse proxy during development! For example:
ProxyPass / http://localhost:8000 retry=0

Answer 2

Run following command
# /usr/sbin/setsebool httpd_can_network_connect 1
OR
# /usr/sbin/setsebool httpd_can_network_connect true
and after that restart httpd
# service httpd restart

Answer 3

Are you sure they're restarting in the correct order? I've had weird issues where Apache starts, then Mongrel starts and although Mongrel is running, Apache still throws the proxy error.
I've solved this in the past with various incantations and restarts of Apache and eventually the gods are happy. It seems that sometimes the Mongrel processes don't properly shut down so you have to manually kill them. Here's a link with some [possible] help.
I ended up adding a "kill" option to my /etc/init.d/ mongrel script because it happened so much. It stop Mongrel, killed all Mongrel sessions, started Mongrel and restarted Apache.
<snip>
kill)
echo "Stopping, killing, starting, and restarting Apache..."
mongrel_cluster_ctl stop -c $CONF_DIR --clean
killall -u mongrel
mongrel_cluster_ctl start -c $CONF_DIR --clean
/etc/init.d/httpd restart
RETVAL=$?
;;
</snip>
Probably not a very good solution but the evil went away.

Answer 4

Try running monit to monitor your mongrels behind Apache, and that way it can restart mongrels for you if they die or get too hungry for memory. If for any reason Apache still gets confused you may just have to gracefully restart apache and it should resolve itself, but for 99% of cases having monit watch over your mongrels should avoid this happening again. The other option is look into Phusion Passenger.

Answer 5

First check whether the port 8080 is listening or not by the following command
netstat -tlpn
If not than restart the jenkins server by the following command
sudo /etc/init.d/jenkins start
It should work now. Hope it helps.

Answer 6

Fist, you must install selinux: (SELinux stands for Security-Enhanced Linux.)
apt-get install selinux
After that, you can enable Security Policy of SElinux by follow command:
sed -i 's/SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
Notice:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
Final,restart apache!