I am getting the error in the error_log. I was able to figure out that I need to increase ProxyTimeout.
However, I was unable to find where may I change it. All I could do was adding this to the server.xml:
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="600000"
redirectPort="8443" />
Any idea how to exactly increase the ProxyTimeout?
I am using Ubuntu from AWS and Apache Tomcat v7
I have tried to add this into proxy-html.conf:
ProxyRequests On
<Proxy *>
AddDefaultCharset off
Order deny,allow
Allow from all
</Proxy>
ProxyPreserveHost Off
ProxyTimeout 1200
I have also added this into httpd.conf:
TimeOut 600
I have restarted the Tomcat server but neither of the above configurations helped.
Btw this is the exact error, I don't understand why there is proxy mentioned. I am not connecting through any proxy server... The issue happens when I submit a form, which triggers some tough processing. Then it times out in approx. 60 seconds. However, the program still runs, only the request times out.
Error:
[Thu Aug 13 07:34:21.677693 2015] [proxy_ajp:error] [pid 1515] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
[Thu Aug 13 07:34:21.677769 2015] [proxy_ajp:error] [pid 1515] [client 212.130.108.58:52206] AH00992: ajp_read_header: ajp_ilink_receive failed, referer: http://52.17.109.177/Visma_UploadInterface/MappingServlet
[Thu Aug 13 07:34:21.677782 2015] [proxy_ajp:error] [pid 1515] (70007)The timeout specified has expired: [client 212.130.108.58:52206] AH00878: read response failed from 127.0.0.1:8009 (localhost), referer: http://52.17.109.177/Visma_UploadInterface/MappingServlet
I have solved it by adding this two simple lines into httpd.conf file:
Timeout 600
ProxyTimeout 600
I also rebooted the whole server, not just Tomcat. No idea if that was necessary, but seems like it.
I have resolved it modified existing virtual host configuration.
<Proxy "unix:/run/php/php7.2-fpm-example.sock|fcgi://localhost">
ProxySet timeout= 600
</Proxy>
<FilesMatch \.php$>
SetHandler "proxy:fcgi://localhost"
</FilesMatch>
Related
I have installed Apache 2.4.6 on Centos7 official.
httpd.conf was not changed from default.
Configured webdav.conf with the following:
DavLockDB /var/www/html/DavLock
<VirtualHost *:80>
ServerAdmin webmaster#localhost
DocumentRoot /var/www/html/webdav/
ErrorLog /var/log/httpd/error.log
CustomLog /var/log/httpd/access.log combined
Alias /webdav /var/www/html/webdav
<Directory /var/www/html/webdav>
DAV On
</Directory>
</VirtualHost>
webdav directory has 755 rights. DavLock not created.
I keep getting this error and not sure what the cause might be. Have tried many things without success:
[Fri Jan 29 11:41:17.027110 2021] [dav:error] [pid 15068] [client xxx.xxx.xxx.xxx:53946] The locks could not be queried for verification against a possible "If:" header. [500, #0]
[Fri Jan 29 11:41:17.027134 2021] [dav:error] [pid 15068] [client xxx.xxx.xxx.xxx:53946] Could not open the lock database. [500, #400]
[Fri Jan 29 11:41:17.027137 2021] [dav:error] [pid 15068] (13)Permission denied: [client xxx.xxx.xxx.xxx:53946] Could not open property database. [500, #1]
any ideas are welcome
Got it working now.
all files/folders under /var/www/ are chown to apache
moved DavLock to here: /var/lib/dav/
and added this line under httpd.conf:
<IfModule mod_dav_fs.c>
# Location of the WebDAV lock database.
DAVLockDB /var/lib/dav/DavLock
</IfModule>
At this point, I am getting 403 error. Solved by executing the following:
chcon -R -t httpd_sys_content_rw_t /var/www/html/webdav/
I have two apache servers setup on two separate physical machines. My current setup is:
Apache 1 (Reverse Proxy) <===> Apache 2
Both apache server versions are Apache/2.4.29 (Ubuntu) running on Ubuntu 18.04.4 LTS and their /etc/apache2/apache.conf files are identical.
Apache 1 sites-enabled config:
<VirtualHost *:80>
ServerName subdomain.domain.tld
ServerAlias www.subdomain.domain.tld
ServerAdmin webmaster#domain.tld
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyRequests off
ProxyPreserveHost On
ProxyPass /maintenance_page !
ProxyPass / http://[apache2-ip-address]:27300/
ProxyPassReverse / http://[apache2-ip-address]:27300/
</VirtualHost>
Apache 2 sites-enabled config:
<VirtualHost *:27300>
ServerName subdomain.domain.tld
ServerAlias www.subdomain.domain.tld
ServerAdmin webmaster#domain.tld
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
ErrorDocument 400 /notfound.html
ProxyRequests off
ProxyPreserveHost on
</VirtualHost>
If I directly hit http://[apache2-ip-address]:27300/ from the web browser the apache server landing page comes up fine. If I enter http://subdomain.domain.tld into the browser I get a proxy error:
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
I logged a trace on both apache servers. Apache server 2 is receiving the proxied request from apache server 1 and is returning a 200 status response perfectly fine to apache server 1. The flow breaks at apache server 1 where I am seeing the following logs:
[Sat Jul 11 20:34:08.671267 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(3075): AH00962: HTTP: connection complete to [apache2-ip-address]:27300 ([apache2-ip-address])
[Sat Jul 11 20:34:08.671333 2020] [core:trace6] [pid 32275:tid 140388069250816] core_filters.c(525): [remote [apache2-ip-address]:27300] core_output_filter: flushing because of FLUSH bucket
[Sat Jul 11 20:34:08.677508 2020] [proxy_http:error] [pid 32275:tid 140388069250816] (104)Connection reset by peer: [client xx.xxx.xxx.xx:39014] AH01102: error reading status line from remote server [apache2-ip-address]:27300
[Sat Jul 11 20:34:08.677575 2020] [proxy_http:debug] [pid 32275:tid 140388069250816] mod_proxy_http.c(1324): [client xx.xxx.xxx.xx:39014] AH01105: NOT Closing connection to client although reading from backend server [apache2-ip-address]:27300 failed.
[Sat Jul 11 20:34:08.677624 2020] [proxy:error] [pid 32275:tid 140388069250816] [client xx.xxx.xxx.xx:39014] AH00898: Error reading from remote server returned by /
[Sat Jul 11 20:34:08.677681 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(2192): AH00943: HTTP: has released connection for ([apache2-ip-address])
[Sat Jul 11 20:34:08.677724 2020] [http:trace3] [pid 32275:tid 140388069250816] http_filters.c(1128): [client xx.xxx.xxx.xx:39014] Response sent with status 502, headers:
Things I've tried, from few other discussions I could find online, are the following changes to apache server 1 sites-enabled config :
SetEnv proxy-initial-not-pooled 1
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
ProxyTimeout 600
ProxyPass / http://[apache2-ip-address]:27300/ timeout=600
ProxyPass / http://[apache2-ip-address]:27300/ nocanon
I've pretty much bruteforced the situation with several combinations of the above settings, but nothing seems to work. Any help is appreciated.
An additional check I ran is, if I run a nodejs application or python flask service on the same machine as either apache servers and proxy the service using ProxyPass / http://localhost:[port]/, the setup works properly. So both apache servers are running fine and are able to proxy services on their respective localhosts. Whatever is breaking has to do with the communication between the two apache servers.
UPDATE : Upon further triaging using curl with a networking person, the issue seems to be that the org firewall is only allowing inbound traffic to apache server 2 and blocking outbound traffic which may be causing 502 errors on apache server 1. This didn't seem like the issue up until I realized that my laptop was VPN'ed into the org network all along while testing and apache server 1 is sitting outside the org network. If this turns out to be the issue it's going to be a real bummer.
Adding following parameter in the http.conf file solves my issue of "proxy: error reading status line from remote server":
SetEnv proxy-initial-not-pooled 1
I go the reference from Apache URL https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html
Note: restart http server and try again.
In my case, an error with the database's connection was triggering this Apache's Reverse Proxyng error.
Application is not working and failing with below reason. Could someone please help on this.
connectivity apache -> tomcat
Error logs
[Mon Sep 10 03:00:07.123556 2018] [proxy_ajp:error] [pid 12345]
(104)Connection reset by peer: AH01030: ajp_ilink_receive() can't
receive header
ssl_error_log
[Mon Sep 10 03:00:07.123456 2018] [proxy_ajp:error] [pid 12345]
[client 10.10.34.901:56789] AH00992: ajp_read_header:
ajp_ilink_receive failed,
Tomcat configuration
<Connector port="8009"
protocol="AJP/1.3"
maxConnections="256"
keepAliveTimeout="30000"
redirectPort="8443">
Apache configuration
ProxyPass / ajp://localhost:8009/ timeout=600
ProxyPassReverse / ajp://localhost:8009/ timeout=600
ProxyPass needs an additional argument, e.g.
ProxyPass / ajp://localhost:8009/ ...
I have an apache, which works as reverse proxy for a Netscale Gateway. Sometimes it works, but in many cases the request ends with a 500. The log always give the same error type:
[Mon Aug 20 12:51:24.541905 2018] [http:error] [pid 4919:tid 140600024221440] [client 192.168.22.194:40187] AH02430: Response header 'Set-Cookie' value of 'NSC_TASS=\x0fi\xd4a\xbd\x8e\xcf\xdek\x18\xcd:\x01\xc6d\xf1\xe6;HttpOnly;Path=/;Secure' contains invalid characters, aborting request
[Mon Aug 20 13:03:09.550947 2018] [http:error] [pid 5023:tid 140354590320384] [client 192.168.22.194:24541] AH02430: Response header 'Set-Cookie' value of 'NSC_TASS=le8M1TpPxu5GG1h8nEom8vsA\xe3\x06\x87\x8fnId=&janusWebEvent=PDClass.getJanusServerPage_webEvent_nextPhaseGC2%2C114078_pid_pdPreview_imgPath_res133136%2Fimg%2F_cid_10_clName_ADV_oidHi_10_oidLow_114073;HttpOnly;Path=/;Secure' contains invalid characters, aborting request
[Mon Aug 20 13:09:15.239058 2018] [http:error] [pid 5330:tid 140134346917632] [client 192.168.22.194:40606] AH02430: Response header 'Set-Cookie' value of 'NSC_TASS=\x95\x15\xa91e\xf8\xc8\x96\xdfI\x02\x89\xf4y\x05\xf2&;HttpOnly;Path=/;Secure' contains invalid characters, aborting request
I think it is the backslash in the cookie value, because backslashes are not allowed.
Can apache skip the validating of this or can I write a rule to rewrite the cookie value to a url-encoding\utf-8... value?
My config:
<VirtualHost *:80> # a balancer managed the ssl
ServerName ng.subdomain.domain.tld
SSLProxyEngine On
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
ProxyRequests Off
#ProxyPreserveHost On
ProxyPass / https://ng.domain2.tld/
ProxyPassReverse / https://ng.domain2.tld/
#ProxyPass / wss://ng.domain2.tld/
#ProxyPassReverse / wss://ng.domain2.tld/
ProxyPassReverseCookieDomain ng.subdomain.domain.tld ng.domain2.tld
</VirtualHost>
Perhaps you can set the HttpProtocolOptions to LenientMethods, it might be set at Strict by default. This will only work if your proxy targter returns 501 for invalid restuls.
For more information see the Apache documentation on HttpProtocolOptions.
An easy fix would be to simply have the netscaler encrypt its session cookies.
https://support.citrix.com/article/CTX220162
To Encrypt session cookies as suggested above:
set lb parameter -useSecuredPersistenceCookie Enabled-cookiePassphrase
Example :
set lb parameter -useSecuredPersistenceCookie Enabled-cookiePassphrase test
Here are just a few lines from my Apache 2.0 error_log:
[Sun Nov 25 08:22:04 2012] [error] [client 64.34.195.190] File does not exist: /var/www/vhosts/default/htdocs/admin
[Sun Nov 25 14:14:32 2012] [error] [client 96.254.171.2] File does not exist: /var/www/vhosts/default/htdocs/azenv.php
[Wed Nov 28 03:02:01 2012] [error] [client 91.205.189.15] File does not exist: /var/www/vhosts/default/htdocs/user
[Wed Nov 28 03:44:35 2012] [error] [client 66.193.171.223] File does not exist: /var/www/vhosts/default/htdocs/vtigercrm
[Mon Dec 03 00:09:16 2012] [error] [client 82.223.239.68] File does not exist: /var/www/vhosts/default/htdocs/jmx-console
[Mon Dec 03 20:48:44 2012] [error] [client 221.2.209.46] File does not exist: /var/www/vhosts/default/htdocs/manager
[Thu Dec 06 07:37:04 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/w00tw00t.at.blackhats.romanian.anti-sec:)
[Thu Dec 06 07:37:05 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/phpMyAdmin
[Thu Dec 06 07:37:05 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/phpmyadmin
[Thu Dec 06 07:37:06 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/pma
[Thu Dec 06 07:37:06 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/myadmin
[Thu Dec 06 07:37:07 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/MyAdmin
[Thu Dec 13 02:19:53 2012] [error] [client 96.254.171.2] File does not exist: /var/www/vhosts/default/htdocs/judge.php
The most common errors are requests for the "phpMyAdmin" file, and "w00tw00t.at.blackhats.romanian.anti-sec:)".
I can see the IP address that the requests are coming from. But who is "client"?
Thanks,
Shane.
This is just an automatic script deployed by many Script Kiddies looking for a security breach in your apache version/configuration. The signature w00tw00t is usually left by DFind.
Just use a program like fail2ban configured such as this example explains to avoid being flooded by these requests :
https://web.archive.org/web/20160617020600/http://www.userdel.com/post/18618537324/block-w00tw00t-scans-with-fail2ban
This does not necessarily mean you've been hacked, but the server has been scanned for vulnerabilities. However, if you use any of the software that you saw in those logs and it is an older version having known vulnerabilities, you should check your server for unusual files and login activities.
Requests for this are usually sent without a server header. Just create a default virtual host for requests that don't have a server header you expect and blackhole it. Also fun to log broken traffic and do reverse DNS to see if it's coming from another webserver (compromised?) and contact the owner based on whois database. You never know who's running silly scripts from a publicly identifiable server to scan for vulnerabilities and later exploit them over ToR tunnel. Use burner contact information if you don't want to bring attention to yourself.
To follow up on the answer given by #user823629, here is a default virtual host configuration I use on Apache 2.4:
<VirtualHost *:80>
# Default vhost for requests not matching IP or Host of other vhosts
ServerName blackhole
ErrorLog logs/error_log_default
CustomLog logs/access_log_default combined
Redirect 404 /
</VirtualHost>
<VirtualHost *:443>
# Default vhost for requests not matching IP or Host of other vhosts
ServerName blackhole
ErrorLog logs/ssl_error_log_default
CustomLog logs/ssl_access_log_default combined
CustomLog logs/ssl_request_log_default "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
Redirect 404 /
</VirtualHost>
It redirects all requests to the default 404 page. SSL requests that do not match any other site will end up at the second VirtualHost definition, and will of course result in a certificate error, but this is expected and fine.
I put this in conf.d and give it a name of conf.d/0_default.conf so that it comes before other vhosts definitions and it is the default virtual host. This can be verified via:
apachectl -t -D DUMP_VHOSTS
or on Redhat/Fedora/CentOS distros:
httpd -t -D DUMP_VHOSTS
Other virtual hosts will match before this default vhost if:
Their IP address and port matches the VirtualHost definition more explicitly (IP-based virtual host), or
The request contains a Host header that matches the request (name-based virtual host). Otherwise, the request will fall back to the default blackhole virtual host defined above.
Be careful with VirtualHost definitions with IP addresses specified. Since these match before the blackhole, the wrong configuration can become the default for that IP. List the specific IPs in the blackhole if necessary.
See http://httpd.apache.org/docs/current/vhosts/details.html for more details on virtual host matching.
Unless you actually use /var/www/vhosts/default/ for hosting a website, this means you have requests going to the default host that are not being caught by your virtualhosts setup.
Disregard for a moment that these are malicious requests, because the underlying reason for these vhosts/default/ errors is you probably have SSL disabled for a virtualhost, & these are HTTPS requests caught up in the default server config.
You can add %v %V %p to your Apache access logging parameters in httpd.conf to see more info on what these requests are & what virtualhost/servername is handling them (%v %V) & on what port (%p) the requests are being made through (typically port 443 if it's HTTPS).
To fix the HTTPS aspect, I'd enable SSL & then put in a RewriteRule to send HTTPS requests to HTTP (if that's the intended behavior). More info about how to do that here.
Otherwise to help with script kiddies, the blackhole mentioned above is the way to go. Just make sure you aren't mistakenly sending legitimate web crawler/spiders requesting HTTPS to the same galactic death -- for instance Googlebot tests legitimate pages via HTTPS since that's the direction Google wants the web to head in.