Am I being hacked? - apache

Here are just a few lines from my Apache 2.0 error_log:
[Sun Nov 25 08:22:04 2012] [error] [client 64.34.195.190] File does not exist: /var/www/vhosts/default/htdocs/admin
[Sun Nov 25 14:14:32 2012] [error] [client 96.254.171.2] File does not exist: /var/www/vhosts/default/htdocs/azenv.php
[Wed Nov 28 03:02:01 2012] [error] [client 91.205.189.15] File does not exist: /var/www/vhosts/default/htdocs/user
[Wed Nov 28 03:44:35 2012] [error] [client 66.193.171.223] File does not exist: /var/www/vhosts/default/htdocs/vtigercrm
[Mon Dec 03 00:09:16 2012] [error] [client 82.223.239.68] File does not exist: /var/www/vhosts/default/htdocs/jmx-console
[Mon Dec 03 20:48:44 2012] [error] [client 221.2.209.46] File does not exist: /var/www/vhosts/default/htdocs/manager
[Thu Dec 06 07:37:04 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/w00tw00t.at.blackhats.romanian.anti-sec:)
[Thu Dec 06 07:37:05 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/phpMyAdmin
[Thu Dec 06 07:37:05 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/phpmyadmin
[Thu Dec 06 07:37:06 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/pma
[Thu Dec 06 07:37:06 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/myadmin
[Thu Dec 06 07:37:07 2012] [error] [client 116.254.203.24] File does not exist: /var/www/vhosts/default/htdocs/MyAdmin
[Thu Dec 13 02:19:53 2012] [error] [client 96.254.171.2] File does not exist: /var/www/vhosts/default/htdocs/judge.php
The most common errors are requests for the "phpMyAdmin" file, and "w00tw00t.at.blackhats.romanian.anti-sec:)".
I can see the IP address that the requests are coming from. But who is "client"?
Thanks,
Shane.

This is just an automatic script deployed by many Script Kiddies looking for a security breach in your apache version/configuration. The signature w00tw00t is usually left by DFind.
Just use a program like fail2ban configured such as this example explains to avoid being flooded by these requests :
https://web.archive.org/web/20160617020600/http://www.userdel.com/post/18618537324/block-w00tw00t-scans-with-fail2ban
This does not necessarily mean you've been hacked, but the server has been scanned for vulnerabilities. However, if you use any of the software that you saw in those logs and it is an older version having known vulnerabilities, you should check your server for unusual files and login activities.

Requests for this are usually sent without a server header. Just create a default virtual host for requests that don't have a server header you expect and blackhole it. Also fun to log broken traffic and do reverse DNS to see if it's coming from another webserver (compromised?) and contact the owner based on whois database. You never know who's running silly scripts from a publicly identifiable server to scan for vulnerabilities and later exploit them over ToR tunnel. Use burner contact information if you don't want to bring attention to yourself.

To follow up on the answer given by #user823629, here is a default virtual host configuration I use on Apache 2.4:
<VirtualHost *:80>
# Default vhost for requests not matching IP or Host of other vhosts
ServerName blackhole
ErrorLog logs/error_log_default
CustomLog logs/access_log_default combined
Redirect 404 /
</VirtualHost>
<VirtualHost *:443>
# Default vhost for requests not matching IP or Host of other vhosts
ServerName blackhole
ErrorLog logs/ssl_error_log_default
CustomLog logs/ssl_access_log_default combined
CustomLog logs/ssl_request_log_default "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
Redirect 404 /
</VirtualHost>
It redirects all requests to the default 404 page. SSL requests that do not match any other site will end up at the second VirtualHost definition, and will of course result in a certificate error, but this is expected and fine.
I put this in conf.d and give it a name of conf.d/0_default.conf so that it comes before other vhosts definitions and it is the default virtual host. This can be verified via:
apachectl -t -D DUMP_VHOSTS
or on Redhat/Fedora/CentOS distros:
httpd -t -D DUMP_VHOSTS
Other virtual hosts will match before this default vhost if:
Their IP address and port matches the VirtualHost definition more explicitly (IP-based virtual host), or
The request contains a Host header that matches the request (name-based virtual host). Otherwise, the request will fall back to the default blackhole virtual host defined above.
Be careful with VirtualHost definitions with IP addresses specified. Since these match before the blackhole, the wrong configuration can become the default for that IP. List the specific IPs in the blackhole if necessary.
See http://httpd.apache.org/docs/current/vhosts/details.html for more details on virtual host matching.

Unless you actually use /var/www/vhosts/default/ for hosting a website, this means you have requests going to the default host that are not being caught by your virtualhosts setup.
Disregard for a moment that these are malicious requests, because the underlying reason for these vhosts/default/ errors is you probably have SSL disabled for a virtualhost, & these are HTTPS requests caught up in the default server config.
You can add %v %V %p to your Apache access logging parameters in httpd.conf to see more info on what these requests are & what virtualhost/servername is handling them (%v %V) & on what port (%p) the requests are being made through (typically port 443 if it's HTTPS).
To fix the HTTPS aspect, I'd enable SSL & then put in a RewriteRule to send HTTPS requests to HTTP (if that's the intended behavior). More info about how to do that here.
Otherwise to help with script kiddies, the blackhole mentioned above is the way to go. Just make sure you aren't mistakenly sending legitimate web crawler/spiders requesting HTTPS to the same galactic death -- for instance Googlebot tests legitimate pages via HTTPS since that's the direction Google wants the web to head in.

Related

Apache server reverse-proxying another apache server, getting "AH01102: error reading status line from remote server"

I have two apache servers setup on two separate physical machines. My current setup is:
Apache 1 (Reverse Proxy) <===> Apache 2
Both apache server versions are Apache/2.4.29 (Ubuntu) running on Ubuntu 18.04.4 LTS and their /etc/apache2/apache.conf files are identical.
Apache 1 sites-enabled config:
<VirtualHost *:80>
ServerName subdomain.domain.tld
ServerAlias www.subdomain.domain.tld
ServerAdmin webmaster#domain.tld
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyRequests off
ProxyPreserveHost On
ProxyPass /maintenance_page !
ProxyPass / http://[apache2-ip-address]:27300/
ProxyPassReverse / http://[apache2-ip-address]:27300/
</VirtualHost>
Apache 2 sites-enabled config:
<VirtualHost *:27300>
ServerName subdomain.domain.tld
ServerAlias www.subdomain.domain.tld
ServerAdmin webmaster#domain.tld
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
ErrorDocument 400 /notfound.html
ProxyRequests off
ProxyPreserveHost on
</VirtualHost>
If I directly hit http://[apache2-ip-address]:27300/ from the web browser the apache server landing page comes up fine. If I enter http://subdomain.domain.tld into the browser I get a proxy error:
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
I logged a trace on both apache servers. Apache server 2 is receiving the proxied request from apache server 1 and is returning a 200 status response perfectly fine to apache server 1. The flow breaks at apache server 1 where I am seeing the following logs:
[Sat Jul 11 20:34:08.671267 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(3075): AH00962: HTTP: connection complete to [apache2-ip-address]:27300 ([apache2-ip-address])
[Sat Jul 11 20:34:08.671333 2020] [core:trace6] [pid 32275:tid 140388069250816] core_filters.c(525): [remote [apache2-ip-address]:27300] core_output_filter: flushing because of FLUSH bucket
[Sat Jul 11 20:34:08.677508 2020] [proxy_http:error] [pid 32275:tid 140388069250816] (104)Connection reset by peer: [client xx.xxx.xxx.xx:39014] AH01102: error reading status line from remote server [apache2-ip-address]:27300
[Sat Jul 11 20:34:08.677575 2020] [proxy_http:debug] [pid 32275:tid 140388069250816] mod_proxy_http.c(1324): [client xx.xxx.xxx.xx:39014] AH01105: NOT Closing connection to client although reading from backend server [apache2-ip-address]:27300 failed.
[Sat Jul 11 20:34:08.677624 2020] [proxy:error] [pid 32275:tid 140388069250816] [client xx.xxx.xxx.xx:39014] AH00898: Error reading from remote server returned by /
[Sat Jul 11 20:34:08.677681 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(2192): AH00943: HTTP: has released connection for ([apache2-ip-address])
[Sat Jul 11 20:34:08.677724 2020] [http:trace3] [pid 32275:tid 140388069250816] http_filters.c(1128): [client xx.xxx.xxx.xx:39014] Response sent with status 502, headers:
Things I've tried, from few other discussions I could find online, are the following changes to apache server 1 sites-enabled config :
SetEnv proxy-initial-not-pooled 1
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
ProxyTimeout 600
ProxyPass / http://[apache2-ip-address]:27300/ timeout=600
ProxyPass / http://[apache2-ip-address]:27300/ nocanon
I've pretty much bruteforced the situation with several combinations of the above settings, but nothing seems to work. Any help is appreciated.
An additional check I ran is, if I run a nodejs application or python flask service on the same machine as either apache servers and proxy the service using ProxyPass / http://localhost:[port]/, the setup works properly. So both apache servers are running fine and are able to proxy services on their respective localhosts. Whatever is breaking has to do with the communication between the two apache servers.
UPDATE : Upon further triaging using curl with a networking person, the issue seems to be that the org firewall is only allowing inbound traffic to apache server 2 and blocking outbound traffic which may be causing 502 errors on apache server 1. This didn't seem like the issue up until I realized that my laptop was VPN'ed into the org network all along while testing and apache server 1 is sitting outside the org network. If this turns out to be the issue it's going to be a real bummer.
Adding following parameter in the http.conf file solves my issue of "proxy: error reading status line from remote server":
SetEnv proxy-initial-not-pooled 1
I go the reference from Apache URL https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html
Note: restart http server and try again.
In my case, an error with the database's connection was triggering this Apache's Reverse Proxyng error.

Debugging 400 when using VirtualHost with custom hostname (apache)

This setup was working on an ubuntu 14.04 until a recent update (probably the update/ change in package itself is old).
An internal DNS resolves "piwik.7l" to 192.168.1.17 .
192.168.1.17 serves an apache with piwik on it and following conf
<VirtualHost *:80>
ServerName "piwik.7l"
ServerAlias www.piwik.7l
</VirtualHost>
The setup now does work if addressed by IP, but responds with a 400 Bad Request if browsed to via its domain (piwik.7l). The 400 page does contain
Apache/2.4.7 (Ubuntu) Server at piwik.7l Port 80 and after setting the LogLevel to debug, I see following messages in the error.log:
[Mon May 22 15:12:33.566249 2017] [core:debug] [pid 1559] vhost.c(794): [client 192.168.1.112:38587] AH02415: [strict] Invalid host name 'piwik.7l', problem near: .7l
[Mon May 22 15:12:33.566316 2017] [core:debug] [pid 1559] vhost.c(889): [client 192.168.1.112:38587] AH00550: Client sent malformed Host header: piwik.7l
[Mon May 22 15:12:33.566326 2017] [core:debug] [pid 1559] protocol.c(1356): [client 192.168.1.112:38587] AH00569: client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
In scroll-friendly
[strict] Invalid host name 'piwik.7l', problem near: .7l
AH00550: Client sent malformed Host header: piwik.7l
AH00569: client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
I tried various combinations of IP, port and domain name usage in the VirtualHost/ServerName/ServerAlias blocks and disabled all other VirtualHost-definitions.
To me it sounds as if Apache would try to validate the "tld" part of the domain (7l) in a strict way. Is there any way I can disable that or debug it further?
Edit
The relevant parts of apache2ctl -S:
VirtualHost configuration:
*:80 piwik.7l (/etc/apache2/sites-enabled/000-default.conf:1)
Edit 2
ping piwik.7l does resolve fine from the webserver and from my machine.
Thanks to #savedarios answer here: https://serverfault.com/questions/658537/apache-virtualhost-error-invalid-host-name/841984#841984
If found the solution to be the following change to my apache2.conf:
HttpProtocolOptions Unsafe

how to configure apache server to talk to HTTPS backend server?

I configured apache server as a reverse proxy and it works fine if I point a backend server as HTTP. That is:
I configured virtual host 443 like:
ProxyPass /primary/store http://localhost:9763/store/
ProxyPassReverse /primary/store http://localhost:9763/store/
Here users will access the server like https://localhost/primary/store
And this works fine... But I want to configure HTTP server like;
ProxyPass /primary/store https://localhost:9443/store/
ProxyPassReverse /primary/store https://localhost:9443/store/
When I configure like apache server gives 500 internal server error. What am I doing wrong here?
The error I get is:
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, you#example.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
In the apache error log it states;
nt: SSLProxyEngine]
[Mon Aug 04 00:03:26 2014] [error] proxy: HTTPS: failed to enable ssl support for [::1]:9443 (localhost)
[Mon Aug 04 00:03:31 2014] [error] [client ::1] SSL Proxy requested for localhost:443 but not enabled [Hint: SSLProxyEngine]
[Mon Aug 04 00:03:31 2014] [error] proxy: HTTPS: failed to enable ssl support for [::1]:9443 (localhost)
[Mon Aug 04 00:03:51 2014] [error] [client ::1] SSL Proxy requested for localhost:443 but not enabled [Hint: SSLProxyEngine]
[Mon Aug 04 00:03:51 2014] [error] proxy: HTTPS: failed to enable ssl support for [::1]:9443 (localhost)
How to configure http server to talk to HTTPS server?
Your server tells you exactly what you need : [Hint: SSLProxyEngine]
You need to add that directive to your VirtualHost before the Proxy directives :
SSLProxyEngine on
ProxyPass /primary/store https://localhost:9763/store/
ProxyPassReverse /primary/store https://localhost:9763/store/
See the doc for more detail.
In my case, my server was configured to work only in https mode, and error occured when I try to access http mode. So changing http://my-service to https://my-service helped.

Laravel route not working in Apache using custom port in virtual host

I have a server running Linux. Its a VM with ip 10.61.0.3. Currently used to serve my web located at /var/www/html at port 80.
I want to serve Laravel app from this server located at /var/www/laraapp to port 8080.
My current apache conf for virtual host is:
<VirtualHost *:8080>
ServerName 10.61.0.3:8080
DocumentRoot /var/www/laraapp/public
</VirtualHost>
When I access 10.61.0.3:8080 nothing is shown up. Here is what show on apache error_logs:
[Mon Mar 17 17:28:43 2014] [notice] caught SIGTERM, shutting down
[Mon Mar 17 17:28:48 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Mon Mar 17 17:28:48 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Mar 17 17:28:48 2014] [notice] Digest: generating secret for digest authentication ...
[Mon Mar 17 17:28:48 2014] [notice] Digest: done
[Mon Mar 17 17:28:48 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.7 configured -- resuming normal operations
[Mon Mar 17 17:28:48 2014] [error] avahi_entry_group_add_service_strlst("10.61.0.3") failed: Invalid host name
What should I do?
Note:
I can run my larapp using builtin PHP web server.
LoadModule rewrite_module modules/mod_rewrite.so is uncommented in /etc/httpd/conf/httpd.conf
Listen 8080 already declared in /etc/httpd/conf/httpd.conf
In Apache documentation, ServerName should something like domain name http://httpd.apache.org/docs/2.2/mod/core.html#servername
Try to change ServerName 10.61.0.3 to ServerName example.com.
Don't forget to add 127.0.0.1 example.com into your /etc/hosts.
If you want VirtualHost runnning on port 8080, you must define NameVirtualHost http://httpd.apache.org/docs/2.2/vhosts/examples.html#port
For example:
Listen 8080
NameVirtualHost 10.61.0.3:8080
<VirtualHost 10.61.0.3:8080>
ServerName example.com
DocumentRoot /var/www/laraapp/public
</VirtualHost>

apache server keeps crashing on ubunutu, ssl: caught SIGTERM, shutting down

i have a windows machine and have ubuntu as a guest OS on VM. i set up apache onubuntu and im trying to configure ssl on that server but apache keeps crashing after following all instructions i found on the internet.
I have my ssl files in
/etc/apache2/ssl/server.crt
/etc/apache2/ssl/server.key
I have a default-ssl conf file with:
DocumentRoot /var/www-ssl/html/
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/server.crt
SSLCertificateKeyFile /etc/apache2/ssl/server.key
In the vhost and I also have the same in a vhost i am setting up(forums) forums-ssl conf file.
Mod ssl is already enabled
In my guest host file I have: 127.0.0.1 localhost test tribunal
When I restart apache, it asks me for my pass phrase, I enter it and it says ok, although it spits out those notices. But when I navigate to the page it times out.
and here is the error in apache error_log:
[Sun Mar 04 20:23:59 2012] [notice] caught SIGTERM, shutting down
[Sun Mar 04 20:24:04 2012] [notice] Apache/2.2.17 (Ubuntu) PHP/5.3.5-1ubuntu7.7 with Suhosin-Patch mod_ssl/2.2.17 OpenSSL/0.9.8o configured -- resuming normal operations
does anyone know why this is happening?