The servers (Ubuntu Server) it's going down because a 360Spider it's running too many request per seconds, I am trying to resolve this using this configuration in the .htaccess file:
BrowserMatchNoCase "360Spider" bots
BrowserMatchNoCase ^360Spider bots
Order Allow,Deny
Allow from ALL
Deny from env=bots
And works partially because the error.log logs some of this events:
[Sun Jul 20 23:30:15 2014] [error] [client 10.183.200.5] client denied by server configuration: /var/www/view, referer: http://www.mysite.org/
But the access.log it's still saving information about the 360Spider:
10.183.200.5 - - [20/Jul/2014:23:31:33 -0400] "GET /view/article/154967 HTTP/1.1" 403 536 "http://www.mysite.org/view/article/154967/" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider"
I want to block all that have the word 360Spider.
PD: I can't block the bot using the ip because all the traffic come with the same ip. I just can work with the .htaccess file.
Any IP Address, or bot going to a url/website will most likely make a GET request — Apache logs it. Just because you see it in the log does not mean it isn't blocked; Your access.log clearly shows that it is.
When the bot tried to GET /view/article/154967 it was denied (403 Forbidden).
Related
I use apache 2.4.41 and I would like to deny direct access to image files on my server.
I've implemented the following code in my apache configuration file :
SetEnvIf Referer "(www\.)?mywebsite\.net" localreferer
<FilesMatch "\.(jpg|png|gif)$">
Require env localreferer
</FilesMatch>
However I have a strange behavior. In a web browser, when a try to access an image file directly with its url, I don't get a 403 error code (as expected) and the image is displayed. But, when reloading the page (F5 or cmd+R on a mac), the ressource is blocked and a 403 error status is displayed.
When trying a curl -I, I have the following result :
HTTP/1.1 302 Found
Date: Wed, 06 Jul 2022 14:31:35 GMT
Server: Apache/2.4.41 (Ubuntu)
Location: http://www.mywebsite.net/error/403.php
But I should get HTTP/1.1 403 Forbidden...
Could someone help me with this issue ?
Many thanks in advance.
I have a simple Perl script that uploads a file from an HTML form, and it does works. i.e Uploads a file from my local Mac HD to my web server via a webpage.
What I have noticed, however, if I try to upload files from Microsoft's OneDrive I am more likely to get the following info below. I have no problems using my OneDrive via Mac's Finder, or my iPhone, etc.
access.log
[14/Feb/2022:23:36:51 -0500] "POST /cgi-bin/upload2.pl HTTP/1.1" 408
487 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2
Safari/605.1.15"
error.log
[Mon Feb 14 23:37:02.121496 2022] [cgi:error] [pid 3734:tid
140367391328000] (70007)The timeout specified has expired: [client
-.-.-.-:58184] AH01225: Error reading request entity data, referer: http://example.com/
My Apache2 settings are:
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
At my slim level of knowledge of Apache, I am assuming that my problem is all about timing. If that's the case, can I change settings above to help? Or am I off base?
*One small thing I noticed, but I don't know if it means anything
My web browser says:
Server timeout waiting for the HTTP request from the client. and it
mentions port 80.
However, in the error.log it mentions port 58184. I don't know if that's normal, due to routers, other routine behavior.
Set KeepAlive to Off - it seems to have a detrimental effect with busy Apache servers.
We are seeing random letters appear in access logs. The requests 404 since the content does not exist. The requests are made by a variety of users and other requests from the same ip usually look genuine. There is no way to request these from the site. Some of these requests even appear from internal traffic on our network.
Example:
157.203.177.191 - - [04/Feb/2018:23:51:20 +0000] "GET /VLTRP/content/dam/example/dotcom/images/ABtest/existing-customer-thumb.jpg HTTP/1.1" 404 60294 39082 "http://www.example.com/shop.html" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 2
Without the /VLTRP this is a genuine request. Has anyone seen something similar before?
For info we are running Apache/2.2.15 (Unix) with ModSec enabled. We do see similar behaviour on another site where we do not have ModSec configured. We see similar requests for internal, external and bot traffic.
We have some public content on the cloud and I am trying to use Basic Auth to deter people from accessing our non-production web-site unless they are on our corporate network.
Everything is working until I submit a request that is just an extension such as '/.js'.
$ wget https://test.domain.com/.js
HTTP request sent, awaiting response... 401 Authorization Required
Authorization failed.
If I add a file name, I am not asked for to authenticate. I get a 200 if the file actually exists.
$ wget https://test.domain.com/x.js
HTTP request sent, awaiting response... 404 Not Found
2014-05-29 16:59:35 ERROR 404: Not Found.
Here is the Basic Auth configuration. It is the .htaccess directory in the document root.
SetEnvIf HOST "^www.domain.com$" allowProduction
SetEnvIF X-Forwarded-For "123\.123\.123\.123" allowEmployee
AuthType Basic
AuthName "Authentication Required"
AuthUserFile /location/of/passwords/.htpasswd
Order Deny,Allow
Deny from all
Satisfy any
Require valid-user
Allow from env=allowProduction
Allow from env=allowEmployee
I modified the log format to display X-Forwarded-For since Apache sits behind a load balancer.
123.123.123.123 test.domain.com - - [29/May/2014:16:54:48 -0400] "GET /.js HTTP/1.0" 401 497 "-" "Wget/1.12 (linux-gnu)"
123.123.123.123 test.domain.com - - [29/May/2014:16:59:35 -0400] "GET /x.js HTTP/1.0" 404 287 "-" "Wget/1.12 (linux-gnu)"
I even tried adding some rule based on the request URI. I tried a few variations with and without the leading '/'. None of them worked.
SetEnvIf Request_URI "^/\..*$" allowJunk
And then added the following.
Allow from env=allowJunk
We do have a rewrite rule to forbid requests that have a leading '.' but that is a 403 and not a 401. A 403 is acceptable since the browser can ignore it. A 401 requires a user to enter the user name and password.
RewriteRule "(^|/)\." - [F]
An example of a forbidden request.
$ wget https://test.domain.com/x/.js
HTTP request sent, awaiting response... 403 Forbidden
2014-05-29 17:04:18 ERROR 403: Forbidden.
The simple solution would be to not request any leading '.' URLs but we are using someone else's framework and we get these requests on some browser. Rewriting the framework for a small percentage of our customers is not desirable. Prompting them for a password is even less desirable.
Any suggestions?
Thanks,
Wes.
Why can't I see why Apache returns 403?!
If I look in the access log the only information I get is
193.162.142.166 - - [29/Jan/2014:18:34:26 +0100] "POST /api_test/callback.php HTTP/1.1" 403 2293
How can I get more information about why the request is forbidden/rejected?
The call is made from a payment gateway...
If the callback URL is a http request there are no problems and returns 200 OK
If the callback URL is a https my server returns 403.. I need to know why?
The server has SSL and openSSL installed and it works!
Have tried to do the https request from http://web-sniffer.net/ and then there are no problems..
I don't get it.. There must be something in the request headers from the payment gateway which results in 403
update
error log
[Wed Jan 29 20:45:55 2014] [error] No hostname was provided via SNI for a name based virtual host
solution
Ok it looks like the client doesn't support SNI
http://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI
Use the LogLevel directive to adjust how verbose the error logs are and increase until you can see what you want.
httpd 2.4 has better messages in a lot of respect and expensive list of LogLevel settings than 2.2. So if you're using 2.2 it may be a bit harder to figure this out.