On apache access log, I find out that https return bigger size than http
210.10.0.156 - - [29/Apr/2019:12:22:46 +0800] "GET /robots.txt HTTP/1.1" 200 5837 "-" "curl/7.52.1"
If you can see it is 5837 bytes where is for http less than 1000 bytes
my robots.txt content is only
User-agent: *
Disallow: /
Is this a normal things?
When i tried to do same thing on other server with cpanel installed, the size is much more lower, im not sure what configuration i missed, any advise?
Yes, this is perfectly normal.
Your website may not be configured for HTTP, which will redirect people to HTTPS with a 301 or 302 rule, meaning all they see when they access your site through HTTP is a redirect page, which is usually smaller than the regular webpage which they were expecting.
You can go to your website with http:// or https:// at the start of the URL and see if it looks any different.
Related
I have an HTTPS-based site that loads CSS via HTML just fine, but not via HTTPS.
http://site/foo.css
... loads the asset fine. But...
https://site/foo.css
Does not. I get an Apache 502 error. The Apache access log shows:
[07/Nov/2018:10:17:20 -0800] "GET /foo.css HTTP/1.0" 200 95568 "-" ...
That tells me that it's trying to load my foo.css as HTTP even though I specified HTTPS. Also note that while my browser gives a 502 error, I get a 200 response in the logs.
Seems like some sort of HTTPS misconfiguration but I'm not sure what. Help?
Use
//site/foo.css
instead of adding protocol when linking your css.
link everything with https as secure sites does not support mixed content.
I try to do, but when i turn to bypass my domain A record, the protocol h2 is working.
All correct in browser and apache logs.
When i turn on the cloudflare on the domain, the browser is working fine(I know the CF is working on HTTP2 protocol with the clients).
But I saw these in the apache logs:
“GET / HTTP/1.1” 302 5067
"GET /en/ HTTP/1.1" 200 5068
"POST /en/ajax/user HTTP/1.1" 200 77
It's using http/1.1, but I don't know why.
I try to force http2 protocol, but no difference.
Any idea ?:slightly_smiling_face:
Thank you!
Cloudflare plans to use HTTP 1.1 to the origin for the foreseeable future.
https://support.cloudflare.com/hc/en-us/articles/214534978-Are-the-HTTP-2-or-SPDY-protocols-supported-between-Cloudflare-and-the-origin-server-
You can use HTTP/2 to origin as of June 14th, 2022:
1. Log into the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account.
2. Choose the domain that will use HTTP/2 to Origin.
3. Click **Network**.
4. Under **HTTP/2 to Origin** set the toggle to On.
Refs are:
https://github.com/cloudflare/cloudflare-docs/blob/production/content/cache/how-to/enable-http2-to-origin.md
https://developers.cloudflare.com/cache/how-to/enable-http2-to-origin/
Do you know if it is possible to force the robots crawl on www.domaine.com and not domaine.com ? In my case, I have a web app that has enabled cached urls with prerender.io (to view the HTML code), but only on www.
So, when the robots crawl on domaine.com, it has no data.
The redirection is automatic (domaine.com> http://www.domaine.com) on Nginx, but no results.
I said that my on my sitemap, urls have all www.
My Nginx redirect :
server {
listen *:80;
server_name stephane-richin.fr;
location / {
if ($http_host ~ "^([^\.]+)\.([^\.]+)$"){
rewrite ^/(.*) http://www.stephane-richin.fr/$1 redirect;
}
}
}
Do you have an idea ?
Thank you !
If you submitted a sitemap with the correct URLs a week ago, it seems strange that the Google keeps requesting the old ones.
Anyway - you’re sending the wrong status code in your non-www to www redirect. You are sending a 302 but should be sending a 301. Philippe explains the difference in this answer:
Status 301 means that the resource (page) is moved permanently to a new location. The client/browser should not attempt to request the original location but use the new location from now on.
Status 302 means that the resource is temporarily located somewhere else, and the client/browser should continue requesting the original url.
Could you have a robots.txt file with
User-agent: *
Disallow: /
on domaine.com and a different one with
User-agent: *
Disallow:
on www.domaine.com?
I have apache with version 2.22 and have these rare logs with IPs from Japan, Taiwan ...
218.52.178.3 - - [10/Feb/2015:13:48:48 +0100] "\xa6Q\xb0\xce\xf5\b\x15\x05\xd3\xaf\xbe\x1d\xac\xbf\x99lx\xbf\xb0\x16w{\xb3\xf4\xa5%\xd6\x16\xd0\x15\xd5\xfa\xcbqo\xae9\xc2\x13\xf5&\x91\xb7\x90\xa1\xe5W\xe3" 200 62
98.119.71.85 - - [10/Feb/2015:15:48:47 +0100] "\x86}\x12%\xe8<\xc9\xb8\xed\xb9\x9d4j+0-\x19\xb9\xd3\xafA\xa9#7\x04q\x17\xe3\xa4\x87\xd3\t\x04\x92\x8493X\xd3H?\xf1\xb8\x96\x11\xbf\xc9\xddQ" 400 226
Does anyone know what they mean?
This is some kind of crawler, checking for vulnerabilities. You can block them by IP. Howevere They change the IP constantly.
Create a .htaccess file in your Document root and insert this in to it
Order Deny,Allow
Deny from 218.52.178.3
Deny from 98.119.71.85
If you have access to your firewall you can block hem from there. Each firewall has a different way to block IPs.
This is the requests made in Eastern languages by UTF encoding. Hieroglyphs . Apache converts it to english letters. They can be decoded to origin using some online services.
In my apache instillation, I keep seeing the following line in my access logs:
"POST http://yourinfo.allrequestsallowed.net/ HTTP/1.1" 200
It's really freaking me out because this site is not being hosted on my server (I checked the IP just to be 100% sure). I added a "Deny all" line since the site is still in development, and now the HTTP 200 response changed to 403, like the domain is being hosted on my server.
I'm incredibly confused and scared. Does anybody know what's going on? Can I Deny all to this domain that's apparently pointing to my server?
You may want to check to make sure you don't have ProxyRequests On set anywhere where it's not supposed to. Typically a request like that is for a forward proxy and the troubling bit is that you returned a 200 response which could indicate that the request was successfully proxied.
Take a look at this wiki page about Proxy abuse.
My server is properly configured not to proxy, so why is Apache returning a 200 (Success) status code?
That status code indicates that Apache successfully sent a response to the client, but not necessarily that the response was retrieved from the foreign website.
RFC2616 section 5.1.2 mandates that Apache must accept requests with absolute URLs in the request-URI, even for non-proxy requests. This means that even when proxying is turned off, Apache will accept requests that look like proxy requests. But instead of retrieving the content from the foreign site, Apache will serve the content at the corresponding location on your website. Since the hostname probably doesn't match a name for your site, Apache will look for the content on your default host.
But it's probably worthwhile to check that you aren't proxying. Otherwise, it's not really that big of a deal.
After Jon Lin pointed me in the right direction, I figured it out.
After disabling mod_proxy and enabling mod_security, I added the following to my virtual host configuration:
SecRuleEngine On
SecRule REQUEST_LINE "://" drop,phase:1
And then restarted apache. It quits the connection and returns any amount of data, which uses less resources and bandwidth during Brute Force and DDOS attacks.
Also, it shows as an HTTP 404 Response in the access logs.
EDIT: I updated the rule to drop all types or proxies (https,https,ftp). I don't know how many protocols can be used this way, but I'd rather be safe than sorry.