Conditional request not honored in Includes - apache

I'm trying to dynamically concatenate a bunch of javascript files into a single file using the INCLUDE filter. The include.shtml.js test script is
<!--#include virtual="/static/script2.js" -->
<!--#include virtual="/static/script1.js" -->
The virtual server config has both SSIETag and SSILastModified set to On for that file
<VirtualHost *:80>
ServerName test.dkt
ServerAlias test.com
UseCanonicalName Off
ErrorLog logs/test.dkt-error_log
CustomLog logs/test.dkt-access_log combined
LogLevel info
FileEtag All
AddType application/javascript .js
DocumentRoot /var/www/html/test.com
<Directory /var/www/html/test.com>
Options -Indexes
ExpiresActive Off
ExpiresDefault "access plus 1 years"
Header append Cache-Control "public"
Order deny,allow
Allow from all
</Directory>
<Directory /var/www/html/test.com/static>
<FilesMatch "\.shtml\.js$">
SSIETag On
SSILastModified On
Options +Includes
SetOutputFilter INCLUDES
</FilesMatch>
</Directory>
</VirtualHost>
It correctly serves the concatenated scripts but is always a full 200 OK in instead of a 304 Not Modified. The Firebug log
Response Headers
HTTP/1.1 200 OK
Date: Fri, 24 Jan 2014 16:57:12 GMT
Server: Apache/2.2.15 (CentOS)
Last-Modified: Fri, 24 Jan 2014 16:53:32 GMT
Etag: "460bbc-5c-4f0ba32b7447d"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Cache-Control: public
Content-Length: 40
Connection: close
Content-Type: application/javascript
Request Headers
GET /static/include.shtml.js HTTP/1.1
Host: test.dkt
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: pt-br,en-us;q=0.9,es;q=0.7,en;q=0.6,zh-tw;q=0.4,ar-sa;q=0.3,ar;q=0.1
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
If-Modified-Since: Fri, 24 Jan 2014 16:53:32 GMT
If-None-Match: "460bbc-5c-4f0ba32b7447d"
Cache-Control: max-age=0
Is there a hard coded restriction on conditional requests for the INCLUDE filter?
I'm aware that I should "touch" the including script whenever there is a change in any of the included scripts. The Apache version is 2.2 running in Centos 6
EDIT
Using the #covener answer I made it work setting the group execute permission of the file and adding the XBitHack full directive

Even though you've opted into the etags, it seems you need to separately enable xbithack to allow a 304 to be generated (ap_meets_conditions in the core checks no_local_copy flag referenced in mod_include
http://httpd.apache.org/docs/current/mod/mod_include.html#xbithack
/* When our xbithack value isn't set to full or our platform isn't
* providing group-level protection bits or our group-level bits do not
* have group-execite on, we will set the no_local_copy value to 1 so
* that we will not send 304s.
*/
if ((conf->xbithack != XBITHACK_FULL)
|| !(f->r->finfo.valid & APR_FINFO_GPROT)
|| !(f->r->finfo.protection & APR_GEXECUTE)) {
f->r->no_local_copy = 1;
}

When having
SSILastModified on
XBitHack full
together in configuration file, the setting "SSILastModified On" is a silent misconfiguration, because whether "SSILastModified" is on or not, it does not change any program behavior.
By tracking back to the source code of Apache, we can see the root cause of this misconfiguration is that the semantics enabled by "Xbithack Full" implicitly overwrite the semantics enabled by "SSILastModified On".
if (conf->lastmodified > 0) {
... {
ap_update_mtime(r, r->finfo.mtime);
ap_set_last_modified(r);}}
else if (((conf->xbithack == XBITHACK_FULL ||
(conf->xbithack == XBITHACK_UNSET &&
DEFAULT_XBITHACK == XBITHACK_FULL))
...)) {
ap_update_mtime(r, r->finfo.mtime);
ap_set_last_modified(r);
}
So one possible solution would be just keep this
Xbithack full

Related

Apache (not the browser) is caching my file

The browser is not caching it. It gets the response headers:
Accept-Ranges:bytes
Cache-Control:max-age=0, no-cache, no-store, must-revalidate
Connection:Keep-Alive
Content-Length:425169
Content-Type:application/javascript
Date:Thu, 09 Mar 2017 20:06:53 GMT
Expires:Wed, 11 Jan 1984 05:00:00 GMT
Keep-Alive:timeout=5, max=100
Last-Modified:Thu, 09 Mar 2017 20:06:49 GMT
Pragma:no-cache
Server:Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16
My settings in Apache:
<VirtualHost *:80>
<Directory "/webapps/apps/devsite">
Allow from all
AllowOverride All
Order allow,deny
</Directory>
DocumentRoot /webapps/apps/devsite
ServerName testing.devsite.com
SSLEngine off
</VirtualHost>
My .htaccess:
<FilesMatch "\.(html|htm|js|css)$">
FileETag None
<IfModule mod_headers.c>
Header unset ETag
Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
Header set Pragma "no-cache"
Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT"
</IfModule>
</FilesMatch>
The following loads a new, non-cached version:
on the server run: rm -f /webapps/apps/devsite/scripts/script.js
Reload in the web browser (thus getting a 404)
Copy the file back on to server
Reload in browser
The following does loads an old, CACHED version!:
On the server run: rm -f /webapps/apps/devsite/scripts/script.js
Copy the file back on to server (NOTE: I did not reload in browser yet)
Reload in browser
This shows that Apache is somehow caching it until it gets a new request and cannot find it. Why? How do i fix this?
The issue was it was using the kernel's SendFile which caused it to miss the file being changed. This is a Virtual Machine shared folder. Adding the following fixes it:
EnableSendfile off
(the "file" is lowercase)
More info here: https://www.vagrantup.com/docs/synced-folders/virtualbox.html
http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile
Apache does not permanently watch all files, only when you request a specific resource.
When you hit the 404 error, Apache loses the information about the file it has had found before.
The last modified timestamp does not change when you don't request a resource in the meantime.

Apache deflate javascript compress ratio 1.00x

If I ran gzip in CLI, I can get a good compression ratio:
bundle.js: 75.3% -- replaced with bundle.js.gz
But in Apache, even I set deflate, it did compress, but with same file size. Below is my Apache config:
LoadModule deflate_module libexec/apache2/mod_deflate.so
<IfModule deflate_module>
DeflateCompressionLevel 9
AddOutputFilterByType DEFLATE application/javascript text/plain text/css
CustomLog /var/log/deflate_log DEFLATE
</IfModule>
Below is the response:
ETag "8342b-53dc33d01d2c0-gzip"
Server Apache/2.4.23 (Unix)
Content-Type application/javascript
Last-Modified Sat, 01 Oct 2016 01:00:35 GMT
Date Sun, 02 Oct 2016 01:14:20 GMT
Connection Keep-Alive
Vary Accept-Encoding
Accept-Ranges bytes
Keep-Alive timeout=5, max=98
Content-Encoding gzip
Transfer-Encoding Identity
The network transfer size is same as before, and the ratio is 1.00x. I narrowed it down to only js get not compressed, instead css get good compression ratio of 6.22x. Is there something wrong with the js file?
I got it!
I noticed there is no "content-length" header in the response. So I went back to check Apache document. It says:
The DeflateBufferSize directive specifies the size in bytes of the
fragments that zlib should compress at one time. If the compressed
response size is bigger than the one specified by this directive then
httpd will switch to chunked encoding (HTTP header Transfer-Encoding
set to Chunked), with the side effect of not setting any
Content-Length HTTP header. This is particularly important when httpd
works behind reverse caching proxies or when httpd is configured with
mod_cache and mod_cache_disk because HTTP responses without any
Content-Length header might not be cached.
As my js file is 500k, much over the default 8k setting, so I added the following in conf file, now everything is good:
<IfModule deflate_module>
SetOutputFilter DEFLATE
AddOutputFilterByType DEFLATE application/javascript text/plain text/css
DeflateBufferSize 8096000
</IfModule>

Setting cache-control max-age using Apache not working

I'm trying to setup HTTP Caching for my website. Following is my configuration settings
# 1 YEAR
<FilesMatch "\.(ico|svg|woff|eot|ttf)$">
Header set Cache-Control "max-age=31536000, public"
</FilesMatch>
# 1 WEEK
<FilesMatch "\.(jpg|png|gif|css|js)$">
Header set Cache-Control "max-age=604800, public"
</FilesMatch>
Does it make a difference if I place this in my <VirtualHost> settings or outside it? I've placed it inside the <VirtualHost>.
I tried checking the HTTP response for one of the png image using redbot.org and this is what it returned.
HTTP/1.1 200 OK
Date: Fri, 12 Sep 2014 09:28:33 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Tue, 26 Aug 2014 05:43:32 GMT
ETag: 1409031812.69
Content-Length: 23907
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: image/png
Why is there no Cache-Control max-age header tag?
I also checked using the Google PageSpeed Insights and it still says expiration not specified for all the files.
Did I miss something?
The .htaccess file was not being taken into account due to some missing configuration settings in my apache2.conf file. Making the required changes in the conf file solved the issue.

Bypassing authentication for "Options request" (so all headers are sent in the response)

This is in the context of Cross-origin resource sharing. For the preflight request, the server is not sending the headers set.
When a valid cookie is not passed with the "Options request", the server in it's response is not sending the headers I set, however, it's sending "200 OK". I checked this with curl as can be seen below (obviously, I replaced my valid cookie with a dummy "xyzabcde" here)
The curl request WITHOUT cookie:
curl -H "Origin: app2_url" -H "Access-Control-Request-Method: POST" -H "Access-Control-Request-Headers: accept, origin, content-type" -X OPTIONS --verbose app1_url/jsonrpc.cgi
(sends below response...)
HTTP/1.1 200 OK
Date: Tue, 01 Oct 2013 11:37:36 GMT
Server: Apache
Expires: Tue, 01 Oct 2013 11:37:36 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Expires: Tue, 01 Oct 2013 11:37:36 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Length: 4531
Content-Type: text/html; charset=utf-8
with "-H Cookie:xyzabcde":
curl -H "Origin: app2_url" -H "Access-Control-Request-Method: POST" -H "Access-Control-Request-Headers: accept, origin, content-type" "-H Cookie:xyzabcde" -X OPTIONS --verbose app1_url/jsonrpc.cgi
(sends below response...)
HTTP/1.1 403 Forbidden
Date: Wed, 02 Oct 2013 18:48:34 GMT
Server: Apache
X-frame-options: ALLOW-FROM app2_url
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: accept, origin, content-type, Man, Messagetype, Soapaction, X-Requested-With
Access-Control-Allow-Methods: GET, POST, HEAD, PUT, OPTIONS
Access-Control-Allow-Origin: app2_url
Access-Control-Max-Age: 1800
Transfer-Encoding: chunked
Content-Type: application/json; charset=UTF-8
The apache config looks something like...
<VirtualHost *:443>
.
.
Header always set X-Frame-Options "ALLOW-FROM app2_url"
Header always set Access-Control-Allow-Credentials "true"
Header always set Access-Control-Allow-Headers "accept, origin, content-type, Man, Messagetype, Soapaction, X-Requested-With"
Header always set Access-Control-Allow-Methods "GET, POST, HEAD, PUT, OPTIONS"
Header always set Access-Control-Allow-Origin "app2_url"
Header always set Access-Control-Max-Age "1800"
.
.
.
<Directory /app1/dir/>
Options Includes FollowSymLinks ExecCGI MultiViews
AllowOverride None
Order allow,deny
allow from all
AuthType Net
PubcookieInactiveExpire -1
PubcookieAppID app1.company.com
require valid-user
</Directory>
.
.
</VirtualHost>
How can I make all the headers be sent in response to unauthenticated requests?
I guess, Options requests ideally are supposed to not require any authentication.
We solved this with different configuration. Below is the snippet from myApplication.conf file at /usr/local/apache/conf/extra
<Location "/myService">
SetEnvIf Request_URI "/healthCheck" REDIRECT_noauth=1
SetEnvIf Request_Method "OPTIONS" REDIRECT_noauth=1
AuthType Basic
AuthName "myService"
AuthUserFile /usr/local/apache/conf/passwd/passwords
AuthGroupFile /usr/local/apache/conf/passwd/groups
Require group GroupName
Order allow,deny
Allow from env=REDIRECT_noauth
Satisfy any
</Location>
So, we can bypass the authentication:
Based on particular URI, in above example /healthCheck is bypassed
Based on HTTP method, in above example OPTIONS is bypassed and auth will be prompted for other HTTP methods
Hope it helps someone to resolve the issues.
"LimitExcept" directive solved it. In fact, prior to posting the question I tried the directive, however the mistake earlier was including the first two lines ("Options Includes..." and "Alowoverride...") within the "LimitExcept" block.
<Directory /app1/dir/>
Options Includes FollowSymLinks ExecCGI MultiViews
AllowOverride None
<LimitExcept OPTIONS>
Order allow,deny
allow from all
AuthType Net
PubcookieInactiveExpire -1
PubcookieAppID app1.company.com
require valid-user
</LimitExcept> #<- syntax error fixed.
</Directory>

How do I use mod_deflate in Apache 2.4.1 with Haproxy?

I have a strange issue whereby including the following syntax in my Apache 2.4.1 httpd.conf causes "502 Bad Gateway" errors when retrieving swf files via HAproxy:
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript text/javascript
When I remove this config line the 502 Bad Gateway error goes away.
The server returns these response headers on a successful request:
Date: Wed, 11 Apr 2012 20:24:12 GMT
Server: Apache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
200 OK
I fixed this by updating to Apache 2.4.2 (there was a mod_deflate seg fault bug in 2.4.1) and adding:
Header append Vary User-Agent
Beneath the AddOutputFilterByType line.