Is it possible to DiskCache non-image files without losing their type when rendered in the browser?
I followed the instructions on this page:
http://imageresizing.net/docs/v4/howto/cache-non-images
Per the instructions, I set PostAuthorizeRequestStart = True and cache = Always in the PostAuthorizeRequestStart event. I also added the .unknown mimeType in the config.
However, when an xml file is requested, it's returned as content-type "application/octet-stream" instead of "text/xml".
Is there anyway to preserve the original content-type of non-image files?
I'm afraid not - at least not without modifying ImageResizer source code.
We made the decision to prioritize security, and save all files with the ".unknown" extension to prevent them accidentally being executed as scripts by IIS. IIS sends the content-type based on the file extension, and (depending on your IIS configuration), the extension determines if the file should be executed as code.
I see no harm in expanding the "whitelist" of extensions to include non-image file types, as long as we're reasonably confident that other users haven't allowed IIS to consider those file types executable.
The code that would need to be modified (in v4+), would be HttpModuleRequestAssistant.EstimateResponseInfo. Instead of falling back to "unknown" immediately, a second whitelist could be consulted.
If you file an issue on GitHub about this, you can subscribe to notifications. We'd definitely accept a pull request addressing this feature request, particularly during the current v4 prerelease phase when changes to the pipeline are less risky.
Related
I am rather new to TYPO3. Recently I noticed some very weird behavior in my installation: Some CSS-files in the directory typo3temp/assets/compressed got the MIME-type text/html instead of the expected text/css. Therefore my browser received a 403 Forbidden status code from the webserver for these resources. That resulted in some parts of the backend being shown without styling.
I tried clearing all caches and deleting the typo3temp/assets/compressed directory, however now all the stuff in there (CSS and JS) is served with MIME-type text/html. Getting the backend without JavaScript means, that I am now basically locked out of the backend. I can however still reach and use the install tool.
Do you have any ideas how this might happen and how to fix it?
Some details of my setup:
TYPO3 v10.4.13 (recently updated from 10.4.9)
Apache web server (I don't have access to its config and have to rely on .htaccess files)
I suggest to set
TYPO3_CONF_VARS/FE/compressionLevel=0
TYPO3_CONF_VARS/BE/compressionLevel=0
in order not have these kind of problems. The problem is that this compression creates compressed files but relies on webserver configuration in order to deliver them as text/css and NOT applying the default webserver's transport compression to them (or they could end up double-compressed and you might not even easily notice - some browsers can deal with that, others not).
It is a kind of micro-optimization that sounded useful in times when we avoided https:// because of the processing overhead...
Here's some docs (the first statement is outdated in my oppinion): https://docs.typo3.org/m/typo3/reference-skinning/master/en-us/BackendCssApi/CssCompression/Index.html
I have a Dynamics 365 instance that makes heavy use of custom front-end interfaces using a modern Nodejs-based build pipeline involving the usual suspects such as webpack/babel/etc. I'm hosting these files as webresources in Dynamics (one html file and one bundle.js file per SPA).
As my team nears production, I'm trying to set up a nice production build for our front-end stuff to reduce load times. Unfortunately, I can't find a good way to serve our bundle.js files encoded as gzip because Dynamics does not return the Content-Encoded: gzip header when a request is made and therefore the browser doen't decompress the file and tries to read the compressed file as plain JavaScript.
Of course, we can serve the uncompressed file just fine but we would like to provide the smaller, faster loading file if possible as it's generally about 1/3 the size.
Does anyone have any brilliant ideas for how to override the default response headers coming back from dynamics when I request a web resource? Or any other clever solutions to this problem?
Thanks, and let me know if any clarification is needed.
I don't know of any way to serve gzipped content via a web resource.
If the download size is a huge concern perhaps encode the gzipped code to base64 and store it as a string variable in JS.
Then during execution you could decode, unzip, and eval() the code.
You could also store base64 gzipped code as a file attachment via an annotation record or within an XML web resource, though those options would require an additional API call to get the code, so a string variable may be your best bet.
I sense I'm going to end up embarrassed for asking such a simple question, but I've been researching for days and can't any useful information.
What determines the HTTP response header that a server sends? If I control the server (if we need concreteness, let's say Apache), then what file can I edit to change the response header? For example, to set it to include Content-Length instead of Transfer-Encoding: chunked?
I'm aware that PHP and Java Servlets can be used to manipulate headers. The existence and content of response headers is fundamental to HTTP, though, so there ought to exist a way to edit these without using outside technology, no?
Certain headers are set automatically. They are part of the HTTP spec and the server takes care of them for you. That’s what a web server is for and why it differs from, say, an FTP server or a fileshare. For example Content-Length is easily calculated by the webserver and needs to be set so the server just does it.
Certain other headers are set based on config. Apache usually loads a main config file (often called httpd.conf or apache2.conf) but then, to save this file getting into a big unwieldy mess it often loads other files from within that. Those files are just text files with lines of configuration text to change behaviour of the server. Other web servers may use XML configuration files and may have a GUI to control the config (e.g. IIS)
So, for some of the headers, you might not explicitly set the header value but you basically configure the server and it then uses that config to figure out the appropriate headers to send. For example you can configure the server to gzip certain files (e.g. text files but not jpgs which are already compressed). In Apache this is handled by the mod_deflate module and the config options it gives you. Once the appropriate config is added to the server config, the server will do the necessarily processing (e.g. gzip the file or not depending on type) and then automatically add the headers. So an Apache module is basically something that changes how the server works and this may or may not the also set headers. Another example is for sending caching headers to tell the browser how long to cache files for. This is controlled by adding the mod_expiries module and all the config options it allows. While some of these headers could be hardcoded (e.g. Cache-Control) others depend on Apache doing calculations (e.g. Expires) so better to use the module to do this for you based on your config.
And finally you can explicitly set headers in your server (in Apache this is done using the mod_headers module). This is useful for new features added to browsers for example (e.g. HSTS, CSP or HPKP) where the server doesn't need to do anything but just add the header and the client (e.g. the web browser) knows what to do with them. You could add a JonahHuron header for example by adding this config to httpd.conf:
Header always set JonahHuron "Some Value"
As to whether that header is used depends entirely on the program receiving the response.
One of the responsibilities of my Rails application is to create and serve signed xmls. Any signed xml, once created, never changes. So I store every xml in the public folder and redirect the client appropriately to avoid unnecessary processing from the controller.
Now I want a new feature: every xml is associated with a date, and I'd like to implement the ability to serve a compressed file containing every xml whose date lies in a period specified by the client. Nevertheless, the period cannot be limited to less than one month for the feature to be useful, and this implies some zip files being served will be as big as 50M.
My application is deployed as a Passenger module of Apache. Thus, it's totally unacceptable to serve the file with send_data, since the client will have to wait for the entire compressed file to be generated before the actual download begins. Although I have an idea on how to implement the feature in Rails so the compressed file is produced while being served, I feel my server will get scarce on resources once some lengthy Ruby/Passenger processes are allocated to serve big zip files.
I've read about a better solution to serve static files through Apache, but not dynamic ones.
So, what's the solution to the problem? Do I need something like a custom Apache handler? How do I inform Apache, from my application, how to handle the request, compressing the files and streaming the result simultaneously?
Check out my mod_zip module for Nginx:
http://wiki.nginx.org/NgxZip
You can have a backend script tell Nginx which URL locations to include in the archive, and Nginx will dynamically stream a ZIP file to the client containing those files. The module leverages Nginx's single-threaded proxy code and is extremely lightweight.
The module was first released in 2008 and is fairly mature at this point. From your description I think it will suit your needs.
You simply need to use whatever API you have available for you to create a zip file and write it to the response, flushing the output periodically. If this is serving large zip files, or will be requested frequently, consider running it in a separate process with a high nice/ionice value / low priority.
Worst case, you could run a command-line zip in a low priority process and pass the output along periodically.
it's tricky to do, but I've made a gem called zipline ( http://github.com/fringd/zipline ) that gets things working for me. I want to update it so that it can support plain file handles or paths, right now it assumes you're using carrierwave...
also, you probably can't stream the response with passenger... I had to use unicorn to make streaming work properly... and certain rack middleware can even screw that up (calling response.to_s breaks it)
if anybody still needs this bother me on the github page
Is it possible to send pre-compressed files that are contained within an EARfile? More specifically, the jsp and js files within the WAR file. I am using Apache HTTP as the web server and although it is simple to turn on the deflate module and set it up to use a pre-compressed version of the files, I would like to apply this to files that are contained within an EAR file that is deployed to JBoss. The reason being that the content is quite static and compressing it on the fly each time is quite costly in terms of cpu time.
Quite frankly, I am not entirely familiar with how JBoss deploys these EAR files and 'serves' them. The gist of what I want to do is pre-compress the files contained inside the war so that when they are requested they are sent back to the client with gzip for Content-Encoding.
In theory, you could compress them before packging them in the EAR, and then serve them up with a custom controller which adds the http header to the response which tells the client they're compressed, but that seems like a lot of effort to go to.
When you say that on-the-fly compression is quite costly, have you actually measured it? Have you tried requesting a large number of uncompressed pages, measured the cpu usage, then tied it again with compressed pages? I think you may be over-estimating the impact. It uses quite low-intensity stream compression, designed to use little CPU resources.
You need to be very sure that you have a real performance problem before going to such lengths to mitigate it.
I don't frequent this site often and I seem to have left this thread hanging. Sorry about that. I did succeed in getting compression to my javascript and css files. What I did was I precompress them in the ant build process using the gzip. I then had to spoof the name to get rid of the gzip extension. So I had foo.js and compressed it into foo.js.gzip. I renamed this foo.js.gzip to foo.js and this is the file that gets packaged into the WAR file. So that handles the precompression part. To get this file served up properly, we just have to tell the browser that this file is compressed, via the content-encoding header of the http response. This was done via a output filter that is applied to files that matched the *.js extension (some Java/JBoss, WEB-INF/web.xml if it helps. I'm not too familiar with this so sorry guys).