I have a restful java api that provides data to a Node.js client (that gzip data to users). The question is, If they are running in the same machine, should I Gzip the data from the java api to the node.js application?
I'm asking this because this case, I dont have to worry to network latency, but Gzip compression may increase CPU utilization.
Does it worth use gzip this situation?
If the objective is to increase speed of the overall system, then using gzip to transfer across processes boundaries would not be very useful, particularly if the message size is small enough to fit within memory. If the message is too large to fit in memory, and some paging overhead is incurred, the benefit of gzip may be greater but still not anywhere near enough to justify using it. Gzip only makes sense when the speed of compression is significantly greater than the speed of communication. This is usually not the case with inter-process communication (even if it incurs pagefault overhead.)
Related
We are currently testing out Image Resizer library and one of the questions is, how do we avoid malicious attacks to the site if someone programmically send thousands of resizing requests of images with arbitrary sizes to the server, overloading the CPU/RAM of server and potentially causing disk space to run out due to tremendous caching files.
Is there any way to whitelisting certain dimensions? Or what is the best practice to avoid this scenario?
Thanks!
Stephen
Neither CPU or RAM can generally be overloaded during a (D)DOS attack to ImageResizer. Memory allocation is contiguous, meaning an image cannot be processed unless there is around 15-30% free RAM remaining. Under the default pipeline, only 2 cores are used for image processing, so a regular server will not see CPU saturation either.
In general, there are far more effective ways to attack an ASP.NET website than though ImageResizer. Any database-heavy page is more likely to be a weak point, as the memory allocations are smaller and easier to saturate the server with.
Disk space starvation can be mitigated by enabling autoClean="true".
If you're a high-profile site with lots of determined ill-wishers, you can also consider the following:
Use request signing - only URLs generated by your server will be accepted.
Use the Presets plugin to white-list defined permitted command combinations.
Both of these reduce development agility and limit your options for responsive web design, so unless you have actually been attacked in the past, I don't suggest them.
In practice, (D)DOS attacks against dynamic imaging software are rarely useful at bringing down anything except — temporarily — uncached images — even when running under the same application pool. Since visited images tend to be cached, the actual effect is rather laughable.
Redis is "memory monster". Storing data as "compressed json string" minimizes memory usage.
Is there any built-in compression option in Redis Db?
Redis uses LZF light data compressor at the dump time, so it won't lessen the memory consumption. Implying that the redis does not compresses the data in memory and stores it as a string.You must deploy your own client side compression code.
The lua scripting also provides the compression algorithm but the branch is relatively new and therefore won't be advisable to use at production level.
No, there isn't any runtime compression option.
However, as dan-boa said - it might be a good idea to implement compression on your application side. Doing it that way will let to save CPU on the Redis server. Your Database server won't be affected of cpu time needed for compression.
In one of our Redis cluster we saved like 82% of memory (from circa 340GB to 60GB) thanks to GZIPing our json-based blobs. Some more thoughts about it and other ways of optimizing memory usage can be found in our article:
http://labs.octivi.com/how-we-cut-down-memory-usage-by-82/
Note: link moved to archive.org backup
Enabling compressing (gzip/deflate) in the Apache server will reduce the size of the response but will add more CPU cycles, I will run a stress test with various response sizes but
I wanted to ask if in terms of server load is there any suggestion on when should I turn compressing on or off?
Thank you
In most cases web servers are limited by io (be it memory, network bandwidth, database, hard drive, ...), and have plenty of spare cpu cycle to use for compressing the pages before serving them, especially since this isn't even really that much cpu intensive, while it provide a huge usability boost for your users and save you bandwidth.
I think that as long as the server has a powerful CPU, use compression. Speed is usually the best feature that servers should have after security and stability.
It depends on what you want to achieve. Tipically, turning deflate on won't add a very significant footprint to your CPU performance and if your website/s include large text files (html, js, css, etc.) it's likely to make an important difference in bandwidth usage and page loading times. Of course, if what you want is to reduce system load and don't care much for bandwidth, this wouldn't be the right choice for you.
Another option you might find useful is installing a lightweight web server/proxy like Nginx, lighttpd or Varnish (I personally prefer the first one), and serve compressed static content with that (leaving heavier Apache processes only to handle the dynamic content). That would also be likely to result in a better overall performance of your server. But, again, these all depends on your scenario, what's your website/web application like, and what you want to achieve.
We have web based j2ee application which allows file upload/download. Due to latency issue upload/download is slower for many users.
1) I read that sending data using UDP can improve data transfer speed. How can we send file data using UDP?
2) We are zipping file using GZIP before upload/download to reduce amount data transfer. Is there better method available improve data compression?
UDP is a protocol that does not guarantee the arrival of messages. You are most likely using a standard file transfer protocol like ftp which should suit you fine. Are your issues with latency or with bandwidth? You might be better of investigating why the link has a high latency or bandwidth issues, as this could prove to be an issue with other parts of your web application.
GZIP and other zipping tools are good for reducing the amount of data that is sent if you're willing up put up with the initial cost of compressing. These tools should have options so you can tweak the level of compression (i.e. take a long amount of time and compress optimally, or compress it quickly but have a larger zipped file). You will probably need to experiment and see what balance works the best for you.
1) Are there protocols faster than TCP on high latency links?
Yes, UDT is the primary example, but it is not a free trade, for instance consider you now need a custom frontend application to download files.
2) Is there better file compression than GZIP?
Yes, view the exhaustive list at http://www.maximumcompression.com/index.html, bzip2 and 7-zip are popular alternatives to gzip.
Note for specific domains, such as text, photographic images, scanned text, there are domain specific codecs which are more preferable.
We have a Win CE 6.0 device that is required to consume services that will be provided using WCF. We are attempting to reduce bandwidth usage as much as possible and with a simple test we have found that using UDP instead of HTTP saved significant data usage.
I understand there are limitations regarding WCF on .NET Compact Framework 3.5 devices and was curious what people thought would be the appropriate way forward. Would it make sense to develop a custom UDP binding, and would that work for both sides?
Any feedback would be appreciated. Thanks.
While http does have some overhead, if this is becoming a significant part of your data usage, then I would suspect that your API is too "chatty", and maybe fewer messages (each carrying more payload) should be considered.
The next point would be; how can we reduce the bandwidth for a given amount of payload? Compression is an option, but can be a problem on some platforms. Another is to use a serialization format that is inherently dense and efficient to process (in terms of CPU cycles, since you are using low-power devices). For that purpose, something like "protocol buffers" would be ideal.
protobuf-net is a CF-compatible implementation of protocol buffers for .NET; the CF build doesn't have all the nice WCF features (because CF doesn't support them), but it can work very effectively.
Additionally, if you do go http, then MTOM should be considered, as this reduces the encoding overhead of binary data (i.e. what protobuf-net would use).
Moving to UDP can be an option, but I would try something like http + protobuf-net + MTOM first (combined with a less "chatty" API), and see how it stacks up.
I should also note that the current (downloadable) version of protobuf-net has some "kinks" with CF; it works, but it isn't as fast etc as it could be (due to limitations in meta-programming on CF). The "v2" product (not yet released) addresses all these points, allowing fully static (and fast) execution on CF. And best of all, it is free.