WCF Compression with .net 4 - wcf

I have read this WCF Compression article
I understand that for .net 4.0 WCF compression is available out of the box.
I can't find any clear explanation of how to use it, do I need to define any settings or change a binding? or it is compressed automatically?
I am using basicHttpBinding in IIS7. The option "enable dynamic compression" is set to true, but I don't get how the client knows to compress the requests and to decompress the response?
Any explanation including setting the binding to reduce message size will be appreciated. I am experiencing very bad performance when working on remote server with 4MB bandwidth.

but I don't get how the client knows to compress the requests and to decompress the response??
It's all part of the HTTP spec. Since WCF uses HTTP & IIS, it can leverage the built-in compression of the web server and client HTTP stack.
Check out section 14.3:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
Basically, your client needs to send a header saying it supports compression. Example: Accept-Encoding: gzip, deflate. You can set this by following the instructions in your article for the WCF Client section. Your client will then send out the right headers to the server.
Now on the server side, IIS will see that header, and it will compress the response...if configured to do so. The article you linked tells you how to set up IIS for compression for WCF services. The server will then send back a header to the client telling it the content is compressed: Content-Encoding: gzip. The client will then decompress the response and go on its merry way.
That's pretty much it; it's just a matter of getting the client headers right and the server configured to send back a compressed response. The article tells you how to do just that. Hope that helps

Note that compression has been added to WCF 4.5. It's covered here: http://msdn.microsoft.com/en-us/library/aa751889(v=vs.110).aspx
You have to use a custom binding to enable it:
<customBinding>
<binding name="BinaryCompressionBinding">
<binaryMessageEncoding compressionFormat="GZip"/>
<httpTransport />
</binding>
</customBinding>
It only works with binary encoding. Also, you have to be aware of your scenario. If you are hosted in IIS, compression may already be on. See here: http://blogs.msdn.com/b/dmetzgar/archive/2011/04/29/automatic-decompression-in-wcf.aspx

The compression sample is provided in .NET 4 WCF sample,
http://www.microsoft.com/en-us/download/details.aspx?id=21459
This blog post explains it with more information,
http://blogs.msdn.com/b/dmetzgar/archive/2011/03/10/compressing-messages-in-wcf-part-one-fixing-the-gzipmessageencoder-bug.aspx
There are other posts on MSDN blogs, like
http://blogs.msdn.com/b/dmetzgar/archive/2011/04/29/automatic-decompression-in-wcf.aspx

When using an http encoding, a good way to enable the compression of responses (only) is by using the dynamic compression built-in to IIS7 and higher.
but I don't get how the client knows to compress the requests and to decompress the response?
What follows is a description of what HTTP offers out of the box, which can be used together with the WCF HTTP(S) encodings. In addition to that, WCF 4.5 provides gzip and deflate compression of its binary encoding.
Compressed responses are part of the HTTP standard. In its request, the client signals to the server which compression methods (gzip, deflate, ...) it supports by means of the following header:
Accept-Encoding: gzip, deflate
The server, in its sole discretion, infinite wisdom and mysterious ways, is free to ignore that header and send the response uncompressed, or it may choose any one of the algorithms offered by the client, answering, say, with the following header and compress the response body.
Content-Encoding: gzip
To make matters more complicated, the server will probably also set the following header:
Transfer-Encoding: chunked
This allows the server to omit the otherwise mandatory Content-Length header that, as HTTP headers in general, has to precede the HTTP body. (Setting the chunked encoding affects the way the body gets encoded.) So now it can compress the response body on-the-fly, i.e. spitting out bytes as they are compressed, without having to wait for the compression of the whole body to finish, just to be able to determine the content length of the compressed result. This can save a lot of memory on the server side. (The client side, however, is now left in the dark as to the total size of the compressed response, until it finished receiving the whole response, making it's decompression slightly less efficient)
Note however that using Accept-Encoding and Content-Encoding, as I just described, to transparently compress responses was actually a stupid idea, according to http co-author Roy Fielding and what should have been used instead is the following header in the request:
TE: gzip, deflate
And the server, if it chooses to perform compression, would add the following header to its response:
Transfer-Encoding: gzip, chunked
As before, chunked is necessary if the server wants to omit the Content-Length.
Otherwise, the TE/Transfer-Encoding combo is syntactically identical to the Accept-Encoding/Content-Encoding combo, but the meaning is different, as can be gleaned from this longish discussion.
The gist of the problem: TE/Transfer-Encoding makes compression a transportation detail, whereas Accept-Encoding/Content-Encoding denotes the compressed version as the actual data (entity in HTTP parlance), with a number of unfortunate ramifications regarding caching of requests, proxy-ing, etc. of the latter.
However, the TE/Transfer-Encoding ship sailed long time ago, and we are stuck with the AE/CE combo, which is supported by most clients and servers with a meaning that is in actuality closer to that of TE/TE
When it comes to compressed requests in HTTP, they are rarely used in practice, and there is no standard way for the client to figure out if a server supports it. Either you tell the client out-of-band (e.g. hard-coding) that the server understands compressed requests (and configure the server appropriately). Or you have your client proactively try compression once, and if it yields a 400 Bad Request (at least that's what an IIS 7.5 would return), you fall back to non-compressed requests.

Related

ASP.NET Core - difference between Output caching and Response caching

ASP.NET Core 7 preview 6 just introduced Output caching which caches the endpoint output. However ASP.NET already has Response caching which seems to already provide the same feature.
What is the difference between the two and when should one be used and when should the other be used?
I was looking for aswers and trying to understand the differences between both, and really took a huge amount of time to understand the diferences between the two, and when (or not) to use each other.
As of November 2022 .Net 7 has been released, but the documentation is not very clear about the differences between them. The documentation and all videos only talk about the OutputCache as a replacement for the ResponseCache.
Also searching for OutputCache, it comes up with a lot of results from the old AspNet (Full framework) MVC 5.
So let´s clarify the differences and how we could use each other.
ResponseCache
First, the ResponseCache can be divided in 2 parts, that work independently and are different concepts of how and where the information would be cached. Let´s catch them up:
ResponseCacheAttribute: Basically it manipulates cache header like Vary, Cache-Control and others. It works telling the browsers or proxies to store (or not) the response content. This technique can reduce the number of requests done to the server, if used correctly.
The ResponseCache attribute sets response caching headers. Clients and
intermediate proxies should honor the headers for caching responses
under the HTTP 1.1 Caching specification
Response Caching Middleware: Basically it is used to make server side caching based on headers defined by ResponseCacheAttribute. Depending on the Request Headers sent to the server, the response would never be cached on server side.
Enables caching server responses based on HTTP cache headers.
Implements the standard HTTP caching semantics. Caches based on HTTP
cache headers like proxies do.
Is typically not beneficial for UI apps such as Razor Pages because
browsers generally set request headers that prevent caching. Output
caching, which is available in ASP.NET Core 7.0 and later, benefits UI
apps. With output caching, configuration decides what should be cached
independently of HTTP headers.
And at this point that OutputCache comes as a replacement for Response Caching Middleware.
OutputCache (available in ASP.NET Core 7.0 and later)
The OutputCache configuration decides what should be cached (server side) independently of HTTP headers. Also it comes with a lot of new features like cache entry invalidation, storage medium extensibility and others.
Conclusion
To take the benefits from both worlds you can use:
ResponseCacheAttribute: To manipulate response headers and enable the clients/proxies to store content on client side;
OutputCache: To store responses on server side and increase throuthput when responses are cached.
Both work independently. You can choose the one that fits best you application.
I haven't watch the video CodingMytra provided. But I think Output caching has some enhancements over Response caching. For example, you can specify a few seconds of caching.
I found a useful video, and it has some demos you learn more about the Output caching in .Net7. I think you can find the difference in this video.
We can find out why there is a need for Output caching in this github issue.
Link : Add support for Output Caching #27387

Under what conditions are HTTP request headers removed by proxies?

I'm looking at various methods of RESTfully versioning APIs, and there are three major contenders. I believe I've all but settled on using X-API-Version. Putting that debate aside, one of the arguments against using that header, and custom headers in general, is that you can't control when headers are manipulated by proxy servers. I'm curious about what real-world examples there are of this, when it happens on the internet at large, or when it might be used on an intranet or server cluster, or when it might occur in any other situation.
The Guidelines for Web Content Transformation Proxies 1.0 is pretty much the definitive guide to understanding and predicting standards-compliant proxy server behavior. In terms of your question, the Proxy Forwarding of Request portion of the document might be especially helpful.
Each proxy software package and their individual configurations will be vary but, HTTP proxies are generally expected to follow the W3C Guidelines. Here are some highlights.
4.1 Proxy Forwarding of Request:
Other than to convert between HEAD and GET proxies must not alter request methods.
If the request contains a Cache-Control: no-transform directive, proxies must not alter the request other than to comply with transparent HTTP behavior defined in [RFC 2616 HTTP] sections section 14.9.5 and section 13.5.2 and to add header fields as described in 4.1.6 Additional HTTP Header Fields below.
4.1.3 Treatment of Requesters that are not Web browsers
Before altering aspects of HTTP requests and responses proxies need to take account of the fact that HTTP is used as a transport mechanism for many applications other than "Traditional Browsing". Increasingly browser based applications involve exchanges of data using XMLHttpRequest (see 4.2.8 Proxy Decision to Transform) and alteration of such exchanges is likely to cause misoperation.
4.1.5 Alteration of HTTP Header Field Values
Other than the modifications required by [RFC 2616 HTTP] proxies should not modify the values of header fields other than the User-Agent, Accept, Accept-Charset, Accept-Encoding, and Accept-Language header fields and must not delete header fields (see 4.1.5.5 Original Header Fields).
Other than to comply with transparent HTTP operation, proxies should not modify any request header fields unless one of the following applies:
the user would be prohibited from accessing content as a result of the server responding that the request is "unacceptable" (see 4.2.4 Server Rejection of HTTP Request);
the user has specifically requested a restructured desktop experience (see 4.1.5.3 User Selection of Restructured Experience);
the request is part of a sequence of requests comprising either included resources or linked resources on the same Web site (see 4.1.5.4 Sequence of Requests).
These circumstances are detailed in the following sections.
Note:
It is emphasized that requests must not be altered in the presence of Cache-Control: no-transform as described under 4.1.2 no-transform directive in Request.
The URI referred to in the request plays no part in determining whether or not to alter HTTP request header field values. In particular the patterns mentioned in 4.2.8 Proxy Decision to Transform are not material.
4.1.6 Additional HTTP Header Fields
Irrespective of the presence of a no-transform directive:
proxies should add the IP address of the initiator of the request to the end of a comma separated list in an X-Forwarded-For HTTP header field;
proxies must (in accordance with RFC 2616) include a Via HTTP header field (see 4.1.6.1 Proxy Treatment of Via Header Field).
There is also lots of information regarding the alteration of response headers and being able to detect those changes.
As for web service REST API versioning, there is a very lucid and useful SO thread at Best practices for API versioning? that should provide a wealth of helpful insight.
I hope all of this helps. Take care.
This isn't an answer per se, but rather a mention of real-world scenario.
My current environment uses a mixed CAS/AD solution in order to allow SSO across several different platforms (classic ASP, ASP.NET, J2EE, you name it).
Recently we identified some issues - part of the solution involves aggregating Auth tokens to HTTP headers whenever necessary to propagate credentials. One specific solution, making considerable heavy usage of cookies, was chained with an nginx implementation, whose HTTP header limit was set to 4KiB. If the cookie payload went over 2KiB, it would start leaking out headers.
Consequently, applications that had some sort of state/scope control being coordinated via HTTP headers (session cookies included) suddenly started behaving erratically.
On an interesting, related note, REST services using URL versioning (http://server/api/vX.X/resource, for example) were unaffected.

HTTP POST/PUT files with headers that are guaranteed to be returned on GET operations

I would like to POST/PUT files on a web server with certain HTTP headers, and have the web server return those headers on each GET requests for those files.
Is there a way included inside the protocol to guarantee this, or is this behavior dependent on having a proprietary web server?
Thanks.
It depends on the web server. Most won't store data from request header fields.

Why is chunked transfer encoding not allowed to web applications running in CloudBees run#cloud?

I'm using an application that sends SOAP requests (HTTP POST) to my application running in CloudBees PaaS (run#cloud). The SOAP sender application gets the following error from the server: Transport error: 411 Error: Length Required. This means that it should not use chunked transfer encoding, because chunked doesn't send Content-length header which the server seems to want.
Is there some reason why chunked cannot be used? I'm aware that some web servers, like Apache, might have DOS vulnerabilities related to chunked transfer encoding. Is this the reason? Or is it because run#cloud uses Nginx as a proxy?
You can now set the httpVersion end to end for your app:
To enable: httpVersion=1.1
for example, this is how WebSocket works:
https://developer.cloudbees.com/bin/view/RUN/WebSockets
You can and should also set proxyBuffering=false - (this is default for new apps).
Cloudbees Nginx router indeed uses http 1.0 so don't have chunked transfert support. As we are working on websocket support, with a new version of NGinx, this may be available soon

Injecting data caching and other effects into the WCF pipeline

I have a service that always returns the same results for a given parameter. So naturally I would like to cache those results on the client.
Is there a way to introduce caching and other effect inside the WCF pipeline? Perhaps a custom binding class that could site between the client and the actual HTTP binding.
EDIT:
Just to be clear, I'm not talking about HTTP caching. The endpoint may not necessarily be HTTP and I am looking at far more effects than just caching. For example, one effect I need is to prevent multiple calls with the same parameters.
The WCF service can use Cache-Control directives in the HTTP header to say the client how it should use the client side cache. There are many options, which are the part of HTTP protocol. So you can for example define how long the client can just get the data from the local cache instead of making requests to the server. All clients implemented HTTP, like all web browsers, will follow the instructions. If your client use ajax requests to the WCF server, then the corresponding ajax call just return the data from the local cache.
Moreover one can implement many interesting caching scenarios. For example if one set "Cache-Control" to "max-age=0" (see here an example), then the client will always make revalidation of the cache by the server. Typically the server send so named "ETag" in the header together with the data. The "ETag" represent the MD5 hash or any other free information which will be changed if the data are changed. The client send automatically the "ETag", received previously from the server, together inside the header of the GET request to the server. The server can answer with the special response HTTP/1.1 304 Not Modified (instead of the typical HTTP/1.1 200 OK response) and with the body having no data. In the case the client will safe to get the data from the local cache.
I use "Cache-Control:max-age=0" additionally with Cache-Control: private which switch off caching the data on the proxy and declare that the data could be cached, but not shared with another users.
If you want read more about caching control with respect of HTTP headers I'll recommend you to read the following Caching Tutorial.
UPDATED: If you want implement some general purpouse caching you can use Microsoft Enterprise Library which contains Caching Application Block. The Microsoft Enterprise Library are published on the CodePlex with the source code. As an alternative in .NET 4.0 you can use System.Runtime.Caching. It can be used not only in ASP.NET (see here)
I continue recommend you to use HTTP binding with HTTP caching if it only possible in your environment. In the way you could save many time of development and receive at the end more simple, scalable and effective application. Because HTTP is so important, one implemened already so much useful things which you can use out-of-the-box. Caching is oly one from the features.