restsharp accept-encoding disabling compression - restsharp

In a particular case I need to be able to disable compression in the requst/response.
Using Firefox RestClient I am able to post some xml to a web service and get some response xml successfully with a single header parameter "Accept-Encoding" : " "
which if I do not set this header, the response body would come back compressed with some binary data in the response body(that's why I want to disable gzip in response)
Now using the same header value in my app (using RestSharp in C#), I still get the binary data (gzip) in response.
Can someone please shed some light? Is it supported in RestSharp?

RestSharp does not support disabling compression.
If you look at the source code in Http.Sync.cs line 267 (assuming a sync request, async has the same code duplicated in Http.Async.cs line 424)
webRequest.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
that is, the underlying WebRequest that Restsharp uses to make the http call has the compression options hardcoded. There is an open issue that documents this

The feature (only just) seems to have been added, but stealthily - without a note on the issue's status nor on the changelogs. Possibly as it hasn't been sufficiently tested?
Nevertheless I recently had a need for this functionality and tested it - and it works. Just set the RestClient instance's AutomaticDecompression property to false.
If you intend to keep your RestClient instance long-lived remember to do this before its first use - the setting seems to be 'locked in' after use and cannot change after. In my case I needed to make calls with and without AutomaticDecompression so i simply created two different RestClient instances.

Using RestSharp v106.11.4, I was unable to turn off automatic decompression as Bo Ngoh suggested. I set the AutomaticDecompression on the RestClient instance at the moment it gets instantiated, but still the Accept-Encoding header was added.
The way to set this & disable the decompression is through the ConfigureWebRequest method, which is exposed on the RestClient. Below snippet allowed me to turn off this feature:
var client = new RestClient();
client.ConfigureWebRequest(wr =>
{
wr.AutomaticDecompression = DecompressionMethods.None;
});

Not sure if this relevant anymore, but for maybe future references
RestRequest has IList<DecompressionMethods> AllowedDecompressionMethods, and when creating new RestRequest the list is empty. Only when calling the Execute method it fills with the default values (None, Deflate, and GZip) unless it's not empty
To update the wanted decompression method, simply use the method named AddDecompressionMethod and add the wanted decompression method - and that's that
Example:
var client = new RestClient();
var request = new RestRequest(URL, Method.GET, DataFormat.None);
request.AddDecompressionMethod(DecompressionMethods.GZip);
var response = client.Execute(request);

As of RestSharp version 107, the AddDecompressionMethod has been removed and most of the client options has been move to RestClientOptions. Posting here the solution that worked for me, in case anyone needs it.
var options = new RestClientOptions(url)
{
AutomaticDecompression = DecompressionMethods.None
};
_client = new RestClient(options);

Related

In karate mocking (karate-netty), how can we override request header value?

Objective:
We want few API calls should go to mock-server(https://192.x.x.x:8001) and others should go to an actual downstream application server(https://dev.api.acme.com).
Setup :
On local, mock server is up with standalone jar on port 8001. e.g https://192.x.x.x:8001
In application config file (config.property)downstream system(which need to mock) defined with mockserver IP i.e https://192.x.x.x:8001
Testing scenario and problem:
1.
Scenario: pathMatches('/profile/v1/users/{id}/user')
* karate.proceed('https://dev.api.acme.com')
* def response = read ('findScope.json')
* def responseStatus = 200ˀˀ
* print 'created response is: ' + response
Now, when we hit API request via postman or feature file then it does karate.proceed properly to https://dev.api.acme.com/profile/v1/users/123/user instead of 192.x.x.x. However, in this request, host is referring to https://192.x.x.x:8001 instead of https://dev.api.acme.com which create a problem for us.
How can we override request header in this case? I did try with karate.set and also with header host=https://192.x.x.x:8001 but no luck.
Thanks!
Please see if the 1.0 version works: https://github.com/intuit/karate/wiki/1.0-upgrade-guide
Unfortunately https proxying may not work as mentioned. If you are depending on this, we may need your help (code contribution) to get this working
If the Host header is still not mutable, that also can be considered a feature request, and here also I'd request you to consider contributing code

ASP.NET Web API - Reading querystring/formdata before each request

For reasons outlined here I need to review a set values from they querystring or formdata before each request (so I can perform some authentication). The keys are the same each time and should be present in each request, however they will be located in the querystring for GET requests, and in the formdata for POST and others
As this is for authentication purposes, this needs to run before the request; At the moment I am using a MessageHandler.
I can work out whether I should be reading the querystring or formdata based on the method, and when it's a GET I can read the querystring OK using Request.GetQueryNameValuePairs(); however the problem is reading the formdata when it's a POST.
I can get the formdata using Request.Content.ReadAsFormDataAsync(), however formdata can only be read once, and when I read it here it is no longer available for the request (i.e. my controller actions get null models)
What is the most appropriate way to consistently and non-intrusively read querystring and/or formdata from a request before it gets to the request logic?
Regarding your question of which place would be better, in this case i believe the AuthorizationFilters to be better than a message handler, but either way i see that the problem is related to reading the body multiple times.
After doing "Request.Content.ReadAsFormDataAsync()" in your message handler, Can you try doing the following?
Stream requestBufferedStream = Request.Content.ReadAsStreamAsync().Result;
requestBufferedStream.Position = 0; //resetting to 0 as ReadAsFormDataAsync might have read the entire stream and position would be at the end of the stream causing no bytes to be read during parameter binding and you are seeing null values.
note: The ability of a request's content to be read single time only or multiple times depends on the host's buffer policy. By default, the host's buffer policy is set as always Buffered. In this case, you will be able to reset the position back to 0. However, if you explicitly make the policy to be Streamed, then you cannot reset back to 0.
What about using ActionFilterAtrributes?
this code worked well for me
public HttpResponseMessage AddEditCheck(Check check)
{
var request= ((System.Web.HttpContextWrapper)Request.Properties.ToList<KeyValuePair<string, object>>().First().Value).Request;
var i = request.Form["txtCheckDate"];
return Request.CreateResponse(HttpStatusCode.Ok);
}

Is it possible to remove a Pragma no-cache response header once it has been set by SetCacheability method?

I have an MVC4 GET action method that returns a FileStreamResult. A requirement exists to only use SSL and to not allow caching of the served document so SSL it is and I've also used the OutputCache filter with the following properties:
[OutputCache(NoStore = true, Duration = 0, VaryByParam = "None", Location = OutputCacheLocation.None)]
This behaves as expected and produces the following response headers:
Cache-Control: no-cache, no-store
Expires: -1
Pragma: no-cache
All was well until asked to also support IE8 and as many here have also encountered the documents just won't download with both no-cache set and SSL in the mix. The workaround for IE8 and below is to add some registry setting which is not really viable, or to remove the no-cache headers which breaks a fundamental requirement.
I experimented with Fiddler and IE8 and was able to download a document if I just removed the pragma: no-cache header but left the Cache-Control header intact. This didn't appear to leave a copy of the document in my temporary internet files but I might need to test this some more.
With this information in mind I thought it might be a simple task to remove the pragma using a filter on the action but it seems no matter what I do I cannot change whatever the OutputCache is going to set. I've even removed the OutputCache attribute and used:
Response.Cache.SetCacheability(HttpCacheability.NoCache)
Using this method alone ensures I get the same cache settings as before but they are not set at the point of this method call. This merely sets up the cache policy which gets applied at some point in the response pipeline but I just don't know where.
Does anyone know if there is a way of hooking into the response pipeline to alter the cache headers as they are being written?
EDIT
I've added a simple custom IHttpModule into the pipeline that looks for and removes any pragma header in the response NameValueCollection and whilst the cache-control is set the pragma is not there. Does this mean that IIS 7.5 is inserting the pragma itself based upon what it sees in the cache-control perhaps? I know for sure I have not set anything beyond defaults for a simple web site.
EDIT
Checked the Cache-Control header value within the module and it is set private so the cache headers haven't been applied to the response yet. So it would appear the cache headers get added after modules are executed perhaps?
I was troubleshooting this same issue and ran into the same issue removing the pragma header. When .NET renders a Page object, it outputs the cache headers. The cache handling is controlled by an HttpModule. I've tried several ways to remove the pragma header, but to no avail.
One method I haven't tried yet that looks like it might work, but also looks like a PITA is to implement a filter on the Response output stream via Response.Filter = new MyCustomFilter(...).
Prior to this I tried checking the headers in various locations, but the output cache processing had not been executed yet and pragma header did not exist and so could not be removed. Notably the HttpApplication event PreSendRequestHeaders did not work.
Some other options include implementing your own OutputCache module instead of using the built-in framework version, or somehow overriding the System.Web.HttpCachePolicy class where the pragma header is rendered.
The pragma header is rendered as part of the HttpCacheability.NoCache option:
if (httpCacheability == HttpCacheability.NoCache || httpCacheability == HttpCacheability.Server)
{
if (HttpCachePolicy.s_headerPragmaNoCache == null)
HttpCachePolicy.s_headerPragmaNoCache = new HttpResponseHeader(4, "no-cache");
this._headerPragma = HttpCachePolicy.s_headerPragmaNoCache;
if (this._allowInHistory != 1)
{
if (HttpCachePolicy.s_headerExpiresMinus1 == null)
HttpCachePolicy.s_headerExpiresMinus1 = new HttpResponseHeader(18, "-1");
this._headerExpires = HttpCachePolicy.s_headerExpiresMinus1;
}
}
The only pragmatic option I've found is to set the cache-control to private and also set a short expiration for the URL. It doesn't address the root cause on either end, but it does end up with almost the same desired effect.

Upload file to Solr with HttpClient and MultipartEntity

httpclient, httpmime 4.1.3
I am trying to upload a file through http to a remote server with no success.
Here's my code:
HttpPost method;
method = new HttpPost(solrUrl + "/extract");
method.getParams().setParameter("literal.id", fileId);
method.getParams().setBooleanParameter("commit", true);
MultipartEntity me = new MultipartEntity();
me.addPart("myfile", new InputStreamBody(doubleInput, contentType, fileId));
method.setEntity(me);
//method.setHeader("Content-Type", "multipart/form-data");
HttpClient httpClient = new DefaultHttpClient();
HttpResponse hr = httpClient.execute(method);
The server is Solr.
This is to replace a working bash script that calls curl like this,
curl http://localhost:8080/solr/update/extract?literal.id=bububu&commit=true -F myfile=#bububu.doc
If I try to set "Content-Type" "multipart/form-data", the receiving part says that there's no boundary (which is true):
HTTP Status 500 - the request was rejected because no multipart boundary was found
If I omit this header setting, the server issues an error description that, as far as I discovered, indicates that the content type was not multipart [2]:
HTTP Status 400. The request sent by the client was syntactically incorrect ([doc=null] missing required field: id).
This is related to [1] but I couldn't determine the answer from it. I was wondering,
I am in the same situation but didn't understand what to do. I was hoping that the MultipartEntity would tell the HttpPost object that it is multipart, form data and have some boundary, and I wouldnt set content type by myself. I didn't quite get how to provide boundaries to the entities - the MultipartEntity doesn't have a method like setBoundary. Or, how to get that randomly generated boundary to specify it in addHeader by myself - no getBoundary methor either...
[1] Problem with setting header "Content-Type" in uploading file with HttpClient4
[2] http://lucene.472066.n3.nabble.com/Updating-the-index-with-a-csv-file-td490013.html
I am suspicious of
method.getParams().setParameter("literal.id", fileId);
method.getParams().setBooleanParameter("commit", true);
In the first line, is fileId a string or file pointer (or something else)? I hope it is a string. As for the second line, you can rather set a normal parameter.
I am trying to tackle the HTTP Status 400. I dont know much Java (or is that .Net?)
http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_Client_Error

Implementing ETag Support in ASP.NET MVC4 WebAPI

In the latest ASP.NET MVC4 beta, how would you support conditional GET support via ETags? The ActionFilter would need to be able to complete the request to generate the ETag for the returned resource in order to compare to the If-None-Match header in the request. And then, regardless of whether the incoming ETag in the If-None-Match header was the same as the generated ETag, add the generated ETag to the ETag response header. But with ASP.NET MVC4, I have no idea where to begin. Any suggestions?
Personally, I'm not a fan of "framework magic" and prefer plain old code in the web methods, else we end up with something more akin to WCF, yuk.
So, within your Get web method, manually create the response like so:
var response = this.Request.CreateResponse(HttpStatusCode.OK, obj);
string hash = obj.ModifiedDate.GetHashCode().ToString();
response.Headers.ETag =
new EntityTagHeaderValue(String.Concat("\"", hash, "\""), true);
return response;
Please note that the ETag produced from the hash code of the timestamp is purely illustrative of a weak entity tagging system. It also shows the additional quotes required.
There is a ETagMessageHandler in the WebApiContrib which does what you need.
UPDATE
I have implemented RFC 2616's server side caching in WebApiContrib. Look for CachingHandler.
More info here.
More Update
This will be actively developed and expanded upon under CacheCow. This will include both client and server components. NuGet packages to be published soon are published now.
WebApiContrib's CachingHandler will still be maintained so any bugs or problems please let me know.
Luke Puplett's answer got me on the right track (+1), but note that you also have to read the ETag on the server side to avoid sending all the data with each request:
string hash = obj.ModifiedDate.GetHashCode().ToString();
var etag = new EntityTagHeaderValue(String.Concat("\"", hash, "\""), true);
if (Request.Headers.IfNoneMatch.Any(h => h.Equals(etag)))
{
return new HttpResponseMessage(HttpStatusCode.NotModified);
}
var response = this.Request.CreateResponse(HttpStatusCode.OK, obj);
response.Headers.ETag = etag;
return response;
It would also be a good idea to respect the If-Modified-Since header. See RFC 2616.
It seems this is what you are looking for (see section "Support for ETags"):
http://blogs.msdn.com/b/webdev/archive/2014/03/13/getting-started-with-asp-net-web-api-2-2-for-odata-v4-0.aspx
In case your model is stored deeper in domain and you are not able to apply the [ConcurrencyCheck] attribute, you can do that using the ODataModelBuilder:
ODataModelBuilder builder = new ODataConventionModelBuilder();
var myEntity = builder.EntitySet<MyEntity>("MyEntities");
myEntity.EntityType.Property(l => l.Version).ConcurrencyToken = true;
this will make it to add the "#odata.etag" property to a response body.