We have a WCF REST service hosted on IIS 7 with .NET Framework 4.5. The client is sending data in GZip compressed format with request headers:
Content-Encoding:gzip
Content-Type: application/xml
But we are getting bad request from the server, if the request is in compressed format. We enabled Request compression by implementation of IHttpModule that will filter/modify incoming requests. From my understanding, this is failing because WCF uses original content length (that of compressed data) instead of Decompressed data. So here are my questions:
Is there any way we can fix this content length issues in IIS7/.NET 4.5? My HTTP module implementation is given below:
httpApplication.Request.Filter = New GZipStream(httpApplication.Request.Filter, CompressionMode.Decompress)`
If fixing the content length issue is not possible at server side, is there any way I can send original content length from client with a compressed request? Client side implementation is as follows:
using (Stream requeststream = serviceRequest.GetRequestStream())
{
if (useCompression)
{
using (GZipStream zipStream = new GZipStream(requeststream, CompressionMode.Compress))
{
zipStream.Write(bytes, 0, bytes.Length);
zipStream.Close();
requeststream.Close();
}
serviceRequest.Headers.Add("Content-Encoding", "gzip");
}
else
{
requeststream.Write(bytes, 0, bytes.Length);
requeststream.Close();
}
}
I was able to get gzip working in WCF using wsHTTPBinding and this as the base of the web request:
private HttpWebRequest GetWebRequest()
{
dynamic objHTTPReq = (HttpWebRequest)WebRequest.CreateDefault(_URI);
objHTTPReq.ContentType = "text/xml; charset=\"utf-8\"";
objHTTPReq.Method = "POST";
objHTTPReq.Accept = "gzip, deflate";
objHTTPReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; OfficeLiveConnector.1.3;OfficeLivePatch.0.0; Zune 3.0; MS-RTC LM 8)";
objHTTPReq.Headers.Add("SOAPAction", "http://xxx.yyyy.zzzz");
return objHTTPReq;
}
So give that a try. Good luck.
Related
Regardless of whether I use WebClient or HttpWebRequest, loading this page times out. What am I doing wrong? It can't be https, since other https sites load just fine.
Below is my latest attempt, which adds all headers that I see in Firefox's inspector.
One interesting behavior is that I cannot monitor this with Fiddler, because everything works properly when Fiddler is running.
Using client As WebClient = New WebClient()
client.Headers(HttpRequestHeader.Accept) = "text/html, image/png, image/jpeg, image/gif, */*;q=0.1"
client.Headers(HttpRequestHeader.UserAgent) = "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
client.Headers(HttpRequestHeader.AcceptLanguage) = "en-US;en;q=0.5"
client.Headers(HttpRequestHeader.AcceptEncoding) = "gzip, deflate, br"
client.Headers(HttpRequestHeader.Referer) = "http://www.torontohydro.com/sites/electricsystem/Pages/foryourhome.aspx"
client.Headers("DNT") = "1"
client.Headers(HttpRequestHeader.KeepAlive) = "keep-alive"
client.Headers(HttpRequestHeader.Upgrade) = "1"
client.Headers(HttpRequestHeader.CacheControl) = "max-age=0"
Dim x = New Uri("https://css.torontohydro.com/")
Dim data as string = client.DownloadString(x)
End Using
All of this is excess code. Boiling it down to just a couple of lines causes the same hang.
Using client as WebClient = New WebClient()
Dim data as string = client.DownloadString("https://css.torontohydro.com")
End Using
And this is the HttpWebRequest code, in a nutshell, which also hangs getting the response.
Dim getRequest As HttpWebRequest = CreateWebRequest("https://css.torontohydro.com/")
getRequest.CachePolicy = New Cache.RequestCachePolicy(Cache.RequestCacheLevel.BypassCache)
Using webResponse As HttpWebResponse = CType(getRequest.GetResponse(), HttpWebResponse)
'no need for any more code, since the above line is where things hang
So this ended up being due to the project still being in .NET 3.5. .NET was trying to load the site, being https, using SSL. Adding this line fixed the problem:
ServicePointManager.SecurityProtocol = 3072
I had to use 3072 since 3.5 does not contain a definition for SecurityProtocolType.Tls12.
I am having difficulties interacting with a website which use basic authentication to authenticate the user.
I am working on visual basic and i have already tried to use
Dim req As HttpWebRequest = HttpWebRequest.Create("http://url.to.website.com")
adding the headers directly to the web request:
req.Headers.Add("Authorization: Basic " & Convert.ToBase64String(Encoding.Default.GetBytes("user" & ":" & "password")))
or using the network credentials:
req.Credentials = New Net.NetworkCredential("user", "password")
receiving always the same response code: 401 Unauthorized
Using Firefox developer tools i can analyze and resend some web requests and only using Firefox i am able to authenticate correctly.
Firefox report these headers:
Host: url.to.website.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http.//url.to.website.com/portal/data/pub
DNT: 1
Authorization: Basic ZmFrZTpwYXNzd29yZA==
Connection: keep-alive
So i have tried to set it manaually this way:
req.Host = "url.to.website.com"
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:11.0) Gecko/20100101 Firefox/11.0"
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
req.Referer = "https://url.to.website.com/some/path/to/file.jsf"
req.ContentType = "application/x-www-form-urlencoded"
req.KeepAlive = True
req.PreAuthenticate = True
req.Method = "POST"
req.Headers.Add("Authorization: Basic " & Convert.ToBase64String(Encoding.Default.GetBytes("user" & ":" & "password")))
with no success (receiving always the same response code: 401 Unauthorized)
Another try was with a web-browser:
WebBrowser1.Navigate("url", Nothing, Nothing, "Authorization: Basic " & Convert.ToBase64String(Encoding.Default.GetBytes(AUTH_USER & ":" & AUTH_PASSWORD)))
My objective is to authenticate, then query some pages and collect responses in order to parse them and use it later in the application.
How can i solve the issue about authentication?
The website is written using JSF and i have no control over it.
Update:
My problem is about authentication, not yet about the jsf application.
While using Firefox all work fine (I can send a request to the website and it will authenticate me right) but while using the HttpWebRequest the authentication fails, even if I set the same headers, as Written before .
I have to figure out the difference between the two requests
I had to get this working for Dukes Forest Java EE Tutorial Port to Wildfly. The code was already written, but the header was case sensitive. Anyway, the code used there is as follows:
/* Client filter for basic HTTP auth */
class AuthClientRequestFilter implements ClientRequestFilter {
private final String user;
private final String password;
public AuthClientRequestFilter(String user, String password) {
this.user = user;
this.password = password;
}
#Override
public void filter(ClientRequestContext requestContext) throws IOException {
try {
requestContext.getHeaders().add(
"Authorization",
"Basic " + DatatypeConverter.printBase64Binary(
(user+":"+password).getBytes("UTF-8"))
);
} catch (UnsupportedEncodingException ex) { }
}
}
The DatatypeConverter is imported from javax.xml.bind. This code was called from the following routine, which has the HTTPClient:
Client client = ClientBuilder.newClient();
client.register(new AuthClientRequestFilter("jack#example.com", "1234"));
Response resp = client.target(ENDPOINT)
.request(MediaType.APPLICATION_XML)
.post(Entity.entity(order, MediaType.APPLICATION_XML), Response.class);
int status = resp.getStatus();
if (status == 200) {
success = true;
}
logger.log(Level.INFO, "[PaymentHandler] Response status {0}", status);
client.close();
return success;
This client code posts to a RESTful service.
can I set a custom User Agent for a WebView?
I need to show mobile style of websites.
It's easy to do:
string ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)" + "AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25";
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, new Uri(url));
httpRequestMessage.Headers.Add("User-Agent",ua);
webView1.NavigateWithHttpRequestMessage(hrm);
Per this MSDN Forum posting you cannot. Could you host a lightweight proxy service (say Azure Web Site) to proxy the request for you?
You can load HTML with custom user agent and then pass the html to WebView
Loading html
var handler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("user-agent",
"Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2;
WOW64; Trident/6.0)");
var response = await client.GetAsync(url);
response.EnsureSuccessStatusCode();
var html = await response.Content.ReadAsStringAsync();
assign html to WebView
WebView.NavigateToString(html);
I have a BasicAuthenticationAttribute that inspects the Authorisation header in the request but despite it being present, it still believes the Authorisation header is null:
public class BasicAuthenticationAttribute : ActionFilterAttribute
{
public override void OnActionExecuting(HttpActionContext actionContext)
{
if (actionContext.Request.Headers.Authorization == null)
{
actionContext.Response = new HttpResponseMessage(HttpStatusCode.Unauthorized);
}
...
If I inspect actionContext.Request.Headers I can see Authorization listed:
{Connection: Keep-Alive
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Authorization: REDACTED_BUT_PRESENT==
Host: localhost:44300
Referer: https://localhost:44300/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E)
}
Update
I have just inspected the full request headers and they look like this... I can see an Authorization header in the first section, but the Authorization header in the second section is clearly null.
request.Headers
{Connection: Keep-Alive
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Authorization: REDACTED_BUT_PRESENT==
Host: localhost:1734
Referer: http://localhost:1734/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E)
}
base {System.Net.Http.Headers.HttpHeaders}: {Connection: Keep-Alive
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Authorization: VXNlcjpQYXNzd29yZA==
Host: localhost:1734
Referer: http://localhost:1734/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E)
}
Accept: {*/*}
AcceptCharset: {}
AcceptEncoding: {gzip, deflate}
AcceptLanguage: {en-gb}
Authorization: null
CacheControl: null
... removed for brevity ...
Warning: {}
If you get stuck on this, you can get the header using:
var header = request.Headers.FirstOrDefault(h => h.Key.Equals("Authorization"));
But not via
var header = request.Headers.Authorization;
I noticed myself that if the Authorization-header only contained the key/token, the request.Headers.Authorization wouldn't be initiated properly because it's looking for a scheme as well in the format <Scheme> <key/token>, i.e. Authorization: Token VXNlcjpQYXNzd29yZA==, then the Authorization wouldn't be null anymore and contain request.Headers.Authorization.Scheme = "Token" and request.Headers.Authorization.Parameter = "VXNlcjpQYXNzd29yZA=="
I've posted my own example of a Basic Authentication Attribute. Maybe this gives you some hints.
I use:
HttpContext.Current.Request.Headers["Authorization"];
And here is the link to the complete solution:
http://remy.supertext.ch/2012/04/basic-http-authorization-for-web-api-in-mvc-4-beta/
Though, this thread is very old but it might help others if I share how did I resolve it in my case:
Request should contain
Authorization: Basic VXNlcjpQYXNzd29yZA==
instead of:
Authorization: VXNlcjpQYXNzd29yZA==
so following change in request may solve the problem:
client.Headers.Add("Authorization", "Basic VXNlcjpQYXNzd29yZA==");
Adding more information to #finstas's answer.
Authorization is null because well defined HTTP headers like Accept, Authorization and many more are parsed when creating the HttpRequestHeaders class. Hence if the request comes in with a format different from what .NET accepts for that header then that specific property will be null.
Below is the decompiled code from the AuthenticationHeaderValue class responsible for parsing the Authorization header. Similarly there are other classes for the different HTTP headers which do the same.
Hope this sheds more info into why there needs to be a space between Token and the value.
internal static int GetAuthenticationLength(string input, int startIndex, out object parsedValue)
{
parsedValue = (object) null;
if (string.IsNullOrEmpty(input) || startIndex >= input.Length)
return 0;
int tokenLength = HttpRuleParser.GetTokenLength(input, startIndex);
if (tokenLength == 0)
return 0;
AuthenticationHeaderValue authenticationHeaderValue = new AuthenticationHeaderValue();
authenticationHeaderValue.scheme = input.Substring(startIndex, tokenLength);
int startIndex1 = startIndex + tokenLength;
int whitespaceLength = HttpRuleParser.GetWhitespaceLength(input, startIndex1);
int index = startIndex1 + whitespaceLength;
if (index == input.Length || (int) input[index] == 44)
{
parsedValue = (object) authenticationHeaderValue;
return index - startIndex;
}
if (whitespaceLength == 0)
return 0;
int startIndex2 = index;
int parameterEndIndex = index;
if (!AuthenticationHeaderValue.TrySkipFirstBlob(input, ref index, ref parameterEndIndex) || index < input.Length && !AuthenticationHeaderValue.TryGetParametersEndIndex(input, ref index, ref parameterEndIndex))
return 0;
authenticationHeaderValue.parameter = input.Substring(startIndex2, parameterEndIndex - startIndex2 + 1);
parsedValue = (object) authenticationHeaderValue;
return index - startIndex;
}
I've never tried before, but now I really need to get through authorization on Sprint's site (www.sprint.com).
Could you guys help me to understand how this actually works?
I'm trying to do like this, but obviously I'm missing something. Either something about cookies
or ssl or other stuff, I don't know.
HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(
"https://sso.sprintpcs.com/sso/Login.do");
CookieContainer cookieContainer = new CookieContainer();
webRequest.CookieContainer = cookieContainer;
webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0;
chromeframe; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729; Tablet PC 2.0; .NET4.0C; .NET4.0E)";
webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml,
image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash,
application/vnd.ms-excel, application/msword, */*";
webRequest.Method = "POST";
webRequest.Host = "manage.sprintpcs.com";
string strUserId = "kindauser";
string strPass = "kindapass";
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "userid=" + strUserId + "&password="
+ strPass + "&userExperince=USC allowlogin=false";
byte[] data = encoding.GetBytes(postData);
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write(data,0,data.Length);
HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse();
I would do the following - and this applies to all cases where you want to interact wit a website.
1) get firefox, along with firebug extension.
2) clear the firefox content and cookie cache
3) use firefox to do the scenario - logging into the website, for eg.
4) At this point, firebug shows you the exact sequence of requests sent along withh the cookie headers, etc.
5) Now try to replicate this using code.