I'm using an ASP.NET Core Azure Web App to provide a RESTful API to a client, and the client doesn't handle chunking correctly.
Is it possible to completely turn off Transfer-Encoding: chunked, either at the controller level or in file web.config?
I'm returning a JsonResult somewhat like this:
[HttpPost]
[Produces("application/json")]
public IActionResult Post([FromBody] AuthRequest RequestData)
{
AuthResult AuthResultData = new AuthResult();
return Json(AuthResultData);
}
How to get rid of chunking in .NET Core 2.2:
The trick is to read the response body into your own MemoryStream, so you can get the length. Once you do that, you can set the content-length header, and IIS won't chunk it. I assume this would work for Azure too, but I haven't tested it.
Here's the middleware:
public class DeChunkerMiddleware
{
private readonly RequestDelegate _next;
public DeChunkerMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task InvokeAsync(HttpContext context)
{
var originalBodyStream = context.Response.Body;
using (var responseBody = new MemoryStream())
{
context.Response.Body = responseBody;
long length = 0;
context.Response.OnStarting(() =>
{
context.Response.Headers.ContentLength = length;
return Task.CompletedTask;
});
await _next(context);
// If you want to read the body, uncomment these lines.
//context.Response.Body.Seek(0, SeekOrigin.Begin);
//var body = await new StreamReader(context.Response.Body).ReadToEndAsync();
length = context.Response.Body.Length;
context.Response.Body.Seek(0, SeekOrigin.Begin);
await responseBody.CopyToAsync(originalBodyStream);
}
}
}
Then add this in Startup:
app.UseMiddleware<DeChunkerMiddleware>();
It needs to be before app.UseMvC().
In ASP.NET Core, this seems to work across hosts:
response.Headers["Content-Encoding"] = "identity";
response.Headers["Transfer-Encoding"] = "identity";
Indicates the identity function (i.e., no compression, nor
modification). This token, except if explicitly specified, is always
deemed acceptable.
Content-Encoding
Transfer-Encoding
This also works when you explicitly disable response buffering:
var bufferingFeature = httpContext.Features.Get<IHttpBufferingFeature>();
bufferingFeature?.DisableResponseBuffering();
It works in .NET Core 2.0. Just set ContentLength before writing the results into the response body stream.
In the startup class:
app.Use(async (ctx, next) =>
{
var stream = new xxxResultTranslatorStream(ctx.Response.Body);
ctx.Response.Body = stream;
await Run(ctx, next);
stream.Translate(ctx);
ctx.Response.Body = stream.Stream;
});
In xxxResultTranslatorStream:
ctx.Response.Headers.ContentLength = 40;
stream.Write(writeTargetByte, 0, writeTargetByte.Length);
I found that all my chunking problems went away if I just returned a FileStream from Get() and let ASP.NET deal with the rest.
Microsoft software tends to work best if you just give up control and trust them. It tends to work worst if you actually try to control the process.
Related
I want to add telemetry either as a log or to Application insights for the Content-Length of a response in dotnet core 2.2. I've tried a number of places in middleware and different filters in the request pipeline. No where that I've checked has the Content-Length materialized--it's always null.
My alternate solution was to check the response stream myself and compute the length but I'd really prefer not to re-read the stream if I don't have to. Is there somewhere in the dotnet core request pipeline that I can hook into for that information?
You will still have to implement custom middleware for this. Here is an example:
public class CustomMiddleware
{
private readonly RequestDelegate _next;
public CustomMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task InvokeAsync(HttpContext context)
{
// store original response and replace it with another
var originalResponse = context.Response.Body;
await using var newResponse = new MemoryStream();
context.Response.Body = newResponse;
await _next(context);
// You can easily access Length property of the stream here
// and log it (logging skipped in the example)
var contentLength = newResponse.Length;
// setting back the original stream
newResponse.Seek(0, SeekOrigin.Begin);
await newResponse.CopyToAsync(originalResponse);
context.Response.Body = originalResponse;
}
}
You can also see this implementation, which uses buffering
I'm in the process of trying to put together a small project which uses Asp.Net Core Identity, Identity Server 4 and a Web API project.
I've got my MVC project authenticating correctly with IdS4 from which I get a JWT which I can then add to the header of a request to my Web API project, this all works as expected.
The issue I have is how I'm actually adding the token to the HttpClient, basically I'm setting it up for every request which is obviously wrong otherwise I'd have seen other examples online, but I haven't been able to determine a good way to refactor this. I've read many articles and I have found very little information about this part of the flow, so I'm guessing it could be so simple that it's never detailed in guides, but I still don't know!
Here is an example MVC action that calls my API:
[HttpGet]
[Authorize]
public async Task<IActionResult> GetFromApi()
{
var client = await GetHttpClient();
string testUri = "https://localhost:44308/api/TestItems";
var response = await client.GetAsync(testUri, HttpCompletionOption.ResponseHeadersRead);
var data = await response.Content.ReadAsStringAsync();
GetFromApiViewModel vm = new GetFromApiViewModel()
{
Output = data
};
return View(vm);
}
And here is the GetHttpClient() method which I call (currently residing in the same controller):
private async Task<HttpClient> GetHttpClient()
{
var client = new HttpClient();
var expat = HttpContext.GetTokenAsync("expires_at").Result;
var dataExp = DateTime.Parse(expat, null, DateTimeStyles.RoundtripKind);
if ((dataExp - DateTime.Now).TotalMinutes < 10)
{
//SNIP GETTING A NEW TOKEN IF ITS ABOUT TO EXPIRE
}
var accessToken = await HttpContext.GetTokenAsync("access_token");
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
return client;
}
My StartUp classes are pretty standard from what I gather, but if they could be useful, then I'll add them in.
I've read many articles and I have found very little information about this part of the flow, so I'm guessing it could be so simple that it's never detailed in guides, but I still don't know!
The problem is that the docs are really spread all over, so it's hard to get a big picture of all the best practices. I'm planning a blog series on "Modern HTTP API Clients" that will collect all these best practices.
First, I recommend you use HttpClientFactory rather than new-ing up an HttpClient.
Next, adding an authorization header is IMO best done by hooking into the HttpClient's pipeline of message handlers. A basic bearer-token authentication helper could look like this:
public sealed class BackendApiAuthenticationHttpClientHandler : DelegatingHandler
{
private readonly IHttpContextAccessor _accessor;
public BackendApiAuthenticationHttpClientHandler(IHttpContextAccessor accessor)
{
_accessor = accessor;
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var expat = await _accessor.HttpContext.GetTokenAsync("expires_at");
var dataExp = DateTime.Parse(expat, null, DateTimeStyles.RoundtripKind);
if ((dataExp - DateTime.Now).TotalMinutes < 10)
{
//SNIP GETTING A NEW TOKEN IF ITS ABOUT TO EXPIRE
}
var token = await _accessor.HttpContext.GetTokenAsync("access_token");
// Use the token to make the call.
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
return await base.SendAsync(request, cancellationToken);
}
}
This can be hooked up via DI:
services.AddTransient<BackendApiAuthenticationHttpClientHandler>();
services.AddHttpClient<MyController>()
.ConfigureHttpClient((provider, c) => c.BaseAddress = new Uri("https://localhost:44308/api"))
.AddHttpMessageHandler<BackendApiAuthenticationHttpClientHandler>();
Then you can inject an HttpClient into your MyController, and it will magically use the auth tokens:
// _client is an HttpClient, initialized in the constructor
string testUri = "TestItems";
var response = await _client.GetAsync(testUri, HttpCompletionOption.ResponseHeadersRead);
var data = await response.Content.ReadAsStringAsync();
GetFromApiViewModel vm = new GetFromApiViewModel()
{
Output = data
};
return View(vm);
This pattern seems complex at first, but it separates the "how do I call this API" logic from "what is this action doing" logic. And it's easier to extend with retries / circuit breakers / etc, via Polly.
You can use HttpRequestMessage
// Create this instance once on stratup
// (preferably you want to keep an instance per base url to avoid waiting for socket fin)
HttpClient client = new HttpClient();
Then create an instance of HttpRequestMessage:
HttpRequestMessage request = new HttpRequestMessage(
HttpMethod.Get,
"https://localhost:44308/api/TestItems");
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", "ey..");
await client.SendAsync(request);
I have a middleware to track performance of my custom developed gateway in ASP.NET Core 2.2 API. I have used the this post from StackOverflow.
Basically the main part is as follows :
public class ResponseRewindMiddleware {
private readonly RequestDelegate next;
public ResponseRewindMiddleware(RequestDelegate next) {
this.next = next;
}
public async Task Invoke(HttpContext context) {
Stream originalBody = context.Response.Body;
/* MY CODE COMES HERE */
try {
using (var memStream = new MemoryStream()) {
context.Response.Body = memStream;
await next(context);
memStream.Position = 0;
string responseBody = new StreamReader(memStream).ReadToEnd();
memStream.Position = 0;
await memStream.CopyToAsync(originalBody);
}
} finally {
context.Response.Body = originalBody;
}
}
This code runs OK. But I want to log the input (a JSON body) to the gateway and I add the following lines :
using (System.IO.StreamReader rd = new System.IO.StreamReader(context.Request.Body))
{
bodyStr = rd.ReadToEnd();
}
This reads the input body from Request but the flow is broken and the rest of the process does not flow resulting in a "HTTP 500 Internal Server Error". I assume reading the Request body via a Stream breaks something.
How can I read the Request body without breaking the flow?
The idea is to call EnableBuffering to enable multiple read, and then to not dispose the request body after you have done reading it. The following works for me.
// Enable the request body to be read in the future
context.Request.EnableBuffering();
// Read the request body, but do not dispose it
var stream = new StreamReader(context.Request.Body);
string requestBody = await stream.ReadToEndAsync();
// Reset to the origin so the next read would start from the beginning
context.Request.Body.Seek(0, SeekOrigin.Begin);
I want to write Custom Middleware in my ASP.NET Core 1.0 project which will replace original framework's Http Response Stream to my own, so I will be able to perform read / seek / write operations on it (first 2 are not possible on the original stream) in the further code i.e. in Actions or Filters.
I've started with the following code:
public class ReplaceStreamMiddleware
{
protected RequestDelegate NextMiddleware;
public ReplaceStreamMiddleware(RequestDelegate next)
{
NextMiddleware = next;
}
public async Task Invoke(HttpContext httpContext)
{
using (var responseStream = new MemoryStream())
{
var fullResponse = httpContext.Response.Body;
httpContext.Response.Body = responseStream;
await NextMiddleware.Invoke(httpContext);
responseStream.Seek(0, SeekOrigin.Begin);
await responseStream.CopyToAsync(fullResponse);
}
}
}
The problem with the following code is that sometimes the fullResponse stream is already closed at the time of invoking await responseStream.CopyToAsync(fullResponse); so it throws an exception Cannot access a closed Stream.
This weird behaviour is easy to observe when I load the page in the browser and then refresh, before it loads completely.
I would like to know:
why this happens?
how to prevent it?
is my solution a good idea or there is another way to replace response stream?
The exception doesn't come from your CopyToAsync. It's from one of your code's callers:
You're not restoring the original response stream in HttpContext. Therefore, whoever calls your middleware will get back a closed MemoryStream.
Here's some working code:
app.Use(async (httpContext, next) =>
{
using (var memoryResponse = new MemoryStream())
{
var originalResponse = httpContext.Response.Body;
try
{
httpContext.Response.Body = memoryResponse;
await next.Invoke();
memoryResponse.Seek(0, SeekOrigin.Begin);
await memoryResponse.CopyToAsync(originalResponse);
}
finally
{
// This is what you're missing
httpContext.Response.Body = originalResponse;
}
}
});
app.Run(async (context) =>
{
context.Response.ContentType = "text/other";
await context.Response.WriteAsync("Hello World!");
});
I am using some custom compression middleware from this repository (pasted below). Upon the first request, the content is compressed just fine. For every request after that, the response comes back as completely empty (with a Content-Length of 0).
This only started happening after migrating from ASP.NET Core RC2 to RTM.
Does anyone know why this is happening?
CompressionMiddleware:
public class CompressionMiddleware
{
private readonly RequestDelegate _next;
public CompressionMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
if (IsGZipSupported(context))
{
string acceptEncoding = context.Request.Headers["Accept-Encoding"];
var buffer = new MemoryStream();
var stream = context.Response.Body;
context.Response.Body = buffer;
await _next(context);
if (acceptEncoding.Contains("gzip"))
{
var gstream = new GZipStream(stream, CompressionLevel.Optimal);
context.Response.Headers.Add("Content-Encoding", new[] { "gzip" });
buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(gstream);
gstream.Dispose();
}
else
{
var gstream = new DeflateStream(stream, CompressionLevel.Optimal);
context.Response.Headers.Add("Content-Encoding", new[] { "deflate" });
buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(gstream);
gstream.Dispose();
}
}
else
{
await _next(context);
}
}
public bool IsGZipSupported(HttpContext context)
{
string acceptEncoding = context.Request.Headers["Accept-Encoding"];
return !string.IsNullOrEmpty(acceptEncoding) &&
(acceptEncoding.Contains("gzip") || acceptEncoding.Contains("deflate"));
}
}
I have found the following in "Add HTTP compression middleware" issue:
I have added gzip and it worked, but first request. I mean in the first request, the response page is null (context.Response.Body) but when you refresh the page (just once) it works correctly after that.(I don't know why but I have to solve it)
And response on question is:
You need to update
context.Response.Headers["Content-Length"] with actual compressed
buffer length.
CompressionMiddleware.cs
And above link to realisation of compression middleware contains:
if (context.Response.Headers["Content-Length"].Count > 0)
{
context.Response.Headers["Content-Length"] = compressed.Length.ToString();
}