I want to add telemetry either as a log or to Application insights for the Content-Length of a response in dotnet core 2.2. I've tried a number of places in middleware and different filters in the request pipeline. No where that I've checked has the Content-Length materialized--it's always null.
My alternate solution was to check the response stream myself and compute the length but I'd really prefer not to re-read the stream if I don't have to. Is there somewhere in the dotnet core request pipeline that I can hook into for that information?
You will still have to implement custom middleware for this. Here is an example:
public class CustomMiddleware
{
private readonly RequestDelegate _next;
public CustomMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task InvokeAsync(HttpContext context)
{
// store original response and replace it with another
var originalResponse = context.Response.Body;
await using var newResponse = new MemoryStream();
context.Response.Body = newResponse;
await _next(context);
// You can easily access Length property of the stream here
// and log it (logging skipped in the example)
var contentLength = newResponse.Length;
// setting back the original stream
newResponse.Seek(0, SeekOrigin.Begin);
await newResponse.CopyToAsync(originalResponse);
context.Response.Body = originalResponse;
}
}
You can also see this implementation, which uses buffering
Related
I'm in the process of trying to put together a small project which uses Asp.Net Core Identity, Identity Server 4 and a Web API project.
I've got my MVC project authenticating correctly with IdS4 from which I get a JWT which I can then add to the header of a request to my Web API project, this all works as expected.
The issue I have is how I'm actually adding the token to the HttpClient, basically I'm setting it up for every request which is obviously wrong otherwise I'd have seen other examples online, but I haven't been able to determine a good way to refactor this. I've read many articles and I have found very little information about this part of the flow, so I'm guessing it could be so simple that it's never detailed in guides, but I still don't know!
Here is an example MVC action that calls my API:
[HttpGet]
[Authorize]
public async Task<IActionResult> GetFromApi()
{
var client = await GetHttpClient();
string testUri = "https://localhost:44308/api/TestItems";
var response = await client.GetAsync(testUri, HttpCompletionOption.ResponseHeadersRead);
var data = await response.Content.ReadAsStringAsync();
GetFromApiViewModel vm = new GetFromApiViewModel()
{
Output = data
};
return View(vm);
}
And here is the GetHttpClient() method which I call (currently residing in the same controller):
private async Task<HttpClient> GetHttpClient()
{
var client = new HttpClient();
var expat = HttpContext.GetTokenAsync("expires_at").Result;
var dataExp = DateTime.Parse(expat, null, DateTimeStyles.RoundtripKind);
if ((dataExp - DateTime.Now).TotalMinutes < 10)
{
//SNIP GETTING A NEW TOKEN IF ITS ABOUT TO EXPIRE
}
var accessToken = await HttpContext.GetTokenAsync("access_token");
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
return client;
}
My StartUp classes are pretty standard from what I gather, but if they could be useful, then I'll add them in.
I've read many articles and I have found very little information about this part of the flow, so I'm guessing it could be so simple that it's never detailed in guides, but I still don't know!
The problem is that the docs are really spread all over, so it's hard to get a big picture of all the best practices. I'm planning a blog series on "Modern HTTP API Clients" that will collect all these best practices.
First, I recommend you use HttpClientFactory rather than new-ing up an HttpClient.
Next, adding an authorization header is IMO best done by hooking into the HttpClient's pipeline of message handlers. A basic bearer-token authentication helper could look like this:
public sealed class BackendApiAuthenticationHttpClientHandler : DelegatingHandler
{
private readonly IHttpContextAccessor _accessor;
public BackendApiAuthenticationHttpClientHandler(IHttpContextAccessor accessor)
{
_accessor = accessor;
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var expat = await _accessor.HttpContext.GetTokenAsync("expires_at");
var dataExp = DateTime.Parse(expat, null, DateTimeStyles.RoundtripKind);
if ((dataExp - DateTime.Now).TotalMinutes < 10)
{
//SNIP GETTING A NEW TOKEN IF ITS ABOUT TO EXPIRE
}
var token = await _accessor.HttpContext.GetTokenAsync("access_token");
// Use the token to make the call.
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
return await base.SendAsync(request, cancellationToken);
}
}
This can be hooked up via DI:
services.AddTransient<BackendApiAuthenticationHttpClientHandler>();
services.AddHttpClient<MyController>()
.ConfigureHttpClient((provider, c) => c.BaseAddress = new Uri("https://localhost:44308/api"))
.AddHttpMessageHandler<BackendApiAuthenticationHttpClientHandler>();
Then you can inject an HttpClient into your MyController, and it will magically use the auth tokens:
// _client is an HttpClient, initialized in the constructor
string testUri = "TestItems";
var response = await _client.GetAsync(testUri, HttpCompletionOption.ResponseHeadersRead);
var data = await response.Content.ReadAsStringAsync();
GetFromApiViewModel vm = new GetFromApiViewModel()
{
Output = data
};
return View(vm);
This pattern seems complex at first, but it separates the "how do I call this API" logic from "what is this action doing" logic. And it's easier to extend with retries / circuit breakers / etc, via Polly.
You can use HttpRequestMessage
// Create this instance once on stratup
// (preferably you want to keep an instance per base url to avoid waiting for socket fin)
HttpClient client = new HttpClient();
Then create an instance of HttpRequestMessage:
HttpRequestMessage request = new HttpRequestMessage(
HttpMethod.Get,
"https://localhost:44308/api/TestItems");
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", "ey..");
await client.SendAsync(request);
I have a middleware to track performance of my custom developed gateway in ASP.NET Core 2.2 API. I have used the this post from StackOverflow.
Basically the main part is as follows :
public class ResponseRewindMiddleware {
private readonly RequestDelegate next;
public ResponseRewindMiddleware(RequestDelegate next) {
this.next = next;
}
public async Task Invoke(HttpContext context) {
Stream originalBody = context.Response.Body;
/* MY CODE COMES HERE */
try {
using (var memStream = new MemoryStream()) {
context.Response.Body = memStream;
await next(context);
memStream.Position = 0;
string responseBody = new StreamReader(memStream).ReadToEnd();
memStream.Position = 0;
await memStream.CopyToAsync(originalBody);
}
} finally {
context.Response.Body = originalBody;
}
}
This code runs OK. But I want to log the input (a JSON body) to the gateway and I add the following lines :
using (System.IO.StreamReader rd = new System.IO.StreamReader(context.Request.Body))
{
bodyStr = rd.ReadToEnd();
}
This reads the input body from Request but the flow is broken and the rest of the process does not flow resulting in a "HTTP 500 Internal Server Error". I assume reading the Request body via a Stream breaks something.
How can I read the Request body without breaking the flow?
The idea is to call EnableBuffering to enable multiple read, and then to not dispose the request body after you have done reading it. The following works for me.
// Enable the request body to be read in the future
context.Request.EnableBuffering();
// Read the request body, but do not dispose it
var stream = new StreamReader(context.Request.Body);
string requestBody = await stream.ReadToEndAsync();
// Reset to the origin so the next read would start from the beginning
context.Request.Body.Seek(0, SeekOrigin.Begin);
Sample code below to write a file stream to Response.Body in an ASP.NET Core middleware doesn't work (emits empty response):
public Task Invoke(HttpContext context)
{
context.Response.ContentType = "text/plain";
using (var fs = new FileStream("/valid-path-to-file-on-server.txt", FileMode.Open)
using (var sr = new StreamReader(fs))
{
context.Response.Body = sr.BaseStream;
}
return Task.CompletedTask;
}
Any ideas what could be wrong with this approach of directly setting the context.Response.Body?
Note: any next middleware in the pipeline is skipped for no further processing.
Update (another example): a simple MemoryStream assignment doesn't work either (empty response):
context.Response.Body = new MemoryStream(Encoding.UTF8.GetBytes(DateTime.Now.ToString()));
No. You can never do that directly.
Note that context.Response.Body is a reference to an object (HttpResponseStream) that is initialized before it becomes available in HttpContext. It is assumed that all bytes are written into this original Stream. If you change the Body to reference (point to) a new stream object by context.Response.Body = a_new_Stream, the original Stream is not changed at all.
Also, if you look into the source code of ASP.NET Core, you'll find the Team always copy the wrapper stream to the original body stream at the end rather than with a simple replacement(unless they're unit-testing with a mocked stream). For example, the SPA Prerendering middleware source code:
finally
{
context.Response.Body = originalResponseStream;
...
And the ResponseCachingMiddleware source code:
public async Task Invoke(HttpContext httpContext)
{
...
finally
{
UnshimResponseStream(context);
}
...
}
internal static void UnshimResponseStream(ResponseCachingContext context)
{
// Unshim response stream
context.HttpContext.Response.Body = context.OriginalResponseStream;
// Remove IResponseCachingFeature
RemoveResponseCachingFeature(context.HttpContext);
}
As a walkaround, you can copy the bytes to the raw stream as below:
public async Task Invoke(HttpContext context)
{
context.Response.ContentType = "text/plain";
using (var fs = new FileStream("valid-path-to-file-on-server.txt", FileMode.Open))
{
await fs.CopyToAsync(context.Response.Body);
}
}
Or if you like to hijack the raw HttpResponseStream with your own stream wrapper:
var originalBody = HttpContext.Response.Body;
var ms = new MemoryStream();
HttpContext.Response.Body = ms;
try
{
await next();
HttpContext.Response.Body = originalBody;
ms.Seek(0, SeekOrigin.Begin);
await ms.CopyToAsync(HttpContext.Response.Body);
}
finally
{
response.Body = originalBody;
}
The using statements in the question causes your stream and stream reader to be rather ephemeral, so they will both be disposed. The extra reference to the steam in "body" wont prevent the dispose.
The framework disposes of the stream after sending the response. (The medium is the message).
In net 6 I found I was getting console errors when I tried to do this e.g.:
System.InvalidOperationException: Response Content-Length mismatch: too many bytes written (25247 of 8863).
The solution was to remove the relevant header:
context.Response.Headers.Remove("Content-Length");
await context.Response.SendFileAsync(filename);
I am using following class to log all request and responses to my API. The code is taken from link https://exceptionnotfound.net/using-middleware-to-log-requests-and-responses-in-asp-net-core/. The problem is when i register this middleware my model binding stops working. The request is always null. I think the problem is into the method "FormatRequest", if i remove the call to that method it starts working, but cannot figure out why it is disrupting model binding process.
public class RequestResponseLoggingMiddleware
{
private readonly RequestDelegate _next;
public RequestResponseLoggingMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
//First, get the incoming request
var request = await FormatRequest(context.Request);
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using (var responseBody = new MemoryStream())
{
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
await _next(context);
//Format the response from the server
var response = await FormatResponse(context.Response);
//TODO: Save log to chosen datastore
//Copy the contents of the new memory stream (which contains the response) to the original stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
}
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
request.Body = body;
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
}
This is how i am registering it,
public class Startup
{
//...
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
//Add our new middleware to the pipeline
app.UseMiddleware<RequestResponseLoggingMiddleware>();
app.UseMvc();
}
}
For this issue, you could try request.Body.Seek(0, SeekOrigin.Begin); to reset body instead of request.Body = body;
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
//request.Body = body;
request.Body.Seek(0, SeekOrigin.Begin);
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
I'm using an ASP.NET Core Azure Web App to provide a RESTful API to a client, and the client doesn't handle chunking correctly.
Is it possible to completely turn off Transfer-Encoding: chunked, either at the controller level or in file web.config?
I'm returning a JsonResult somewhat like this:
[HttpPost]
[Produces("application/json")]
public IActionResult Post([FromBody] AuthRequest RequestData)
{
AuthResult AuthResultData = new AuthResult();
return Json(AuthResultData);
}
How to get rid of chunking in .NET Core 2.2:
The trick is to read the response body into your own MemoryStream, so you can get the length. Once you do that, you can set the content-length header, and IIS won't chunk it. I assume this would work for Azure too, but I haven't tested it.
Here's the middleware:
public class DeChunkerMiddleware
{
private readonly RequestDelegate _next;
public DeChunkerMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task InvokeAsync(HttpContext context)
{
var originalBodyStream = context.Response.Body;
using (var responseBody = new MemoryStream())
{
context.Response.Body = responseBody;
long length = 0;
context.Response.OnStarting(() =>
{
context.Response.Headers.ContentLength = length;
return Task.CompletedTask;
});
await _next(context);
// If you want to read the body, uncomment these lines.
//context.Response.Body.Seek(0, SeekOrigin.Begin);
//var body = await new StreamReader(context.Response.Body).ReadToEndAsync();
length = context.Response.Body.Length;
context.Response.Body.Seek(0, SeekOrigin.Begin);
await responseBody.CopyToAsync(originalBodyStream);
}
}
}
Then add this in Startup:
app.UseMiddleware<DeChunkerMiddleware>();
It needs to be before app.UseMvC().
In ASP.NET Core, this seems to work across hosts:
response.Headers["Content-Encoding"] = "identity";
response.Headers["Transfer-Encoding"] = "identity";
Indicates the identity function (i.e., no compression, nor
modification). This token, except if explicitly specified, is always
deemed acceptable.
Content-Encoding
Transfer-Encoding
This also works when you explicitly disable response buffering:
var bufferingFeature = httpContext.Features.Get<IHttpBufferingFeature>();
bufferingFeature?.DisableResponseBuffering();
It works in .NET Core 2.0. Just set ContentLength before writing the results into the response body stream.
In the startup class:
app.Use(async (ctx, next) =>
{
var stream = new xxxResultTranslatorStream(ctx.Response.Body);
ctx.Response.Body = stream;
await Run(ctx, next);
stream.Translate(ctx);
ctx.Response.Body = stream.Stream;
});
In xxxResultTranslatorStream:
ctx.Response.Headers.ContentLength = 40;
stream.Write(writeTargetByte, 0, writeTargetByte.Length);
I found that all my chunking problems went away if I just returned a FileStream from Get() and let ASP.NET deal with the rest.
Microsoft software tends to work best if you just give up control and trust them. It tends to work worst if you actually try to control the process.