Model binding stopped working when added custom middleware for request logging - asp.net-core

I am using following class to log all request and responses to my API. The code is taken from link https://exceptionnotfound.net/using-middleware-to-log-requests-and-responses-in-asp-net-core/. The problem is when i register this middleware my model binding stops working. The request is always null. I think the problem is into the method "FormatRequest", if i remove the call to that method it starts working, but cannot figure out why it is disrupting model binding process.
public class RequestResponseLoggingMiddleware
{
private readonly RequestDelegate _next;
public RequestResponseLoggingMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
//First, get the incoming request
var request = await FormatRequest(context.Request);
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using (var responseBody = new MemoryStream())
{
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
await _next(context);
//Format the response from the server
var response = await FormatResponse(context.Response);
//TODO: Save log to chosen datastore
//Copy the contents of the new memory stream (which contains the response) to the original stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
}
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
request.Body = body;
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
}
This is how i am registering it,
public class Startup
{
//...
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
//Add our new middleware to the pipeline
app.UseMiddleware<RequestResponseLoggingMiddleware>();
app.UseMvc();
}
}

For this issue, you could try request.Body.Seek(0, SeekOrigin.Begin); to reset body instead of request.Body = body;
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
//request.Body = body;
request.Body.Seek(0, SeekOrigin.Begin);
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}

Related

Centralized place to access response Context-Length for telemetry

I want to add telemetry either as a log or to Application insights for the Content-Length of a response in dotnet core 2.2. I've tried a number of places in middleware and different filters in the request pipeline. No where that I've checked has the Content-Length materialized--it's always null.
My alternate solution was to check the response stream myself and compute the length but I'd really prefer not to re-read the stream if I don't have to. Is there somewhere in the dotnet core request pipeline that I can hook into for that information?
You will still have to implement custom middleware for this. Here is an example:
public class CustomMiddleware
{
private readonly RequestDelegate _next;
public CustomMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task InvokeAsync(HttpContext context)
{
// store original response and replace it with another
var originalResponse = context.Response.Body;
await using var newResponse = new MemoryStream();
context.Response.Body = newResponse;
await _next(context);
// You can easily access Length property of the stream here
// and log it (logging skipped in the example)
var contentLength = newResponse.Length;
// setting back the original stream
newResponse.Seek(0, SeekOrigin.Begin);
await newResponse.CopyToAsync(originalResponse);
context.Response.Body = originalResponse;
}
}
You can also see this implementation, which uses buffering

Reading RequestBody distrupts flow in ASP.NET Core 2.2 gateway

I have a middleware to track performance of my custom developed gateway in ASP.NET Core 2.2 API. I have used the this post from StackOverflow.
Basically the main part is as follows :
public class ResponseRewindMiddleware {
private readonly RequestDelegate next;
public ResponseRewindMiddleware(RequestDelegate next) {
this.next = next;
}
public async Task Invoke(HttpContext context) {
Stream originalBody = context.Response.Body;
/* MY CODE COMES HERE */
try {
using (var memStream = new MemoryStream()) {
context.Response.Body = memStream;
await next(context);
memStream.Position = 0;
string responseBody = new StreamReader(memStream).ReadToEnd();
memStream.Position = 0;
await memStream.CopyToAsync(originalBody);
}
} finally {
context.Response.Body = originalBody;
}
}
This code runs OK. But I want to log the input (a JSON body) to the gateway and I add the following lines :
using (System.IO.StreamReader rd = new System.IO.StreamReader(context.Request.Body))
{
bodyStr = rd.ReadToEnd();
}
This reads the input body from Request but the flow is broken and the rest of the process does not flow resulting in a "HTTP 500 Internal Server Error". I assume reading the Request body via a Stream breaks something.
How can I read the Request body without breaking the flow?
The idea is to call EnableBuffering to enable multiple read, and then to not dispose the request body after you have done reading it. The following works for me.
// Enable the request body to be read in the future
context.Request.EnableBuffering();
// Read the request body, but do not dispose it
var stream = new StreamReader(context.Request.Body);
string requestBody = await stream.ReadToEndAsync();
// Reset to the origin so the next read would start from the beginning
context.Request.Body.Seek(0, SeekOrigin.Begin);

How to directly set response body to a file stream in ASP.NET Core middleware?

Sample code below to write a file stream to Response.Body in an ASP.NET Core middleware doesn't work (emits empty response):
public Task Invoke(HttpContext context)
{
context.Response.ContentType = "text/plain";
using (var fs = new FileStream("/valid-path-to-file-on-server.txt", FileMode.Open)
using (var sr = new StreamReader(fs))
{
context.Response.Body = sr.BaseStream;
}
return Task.CompletedTask;
}
Any ideas what could be wrong with this approach of directly setting the context.Response.Body?
Note: any next middleware in the pipeline is skipped for no further processing.
Update (another example): a simple MemoryStream assignment doesn't work either (empty response):
context.Response.Body = new MemoryStream(Encoding.UTF8.GetBytes(DateTime.Now.ToString()));
No. You can never do that directly.
Note that context.Response.Body is a reference to an object (HttpResponseStream) that is initialized before it becomes available in HttpContext. It is assumed that all bytes are written into this original Stream. If you change the Body to reference (point to) a new stream object by context.Response.Body = a_new_Stream, the original Stream is not changed at all.
Also, if you look into the source code of ASP.NET Core, you'll find the Team always copy the wrapper stream to the original body stream at the end rather than with a simple replacement(unless they're unit-testing with a mocked stream). For example, the SPA Prerendering middleware source code:
finally
{
context.Response.Body = originalResponseStream;
...
And the ResponseCachingMiddleware source code:
public async Task Invoke(HttpContext httpContext)
{
...
finally
{
UnshimResponseStream(context);
}
...
}
internal static void UnshimResponseStream(ResponseCachingContext context)
{
// Unshim response stream
context.HttpContext.Response.Body = context.OriginalResponseStream;
// Remove IResponseCachingFeature
RemoveResponseCachingFeature(context.HttpContext);
}
As a walkaround, you can copy the bytes to the raw stream as below:
public async Task Invoke(HttpContext context)
{
context.Response.ContentType = "text/plain";
using (var fs = new FileStream("valid-path-to-file-on-server.txt", FileMode.Open))
{
await fs.CopyToAsync(context.Response.Body);
}
}
Or if you like to hijack the raw HttpResponseStream with your own stream wrapper:
var originalBody = HttpContext.Response.Body;
var ms = new MemoryStream();
HttpContext.Response.Body = ms;
try
{
await next();
HttpContext.Response.Body = originalBody;
ms.Seek(0, SeekOrigin.Begin);
await ms.CopyToAsync(HttpContext.Response.Body);
}
finally
{
response.Body = originalBody;
}
The using statements in the question causes your stream and stream reader to be rather ephemeral, so they will both be disposed. The extra reference to the steam in "body" wont prevent the dispose.
The framework disposes of the stream after sending the response. (The medium is the message).
In net 6 I found I was getting console errors when I tried to do this e.g.:
System.InvalidOperationException: Response Content-Length mismatch: too many bytes written (25247 of 8863).
The solution was to remove the relevant header:
context.Response.Headers.Remove("Content-Length");
await context.Response.SendFileAsync(filename);

Modify middleware response

My requirement: write a middleware that filters all "bad words" out of a response that comes from another subsequent middleware (e.g. Mvc).
The problem: streaming of the response. So when we come back to our FilterBadWordsMiddleware from a subsequent middleware, which already wrote to the response, we are too late to the party... because response started already sending, which yields to the wellknown error response has already started...
So since this is a requirement in many various situations -- how to deal with it?
Replace a response stream to MemoryStream to prevent its sending. Return the original stream after the response is modified:
public async Task Invoke(HttpContext context)
{
bool modifyResponse = true;
Stream originBody = null;
if (modifyResponse)
{
//uncomment this line only if you need to read context.Request.Body stream
//context.Request.EnableRewind();
originBody = ReplaceBody(context.Response);
}
await _next(context);
if (modifyResponse)
{
//as we replaced the Response.Body with a MemoryStream instance before,
//here we can read/write Response.Body
//containing the data written by middlewares down the pipeline
//finally, write modified data to originBody and set it back as Response.Body value
ReturnBody(context.Response, originBody);
}
}
private Stream ReplaceBody(HttpResponse response)
{
var originBody = response.Body;
response.Body = new MemoryStream();
return originBody;
}
private void ReturnBody(HttpResponse response, Stream originBody)
{
response.Body.Seek(0, SeekOrigin.Begin);
response.Body.CopyTo(originBody);
response.Body = originBody;
}
It's a workaround and it can cause performance problems. I hope to see a better solution here.
A simpler version based on the code I used:
/// <summary>
/// The middleware Invoke method.
/// </summary>
/// <param name="httpContext">The current <see cref="HttpContext"/>.</param>
/// <returns>A Task to support async calls.</returns>
public async Task Invoke(HttpContext httpContext)
{
var originBody = httpContext.Response.Body;
try
{
var memStream = new MemoryStream();
httpContext.Response.Body = memStream;
await _next(httpContext).ConfigureAwait(false);
memStream.Position = 0;
var responseBody = new StreamReader(memStream).ReadToEnd();
//Custom logic to modify response
responseBody = responseBody.Replace("hello", "hi", StringComparison.InvariantCultureIgnoreCase);
var memoryStreamModified = new MemoryStream();
var sw = new StreamWriter(memoryStreamModified);
sw.Write(responseBody);
sw.Flush();
memoryStreamModified.Position = 0;
await memoryStreamModified.CopyToAsync(originBody).ConfigureAwait(false);
}
finally
{
httpContext.Response.Body = originBody;
}
}
Unfortunately I'm not allowed to comment since my score is too low.
So just wanted to post my extension of the excellent top solution, and a modification for .NET Core 3.0+
First of all
context.Request.EnableRewind();
has been changed to
context.Request.EnableBuffering();
in .NET Core 3.0+
And here's how I read/write the body content:
First a filter, so we just modify the content types we're interested in
private static readonly IEnumerable<string> validContentTypes = new HashSet<string>() { "text/html", "application/json", "application/javascript" };
It's a solution for transforming nuggeted texts like [[[Translate me]]] into its translation. This way I can just mark up everything that needs to be translated, read the po-file we've gotten from the translator, and then do the translation replacement in the output stream - regardless if the nuggeted texts is in a razor view, javascript or ... whatever.
Kind of like the TurquoiseOwl i18n package does, but in .NET Core, which that excellent package unfortunately doesn't support.
...
if (modifyResponse)
{
//as we replaced the Response.Body with a MemoryStream instance before,
//here we can read/write Response.Body
//containing the data written by middlewares down the pipeline
var contentType = context.Response.ContentType?.ToLower();
contentType = contentType?.Split(';', StringSplitOptions.RemoveEmptyEntries).FirstOrDefault(); // Filter out text/html from "text/html; charset=utf-8"
if (validContentTypes.Contains(contentType))
{
using (var streamReader = new StreamReader(context.Response.Body))
{
// Read the body
context.Response.Body.Seek(0, SeekOrigin.Begin);
var responseBody = await streamReader.ReadToEndAsync();
// Replace [[[Bananas]]] with translated texts - or Bananas if a translation is missing
responseBody = NuggetReplacer.ReplaceNuggets(poCatalog, responseBody);
// Create a new stream with the modified body, and reset the content length to match the new stream
var requestContent = new StringContent(responseBody, Encoding.UTF8, contentType);
context.Response.Body = await requestContent.ReadAsStreamAsync();//modified stream
context.Response.ContentLength = context.Response.Body.Length;
}
}
//finally, write modified data to originBody and set it back as Response.Body value
await ReturnBody(context.Response, originBody);
}
...
private Task ReturnBody(HttpResponse response, Stream originBody)
{
response.Body.Seek(0, SeekOrigin.Begin);
await response.Body.CopyToAsync(originBody);
response.Body = originBody;
}
A "real" production scenario may be found here: tethys logging middeware
If you follow the logic presented in the link, do not forget to addhttpContext.Request.EnableRewind() prior calling _next(httpContext) (extension method of Microsoft.AspNetCore.Http.Internal namespace).

ASP.NET Core Compression Middleware - Empty Reponse

I am using some custom compression middleware from this repository (pasted below). Upon the first request, the content is compressed just fine. For every request after that, the response comes back as completely empty (with a Content-Length of 0).
This only started happening after migrating from ASP.NET Core RC2 to RTM.
Does anyone know why this is happening?
CompressionMiddleware:
public class CompressionMiddleware
{
private readonly RequestDelegate _next;
public CompressionMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
if (IsGZipSupported(context))
{
string acceptEncoding = context.Request.Headers["Accept-Encoding"];
var buffer = new MemoryStream();
var stream = context.Response.Body;
context.Response.Body = buffer;
await _next(context);
if (acceptEncoding.Contains("gzip"))
{
var gstream = new GZipStream(stream, CompressionLevel.Optimal);
context.Response.Headers.Add("Content-Encoding", new[] { "gzip" });
buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(gstream);
gstream.Dispose();
}
else
{
var gstream = new DeflateStream(stream, CompressionLevel.Optimal);
context.Response.Headers.Add("Content-Encoding", new[] { "deflate" });
buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(gstream);
gstream.Dispose();
}
}
else
{
await _next(context);
}
}
public bool IsGZipSupported(HttpContext context)
{
string acceptEncoding = context.Request.Headers["Accept-Encoding"];
return !string.IsNullOrEmpty(acceptEncoding) &&
(acceptEncoding.Contains("gzip") || acceptEncoding.Contains("deflate"));
}
}
I have found the following in "Add HTTP compression middleware" issue:
I have added gzip and it worked, but first request. I mean in the first request, the response page is null (context.Response.Body) but when you refresh the page (just once) it works correctly after that.(I don't know why but I have to solve it)
And response on question is:
You need to update
context.Response.Headers["Content-Length"] with actual compressed
buffer length.
CompressionMiddleware.cs
And above link to realisation of compression middleware contains:
if (context.Response.Headers["Content-Length"].Count > 0)
{
context.Response.Headers["Content-Length"] = compressed.Length.ToString();
}