IIS Buffering Requests Content Before Controller Action - vb.net

Question

I am trying to setup an "external" web api that will be able to receive in large http posts (+1GB) and forward the stream to another "internal" web api that writes the request contents to a file. I have model my implementation based on examples of using a custom WebHostBufferPolicySelector and using the UseBufferedInputStream method in the controller method. It works as expected when using IIS Express, no significate increase in memory footprint, but as soon as my code is deployed to IIS, the memory footprint is substantial and results in OOM.
I have put tracing statements in my controller methods and in my WebHostBufferPolicySelector.UseBufferedInputStream, and have verified that UseBufferedInputStream always is returning false and that my controller methods are getting hit. The only difference that I noticed is that when I debug, the time stamps between UseBufferedInputStream and my controller method are very close. Where hosted on IIS, the time stamps are very far apart, suggesting that something in between when UseBufferedInputStream is called and my controller method is called is buffering up the request entirely.
I am looking for some tips on to find out what that is causing the request to get buffered and how for it not to buffer and using streaming all the way.
Client is coming at the external web api with a content type of application/octet-stream with Transfer Encoding of Chucked.
Used to build out implementation
https://forums.asp.net/t/2018289.aspx?Web+API2+WebHostBufferPolicySelector+UseBufferedInputStream+override
https://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/
Proxy Web Api Controller Method
<HttpPost, Route("postLargeFile")>
Protected Overridable Async Function PostLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Started {0}", NameOf(MyProxyController.PostLargeFile))
Dim internalHttpClient As HttpClient
Dim fowardingContent As StreamContent = Nothing
Dim fowardingMessage As HttpRequestMessage = Nothing
Dim fowardingResponse As HttpResponseMessage = Nothing
Dim externalResponse As HttpResponseMessage = Nothing
Try
internalHttpClient = New HttpClient()
internalHttpClient.BaseAddress = "https://myinternalService.com"
fowardingMessage = New HttpRequestMessage(HttpMethod.Post, "https://myinternalService.com/saveLargeFile")
fowardingContent = New StreamContent(HttpContext.Current.Request.GetBufferlessInputStream(True))
CopyContentHeaders(Request.Content, fowardingContent)
fowardingMessage.Headers.TransferEncodingChunked = True
fowardingMessage.Content = fowardingContent
fowardingResponse = Await internalHttpClient.SendAsync(fowardingMessage, HttpCompletionOption.ResponseHeadersRead)
externalResponse = New HttpResponseMessage(fowardingResponse.StatusCode)
externalResponse.Content = New StreamContent(Await fowardingResponse.Content.ReadAsStreamAsync)
CopyContentHeaders(fowardingResponse.Content, externalResponse.Content)
Return New Results.ResponseMessageResult(externalResponse)
Catch ex As Exception
Return InternalServerError(ex)
Finally
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Finished {0}", NameOf(MyProxyController.PostLargeFile))
End Try
End Function
Internal Web Api Controller Method
<HttpPost, Route("saveLargeFile")>
Protected Overridable Async Function SaveLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Started {0}", NameOf(MyInternalController.PostLargeFile))
Dim bufferlessStream As IO.Stream
Dim fowardingContent As StreamContent = Nothing
Try
bufferlessStream = HttpContext.Current.Request.GetBufferlessInputStream()
Using fileStream As IO.FileStream = IO.File.Create("MyFile.txt")
bufferlessStream.CopyTo(fileStream)
fileStream.Flush()
End Using
Return New Results.StatusCodeResult(Net.HttpStatusCode.Created, Me)
Catch ex As Exception
Return InternalServerError(ex)
Finally
Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Finished {0}", NameOf(MyInternalController.PostLargeFile))
End Try
End Function
Policy Selector Configuration
Public Class MyBufferPolicySelector
Inherits Http.WebHost.WebHostBufferPolicySelector
Public Property Tracer As ITraceWriter
Public Overrides Function UseBufferedInputStream(hostContext As Object) As Boolean
UseBufferedInputStream = False
Tracer?.Info(Nothing, $"{Me.GetType.Namespace}.{NameOf(MyBufferPolicySelector)}", "{0} UseBufferedInputStream={1}", HttpContext.Current?.Request?.Url?.AbsoluteUri, UseBufferedInputStream)
Return UseBufferedInputStream
End Function
End Class
WebApiConfig for both Internal and External Web APIs
Public Module WebApiConfig
Public Sub Register(ByVal config As HttpConfiguration)
Dim tracer As SystemDiagnosticsTraceWriter
' Web API configuration and services
' Web API routes
config.MapHttpAttributeRoutes()
tracer = config.EnableSystemDiagnosticsTracing
tracer.IsVerbose = True
tracer.MinimumLevel = Tracing.TraceLevel.Debug
GlobalConfiguration.Configuration.Services.Replace(GetType(IHostBufferPolicySelector), New MyBufferPolicySelector() With {.Tracer = tracer})
End Sub
End Module

Answer 1

I was able to figure out what was causing the the buffering in IIS. The below link lead me to the uploadReadAheadSize setting in IIS. This was maxed out. So this would cause IIS to fully read in/buffer in the request before passing it into the module where the web api pipeline exists (web api controllers). After setting it to the default, I saw my large file posts not get buffered, the app pool memory footprint remained low, no more out of memory exceptions, and a large performance boost. Great!
But now I have the same issue as described in the below link. When SSL is required, set in IIS, which it is required in our non development environments, the uploadReadAheadSize needs to be increased so the ssl can work in the ssl module I guess. It might have to do with some SSL renegotiation.
Can anybody describe a way to prevent the buffering in SSL to keep the memory footprint low and prevent out of memory exceptions for large http posts?
Large file upload when using ssl and client certificates (uploadReadAheadSize) but dont want all data to be readahead