WCF Streaming System.OutOfMemoryException - vb.net

Question

In my streamed WCF application I get System.OutOfMemoryExceptions in my service for large messages (querying >7GB MSSQL tables using a datareader, messages might even exceed 7GB) while small messages work just fine. I can observe the memory usage grow constantly during the execution of DataReaderToExcelXml (see below). The weird thing is, that it usually grows fast to 2GB, stays at 2GB +-1GB for a while (1-5 minutes) and then again raises very fast to ~6.5GB which leads to the exception (machine has 8GB memory). At this point it looks to me, as if the Stream isn't passed through anymore but buffered.
I already enabled trace logging, but it seems to stop with the exception. The DataReaderToExcelXml function call is the last visible event in the trace log.
In the WCF message contract I ensured that the message only contains the Stream. On client side, the returned Stream is simply read, written to a filestream and disposed. However, I can never observe the client code being executed or the file being written when I get the exception.
I already tried setting the maxBufferPoolSize to zero, both on the client and server side, as described here https://stackoverflow.com/a/595871/4166885. No success.
Stream writer function on WCF-service side:
Public Shared Function DataReaderToExcelXml(ByRef dr As SqlDataReader) As Stream
Dim ms As New MemoryStream
Dim tw As New IO.StreamWriter(ms)
For Each row As DbDataRecord In dr
'Embed row in ExcelXml, detailed function omitted
tw.write(row.toString()) 'row.toString is just a simplification
End While
tw.Flush()
dr.Close()
ms.Seek(0, SeekOrigin.Begin)
Return ms
End Function
Web.config bindings
<bindings>
<basicHttpBinding>
<binding receiveTimeout="24.00:00:00" sendTimeout="24.00:00:00"
maxBufferPoolSize="9223372036854775807" maxReceivedMessageSize="9223372036854775807"
messageEncoding="Mtom" transferMode="Streamed" bypassProxyOnLocal="True">
<readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647"
maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
<security mode="Transport">
<transport clientCredentialType="None" />
</security>
</binding>
</basicHttpBinding>
</bindings>
App.config bindings
<binding name="BasicHttpBinding_IFileService" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="24.00:00:00" sendTimeout="24.00:00:00"
allowCookies="false" bypassProxyOnLocal="true" hostNameComparisonMode="StrongWildcard"
maxBufferSize="2147483647" maxBufferPoolSize="2147483647"
maxReceivedMessageSize="8589934592" messageEncoding="Mtom"
textEncoding="utf-8" transferMode="Streamed">
<readerQuotas maxDepth="32" maxStringContentLength="2147483647"
maxArrayLength="2147483647" maxBytesPerRead="4096" maxNameTableCharCount="2147483647" />
<security mode="Transport">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
edit: Using .NET memory profiler on my service I found out, that the increasing amount of memory can be traced back to the type byte[]. Personally, that's not telling anything to me, but maybe it's useful information.
edit2: While inspecting the System.OutOfMemoryException I just noticed that it occurs when ms.Capacity = ms.length = 1073741824 = 1GB. So just before the memory stream would double its capacity. I'm still not sure, why w3wp consumes such large amounts of memory in the first place, but now it's clear that the memory stream triggers the Exception.

Answer 1

Well, reading your question, I think that the source of your error, is the key. Where is exactly the error? In transport, or in your server?
If the Exception is raised in DataReaderToExcel, maybe you could consider another ideas (I think it isn't a WCF issue):
By now, I think you would have the same problem in a Windows forms app.
You need memory, so you can fill it only with your SqlDataReader, you have to query by blocks, based on ranges.
We will start computing the "TotalRecords":
SELECT COUNT(*)
FROM YOUR_TABLE
WHERE WATHEVER_YOU_WANT
For example, querying blocks of 10.000 records, you have TotalRecords/10.000 pages (querys) +1. (Example: 30.001 rows, 10.000 row per block = 4 queries )
Iterate building them so:
select top **10.000** * from
(SELECT TOP (100) PERCENT ROW_NUMBER() OVER (ORDER BY YOUR_TABLE_FIELD_1 DESC) ROW_PAGINATED, YOUR_TABLE_FIELD_1, YOUR_TABLE_FIELD_2 , ... , YOUR_TABLE_FIELD_N
FROM YOUR_TABLE
WHERE WATHEVER_YOU_WANT
ORDER BY YOUR_TABLE_FIELD_1 DESC
) YOUR_ALIAS
WHERE YOUR_ALIAS.ROW_PAGINATED BETWEEN min_records_per_page AND max_records_per_page
)
Where min_records_per_page AND max_records_per_page have this values:
Query 1:
min_records_per_page= 1
max_records_per_page = 10000
Query 2:
min_records_per_page= 10001
max_records_per_page = 20000
...
Query N:
min_records_per_page= (N-1)* +1
max_records_per_page = TotalRecords
On each iteration, you will map each datarow into a ExcelXml class.
Doing so, you will avoid to consume all the memory, assuming you will be Disposing the objects you use. You can use the Function SetProcessWorkingSetSize each 2 iterations, for example.
To free memory after a significant number of processed rows:
Private Declare Auto Function SetProcessWorkingSetSize Lib "kernel32.dll" (ByVal procHandle As IntPtr, ByVal min As Int32, ByVal max As Int32) As Boolean
Dim Mem As Process
Mem = Process.GetCurrentProcess()
SetProcessWorkingSetSize(Mem.Handle, -1, -1)
Then, now you have an array with mapped objects. Part 1 finished.
Part 2. Your client WCF may receive this data. Tell us then if you have problems once the data to be sent are ready. Then we can explore communication parameters, and then we would say "increase that parameter, or add this one".
Hope it helps