In my WCF service, I am retrieving data from SQL server using Command.ExecuteReader() method. The data size is very large (around 1+ GB) and transferring this data to client over netTcp binding.
I am planning to implement stream mode instead of buffered mode in WCF. Can anyone point me to any article or document to do the same.
In simple words, my objective is to convert IDataReader to stream object that will transfer to client and client side, want to convert this stream back to dataset/datatable or anything that can be bind with Grid.
I cannot convert IdataReader to IEnumerable as data is coming through SP and no of columns in output set keep changing (I don;t want to add no of column limitation in code).
Ultimately, final communication will be done on dataset from WCF service to client app. If any solution like converting Dataset to stream, sent it to client and at client, convert stream back to dataset will also solve my problem.
You should not try to convert the IDataReader to a stream, but let your data access method return an IEnumerable of a type representing a single row of the query result like this:
public IEnumerable<Order> GetOrders()
{
IDbCommand cmd = ... <<build your command here>> ...
using(var rdr = cmd.ExecuteDataReader())
{
while(rdr.Read())
{
Order order = new Order {Id=rdr.GetDecimal(1), Name=rdr.GetString(2)};
yield return order;
}
}
}
Next you can serialize the result of this method to a stream (as shown by #Mohamed, for example). This way you can send a list of objects to the client without needing the complete resultset to be loaded in memory. And you are still sure the datareader is disposed when the reader has reached the end of the result.
you can convert anything to stream like this:
var stream = new MemoryStream(Encoding.UTF8.GetBytes(Newtonsoft.Json.JsonConvert.SerializeObject(datareader)));
but its not a best practice. you should create an array of objects with your datareader and then return it.
Related
I have a Kafka consumer configured with schema polling from the topic, what I would like to do, is create another Avro schema, on top of the current one, and hydrate data using it, basically I don't need 50% of the information and need to write some logic to change a couple of fields. Thats just an example
val consumer: KafkaConsumer<String, GenericRecord>(props) = createConsumer()
while (true) {
consumer.poll(Duration.ofSeconds(10).forEach {it ->
println(it.value())
}
}
The event returned from stream is pretty complex, so I've modelled a smaller CustomObj as a .avsc file and compiled it to java. And when trying to run the code with the CustomObj, Error deserializing key/value for partition all I want to do is consume an event, and then deserialize it into a much smaller object with just selected fields.
return KafkaConsumer<String, CustomObj>(props)
This didn't work, not sure how can I deserialize it using CustomObj from the GenericRecord? Let me just add that I don't have any access to the stream or its config I can just consume from it.
In Avro, your reader schema needs to be compatible with the writer schema. By giving the smaller object, you're providing a different reader schema
It's not possible to directly deserialize to a subset of the input data, so you must parse the larger object and map it to the smaller one (which isn't what deserialization does)
I am using BufferManager in my WCF service. I created my own class to wrap around the BufferManager which implements IDisposable. Right now my Dispose method looks like this:
public void Dispose()
{
this.bufferManager.Clear();
}
My question: does this accomplish the same thing as calling ReturnBuffer on all of the buffers that have been taken by the manager?
Just for some background: I am using the BufferManager in the following method:
public byte[] ReadAllBufferedBytes(string filePath)
{
using (var fileStream =
new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
byte[] buffer = this.bufferManager.TakeBuffer((int)fileStream.Length);
fileStream.Read(buffer, 0, buffer.Length);
return buffer;
}
}
The reason I am doing this is because I kept getting OutOfMemory exceptions which would tear down the service.
The host server has 3 GB of memory. The service is in InstanceContextMode.Single mode, so images are processed one at a time. The images are received as byte arrays - the biggest might be 100MB, but are typically much smaller - converted, and then returned as a byte array. A lot ends up on the Large Object Heap, and image sizes vary quite a bit.
I am wondering if the issue is heap fragmentation.
As each page of a document gets converted, it is appended to a temp file on disk. After the conversion, I read the entire converted file from disk into a byte array and return it to the client.
The standard File.ReadAllBytes method creates a new byte array when it reads from the file, which inevitably ends up on the LOH due to the image sizes I'm working with (I assume this is what happens). I created the ReadAllBufferedBytes method to do the same thing, but to buffer the byte array and let the BufferManager return the buffer when it is disposed.
Another question is: do I even need to do all this?
The BufferManager is normally used in scenarios where you must prevent GC pressure - where there are a lot of small byte[] assignments, like when receiving or sending data on a very low level (e.g. sockets). The emphasis here is on a lot, otherwise the GC should be able to handle the memory allocations just fine.
To prevent the loading of the entire converted document file into memory you should use the FileStream directly (without reading it's entire content into memory - a byte[]) in combination with the streamed (response) TransferMode, if possible.
I have a file uploading logic and a very specific business rules. And according them I should parse my filemodel to row, which looks like "Header:{processed field1},{processed field2},{processed field3},{processed field4},{processed field5},{processed field6},{processed field7},{processed field8} and so on for 19 params" It's initially custom serialization.
And I also should have possibility to parse this row back to object. So, the question is what is a common idea to codding such staff?
Because now for parsing model to row I just use string.format with many options, and for parsing row to model I split the row by ',' and then manipulation with parts of information assign it to models fields. But in this implementation there are a lot of low level work, some hard coded position and also a lot of things that do not look pretty for me.
There's not going to be any magic involved here, particularly since you are serializing the object to a non standard format. You're probably going to have to live with the 'ugly' code.
You should put your serialization / deserialization inside a custom serializer. You can follow the same pattern as the other serializers in the .net library and implement the IFormatter interface . This would provide you with a common interface that you can use to stream to and from a file (or any stream):
using (var fileStream = new FileStream(fileName, FileMode.Create))
{
var formatter = new CustomFormatter();
formatter.Serialize(fileStream, objectToSerialize);
}
using (var fileStream = new FileStream(fileName, FileMode.Read))
{
var formatter = new CustomFormatter();
return (CustomType)formatter.DeSerialize(fileStream);
}
You can see an example of a custom formatter in this download
So I have a function which returns a pdf stream. The calling procedure gets the stream and writes it to disk (thanks ServiceGuy!). Normal VB code is as follows:
Public Function GetPDF() As System.IO.FileStream
GetPDF = File.OpenRead("C:\Documents and Settings\jspel903\Desktop\3211LD.pdf")
End Function
Now, for whatever reason when I put that code in my WCF (see below) and build/install, it won't return anything. The calling procedure gets nothing after My WCF is hosted in a Windows service (I can't get it to debug or step into). So I'm wondering if, perhaps, WCFs won't return a FileStream?
Here's the IService:
<ServiceContract()> _
Public Interface IService1
<OperationContract()> _
Function GetPDF() As System.IO.FileStream
'Should return a .pdf file as a stream
End Interface
Here's the svc:
Imports System.IO
Public Class Service1
Implements IService1
Public Function GetPDF() As System.IO.FileStream Implements IService1.GetPDF
GetPDF = File.OpenRead("C:\Documents and Settings\jspel903\Desktop\3211LD.pdf")
End Function
End Class
Seems to me like it should work. Any ideas?
You probably want to convert your FileStream to a byte array and return that. WCF is generally limited to returning serializable types.
// fs is your FileStream
byte[] Data = new byte[fs.Length];
fs.Read(Data,0,fs.Length);
It's not going to work... Think about what it would mean if it did. Your WCF client would have access to a file stream on your server device. It'd have to be able to perform all of the operations that you could do locally, over a remote connection.
WCF is a transport mechanism for data, it doesn't actually send object references to the server's objects. The client gets a copy of the object, that has been serialized by the server, then deserialized at the client side. If you could send a stream object, the file handle / memory reference etc wouldn't mean anything to the receiving client.
You need to read the data from the stream on the server and then convert it into an appropriate data object to transmit back to the client.
EDIT:
Apparently you can use streaming with WCF, but you're not going to be able to return a System.IO.FileStream, you can return a Stream.
Take a look at: Large Data And Streaming for a description and some of the restriction's you'll need to be aware of if you take that approach.
I have an existing application that uses Active Record for its data retrieval. It's in VB.NET (first time I'm doing VB.NET; I usually work in C#). And I'm building a method to return a List(Of T) of an object.
The current pattern uses a SQLDataAdapter to populate a datatable. I COULD add the record to the List(Of T) as I fill the datatable, but there HAS to be a better way to do this.
Any ideas? I'm not married to using SQLDataAdapter, if there's a better way that avoids it...
As you've still not had any responses...
I haven't used Active Record myself, so I don't know at what point that makes any kind of difference but it strikes me that reading into a DataTable and a List<T> is duplicating things somewhat.
How about populating the List<T> from a SqlDataReader instead? As you move through the results using Read(), create a new object from the current row and add it to the list. If you need to do it for various different types, I'd write a generic method to do it, along the lines of:
public static List<T> ToList<T>(this SqlDataReader reader,
Func<SqlDataReader, T> rowConverter)
{
List<T> ret = new List<T>();
while (reader.Read())
{
ret.Add(rowConverter(reader));
}
return ret;
}
Then you can implement the converter with a lambda expression or anonymous method.
Alternatively you could make the extension method target the command instead of the SqlDataReader - it could deal with calling ExecuteReader, converting every row and then disposing of the reader afterwards.
(It's a slightly odd signature because SqlDataReader represents both the iterator and the current row; just a quirk of the API.)
I'm probably missing something deeper though - if this doesn't help, could you explain why, perhaps as a question edit?