I'm trying to pull a large amount of data out of a WCF web service. The request is fairly small and the response message will be very big. Currently the web service is throwing SystemOutOfMemory exceptions due a limitation on IIS6 for the memory it can allocated (~1.4GB).
I have read in some blogs that implementing streaming will solve my problem.
Can anybody share their experiences with this topic? I'm most interested in any sample client-side & service-side code that can be shared or any recommendations/best practices. MemoryStream vs FileStream? Return type should be Stream, Message, Byte[]?
My operation looks like the following: (typically it will return a big number of elements in the response array, ~200K elements)
MediumSizeResponseClass[] GetData(SmallSizeRequestClass request)
If you want to stream back only the response, then use transferMode=streamedResponse in your binding configuration. This makes sure only the returned response will be streamed.
The return value of a streaming function must be a Stream. You can then read from that stream and do whatever you need to do with it (store it, analyse it, whatever).
So basically you'd have a service contract something like this:
[ServiceContract]
interface IYourService
{
[OperationContract]
Stream GetData(SmallSizeRequestClass request);
}
On the server, you basically just write to a stream, while on the client, you read from the stream.
Have you consulted the MSDN docs on WCF Streaming?
Related
Requirement:
We have like more than 100 endpoints in our service/Api. Do the following:
--> track all the data like the request body, request url, response from our server and cookies.
--> upload that data to blob which would be further processed to insert in Storage for tracking/auditing purpose.
We are using Web Api .net core project.
Need suggestions:
We were thinking to use middleware in .net core to read the request and response object. But found that the httpContext has request/response in stream objects. And would need memory stream and stream reader to read the data. After adding that code in the middleware, it got bit fuzzy. Concern is if there are any issues in this new code, it might break all Api calls. We can wrap the code in try and catch but then was thinking if there is some better way to do this?
Any suggestions are greatly appreciated. Thanking in advance.
While I have already learned how I can easily write to the response stream, I was wondering whether in the context of a ServiceStack client (e.g. ServiceClientBase) I can gain access to the server's response stream.
Before I start accessing the HttpWebResponse object I wanted to double-check that I do not reinvent the wheel.
See this earlier answer for examples of how to access the underlying raw response types.
Also the C# ServiceClient wiki is the authoritative documentation on ServiceStack's built in C#/.NET service clients. It includes a section on different ways of accessing the underlying raw response, including via the Response stream:
using (Stream responseStream = client.Get<Stream>("/poco/World")) {
var dto = responseStream.ReadFully().FromUtf8Bytes().FromJson<PocoResponse>();
dto.Result //Hello, World
}
I am still confused when it is appropriate to use the Message type in WCF like below
[ServiceContract]
public interface IMyService
{
[OperationContract]
Message GetData();
[OperationContract]
void PutData(Message m);
}
Why would you want to use it?
Can you use it for streaming?
Thanks
MSDN lists the follow reasons for using the message class directly:
When you need an alternative way of creating outgoing message contents (for example, creating a message directly from a file on disk) instead of serializing .NET Framework objects.
When you need an alternative way of using incoming message contents (for example, when you want to apply an XSLT transformation to the raw XML contents) instead of deserializing into .NET Framework objects.
When you need to deal with messages in a general way regardless of message contents (for example, when routing or forwarding messages when building a router, load-balancer, or a publish-subscribe system).
See Using the Message Class for more detailed information.
Edit to address the streaming question
I didn't find a definitive answer in my quick scan via google, but the article above states "All communication between clients and services ultimately results in Message instances being sent and received" - so I would assume it could be used directly in streaming.
While the reasons listed by Tim are valid, we use messages directly in our services to create one uber routing service.
we have one service that can take any method call you throw at it, Clients are generated by wsdls supplied from multiple sources.
This service would take the message, examine its content and route it accordingly.
So in my opinion if you want to get closer to the wire, or when you dont know the type of incoming messages, you can use the message in the signature directly.
Streaming is a separate concept than message signature, streaming is supported by wcf under very specific bindings and security mechanism and the method signature has to be very specific (i.e it should return/accept stream). Also in streaming the actual stream of data travels outside the scope of soap message.
I was having a problem sending video data to a WCF restful service using post, my contract looks like this
[OperationContract]
[WebInvoke(Method = "POST",
ResponseFormat=WebMessageFormat.Json,
BodyStyle = WebMessageBodyStyle.Wrapped,
UriTemplate = "UploadMovie")]
string UploadMovie(Stream stream);
This works ok when I'm sending some text data but does not work when I attempt to send Video data, I have some exception catching in place but it seems like the request is not even being processed, since I get no response and no exceptions get logged... anyone have any input on this?
you can use the svclog app to determine exactly what is happening. You just have enable wcf logging
http://wcfsecurity.codeplex.com/Wiki/View.aspx?title=How%20to%20enable%20WCF%20message%20logging
If you are instantiating all wcf in code, you can just add an app.config with the correct information to your code directory and this will work. Just remember to take it out when you're done. Most likely you have some exception that the WCF framework is catching. That happened to me on a message that seemingly vanished into thin air.
Check out this post on streaming in Restful WCF. It's reverse of what you want to do, but using the AdapterStream class will probably help.
The problem was the buffer size, i ended up splitting the video up into chunks and sending it like that, thanks everyone for their input
I am trying to design an Picture Upload feature into a web site.
I am using ASP.NET 3.5, C#, and WCF.
I have been asked to accomplish the following:
1) Make the Uploader a Web Service
2) Return progress updates to the user as files are uploaded.
3) Log other relevant user-selected options in the database.
So, I have started off by creating a WCF web client with the
below service contract:
IService.UploadPictures(HttpRequest request);
private UploadServiceClient upload;
protected void Page_Load(object sender, EventArgs e)
{
upload = new UploadServiceClient();
upload.UploadPictures(Request.Files);
}
When I compile, I get the below error:
Type 'System.Web.HttpRequest' cannot
be serialized. Consider marking it
with the DataContractAttribute, and
marking all of its members you want
serialized with the
DataMemberAttribute attribute.
So, I went back into my service contract and
changed [OperationContract] to [DataContract]
but the change produced the same error.
Can somebody kindly tell me what I am doing wrong
and provide examples as to how to best move forward?
Thanks for your time.
You cannot use something like a HttpRequest as a WCF parameter. The error messages says it all - the HttpRequest is not serializable, and in order to work with WCF, types have to be serializable.
Also, you need to remember: you're not just passing an object instance to a method here - what you're really doing is having the WCF runtime serialize your request (the method name to call plus all the parameters passed in) into a message (think: e-mail or xml file), sending it to the server, deserialising there and building up a new copy of the given datatype (as defined in your DataContract), and doing something with it.
Your WCF service could well be self-hosted, e.g. running in a NT Service or console app - no HttpRequest available in those circumstances!
You need to definitely rearchitect your solution - you need to either check into WCF streaming to upload files to WCF (google for it - you'll find plenty of hits) or you'll need to find another way to pass the relevant info (e.g. list of filenames) to the WCF service without use of a HttpRequest object.
Marc
You are submitting a request as a parameter to a request. This is not what you want to do. I'm assuming that "Request.Files" is an array of files. This is what you want to upload. So something like:
IService.UploadPictures(List<SomeFileType> request);