When serializing large response, client receives ServiceStack exception, Out of Memory, - serialization

Question

I have a ServiceStack RESTful web service on a linux box with apache/mod_mono.
public DataSetResponse Get(DataRequest req)
{
DataSetResponse Response = new DataSetResponse();
DataSet BigData = new DataSet();
this.Status = this.DataFetcher(ref BigData);
Response.Data = BigData;
Response.Status = this.Status;
System.Threading.Thread.Sleep(30000);
return Response;
}
When the thread sleeps I can see that mono is at 8% of the memory as reported by top. 30+ seconds later when the mono has cpu activity again the memory ramps up to 90% and an Out of Memory Exception is thrown. Mono continues to run but does not release its memory.
On small data sets (1/10 of the size) it seems to work fine and mono has 1% of memory. I think the memory growth occurs as the data object is serialized to Json, before it is streamed to the client.
Is this correct? And more important, how can I solve this?

Answer 1

I don't know how much RAM your server has but if it's maxing out on a single request, and I'm presuming that since it's a web service you expect to serve to multiple clients so you could be getting 2 or more of these requests at around the same time, you may want to consider some way to stream or chunk up the data (ie, the client can request a page of data at a time, and they can keep asking for more pages until they have the entire data set).
So your request DTO might include a page #, your data fetcher would would grab that next page (based upon whatever you decide to be the number of records per page), and return it.
Your response would need to include total # of pages, and page returned so the client could decide to keep fetching data.
The reason you probably see 8% prior to serialization is that the object is in its binary format - converting it to a big, JSON string is going to really balloon it out.
You could also consider some of the other binary formats ServiceStack supports - ProtoBuf and MessagePack.