I have a WCF service which is called in 4 different places in my system. It returns approx 500 records from the database each time it's called.
I would like to use a cache in place of making the call to WCF every time because data in the DB will remain unchanged.
Is there anything built into WCF for this or do I have to create my own solution?
There are several classes in .NET to allow you to cache objects in memory and there are also open source solutions which will do this for you.
If you wish to code this yourself one class you can use is MemoryCache
An open source solution can be found here: Redis.IO
In addition to Jon's answer, you can also use SQL Cache Dependency.
If your WCF Web HTTP service depends on data stored in a SQL database,
you may want to cache the service's response and invalidate the cached
response when data in the SQL database table changes.
I have created a WCF web service that will upload data from SQL Server to our ISeries. When an end user is finished with their data entry (a batch), they will "send" the batch number to the web service. The web service will then upload that data to the ISeries. It cannot be assumed that this will be a quick process and there may be many end users hitting the web service at once. Likewise, because the way the database is setup on the ISeries, I can't upload this data simultaneously because we may run into locks, misfired triggers, etc. So, I want somehow to queue these calls so that they are done in order received.
I have been searching for methods to do this and there's a lot of information in 2011 and earlier discussing MSMQ. Is that the still preferred way to do this? Would Reentrant Concurrency mode be a more "modern" option?
There are a lot of alternative queuing systems. Since you have a SQL Server in place I would recommend using MSMQ. In this combination you can use TransactionScope out-of-the-box to handle transcations spanning your DB and the Queuing system.
From my own experience, MSMQ is a proven and stable technology stack.
I was playing with powerpivot to directly load 3 million rows from a sql database and performance is suprisingly good.
I tried generating a simple oData service by using vs2010 and silverlight RIA services and access that from powerpivot. Which works with small numbers of rows but blows up on the server if a single method tries to return 3 million rows. Not suprising I guess.
Ive often run into the message size issue with WCF and it is a real pain to configure transports to support larger sizes. Plus, ideally I dont want one big message but some sort of data packeting. Adding a layer of RIA and oData ontop of WCF seems to just make the idea of changing max message sizes even more convoluted.
Is there any support in the oData interface for a transport that will stream or packet the data returned from a method?
Is this a limitation of WCF/RIA or oData itself. Is it possible to use powerpivot connected to an oData source that returns millions of rows?
Anybody have ideas for better techniques of exposing large sets of data via WCF / RIA / oData ?
thanks,
Adam
Found it! In the DataService<> class InitializeService method needed to use config.SetEntitySetPageSize.
We've got a smart client that talks to a SQL Server database via WCF, displaying the entities in the database, and allowing the user to edit those entities.
Some of the WCF calls return a large data set. Since this data set doesn't change very often, I'm considering some sort of write-through cache on the client, and only getting the deltas from the WCF service.
That is: the client both reads from the service and writes to the service.
I'm not looking for disconnected/offline operation, but since the majority of the data doesn't change very often, I'd probably implement this with a local data store.
I don't want the local store to get too stale, and I don't think I'm too concerned about conflict resolution, because updates will always go straight to the WCF service -- think of it as a write-through cache.
Would Microsoft's Sync Framework be good for this? Could I use a local SQL-CE cache and perform the updates over WCF? The service end has a SQL Server 2005/2008 backend, but I don't want to talk to it directly. Does Sync Framework integrate well with WCF?
Are there other solutions out there? Should I roll something myself?
I don't think you have to couple it to WCF at all. FeedSync allows you to publish directly to an RSS feed.
The only that I'm not too sure about is if it would be suitable for a "large dataset" though. Since you don't need two way replication, if your dataset is extremely large, you might want to write your own WCF implementation to optimize it; especially for the initial population.
Ok, I have a pretty complex silverlight app that gets its data from a WCF service (asp.net hosted service layer) which in turn calls into a data layer that calls stored procedures in a SQL 2005 DB to extract the needed data. So the round trip goes like this:
Silverlight App --> WCF Service --> Data Layer --> DB --> Data Layer --> WCF Service transforms Data Entity into corresponding DTO (Data Transfer Object) or List<> thereof --> Silverlight App
Much of the data is highly relational (so it needs to exist in the DB), but it will change infrequently. It seems that I have several choices of locations to cache this "semi-constant" data:
I can cache it in the data layer. My data layer is already set up to use the SQLDependency class and cache the results from a stored procedure call. I think that this is or can be an application level cache.
I can cache the resulting DTO in an application level (or session level depending on the call) cache within the WCF service itself.
2(a) I could even take this a step further by serializing the XML for the resulting DTO(s) into a file on the WCF service side so that I could (a) check memory cache, then (b) check file cache and (c) hit the data layer
I could do something similar to 2(a) with isolated storage on the client side within the SL app. I could serialize the data to the local isolated storage with a hash (or a moddate or something) and then just make a call to check that.
One more thing to add: I am hosting this WCF service in IIS7 with dynamic compression turned on so that the (often very large and easily compressed) XML response gets gzip-ed. Ideally, it would seem, I would like IIS to cache this gzip-ed result to avoid all the extra processing. I think that it may do this already but I am not sure.
I am pretty sure that the final answer to this is some flavor of "it depends", but I would love to hear how others are approaching this. A good tactical recipe of Do X, Test Performance with tool Y, the do Z if needed would be great to have.
A few links (I will add to this as I research this):
WCF Caching Approach
If you have data that are user that will change quite rarely and need fast response, going for a custom mechanism bases on local storage is a great advantage quite faster than having to wait for a server roundtrip.
Dino Sposito published an interesting article about local storage and caching on MSDN Magazine there you can find as well an approach to catch assemblies (imagine just loading the minimum package required and just go loadin the rest of assemblies in background, ... performance rocket, more complexity on your code :)).
As you said is matter to go putting in a balance and decide.
HTH
Braulio
My approach would be this:
Determine if there is actually a problem with performance (isn't it alreade acceptable to my users?)
Measure the performance at each teir (how long does it take the database to come up with data? how long does it take the service to respond with data? how much time does it take from the service to the client?)
Based on the measurements I would then determine where to do my caching. Remember that, the closer to your data storage you do caching, the easier it is, but the closer to the client you do caching, the better the performance gain (usually).
Also remember that caching should not be the first thing to do to improve performance. You should also look into other performance gains as well. Are the stored procedures slow? Is there a lot of overhead in the WCF messages? Is there some inefficient processing in the service? Do I realy need all that data in one message?
HTH,
Jonathan
I think #2 is your best bet for maintainability and architecture. IIS provides caching, why not use it?
You don't want to have to reference System.Web from a data layer. Client side is not the best option either, because you'd have to write a bunch of additional code to keep the data synchronized.
Is System.Web caching even available to WCF when it's not running in ASP.NET compatible mode? Probably best not to depend on it and write your own.
On the other hand, look into Microsoft's Velocity project, which looks like it will produce a very interesting caching technology not dependant on ASP.NET.
We just recently implemented #3, the client-side caching using Isolated Storage.
In our app we have lot of drop downs and custom fields which the app used to get from the server every time it loads. Moving these data to IS really helped. The app now makes a call to check if there were any changes on the server, and if not - loads the data from the IS, otherwise ( which is pretty rare ) refreshes IS.
That eliminated a lot of WCF calls and data transfers, the SL pages' loading time is shorter, and the app in general became more scalable because of the reduced network traffic and db access.
Yes, there are some coding involved, but the benefits for the end users are essential.
Andrew
If you use RIA Services, then a simple approach is to have two separate edmx definitions. One for cached entities, one for transactional ones.
One domain context can reference the entities on another domaincontext via AddReference see.
The cached entities could be loaded immediately after user has authenticated. For simplicity, transactional data should not load until cached entities have loaded.
Depending on the size of the cache, you may also wish to consider serializing these values to local storage.