My .net WCF service calls a SSIS package using the Package.Execute(); method.
After I call execute, I set pkg.Dispose() and app = null;
The memory usage keeps climbing, 100mb to 150mb all the way to almost 300mb.
I am recycling the process now, but want to know the source of the problem.
Any ideas?
Update
The application that calls the WCF service is on another server so there is no issue there.
Are you closing your host? Just using a using statement? what does the open/close code look like?
There are a number of ways to do this quite common task (diagnose memory leaks in w3wp work processes). Tess has a great "getting started" post here:
http://blogs.msdn.com/tess/archive/2008/05/21/debugdiag-1-1-or-windbg-which-one-should-i-use-and-how-do-i-gather-memory-dumps.aspx
Oisin
An increase in virtual memory is not necessarily a problem, and 300MB is not very much memory in any case. Does IIS recycle on its own? If not, then I suggest you leave it alone.
Are you running SSIS 05 or 08? I remember 05 having a known mem leak issue when called using the API.
HTH
Related
We are developing a Web API using .Net Core. To perform background tasks we have used Hosted Services.
System has been hosted in AWS Beantalk Environment with the Load Balancer. So based on the load Beanstalk creates/remove new instances of the system.
Our problem is,
Since background services also runs inside the API, When load balancer increases the instances, number of background services also get increased and there is a possibility to execute same task multiple times. Ideally there should be only one instance of background services.
One way to tackle this is to stop executing background services when in a load balanced environment and have a dedicated non-load balanced single instance environment for background services only.
That is a bit ugly solution. So,
1) Is there a better solution for this?
2) Is there a way to identify the primary instance while in a load balanced environment? If so I can conditionally register Hosted services.
Any help is really appreciated.
Thanks
I am facing the same scenario and thinking of a way to implement a custom service architecture that can run normally on all of the instance but to take advantage of pub/sub broker and distributed memory service so those small services will contact each other and coordinate what's to be done. It's complicated to develop yes but a very robust solution IMO.
You'll "have to" use a distributed "lock" system. You'll have to use, for example, a distributed memory cache who put a lock when someone (a node of your cluster) is working on background. If another node is trying to do the same job, he'll be locked by the first lock if the work isn't done yet.
What i mean, if all your nodes doesn't have a "sync handler" you can't handle this kind of situation. It could be SQL app lock, distributed memory cache or other things ..
There is something called Mutex but even that won't control this in multi-instance environment. However, there are ways to control it to some level (may be even 100%). One way would be to keep a tracker in the database. e.g. if the job has to run daily, before starting your job in the background service you might wanna query the database if there is any entry for today, if not then you will insert an entry and start your job.
I have a project with requirements that response time should be under 0.5sec under load as 3000 concurrency users;
I have few API which use some aggregation from SQL Server.
when we testing it with 3000CCU average response time about 15 second. And also 500 error due to SQL can't handle so many requests. Actually requests to SQL Server interrupt with timeout)
Our current instance is r4.2xlarge 8CPU and 61GB Memory.
All code is asynchronous without blocking operations.
We running our app behind load balancer with 10 instances 300 CCU per instance in this case. utilization on instances about 30%. The bottleneck currently is SQL server.
I see few solution. Setup some big SQL, Cluster or Sharding, I'm not really sure. I'm not strong in that.
Or use cache for request. We have mostly read only data, which we can cache after aggregation.
Update:
I need solution to cache exactly sql responses. To order work with it late with LINQ.
But it seems there is no ready solution for that.
I found good try for that called CacheManager. But there are few problems exist with this.
It works with Redis only in sync mode, means use sync command instead of async.
There is no implementation of concurrency lock, which can occur in our case because we have 10 instances. We need solution which work with as distributed cache.
There are few bugs which utilize Redis multiplexor wrong. And you constantly will have connection problem.
Please advice how to overcome this issue. How you solve it. I'm sure there are people who already solve it somehow.
I enable query store on sql and monitor all missing indexes. Ef core generates some of request absolutely different way that expected. After creating missing indexes performance became much better. But i still have problem to handle required CCU. I explore all existing solution which extend ef core to cache. Most of them was written in sync version. Which just can’t utilize all benefits of async. As well i did’t found any distributed cache that implement distributed lock. Finally I create this lib which extend ef core and and distributed cache in redis .cache allow us scale a lot more. And now everything just flight;) leave it here, for someone who have performance issue like me. https://github.com/grinay/Microsoft.EntityFrameworkCore.DistributedCache
I've got a tcpserver serving multiple clients at the same time using multithreading. It works very good but i have got an issue on memory management.
Up to 1500 clients may connect to the server and stay connected for hours and days.
So i am having "Out of memory trouble" because i MUST use 32bit operating system. So i want converting to 64bit os should stay as plan b.
What do you suggest me to do?
A task based asynchronous tcpserver serving multiclient?
Or keep going with multi threads and making reusable 1500 threads?
Thanks.
simply changing "new thread" to "new task" solved my problem. Thank you all.
New Task(Sub() listen(tcp_client))
I have a process using Microsoft Sync Framework 2.0 across a WCF Service (IIS Hosted) to synchronize a SQL 2008 Standard database (server) and SQL CE 3.5 (client). All was working perfectly, until a single user started receiving OutOfMemory Exceptions. As it turns out, this user has a dataset that is significantly larger than any other user.
The dataset in question is 800,000 rows, with a total size when exported to CSV from SSMS of 174MB. Most users are in the 20-30MB range, which works fine.
I am using the DbServerSyncProvider, and SqlCeClientSyncProvider.
I have implemented batching as described in other articles and posted, to no avail. As I understand it, the batching mechanism in the DbServerSyncProvider is just how many revisions of the data to retrieve in one pull. Even with an anchor difference of 1, I still result in the same sized dataset.
I am using transferMode="Streamed" on my service, and I have applied the fix for Streamed when hosting in IIS.
I have tried upping the maxReceivedMessageSize, first from 20MB to 200MB, then to 2GB, and finally to 10GB, all with no success. This was done on both the server and client.
My WCF Trace logs show the Execute of GetChanges, but never logs anything under Process action.
I have read about the SqlSyncProvider, and how it allows batching by memory size. I can't find much information about using this through a WCF Service, though, and before I attempt to rewrite my client and server using this, I wanted to check if I was being an idiot on something and whether the SqlSyncProvider could solve my issue, along with being able to transfer across a WCF Service.
Thanks in advance...
The out of memory is most likely caused by the way Datasets are serialized.
If you want to re-write using the SqlSyncProvider, check out the section Code Specific to N-Tier on this link: http://msdn.microsoft.com/en-us/library/dd918908.aspx#Y3096. That should give you an idea on writing the WCF service component for the SqlSyncProvider.
You may also check out the sample SQL Server and SQL Compact N-Tier with WCF
If you want to retain your existing providers, you can play around using DatasetSurrogates. Check out a sample here: Sync Framework WCF-based Synchronization for Offline scenario – Using custom dataset serialization
I am just wondering why is restarting IIS 7 application pool consider a good practice? I know that it'll clean up orphan resources like threads or session state, but how does other web server deal with this kind of problem? Do java developer/admin have to restart weblogic/websphere/apache/tomcat every so often to clean up junk in memory?
Typically it's used to compensate for bad programming. Sometimes yours, sometimes third parties depending on the libraries you are using.
And, as with all recommendations, it's best to look at your specific environment to see if it's really necessary before doing it.
Start with profiling the app. A simple way is to just let it run for a while under load. Look for things like does memory utilization settle down or does it continually grow?
If it keeps growing, you will need to set IIS to recycle every so often and go fix your app. If it grows to a point then settles down then you should be just fine. Make sure you test even the edge cases of your app.
With IIS it is possible to have non-managed code - the best example is a COM object - that don't behave well. As a result you can set your IIS to recycle processes periodically, to allow for the memory leakage that might occur with such code.
It's not necessary to do it though. You can eliminate the restart thresholds, if you prefer not to restart the processes.
If this idea doesn't exist in other servers, then it may be because other servers allow only managed code.