We are facing out of memory issues when lot of time outs happens with our back end service. Our application processes more than 300K transactions every day and some of our back end service times out when load is at it's peak. In this case, we have custom fault sequence initialized on API and custom fault sequence has respond mediator at the end to return the failed response to the client in case any time outs happens.
Our servers are going down and the heap dumps are pointing custom fault sequence.
Does anybody encountered this issue and also suggest us about best practices to implement custom fault sequence.
Related
When we are testing fault sequence in our flow, we noticed WSO2 is taking around 10seconds to invoke default fault sequence. Is this predictable behaviour in the product or I am missing something here.
Since the calling of default fault sequence is in the control of wso2 engine, we are not able to test it clearely. but we had out some logger before calling the nd point and logger at the entry of fault sequence and notices the time difference.
So the two logger statements have a gap of more than 10seconds. Would like to know what the wso2 engine doing here.
It would be helpful to provide a better explanation if you can share the error in the endpoint that you observe. Is there an error code? Based on what you have mentioned about the 10 seconds delay, this could be because of the callback processing interval. Please refer to http://blog.maheeka.me/2016/10/mocking-endpoint-behaviours-for.html for more details on this.
I have a situation where I have two programs (one exe and one dll loaded into the process space of another third-party exe) communicating requests with each other using a local machine wcf service (using net named pipe binding). There's a third host exe that starts hosting the service. It all works great (so far anyways... I'm still learning), but I got to thinking about what would happen if the channel faults or the service times out. What would be the best practice for checking and handling faults as well as keep the channel alive?
In my case it will be up to the user to keep the applications open or close them and we do have those users who tend to keep them open overnight, over the weekend, etc... It seems to me this could open the possibility of a fault or loss of service and I don't have a clue how to recover. Any help would be greatly appreciated.
Firstly, why would you keep the channel alive indefinitely?
Imagine you are connecting to a database from which you want to read over the course of one day. Would you create the database connection in the morning and then close it in the evening?
It is relatively cheap to construct a channel in WCF for each call, unless you know you are going to be making multiple calls within a few seconds of each other, in which case you should reuse the channel.
EDIT
This post explains how to do it. It's pretty complicated and it may be easier to just set a huge timeout value for the binding in code (as suggested at the end of the post):
Do WCF Callbacks TimeOut
EDIT
There's tons of stuff on google about this: http://bit.ly/10ZPWE2
i've tried to run mule on 3 cases in order to test it's mem usage:
One case is where i had a quartz generator create an event that a filter (right after it in a flow) allways stopped (Returned false) - meaning the flow did absolutly nothing.
In another case i did not use the filter but just used that flow to send a custom object to a WCF service running on another computer (using a cxf endpoint)
Also, i've checked what happened when i leave the flow as is but drop the wcf servce (meaning a lot of socket connection exceptions were thrown).
I did this because i am building a large app that would need this bus to work at all times (weeks at a time).
In all of those cases, the mem usage kept rising. (getting as high as 200mb ram after a few hours)
Any specific reasons this could happen?? What is causing mule to take more memory, in all of these cases?
Off the top of my head I'll stick with thread pool lazy initialization as explanation for this behavior. As time goes on and usage gets higher, the thread pools will get fully initialized.
If you want proof evidences take a look to this approach, or this one (with enableStatistics).
I'm having a problem where the WCF service hangs after 13-14 asynchronous process calls from the client. This occurs all the time. The client is a mobile JavaFX app. There is no specific error outputted in the server as well as in client. Someone suggested that it might be a throttling issue.
I've set the service side .config parameters maxConcurrent calls from 10 to 500
<serviceThrottling maxConcurrentCalls="500" maxConcurrentSessions="500” />
So this means, it should be able to accept more than 10 calls, right? However, it didn't resolve this issue. Still hangs on the 13-14th process call.
Only one client is connecting to this web service.
What do you think is wrong?
Do you close the client after doing your call?
When I encountered this problem, I did not close it, and the open requests blocked the service after a short time.
Edit: Ok, I know nothing about JavaFX =) The code below is C#, sorry. But you can surely do something similar.
Use either
WcfClient client = new WcfClient()
// ...
client.Close()
or
using(WcfClient client = new WcfClient()){
// ...
}
Similar problem here - I have an app calling from one process to another, locally, named pipes.
Calls are really light in code- basically takex an array of serializable objects, queues them on other side. Occasionally it hangs. Restarts afte rtimeout. no data lost, but... as the data is financial data, and the receiving app an autoamted trading system, that may result in very bad financial issues. Not been able to reproduce it yet.
This could very easily be caused by any deadlock condition in your code. If your service locks up and starts eating up 100% or CPU you have a dead lock. Create a dump file and see where your code was at.
I ran into the same issue my first WCF app it was a dictionary that i wasn't making sure was synchronized in logging code.
The SvcTraceViewer is super helpful in figuring out tough wcf
Getting this WCF error, and no idea how to fix it:
System.ServiceModel.CommunicationException: The sequence has been terminated by the remote endpoint. The user specified maximum retry count for a particular message has been exceeded. Because of this the reliable session cannot continue. The reliable session was faulted.
Any ideas welcome :(
From the error message, it would appear that you're using reliable messaging. One of its features is that if a message transfer fails, it will be retried - up to a maximum number of attempts.
Obviously, in your setup, this max number has been maxed out. This might indicate a problem with the network, or your service code, or both. Really hard to tell from here without knowing what you're doing and what your setup is......
I guess the main question would be: do you really need the reliable messaging feature? What are you trying to achieve with this? If you could turn it off, you wouldn't be seeing those errors... can you switch to some other mechanism, maybe message queueing (MSMQ)? Or can you rearchitect your app so you can live with the odd chance that one message might get delivered "out of band" ?