We're trying to track down some performance issues with an application and would like to figure out how many outgoing requests are being triggered by each page load. I can find lots of counters for showing incoming WCF connections, but nothing tracking how many are going out. Any ideas short of retrofitting all of the pages with custom counters?
You could set up tracing and use the Trace View Tool or you could go lower level and use Wireshark to monitor the outgoing network activity.
Related
I am getting my head around messaging. Currently we are spiking a few scenarios using Rebus. We are also considering NServiceBus.
The scenario we are trying to build is a proof of concept for a background task processing system. Today we have a handful of backend services hosted in different ways. (web, windows services, console apps) I am looking to hook them up to rebus and start consuming messages using competing consumer, some mesages will have one listener and some will share the load of messages. Elegant :)
I got a pretty good start from this other question How should I set rebus up for one producer and many consumers and it is working nicely in the proof of concept.
Now I want to start reporting progress. My intital approach is to set up pub/sub as well and spin up a service that listens to progress events from all the services. And if a service is interrested in a specific progress in the future it is easy to subscripe of interrest to the messages and start listening.
But how shall I approach setting up both competing consumer and pub/sub? it is dimply two separate things? (In the rebus case one adapter using UseSqlServerInOneWayClientMode / UseSqlServer and another adapter that is set up for the pub/sub using whatever protocol we want?)
Or is there a better solution then having two "buses" here?
I've built something like that myself a couple of times, and I've had pretty good results with using SignalR to report progress from this kind of backend worker processes.
Our setup had a bunch of WPF clients, one single SignalR hub, and a bunch of backend worker processes. All WPF clients and all backend workers would then establish a connection to the hub, allowing workers to send progress reports while doing their work.
SignalR has some nice properties that makes it very suitable for this exact kind of problem:
The published messages "escape" the Rebus unit of work, allowing progress report messages to be sent several times from within one single message handler even though it could take a long time to complete
It was easy to get the messages all the way to the clients because they subscribe directly
We could use the hub groups functionality to group users so we could target progress/status messages from the backend at either all users or a single user (could also be used for departments, etc.)
The most important point, I guess, is that this progress reporting thing (at least in our case) was not as important as our Rebus messages, i.e. it didn't require the same reliability etc, which we could use to our advantage and then pick a technology with some other nice properties that turned out to be cool.
I'm setting up a web service with pyramid. A typical request for a view will be very long, about 15 min to finish. So my idea was to queue jobs with celery and a rabbitmq broker.
I would like to know what would be the best way to ensure that bad things cannot happen.
Specifically I would like to prevent the task queue from overflow for example.
A first mesure will be defining quotas per IP, to limit the number of requests a given IP can submit per hour.
However I cannot predict the number of involved IPs, so this cannot solve everything.
I have read that it's not possible to limit the queue size with celery/rabbitmq. I was thinking of retrieving the queue size before pushing a new item into it but I'm not sure if it's a good idea.
I'm not used to good practices in messaging/job scheduling. Is there a recommended way to handle this kind of problems ?
RabbitMQ has flow control built into the QoS. If RabbitMQ cannot handle the publishing rate it will adjust the TCP window size to slow down the publishers. In the event of too many messages being sent to the server it will also overflow to disk. This will allow your consumer to be a bit more naive although if you restart the connection on error and flood the connection you can cause problems.
I've always decided to spend more time making sure the publishers/consumers could work with multiple queue servers instead of trying to make them more intelligent about a single queue server. The benefit is that if you are really overloading a single server you can just add another one (or another pair if using RabbitMQ HA. There is a useful video from Pycon about Messaging at Scale using Celery and RabbitMQ that should be of use.
I'm working on a system which amongst other things, runs payroll, a heavy load process. It is likely that soon, there may be so many requests to run payroll at peak times that the batch servers will be overwhelmed.
I'm looking to put together a proof of concept to cope with this by using MSMQ (probably replacing this with a commercial solution like nservicebus later). I using this this example as a basis. I can see how to set up the bindings and stick it together, but I still need a way to tell the subscribers hosted by WAS to only process the 'run heavy payroll process' message if they are not busy. Otherwise the messages on the queue will get picked up straightaway and we have the same problem as before.
Can I set up the subscribing service to say, "I'm busy, I can't take the message, leave it on the queue"? Does the queue need to be transactional?
If you're using WCF then there's no way to conditionally activate the channel thereby leaving the messages on the queue for later.
A better solution is to host the message receiver in a completely different process, for example as a windows service. These can then be enabled/disabled according to your service window requirement.
You also get the additional benefit of being able to very easily scale out the message receivers to handle greater loads (by hosting more instances of your receiver).
One way to do this is to have 2 queues, your polling always checks the high priority queue first, only if there are no items in that queue does it take an item from the other
I know that ZMQ offers all of the flexibility to do your own load-balancing. However I would expect the out-of-the-box broker, about 4 lines of code using the line
zmq_device (ZMQ_QUEUE, frontend, backend);
to load balance quite well as the documentation says it does load balance.
ZMQ_QUEUE creates a shared queue that collects requests from a set of clients, and distributes these fairly among a set of services. Requests are fair-queued from frontend connections and load-balanced between backend connections. Replies automatically return to the client that made the original request.
I have an army of back-end services and yet find that often my front-end clients have to wait several seconds for something that takes < 1/10 of a second in a 1:1 setting (there are same # of client and service machines). I suspect that ZMQ is not load-balancing properly out of the box - it's sending too many requests to the same service even though it doesn't have bandwidth, etc.
I think this is partly because the services are multithreaded in a way that lets them take up to 10 concurrent requests yet it slows down greatly at near the 10th request even though it can still accept them. Random distribution would be ideal. Is there an out-of-the-box way to do this or can it be done in a few lines of code, or do I have to write my own broker from scratch?
Fwiw issue was the workers were taking on work when they didn't have room for it, issue was not in ZMQ layer per se.
I designed one On line Trading Application, which uses blazeds & jetty,
in that i used AMF-LongPooling as channel, with following parameter,
Here is the problem is Each message is not reaching all the user,who are connected, messages are missing to few users (300 recieving out of 600)...
what we need to do to provided instant messages to all Online. ??
Please help me one?
Your question is too generic, it's not possible to give an answer because it depends on too many things: network, size of the messages, your system architecture etc. My suggestion is to invest heavily in reading BlazeDS developer guide and to turn the debug messages on (there is a lot of useful information displayed by BlazeDS). It would also help to study BlazeDS source code.
In case of AMF-longpolling the request is parked on the server and if too many requests are parked at a time, they will consume all available threads for the server. And the next client won't be able to connect.
In your case I am assuming the message size is not very big. And the solution can be one of the followings:
To increase the number of available threads. For that you can have multiple server instances and distribute your clients over them.
You can make use of LCDS.
You don't get that problem in LCDS as it makes use of NIO end points that don't block the thread. I have come to know that this thread restriction is not a problem with Servlet 3.0 and in that case you can support more clients with blazeds itself. You can check more about it HERE.