We are hosting a net.msmq service in IIS7.
The queue is transactional.
Messages arrive in the queue and are picked up correctly by the service.
If an exception occurs, message is put into the retry queue.
The retry delay is set to 1 hour, however when this time elapses the message is not "re-tried".
If we browse to the .svc or send another message to the main queue then the retry messages are also picked up.
So basically messages get stuck in the retry queue until somethin "boots up" the site pool again.
has anyone come across this same problem?
Sounds like your service's AppDomain is getting unloaded due to inactivity. That's always a pain in the neck with anything hosted in IIS, and usually the solution is to create something that will keep the AppDomain alive by pinging it every few minutes (you could easily expose a second MSMQ-based endpoint on your service and just send a message to it every ten seconds on so to keep it alive).
Related
I'm doing a test to see how the flow control behaves. I created a fast producer and slow consumers and set my destination queue policy highwater mark to 60 percent..
the queue did reach 60% so messages now went to the store, now the store is full and blocking as expected..
But now i cannot get my consumer to connect and pull from the queue.. Seem that blocking is also blocking the consumer from getting in to start pulling from the queue..
Is this the correct behavior?
The consumer should not be blocked by flow-control. Otherwise messages could not be consumed to free up space on the broker for producers to send additional messages.
So this issues surfaced when I was using a on demand jms service. The service will queue or dequeue via a REST services. The consumers are created on demand.. If the broker is being blocked as im my case being out of resource, then you cannot create a new consumer.
I've since modified the jms service to use a consumer pool(implemented a object pool pattern). The consumer pool is initialized when the application starts and this resolved the blocking issue
I am using BeginPeek() /no params/ to subscribe to messages coming in to my private queue. This is being done in a service hosted in NServiceBus host. When NServiceBus encounters transport connection timeout exception (i'm seeing circuit breaker armed logs and timeout exception logs), the peek event subscription seems get lost. When database connectivity becomes stable and new messages come in to my queue, the service is no longer notified.
Any ideas or suggestions on how to address this?
I have a WCF service that reads from a message queue. I noticed with our latest deployment that there is a message sitting in the queue that has not been read. The WCF service is up and running and it is correctly establishing a connection to the queue (or I would receive and error).
We made some changes to our code recently and we are wondering if perhaps that is related. It seems odd to us that the service would be running and yet the message not read. At a minimum, I would expect WCF to throw some sort of error if there was a malformed message in the queue.
I looked at the properties of the message and it says there have been zero moves, so I don't think it is being sent to a retry sub-queue. It is just sitting there and the service won't read it.
Is there a circumstance where WCF would ignore a message in the queue? How does WCF handle malformed messages?
Sorry if this isn't a lot of information to go on. At this point I am just trying to understand what is preventing the message from being processed.
This situation seemed just too odd. It didn't make sense that WCF would do nothing at all.
I eventually decided to simply try restarting the MSMQ service on the server. Once I did that, my service immediately picked up the messages and everything started working again.
I have no idea if this had something to do with a Windows Update or some other server change. I am glad it was this easy to fix - it will be the first thing I try in the future.
I know that once a message has been delivered to the MSMQ by a WCF client, the netmsmqbinding provides retries out of the box in case the service faults.
But if my client fails to put the message in the MSMQ in the first place, is there an out of the box client retry available in WCF or do I have to implement a client queue and retry logic in my client code?
Thanks
It's a highly unlikely scenario that your messages sent to the service will not even be placed in the client queue in the first place, if you have MSMQ server running on the client station and the MSMQ listener service is up and running you should have nothing to worry about. I don't think MSMQ offers anything to check this for you, you should code some method on your client to periodically Peek() the local queue and send an acknowledgment receipt for every message that has reached the queue, this is feasible since you can easily access your local queues in code and also every message sent via MSMQ from a client to a service will always go trhough the local queue. You can also tell that the message reaches the queue if your Send() method desn't return an error. But I don't think you really need to worry about message son the client not reaching the local queue first.
Not being an expert on MSMQ or WCF, I have read up a fair bit about it and it sounds and looks great.
I am trying to develop something, eventually but first some theory, which needs to be robust and durable.
MSMQ I guess will be hosted on a seperate server.
There will be 2 WCF services. One for incoming messages and the other for outgoing messages (takes a message, does some internal processing/validation then places it on the outgoing messages queue or maybe sending an email/text message/whatever)
I understand with the right configuration, we can have the system so that it can be transactional (no messages are ever lost) and can be sent exactly once, so no chance of duplication of messages.
The applications/services will be multithreaded to process messages, which there will be hundreds and thousands of them.
BUT during the processing of a message or through the services lifetime, what if the server crashes? What if the server reboots? What if the service throws an exception for whatever reason? How is it possible to not lose that message but some how to put it back on the queue waiting for it to be processed again?
Also how is it possible to make sure that the service is robust in such a way that it will spawn itself again?
I'd appreciate any advice and details here. There is quite alot to take in and WCF/MSMQ exposes quite alot of options.
Your assumption:
MSMQ I guess will be hosted on a seperate server.
is incorrect. MSMQ is installed on all machines which want to participate in message queuing.
There will be 2 WCF services. One for incoming messages and the other
for outgoing messages
In the most typical configuration, the destination queues are local to the listening service.
For example, your ServiceA would have a local queue from which it reads. ServiceB also has a local queue from which it reads. If ServiceA wants to call ServiceB it will put a message into ServiceB's local queue.
I understand with the right configuration, we can have the system so
that it can be transactional (no messages are ever lost)
This is correct. This is because MSMQ uses a messaging pattern called store-and-forward. See here for an explanation.
Essentially the reason it is safe to assume no message loss is because the transmission of a message from one machine to another actually takes place under three distinct transactions.
The first transaction: ServiceA writes to it's own temporary local queue. If this fails the transaction rolls back and ServiceA can handle the exception.
Second transaction: Queue manager on ServiceA machine transmits message to Queue manager on ServiceB machine. If failure then message remains on temporary queue.
Third transaction: ServiceB reads the message off local queue. If ServiceB message handler method throws exception then transaction rolls message back to local queue.
The applications/services will be multithreaded to process messages
This is fine except if you require order to be preserved in the message processing chain. If you need ordered processing then you cannot have multiple threads without implementing a re-sequencer to reapply order.
I thought that MSMQ can be hosted seperately and have x servers share
that queue?
All servers which want to participate in the exchange of messages have MSMQ installed. Each server can then write to any queue on any other server.
The reason for my thinking was because what if the server goes down?
Then how will the messages get sent/received into MSMQ
If the queues are transactional then that means messages on them are persisted to disk. If the server goes down then when it comes back up the messages are still there. While a server is down it obviously cannot participate in the exchange of messages. However, messages can still be "sent" to that server - they just remain local to the sender (in a temporary queue) until the destination server comes back on-line.
so by having one central MSMQ server (and having it mirrored/failover)
then there will be guarentee of uptime
The whole point of using message queueing is it's a fault-tolerant transport, so you don't need to guarantee uptime. If you have a 100% availability then there would be little reason to use message queuing.
how will WCF be notified of messages that are incoming?
Each service will listen on its own local queue. When a message arrives, the WCF runtime causes the handling method to be called and the message to be handled.
how will the service be notified of failures of sending messages
If ServiceA fails to transmit a message to ServiceB then ServiceB will never be notified of that failure. Nor should it be. ServiceA will handle the failure to transmit, not ServiceB. Your expectation in this instance creates a hard coupling between the services, something which message queueing is supposed to remove.
MSMQ can store messages even if temporary shutdown the service or reboot computer.
Main goal of WCF is transport message from source to destination. Doesn't matter what is the transport. In your case MSMQ is transport for WCF and not obvious to have online / available both client and service simultaneously. But when message is received, it's your responsibility to correctly process it, despite what transport was used to send message.