My web app communicates to my mobile app via ActiveMQ durable topics via MQTT (Paho). To deliver messages to the mobile device a separate topic is created for every mobile user. All messages from mobile devices are directed to a single topic.
This configuration seem to work fine, but every once a while something strange happens, the messages are not delivered to a device. Here is what's happening.
The user specific topic for messages sent to the device is dynamically created and looks fine in the console.
The messages sent to the topic seem to be enqueued fine - the count in the console is incremented
The client seems to be active - at least the console shows that the topic has an active subscriber
No messages are delivered to the client - The dequeued counter is stuck at 0
while the topic for this particular user is not working, other topics for other users work fine - messages are delivered as expected
Deleting topic for the user from the console and having it recreated does not work. The topic disappears from the console and reappears after it is recreated with the same counters it had before deleting. The message delivery still does not work
Restarting the broker fixes the problem
Any thoughts?
Related
I am facing a weird problem with connecting Rabbitmq from UI. I use 'xhr-polling' only due to some reason to connect rabbitmq queue from UI and it works fine for quite sometime when user lands on page but it disconnects with rabbitmq at some point without any error.
I have put in some logic to reconnect, once it is disconnected and as per the log it seems it connects but when I look at the RabbitMQ there is no client connected to it. However browser console (connected to server RabbitMQ/3.6.10) and keeps sending calling xhr_send?t=[random-key] and xhr?=[random-key] gets response 204 or 200 (As per developer tool).
When I refresh the whole page, it connects back again fine and see rabbitmq client queue as well something like (connected to server RabbitMQ/3.6.10).
Technology stacks are : Sockjs + Stompjs + RabbitMQ with Stomp plugin
So in summary reconnect logic shows it is connected but as per rabbitmq there is no subscribed client. Normally I see something like this stomp-subscription-rIUXo4Yvmilga2w3g5Lu6g as queue name when connected.
I have a webpage connecting to a rabbit mq broker using javascript/websockets that are exposed by a spring app deployed in tomcat. Messages are produced 1 per second by an external application and are rendered on the webpage. The javascript subscription is durable.
The issue I'm experiencing is that when the network connection is broken on the javascript client for a period of time (say 60 seconds), the first ~24 seconds of messages are missing. I've looked through the logs of the app deployed in tomcat and the missing messages seem to be up until the following log statement:
org.springframework.messaging.simp.stomp.StompBrokerRelayMessageHandler - DEBUG - TCP connection to broker closed in session 14
I think this is the point at which the endpoint realises the javascript client is disconnected and decides to close the connection to the broker resulting in future messages queueing up.
My question is how can I ensure that the messages between the time the network is severed and the time the endpoint realises the client is disconnected are not lost? Should the endpoint put the messages back on the queue somehow? Maybe there's a way to make it transactional?
Thanks in advance.
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
Your Tomcat application should not acknowledge messages from RabbitMQ until it confirms that your Javascript client has received them. This way, any messages that aren't ack-ed by the JS client won't be ack-ed by Tomcat, and RabbitMQ will re-deliver them.
I don't know how your JS app and Tomcat interact, but you may have to implement your own ack process there.
I'm using celery 3.0.18 with RabbitMQ 3.0.2. I have a task sent to another application by using celery.send_task, and I can see the send_task call in my logs, I can see the packets leaving the worker instance, and I can see the packets reaching the RabbitMQ instance when I call tcpflow -ce -i any port 5672, however, only the first message gets to the queue. They all have the same routing key, I tried recreating the exchange and bindings, and even a new RabbitMQ instance, and nothing seems to work. This used to work fine for months, until we had to rebuild the RabbitMQ from scratch after a crash in our AWS infrastructure. Strangely, I have the exact same setup working on other application, using the same broker and the same exchange, binding and queue, and it works perfectly there. Also, it works when I send the messages to the same exchange using the same call from a management script, running from the shell on the same instance, but it doesn't work when it's sent from the celery task in the worker process.
Any ideas on what the problem might be?
Eventually, I figured what's wrong, but it's not clear if this is the expected behavior, a celery bug, or a RabbitMQ bug.
What happens is that besides our application tasks, I have a custom logging handler used to send logs to a central location using RabbitMQ, using celery.send_task. This logging handler sends messages to an exchange named application.logger, with a routing key like application.logger.info, application.logger.warning, etc, and have bindings to route some logging levels to specific queues. This exchange, bindings and queues were created directly in RabbitMQ and not defined in Celery routes.
When the worker tries to send a message to this exchange and it doesn't exist, Celery would log a 404 NOT_FOUND error. After that, tasks sent to other exchanges using the same connection weren't delivered. They were sent by the worker instance, we could see the packets arriving and the RabbitMQ management screen for that connection even shows the data arriving from the client in kb/s, but no messages were delivered.
I've got an iOS application which uses a STOMP Client to talk to RabbitMQ. The application loads a lot of state during startup, and then keeps that state in sync by receiving updates published on STOMP. Of course, if it loses its connection, it can no longer be sure it's in sync, and therefore has to re-load that large initial blob. Any kind of network interruption triggers this behavior and makes my customers sad.
There are a lot of big-picture ways to fix this (and I'm working on them) but in the meantime, I'm trying to use persistent queues to solve this problem. The idea is that the server will create a queue, bind it to the appropriate topics, and then start building the large startup bundle. When finished, it will hand everything off to the client. The client will set itself up with the startup bundle, open a subscription to the queue, and then process any updates which happened while the server was getting things ready. Similarly, if the client should become disconnected, it can simply reconnect and resume reading the messages it finds in the queue.
My problem is that while the client successfully receives messages sent after it connects, if there were any messages in the queue before it connected, they are not read. Likewise, if the client becomes disconnected, when it reconnects, it won't see any messages which arrived while it was away.
Can anyone suggest how I might get the client to be able to read those missing messages?
It turns out what was happening was that the STOMP adapter was consuming the messages but failing to deliver them. Thus, when the client reconnected, it wouldn't have any messages waiting for it.
To fix the problem, I changed the "ack" setting on the subscribe request to "client", meaning that STOMP shouldn't consider the message delivered until the client sends back an ACK frame. By changing my client appropriately, messages now get delivered even after the client has been away.
Live tiles are able to receive push notifications without the associated metro app needing to be running.
However I believe that the app must have run at least once in order for the app to acquire a notification channel and subscribe to a notification server, passing the channel to the server.
My question is -
What happens if the server cuts off the client? If the user turns off their computer I presume the server would start receiving delivery failure errors. The server might then cut off the client.
But what happens when the user turns their computer back on? Is the tile now disconnected until the user starts the app again and it resubscribes with the server for notifications?
Or is there a way for the tile to resubscribe automatically on start up without the app having to run?
The push notifications are not sent directly to the client; they're sent via the Windows Notification service in the cloud. This means your service will be able to just send them. The WNS service will do the right thing with notifications when the machine comes out of sleep / reconnects to the network.
http://msdn.microsoft.com/en-us/library/windows/apps/hh913756.aspx has a overview of the service side of notifications.
It's important to note that the tile channel expires after 30 days, and will need to be (programmatically) renewed. The guidance is that you should renew when the app runs to make sure it doesn't expire.
The only thing I can't seem to locate in the documentation is how many push notifications are queued on the client - I suspect that for a given tag notification, only one is kept.
Maybe another way to think about this is with the bad notification -- e.g a "new items" count. If you push this number while the device is disconnect from the network (off, driven over etc), then your service will succeed in sending the notification, and when that machine reconnects, it will seamlessly see the badge update.
You should handle that in your code that when your clients from the server went offline then you should remove them and disconnect them, the client side will only receive the cached values in the live tiles.
If they went back on, then you should also handle it in your server side to push the new notification data.
Just a quick tip: If you are using WCF as your service, you might want to check the Announcement Service Class there you can handle your clients online/offline scenarios.