I send messages C2D, it has a TTL of 90 seconds (default). I have almost 10k devices.
I want to know what all messages were not delivered and towards which device.
Do we have a concept like a dead letter queue with IOT hub, How can I implement this?
Related
per https://firebase.google.com/docs/cloud-messaging/concept-options#device_throttling, it says...
You can send up to 240 messages/minute and 5,000 messages/hour to a single device. This high threshold is meant to allow for short term bursts of traffic, such as when users are interacting rapidly over chat. This limit prevents errors in sending logic from inadvertently draining the battery on a device.
does this mean a device can only receive 240 messages / minute?
or does it mean it can receive 240 messages sent by a particular device?
say, 4 other devices can send 240 messages to a device?
This is a limit on the number of downstream messages that can be sent to a device for the entire project.
Sending messages to a device should only happen from a trusted environment (your development machine, the Firebase console, a server you control, Cloud Functions). There is no ability to send downstream messages with the Firebase client-side SDKs.
I am creating a bulk video processing system using spring-boot. Here the user will provide all the video related information through an xlsx sheet and we will process the videos in the backend. I am using the Rabbitmq for queuing up the request.
Let say a user has uploaded a sheet with 100 rows,then there will be 100 messages in the Rabbitmq queue. In the back-end, we are auto-scaling the subscribers (servers). So we will start with one subscriber-only and based on the load (number of messages in the queue) we will scale up to 15 subscribers.
But our producer is very fast and it allocating all the messages to our first subscriber (before other subscribers are coming up) and all our new subscriber are not getting any messages from the queue.
If all the subscribers are available before producer started pushing the messages then it is allocating the messages to all servers.
Please provide me a solution of how can our new subscribers pull the messages from the queue that were produced earlier.
You are probably being affected by the listener container prefetchCount property - it defaults to 250 with recent versions (since 2.0).
So the first consumer will get up to 250 messages when it starts.
It sounds like you should reduce it to a small number, even all the way down to 1 so only one message is outstanding at each consumer.
I have a queue producer(NON_PERSISTENT) and a queue consumer connected to my ActiveMQ server. If a producer sends a message to a queue while the consumer is not connected to ActiveMQ, it seems it is stored and delivered when the consumer is up and connected to ActiveMQ.
I want ActiveMQ to ignore the message if the consumer is not connected at the time of delivery. How can I achieve this?
Thanks in advance.
Use a topic instead of a queue - this is the default behaviour for topics (unless a durable subscriber is used).
Otherwise, for queues, you can set a message expiry when sending the message. It will be discarded if not read within that time frame. Make sure to set enough time frame so that clock sync issues between servers won't be a factor. Let's say 2 minutes or so.
Here's my current architecture
I have a bunch of IoT devices, that connects through raw duplex persistent TCP to 1 instances of my "worker" that is connected to a RabbitMQ Queue
My publisher publishes some messages that look like that
{
"iot_device_name" : "A",
"command" : "reboot"
}
The worker is then able to map the iot_device_name to the TCP socket.
All is working nice, but if we want to add HA and to scale out a bit, it would be better to have 4 instances of the worker. Load balancing the TCP question is not a problem (with HaProxy or Nginx).
Now the problem is on how to split the load on the Queue part, as the list of IoT devices handled by a worker is dynamic (i.e a device could disconnect and reconnect to an other worker)
So is there a way for a worker to say: "Hmmm no I can't treat this message because I don't know this device, give me an other" so that an other worker can then take it and handle it ?
Other information that may be of help:
the workers are all in the same network, that is also the same than the publisher
the numbers of workers is not dynamic and even if we extrapolate the number of devices for the next years, 8 workers would takes us VERY FAR, as it simply route message/transcode messages, so their cpu load is ridiculous.
So if I understand your architecture correctly, you have commands sent to your publisher on one side, which are pushed into rabbitmq.
On the consumer side, you have multiple workers, to which the messages are dispatched, and each worker has a bunch of devices connected to it.
If indeed this is your architecture, I'd propose the following for your rabbitmq configuration:
use a direct exchange
each worker has it's own queue (exclusive), and manages the bindings between the exchange and its queue dynamically:
each time a device connects to a worker, that worker adds a binding between its queue and the exchange with as routing key the identifier for the device
each time a worker detects that a device is not connected to it anymore, he removes the related binding from the rabbitmq configuration
related to the detection of disconnected devices, I'd expect it common that it's upon receiving a command to push to the device that a worker realize the device isn't connected to it anymore, in such cases in addition to adapting the bindings, the worker would republish the message to the same exchange with the same routing key, so that it can have another shot at being consumed by the proper worker
I'd also consider configuring a TTL on the queues, no point in consuming a message that's too old
The publisher will of course also need to alter the message, including the intended device identification as routing key
I hope the proposal here makes sense, there are a few other cases to be considered: alternate exchange to make sure we don't lose requests if there is a (short) period when the device hasn't reconnected to a worker and we get a command for it anyway, adding a property to a message republished to ensure we do not add an infinite loop in the system, ... but what is indicated above should be a reasonable starting point to achieve your goal
I am developing an app. and I am using activemq. Is there any way to do that one producer always send messages to one broker but on the opposite side there 3 consumers.Each consumer listens broker and can take any of message from queue.Is this possible?
I am using activemq for writing my app. logs to db.As u know writing logs to db is time taking process.That's why consumer is more and more slow than producer.For ex. I send 100.000 message(huge objects).Producer finishes sending messages in 20 mins.But When the producer finished, consumer has finished 4.000 message processing yet.
Yes, what you are describing is possible. In fact, you can have any number of consumers listening on a single queue. The messages are dispatched in a round-robin fashion between consumers.
What you should be aware of is that ActiveMQ performs much better sending small messages than large ones. If you need to send very large payloads (e.g. 100mb), you are far better off saving the message to a location that is accessible by both the producer and consumers (e.g. a network file system), and sending the location of the message instead. The consumer can then use that to read the message manually. This way you get a relatively small amount of traffic through the message broker.