I'm trying to use ActiveMQPrefetchPolicy but cannot quite understand how to use it.
I'm using queue, there are 3 params that I can define for PrefetchPolicy:
queuePrefetch, queueBrowserPrefetch, inputStreamPrefetch
Actually I don't get the meaning of queueBrowserPrefetch and inputStreamPrefetch so I do not know how to use it.
I assume that you have seen the ActiveMQ page on prefetch limits.
queueBrowserPrefetch sets the maximum number of messages sent to a
ActiveMQQueueBrowser until acks are received.
inputStreamPrefetch sets the maximum number of messages sent
through a jms-stream until acks are received
Both queue-browser and jms-stream are specialized consumers. You can read more about each one of them but if you are not using them it won't matter what you assign to their prefetch limits.
Related
I want to stream real-time sensor data(webcam, laser point cloud, etc.) from one robot to multiple observers.
In this use case, only the newest data is useful. For example, when a new frame of point cloud arrives, the older ones will be useless.
Redis has nice publisher/consumer support, but it has buffers according to (Redis Pubsub and Message Queueing).
So are there better alternatives? Something like ROS's publishers/subscribers. They have a message queue size parameter.
/**
* The subscribe() call is how you tell ROS that you want to receive messages
* on a given topic.
*
* The second parameter to the subscribe() function is the size of the message
* queue. If messages are arriving faster than they are being processed, this
* is the number of messages that will be buffered up before beginning to throw
* away the oldest ones.
*/
ros::Subscriber sub = n.subscribe("chatter", 1000, chatterCallback);
Maybe you can use redis list data structure for your purpose, like a queue. The list data structure in redis is made with linked list and adding a new item is O(1). Whenever your robot produces data it can put it in a list with LPUSH command, and when you want to get the latest item from the list use LRANGE "key-name" 0 0. This command will retrive the latest pushed item. Also if you want to not accumulate the data in queue, you may try to use LTRIM before LRANGE to maintain the latest records. For example LTRIM "key-name" 0 10 will keep the records of last 10 elements. This trim interval should be set according to your observer processing speeds. ref: https://redis.io/docs/data-types/lists/
I am using Redis streams and need to block my clients until there are at least say X number of messages in the stream and return when it reaches that X count..
Is there any way to achieve this?
EG: XREADGROUP GROUP G1 C2 COUNT 10 BLOCK 0 STREAMS L > until all 10 messages have arrived in the stream key
COUNT specifies the maximum number of elements to return per stream, if there are any. If the streams are empty (and the BLOCK option was used), the consumer blocks. Once there's a single incoming message, the blocking is released with that message. So no, you can't block until COUNT messages are read. But you can probably group messages on the application level.
See XREAD for more details.
1. Is it possible to receive multiple messages in one receive call?
Sender pseudo-code:
target = ("xxx.xxx.xxx.xxx", 1234)
sender = new_udp_socket()
sender.send("Hello", target)
sender.send("World", target)
Receiver pseudo-code:
receiver = new_udp_socket()
receiver.bind("", 1234)
while true
data = receiver.recvfrom(512)
print(data)
Is it possible that the receiver will receive "HelloWorld" in one receive call instead of "Hello" and "World" separately?
I have been told that it is possible, but I'd like to make sure.
2. If it is possible to receive multiple messages in one receive call, how do I ensure that my code processes both messages separately?
I've been thinking about this but couldn't come up with any solution.
My first idea was that I would add a byte at the beginning of the send call stating the length of the message.
I don't believe this would be reliable either because if too much data is in the receivers buffer then the beginning (the message length) may be cut off and therefore my program would fail.
Thanks for any help!
After much research I have found an answer to my question.
One recvfrom call will only ever receive one sendto call.
Sources:
https://stackoverflow.com/a/8748884/1541397
https://stackoverflow.com/a/26185032/1541397
I have a real-time RabbitMQ queue that I'm running. I'd like to consume the most recent entry, ignoring all others.
Better yet, is it possible to have a fanout exchange with a singleton queue size?
Yes, this can be done by specifying the maximum queue length limit when declaring the queue.
As the documentation states,
The maximum length of a queue can be limited to a set number of messages, or a set number of bytes (the total of all message body lengths, ignoring message properties and any overheads), or both.
The default behaviour for RabbitMQ when a maximum queue length or size is set and the maximum is reached is to drop or dead-letter messages from the front of the queue (i.e. the oldest messages in the queue). To modify this behaviour, use the overflow setting described below.
If you're using Java, you would do the following:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length", 1);
channel.queueDeclare("myqueue", false, false, false, args);
I use ordered set to true, however when many (1000 or more) messages are sent in a short period of time (< 1 second) the messages received are not all received in the same order.
rtcPeerConnection.createDataChannel("app", {
ordered: true,
maxPacketLifeTime: 3000
});
I could provide a minimal example to reproduce this strange behavior if necessary.
I also use bufferedAmountLowThreshold and the associated event to delay when the send buffered amount is too big. I chose 2000 but I don't know what the optimal number is. The reason I have so many messages in a short period of time is because I don't want to overflow the maximum amount of data sent at once. So I split the data into 800 Bytes packs and send those. Again I don't know what the maximum size 1 message can be.
const SEND_BUFFERED_AMOUNT_LOW_THRESHOLD = 2000; //Bytes
rtcSendDataChannel.bufferedAmountLowThreshold = SEND_BUFFERED_AMOUNT_LOW_THRESHOLD;
const MAX_MESSAGE_SIZE = 800;
Everything works fine for small data that is not split into too many messages. The error occurs randomly for big files only.
In 2016/11/01 , there is a bug that lets the dataChannel.bufferedAmount value change during the event loop task execution. Relying on this value can thus cause unexpected results. It is possible to manually cache dataChannel.bufferedAmount, and to use that to prevent this issue.
See https://bugs.chromium.org/p/webrtc/issues/detail?id=6628