Is it possible to declare a queue in pika (python-pika) as a circular queue ? If yes, how ?
I mean by circular queue (or a ring) is a queue when a message is selected it will be re-inserted at the end of the queue, instead of removing it from the queue. For example, if the queue contains: msg_3, msg_2, msg_1, then after a consumer get msg_1, the latter will be inserted at the end, so the queue will be: msg_1, msg_3, msg_2 (instead of msg_3, msg_2)
Edit: As proposed by IMSoP, I can make the consumer republish every consumed message (for example, at the end of the callback function).
Related
I have a real-time RabbitMQ queue that I'm running. I'd like to consume the most recent entry, ignoring all others.
Better yet, is it possible to have a fanout exchange with a singleton queue size?
Yes, this can be done by specifying the maximum queue length limit when declaring the queue.
As the documentation states,
The maximum length of a queue can be limited to a set number of messages, or a set number of bytes (the total of all message body lengths, ignoring message properties and any overheads), or both.
The default behaviour for RabbitMQ when a maximum queue length or size is set and the maximum is reached is to drop or dead-letter messages from the front of the queue (i.e. the oldest messages in the queue). To modify this behaviour, use the overflow setting described below.
If you're using Java, you would do the following:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length", 1);
channel.queueDeclare("myqueue", false, false, false, args);
How can I create a custom serial queue that runs at high priority?
Right now I'm using myQueue = dispatch_queue_create("com.MyApp.MyQueue", NULL); but this doesn't seem to allow for setting a priority?
Create a serial queue, then use dispatch_set_target_queue() to set its target queue to the high priority queue.
Here's how:
dispatch_set_target_queue(myQueue, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0));
Now myQueue should run serially with high priority.
Here's another SO answer if you want to know more.
The dispatch_queue_attr_make_with_qos_class function may be new since the accepted answer was posted, but something like:
dispatch_queue_attr_t priorityAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INITIATED, -1
);
myQueue = dispatch_queue_create("com.MyApp.MyQueue", priorityAttribute);
could give the queue a high priority ('quality of service'). There is a higher QOS class, but QOS_CLASS_USER_INITIATED is equivalent to DISPATCH_QUEUE_PRIORITY_HIGH.
is it a requirement that you have a custom queue? If not, you could look at dispatching a block to the high priority global queue, which you can retrieve using:
dispatch_queue_t q = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0)
keep in mind that this is the global queue so it may impact other concurrent operations.
There is one Producer and n consumers.
producer us assigning n jobs to n consumers and calling pthread_cond_wait() n times to wait for the assigned job to be completed by consumers.
Each Consumer after consuming job calls pthread_cond_signal() to notify the producer.
My question is "Will n calls to pthread_cond_signal() by consumer makes the producer to come out of pthread_cond_wait() n times? Or is there any case where multiple signals be merged into single signal so that pthread_cond_wait() comes out less than n times?
If the producer isn't actually waiting inside a call to pthread_cond_wait() when a consumer thread calls pthread_cond_signal(), then that signal will get 'lost' (ie., if the producer thread later rolls into the pthread_cond_wait(), it will block until another signal is sent).
That is why condition variables must be used in conjunction with some other "boolean predicate" that is checked while holding the mutex used with the condition variable. That predicate is the actual final word on whether or not the thread deciding whether or not to wait should wait. Another reason that the predicate is the final word is that a thread blocked in pthread_cond_wait() can be spuriously awakened.
From the POSIX docs on pthread_cond_wait():
When using condition variables there is always a Boolean predicate
involving shared variables associated with each condition wait that is
true if the thread should proceed. Spurious wakeups from the
pthread_cond_timedwait() or pthread_cond_wait() functions may occur.
Since the return from pthread_cond_timedwait() or pthread_cond_wait()
does not imply anything about the value of this predicate, the
predicate should be re-evaluated upon such return.
I'm trying to use ActiveMQPrefetchPolicy but cannot quite understand how to use it.
I'm using queue, there are 3 params that I can define for PrefetchPolicy:
queuePrefetch, queueBrowserPrefetch, inputStreamPrefetch
Actually I don't get the meaning of queueBrowserPrefetch and inputStreamPrefetch so I do not know how to use it.
I assume that you have seen the ActiveMQ page on prefetch limits.
queueBrowserPrefetch sets the maximum number of messages sent to a
ActiveMQQueueBrowser until acks are received.
inputStreamPrefetch sets the maximum number of messages sent
through a jms-stream until acks are received
Both queue-browser and jms-stream are specialized consumers. You can read more about each one of them but if you are not using them it won't matter what you assign to their prefetch limits.
I am fairly new to pthread programming and am trying to get my head around cond_signal & mutex_lock. I am writing a sample program which has One producer thread and Two consumer threads.
There is a queue between producer and the first consumer and a different queue between producer and the second consumer. My producer is basically a communication interface which reads packets from the network and based on a configured filter delivers the packets to either of the consumers.
I am trying to use pthread_cond_signal & pthread_mutex_lock the following way between producer and consumer.
[At producer]
0) Wait for packets to arrive
1) Lock the mutex pthread_mutex_lock(&cons1Mux)
2) Add the packet to the tail of the consumer queue
3) Signal the Consumer 1 process pthread_cond_signal(&msgForCons1)
4) Unlock the mutex pthread_mutex_lock(&cons1Mux)
5) Go to step 0
[At consumer]
1) Lock the mutex pthread_mutex_lock(&cons1Mux)
2) Wait for signal pthread_cond_wait(&msgForCons1,&cons1Mux)
3) After waking up, read the packet
4) Delete from queue.
5) Unlock the mutex pthread_mutex_unlock(&cons1Mux)
6) Goto Step 1
Are the above steps correct? If a switch happens from the consumer thread exactly after step 5 to the producer thread, then is it possible that the producer may signal a packet is waiting even though the consumer hasn't yet started listening for that signal. Will that cause a "missed signal"?
Are there any other problems with these steps?
Yes, you're correct you could have a problem there: if there are no threads waiting, the pthread_cond_signal is a no-op. It isn't queued up somewhere to trigger a subsequent wait.
What's you're supposed to do is, in the consumer, once you've acquired the mutex, test whether there is any work to do. If there is, well, you have a mutex; take ownership, update state, and do it. You only need to wait if there is nothing to do.
The cannonical example is:
decrement_count()
{ pthread_mutex_lock(&count_lock);
while (count == 0)
pthread_cond_wait(&count_nonzero, &count_lock);
count = count - 1;
pthread_mutex_unlock(&count_lock);
}
increment_count()
{ pthread_mutex_lock(&count_lock);
if (count == 0)
pthread_cond_signal(&count_nonzero);
count = count + 1;
pthread_mutex_unlock(&count_lock);
}
Note how the "consumer" decrementing thread doesn't wait around if there's something to decrement. The pattern applies equally well to the case where count is replaced by a queue size or the validity of a struct containing a message.