Multiplexing RabbitMQ message - rabbitmq

For example i have 4 sources which publish meterics.
I would like to multiplex / merge all theses messages in one queue/exchange
--------+----+----+----+----+ -------+---------+----+---------+---------+
Source1 | M1 | M2 | M3 | | => Result | M1 | M4 | M2 | M3 | M6 | M5 | M7 |
Source2 | M4 | | | M5 |
Source3 | | | M6 | |
Source4 | | | | M7 |
For each queue:
* Read one message
* Publish message to the Result queue
Is there a "native" way to do this in RabbitMQ or should i write my own Consumer/Publisher ?
EDIT 1
Some example to clarify, let's say after some time I have
Processing "window"
+-+
Source1 |X|XXXXXXXXXXXXX
Source2 |Y|YYYYYYY
Source3 |Z|ZZZZZZZZZZ
Source4 |W|WW
+-+
And then later
Processing "window"
+-+
Source1 XXX|X|XXXXXXXXXX
Source2 YYY|Y|YYYY
Source3 ZZZ|Z|ZZZZZZZ
Source4 WWW| |
+-+
And then later
Processing "window"
+-+
Source1 XXXXXXXXX|X|XXXX
Source2 YYYYYYYY | |
Source3 ZZZZZZZZZ|Z|Z
Source4 WWW | |
+-+
The result consuming order will be:
X Y Z W X Y Z W X Y Z W X Y Z X Y Z X Y Z X Y Z X Y Z X Z X Z X Z X X X
X,Y,Z,W then
X,Y,Z,W then
X,Y,Z,W then
X,Y,Z
...
X,Z
...
This way, even if a source is "spamming" all other messages from other sources have a chance to be consumed.
For technical/financial reasons I need to consume only 1 message a time.
The consumer is way slower than the producers but the producers publish a lot but occasionaly.
If each source published to an exchange bound to the same queue, the result might be XXXXXXXXXXXXXX YYYYYYYY ZZZZZZZZZZZ WWW or
XXXXX Y XXXXX YYY XXX YYYY ZZZZZZZZZZZ WWW (depending on the publish rate of each source)

I think what you want can be achieved simply by running a single script that subscribes to all the queues.
The key requirement is to use a single application thread to handle all messages, regardless of which queue they arrive from. What that looks like will vary depending on what language and client library you're using - if you're using PHP, you'd have to really go out of your way not to be single-threaded, but maybe there are some client libraries that assume each callback is on a separate worker thread, and you'll need some shared resource for them to block on.
In terms of the actual RabbitMQ side of things, you will need to:
register a subscription for the server to push messages to you, with basic.consume; this is generally recommended over explicitly polling with basic.get anyway
use a single "channel" for all the basic.consume calls
use manual acknowledgements so that messages remain in the queue until your process has finished
set a per-queue prefetch limit of 1 with basic.qos
If you have 4 queues, A, B, C, and D, which have varying amounts of messages when you start the consumer:
When you first subscribe, the prefetch limit will mean that one message from each queue will be sent to the channel; call them A1, B1, C1, and D1
The client library will raise an asynchronous event in your application for each of these in turn
Your single worker thread will handle the first of these events, and start processing message A1
Until you manually acknowledge that message, no other messages can arrive
Once you acknowledge the first message (A1), a new message can be pre-fetched from that queue (A2)
Meanwhile, your worker thread will unblock and handle the next event which was already raised, for message B1
Only once you've processed the pending events for B1, C1, and D1 will the worker thread see the event for message A2
As long as the queues have messages waiting, they will be processed in a round-robin fashion. Even if all but one of the queues become empty, they will slot back into rotation as soon as a message arrives, because only one message from the busy queue will have been pre-fetched, the rest will just be waiting on the RabbitMQ server.

Related

Can RTMP multiplex messages within a single chunk stream?

Reading the RTMP specification, in an effort to write a rudimentary RTMP server, I can't tell whether multiple messages (message stream id) can be sent over the same chunk stream (chunk stream id).
Section 5.3.2 shares two examples: one where multiple messages with the same stream id are sent sequentially over multiple chunks for a single chunk stream id and one where a single message is sent over multiple chunks for a single chunk stream id.
But there's no example demonstrating multiple messages with different stream ids being sent concurrently over multiple chunks for a single chunk stream id. I can't find anything that would prevent this, but I'd like confirmation.
For example, say you have two messages like in example 2
+-----------+-------------------+-----------------+-----------------+
| | Message Stream ID | Message TYpe ID | Time | Length |
+-----------+-------------------+-----------------+-----------------+
| Msg # 1 | 27 | 9 (video) | 1000 | 307 |
+-----------+-------------------+-----------------+-----------------+
| Msg # 2 | 42 | 9 (video) | 1000 | 197 |
+-----------+-------------------+-----------------+-----------------+
Can the RTMP client send the following sequence of chunks?
Type 0 message for 27
Type 0 message for 42
Type 3 message for 27
Type 3 message for 27 (completely sent Msg # 1)
Type 3 message for 42 (completely sent Msg # 2)
In other words, is chunk 3 expected to use the header from 1 or from 2 (ie. based on message stream id)?

What is the meaning of the "Load" column in Apache balancer-manager?

I've set up the Apache (2.4) load-balancer which is working okay. To monitor its performance, I enabled the balancer-manager handler, which shows the status of the balancers.
I noticed a "Load" column, which was not present in version 2.2, with a value that may be negative, but I don't understand its meaning nor I was able to find documentation relative to this.
Can anyone explain the meaning of that value or point me to the right documentation?
I now understood, how the calculation of "Load" works. Here is a I think more simpler example than on the apache documents page.
Let's say we have 3 worker and a configured load factor of 1.
1) Start
a | b | c
--+---+---
0 | 0 | 0
add the load factor of 1 to all workers
a | b | c
--+---+---
1 | 1 | 1
now select the one with highest value --> a and decrease by the sum of the factor of all (=3) - this is the selected worker
a | b | c
---+---+---
-2 | 1 | 1
2) next round, add again 1 to all
a | b | c
---+---+---
-1 | 2 | 2
now select the one with highest value --> b and decrease by the sum of the factor of all (=3) - this is the selected worker
a | b | c
---+----+----
-1 | -1 | 2
3) next round, add again 1
a | b | c
---+----+----
0 | 0 | 3
now select the one with highest value --> c and decrease by the sum of the factor of all (=3) - this is the selected worker
a | b | c
---+----+----
0 | 0 | 0
startover again :)
I hope this helps others.
The Load value is populated by lbstatus based on this line of code:
ap_rprintf(r, "<td>%d</td><td>", worker->s->lbstatus);
in https://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy_balancer.c?view=markup#l1767 (line might changed when the code modified)
Since your method is by request, lbstatus is specified by mod_lbmethod_byrequests which define:
lbstatus is how urgent this worker has to work to fulfill its quota of
work.
Details on the algorithm can be found here: https://httpd.apache.org/docs/2.4/mod/mod_lbmethod_byrequests.html
i too want to know to description for others column like BUSY, ELECTED etc.. my LB has BUSY over 100 already.. i though BUSY should not exceed 100 ( as in 100% server busyness or something )

ActiveMQ ActiveMQ.Advisory.TempQueue getting bigger and bigger

Problem,
on ActiveMQ for some reasons(I dont know why) the ActiveMQ.Advisory.TempQueue is getting bigger and bigger (1GB per day).
here is a snapshot:
Name Producer # Consumer # Enqueue # Dequeue # Memory % Dispatch # Always retroactive Average blocked time Average enqueue time Average message size Blocked producer warning interval Blocked sends Dlq Expired count Forward count In flight count Max audit depth Max enqueue time Max message size Max page size Max producers to audit Memory limit Memory usage byte count Memory usage portion Min enqueue time Min message size Options Prioritized messages Producer flow control Queue size Slow consumer strategy Store message size Total blocked time Use cache Object name
ActiveMQ.Advisory.TempQueue | 0 | 816 | 187550135 | 0 | 0 | 187836323 | FALSE | 0 | 0.3694736 | 1024 | 30000 | 0 | FALSE | 0 | 0 | 187836323 | 2048 | 1233 | 1024 | 200 | 1024 | 668309914 | 0 | 1 | 0 | 1024 | FALSE | TRUE | 0 | 0 | 0 | TRUE | org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Topic,destinationName=ActiveMQ.Advisory.TempQueue
Any idea?
Advisory Topics in ActiveMQ don't accumulate data, they are Topics and as such when there are no consumers on the Topics the messages sent to them are dropped. If you have a consumer on the advisory Topic then message pass through it but are not stored in the persistent storage on the broker. The stats can sometimes be deceiving given the enqueue count keeps ticking up.
Without knowing more about what you are seeing there's not much more help that can be offered.
If you are seeing growth in the KahaDB logs then it is unrelated to your Advisory Topics as I've stated they don't store messages ever so there is something else going on. There's some nice instructions on the ActiveMQ WebSite on how to take a look at what is keeping the KahaDB journal files alive which you should use to help debug your issue.

Average wait time for pre-emptive priority scheduling

Given the following table for calculating and average waiting time for the processes for priority based preemptive scheduling.
Process Burst Time Priority
P1 10 3
P2 1 1
P3 2 4
P4 1 5
P5 5 2
The gantt chart would be as follows:
| P2 | P5 | P1 | P3 | P4 |
0 1 6 16 18 19
I have following questions:
1) Is the turn around time = 19 units?
2) How do i calculate average waiting time? Is there a formula?
3) What if few processes have the same priority?
I am new to OS. I have viewed some other similar questions but I did not get exactly how to do it.
Given the data,before you have to implement priority based preemptive scheduling, you should know the following facts :-
priorities are usually numeric over a range
high numbers may indicate low priority (system dependent)
associate a priority with each process, allocate the CPU to the process with the highest priority
any 2 processes with the same priority are handled FCFS
Proceeding with this much of knowledge,the required Gantt chart would be the same as what you've drawn:-
| P2 | P5 | P1 | P3 | P4 |
0 1 6 16 18 19
1) Is the turn around time = 19 units?
No, the turnaround time will be 16 + 1 + 18 + 19 + 6 = 60.
Average turnaround time = 60 / 5 = 12.
2) How do i calculate average waiting time? Is there a formula?
Average waiting time is defined as the sum of total time waited before starting of the processes divided by the total number of processes.
Here, average waiting time = (6 + 0 + 16 + 18 + 1) / 5 = 41 / 5 = 8.2.
3) What if few processes have the same priority?
If the few processes will have same priority then the scheduling would be handled using First-Come First-Serve (FCFS) as mentioned in the 4th point above. So, everywhere including Gantt chart, the process coming first will be scheduled first and the other similar-priority process would be scheduled late as it came arrived late.
I hope it is crystal clear from my steps and doesn't need any further explanation.

Priority Preemptive Scheduling

When using Priority Preemptive Scheduling, does a higher priority yield way to a process with a lower priority but with a shorter burst time?
For example, if I had:
Arrival Time Burst Time Priority
P1 0 5 3
P2 2 6 1
P3 3 3 2
Would the Gannt chart look like this?
| P1 | P2 | P3 | P1 |
0 2 8 11 16
Priority Scheduling always selects the process(es) with the highest priority currently ready to run. If there is more than one process having the currently highest priority, you need a second scheduling algorithm to choose among these processes. Non-preemptive Priority Scheduling only selects a new process to run if the running process finished its work or yields (voluntarily) to the scheduler.
Preemptive Priority Scheduling is the same algorithm but if a new process having a higher priority than the currently running process arrives, it gets selected immediately. The new process has not to wait until the currently running process finishes or yields.
In your sample, the Gantt chart for Preemptive Priority Scheduling and 3 being the highest and 1 the lowest priority would look like:
| P1 | P3 | P2 |
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
| p1 | p2 | p3 | p1 |
0....2....8....11...14
taking 1 as the highest priority.
|p1 |p2 |p3 |p1 |
0 2 8 11 14
because preemptive approach will preempt if the priority of the newly-arrived process is higher than the priority of currently running process..