Is there a way to filter the visible queues within the ActiveMQ queue overview?
We have a lot of different queues and thus the overview page is very full. It would be great to filter away not so important queues by default.
If you're talking about the web admin console configuration, then No - it's not built to scale with a lot of queues.
But then again, the console is a simple java web app , so you could easily implement a filter yourself if you really need it.
The most quick n dirty way I can think of is open webapps/admin/queues.jsp and add an JSP-if statement in the forEach-loop. It will be very hard coded, but is very quick and does not even require a server restart, just a simple browser reload.
<c:forEach items="${requestContext.brokerQuery.queues}" var="row">
<!-- Only show queues starting with Important. -->
<c:if test="${fn:startsWith(row.name,'Important.')}">
..
</c:if>
</c:forEach>
There are, of course, much more sophisticated and better ways to achieve this if you're up to the task.
Related
I've spent quite a bit of time trying to figure out whether I should use the RabbitMQ federation plugin or shovel.
Basically I have two microservices. I want one of them to send a message to another. Each microservice has a different rabbitMQ cluster, so I need to use Federation/shovel.
I read this post When to use RabbitMQ shovels and when Federation plugin? and still couldn't figure it out / make a decision.
I want to satisfy the following:
Loose coupling
Microservices don't know about each other -- I.e the first microservice emits a message saying "i'm done doing x". And the second microservice just listens to that 'event' and acts accordingly..
In the future I 'might' want to add more microservices, each with their own rabbitMQ cluster / vhost.
Based on this information - what do you recommend, shovel or federation?
Why not just have one cluster for everything? RabbitMQ is build for handling 10k+ exchanges and queues, actually there is no upper limit except memory or disk space. Setting up a cluster for each microservice is too much work and creates unnecessary overhead. Using vhost should also not be used for this, but for each business area.
I'm only using shovels and I use them to transfer messages from my production environment to test, so I can test with real data. It's very easy to setup with scripts. And yes, you should only do this with scripts. Using the UI is too slow.
I know this doesn't answer your question directly, but I hope it has given you some food for thought.
Happy messaging!
We are using Akka.Net and in some cases we need actors to communicate reliably while preserving order over a message queue (i.e. Oracle Advanced Queues or WebSphere MQ, but any message queuing system would work such as RabbitMQ).
We have various requirements why we are using the message queue, so the question isn't if we should be using this with Akka, the question is how.
How would we go about connecting the queue up to Akka so that it is as seamless as possible?
Is a a custom Mailbox the route to go down? Do we need to right a custom IMessageQueue implementation? Or maybe we need a custom router? Are there any specific tests we can run to be sure our Mailbox/IMessageQueue works well with Akka.Net?
EDIT:
Should we maybe looking to implement a custom Transport?
Can any pointers be offered on where to start?
In general implementing custom mailbox based on some reliable queue is not feasible solution - actually it has been already done on the Akka JVM side, and it failed all hopes.
One of the basic reasons is usually the misunderstanding of the basic idea - when people are talking about reliable delivery (that MQ-systems offers), what they really mean, is reliable processing. What if your messages has been send with 100% delivery ratio, but ultimately receiving actor/node has crashed while processing them? From the mailbox point of view everything went smooth...
For this reason, usually the way to go is a dedicated actor - or hierarchy of them - working as a gateway to external messaging system. This way you can not only send message them but also mark them as receive after explicit acknowledgement from successfully completed process. One of the examples may be akka-rabbitmq (written in Scala).
I am a newbie to real-time application development and am trying to wrap my head around the myriad options out there. I have read as many blog posts, notes and essays out there that people have been kind enough to share. Yet, a simple problem seems unanswered in my tiny brain. I thought a number of other people might have the same issues, so I might as well sign up and post here on SO. Here goes:
I am building a tiny real-time app which is asynchronous chat + another fun feature. I boiled my choices down to the following two options:
LAMP + RabbitMQ
Node.JS + Redis + Pub-Sub
I believe that I get the basics to start learning and building this out. However, my (seriously n00b) questions are:
How do I communicate with the end-user -> Client to/from Server in both of those? Would that be simple Javascript long/infinite polling?
Of the two, which might more efficient to build out and manage from a single Slice (assuming 100 - 1,000 users)?
Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )
I hope this isn't a crazy question and won't get flamed right away. Would love some constructive feedback, love this community!
Thank you.
Architecturally, both of your choices are the same as storing data in an Oracle database server for another application to retrieve.
Both the RabbitMQ and the Redis solution require your apps to connect to an intermediary server that handles the data communications. Redis is most like Oracle, because it can be used simply as a persistent database with a network API. But RabbitMQ is a little different because the MQ Broker is not really responsible for persisting data. If you configure it right and use the right options when publishing a message, then RabbitMQ will actually persist the data for you but you can't get the data out except as part of the normal message queueing process. In other words, RabbitMQ is for communicating messages and only offers persistence as a way of recovering from network problems or system crashes.
I would suggest using RabbitMQ and whatever programming languages you are already familiar with. Since the M in LAMP is usually interpreted as MySQL, this means that you would either not use MySQL at all, or only use it for long term storage of data, not for the realtime communications.
The RabbitMQ site has a huge amount of documentation about building apps with AMQP. I suggest that after you install RabbitMQ, you read through the docs for rabbitmqctl and then create a vhost to experiment in. That way it is easy to clean up your experiments without resetting everything. I also suggest using only topic exchanges because you can emulate the behavior of direct and fanout exchanges by using wildcards in the routing_key.
Remember, you only publish messages to exchanges, and you only receive messages from queues. The exchange is responsible for pattern matching the message's routing_key to the queue's binding_key to determine which queues should receive a copy of the message. It is worthwhile learning the whole AMQP model even if you only plan to send messages to one queue with the same name as the routing_key.
If you are building your client in the browser, and you want to build a prototype, then you should consider just using XHR today, and then move to something like Kamaloka-js which is a pure Javascript implementation of AMQP (the AMQ Protocol) which is the standard protocol used to communicate to a RabbitMQ message broker. In other words, build it with what you know today, and then speed it up later which something (AMQP) that has a long term future in your toolbox.
Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )
This is usually called RAD (rapid application design/development) and it is what I would recommend right now. This lets you build the proof of concept that you can use to work off of later to get what you want to happen.
As for how to talk to the clients from the server, and vice versa, have you read at all on websockets?
Given the choice between LAMP or event based programming, for what you're suggesting, I would tell you to go with the event based programming, so nodejs. But that's just one man's opinion.
Well,
LAMP - Apache create new process for every request. RabbitMQ can be useful with many features.
Node.js - Uses single process to handle all request asynchronously with help of event looping. So, no extra overhead process creation like apache.
For asynchronous chat application,
socket.io + Node.js + redis pub-sup is best stack.
I have already implemented real-time notification using above stack.
I'd like to have a real-time 'system status'/'activity monitor' console for my Twisted application.
The app is basically a protocol.ServerFactory which accepts connections performs different jobs.
Kind of like the twisted.manhole, I'm looking for the simplest way to create a admin application where I can check the current stats of my app.
The admin can be a simple ascii-based shell or html/json setup.
I'm aware that I could build this with a bunch of counters, a separate protocol for authenticating and monitoring these, but I'm thinking Twisted might already have such thing since it at least knows the number of connections, protocol types, etc etc.
Tips?
There's the unmaintained, slowly rotting twisted.internet.gladereactor. If you're using twistd, then you can use this trivally:
twistd --reactor debug-gui ...
If you're running the reactor directly yourself, then it's only slightly more effort:
from twisted.manhole import gladereactor
gladereactor.install()
from twisted.internet import reactor
...
The Inspect feature appears to have been broken for some time, but it will still show you a list of established connections and what state they are in, and it will also apparently give you a traffic log for each connection. Fixing Inspect may also be a fairly straightforward effort, in case you're looking for a little project. :)
On my team at work, we use the IBM MQ technology a lot for cross-application communication. I've seen lately on Hacker News and other places about other MQ technologies like RabbitMQ. I have a basic understanding of what it is (a commonly checked area to put and get messages), but what I want to know what exactly is it good at? How will I know where I want to use it and when? Why not just stick with more rudimentary forms of interprocess messaging?
All the explanations so far are accurate and to the point - but might be missing something: one of the main benefits of message queueing: resilience.
Imagine this: you need to communicate with two or three other systems. A common approach these days will be web services which is fine if you need an answers right away.
However: web services can be down and not available - what do you do then? Putting your message into a message queue (which has a component on your machine/server, too) typically will work in this scenario - your message just doesn't get delivered and thus processed right now - but it will later on, when the other side of the service comes back online.
So in many cases, using message queues to connect disparate systems is a more reliable, more robust way of sending messages back and forth. It doesn't work well for everything (if you want to know the current stock price for MSFT, putting that request into a queue might not be the best of ideas) - but in lots of cases, like putting an order into your supplier's message queue, it works really well and can help ease some of the reliability issues with other technologies.
MQ stands for messaging queue.
It's an abstraction layer that allows multiple processes (likely on different machines) to communicate via various models (e.g., point-to-point, publish subscribe, etc.). Depending on the implementation, it can be configured for things like guaranteed reliability, error reporting, security, discovery, performance, etc.
You can do all this manually with sockets, but it's very difficult.
For example: Suppose you want to processes to communicate, but one of them can die in the middle and later get reconnected. How would you ensure that interim messages were not lost? MQ solutions can do that for you.
Message queueuing systems are supposed to give you several bonuses. Among most important ones are monitoring and transactional behavior.
Transactional design is important if you want to be immune to failures, such as power failure. Imagine that you want to notify a bank system of ATM money withdrawal, and it has to be done exactly once per request, no matter what servers failed temporarily in the middle. MQ systems would allow you to coordinate transactions across multiple database, MQ and other systems.
Needless to say, such systems are very slow compared to named pipes, TCP or other non-transactional tools. If high performance is required, you would not allow your messages to be written thru disk. Instead, it will complicate your design - to achieve exotic reliable AND fast communication, which pushes the designer into really non-trivial tricks.
MQ systems normally allow users to watch the queue contents, write plugins, clear queus, etc.
MQ simply stands for Message Queue.
You would use one when you need to reliably send a inter-process/cross-platform/cross-application message that isn't time dependent.
The Message Queue receives the message, places it in the proper queue, and waits for the application to retrieve the message when ready.
reference: web services can be down and not available - what do you do then?
As an extension to that; what if your local network and your local pc is down as well?? While you wait for the system to recover the dependent deployed systems elsewhere waiting for that data needs to see an alternative data stream.
Otherwise, that might not be good enough 'real time' response for today's and very soon in the future Internet of Things (IOT) requirements.
if you want true parallel, non volatile storage of various FIFO streams(at least at some point along the signal chain) use an FPGA and FRAM memory. FRAM runs at clock speed and FPGA devices can be reprogrammed on the fly adding and taking away however many independent parallel data streams are needed(within established constraints of course).