We are developing a system which uses rabbitMQ for sending and receiving data between its clients and servers.
The internet connection may sometimes be lost.
1- Can all the messages in the queue be exported to a file ? And somehow be imported to the client using this file?
2- In a different scenario, a client wants to send some messages to the queue but it has no internet connection! So we want to export all the message from client and make a file and somehow send it to the server (eg. transfer it to another location which has internet), Is this possible to import this file to the queue?
I had the same questions as I wanted to replay messages for testing / load testing purposes.
I made RabbitDump, a dotnet tool, to do this. It allows you to do all possible transfers from AMQP to and from Zip (bunch of messages). Examples: AMQP => ZIP, AMQP => AMQP, ZIP => AMQP and ZIP => ZIP (because why not ..).
The tool can be found here. It's installable as a dotnet tool, using dotnet tool install --global MBW.Tools.RabbitDump.
This tool will be useful to export messages from the remote queue and push them on a local RabbitMQ.
https://github.com/jecnua/rabbitmq-export-to-local
You can import/export messages using QueueExplorer.
Disclaimers: I'm the author, it's a commercial tool, and for now on Linux it runs under Wine.
https://www.cogin.com/QueueExplorer/rabbitmq/
Related
I am running a production application with fuse esb and using fuse provided activemq queues. There are 100k messages in one of my queues and I need to get a dump of those messages without removing them from the queue. What is the method to get a dump of those messages.
I used activemq:browse karaf command and directed output to file.But it did not give me all the messages. Only 4000 messages were written to a file.
ActiveMQ cannot browse extremely deep Queues so you won't likely be able to view them all. The browse operation is limited to what can fit into the broker memory and by the maxBrowsePageSize setting.
There is no tooling to dump the contents of the message store offered in ActiveMQ. A broker is not a database and should not be treated as one, messages are meant for consumers to consume.
Being new to the RabbitMQ I was wondering how to deal with an offline target node.
As an example this scenario:
1 log recording application that stores logs to some persistent storage
N log publishing applications that want their logs to be written to the persistent storage via the log recording server.
There would be two options:
Each publishing application publishes it's log messages to it's local RabbitMQ instance and the log recording server must subscribe to each of these
The log recording application has it's local RabbitMQ instance on which each log publishing application delivers it's messages.
Option 1 would require me to reconfigure/recode/notify the recording application each time a new application appears or moves. Therefore I would think Option 2 is the right one, each new publishing application simply writes to the RabbitMQ Node of the recording application.
The only thing I am struggling with is how to deal with a situation in which the Node of the recording application is down. Do I need to build my own system to store the messages until it's back online or can I use some functionality of RabbitMQ to deal with that? I.e. could the local RabbitMQ of each of the publishing applications just receive the messages and forward them to the recording application RabbitMQ as soon as it's back online?
I found something about the Federated plugin be couldn't understand if that's the solution. Maybe I need something different or maybe I have to write my own local queueing system (which I hope I don't have to) to queue messages when the target Node is offline.
Any links to architectural examples or solutions are more than welcome.
BTW: https://groups.google.com/forum/#!topic/easynetq/nILIKSjxyMg states that you shouldn't be installing a RabbitMQ Node for each application, so maybe I should resort to something like MSQM or ZeroMQ (?)
From experience in what sounds like a similar situation, I would suggest using something other than a queue to store the messages locally, when offline.
Years ago, I built a system that had to work offline - no network connection at all - and then had to push messages through a message queue to the central server, when the laptop was brought back to the office.
I solved this by using a local database (sqlite at the time) to store my messages when the message queue was not available.
You should do something similar. Use a local database or even a plain text file or CSV file to store your messages when RabbitMQ is offline. When it reconnects, read the messages from your local file system and send them through RabbitMQ.
This is a good strategy to use, even if you do not expect RabbitMQ to go offline. Frankly, it will go offline at some point and you will have to deal with it. You should be prepared for that situation, and having a local store for your messages will help that.
...
regarding rqm node per application: bad idea. this adds a ton of complexity to your system. You want as few RabbitMQ nodes as you can get away with. Meaning, 1 per system (a system being comprised of many applications) when possible... with the exception of RabbitMQ clusters for availability - but that's another line of questions and design, entirely.
...
I did an interview with Aria Stewart about designing for failure with RabbitMQ and messaging systems, and have a small excerpt where she talks about how networks fail.
The point is, the network or RabbitMQ or something will fail and you will need a solution like a local datastore so that you can recover when RabbitMQ comes back online.
I'm new to RabbitMQ and i need some help.
how to do backup and restore to RabbitMQ, and what is the important data i need to save.
thanks!
If you have the management plugin installed you can back-up and restore the broker on the Overview page. At the bottom you will see Import/Export Definitions and you can use this to download a JSON representation of your broker.
This will restore Exchanges, Queues, Virtual Hosts, Policies and Users.
Hope that helps.
For those looking for the HTTP API endpoint, it is:
http://rabbit:15672/api/definitions
Another way to automate this is to use the command line tool rabbitmqadmin (http://rabbit:15672/cli/) add pass the export subcommand, e.g.
rabbitmqadmin export rabbit-backup.config
I have a Java API on my server, and I want it to create tasks and add them to Celery via RabbitMQ. I followed the following tutorial, http://www.rabbitmq.com/tutorials/tutorial-two-python.html, where I used java for the client (send.java) and python to receive (receive.py). In receive.py, where the callback method is invoked, I call a method that I've annotated with #celery.task so that the task is added to celery.
I'm wondering how all of this is deployed on a server though, specifically, why there is a receive.py file. Is receive.py a process that must continually run on the server? Is there a way to configure RabbitMQ so that it automatically routes java client tasks to celery?
Thanks!
RabbitMQ is just a message queue. Producers put messages and consumers get them on demand. You can only restrict access for specific queues via RabbitMQ's auth options.
As for deployment: yes, receive.py needs to continuously run. It is Celery's job to do that. See the Workers Guide for info on running a worker.
I'm looking for a client (as in GUI client, not client library) to play with our MQ server and familiarize myself with its semantics. Something that will send and receive messages at the press of a button (or a text command) and maybe even update me about the status of the server queues and messages. Administration would be a bonus. The UI doesn't have to be graphical (i.e. command line clients are fine).
The server will probably run RabbitMQ so anything RabbitMQ-specific is fine, as is ActiveMQ. But I'd rather have a generic AMQP or STOMP tool.
So, does anything of the sort exist?
I know some management and monitoring tools come with both server distributions, but no clients, right?
For Apache ActiveMQ, there is
the web admin console at http://localhost:8161/admin/
the ApacheActiveMQBrowser project on Sourceforge:
An open source project of developing
Message admin gui based tools for
Apache ActiveMQ.
HermesJMS, it does not mention ActiveMQ 5 (only 3 and 4) on the plugin page, but there is an active user forum
The rabbitmq-management plugin that comes with RabbitMQ (and enabled by rabbitmq-plugins enable rabbitmq_management) has a web-based interface that listens on the port 15672 and can do everything you are (I was) asking for.
Check out the BQL RabbitMQ plugin.
It gives you an SQL-style language for AMQP. For instance,
BQL> create exchange myexchange;
ok
BQL> create durable queue 'myqueue'
ok
BQL> select name,messages from queues where 'durable'=true order by name
----------------------
| name | messages |
----------------------
| myqueue | 0 |
Obviously, it's RabbitMQ specific.
If you're willing to do a bit of coding, you could take a look at the examples in the RabbitMQ Java and .NET clients:
Java examples
.NET examples
They're not quite graphical, but trying to understand them forces you to ask the right questions.
It's been a while, but I remember thinking that the best way to familiarize yourself with AMQP is to read the 0-9-1 spec and write some simple programs; in particular, the protocol documentation on that site gives a lot of examples.
Command line tools (written in C) to send and receive AMQP messages: http://github.com/rmt/amqptools