We use rabbitmq in production and really happy with it. we are planning to use rabbitmq fanout feature, but would like to understand the internals.
Here are my questions:
1. When a message is fan'ed out to multiple queues, is the message copied to different queues rather than just the reference?
2. if the entire message is duplicated on all the fan'ed out queues, Is there a way to make it efficient?
Thank you,
-Bala
To store AMQP message content, RabbitMQ uses "binaries" which are a particular type in Erlang.
The Erlang VM only copies those binaries if they are small enough (the limit is 64 bytes if I remember correctly), otherwise, it uses reference counting. There is a chapter in the Erlang documentation about how binaries are implemented. Paragraphe 4.1 should be of interest to you.
Related
I'm using RabbitMQ as message broker in first time and now I have a question about when to declare queues and exchanges using rabbit's own management tool and when to do it in the code of the software? In my opinion is that it is much better to create queues and exchanges using the management tool, because it's a centralized place to add new or remove useless queues without the need to modify the actual software. I am asking some advice and opinions.
Thank you.
The short answer is: whatever works best for you.
I've worked with message brokers that required external tools for defining the topology (exchanges, queues, bindings, etc) and with RabbitMQ that allows me to define them at runtime, as needed.
I don't think either scenario is "the right way". Rather, it depends entirely on your situation.
Personally, I see a lot of value in letting my software define the topology at runtime with RabbitMQ. But there are still times when it gets frustrating because I often end up duplicating my definitions between producers and consumers.
But then, moving from development to production is easier when the software itself defines the topology. No need to pre-configure things before moving code to production.
It's all tradeoffs.
Try it however you're comfortable. Then try it the other way. See what happens, and learn which you prefer and when. Just remember that you don't have to do one or the other. You can do both if you want.
I am trying to learn messaging system. I have found that RabbitMq and NServiceBus are using together in few places. My questions are
If I am using the RabbitMQ then why do i need NServiceBus? and vice versa
What NServiceBus can do but RabbitMQ or Kafka cannot?
Can I use NServiceBus and kafka together? Or Apache-Kafka does not require NServiceBus
Years ago, I asked myself the same question. I was looking at NServiceBus to work with a different message queue, but the question was the same.
I decided not to use NServiceBus.
6 Month later, I realized I had re-built half of what NServiceBus did... only much more poorly.
The equivalent question of why would you need NServiceBus with RabbitMQ, is to ask why you would need the .NET Framework with ASP.NET MVC, or WinForms, or XAML, or any of the built-in libraries that .NET ships with, when you have the Common Language Runtime.
Shouldn't the CLR be enough, after all?
Of course not. Having the runtime on which code can execute - the MSIL interpreter and execution engine - is not nearly enough to be productive.
Sure, you can write command-line applications that take input and produce output. But try to build a real application without the common libraries - without the built-in SQL Server drivers; without any 3rd party controls or libraries. Build a Windows Desktop app without the System.Windows namespace.
You need those libraries to give you collections, and database access, and window objects and UI controls.
Similarly, RabbitMQ gives you everything you need to get started and working, but not enough to maintain productivity.
Sure, you can grab the .NET driver for RabbitMQ and start producing and consuming messages.
For a while, this will work just fine.
Pretty soon, you'll find yourself creating a wrapper around the driver, so you can reduce the amount of code you need to write.
Then you'll find yourself needing to deal with ack vs nack, and you'll create a simple API for that.
Then the need for dead-letter queues will pop up with nack calls, and you'll wrap that up in your API - simplified compared to the rabbitmq driver, of course.
Eventually, you'll want to deal with poison messages - messages that are malformed and causing exceptions. Once again, you don't want to write one-off code for this, so you'll write a library to handle it.
The list goes on and on.
6 months from now, you'll find yourself working with a half-written, barely specified, untestable library that only mimics the value and capabilities of NServiceBus (or MassTransit or whatever other service bus library you choose).
I won't say you have to use NServiceBus. And I would say you should learn how RabbitMQ works, without it. But once you get beyond the basics of sending and receiving messages, the value of NServiceBus and other service bus implementations, becomes very apparent, very quickly.
It seems there is community support for Kafka transport in NServiceBus now: https://docs.particular.net/nservicebus/kafka/
(haven't tried it myself yet).
Can we migrate from IBM MQ to Rabbit MQ ? Is this possible?
Are there any dependencies.What are the factors that we need to look at.
That's like asking can you migrate from Java to c++. Both things do the same thing but are different by structure (at least).
You can do it of course, but I wouldn't call it migrating, but setting up new. They are both message queue brokers and you could keep some concepts (architecturally looking) but the whole "physical" infrastructure would needed to be set up from scratch.
You need to re-look at the current messaging design pattern and re-design it for RabbitMQ. And design the roll-out strategy. Secondly all the producers and consumers need to be re-written. You need to take into consideration how much effort the producer and consumer applications need to migrate to the new messaging system and secondly Messaging Format also slightly different here in RabbitMQ. So you need to take various aspect into consideration before migrating.
In the distributed message transaction world, am trying to understand the different parts that are involved in developing distributed systems. From what I understand you can design messaging system using enterprise bus backed with a message queue system. Why is it a good idea to use both? Can the same be achieved by programming against just the message queuing system? What are the advantages of using both together?
You certainly can code directly against the messaging infrastructure and you will find that there are pros and cons w.r.t. each transport. There are many decisions that you will need to make along the way, though, and this is where a service bus may assist.
Developing directly against the queuing system will inevitably lead to various abstractions that you will require to prevent duplication.
A service bus will provide opinions/implementations for:
Message delivery
exactly-once (distributed transactions - distributed transactions are not supported by all queuing systems)
at-least-once (non-transactional)
at-most-once (will probably require some transactional processing but you can get away with no distributed transactions)
Retrying failed messages
Request / Response
Message distribution
Publish/Subscribe (probably quite easy with RabbitMQ directly, not so much with MSMQ directly)
Message Idempotence
Dependency Injection
Some service bus implementations provide a framework for implementing process managers (called sagas by most). My current opinion is that a process manager needs to be a first-class citizen as any other entity is but that may change :)
Anyhow, if you as still evaluating options you could also take a look at my FOSS project: http://shuttle.github.io/shuttle-esb/
So a service bus may buy you quite a bit out-of-the-box whereas coding against the queues directly may be a bit of work to get going.
I can't comment directly on MassTransit, having only tinkered with it.
I use NServiceBus and am a fan of it. I think there are valid reasons for directly using queuing technology, but I think rolling your own ESB using MSMQ/RabbitMQ would cost a lot more than simply using a commercial product (or open source product e.g. MassTransit).
So do you need it? No. Will it make your life much easier if the features match your requirements? Absolutely.
I'm considering using AMQP (using qpid) to enable a mixture of Python and Java services communicate with each other. Basic text messaging seems simple enough but, as with every other messaging technology I've investigated, that's where it seems to stop. Except for building instant messaging applications, I would have thought sending strings wasn't a particularly useful thing to do yet example after example demonstrates sending unformatted text around.
My instinct then is to use XML (de-)serialization or something similar (JSON, YAML, Protocol Buffers etc.) which has good library support in both languages. Is this a best practice and, if so, which (de-)serialization protocol would people recommend? Or am I missing the point somewhere and should be quite content sending small bits of text?
Owen, may I offer a few words about RabbitMQ.
AMQP is a binary protocol and you can certainly do much more than send strings around! Which Python client do you plan to use? We recommend Barry Pederson's client for most uses: http://barryp.org/software/py-amqplib/ You are most welcome to come to the RabbitMQ list and ask any questions you like about anything in relation to your post and the comments :-)
As James points out, JSON is goodness. RabbitMQ supports JSON-RPC over HTTP connecting to an AMQP back end. People also use RabbitMQ with Orbited for comet type apps.
In addition we are fans of, and support XMPP, and STOMP too which James invented. STOMP is handy for a certain class of messaging apps and RabbitMQ supports it for both direct and topic based routing. We've found it a fine way to interop with ActiveMQ, preferring it to JMS in that scenario.
I hope you find the right server for your use cases, and recommend you try out different combinations, for best results.
Cheers,
alexis
For what it's worth, I've been having a good experience using AMQP + Protocol Buffers.
If the sender is serializing the messages, you will probably need to include an id in the header so that you know how to de-serialize on the receiving side. However, this isn't too much trouble to accomplish.
XML or JSON are probably the easiest. Protocol buffers is cool but I'd treat it as an optimisation to think of later on if you really need to (as its a bit harder to use being essentially a binary wire format).
BTW you might want to look at Stomp rather than AMQP; its got way more client libraries and supported message brokers. e.g. Apache ActiveMQ which is way more popular than qpid or rabbitmq - or indeed any JMS provider would work just fine.