Is there a option in RabbitMQ to send multipart messages?
This fact allows for using multi-part messages for adding
coarse-grained structure to your message. The example with two
matrices illustrates the point. You send the two matrices as two
message parts and thus avoid the copy. However, at the same time the
matrices are cleanly separated, each residing in its own message part
and you are guaranteed that the separation will be preserved even on
the receiving side. Consequently you don't have to put matrix size
into the message or invent any kind of "matrix delimiters".
There is no such feature like multipart content publish in AMQP protocol (see section 2.1.2 Message Flow) at all.
Multipart message sending and receiving can be implemented on application level, but there are no known use cases for it.
Related
In our team, we exchange messages via RabbitMQ between two systems. The messages are encoded in protobuf (v3). We use NServiceBus on the sending and receiving side. We use the RabbitMQ management UI for monitoring the error queue. In production, we noticed that it is not easy to make sense of the payload of the messages in the error queue, which are base64-encoded.
What is the easiest way to get human-readable access to the messages in the error queue? We have full control over the decisions in both systems, and also discussed a switch to JSON-encoded messages (instead of protobuf). But we are otherwise happy with our protobuf-based implementation. Which is already implemented after all.
Protobuf v3 supports formatting as json, once you have the data parsed as IMessage (the base type for in-memory protobuf objects).
So you can convert a single message to be human readable as follows:
Use the webUI GetMessage function to get the message as base64 then requeue it
Convert the message back to protobuf binary via Convert.FromBase64String
Parse it back to an IMessage via ProtoMessageTypeGoesHere.Parser.ParseFrom(binaryData)
You can then convert the parsed message to Json via ToString() or Google.Protobuf.JsonFormatter.
As long as your error queue isn't going to be disrupted by the re-queuing (e.g. resetting of timestamps or reprocessing), you should be able to do this for all messages in the queue.
I wouldn't recommend using the management UI for this. A simple script or html page with a stomp client would be a lot easier to use and more error-proof, in my opinion.
However, to answer your question: to simply decode the message and replace the text, a simple javascript solution will work fine.
$(".msg-payload").text(atob($(".msg-payload").text()))
This will simply select the message field on the queue page on the RabbitMQ management UI and replace it with the decoded value (that's the function atob).
To use this, you can either run it from the console or add it as a bookmark in your browser. Simply use the code prefixed with javascript:, like so:
javascript:$(".msg-payload").text(atob($(".msg-payload").text()))
What is the recommended way to deal with message versioning? The main schools of thought appear to be:
Always create a new message class as the message structure changes
Never use (pure) serialized objects as a message. Always use some kind of version header field and a byte stream body field. In this way, the receiver can always accept the message and check the version number before attempting to read the message body.
Never use binary serialized objects as a message. Instead, use a textual form such as JSON. In this way, the receiver can always accept the message, check the version number, and then (when possible) understand the message body.
As I want to keep my messages compact I am considering using Google Protocol Buffers which would allow me to satisfy both 2 & 3.
However I am interested in real world experiences and advice on how to handle versioning of messages as their structure changes?
In this case "version" will be basically some metadata about the message, And these metadata are some instruction/hints to the processing algorithm. So I willsuggest to add such metadata in the header (outside of the payload), so that consumer can read the metadata first before trying to read/understand and process the message payload. For example, if you keep the version info in the payload and due to some reason your (message payload is corrupted) then algorithm will fail parse the message, then it can not event reach the metadata you have put there.
You may consider to have both version and type info of the payload in one header.
I am newbee in message broker area . Currently there are quiet a good no. of message broker are there(Rabbit-mq ,zeromq ,kafka and many more).
Want to know which thing to consider while opting for any message broker for backend architecture .
Route messages to one or more of many destinations
Transform messages to an alternative representation
Perform message aggregation, decomposing messages into multiple messages and sending them to their destination, then recomposing the responses into one message to return to the user
Interact with an external repository to augment a message or store it
Invoke Web services to retrieve data
Respond to events or errors
Provide content and topic-based message routing using the publish–subscribe pattern
https://en.wikipedia.org/wiki/Message_broker
Try to use RabbitMq, simple and fast.
I'm using the web console against my AMQ 5.2 instance successfully, except for I cannot see the content of all of my messages.
If I send a test message using the web console, I can see the sample text content, but I believe the vendor app I am working with has binary or byte array message content.
Is there something I need to do to be able to view this raw data?
Thanks,
To my knowledge, it is not possible to inspect messages in the Admin Console. You can get some statistics (like how many messages have been sent etc.).
ActiveMQ does not unmarshal messages when receiving them (for performance reasons, unmarshalling is rather expensive).
Thus, if you want to have some way to inspect messages for their content, you can basically do 2 things:
Write a consumer which registers for all topics/queues, through which you can see messages' content. Drawback: if you're using queue-based interaction, your "real" consumers will not get all messages
Write an activeMQ plugin which looks at the messages. Have a look at ActiveMQ's Logger Plugin. Then write your own (you'll need the sources to compile it) and load it with ActiveMQ (see the documentation on how to configure ActiveMQ to load plugins). You want to override the send() method which is called whenever someone sends a message to the broker. There you get a reference to the message and can access its content.
Neither of the two messages provides a convenient viewing-mechanism though. You'll have to resort to standard out, or write your own web-based access.
hawtio now shows first 256 chars of messages. Don't know if that is enough for you. Use browse() method.
Does NServiceBus 2.0 allow for defining serializer for given message type?
I want for all but one of my messaages to be serialized using XmlSerializer. The remaining one should be serialized using BinarySerializer.
Is it possible with NServiceBus 2.0?
I believe the serializer is specified on an endpoint basis, so all messages using that endpoint would use the same serializer.
However, if you follow the rote NServiceBus recommendation of one message type per endpoint/queue then you could effectively isolate one message type and use a different serializer for it.
I'm curious, however, what is special about the one message type that requires binary serialization?
Edit in response to comment
The Distributor info indirectly mentions this under Routing with the Distributor. Udi Dahan also frequently advises this in the NServiceBus Yahoo Group although it's difficult to provide links because the search there is poor.
Basically, the idea is that you wouldn't want high priority messages to get stuck behind lower-priority ones, and also that this provides you with the greatest flexibility to scale out certain message processing if necessary.
Because the MsmqTransportConfig only allows for one InputQueue to be specified, having one message type per queue also means that you only have one message handler per endpoint.
To address the image, you may still be able to encapsulate it in an XML-formatted message if you encode the byte array as a Base64-encoded string. It's not ideal, but if your images aren't too large, it may be easier to do this than to go to the trouble of using a different serializer on only one message type.
Another option is to store the image data out-of-band in a database or filesystem and then refer to it by an ID or path (respectively).
Not possible in Version 2. But it can be done using the pipeline in versions 5 and above http://docs.particular.net/samples/pipeline/multi-serializer/