In our team, we exchange messages via RabbitMQ between two systems. The messages are encoded in protobuf (v3). We use NServiceBus on the sending and receiving side. We use the RabbitMQ management UI for monitoring the error queue. In production, we noticed that it is not easy to make sense of the payload of the messages in the error queue, which are base64-encoded.
What is the easiest way to get human-readable access to the messages in the error queue? We have full control over the decisions in both systems, and also discussed a switch to JSON-encoded messages (instead of protobuf). But we are otherwise happy with our protobuf-based implementation. Which is already implemented after all.
Protobuf v3 supports formatting as json, once you have the data parsed as IMessage (the base type for in-memory protobuf objects).
So you can convert a single message to be human readable as follows:
Use the webUI GetMessage function to get the message as base64 then requeue it
Convert the message back to protobuf binary via Convert.FromBase64String
Parse it back to an IMessage via ProtoMessageTypeGoesHere.Parser.ParseFrom(binaryData)
You can then convert the parsed message to Json via ToString() or Google.Protobuf.JsonFormatter.
As long as your error queue isn't going to be disrupted by the re-queuing (e.g. resetting of timestamps or reprocessing), you should be able to do this for all messages in the queue.
I wouldn't recommend using the management UI for this. A simple script or html page with a stomp client would be a lot easier to use and more error-proof, in my opinion.
However, to answer your question: to simply decode the message and replace the text, a simple javascript solution will work fine.
$(".msg-payload").text(atob($(".msg-payload").text()))
This will simply select the message field on the queue page on the RabbitMQ management UI and replace it with the decoded value (that's the function atob).
To use this, you can either run it from the console or add it as a bookmark in your browser. Simply use the code prefixed with javascript:, like so:
javascript:$(".msg-payload").text(atob($(".msg-payload").text()))
Related
What is the recommended way to deal with message versioning? The main schools of thought appear to be:
Always create a new message class as the message structure changes
Never use (pure) serialized objects as a message. Always use some kind of version header field and a byte stream body field. In this way, the receiver can always accept the message and check the version number before attempting to read the message body.
Never use binary serialized objects as a message. Instead, use a textual form such as JSON. In this way, the receiver can always accept the message, check the version number, and then (when possible) understand the message body.
As I want to keep my messages compact I am considering using Google Protocol Buffers which would allow me to satisfy both 2 & 3.
However I am interested in real world experiences and advice on how to handle versioning of messages as their structure changes?
In this case "version" will be basically some metadata about the message, And these metadata are some instruction/hints to the processing algorithm. So I willsuggest to add such metadata in the header (outside of the payload), so that consumer can read the metadata first before trying to read/understand and process the message payload. For example, if you keep the version info in the payload and due to some reason your (message payload is corrupted) then algorithm will fail parse the message, then it can not event reach the metadata you have put there.
You may consider to have both version and type info of the payload in one header.
Is it a good idea to send files with size about 1Mb through RabbitMQ? I want to send message in json format with binary fields corresponding to the files.
And how to do it properly using spring-amqp? Just by publishing object with next class?
class Message {
String field1;
byte[] fileField1;
byte[] fileField2;
}
I would suggest not only reading those links that were posted but also, doing some of your own experimentation. The thing I would be concerned about is performance at the service level and at the client level.
You might want to consider having a server host the files/data and allow rabbitmq just send the message to the consumer with the id of the message in it. So when your consumer gets the message, it sends an HTTP GET request to a service that requests the actual message payload. That way RabbitMQ stays lightweight. You can always add consumers and servers if you need.
That's my opinion without experimenting. You might find that it's still lighting fast with 1MB payloads. That's why I would say to experiment and find out for yourself.
Hope you find this helpful!
Is there a option in RabbitMQ to send multipart messages?
This fact allows for using multi-part messages for adding
coarse-grained structure to your message. The example with two
matrices illustrates the point. You send the two matrices as two
message parts and thus avoid the copy. However, at the same time the
matrices are cleanly separated, each residing in its own message part
and you are guaranteed that the separation will be preserved even on
the receiving side. Consequently you don't have to put matrix size
into the message or invent any kind of "matrix delimiters".
There is no such feature like multipart content publish in AMQP protocol (see section 2.1.2 Message Flow) at all.
Multipart message sending and receiving can be implemented on application level, but there are no known use cases for it.
Hy,
Is there a way to see the messages that arr waiting in the Celery/RabbitMQ queues?
I would like to see what parameters has each waiting task.
Tx!
You can do it with Management Plugin
UPD:
Message props and headers are plain strings or numbers so you can view them without any extra job.
If you need to see message body and Management Plugin doesn't fits you, another approach is to consume messages from queue, output their content and push them back in queue. There is some chance to loose message during such action if your script fail before message will be requeued back.
Again, if you need to see only parameters or headers then Management Plugin is your best friend.
If you send JSON or XML encoded message body i think it is trivial to write js script and use it with Userscripts to show you message original content.
I'm using the web console against my AMQ 5.2 instance successfully, except for I cannot see the content of all of my messages.
If I send a test message using the web console, I can see the sample text content, but I believe the vendor app I am working with has binary or byte array message content.
Is there something I need to do to be able to view this raw data?
Thanks,
To my knowledge, it is not possible to inspect messages in the Admin Console. You can get some statistics (like how many messages have been sent etc.).
ActiveMQ does not unmarshal messages when receiving them (for performance reasons, unmarshalling is rather expensive).
Thus, if you want to have some way to inspect messages for their content, you can basically do 2 things:
Write a consumer which registers for all topics/queues, through which you can see messages' content. Drawback: if you're using queue-based interaction, your "real" consumers will not get all messages
Write an activeMQ plugin which looks at the messages. Have a look at ActiveMQ's Logger Plugin. Then write your own (you'll need the sources to compile it) and load it with ActiveMQ (see the documentation on how to configure ActiveMQ to load plugins). You want to override the send() method which is called whenever someone sends a message to the broker. There you get a reference to the message and can access its content.
Neither of the two messages provides a convenient viewing-mechanism though. You'll have to resort to standard out, or write your own web-based access.
hawtio now shows first 256 chars of messages. Don't know if that is enough for you. Use browse() method.