What is the recommended way to deal with message versioning? The main schools of thought appear to be:
Always create a new message class as the message structure changes
Never use (pure) serialized objects as a message. Always use some kind of version header field and a byte stream body field. In this way, the receiver can always accept the message and check the version number before attempting to read the message body.
Never use binary serialized objects as a message. Instead, use a textual form such as JSON. In this way, the receiver can always accept the message, check the version number, and then (when possible) understand the message body.
As I want to keep my messages compact I am considering using Google Protocol Buffers which would allow me to satisfy both 2 & 3.
However I am interested in real world experiences and advice on how to handle versioning of messages as their structure changes?
In this case "version" will be basically some metadata about the message, And these metadata are some instruction/hints to the processing algorithm. So I willsuggest to add such metadata in the header (outside of the payload), so that consumer can read the metadata first before trying to read/understand and process the message payload. For example, if you keep the version info in the payload and due to some reason your (message payload is corrupted) then algorithm will fail parse the message, then it can not event reach the metadata you have put there.
You may consider to have both version and type info of the payload in one header.
Related
In our team, we exchange messages via RabbitMQ between two systems. The messages are encoded in protobuf (v3). We use NServiceBus on the sending and receiving side. We use the RabbitMQ management UI for monitoring the error queue. In production, we noticed that it is not easy to make sense of the payload of the messages in the error queue, which are base64-encoded.
What is the easiest way to get human-readable access to the messages in the error queue? We have full control over the decisions in both systems, and also discussed a switch to JSON-encoded messages (instead of protobuf). But we are otherwise happy with our protobuf-based implementation. Which is already implemented after all.
Protobuf v3 supports formatting as json, once you have the data parsed as IMessage (the base type for in-memory protobuf objects).
So you can convert a single message to be human readable as follows:
Use the webUI GetMessage function to get the message as base64 then requeue it
Convert the message back to protobuf binary via Convert.FromBase64String
Parse it back to an IMessage via ProtoMessageTypeGoesHere.Parser.ParseFrom(binaryData)
You can then convert the parsed message to Json via ToString() or Google.Protobuf.JsonFormatter.
As long as your error queue isn't going to be disrupted by the re-queuing (e.g. resetting of timestamps or reprocessing), you should be able to do this for all messages in the queue.
I wouldn't recommend using the management UI for this. A simple script or html page with a stomp client would be a lot easier to use and more error-proof, in my opinion.
However, to answer your question: to simply decode the message and replace the text, a simple javascript solution will work fine.
$(".msg-payload").text(atob($(".msg-payload").text()))
This will simply select the message field on the queue page on the RabbitMQ management UI and replace it with the decoded value (that's the function atob).
To use this, you can either run it from the console or add it as a bookmark in your browser. Simply use the code prefixed with javascript:, like so:
javascript:$(".msg-payload").text(atob($(".msg-payload").text()))
I've got a consumer that rejects messages and knows exactly why those messages were rejected. She'd like to provide the "why" as well as the "what" to the producer when rejecting a message.
What's a good queue architecture for nack'ing messages but also sending back metadata describing why the message failed?
(At a higher level, if the producer isn't doing anything with the 'nacked reason codes, I'm thinking logging the reason codes from the consumer would suffice for visibility, so the question becomes moot. Still, seems like an interesting question assuming otherwise.)
You can use the RPC model as described here:
https://www.rabbitmq.com/tutorials/tutorial-six-java.html
In this way you can send-back to the publisher a message with the reason.
You can also considerer Dead Letter Exchanges extension, but you can't change the message, so you are just informed that your message has been rejected.
With a little work, you can create an exchange where you redirect the nack messages, and using the header property message to write the reason, like that:
Map<String, Object> myHeader = new HashMap<String, Object>();
myHeader("reason", "can't access to database");//<-- just an example
AMQP.BasicProperties.Builder bob = new AMQP.BasicProperties.Builder();
bob.headers(myHeader);
In this way you can maintain the original message and modify only the header. (similar to Dead Letter Message)
hope it helps
I fall into similar issue. My solution was to assign unique ID to each message on sending (using properties) and then on rejection save error (associating it with assigned ID) into redis / memcached (I also used time expiration in redis to not overload storage). It is possible in my case, because I quickly handle all these dead messages so errors should not be keeped for a long time.
Probably not so elegant, but I didn't want publish anything manually and preferred rely on native rabbit functionality and I didn't nee many changes to the code.
Our system receives ISO8583 messages which we decode and handle appropriately. Now we are getting invalid ISO messages in between which our system can't handle. In fact, it sends nothing in return. This causes a timeout on the other side. As a consequence, the (invalid) transaction is reverted which then causes quite a messup as there is nothing to be reverted.
Can anyone give me a clue on how to deal with/answer an invalid/undecodable ISO8583 message? Is there a standard answer (e.g. 'NAK' like)?
According to the ISO-8583 spec, 6XX (or 16XX, if you're using the '93 version)-class messages are appropriate for administrative notifications. Generally, a 644 or 1644 MTI is prescribed for notifying the sender of a problem processing a message, where
X6XX - Indicates an administrative message, often containing the details of a failure
XX4X - Indicates that the message is a notification; the sender is not to repeat the message
XXX4 - Indicates the source of the message (acquirer/issuer/other); here, it's Other
Putting it all together, your message should have at least the following fields
MTI: 1644
DE-24 (Function code): 650 (Unable to parse message)
Of course, you're to include the standard message identification fields: DE-7,11,12,39. These fields will be necessary for the message sender to match your response with the request.
I don't think there is a standard way of handling invalid ISO 8583 request messages. You didn't say why you are receiving invalid request messages, and without knowing that it is difficult to suggest how you should handle them.
Depending on the situation it may be best not to answer invalid ISO 8583 requests. In fact I know of systems that not only don't answer invalid request messages but will also blacklist the device that sent the invalid message and refuse to answer all other messages from it.
If you do decide not to respond to invalid request messages then as you have found out the client is likely to time out and then attempt to reverse the transaction. This is not usually a problem because servers will usually respond with an approval message to reversal request for transactions that don't exist. Remember that when a client times out after sending a request, it doesn't know if the request was processed or even if the request was received. So a server has to be prepared to handle both 1. a request that was received and processed (by undoing the transaction and then responding with an approval), and 2. a request that was never received/processed (by sending an approval). NOTE: In case 2 there is no need to undo the transaction because the transaction never took place.
From my experience with integrating ISO links, invalid ISO messages are usually, by industry standard, handled by a dropping down of the acquring host's connection followed by an angry mail from the acquirer's service provider accusing you of segfaulting down their mainframe.
Other than that different implementations, when implemented well, will handle invalid messages differently, from what #kolossus said in case the parser fails completely, to a normal **10 response with a specific response code such as RC 12 "Invalid transaction" when just some subfields don't make sense (such as problems with packaging of the complex subfields with tokens, track2 parsing etc)
The practical reason why #kolossus solution doesn't really make sense and why Stuard has a point is, if the client has issues of forming the ISO messages, then it almost certainly has a problem with parsing them too, so another ISO message doesn't really tell the client anything except provoking a parser exception on his side too.
End result will be the same - a technical reversal by the client, just not after a timeout. Basically, with iso8583, the best way to handle invalid messages is to not have them, there's no clean way.
Is there a option in RabbitMQ to send multipart messages?
This fact allows for using multi-part messages for adding
coarse-grained structure to your message. The example with two
matrices illustrates the point. You send the two matrices as two
message parts and thus avoid the copy. However, at the same time the
matrices are cleanly separated, each residing in its own message part
and you are guaranteed that the separation will be preserved even on
the receiving side. Consequently you don't have to put matrix size
into the message or invent any kind of "matrix delimiters".
There is no such feature like multipart content publish in AMQP protocol (see section 2.1.2 Message Flow) at all.
Multipart message sending and receiving can be implemented on application level, but there are no known use cases for it.
Does NServiceBus 2.0 allow for defining serializer for given message type?
I want for all but one of my messaages to be serialized using XmlSerializer. The remaining one should be serialized using BinarySerializer.
Is it possible with NServiceBus 2.0?
I believe the serializer is specified on an endpoint basis, so all messages using that endpoint would use the same serializer.
However, if you follow the rote NServiceBus recommendation of one message type per endpoint/queue then you could effectively isolate one message type and use a different serializer for it.
I'm curious, however, what is special about the one message type that requires binary serialization?
Edit in response to comment
The Distributor info indirectly mentions this under Routing with the Distributor. Udi Dahan also frequently advises this in the NServiceBus Yahoo Group although it's difficult to provide links because the search there is poor.
Basically, the idea is that you wouldn't want high priority messages to get stuck behind lower-priority ones, and also that this provides you with the greatest flexibility to scale out certain message processing if necessary.
Because the MsmqTransportConfig only allows for one InputQueue to be specified, having one message type per queue also means that you only have one message handler per endpoint.
To address the image, you may still be able to encapsulate it in an XML-formatted message if you encode the byte array as a Base64-encoded string. It's not ideal, but if your images aren't too large, it may be easier to do this than to go to the trouble of using a different serializer on only one message type.
Another option is to store the image data out-of-band in a database or filesystem and then refer to it by an ID or path (respectively).
Not possible in Version 2. But it can be done using the pipeline in versions 5 and above http://docs.particular.net/samples/pipeline/multi-serializer/