Is it a good idea to send files with size about 1Mb through RabbitMQ? I want to send message in json format with binary fields corresponding to the files.
And how to do it properly using spring-amqp? Just by publishing object with next class?
class Message {
String field1;
byte[] fileField1;
byte[] fileField2;
}
I would suggest not only reading those links that were posted but also, doing some of your own experimentation. The thing I would be concerned about is performance at the service level and at the client level.
You might want to consider having a server host the files/data and allow rabbitmq just send the message to the consumer with the id of the message in it. So when your consumer gets the message, it sends an HTTP GET request to a service that requests the actual message payload. That way RabbitMQ stays lightweight. You can always add consumers and servers if you need.
That's my opinion without experimenting. You might find that it's still lighting fast with 1MB payloads. That's why I would say to experiment and find out for yourself.
Hope you find this helpful!
Related
In ActiveMQ while using blob messages we use this as broker
String broker1 = "tcp://localhost:7005?jms.blobTransferPolicy.UploadUrl=http://localhost:7005/fileserver/"
Can anybody explain what is UploadUrl and why we need to configure for blob messages(we don't need to configure for text messages). Why it doesn't allow tcp protocol?
So plain text messages are good and easy to use, but needs to be in memory at all times. It works well with a KBs of data, or even a few MB. However, sending very large files, such as initial data loads, large media files or BI data, is not nice to keep around in memory. There may still be a need to pass the message around, route/filter based on message properties, use transactions and similar.
Blob messages is an attempt to solve the need to pass around GBs of data through the semantics of messaging. The trade off is that you have to define a streaming based server somewhere that both sender and receiver can reach. It can be HTTP, FTP, a local file, WebDAV or similar. ActiveMQ comes with a HTTP based fileserver if you have no other file area around.
I've got a consumer that rejects messages and knows exactly why those messages were rejected. She'd like to provide the "why" as well as the "what" to the producer when rejecting a message.
What's a good queue architecture for nack'ing messages but also sending back metadata describing why the message failed?
(At a higher level, if the producer isn't doing anything with the 'nacked reason codes, I'm thinking logging the reason codes from the consumer would suffice for visibility, so the question becomes moot. Still, seems like an interesting question assuming otherwise.)
You can use the RPC model as described here:
https://www.rabbitmq.com/tutorials/tutorial-six-java.html
In this way you can send-back to the publisher a message with the reason.
You can also considerer Dead Letter Exchanges extension, but you can't change the message, so you are just informed that your message has been rejected.
With a little work, you can create an exchange where you redirect the nack messages, and using the header property message to write the reason, like that:
Map<String, Object> myHeader = new HashMap<String, Object>();
myHeader("reason", "can't access to database");//<-- just an example
AMQP.BasicProperties.Builder bob = new AMQP.BasicProperties.Builder();
bob.headers(myHeader);
In this way you can maintain the original message and modify only the header. (similar to Dead Letter Message)
hope it helps
I fall into similar issue. My solution was to assign unique ID to each message on sending (using properties) and then on rejection save error (associating it with assigned ID) into redis / memcached (I also used time expiration in redis to not overload storage). It is possible in my case, because I quickly handle all these dead messages so errors should not be keeped for a long time.
Probably not so elegant, but I didn't want publish anything manually and preferred rely on native rabbit functionality and I didn't nee many changes to the code.
I am new to the formum and new to MSMQ.
I have been asked to do research on it, to see if it will help out business, but still not sure if and how it works. Here is a short Summary.
We have a service provider that will receive messages via a mobile phone, to which they will pass certain info(Such as the cell number, text in message, etc.) to a URL that we have given them, which is an app we created, which will then process the data and store into our Database etc etc.
However, as we at times receive between a few hundred to a few thousand at any given time(Spread out or at once) - We get timeouts.
What I would like to know, is it possible to get this info stored into a que using MSMQ, before it hits our URL(That we provided to our service provider), so that we can avoid timeouts ?
I hope this makes sense and that someone can help!
Thank you!
MSMQ is a transport protocol. It is designed to get data from A to B as fast and as reliably as possible. As it uses store-and-forward, it can be used as a data buffer at the sender. In your case, though, that's not important. I would recommend:
ISP sends MSMQ message over HTTP(S) to your server.
Message is stored on server's local queue.
Your app reads messages from queue.
Your app writes to database.
Goto 3
So you would be buffering the messages AFTER they reach your server.
The messages would also be buffered at the ISP in case of network outage between them and you.
Let's say I have a ClientRequestMessage message that contains a request for a specific Client. A web application will generate these requests and they need to be sent to the correct Client for handling. I can think of a few options for this.
I could have a single queue that all messages go to and specific client handlers check a property (like ClientId) to decide whether they care about it. This feels wrong on many levels to me though.
I could publish a message to all of the clients and they could decide whether or not they care about it during handling. This seems like too much traffic and wastes each client's time handling messages they shouldn't care about in the first place though.
I could have client specific queues that these messages get routed too. This one feels the best to me, but I am unsure of how to do it. I'd like to keep it simple and avoid client specific message types, but I am not sure how to tell NServiceBus "for client A send it to client A's queue and for client B send it to client B's queue".
So my question is, what is the best (most efficient? easiest to manage?) way to set this up? I am pretty sure I need to use the distributor, but not positive so thought I would ask.
BONUS QUESTION:
Let's say each client has multiple handlers. How can I make sure only one of them handles a given message? Would I need a distributor per client?
If what you really want is the solution that allows you to have just a single message where you can place a specific filter on the message based on clientId and only route the message to the client when it relates to them then I would use PServiceBus(pservicebus.codeplex.com). It will make it easier for you specific a set of subscriptions for each of your client where their messages are all filtered by clientId into a specific queue or what transport you have available. The below example shows filtering a ChatTopic by the UserName Property and the subscriber only receives the message at the specified transport when the message been published UserName property is not TJ. You are also allowed to use complex filter where you do thing such as GreaterThan("MyComplexProperty.Blah.ID", 5)
Subscriber.New("MyUserName").Durable(false)
.SubscribeTo(Topic.Select<ChatTopic>().NotEqual("UserName", "TJ"))
.AddTransport("Tcp",
Transport.New<TcpTransport>(
transport => {
transport.Format = TransportFormat.Json;
transport.IPAddress = "127.0.0.1";
transport.Port = port;
}), "ChatTopic")
.Save();
You can tell NSB where to put messages by using the MessageEndpointMappings configuration section. You can map a specific message type or a whole assembly to a queue. If you don't want to create specific message types and map them, then I would recommend the publish approach. The overhead of removing a message from the queue is pretty minimal.
If your "client" has many instances of NSB to pick up messages then you will need to use a Distributor. Check out the distributed Pub/Sub documentation.
I'm using the web console against my AMQ 5.2 instance successfully, except for I cannot see the content of all of my messages.
If I send a test message using the web console, I can see the sample text content, but I believe the vendor app I am working with has binary or byte array message content.
Is there something I need to do to be able to view this raw data?
Thanks,
To my knowledge, it is not possible to inspect messages in the Admin Console. You can get some statistics (like how many messages have been sent etc.).
ActiveMQ does not unmarshal messages when receiving them (for performance reasons, unmarshalling is rather expensive).
Thus, if you want to have some way to inspect messages for their content, you can basically do 2 things:
Write a consumer which registers for all topics/queues, through which you can see messages' content. Drawback: if you're using queue-based interaction, your "real" consumers will not get all messages
Write an activeMQ plugin which looks at the messages. Have a look at ActiveMQ's Logger Plugin. Then write your own (you'll need the sources to compile it) and load it with ActiveMQ (see the documentation on how to configure ActiveMQ to load plugins). You want to override the send() method which is called whenever someone sends a message to the broker. There you get a reference to the message and can access its content.
Neither of the two messages provides a convenient viewing-mechanism though. You'll have to resort to standard out, or write your own web-based access.
hawtio now shows first 256 chars of messages. Don't know if that is enough for you. Use browse() method.