What's the best way to store custom messages into the queue? I mean if I have a queue that can store different types of messages should I store them in binary format or json?
What do you think?
Windows Azure Storage Client Library provides overloads for binary and string that handle encoding for you. As such you can make use of any serialization mechanism you like, given that the serialized form is less than 64 KB.
Hence, the answer to your question actually depends on your specific scenario. Handling JSON data would be much easier, but if you have a specific need to send the data in another format, please consider such alternatives. For larger scenarios some users augment queue messages to simply point to blob or table storage as a more flexible and verbose option while using the queue messages to provide for reliable message delivery.
Related
In ActiveMQ while using blob messages we use this as broker
String broker1 = "tcp://localhost:7005?jms.blobTransferPolicy.UploadUrl=http://localhost:7005/fileserver/"
Can anybody explain what is UploadUrl and why we need to configure for blob messages(we don't need to configure for text messages). Why it doesn't allow tcp protocol?
So plain text messages are good and easy to use, but needs to be in memory at all times. It works well with a KBs of data, or even a few MB. However, sending very large files, such as initial data loads, large media files or BI data, is not nice to keep around in memory. There may still be a need to pass the message around, route/filter based on message properties, use transactions and similar.
Blob messages is an attempt to solve the need to pass around GBs of data through the semantics of messaging. The trade off is that you have to define a streaming based server somewhere that both sender and receiver can reach. It can be HTTP, FTP, a local file, WebDAV or similar. ActiveMQ comes with a HTTP based fileserver if you have no other file area around.
basically I want to understand both high level and also technical point of view as what constitutes a streaming API, there are all sorts of data available but I could not find a satisfactory explanation of streaming API, also how does it differ from general APIs (REST if applicable)
PS:I am not asking about multimedia streaming.
Kind of a vague question. I guess streaming usually means one of the following (or a combination)
downloading data for immediate consumption, rather than a whole file for storage, potentially with support for delivering partial data (lower quality, only relevant pieces etc), sometimes even without any storage at all in between producer and consumer
a persistent connection that continues to deliver new data as it becomes available, rather than having the client poll
A good example (for the first pattern) are streaming XML parsers (such as SAX). They allow you to handle XML data that is too big to fit into memory (which a DOM parser likes to do).
I just find another good answer here:
https://www.quora.com/What-is-meant-by-streaming-API
A streaming API differs from the normal REST API in the way that it leaves the HTTP connection open for as long as possible(i.e. "persistent connection"). It pushes data to the client as and when it's available and there is no need for the client to poll the requests to the server for newer data. This approach of maintaining a persistent connection reduces the network latency significantly when a server produces continous stream of data like say, today's social media channels. These APIs are mostly used to read/subscribe to data.
I have limited knowledge in WCF as well as sending binary data via WCF, so this question may be somewhat rudimental.
I would like to know the difference between sending data with BinaryMessageEncodingBindingElement and MtomMessageEncodingBindingElement. It is still not clear to me when to use which approach after reading this page from MSDN on Large Data and Streaming.
Also, a small question: are a message with attachments and an MTOM message the same thing?
MTOM is a standard that uses multi-part mime-encoded messages to send portions of the message that are large and would be too expensive to base64 encode as pure binary. The SOAP message itself is sent as the initial part of the message and contains references to the binary parts which a web service software stack like WCF can then pull back together to create a single representation of the message.
Binary encoding is entirely proprietary to WCF and really doesn't just have to do with large messages. It presents a binary representation of the XML Infoset which is far more compact across the wire and faster to parse than text based formats. If you happen to be sending large binary chunks of data then it just fits right in with the other bytes that are being sent.
Streaming can be done used with any message format. That's more about when the data is written across the network vs. being buffered entirely in memoery before being presented to the network transport. Smaller messages make more sense to buffer up before sending and larger messages, especially those containing large binary chunks or streams, necessitate being streamed or will exhaust memory resources.
Does NServiceBus 2.0 allow for defining serializer for given message type?
I want for all but one of my messaages to be serialized using XmlSerializer. The remaining one should be serialized using BinarySerializer.
Is it possible with NServiceBus 2.0?
I believe the serializer is specified on an endpoint basis, so all messages using that endpoint would use the same serializer.
However, if you follow the rote NServiceBus recommendation of one message type per endpoint/queue then you could effectively isolate one message type and use a different serializer for it.
I'm curious, however, what is special about the one message type that requires binary serialization?
Edit in response to comment
The Distributor info indirectly mentions this under Routing with the Distributor. Udi Dahan also frequently advises this in the NServiceBus Yahoo Group although it's difficult to provide links because the search there is poor.
Basically, the idea is that you wouldn't want high priority messages to get stuck behind lower-priority ones, and also that this provides you with the greatest flexibility to scale out certain message processing if necessary.
Because the MsmqTransportConfig only allows for one InputQueue to be specified, having one message type per queue also means that you only have one message handler per endpoint.
To address the image, you may still be able to encapsulate it in an XML-formatted message if you encode the byte array as a Base64-encoded string. It's not ideal, but if your images aren't too large, it may be easier to do this than to go to the trouble of using a different serializer on only one message type.
Another option is to store the image data out-of-band in a database or filesystem and then refer to it by an ID or path (respectively).
Not possible in Version 2. But it can be done using the pipeline in versions 5 and above http://docs.particular.net/samples/pipeline/multi-serializer/
I have a particular service which returns large amount of data. What' is the best practice and options available in WCF to handle this?
These large data is returned after doing all the filtering, so no more filtering is possible.
The data could go into GB's. I do understand there is a limit to how much data a system can handle.
But give the above scenario what options/alternatives would you recommend.
Use streaming MSDN
MTOM is a mechanism for transmitting large binary attachments with SOAP messages as raw bytes, allowing for smaller messages.
see: http://msdn.microsoft.com/en-us/library/aa395209.aspx for details.