We are using Kafka streams to send image files from one web service to other to maintain persistency. During the process we convert image into array of bytes and then convert them into Base64 encoded string and send it as a message along with other file metadata. Due to large image file size our Kafka is going down.
Is there any better option to stream these images through Kafka along with the file metadata?
You might want to use compression of data by the Kafka producer. Spring Cloud Stream Kafka binder allows specifying the compression type using the property: spring.cloud.stream.kafka.bindings.<channelName>.producer.compressionType
This property accepts snappy and gzip as values for compression and none for no compression (which is default).
Related
strong textI create a aws Cross-Account Log Data Sharing with Subscriptions.
By follow this link
After create kinesis stream create Kinesis Data Firehose delivery streams to save logs in s3 bucket.
logs files creating in S3 bucket but in encrypted form .
And at sender side no KMS key id ..
How can i see the logs..
Also not able to decrypt in base64 manually..
Updated:
I found that logs store in S3 bucket have "Content-Type application/octet-stream". when i update content-type to "text/plain" ..
is there any way set in bucket level content type or configure in kinesis data stream or firehose
Is there any way to set content-type kinesis stream or set the default content-type for s3 folder?
The data you posted appears to be compressed (I'd need a short file sample to be able to say that for certain). If I were you I'd look into compression settings for the log stream.
Here, it references some different compressions available for data: https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html
In a pinch, if you save those data in a file and give it the ".gz" extension, does the file become readable? (I'm not too hopeful since it says that the default compression scheme is Snappy and not GZIP, and I might be mistaken but I think I see a ZIP preamble after some kind of header in your screenshot).
I want to transfer an image from a client to the server using RabbitMQ. However, from whatever I read, I understood that RabbitMQ can only transfer a stream of text. So how to transfer an image?
The amqp body is a buffer, you can send what you want.
In general, if you want to send a file, you have to read it and send the buffers.
You should send the file using more publish and not just one, then recreate the file on the consumer side.
You should avoid sending big buffer.
In ActiveMQ while using blob messages we use this as broker
String broker1 = "tcp://localhost:7005?jms.blobTransferPolicy.UploadUrl=http://localhost:7005/fileserver/"
Can anybody explain what is UploadUrl and why we need to configure for blob messages(we don't need to configure for text messages). Why it doesn't allow tcp protocol?
So plain text messages are good and easy to use, but needs to be in memory at all times. It works well with a KBs of data, or even a few MB. However, sending very large files, such as initial data loads, large media files or BI data, is not nice to keep around in memory. There may still be a need to pass the message around, route/filter based on message properties, use transactions and similar.
Blob messages is an attempt to solve the need to pass around GBs of data through the semantics of messaging. The trade off is that you have to define a streaming based server somewhere that both sender and receiver can reach. It can be HTTP, FTP, a local file, WebDAV or similar. ActiveMQ comes with a HTTP based fileserver if you have no other file area around.
I have limited knowledge in WCF as well as sending binary data via WCF, so this question may be somewhat rudimental.
I would like to know the difference between sending data with BinaryMessageEncodingBindingElement and MtomMessageEncodingBindingElement. It is still not clear to me when to use which approach after reading this page from MSDN on Large Data and Streaming.
Also, a small question: are a message with attachments and an MTOM message the same thing?
MTOM is a standard that uses multi-part mime-encoded messages to send portions of the message that are large and would be too expensive to base64 encode as pure binary. The SOAP message itself is sent as the initial part of the message and contains references to the binary parts which a web service software stack like WCF can then pull back together to create a single representation of the message.
Binary encoding is entirely proprietary to WCF and really doesn't just have to do with large messages. It presents a binary representation of the XML Infoset which is far more compact across the wire and faster to parse than text based formats. If you happen to be sending large binary chunks of data then it just fits right in with the other bytes that are being sent.
Streaming can be done used with any message format. That's more about when the data is written across the network vs. being buffered entirely in memoery before being presented to the network transport. Smaller messages make more sense to buffer up before sending and larger messages, especially those containing large binary chunks or streams, necessitate being streamed or will exhaust memory resources.
I want to send a large image to server through Wcf.I may pass it as a byte[].How can i do that.What is MTOM...where can i find a sample...?
MTOM is just an optimized message encoding format. It stands for Message Transmission Optimization Mechanism and transmits as bytes instead of angle brackets (aka XML). Here is a basic introduction to using MTOM with WCF.
This small sample should get you started if you want to use streaming, which is a nice option if you want to transmit large files.