I have a task to use Apache Flume to send messages to a Kafka topic. The caveat is that I have to specify a partition based on an IP address that will be in the message.
Is there a way to configure Apache Flume to do this dynamically, or do I have to implement a custom Producer plugin?
Thank you.
After some research, the only real custom way of configuring the producer is to roll your own.
I implemented the AbstractSink interface and put in my own hash algorithm to generate a partition to send the message to.
Related
I was going through the documentation but couldn't find anything related to this, so I'm here just to confirm with you.
We'd like to partition Rabbit queues using Spring Cloud Stream, but would like to do so without using routing keys for it. Is there a way to do it by using headers or properties in the messages?
RabbitMQ plugins allow this but, does spring cloud stream allow it in some way?
Thanks and regards.
You can manually provision whatever exchange type you want and set the declareExchange property to false.
I am getting my data through ActiveMQ which I want to process in real time with Apache Flink DataStreams. There is support for many messaging services like RabbitMQ and Kafka but I can't see any support for ActiveMQ. How can I use it?
Since there is not support for ActiveMQ, I would recommend implement a custom source.
You basically have to implement the SourceFunction interface.
If you want to have exactly-once semantics, you can base your implementation on the MultipleIdsMessageAcknowledgingSourceBase class.
I would recommend you to start with implementing a SourceFunction
Found a JMS connector for Flink:
https://github.com/jkirsch/senser/blob/master/src/main/java/edu/tuberlin/senser/images/flink/io/FlinkJMSStreamSource.java
Is not Apache Kafka another implementation of JMS?
I am using JMS+AMQ in my application, and migrating to Apache Kafka. Do I have to change all JMS codes?
No, Kafka is different from JMS systems such as ActiveMQ.
see ActiveMQ vs Apollo vs Kafka
Kafka has less features than ActiveMQ, as the stress has been put on performances. So before migrating, check that the features you use in AMQ are in Kafka.
However, there is an open suggestion for a bridge between JMS and Kafka, to allow exactly what you need. Maybe the provided links can help you
https://issues.apache.org/jira/browse/KAFKA-1995
Actually, the two are not the same. And with a little more time seeing the two co-exist - and listening to problems and happy points from those deploying each in the field - there is a little more to say about each one.
Firstly, JMS supports both point-to-point messaging (where messages are sent to single consumers; the consumers themselves maintain their message queues) and the publish-and-subscribe (pub/sub) model (where messages are written to a single topic, and consumers, independently, decide which messages to consume).
In a point-to-point messaging architecture, message producers and consumers know each other, where as in a pub/sub model they do not. Apache Kafka focuses on a pub/sub model, maintaining a separate log/topic from which consumers read from offsets. Kafka is also built for the cloud, with high-throughput a core consideration.
Many in our community and at meetups throw their hands up in frustration at MOMs (message-oriented middlewares) like JMS and switch to Kafka, for, what boils down to one reason: scalability. They argue that Kafka is better suited for scale than other MOMs because Kafka maintains a partitioned topic log. In so doing, Kafka can split up message flow to groups of consumers by partition and batch transmit the messages.
This concept also allows Kafka to have more granular control over ACLs (access control) to Kafka Consumers, although there are some issues there, which Apache Pulsar is addressing.
Finally, on Kafka, since the client/consumer decides which messages to consume (by offset in the topic), this removes some of the producer-side complexity of routing rules built into MOMs like JMS.
There's more differences than that, but this is a distillation of some of the ones that keep coming up! Hope this helps.
No, Kafka uses its own non-standard protocol and clients.
However, there's a 3rd-party JMS Client for Kafka from Confluent.
Both are open source ESB. So what are distinct features each of them have and in which scenarios/usecases, each should be used?
Apache camel is basically a Mediation engine which will help you to define
mediation and routing rules for your application,
whereas Activemq is a message broker that allows you to
send messages between producers and consumers using Java Messaging System (JMS)
Apache Camel can use ActiveMQ for forwarding of messages.
Both are altogether different types of systems, with different functionalities, and can not be compared
hope this helps!
Good luck!
We have a use case where we would like to get a certain messages in a queue distributed into other queues after we browse the queue and get the message properties. Can that be done with JMS API for hornetq or should we use a JMX client as that seems to be a possible operation in jvisualvm?
I don't know if I understand You correctly but if You would like to copy from one queue to another only filtered messages You can use bridge.
http://docs.jboss.org/hornetq/2.3.0.CR2/docs/user-manual/html/core-bridges.html
In bridge configuration You can define which messages should be copied from one queue to another
It seems that org.hornetq.api.jms.management.JMSQueueControl from the HornetQ Management API will do the trick. I was hoping for some generic JMS API that would allow that as well but this will work.