I am a new learner of kafka.But what make me confused is that there seems to be two packages of kafka clients.
One is kafka.javaapi.* like
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
the other is org.apache.kafka.*. like
import org.apache.kafka.clients.producer.KafkaProducer<K,V>
which is shown is page http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer
I don't know what's their differences.
Before Kafka 0.8.2, kafka.javaapi.producer.Producer was the only official Java client (producer) which is implemented with Scala.
From Kafka 0.8.2, there comes a new Java producer API, org.apache.kafka.clients.producer.KafkaProducer, which is fully implemented with Java.
Kafka 0.8.2 Documentation says
We are in the process of rewritting the JVM clients for Kafka. As of 0.8.2 Kafka includes a newly rewritten Java producer. The next release will include an equivalent Java consumer. These new clients are meant to supplant the existing Scala clients, but for compatability they will co-exist for some time. These clients are available in a seperate jar with minimal dependencies, while the old Scala clients remain packaged with the server.
If you are interested in kafka.javaapi.producer.Producer, refer to 2.1 Producer API in Kafka 0.8.1 Documentation.
Related
I am currently developing a typical IoT service. At the moment multiple devices connect to one MQTT broker (mosquitto) and my java backend also connects to the broker (Paho).
The problem i see is the following:
When i am going to have multiple instances of my java backend every backend will receive and process every message received. That`s a big issue. I just want to deliver a message to only one java backend. Anybody an idea how to deal with this problem?
Btw: Java backends will be added or removed depending on the load.
There are a couple of options
Place a queuing system between your application and the MQTT broker, possibly something like Apache Kafka
HiveMQ and IBM MessageSight brokers support (different implementations) of something called shared subscriptions. This allows messages to be shared out between more than one client. Shared subscriptions is likely to be formally added to the MQTTv5 spec which should mean that it will be added to more broker and have a standard implementation.
I'm building a spring-websocket application that currently uses RabbitMQ as a message broker via the STOMP protocol. The rest of our organization mostly uses IBM Websphere MQ as a message broker, so we'd like to convert it away from RabbitMQ. However Websphere MQ doesn't support the STOMP protocol, which is spring-websocket's default. MQTT seems like the easiest supported protocol to use instead. Ideally our front-end web clients will continue to use STOMP, but I'm also OK with migrating them to MQTT if needed.
What classes do I need to overwrite to make spring-websocket interface with the broker via MQTT instead of STOMP? This article provides some general guidance that I should extend AbstractMessageBrokerConfiguration, but I'm unclear where to begin.
Currently I'm using the standard configuration methods: registry.enableStompBrokerRelay and registerStompEndpoints in AbstractWebSocketMessageBrokerConfigurer
Ryan has some good pointers.
The main work is going to be creating a replacement for StompBrokerRelayMessageHandler with an MqttBrokerMessageHandler that not only talks to an MQTT broker but also adapts client STOMP frames to MQTT and vice versa. The protocols are similar enough that it may be possible to find common ground but you won't know until you try.
Note that we did have plans for for MQTT support https://jira.spring.io/browse/SPR-12581 but the key issue was that SockJS which is required over the Web for fallback support does not support binary messages.
Here's my stab at this after reviewing the spring-websocket source code:
Change WebSocketConfig:
Remove #EnableWebSocketMessageBroker
Add new annotation: #EnableMqttWebSocketMessageBroker
Create MqttBrokerMessageHandler that extends AbstractBrokerMessageHandler -- suggest we copy and edit StompBrokerRelayMessageHandler
Create a new class that EnableMqttWebSocketMessageBroker imports: DelegatingMqttWebSocketMessageBrokerConfiguration
DelegatingMqttWebSocketMessageBrokerConfiguration extends AbstractMessageBrokerConfiguration directly and routes to MqttBrokerMessageHandler
Add this to server.xml on WebSphere Liberty:
<feature>websocket-1.1</feature>
Can anyone point me to decent example where Java stomp client is used to connect to ActiveMQ.
Also I am interested in following:
Is failover supported over stomp?
How to create durable subscription?
Does stomp support asynchronous messaging? Examples? I think I have to implement MessageListener interface for it, but I wasn't able to find example for this.
If you really want to use STOMP from Java then you could look at StompJMS which maps quite a bit of the JMS API to STOMP. It doesn't support failover but there aren't a lot of stomp client's that do. When using Java you are better off to use the native JMS client from the ActiveMQ broker as it is going to be the most robust and feature complete client library you will find.
RabbitMQ supports multiple protocols, AMQP, MQTT, STOMP, ....
When using PHP for example, it's easier to publish using the STOMP library since the PHP AMQP libraries requires compiled C code and is somewhat of a mission to setup if you don't have to.
On the JAVA side, apache camel with AMQP on spring is pretty straight forward.
Is it possible to setup a queue, publish to it via STOMP and then consume via AMQP and then again publish via AMQP and consume via STOMP if the message broker is RabbitMQ?
Yes, this should work, given that you have installed RabbitMQ's STOMP plugin on your RabbitMQ node(s).
The protocol only defines the communication between client and server and has no impact on a message itself.
You should note that using protocols other than AMQP will most likely come along with limitations and/or worse performance.
There also exist native PHP libraries for RabbitMQ that don't require compiling C code. Unfortunately, I cannot tell you which one is the best, because I am a Java guy ;-).
How to use RabbitMQ message broker in Java application? I have not found any document or link to understand how to do it.
There's plenty or RabbitMQ documentation.
In particular, you may be interested in the Java client API guide and the javadocs.