Apache Beam : is it possible to comsume messages of RabbitMQ with exchange and routing key - rabbitmq

Question

I defined a pipeline in Apache Beam to consume messages of a given queue in RabbitMQ message broker.
I defined an exchange and routing key in RabbitMQ.
I used AmqpIO.read() in Beam (version 2.9.0) but I did not found any API to set the echange and the routing key.
(Following this doc : https://beam.apache.org/releases/javadoc/2.4.0/org/apache/beam/sdk/io/amqp/AmqpIO.html)
Is there any possibility to do that ? Even with any other plugin.
Regards,
Ali

Answer 1

There is a new (experimental) IO connector for RabbitMQ shipped with the latest v2.9.0 Apache Beam release. The AMQP connector will not work for RabbitMQ.
If you are using Maven add the following dependency in your POM
<!-- Beam MongoDB I/O -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-mongodb</artifactId>
<version>2.9.0</version>
</dependency>
and you can use it in a pipeline like
public class RabbitMQPipeline {
final static Logger log = LoggerFactory.getLogger(RabbitMQPipeline.class);
/**
* Mongo Pipeline options.
*/
public interface RabbitMQPipelineOptions extends PipelineOptions {
#Description("Path of the file to read from")
#Default.String("amqp://localhost")
#Required
String getUri();
void setUri(String uri);
}
/**
* #param args
*/
public static void main(String[] args) {
RabbitMQPipelineOptions options = PipelineOptionsFactory.fromArgs(args).withValidation()
.as(RabbitMQPipelineOptions.class);
Pipeline pipeline = Pipeline.create(options);
PCollection<RabbitMqMessage> messages = pipeline
.apply(RabbitMqIO2.read().withUri(options.getUri()).withQueue("test"));
messages.apply(ParDo.of(new DoFn<RabbitMqMessage, String>() {
#ProcessElement
public void process(#Element RabbitMqMessage msg) {
System.out.println(msg.toString());
}
}));
pipeline.run().waitUntilFinish();
}
}
The RabbitMqIO Javadoc has examples of how to use the reader and writer.
A word of caution
There is a known bug that has been fixed but scheduled for release in v2.11.0 that blocks the connector from working even in the simplest scenarios. The fix is really simple (see JIRA issue) but you will need to rebuild a new version of the class. In case you want to give it a try make sure you add the following Maven dependency
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value</artifactId>
<version>1.5.2</version>
<scope>provided</scope>
</dependency>
and add the following configuration in Maven Compiler plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<annotationProcessors>
<annotationProcessor>com.google.auto.value.processor.AutoValueProcessor</annotationProcessor>
</annotationProcessors>
</configuration>
</plugin>
If you are using Eclipse make sure you install the m2-apt Maven plugin. Good luck!