I'm trying to add Spring Cloud Stream to the existing project that uses Spring AMQP with RabbitMQ.
I have the following rabbit configuration:
Producer exchange name: producer.mail-sent.exchange
Consumer queue name: consumer.mail-sent.queue
On the producer's side I configure like this:
spring:
cloud:
stream:
bindings:
output:
contentType: application/json
destination: producer.mail-sent.exchange
And using the following code:
#Autowired
private Source source;
...
source.output().send(MessageBuilder.withPayload(someStuff).build());
...
On the consumer's side I have the following config:
spring:
cloud:
stream:
bindings:
input:
contentType: application/json
destination: producer.mail-sent.exchange
group: consumer.mail-sent.queue
With the following code:
#EnableBinding(Sink.class)
...
#StreamListener(Sink.INPUT)
public void handle(String someStuff) {
log.info("some stuff is received: " + someStuff);
}
And it seems that it works. :)
But! On the rabbit's side I have a new queue named producer.mail-sent.exchange.consumer.mail-sent.queue, but I want it to use the existing queue named consumer.mail-sent.queue.
Is there any way to achieve this?
It's not currently supported; while many properties are configurable (routing key etc), the queue name is always <destination>.<group>.
If you want to consume from an existing application, consider using a #RabbitListner instead of a #StreamListener.
Feel free to open a GitHub Issue referencing this post - many other "opinionated" configuration settings (such as routing key) are configurable, but not the queue name itself. Perhaps we could add a boolean includeDestInQueueName. Reference this question in the issue.
Related
I have an ActiveMQ AWS service with protocol AMQP. AWS returns to me:
failover:(amqp+ssl://b-ca138bd4-e6c4-4596-8329-f11bebf40111-1.mq.us-east-1.amazonaws.com:5671,amqp+ssl://b-ca138bd4-e6c4-4596-8329-f11bebf40111-2.mq.us-east-1.amazonaws.com:5671)
I am trying to implement using Spring Boot the connection with that endpoint, but I have many problems. I have tried with many ways, but I can't connect to the ActiveMQ using Spring.
I have tried:
Creating many configuration Beans, like:
#Bean
fun connectionFactory(): ConnectionFactory {
val activeMQConnectionFactory = ActiveMQConnectionFactory()
activeMQConnectionFactory.brokerURL = "amqp+ssl://b-ca138bd4-e6c4-4596-8329-f11bebf40111-1.mq.us-east-1.amazonaws.com:5671"
activeMQConnectionFactory.trustedPackages = listOf("com.rappi.scaffolding")
return activeMQConnectionFactory
}
and using many dependencies like:
implementation("org.apache.activemq:activemq-spring:5.17.0")
implementation("org.springframework:spring-jms")
and
implementation("org.springframework.boot:spring-boot-starter-artemis")
But is not possible for me establish the connection. At this moment I am seeing this error:
Reason: java.io.IOException: Transport scheme NOT recognized: [amqp+ssl]
There are some example in Java or Kotlin or guide to connect me with AWS using AMQP protocol? I didn't find any in Google.
I have read that using QPid, but it not works for me.
I have found many examples using Rabbit, but not Apache ActiveMQ protocol amqp+ssl.
Finally It works using the Bean:
#Bean
fun connectionFactory(): ConnectionFactory {
return JmsConnectionFactory(
"failover:(amqps://b-ca138bd4-e6c4-4596-8329-f11bebf40111-1.mq.us-east-1.amazonaws.com:5671,amqps://b-ca138bd4-e6c4-4596-8329-f11bebf40111-2.mq.us-east-1.amazonaws.com:5671)").apply {
this.username = user
this.password = passwordAQ
}
There are many things wrong with your code and configuration.
First, the URL you're using for your client is incorrect. The amqp+ssl scheme is not valid for any client. That's the scheme used to define the connector in the ActiveMQ broker configuration.
Second, your dependencies are wrong. As far as the client goes you just need:
implementation("org.apache.qpid:qpid-jms-client:1.6.0")
Of course, if you're using Spring you'll need all the related Spring dependencies, but as far as the client itself goes this is all you need.
Third, your code is wrong. You should be using something like this:
#Bean
fun connectionFactory(): ConnectionFactory {
return new org.apache.qpid.jms.JmsConnectionFactory("failover:(amqps://b-ca138bd4-e6c4-4596-8329-f11bebf40111-1.mq.us-east-1.amazonaws.com:5671,amqps://b-ca138bd4-e6c4-4596-8329-f11bebf40111-2.mq.us-east-1.amazonaws.com:5671)");
}
I have a server which publishes rabbitmq messages on a exchange, so I tried to create following async api specs for this -
asyncapi: 2.3.0
info:
title: Hello World
version: 1.0.0
description: Get Hello World Messages
contact: {}
servers:
local:
url: amqp://rabbitmq
description: RabbitMQ
protocol: amqp
protocolVersion: 0.9.1
defaultContentType: application/json
channels:
hellow_world:
subscribe:
operationId: HelloWorldSubscriber
description:
message:
$ref: '#/components/messages/HellowWorldEvent'
bindings:
amqp:
ack: true
cc: ["hello_world_routing_key"]
bindingVersion: 0.2.0
bindings:
amqp:
is: routingKey
exchange:
name: hello_world_exchange
type: direct
durable: true
vhost: /
bindingVersion: 0.2.0
components:
messages:
HellowWorldEvent:
payload:
type: object
properties: []
Based on my understanding what it means is that MyApp will publish helloworldevent message on hello_world_exchange exchange using routing key hello_world_routing_key
Question -
How can consumer/subscriber can define which queue he will be using for consuming this message ?
Do I need to define new schema for subscriber and define queue element there ?
I can define another queue.** elements in channel element, but that can only specify 1 queue element, what if there are more than 1 subscriber/consumer, so how we can specify different queues for them ?
Reference -
https://github.com/asyncapi/bindings/tree/master/amqp
I see you have not yet approved any of the responses as a solution. Is this still an issue? Are you using the AsyncAPI generator to generate your code stubs?
If so the generator creates a consumer/subscriber. If you want different processing/business logic you would generate new stubs and configure the queues they listen from. The queue is an implementation detail. I had an issue with the node.js generator for AMQP and RabbitMQ and so I decided to test the spec against Python to see if it was me or the generator.
Try the generator and you can try my gist: https://gist.github.com/adrianvolpe/27e9f02187c5b31247aaf947fa4a7360. I did do this for version 2.2.0 so hopefully it works for you.
I also did a test with the Python pika library however I did not assign a binding to the queue.
I noticed in the above spec you are setting your exchange type to Direct. You can have the same binding with multiple consumers with both Direct and Topic exchanges however you may want Topic as quoted from the RabbitMQ docs:
https://www.rabbitmq.com/tutorials/tutorial-five-python.html
Topic exchange is powerful and can behave like other exchanges.
When a queue is bound with "#" (hash) binding key - it will receive all the messages, regardless of the routing key - like in fanout exchange.
When special characters "*" (star) and "#" (hash) aren't used in bindings, the topic exchange will behave just like a direct one.
Best of luck!
I have an application which connects to ActiveMQ using a "failover" URL string. The admins are adding authentication to the brokers. Is it possible to put jms.userName and jms.password into the URL string? An example with dummy values would be most helpful.
Yes, exactly how you specified it works. The jms. prefix configures any of the setters on the ActiveMQConnectionFactory.
failover:(tcp://127.0.0.1:61616)?jms.userName=admin&jms.password=admin
Log confirmation:
09:41:53.429 INFO [ActiveMQ Task-1] Successfully connected to tcp://127.0.0.1:61616
09:41:53.481 INFO [Blueprint Event Dispatcher: 1] Route: route1 started and consuming from: amq://queue:VQ.ORDER.VT.ORDER.EVENT
I am experimenting with Spring Cloud APIs as part of microservices course.
To setup server-less task, I am using Cloud Task, Cloud Stream(RabbitMQ), and Spring Web.
For this I have setup following projects:
Serverless task to be executed -
https://github.com/Omkar-Shetkar/pluralsight-springcloud-m3-task
Component to receive Http request from user and submit to RabbitMQ -
https://github.com/Omkar-Shetkar/pluralsight-springcloud-m3-taskintake
Sink component to receive TaskLaunchRequest and forward to cloud task - https://github.com/Omkar-Shetkar/pluralsight-springcloud-m3-tasksink
Having setup above components, ensured that task component is available in local maven repository.
After initiating a POST request onto /tasks in pluralsight.com.TaskController.launchTask(String) I see a HTTP response.
But, I couldn't see any update in tasklogs DB associated with serverless task.
This means, task it self is not called.
In RabbitMQ console I could see connections are established from intake and sink components but I don't see any message exchange happening.
Queue with name tasktopic is having ZERO message count.
Appreciate any pointers and suggestions on how to proceed on this to resolve this issue.
Thanks.
There were two issue with my current implementation:
In intake and sink modules -> application.properties, binding property key was wrong.
It should be:
In intake module
spring.cloud.stream.bindings.output.destination=tasktopic
In sink module
spring.cloud.stream.bindings.input.destination=tasktopic
Also, local cloud deployer versions were incompatible in sink modules pom.xml.
Updated the same to:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-deployer-local</artifactId>
<version>1.3.0.RELEASE</version>
</dependency>
With these changes, I am able to get RabbitMQ messages.
#EnableTaskLauncher annotation is missing in TaskIntakeApplication.
#SpringBootApplication
#EnableTaskLauncher
public class PluralsightSpringcloudM3TaskintakeApplication {
public static void main(String[] args) {
SpringApplication.run(PluralsightSpringcloudM3TaskintakeApplication.class, args);
}
}
I have a .net micro-service receiving messages using RabbitMQ client, I need to test the following:
1- consumer is successfully connected to rabbitMq host.
2- consumer is listening to queue.
3- consumer is receiving messages successfully.
To achieve the above, I have created a sample application that sends messages and I am debugging consumer to be sure that it is receiving messages.
How can I automate this test? hence include it in my micro-service CI.
I am thinking to include my sample app in my CI so I can fire a message then run a consumer unit test that waits a specific time then passes if the message received, but this seems like a wrong practice to me because the test will not start until a few seconds the message is fired.
Another way I am thinking of is firing the sample application from the unit test itself, but if the sample app fails to work that would make it the service fault.
Is there any best practices for integration testing of micro-services connecting through RabbitMQ?
I have built many such tests. I have thrown up some basic code on
GitHub here with .NET Core 2.0.
You will need a RabbitMQ cluster for these automated tests. Each test starts by eliminating the queue to ensure that no messages already exist. Pre existing messages from another test will break the current test.
I have a simple helper to delete the queue. In my applications, they always declare their own queues, but if that is not your case then you'll have to create the queue again and any bindings to any exchanges.
public class QueueDestroyer
{
public static void DeleteQueue(string queueName, string virtualHost)
{
var connectionFactory = new ConnectionFactory();
connectionFactory.HostName = "localhost";
connectionFactory.UserName = "guest";
connectionFactory.Password = "guest";
connectionFactory.VirtualHost = virtualHost;
var connection = connectionFactory.CreateConnection();
var channel = connection.CreateModel();
channel.QueueDelete(queueName);
connection.Close();
}
}
I have created a very simple consumer example that represents your microservice. It runs in a Task until cancellation.
public class Consumer
{
private IMessageProcessor _messageProcessor;
private Task _consumerTask;
public Consumer(IMessageProcessor messageProcessor)
{
_messageProcessor = messageProcessor;
}
public void Consume(CancellationToken token, string queueName)
{
_consumerTask = Task.Run(() =>
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: queueName,
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
_messageProcessor.ProcessMessage(message);
};
channel.BasicConsume(queue: queueName,
autoAck: false,
consumer: consumer);
while (!token.IsCancellationRequested)
Thread.Sleep(1000);
}
}
});
}
public void WaitForCompletion()
{
_consumerTask.Wait();
}
}
The consumer has an IMessageProcessor interface that will do the work of processing the message. In my integration test I created a fake. You would probably use your preferred mocking framework for this.
The test publisher publishes a message to the queue.
public class TestPublisher
{
public void Publish(string queueName, string message)
{
var factory = new ConnectionFactory() { HostName = "localhost", UserName="guest", Password="guest" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish(exchange: "",
routingKey: queueName,
basicProperties: null,
body: body);
}
}
}
My example test looks like this:
[Fact]
public void If_SendMessageToQueue_ThenConsumerReceiv4es()
{
// ARRANGE
QueueDestroyer.DeleteQueue("queueX", "/");
var cts = new CancellationTokenSource();
var fake = new FakeProcessor();
var myMicroService = new Consumer(fake);
// ACT
myMicroService.Consume(cts.Token, "queueX");
var producer = new TestPublisher();
producer.Publish("queueX", "hello");
Thread.Sleep(1000); // make sure the consumer will have received the message
cts.Cancel();
// ASSERT
Assert.Equal(1, fake.Messages.Count);
Assert.Equal("hello", fake.Messages[0]);
}
My fake is this:
public class FakeProcessor : IMessageProcessor
{
public List<string> Messages { get; set; }
public FakeProcessor()
{
Messages = new List<string>();
}
public void ProcessMessage(string message)
{
Messages.Add(message);
}
}
Additional advice is:
If you can append randomized text to your queue and exchange names on each test run then do so to avoid concurrent tests interfering with each other
I have some helpers in the code for declaring queues, exchanges and bindings also, if your applications don't do that.
Write a connection killer class that will force close connections and check your applications still work and can recover. I have code for that, but not in .NET Core. Just ask me for it and I can modify it to run in .NET Core.
In general, I think you should avoid including other microservices in your integration tests. If you send a message from one service to another and expect a message back for example, then create a fake consumer that can mock the expected behaviour. If you receive messages from other services then create fake publishers in your integration test project.
I was successfully doing such kind of test. You need test instance of RabbitMQ, test exchange to send messages to and test queue to connect to receive messages.
Do not mock everything!
But, with test consumer, producer and test instance of rabbitMQ there is no actual production code in that test.
use test rabbitMQ instance and real aplication
In order to have meaniningfull test I would use test RabbitMQ instance, exchange and queue, but leave real application (producer and consumer).
I would implement following scenario
when test application does something that test message to rabbitMQ
then number of received messages in rabbitMQ is increased then
application does something that it should do upon receiving messages
Steps 1 and 3 are application-specific. Your application sends messages to rabbitMQ based on some external event (HTTP message received? timer event?). You could reproduce such condition in your test, so application will send message (to test rabbitMQ instance).
Same story for verifying application action upon receiving message. Application should do something observable upon receiving messages.
If application makes HTTP call- then you can mock that HTTP endpoint and verify received messages. If application saves messages to the database- you could pool database to look for your message.
use rabbitMQ monitoring API
Step 2 can be implemented using RabbitMQ monitoring API (there are methods to see number of messages received and consumed from queue https://www.rabbitmq.com/monitoring.html#rabbitmq-metrics)
consider using spring boot to have health checks
If you are java-based and then using Spring Boot will significantly simpify your problem. You will automatically get health check for your rabbitMQ connection!
See https://spring.io/guides/gs/messaging-rabbitmq/ for tutorial how to connect to RabbitMQ using Spring boot.
Spring boot application exposes health information (using HTTP endpoint /health) for every attached external resource (database, messaging, jms, etc)
See https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html#_auto_configured_healthindicators for details.
If connection to rabbitMQ is down then health check (done by org.springframework.boot.actuate.amqp.RabbitHealthIndicator) will return HTTP code 4xx and meaninfull json message in JSON body.
You do not have to do anything particular to have that health check- just using org.springframework.boot:spring-boot-starter-amqp as maven/gradle dependency is enough.
CI test- from src/test directory
I have written such test (that connect to external test instance of RabbitMQ) using integration tests, in src/test directory. If using Spring Boot it is easiest to do that using test profile, and having details of connection to test RabbitMQ instance in application-test.properties (production could use production profile, and application-production.properties file with production instance of RabbitMQ).
In simplest case (just verify connection to rabbitMQ) all you need is to start application normally and validate /health endpoint.
In this case I would do following CI steps
one that builds (gradle build)
one that run unit tests (tests without any external dependenices)
one that run integration tests
CI test- external
Above described approach could also be done for application deployed to test environment (and connected to test rabbitMQ instance). As soon as application starts, you can check /health endpoint to make sure it is connected to rabbitMQ instance.
If you make your application send message to rabbitMQ, then you could observe rabbbitMQ metrics (using rabbitMQ monitoring API) and observe external effects of message being consumed by application.
For such test you need to start and deploy your application from CI befor starting tests.
for that scenario I would do following CI steps
step that that builds app
steps that run all tests in src/test directory (unit, integration)
step that deploys app to test environment, or starts dockerized application
step that runs external tests
for dockerized environment, step that stops docker containers
Consider dockerized enevironment
For external test you could run your application along with test RabbitMQ instance in Docker. You will need two docker containers.
one with application
one with rabbitMQ . There is official docker image for rabbitmq https://hub.docker.com/_/rabbitmq/ and it is really easy to use
To run those two images, it is most reasonable to write docker-compose file.