performance test on 1000 messages in rabbitmq queue using Gatling - rabbitmq

I have 2 rabbitmq queues, one is message queue and other is result queue. I want to test the performance of message queue
When i send list of json to message queue one by one, this message is processed and then i get response in result queue in this format
for one message 5 response messages
{id:1234,status:processing}
{id:1234,status:processing}
{id:1234,status:processing}
{id:1234,status:processing}
{id:1234,status:complete}
Now I want to check the complete status and find the time required to get a response with complete status
this is the scenario
val scn: ScenarioBuilder = scenario("test1")
.exec {
s: Session =>
val list = data.toList *// data is the list of messages(they can me from 100 to 1000 or more)*
s.set("list", list)
}
.foreach(s=>s("list").as[Seq[String]],"message") {
exec(
amqp("publish message to exchange").requestReply
.queueExchange("message")
.replyExchange("result")
.textMessage("${message}")
.priority(0)
.contentType("application/json")
.headers("test" -> "performance")
.check(
jsonPath("$.id").exists,
jsonPath("$.id").saveAs("co-id"),
jsonPath("$.status").is("Complete"), *//this one fails with error msg- found processing*
jsonPath("$.status").saveAs("status")
)
).pause(20)
.exec({session =>
val status = session("status").as[String]
val coId = session("co-id").as[String]
println("Response body:::",coId,status)
session
}).pause(5)
}.pause(10)

Take a look at https://github.com/TinkoffCreditSystems/gatling-amqp-plugin/blob/master/src/test/scala/ru/tinkoff/gatling/amqp/examples/RequestReplyWithOwnMatchingExample.scala . There you will find an example how to write a test with your own matching method.
If I understand your question correct you could return from the method "matchByMessage"
for sending methods something like "messageId+"_"+complete"
for receiving message you have to extract the state from the message
If you don't want to run the plugin in debug-mode locally you you could print the message parameter from the method.

Related

Spring Webflux - initial message without subscriber

I am trying to make an SSE Spring application, using Webflux. According to the documentation, the message is not sent to the sink if there is no subscriber. In my use case, I would like that the subscriber would receive the last message when calling for subscription. I have found that Sink can be configured in following way:
Sinks.many().replay().latest();
And when I have both publisher and subscriber, and the next subscriber calls for subscription, he receives the last sent message, which is great. However if I don't have any subscribers, publisher sends the message and then first subscriber comes in, it receives none. Which is just as documentation above says actually, but I am thinking how to solve that issue to meet my needs. As a workaround I did something like this:
if (shareSinks.currentSubscriberCount() == 0) {
shareSinks.asFlux().subscribe();
}
shareSinks.tryEmitNext(shareDTO);
But subscribing the publisher to its own subscription doesn't sound like a clean way to do this...
This is a matter of hot and cold publishers. Currently, your publisher (Sinks.many().replay().latest()) is a cold publisher. Events that are being emitted while there is no subscriber, will just vanish.
What you need is a so called hot publisher. Hot publishers cache the events and a new subscriber will receive all previously cached events.
This will do the trick:
final Sinks.Many<String> shareSinks = Sinks.many()
.replay()
.all(); // or .limit(10); to keep only the last 10 emissions
final Flux<String> hotPublisher = shareSinks.asFlux()
.cache(); // .cache() turns the cold flux into a
// hot flux
shareSinks.tryEmitNext("1");
shareSinks.tryEmitNext("2");
shareSinks.tryEmitNext("3");
shareSinks.tryEmitNext("4");
hotPublisher.subscribe(message -> System.out.println("received: " + message));
The console print out would be:
received: 1
received: 2
received: 3
received: 4
The Reactor docu also has a chapter on hot vs. cold.

ActiveMQ consumer timed out

We are facing problem of ActiveMQ consumer timedout for one of our Queue(ServiceResponses). This problem started to occur after a couple of days. Here are Queues information from one Node AMQ.
Number of Consumers for Response Queue either 0 or 1 and also Number of Pending Messages exist on this Queue and seem to grow with time.
JMSTemplate with connection pooling(jms-pool) is being used for send and receive. Here is the code snippet which sends to request queue and receive from response queue
val out = mc.andThen { msg =>
msg.setJMSType(mt.jmsType)
msg.setJMSCorrelationID(correlationId)
msg.setJMSReplyTo(responseQueue)
msg
}
jmsTemplate.send(requestQueue, out)
jmsTemplate.setReceiveTimeout(timeoutInMillies)
val response = jmsTemplate.receiveSelected(responseQueue, s"JMSCorrelationID = '$correlationId'")
Timeout value is 5 seconds.
Client Connection Settings are:
spring.activemq.broker-url=failover:(ssl://HOST1:PORT?keepAlive=true&connectionTimeout=500&soTimeout=360000,ssl://HOST2:PORT?keepAlive=true&connectionTimeout=500&soTimeout=360000)?initialReconnectDelay=10&maxReconnectDelay=9000&randomize=true&timeout=1000&backup=false&priorityBackup=true
Does anyone has idea what could be the reason and how can we fix this problem?

SpringAMQP delay

I'm having trouble to identify a way to delay message level in SpringAMQP.
I call a Webservice if the service is not available or if it throws some exception I store all the requests into RabbitMQ queue and i keep retry the service call until it executes successfully. If the service keeps throwing an error or its not available the rabbitMQ listener keeps looping.( Meaning Listener retrieves the message and make service call if any error it re-queue the message)
I restricted the looping until X hours using MessagePostProcessor however i wanted to enable delay on message level and every time it tries to access the service. For example 1st try 3000ms delay and second time 6000ms so on until i try x number of time.
It would be great if you provide a few examples.
Could you please provide me some idea on this?
Well, it isn't possible the way you do that.
Message re-queuing is fully similar to transaction rallback, where the system returns to the state before an exception. So, definitely you can't modify a message to return to the queue.
Probably you have to take a look into Spring Retry project for the same reason and poll message from the queue only once and retries in memory until successful answer or retry policy exhausting. In the end you can just drop message from the queue or move it into DLQ.
See more info in the Reference Manual.
I added CustomeMessage delay exchange
#Bean
CustomExchange delayExchange() {
Map<String, Object> args = new HashMap<>();
args.put("x-delayed-type", "direct");
return new CustomExchange("delayed-exchange", "x-delayed-message", true, false, args);
}
Added MessagePostProcessor
if (message.getMessageProperties().getHeaders().get("x-delay") == null) {
message.getMessageProperties().setHeader("x-delay", 10000);
} else {
Integer integer = (Integer) message.getMessageProperties().getHeaders().get("x-delay");
if (integer < 60000) {
integer = integer + 10000;
message.getMessageProperties().setHeader("x-delay", integer);
}
}
First time it delays 30 seconds and adds 10seconds each time till it reaches 600 seconds.This should be configurable.
And finally send the message to
rabbitTemplate.convertAndSend("delayed-exchange", queueName,message, rabbitMQMessagePostProcessor);

How RabbitMQ handle if queue message bytes length large than x-max-length-bytes?

I had declare a queue like below:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length-bytes", 2 * 1024 * 1024); // Max length is 2G
channel.queueDeclare("queueName", true, false, false, args);
When the queue messages count bytes is large than 2G, It will auto remove the message on the head of the queue.
But what I expected is That it reject produce the last message and return exception to the producer.
How can I get it?
A possible workaround is check the queue size before send your message using the HTTP API.
For example if you have a queue called: myqueuetest with max size = 20.
Before send the message you can call the HTTP API in this way:
http://localhost:15672/api/queues/
the result is a JSON like this:
"message_bytes":10,
"message_bytes_ready":10,
"message_bytes_unacknowledged":0,
"message_bytes_ram":10,
"message_bytes_persistent":0,
..
"name":"myqueuetest",
"vhost":"test",
"durable":true,
"auto_delete":false,
"arguments":{
"x-max-length-bytes":20
},
then you cloud read the message_bytes field before send your message and then decide if send or not.
Hope it helps
EDIT
This workaround could kill your application performance
This workaround is not safe if you have multi-threading/more publisher
This workaround is not a very "best practise"
Just try to see if it is ok for your application.
As explained on the official docs:
Messages will be dropped or dead-lettered from the front of the queue to make room for new messages once the limit is reached.
https://www.rabbitmq.com/maxlength.html
If you think RabbitMQ should drop messages form the end of the queue, feel free to open an issue here so we can discuss about it https://github.com/rabbitmq/rabbitmq-server/issues

Query for Number of Messages in Mule ESB VM Queue

In a Mule flow, I would like to add an Exception Handler that forwards messages to a "retry queue" when there is an exception. However, I don't want this retry logic to run automatically. Instead, I'd rather receive a notification so I can review the errors and then decide whether to retry all messages in the queue or not.
I don't want to receive a notification for every exception. I'd rather have a scheduled job that runs every 15 minutes and checks to see if there are messages in this retry queue and then only send the notification if there are.
Is there any way to determine how many messages are currently in a persistent VM queue?
Assuming you use the default VM queue persistence mechanism and that the VM connector is named vmConnector, you can do this:
final String queueName = "retryQueue";
int messageCount = 0;
final VMConnector vmConnector = (VMConnector) muleContext.getRegistry()
.lookupConnector("vmConnector");
for (final Serializable key : vmConnector.getQueueProfile().getObjectStore().allKeys())
{
final QueueKey queueKey = (QueueKey) key;
if (queueName.equals(queueKey.queueName))
{
messageCount++;
}
}
System.out.printf("Queue %s has %d pending messages%n", queueName, messageCount);