I would like to know how to use/control the Quarkus reactive Redis client with pipelining (ReactiveRedisDataSource). I understand the client does pipelining by default, but how can we control how many commands are pipelined before the request is sent? For example:
#Inject
ReactiveRedisDataSource reactiveDS;
// some code below ...
reactiveDS.list(String.class).lpush("myList", "randomString")
.chain(() -> reactiveDS.hash(String.class).hset("myKey", "field1", "randomValue"))
Does the code above pipelines those 2 commands? If not then how can I pipeline? If yes, how can I control how many messages can I enqueue before sending and how can I get the array response?
UPDATE
This is the suggested way to do it, it works! (see this thread: https://github.com/smallrye/smallrye-mutiny/issues/1099):
rds.withConnection(redis -> {
for (int i = 0; i < 5000; i++) {
unis.add(redis.value(Integer.class).set("key-" + i, i));
}
return Uni.join().all(unis).andCollectFailures()
.replaceWithVoid();
}).await().indefinitely();
Related
My, already "legacy" by now, implementation of a pub/sub solution using ServiceStack quickly ran out of clients, when it reached the 20 client limit.
We do something like:
_redisConsumer = MqClientFactory.Instance.GetRedisClient(); // Returns a IRedisClient
_subscription = _redisConsumer.CreateSubscription();
_subscription.OnSubscribe = channel => CoreLog.Instance.Info($"Subscription started on {eventChannelName}");
_subscription.OnUnSubscribe = channel => CoreLog.Instance.Warning($"Unsubscribed from {eventChannelName}");
_subscription.OnMessage = (channel, msg) =>
{
try
{
onMessageReceived(CoreRequestJsonEnvelope.CreateCoreRequestFromJson(msg));
}
catch (Exception ex)
{
CoreLog.Instance.Exception(ex);
}
};
// Since it blocks execution, we put this in a Task:
Task.Run(() =>
{
try
{
_subscription.SubscribeToChannels(eventChannelName); // blocking
}
catch(Exception e)
{
}
});
and when we have enough different channels to listen too, it runs out.
I then thought, that maybe instead of taking a new IRedisClient for each subscription, I could use the same IRedisClient for all of them, so:
_redisConsumer = mySavedRedisClient;
...
but that returns Unknown reply on multi-request after a few seconds/executions.
Lastly, I looked at the RedisPubSubServer, but it seems that I need to specify the channels in the constructor, and I cannot change after that. I do need to add and remove channels in runtime, and channels are not known from start.
What is the recommended approach?
Is it to increaase the Max-limit and continue as before?
Is it to use RedisPubSub, but how to handle dynamic channels?
What does "unknown reply on multi-request" actually mean?
Thanks!
It's not clear what 20 client limit you're referring to & how the client limit is dependent on channels or subscribers, but if this is your App own limit than sounds like increasing it would be the easiest solution.
ServiceStack.Redis doesn't support changing the subscribed channels after a subscription has started. Instead of managing the IRedisSubscription yourself you may want to consider ServiceStack.Redis Managed Pub/Sub Server which manages the background subscription thread with added resiliency and support for auto retries.
Whilst you can't change the subscribed channels at runtime, you can modify the modify the Channels collection and restart the subscription to create a new subscription to the updated channels list, e.g:
var pubSub = new RedisPubSubServer(clientsManager, chan1);
pubSub.Start();
//...
pubSub.Channels = new[] { chan1, chan2 };
pubSub.Restart();
Otherwise depending on your use-case you may be able to subscribe to a channel pattern which allows you to subscribe to a multiple dynamic channels matching wildcard channel pattern:
var pubSub = new RedisPubSubServer(clientsManager) {
ChannelsMatching = new[] { "chan:*" }
}
.Start();
Where it will handle any messages clients send that matches the channel pattern.
I try to override the set Redis built-in command by implementing a module using the Redis API. I would like to publish the set value on a specific channel. notify-keyspace-events KEAg would have been a good solution if the value was sent on the channel but it is not the case.
I also tried to override the set command directly with the API, but RedisModule_CreateCommand seems only work for new commands, not for built-in commands.
I also tried to create a command like setpub and use rename-command SET SETPUB but SETPUB doesn't seem to be recognize, even if we userename-commandafterloadmodule setpub.so`.
Is there a way to override the built-in commands?
No, there isn't. There will probably soon be a way to wrap native redis commands with hooks, but that's still not available.
But you can use the new module notification API in 4.0.9, and register a notification handler inside your module, and in it extract the value of the key being set - then publish it.
The notification handler should look something likes this (not testing for validity):
int NotifyCallback(RedisModuleCtx *ctx, int type, const char *event,
RedisModuleString *key) {
// We only care about SET events, right?
if (!strcasecmp(event, "SET")) {
// Open the key to get the string in it. We could have used RedisModule_Call but this is faster:
RedisModuleKey *k = RedisModule_OpenKey(ctx, key, REDISMODULE_READ);
// Just to be safe...
if (k && RedisModule_KeyType(k) == REDISMODULE_KEYTYPE_STRING) {
// Get direct access to the string. Careful now!
size_t len;
char *str = RedisModule_StringDMA(ctx, k, &len, REDISMODULE_READ);
// Sadly PUBLISH is only supported with RM_Call at the moment...
// Do change what you're publishing here of course
RedisModule_Call(ctx, "PUBLISH", "cb", MY_PUBSUB_CHANNEL, str, len);
// Cleanup
RedisModule_CloseKey(k);
}
}
return REDISMODULE_OK;
}
And you can register the handler in your module init call:
RedisModule_SubscribeToKeyspaceEvents(ctx, REDISMODULE_NOTIFY_STRING, NotifyCallback);
Is there a way to receive multiple message using a single synchronous call ?
When I know that there are N messages( N could be a small value less than 10) in the queue, then I should be able to do something like channel.basic_get(String queue, boolean autoAck , int numberofMsg ). I don't want to make multiple requests to the server .
RabbitMQ's basic.get doesn't support multiple messages unfortunately as seen in the docs. The preferred method to retrieve multiple messages is to use basic.consume which will push the messages to the client avoiding multiple round trips. acks are asynchronous so your client won't be waiting for the server to respond. basic.consume also has the benefit of allowing RabbitMQ to redeliver the message if the client disconnects, something that basic.get cannot do. This can be turned off as well setting no-ack to true.
Setting basic.qos prefetch-count will set the number of messages to push to the client at any time. If there isn't a message waiting on the client side (which would return immediately) client libraries tend to block with an optional timeout.
You can use a QueueingConsumer implementation of Consumer interface which allows you to retrieve several messages in a single request.
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
channel.basicConsume(plugin.getQueueName(), false, queueingConsumer);
for(int i = 0; i < 10; i++){
QueueingConsumer.Delivery delivery = queueingConsumer.nextDelivery(100);//read timeout in ms
if(delivery == null){
break;
}
}
Not an elegant solution and does not solve making multiple calls but you can use the MessageCount method. For example:
bool noAck = false;
var messageCount = channel.MessageCount("hello");
BasicGetResult result = null;
if (messageCount == 0)
{
// No messages available
}
else
{
while (messageCount > 0)
{
result = channel.BasicGet("hello", noAck);
var message = Encoding.UTF8.GetString(result.Body);
//process message .....
messageCount = channel.MessageCount("hello");
}
First declare instance of QueueingBasicConsumer() wich wraps the model.
From the model execute model.BasicConsume(QueueName, false, consumer)
Then implement a loop that will loop around messages from the queue which will then processing
Next line - consumer.Queue.Dequeue() method - waiting for the message to be received from the queue.
Then convert byte array to a string and display it.
Model.BasicAck() - release message from the queue to receive next message
And then on the server side can start waiting for the next message to come through:
public string GetMessagesByQueue(string QueueName)
{
var consumer = new QueueingBasicConsumer(_model);
_model.BasicConsume(QueueName, false, consumer);
string message = string.Empty;
while (Enabled)
{
//Get next message
var deliveryArgs = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
//Serialize message
message = Encoding.Default.GetString(deliveryArgs.Body);
_model.BasicAck(deliveryArgs.DeliveryTag, false);
}
return message;
}
If I'm connected to RabbitMQ and listening for events using an EventingBasicConsumer, how can I tell if I've been disconnected from the server?
I know there is a Shutdown event, but it doesn't fire if I unplug my network cable to simulate a failure.
I've also tried the ModelShutdown event, and CallbackException on the model but none seem to work.
EDIT-----
The one I marked as the answer is correct, but it was only part of the solution for me. There is also HeartBeat functionality built into RabbitMQ. The server specifies it in the configuration file. It defaults to 10 minutes but of course you can change that.
The client can also request a different interval for the heartbeat by setting the RequestedHeartbeat value on the ConnectionFactory instance.
I'm guessing that you're using the C# library? (but even so I think the others have a similar event).
You can do the following:
public class MyRabbitConsumer
{
private IConnection connection;
public void Connect()
{
connection = CreateAndOpenConnection();
connection.ConnectionShutdown += connection_ConnectionShutdown;
}
public IConnection CreateAndOpenConnection() { ... }
private void connection_ConnectionShutdown(IConnection connection, ShutdownEventArgs reason)
{
}
}
This is an example of it, but the marked answer is what lead me to this.
var factory = new ConnectionFactory
{
HostName = "MY_HOST_NAME",
UserName = "USERNAME",
Password = "PASSWORD",
RequestedHeartbeat = 30
};
using (var connection = factory.CreateConnection())
{
connection.ConnectionShutdown += (o, e) =>
{
//handle disconnect
};
using (var model = connection.CreateModel())
{
model.ExchangeDeclare(EXCHANGE_NAME, "topic");
var queueName = model.QueueDeclare();
model.QueueBind(queueName, EXCHANGE_NAME, "#");
var consumer = new QueueingBasicConsumer(model);
model.BasicConsume(queueName, true, consumer);
while (!stop)
{
BasicDeliverEventArgs args;
consumer.Queue.Dequeue(5000, out args);
if (stop) return;
if (args == null) continue;
if (args.Body.Length == 0) continue;
Task.Factory.StartNew(() =>
{
//Do work here on different thread then this one
}, TaskCreationOptions.PreferFairness);
}
}
}
A few things to note about this.
I'm using # for the topic. This grabs everything. Usually you want to limit by a topic.
I'm setting a variable called "stop" to determine when the process should end. You'll notice the loop runs forever until that variable is true.
The Dequeue waits 5 seconds then leaves without getting data if there is no new message. This is to ensure we listen for that stop variable and actually quit at some point. Change the value to your liking.
When a message comes in I spawn the handling code on a new thread. The current thread is being reserved for just listening to the rabbitmq messages and if a handler takes too long to process I don't want it slowing down the other messages. You may or may not need this depending on your implementation. Be careful however writing the code to handle the messages. If it takes a minute to run and your getting messages at sub-second times you will run out of memory or at least into severe performance issues.
I have a handler similar to the following, which essentially responds to a command and sends a whole bunch of commands to a different queue.
public void Handle(ISomeCommand message)
{
int i=0;
while (i < 10000)
{
var command = Bus.CreateInstance<IAnotherCommand>();
command.Id = i;
Bus.Send("target.queue#d1555", command);
i++;
}
}
The issue with this block is, until the loop is fully completed none of the messages appear in the target queue or in the outgoing queue. Can someone help me understand this behavior?
Also if I use Tasks to send messages within the Handler as below, messages appear immediately. So two questions on this,
What's the explanation on Task based Sends to go through immediately?
Are there are any ramifications on using Tasks with in message handlers?
public void Handle(ISomeCommand message)
{
int i=0;
while (i < 10000)
{
System.Threading.ThreadPool.QueueUserWorkItem((args) =>
{
var command = Bus.CreateInstance<IAnotherCommand>();
command.Id = i;
Bus.Send("target.queue#d1555", command);
i++;
});
}
}
Your time is much appreciated!
First question: Picking a message from a queue, running all the registered message handlers for it AND any other transactional action(like writing new messages or writes against a database) is performed in ONE transaction. Either it all completes or none of it. So what you are seeing is the expected behaviour: picking a message from the queue, handling ISomeCommand and writing 10000 new IAnotherCommand is either done completely or none of it. To avoid this behaviour you can do one of the following:
Configure your NServiceBus endpoint to not be transactional
public class EndpointConfig : IConfigureThisEndpoint, AsA_Publisher,IWantCustomInitialization
{
public void Init()
{
Configure.With()
.DefaultBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(false)
.UnicastBus();
}
}
Wrap the sending of IAnotherCommand in a transaction scope that suppresses the ambient transaction.
public void Handle(ISomeCommand message)
{
using (new TransactionScope(TransactionScopeOption.Suppress))
{
int i=0;
while (i < 10000)
{
var command = Bus.CreateInstance();
command.Id = i;
Bus.Send("target.queue#d1555", command);
i++;
}
}
}
Issue the Bus.Send on another thread, by either starting a new thread yourself, using System.Threading.ThreadPool.QueueUserWorkItem or the Task classes. This works because an ambient transaction is not automatically carried over to a new thread.
Second question: The ramifications of using Tasks, or any of the other methods I mentioned, is that you have no transactional quarantee for the whole thing.
How do you handle the case where you have generated 5000 IAnotherMessage and the power suddenly goes out?
If you use 2) or 3) the original ISomeMessage will not complete and will be retried automatically by NServiceBus when you start up the endpoint again. End result: 5000 + 10000 IAnotherCommands.
If you use 1) you will lose IAnotherMessage completely and end up with only 5000 IAnotherCommands.
Using the recommended transactional way, the initial 5000 IAnotherCommands would be discarded, the original ISomeMessage comes back on the queue and is retried when the endpoint starts up again. Net result: 10000 IAnotherCommands.
If memory serves NServiceBus wraps the calls to the message handlers in a TransactionScope if the transaction option is used and TransactionScope needs some help to be cross-thread friendly:
TransactionScope and multi-threading
If you are trying to reduce overhead you can also bundle your messages. The signature for the send is Bus.Send(IMessage[]messages). If you can guarantee that you don't blow up the size limit for MSMQ, then you could Send() all the messages at once. If the size limit is an issue, then you can chunk them up or use the Databus.