I want to use StackExchange.Redis to develop a PUB/SUB message queue.
In StackExchange.Redis documentation it is specified that:
-The data is not sent via pub/sub; the pub/sub API is only used to notify workers to check for more work
So I understand that I may need to use other redis operations like LPOP to check for work once a subscribed worker detects a notification via PUB/SUB.
However Redis reference specifies:
Once the client enters the subscribed state it is not supposed to
issue any other commands, except for additional SUBSCRIBE, PSUBSCRIBE,
UNSUBSCRIBE and PUNSUBSCRIBE commands.
With StackExchange.Redis am I supposed to create another Multiplexer in my subscribe handler to issue commands like LPOP?
Is it possible to share the same multiplexer for both Subscribe and other commands?
Should my worker unsubscribe when it receives a notification to check for work?
Did you try to use a shared multiplexer?
No there is absolutely no issue in using only one ConnectionMultiplexer and subscribe to as many things you want and do other stuff with it, as the Stackexchange.Redis client does all the stuff behind the scenes for you. It does actually handle a new connection for the subscriber internally.
Meaning, no, you don't have to create a new Multiplexer every time, and no, you don't have to issue any other commands to use the subscriber.
Just create a new subscriber
ISubscriber sub = redis.GetSubscriber();
then subscribe to whatever you want
sub.Subscribe("messages", (channel, message) => {
Console.WriteLine((string)message);
});
or send messages
sub.Publish("messages", "hello");
As explained also in Marc's docs
Related
I am working on a node JS app that connect to redis server and subscribe to a channel to get the messages.
There is a bit of confusion, should we really enable "key space notifications" on redis config to get the events in client
The same scenario I have tried using rdis cli, with which i see "key space notifications" are not enabled at the same time I have subscribed to a channel with a pattern, so when ever I publish a message from the other client, I am able to capture that event in subscribed client.
Is the "key space notifications" mandatory , but the POC says other way.
Does any one know what should be the right approach here, subscribing to channel is suffice to get messages, and its nothing to do with "key-space-notifications" ??
From Redis Keyspace Notifications
Keyspace notifications allow clients to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way.
Examples of events that can be received are:
All the commands affecting a given key.
All the keys receiving an LPUSH operation.
All the keys expiring in the database 0.
Events are delivered using the normal Pub/Sub layer of Redis, so clients implementing Pub/Sub are able to use this feature without modifications.
So, if you need just pub/sub, there is no need of extra configuration regarding Keyspace Notifications
We have some Redis keys with a given TTL that we would like to subscribe to and take action upon once the TTL expires (a la job scheduler).
This works well in a single-host environment, and when you subscribe in ServiceStack, using its Redis client, to '__keyspace#0__:expired', that service will pick it and take action. That's fantastic...
... until you have a high-availability topology set up, with more than one API instance in that cluster. Then every single host appears to be picking up on that message and potentially doing things with it.
I know keyspace notifications don't work exactly the same as traditional pub/sub or messaging-layer events, but is there a way to perform some kind of acknowledgement on these kinds of events, so that, at the end of the day, only one host will carry on with the task?
Otherwise, is there a way to delay a message publishing?
Thanks!
As describe in https://redis.io/topics/notifications
very node of a Redis cluster generates events about its own subset of the keyspace as described above. However, unlike regular Pub/Sub communication in a cluster, events' notifications are not broadcasted to all nodes. Put differently, keyspace events are node-specific. This means that to receive all keyspace events of a cluster, clients need to subscribe to each of the nodes.
So client should create separate connection to each node to get redis keyspace notification.
My understanding of your question: You need an event based unicast notification whenever a key is expired.
This solution will be helpful to you if above assumption is correct. It's kind of crude solution but works!
Solution:
You need to put(may be using a service/thread) the expired keys in the Redis List/queue. Then blocking B*POP operation from the client instances on this list/queue will give you what you want!
How does it work?
Let's assume, a single background thread will continuously push the expired keys into a redis list/queue. The cluster of API instances will be calling blocking pop on this list/queue.
Since, blocking pop operation on each item of redis list will be consumed by only one client, only one API instance will the get the notification of expired key!!!
Ref:
List pop operation: https://redis.io/commands/lpop
Similar problem with pub/sub: Competing Consumer on Redis Pub/Sub supported?
I'm reading here, and I see a warning stating that PUB/SUB subscribers in Redis should not issue other commands:
A client subscribed to one or more channels should not issue commands,
although it can subscribe and unsubscribe to and from other channels.
I have two questions:
Why is this limitation?
For the scope of the paragraph, what's a client? A whole process? A Redis connection? A complete Redis instance? Or is it a bad idea in general to issue commands and subscribe to channels, and the admonition goes for every and any scope I can think of?
A client, in this case, is an instance of a connection to Redis. An application could well have multiple clients, each with different responsibilities or as a way to provide higher degrees of parallelism to the application.
What they are suggesting here, however, is that you use an individual client (think 'connection') to handle your incoming subscription messages and to react to those messages as its sole responsibility. The reason it's recommended not to make calls with this connection is because while it is waiting on incoming messages from subscribed channels, the client is in a blocked state.
Trying to make a call on a given client won't work while it's awaiting response from a blocking call.
I have a situation where the Maxtries in my MSMQ is 5. After 5 times nservicebus sends the message to the Error que that I have defined. Now I want to perfomr some further action when this happens (I have to update status of some processes to Error)
Is it possible to write a handler in my Saga class to read these error queues?
Thanks in Advance
Haris
If your are using 2.x you may want to consider writing a separate endpoint where the error queue is its input queue. The downside to this is that the messages will come off the queue. Assuming you still want to store them, you'll have to push them off to a database or some other type of storage.
You could also write a Saga that polls the error queue to check for messages and updates the appropriate status. After each time you check the queue, you would need to request another Timeout.
In 3.0, you have more control over the exceptions, and can implement your own way to handle the errors. If you implement the interface IManageMessageFailures, you can do your work there.
As an alternative to the solutions provided by Adam, you can subscribe to events raised by ServiceControl which are raisesd when a messages is sent to the errorqueue. See the official documentation about this here: http://docs.particular.net/servicecontrol/contracts
Another approach would be the notification API as described here: http://docs.particular.net/nservicebus/errors/subscribing-to-error-notifications. It allows you to subscribe to certain events (not event messages) like "MessageSentToErrorQueue" directly on the endpoint, so you wouldn't need to consume the error queue.
In a web application, if I need to write an event to a queue, I would make a connection to redis to write the event.
Now if I want another backend process (say a daemon or cron job) to process the or react the the publishing of the event in redis, do I need a persistant connection?
Little confused on how this pub/sub process works in a web application.
Basically in Redis there are two different messaging models:
Fire and Forget / One to Many: Pub/Sub. At the time a message is PUBLISH-ed all the subscribers will receive it, but this message is then lost forever. If a client was not subscribed there is no way it can get it back.
Persisting Queues / One to One: Lists, possibly used with blocking commands such as BLPOP. With lists you have a producer pushing into a list, and one or many consumers waiting for elements, but one message will reach only one of the waiting clients. With lists you have persistence, and messages will wait for a client to pop them instead of disappearing. So even if no one is listening there is a backlog (as big as your available memory, or you can limit the backlog using LTRIM).
I hope this is clear. I suggest you studying the following commands to understand more about Redis and messaging semantics:
LPUSH/RPUSH, RPOP/LPOP, BRPOP/BLPOP
PUBLISH, SUBSCRIBE, PSUBSCRIBE
Doc for this commands is available at redis.io
I'm not totally sure, but I believe that yes, pub/sub requires a persistent connection.
For an alternative I would take a peek at resque and how it handles that. Instead of using pub/sub it simply adds an item to a list in redis, and then whatever daemon or cron job you have can use the lpop command to get the first one.
Sorry for only giving a pseudo answer and then a plug.