I have a requirement where i am pushing my keys to redis with some expiration time. Also have a subscriber for listening key expiration events and then have a callback to my other system which can perform some business rules on it. Is it a good design to have faith in redis pub-sub for this usecase?
Average TTL for keys will be in range ~15 minutes.
Using other design will make me having a scheduler/cron(every minute) or some polling system.
Yes, I have been using Redis pub/sub for exactly the same use case in production and has been working quite well for me without any issues.
Related
I'm looking to build a subscription system using redis. I was thinking about to put a key with the subscription ending date as an expire time and then using redis keyspace notifications to proceed to remove/renew subscription based on that but I was reading about Pub/Sub reliability and found out it hasn't. So I don't know if it's the best choice for a subscription system.
Is there a better approach to accomplish this using redis?
Redis has specific commands to implement a subscription system:
https://redis.io/topics/pubsub when you don't need persistence
https://redis.io/topics/streams-intro when you need persistence (since 5.0)
we are developing data pipeline app using Kafka, storm and redis. Realtime events from different systems will be published to Kafka and storm do the event processing based on rules configured. State is managed in redis.
we have a requirement to implement different WAIT_TIME before processing for different events. we are looking at following options.
we initially looked at storm windowing [sliding or tumbling window] but provides option only to configure fixed intervals. we need varying wait_time based on rules
we are exploring other options of storing the events in a redis cache for varying duration [TTL] and once each events are evicted we need to have a callback back to storm to process it.
Do redis support callback on eviction ? Is there a better way to do this with storm and redis ?
we resolved the problem by calculating the expiry time for each streaming events & storing the events in redis against expiry time [expiry as the key] , on top storm scheduler will query the events which qualifies for eviction and process it.
Currently I'm working on a distributed test execution and reporting system. I'm planning to use Redis PUB/SUB as a message queue and message distribution system.
I'm new to Redis, so I'm trying to read as many docs as I can and play around with it. One of the most important topics is high availability. As I said, I'm not an expert, but I'm aware of the possible options - using Sentinel, replication, clustering, etc.
What's not clear for me is how the Pub/Sub feature and the HA options are related each other. What's the best practice to build a reliable messaging system with Redis? By reliable I mean if my Redis message broker is down there should be some kind of a backup node (a slave?) that should be able to take over this role.
Is there a purely server-side solution? Or do I need to create a smart wrapper around the Redis client to handle this? Will a Sentinel-driven setup help me?
Doing pub sub in Redis with failover means thinking about additional factors in the client side. A key piece to understand is that subscriptions are per-connection. If you are subscribed to a channel on a node and it fails, you will need to handle reconnect and resubscribe. Because subscriptions are done at the connection level it is not something which can be replicated.
Regarding the details as to how it works and what you can expect to see, along with ways around it see a post I made earlier this year at https://objectrocket.com/blog/how-to/reliable-pubsub-and-blocking-commands-during-redis-failovers
You can lower the risk surface by subscribing to slaves and publishing to the master, but you would then need to have non-promotable slaves to subscribe to and still need to handle losing a slave - there is just as much chance to lose a given slave as there is a master.
IMO, PUB/SUB is not a good choice, may be disque (comes from antirez, author of the Redis) fits better:
Disque, an in-memory, distributed job queue
I'm using Redis as a session store in my app. Can I use the same instance (and db) of Redis for my job queue? If it's of any significance, it's hosted with redistogo.
It is perfectly fine to use the same redis for multiple operations.
We had a similar use case where we used Redis as a key value store as well as a job queue.
However you may want to consider other aspects like the performance requirements for your application. Redis can ideally handle around 70k operations per second and if at some time in future you think you may hit these benchmarks it's much better to split your operations to multiple redis instances based on the kind of operations you perform. This will allow you to make decisions about availability and replication at a more finer level depending on the requirements. As a simple use case once your key size grows you may be able to flush your session app redis or shard your keys using redis cluster without affecting job queing infrastructure.
-User can prepare Post to be published for future.
So
Post.PostState is PostState.Scheduled.
Post.PublishDate is FutureDate
When futuredate comes PostState will be PostState.Published.
How can I implement this in Redis.
Sorry for duplication: I found that Delayed execution / scheduling with Redis?
Delayed execution / scheduling with Redis?
It seems an answer will be more related with code than db, so
c# reliable delayed/scheduled execution best practice
There is no scheduling as such, but you could set the values for both keys and put an expire on the scheduled date. Always lookup both keys and prefer the first. When the schedule expires, you will get back the actual as the first (and only) result.
You could also hide all that behind a lua script.
Sorry but using REDIS key expiration for scheduling won't work.
expiration can happen before, or very far in the future (eg. depending on available memory).
I think you might want to use another tool for delayed execution depending on you development platform. (eg. polling a REDIS queue, linux cron, timers, etc.)