We are trying to implement a pattern where we update the Redis in 2 cases
1. from the db every 5-10 minutes.
2. on specific use cases we update the current Redis data according to the use case (from time to time may need to ask the DB for the data again)
Question: What is the way to implement optimistic locking in ServiceStack's Redis?
I'm new to Redis but have recently been reading a lot about issues like this.
I think you want WATCH with MULTI/EXEC.
The redis docs have the reference for these commands, but they're pretty minimal:
http://redis.io/commands/watch
http://redis.io/commands/multi
http://redis.io/commands/exec
The "transactions" page ties them together a lot better:
http://redis.io/topics/transactions
I also recommend the "little Redis book":
http://openmymind.net/2012/1/23/The-Little-Redis-Book/
Related
I have three microservice that needs to communicate between each other. Microservice-1 is incharge of the data and the database(he writes and read to it). I will add a redis cache store for Microservice-1 to cache data there. I want to put a redis-slave for the other 2 microservices to reduce communication with the actual microservice, if the data is already in the cache store. Since all updates to the data, has to go thru the Microservice-1 and he will always update the cache, redis replication will make sure the other two microservices will get it too. Ofcourse, if the data is not in cache, it will call the Microservice-1 for the data, which will update the cache.
Am i missing something, with this approach ?
This will definitely work in the "sunny day" case.
But sometimes there are storms, and in storms there's a chance of losing cache coherency (i.e. the DB and Redis disagree on the data).
For example, lets say that you have Microservice-1 update the DB and then update Redis. What happens if there's a crash between updating the DB and updating Redis?
On the other hand, what if you reverse the ordering (update Redis and then the DB)? Now Redis could be updated and not the DB.
Neither of these in insurmountable, but absent a means of having a transaction which ensures that 0 or 2 of Redis and the DB are updated, there will always be a time window where the change is in one but not the other. In that situation, it's probably worth embracing eventual consistency (e.g. periodically scan the DB and update redis with recently updated records).
As an elaboration on that, a Command Query Responsibility Segregation with Event Sourcing (CQRS/ES) approach may prove useful: Microservice-1 gets split into two services, one which takes commands (requests to update) and another which handles queries. Instead of updating a row in a DB, the command service now appends (in a typical DB, an INSERT) an event which describes what changed. The query service can subscribe to those events and update Redis. Other microservices can also subscribe to the stream of events and update their own views (which can be remixed in any way they want) of Microservice-1's state.
Redisson provides support for locking backed by Redis. It also provides implementation for working with spring cache framework. But based on what I saw locking is not called by default when try to update a key in a cache using spring cache framework. Redisson has separate APIs for locking a particular key. Is that correct?
Also the locking APIs seem to take key as an input so I am not clear how locking works. For locking I am assuming you need both cache name and key.
I am new to redis so any help in throwing some light on this is really appreciated. Thanks
Firstly, locking in Redisson is implement by Redis, but not only used for Redis updating.
For example if you want to implement an atomic operation like this:
Get key value from Redis
Calculate a new value based on some logic
Save the new value to Redis and Mysql
You can use Redisson lock to make the operation atomically.
Secondly, in Redis, set/update command is atomic and you don't need to lock the key if you only update the value.
And for locking API, Redisson implement lock by Redis key/value, so you only need to provide lock-key, which generally contains a resource id and resource type(like "lock:user:31352")
I've done some projects with Redis and MongoDB but I'm not comfortable at all. I'm currently using MongoDB for storing player datas and Redis for temporary and sorted datas. I'd want to use Redis more to my projects.
My questions
Should I use Redis more for persistent datas? I'd like to know a question about this case; if I make a project that ban players from the game server, is Redis good option to use for this case?
What are the best use cases for Redis?
As I mention it above, I use MongoDB for storing player datas and map for cache their information when they're online. From what I know redis is one of the best NoSQL database for caching. Should I use Redis for caching player datas?
If you have any other idea about the topic, I'd like to know that with details.
Should I use Redis more for persistent datas?
Redis is way more than Cache and is acting as Main database in many enterprises, and also supports few methods persistency like RDB and AOF.
if I make a project that ban players from the game server, is Redis good option to use for this case?
Redis support a nice set of plugins (Modules), one of them is RedisBloom, especially suited for quick filtering.
Two issues
Do lua scripts really solve all cases for redis transactions?
What are best practices for asynchronous transactions from one client?
Let me explain, first issue
Redis transactions are limited, with an inability to unwatch specific keys, and all keys being unwatched upon exec; we are limited to a single ongoing transaction on a given client.
I've seen threads where many redis users claim that lua scripts are all they need. Even the redis official docs state they may remove transactions in favour of lua scripts. However, there are cases where this is insufficient, such as the most standard case: using redis as a cache.
Let's say we want to cache some data from a persistent data store, in redis. Here's a quick process:
Check cache -> miss
Load data from database
Store in redis
However, what if, between step 2 (loading data), and step 3 (storing in redis) the data is updated by another client?
The data stored in redis would be stale. So... we use a redis transaction right? We watch the key before loading from db, and if the key is updated somewhere else before storage, storage would fail. Great! However, within an atomic lua script, we cannot load data from an external database, so lua cannot be used here. Hopefully I'm simply missing something, or there is something wrong with our process.
Moving on to the 2nd issue (asynchronous transactions)
Let's say we have a socket.io cluster which processes various messages, and requests for a game, for high speed communication between server and client. This cluster is written in node.js with appropriate use of promises and asynchronous concepts.
Say two requests hit a server in our cluster, which require data to be loaded and cached in redis. Using our transaction from above, multiple keys could be watched, and multiple multi->exec transactions would run in overlapping order on one redis connection. Once the first exec is run, all watched keys will be unwatched, even if the other transaction is still running. This may allow the second transaction to succeed when it should have failed.
These overlaps could happen in totally separate requests happening on the same server, or even sometimes in the same request if multiple data types need to load at the same time.
What is best practice here? Do we need to create a separate redis connection for every individual transaction? Seems like we would lose a lot of speed, and we would see many connections created just from one server if this is case.
As an alternative we could use redlock / mutex locking instead of redis transactions, but this is slow by comparison.
Any help appreciated!
I have received the following, after my query was escalated to redis engineers:
Hi Jeremy,
Your method using multiple backend connections would be the expected way to handle the problem. We do not see anything wrong with multiple backend connections, each using an optimistic Redis transaction (WATCH/MULTI/EXEC) - there is no chance that the “second transaction will succeed where it should have failed”.
Using LUA is not a good fit for this problem.
Best Regards,
The Redis Labs Team
I'm using Redis as a session store in my app. Can I use the same instance (and db) of Redis for my job queue? If it's of any significance, it's hosted with redistogo.
It is perfectly fine to use the same redis for multiple operations.
We had a similar use case where we used Redis as a key value store as well as a job queue.
However you may want to consider other aspects like the performance requirements for your application. Redis can ideally handle around 70k operations per second and if at some time in future you think you may hit these benchmarks it's much better to split your operations to multiple redis instances based on the kind of operations you perform. This will allow you to make decisions about availability and replication at a more finer level depending on the requirements. As a simple use case once your key size grows you may be able to flush your session app redis or shard your keys using redis cluster without affecting job queing infrastructure.