I have an operation I need to get done N times and no more. The operation is done by many possibly parallel processes that recieve requests. A process has to check if the counter has exceeded N and, if not, then increment the counter and execute the operation. I suppose I could use a Redis counter for that.
Howerver if I just GET and then INCR a value I might run into a race condition that will result in the operation being done more than N times. How do I perform some kind of test-and-incr operation against Redis?
I know I could use WATCH but that's an optimistic lock. I expect there going to be very many collisions each second. This will result in a lot of failures. Maybe I could just wrap simple GET and INCR with some kind of external mutex. But I am not sure if it's good enough for performance.
You can use Redis INCR safely here.
INCR returns the value post the increment.
You check the value returned by INCR first ( see there is no need to do a GET )
& proceed to do the operation based on that value.
Only thing is you would have to set your INCR return value threshold as N+1 for limiting to N operations i.e. one extra redis operation than N.
For example, if we want to limit the operation to happen 3 times only, if INCR returns 4 after an increment, then
you stop doing further operation as it has already happened 3 times.
Related
I want to increment a sequence by some batchSize in PostgreSQL but I am not sure if this query is safe from concurrent calls made to the same sequence.
select setval('my_sequence', nextval('my_sequence') + batchSize);
What I am worried bout given the query above is that the call for setval() of the sequence would set the sequence back or perhaps abort the query since if another thread has fetched it nextval() concurrently.
For example:
Thread 1. nextval = 1001 batchSize = 100 sets sequence to 1101
Tread 2. nextval = 1002 batchSize = 20 sets sequence to 1022
This would mean that the sequence could ultimately return duplicate sequenceIds for for ids between 1002 and 1101.
The same could be achieved using the generate_series() function. The drawback would be that the series is not guaranteed to be in sequence since other threads can be calling nextval() concurrently to the same sequence which means that I have to fetch and parse the generated series.
From the PG Documentation
To avoid blocking concurrent transactions that obtain numbers from the same sequence, a nextval operation is never rolled back; that is, once a value has been fetched it is considered used and will not be returned again. This is true even if the surrounding transaction later aborts, or if the calling query ends up not using the value. For example an INSERT with an ON CONFLICT clause will compute the to-be-inserted tuple, including doing any required nextval calls, before detecting any conflict that would cause it to follow the ON CONFLICT rule instead. Such cases will leave unused “holes” in the sequence of assigned values. Thus, PostgreSQL sequence objects cannot be used to obtain “gapless” sequences.
Likewise, any sequence state changes made by setval are not undone if the transaction rolls back.
I would assume the above states that is thread safe.
According to the PostgreSQL documentation, it is:
This is done atomically: even if multiple sessions execute nextval concurrently, each will safely receive a distinct sequence value
Which means, it's totally safe to call NEXTVAL concurrently, every single invocation will get its own unique value, guarranted to not to repeat.
According to answer in PostgreSQL mailing lists SELECT setval(nextval() + N) isn't safe in concurrent scenario. Safe and still somewhat performant alternative would be:
SELECT nextval('my_seq')
FROM generate_series(1, N)
I have an UPDATE command that is failing due to a timeout on SQL Server. It's a large table used for storing events, and the primary key is a NON-clustered index (shouldn't really matter anyway as it's a uniqueidentifier column).
It's updating a single row: update [table] set [field] = 1 where [primary_key] = [primary_key_value];. The problem is likely to be that there are over 200,000,000 rows so it takes a long time to find each one.
In terms of SQL Server's preference, am I better off increasing the command timeout (yes, I know the default of 30 seconds is already 30 times longer than any reasonable query should take), or should I introduce maybe a shorter timeout, but retry over and over again (to a limit)?
So is one long 60 second command timeout better than 6 retries with 5 second timeouts?
In practice there'd be some sort of exponential backoff and circuit breaker implementation, of course.
This is a bit long for a comment.
Retrying is only going to help if the reason for the timeout is due to resource contention. In that case, you are probably better off with some sort of exponential backoff, rather than just running the queries immediately.
However, I would speculate that the reason is due to either:
An unoptimized query.
A large number of rows being updated.
If the number of rows is the issue, you can limit the update to, say, 100 rows and repeat the update until all are completed. However, I would first investigate the query to see if it can be can be optimized in other ways.
I want to delete multiple redis keys using a single delete command on redis client.
Is there any limit in the number of keys to be deleted?
i will be using del key1 key2 ....
There's no hard limit on the number of keys, but the query buffer limit does provide a bound. Connections are closed when the buffer hits 1 GB, so practically speaking this is somewhat difficult to hit.
Docs:
https://redis.io/topics/clients
However! You may want to take into consideration that Redis is single-threaded: a time-consuming command will block all other commands until completed. Depending on your use-case this may make a good case for "chunking" up your deletes into groups of, say, 1000 at a time, because it allows other commands to squeeze in between. (Whether or not this is tolerable is something you'll need to determine based on your specific scenario.)
I am trying to build FIFO queue in Redis, but I am just worried about concurrency. What if 2 clients try to do RPOP operation simultaneously?
If RPOP/LPOP is not atomic then how can I achieve atomicity using MULTI/EXEC ?
Is Redis LPOP / RPOP operation atomic?
Yes, both LPOP and RPOP are atomic.
What if 2 clients try to do RPOP operation simultaneously?
If the size of the LIST is equal to or greater than 2, both clients get a different item. If the LIST has only one item, only one client gets the item, and the other client gets null reply. If the LIST is empty, both clients get null reply.
Another Solution
You can also use BLPOP or BRPOP to implement the FIFO. These two commands are also atomic and will block for empty LIST. See the doc for details.
there are two systems sharing a redis database, one system just read the redis, the other update it.
the read system is so busy that the redis can't handle it, to reduce the count of requests to redis, I find "mget", but I also find "multi".
I'm sure mget will reduce the number of requests, but will "multi" do the same? I think "multi" will force the redis server to keep some info about this transaction and collect requests in this transaction from the client one by one, so the total number of requests sent stays the same, but the results returned will be combined together, right?
So If I just read KeyA, keyB, keyC in "multi" when the other write system changed KeyB's value, what will happen?
Short Answer: You should use MGET
MULTI is used for transaction, and it won't reduces the number of requests. Also, the MULTI command MIGHT be deprecated in the future, since there's a better choice: lua scripting.
So If I just read KeyA, keyB, keyC in "multi" when the other write system changed KeyB's value, what will happen?
Since MULTI (with EXEC) command ensures transaction, all of the three GET commands (read operations) executes atomically. If the update happens before the read operation, you'll get the old value. Otherwise, you'll get the new value.
By the way, there's another option to reduce RTT: PIPELINE. However, in your case, MGET should be the best option.