How to remove old data in redis when updating data? - redis

I'm setting data in redis, but I update that data every minute, and I need redis to replace the old data with the new updated one. I am very confused as to how I might be able to achieve that. I tried to use client.expire('key', 60), but that doesn't seem to work. Below is my code. Any help would be appreciated. Thanks in advance.
var redis = require("redis"),
client = redis.createClient();
const { promisify } = require("util");
const setAsync = promisify(client.set).bind(client);
async function run(){
//do something
const success = await setAsync('mykey', list);
console.log({success});
client.expire('mykey', 59);
}

It seems like the key will be deleted (or more precisely, marked for deletion) 59 seconds after calling client.expire, but there are 2 things you need to be aware of:
If you call the EXPIRE command on a key that already have a TTL, it'll override the pervious TTL.
The expiration timeout is cleared once the value of the key is overwritten (using SET, DEL, etc.).
BTW, in case you're updating the exact same key, the data will be overwritten anyway.. for example:
await setAsync('key', 'value1'); // set 'key' to 'value1'
await setAsync('key', 'value2'); // override 'key' to 'value2'
await getAsync('key'); // 'value2'
Also, instead of running two commands, one for setting the value and one for setting the TTL, you can use the EX option like that:
setAsync('key', 'value', 'EX', 60); // set `key` to `value` with TTL of 60 seconds

Related

Reading a large number of keys in high Redis traffic

I have used Redis to store the statistics of a website with relatively high traffic. I get a Redis variable for each statistical item and increment it with each call.
Now, I need to read this information every minute and update it in SQL and reset the keys to zero. Since the number of keys and traffic is high, what is the optimal way to read the keys?
The easiest method is to scan the Redis database every minute. But since scanning is costly and it will include new keys that are alive for less than a minute, this method does not seem suitable.
In another method, I first enabled the keyspace events to receive a notification when the keys expires.
CONFIG SET notify-keyspace-events Ex
Then, for each key, I created an empty key with a TTL of one minute, and at the time of receiving its expiration notification, I read the associated key and its value and put it in the output queue. This method also has disadvantages, such as the possibility of losing notifications when reconnecting.
public void SubscribeOnStatKeyExpire()
{
sub.Subscribe("__keyevent#0__:expired").OnMessage(async channelMessage => {
await ExportExpiredKeys((string)channelMessage.Message);
});
}
private async Task ExportExpiredKeys(string expiredKey)
{
if(expiredKey.StartsWith(_notifyKeyPrefix))
{
var key = expiredKey.Remove(0, _notifyKeyPrefix.Length);
var value = await _cache.StringGetDeleteAsync(key);
if (!value.HasValue || value.IsNullOrEmpty) return;
var count = (int)value;
if(count <= _negligibleValue)
{
await _cache.StringIncrementAsync(key, count);
return;
}
var listLength = await Enqueue(_exportQueue, key + "." + value);
if (listLength > _numberOfKeysInOneMessage) // _numberOfKeysInOneMessage for now is 500
{
var list = await Dequeue(_exportQueue,_numberOfKeysInOneMessage);
await SendToSqlServiceBrocker(list);
}
}
}
Is there a better way? For example, using a stream, sorted set or etc.

Redis StackExchange LuaScripts with parameters

I'm trying to use the following Lua script using C# StackExchange library:
private const string LuaScriptToExecute = #"
local current
current = redis.call(""incr"", KEYS[1])
if current == 1 then
redis.call(""expire"", KEYS[1], KEYS[2])
return 1
else
return current
end
Whenever i'm evaluating the script "as a string", it works properly:
var incrementValue = await Database.ScriptEvaluateAsync(LuaScriptToExecute,
new RedisKey[] { key, ttlInSeconds });
If I understand correctly, each time I invoke the ScriptEvaluateAsync method, the script is transmitted to the redis server which is not very effective.
To overcome this, I tried using the "prepared script" approach, by running:
_setCounterWithExpiryScript = LuaScript.Prepare(LuaScriptToExecute);
...
...
var incrementValue = await Database.ScriptEvaluateAsync(_setCounterWithExpiryScript,
new[] { key, ttlInSeconds });
Whenever I try to use this approach, I receive the following error:
ERR Error running script (call to f_7c891a96328dfc3aca83aa6fb9340674b54c4442): #user_script:3: #user_script: 3: Lua redis() command arguments must be strings or integers
What am I doing wrong?
What is the right approach in using "prepared" LuaScripts that receive dynamic parameters?
If I look in the documentation: no idea.
If I look in the unit test on github it looks really easy.
(by the way, is your ttlInSeconds really RedisKey and not RedisValue? You are accessing it thru KEYS[2] - shouldnt that be ARGV[1]? Anyway...)
It looks like you should rewrite your script to use named parameters and not arguments:
private const string LuaScriptToExecute = #"
local current
current = redis.call(""incr"", #myKey)
if current == 1 then
redis.call(""expire"", #myKey, #ttl)
return 1
else
return current
end";
// We should load scripts to whole redis cluster. Even when we dont have any.
// In that case, there will be only one EndPoint, one iteration etc...
_myScripts = _redisMultiplexer.GetEndPoints()
.Select(endpoint => _redisMultiplexer.GetServer(endpoint))
.Where(server => server != null)
.Select(server => lua.Load(server))
.ToArray();
Then just execute it with anonymous class as parameter:
for(var setCounterWithExpiryScript in _myScripts)
{
var incrementValue = await Database.ScriptEvaluateAsync(
setCounterWithExpiryScript,
new {
myKey: (RedisKey)key, // or new RedisKey(key) or idk
ttl: (RedisKey)ttlInSeconds
}
)// .ConfigureAwait(false); // ? ;-)
// when ttlInSeconds is value and not key, just dont cast it to RedisKey
/*
var incrementValue = await
Database.ScriptEvaluateAsync(
setCounterWithExpiryScript,
new {
myKey: (RedisKey)key,
ttl: ttlInSeconds
}
).ConfigureAwait(false);*/
}
Warning:
Please note that Redis is in full-stop mode when executing scripts. Your script looks super-easy (you sometimes save one trip to redis (when current != 1) so i have a feeling that this script will be counter productive in greater-then-trivial scale. Just do one or two calls from c# and dont bother with this script.
First of all, Jan's comment above is correct.
The script line that updated the key's TTL should be redis.call(""expire"", KEYS[1], ARGV[1]).
Regarding the issue itself, after searching for similar issues in RedisStackExchange's Github, I found that Lua scripts do not work really well in cluster mode.
Fortunately, it seems that "loading the scripts" isn't really necessary.
The ScriptEvaluateAsync method works properly in cluster mode and is sufficient (caching-wise).
More details can be found in the following Github issue.
So at the end, using ScriptEvaluateAsync without "preparing the script" did the job.
As a side note about Jan's comment above that this script isn't needed and can be replaced with two C# calls, it is actually quite important since this operation should be atomic as it is a "Rate limiter" pattern.

Page stored using Cloudflare Cache API expires earlier than expected

I am developing a backend API using Cloudflare Workers to cache the tokens into respective individual pages, something like http://cache.api/[tokenId] with token value itself as body content, using Cache API.
const tokenId = 'fakeJWT';
const internalUrl = ''.concat(
'http://cache.api/',
tokenId // the query string
);
const cacheUrl = new URL(internalUrl);
const cacheKey = new Request(cacheUrl.toString());
const cache = caches.default;
let response = new Response(tokenId);
response.headers.append('Cache-Control', 's-maxage=86400'); // 24 hours
await cache.put(cacheKey, response.clone());
I've configured Cache-Control header with 24 hours expiry. Then, I wrote another API in the same Cloudflare Workers to check the existence of the cache, it exists after 10 minutes but does not exist after 15 minutes.
const internalUrl = ''.concat(
'http://cache.api/',
tokenId // the query string
);
const cacheUrl = new URL(internalUrl);
const cacheKey = new Request(cacheUrl.toString());
const cache = caches.default;
let response = await cache.match(cacheKey);
if (!response) {
console.log(
`Cache url for ${cacheUrl} is not present`
);
return unauthorised(`${pathName}: This user session has expired! cache url is ${cacheUrl}`);
}
else
{
let response = new Response(tokenId);
response.headers.append('Cache-Control', 's-maxage=86400'); // in seconds
await cache.put(cacheKey, response.clone());
}
Did I miss anything?
What may be happening is a cache miss rather than an expiration. When using Cloudflare for caching, you shouldn't expect caching to be guaranteed, in particular using the cache API the docs mention it being data-center specific (not propagated globally) -
https://developers.cloudflare.com/workers/runtime-apis/cache/#background
Caching should be mechanism for improving performance, but not relied upon for guaranteed storage in other words. If your use-case requires this, it would be better to use the Workers Key Value -
https://developers.cloudflare.com/workers/learning/how-kv-works/

How does node-redis method setex() convert Buffer to string?

I'm using node-redis and I was hoping that someone could help me to figure out how this library converts Buffer to string. I gzip my data before I store it in redis with node-gzip and this call returns Promise<Buffer>
const data = JSON.stringify({ data: 'test' });
const compressed = await gzip(data, { level: 9 });
I tested following 2 approaches of saving buffer data into redis
Without .toString() - I pass the Buffer to the library and it will take care of the conversion
const result = await redisClient.setex('testKey', 3600, compressed);
and with .toString()
const result = await redisClient.setex('testKey', 3600, compressed.toString());
When I try these 2 approaches I don't get the same value saved in redis. I tried to use different params for .toString() to match the output of 1) but it didn't work
Reason why I need the saved value in 1) format is that I'm matching value format that what one of php pages generates
My code is working fine without .toString() but I would like to know how node-redis handles it internally
I've tried to find the answer in the source code and to debug and step into library calls but I didn't find the answer that I was looking for and I hope that someone can help me with this
It looks like happens in utils.js file:
utils.js
if (reply instanceof Buffer) {
return reply.toString();
}
Also use the proper options (i.e. return_buffers):
node-redis README
redis.createClient({ return_buffers: true });

Booksleeve setting expiration on multiple key/values

Unless I am missing something, I don't see a Multiple Set/Add overload that allows you to set multiple keys with an expiration.
var conn = new RedisConnection("server");
Dictionary<string,string> keyvals;
conn.Strings.Set(0,keyvals,expiration);
or even doing it with multiple operations
conn.Strings.Set(0,keyvals);
conn.Expire(keyvals.Keys,expiration);
No such redis operation exists - expire is not varadic. However, since the api is pipelined, just call the method multiple times. If you want to ensure absolute best performance, you can suspend eager socket flushing while you do this:
conn.SuspendFlush();
try {
foreach(...)
conn.Keys.Expire(...);
} finally {
conn.ResumeFlush();
}
Here is my approach:
var expireTime = ...
var batchOp = redisCache.CreateBatch();
foreach (...) {
batchOp.StringSetAsync(key, value, expireTime);
}
batchOp.Execute();