Redis hooking (publish-subscribe) under stress tests - performance under load - redis

Question

Based on the suggested solusion and following the example, I'm trying to delete a key right after I get a notification that another key has expired.
The problem is that under stress test with heavy load of seting 600K new keys and setting half of them with expiration time of 2 seconds, I get the following exception:
Exception in thread "main" redis.clients.jedis.exceptions.JedisConnectionException: Unknown reply: t
The question is what will be the best practice to write such listener? (thread pool? if so in what context to implement it?)
Jedis version: 2.7.2
Redis version: 2.8.19
My code so far:
Subscriber class:
public class Subscriber {
public static void main(String[] args) {
JedisPool pool = new JedisPool(new JedisPoolConfig(), "localhost");
Jedis jedis = pool.getResource();
jedis.psubscribe(new KeyExpiredListener(), "__key*__:*");
}
}
Listener class:
public class KeyExpiredListener extends JedisPubSub {
private String generalKeyTimeoutPrefix = "TO_";
#Override
public void onPMessage(String pattern, String channel, String message) {
String originalKey = null;
try {
if(channel.endsWith("__keyevent#0__:expired") && message.startsWith(generalKeyTimeoutPrefix)) {
originalKey = message.substring(generalKeyTimeoutPrefix.length());
del(originalKey);
}
} catch(Exception e) {
logger.error("..", e);
}
}
private void del(String key) {
Jedis jedis = new Jedis("localhost");
jedis.connect();
try {
jedis.del(key);
} catch (Exception e) {
logger.error("...");
} finally {
jedis.disconnect();
}
}
}
Key generator:
public class TestJedis {
public static void main(String[] args) throws InterruptedException {
JedisPool pool = new JedisPool(new JedisPoolConfig(), "localhost");
Jedis jedis = pool.getResource();
String prefixForlKeys = "token_";
String prefixForTimeoutKeys = "TO_";
for (int i = 0; i < 300000; i++) {
String key = prefixForlKeys + i;
String timeoutKey = prefixForTimeoutKeys + key;
jedis.set(key, "some_data");
jedis.set(timeoutKey, "");
jedis.expire(timeoutKey, 2);
}
System.out.println("Finished to create the keys");
}
}

Answer 1

the problem is with your implementation of del() method: it does not use connection pooling, does not reuse connections, so it finally occupies all available local ports. Try to use something like this:
private void del(String key) {
Jedis jedis = pool.getResource();
jedis.connect();
try {
jedis.del(key);
} catch (Exception e) {
e.printStackTrace( );
} finally {
jedis.close();
}
}
instead of opening/closing a connection for each expired key.