I have a rather large database (5 dbs of about a million keys each), and each key has the environment namespace in it. For example: "datamine::production::crosswalk==foobar"
I need to sync my development environment with this data copied from the production RDB snapshot.
So what I'm trying to do is batch rename every key, changing the namespace from datamine::production to datamine::development. Is there a good, way to achieve this?
What I've tried so far
redis-cli command of keys "datamine::production*", piped into sed, then back to redis-cli. This takes forever, and bombs for some reason on many keys (combining several in the same line, sporadically). I'd prefer a better option.
Perl search/replace on the .rdb file. My local redis-server flat refuses to load the modified RDB.
The solution:
Ok, here's the script I wrote to solve this problem. It requires the "Redis" gem. Hopefully someone else finds this useful...
#!/usr/bin/env ruby
# A script to translate the current redis database into a namespace for another environment
# GWI's Redis keys are namespaced as "datamine::production", "datamine::development", etc.
# This script connects to redis and translates these key names in-place.
#
# This script does not use Rails, but needs the "redis" gem available
require 'Benchmark'
require 'Redis'
FROM_NAMESPACE = "production"
TO_NAMESPACE = "development"
NAMESPACE_PREFIX = "datamine::"
REDIS_SERVER = "localhost"
REDIS_PORT = "6379"
REDIS_DBS = [0,1,2,3,4,5]
redis = Redis.new(host: REDIS_SERVER, port: REDIS_PORT, timeout: 30)
REDIS_DBS.each do |redis_db|
redis.select(redis_db)
puts "Translating db ##{redis_db}..."
seconds = Benchmark.realtime do
dbsize = redis.dbsize.to_f
inc_threshold = (dbsize/100.0).round
i = 0
old_keys = redis.keys("#{NAMESPACE_PREFIX}#{FROM_NAMESPACE}*")
old_keys.each do |old_key|
new_key = old_key.gsub(FROM_NAMESPACE, TO_NAMESPACE)
redis.rename(old_key, new_key)
print "#{((i/dbsize)*100.0).round}% complete\r" if (i % inc_threshold == 0) # on whole # % only
i += 1
end
end
puts "\nDone. It took #{seconds} seconds"
end
I have a working solution:
EVAL "local old_prefix_len = string.len(ARGV[1])
local keys = redis.call('keys', ARGV[1] .. '*')
for i = 1, #keys do
local old_key = keys[i]
local new_key = ARGV[2] .. string.sub(old_key, old_prefix_len + 1)
redis.call('rename', old_key, new_key)
end" 0 "datamine::production::" "datamine::development::"
Two last parameters are respectively an old prefix and a new prefix.
Related
Can I duplicate a key using the redis-cli connected, is there any command predefined in redis or not?
Duplicate FSS_SYSAGENT to FSS_SYSAGENTDuplicate.
10.44.112.213:6403> hgetall FSS_SYSAGENT
1) "SYSTEM_01"
2) "{\"port\":\"4407\",\"ipAddress\":\"10.44.112.213\",\"symbolicName\":\"SYSTEM_01\",\"eventLogEnabled\":\"1110\",\"status\":1,\"wcPort\":\"6029\",\"activeSystem\":\"N\",\"createdBy\":\"\",\"createdDate\":\"2018-11-20 13:11:16\",\"modifiedBy\":\"\",\"modifiedDate\":\"\",\"institution\":\"FSS\",\"delFlag\":0,\"accessID\":0,\"rowCount\":0,\"endCount\":0}"
You can use the DUMP and RESTORE commands to duplicate the key:
use the DUMP command to serialize the value of a key.
use the RESTORE command to restore the serialized value to another key.
You can wrap these two steps into a Lua script:
-- duplicate.lua
local src = KEYS[1]
local dest = KEYS[2]
local val = redis.call('DUMP', src)
if val == false then
return 0
else
-- with RESTORE command, you can also set TTL for the new key, and use the [REPLACE] option to set the new key forcefully.
redis.call('RESTORE', dest, 0, val)
return 1
end
Run the Lua script with redis-cli: ./redis-cli --eval duplicate.lua FSS_SYSAGENT FSS_SYSAGENTDuplicate ,
UPDATE
Since Redis 6.2.0, you can use the COPY command to do the job.
I want to do the following in redis LUA scripting:
SPOP 1+ items from "source" set
SADD elements from #1 into "target" set
I'm using redis 5.
I have the below lua, but this is just for a single element:
local source = KEYS[1]
local target = KEYS[2]
local num = KEYS[3]
local ele = redis.call("SPOP", "source")
redis.call("SADD", target, ele)
return "OK"
How can I update the above with:
handle 1+ elements using the passed in param KEY[3]
ensure if 0 elements were returned from POP, it doesn't try and add to target set.
In Redis v5 and above this should "just work" due to the move to script effects replication as a default.
In v4 you'll have to execute redis.replicate_commands() before any random command in the script.
EDIT: per your edits and comment, here's an example:
-- uncomment the next line for Redis v4
-- redis.replicate_commands()
local source = KEYS[1]
local target = KEYS[2]
local num = ARGV[1]
local elems = redis.call("SPOP", source, num)
if #elems > 0 then
redis.call("SADD", target, unpack(elems))
end
return redis.status_reply("OK")
My redis collection contains many keys
I want to be able to flush them all except all the keys that start with:
"configurations::"
is this possible?
You can do this
redis-cli KEYS "*" | grep -v "configurations::" | xargs redis-cli DEL
List all keys into the redis, remove from the list keys that contains "configurations::" and delete them from the redis
Edit
As #Sergio Tulentsev notice it keys is not for use in production. I used this python script to remove keys on prodution redis. I stoped replication from master to slave before call the script.
#!/usr/bin/env python
import redis
import time
pattern = "yourpattern*"
poolSlave = redis.ConnectionPool(host='yourslavehost', port=6379, db=0)
redisSlave = redis.Redis(connection_pool=poolSlave)
poolMaster = redis.ConnectionPool(host='yourmasterhost', port=6379, db=0)
redisMaster = redis.Redis(connection_pool=poolMaster)
cursor = '0'
while cursor != 0:
cursor, data = redisSlave.scan(cursor, pattern, 1000)
print "cursor: "+str(cursor)
for key in data:
redisMaster.delete(key)
print "delete key: "+key
# reduce call per second on production server
time.sleep(1)
The SCAN & DEL approach (as proposed by #khanou) is the best ad-hoc solution. Alternatively, you could keep an index of all your configurations:: key names with a Redis Set (simply SADD the key's name to it whenever you create a new configurations:: key). Once you have this set you can SSCAN it to get all the relevant key names more efficiently (don't forget to SREM from it whenever you DEL though).
Yes, it's possible. Enumerate all the keys, evaluate each one and delete if it fits the criteria for deletion.
There is no built-in redis command for this, if this is what you were asking.
It might be possible to cook up a Lua script that will do this (and it'll look to your app that it's a single command), but still it's the same approach under the hood.
I have an admin server, NodeManager, and 1 managed server, all on the same machine.
I am trying to enter something similar to this to the arguments field in the Server Start tab:
-Dmy.property=%USERPROFILE%\someDir\someJar.jar
But when the managed server is started it throws this exception:
Error opening zip file or JAR manifest missing : %USERPROFILE%\someDir\someJar.jar
It appears that the environment variable is not being translated into it's value. It is just passed on to the managed server as plain-text.
I tried surrounding the path with double quotes (") but the console validates the input and does not allow this: "Arguments may not contain '"'"
Even editing the config.xml file manually cannot work, as the admin server fails to startup after this:
<Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: [Management:141266]Parsing failure in config.xml: java.lang
.IllegalArgumentException: Arguments may not contain '"'.>
I also tried using %20 to no avail, it is just passed as %20.
I thought that perhaps this had something to do with the spaces in the value of %USERPROFILE% (which is "C:\documents and settings.."), but the same thing happens with other env. variables which point to other directories with no spaces.
My question:
Is there any supported way of :
using double quotes? what if i have to reference a folder with spaces in it's name?
reference an environment variable? What if i have to rely on it's value for distributed servers where i do not know in advance the variable's value?
Edit based on comments:
Approach 1:
Open setDomainEnv.cmd and search for export SERVER_NAME in Linux or for set SERVER_NAME in Windows. Skip to next to next line (i.e skip current and the next line)
On the current line, insert:
customServerList="server1,server2" #this serverList should be taken as input
isCurrServerCustom=$(echo ${customServerList} | tr ',' '\n' | grep ${SERVER_NAME} | wc -l)
if [ $isCurrServerCustom -gt 0 ]; then
# add customJavaArg
JAVA_OPTIONS="-Dmy.property=${USERPROFILE}/someDir/someJar.jar"
fi
Save the setDomainEnv.sh file and re-start servers
Note that I have only given logic for Linux , for Windows similar logic can be used but with batch scripting syntax.
Approach 2:
Assuming domain is already installed and user provides the list of servers to which the JVM argument -Dmy.property need to be added. Jython script (use wlst.sh to execute). WLST Reference.
Usage: wlst.sh script_name props_file_location
import os
from java.io import File
from java.io import FileInputStream
# extract properties from properties file.
print 'Loading input properties...'
propsFile = sys.argv[1]
propInputStream = FileInputStream(propsFile)
configProps = Properties()
configProps.load(propInputStream)
domainDir = configProps.get("domainDir")
# serverList in properties file should be comma seperated
serverList = configProps.get("serverList")
# The current machine's logical name as mentioned while creating the domain has to be given. Basically the machine name on which NM for current host is configured on.
# This param may not be required as an input if the machine name is configured as same as the hostname , in which case , socket module can be imported and socket.getHostName can be used.
currMachineName = configProps.get("machineName")
jarDir = os.environ("USERPROFILE")
argToAdd = '-Dmy.property=' + jarDir + File.separator + 'someDir' + File.separator + 'someJar.jar'
readDomain(domainDir)
for srvr in serverList.split(",") :
cd('/Server/' + srvr)
listenAddr = get('ListenAddress')
if listenAddr != currMachineName :
# Only change current host's servers
continue
cd('/Server/' + srvr + '/ServerStart/' + srvr)
argsOld = get('Arguments')
if argsOld is not None :
set('Arguments', argsOld + ' ' + argToAdd)
else:
set('Arguments', argToAdd)
updateDomain()
closeDomain()
# now restart all affected servers (i.e serverList)
# one way is to connect to adminserver and shutdown them and then start again
Script has to be run from all hosts where the managed servers are going to be deployed in order to have the host specific value of "USERPROFILE" in the JVM argument.
BTW, to answer your question in a line : looks like the JVM arguments have to be supplied with the literal text eventually. But looks like WLS doesn't translate the environment variables if provided as JVM arguments. It gives an impression that it is translating when its done from startWebLogic.cmd (ex: using %DOMAIN_HOME% etc.) but its the shell/cmd executor that translates and then starts the JVM.
In my setup, the info command shows me the following:
[keys] => 1128
[expires] => 1125
I'd like to find those 3 keys without an expiration date. I've already checked the docs to no avail. Any ideas?
Modified from a site that I can't find now.
redis-cli keys "*" | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq -1 ]; then echo "$LINE"; fi; done;
edit: Note, this is a blocking call.
#Waynn Lue's answer runs but uses the Redis KEYS command which Redis warns about:
Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases.
Redis documentation recommends using SCAN.
redis-cli --scan | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq -1 ]; then echo "$LINE"; fi; done;
If you want to scan for a specific key pattern, use:
redis-cli --scan --pattern "something*"
In case somebody is getting bad arguments or wrong number of arguments error,
put double quotes around $LINE.
So,it would be
redis-cli keys "*" | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq -1 ]; then echo "$LINE"; fi; done;
This happens when there are spaces in the key.
To me the accepted answer appears unusable for a medium-sized dataset, as it will run a redis-cli command for each and every key.
Instead I used this lua script to filter the keys inside the redis server:
local show_persistent = ARGV[1] ~= "expiring"
local keys = {}
for i, name in ipairs(redis.call("keys", "*")) do
local persistent = redis.call("pttl", name) < 0
if persistent == show_persistent then
table.insert(keys, name)
end
end
return keys
This can be called as
$ redis-cli --eval show-persistent-keys.lua
to get all keys without an expiration time. It also can be called as
$ redis-cli --eval show-persistent-keys.lua , expiring
to find the opposite key set of all keys with an expiration time set.
On the downside this may block for too long (appears fine for 1 M keys). I'd use scan instead but I happen to have to run this against a legacy Redis at version 2.6, which does not have scan available.
I needed to extract non-expiring keys from bigger (40GB) dataset, so using keys command was not suitable for me. So in case someone is looking for offline/non-blocking solution, you can use https://github.com/sripathikrishnan/redis-rdb-tools for extraction of non-expiring keys from redis rdb dump.
You can just install libraries via pip:
pip install rdbtools python-lzf
Then create simple parser which extracts keys and values which has expiration set to None:
from rdbtools import RdbParser, RdbCallback
from rdbtools.encodehelpers import bytes_to_unicode
class ParserCallback(RdbCallback):
def __init__(self):
super(ParserCallback, self).__init__(string_escape=None)
def encode_key(self, key):
return bytes_to_unicode(key, self._escape, skip_printable=True)
def encode_value(self, val):
return bytes_to_unicode(val, self._escape)
def set(self, key, value, expiry, info):
if expiry is None:
print('%s = %s' % (self.encode_key(key), self.encode_value(value)))
callback = ParserCallback()
parser = RdbParser(callback)
parser.parse('/path/to/dump.rdb')
#!/usr/bin/env python
import argparse
import redis
p = argparse.ArgumentParser()
p.add_argument("-i", '--host', type=str, default="127.0.0.1", help="redis host", required=False)
p.add_argument("-p", '--port', type=int, default=6379, help="redis port", required=False)
p.add_argument("-n", '--db', type=int, default=0, help="redis database", required=False)
args = p.parse_args()
r = redis.Redis(host=args.host, port=args.port, db=args.db)
try:
keys = r.keys()
for key in keys:
if r.ttl(key) < 0:
print(key)
except KeyboardInterrupt:
pass