redis move all keys - redis

is it possible to use redis's MOVE command to move all keys from 1 database to another? The move command only moves 1 key, but I need to move all the keys in the database.

I would recommend taking a look at the following alpha version app to backup and restore redis databases.. (you can install it via gem install redis-dump). You could redis-dump your databaseand then redis-load into another database via the --database argument.
redis-dump project
If this doesn't fit your purposes, you may need to make use of a scripting language's redis bindings (or alternatively throw something together using bash / redis-cli / xargs, etc). If you need assistance along these lines then we probably need more details first.

I've wrote a small python script to move data between two redis servers:(only support list and string types, and you must install python redis client):
'''
Created on 2011-11-9
#author: wuyi
'''
import redis
from optparse import OptionParser
import time
def mv_str(r_source, r_dest, quiet):
keys = r_source.keys("*")
for k in keys:
if r_dest.keys(k):
print "skipping %s"%k
continue
else:
print "copying %s"%k
r_dest.set(k, r_source.get(k))
def mv_list(r_source, r_dest, quiet):
keys = r_source.keys("*")
for k in keys:
length = r_source.llen(k)
i = 0
while (i<length):
print "add queue no.:%d"%i
v = r_source.lindex(k, i)
r_dest.rpush(k, v)
i += 1
if __name__ == "__main__":
usage = """usage: %prog [options] source dest"""
parser = OptionParser(usage=usage)
parser.add_option("-q", "--quiet", dest="quiet",
default = False, action="store_true",
help="quiet mode")
parser.add_option("-p", "--port", dest="port",
default = 6380,
help="port for both source and dest")
parser.add_option("", "--dbs", dest="dbs",
default = "0",
help="db list: 0 1 120 220...")
parser.add_option("-t", "--type", dest="type",
default = "normal",
help="available types: normal, lpoplist")
parser.add_option("", "--tmpdb", dest="tmpdb",
default = 0,
help="tmp db number to store tmp data")
(options, args) = parser.parse_args()
if not len(args) == 2:
print usage
exit(1)
source = args[0]
dest = args[1]
if source == dest:
print "dest must not be the same as source!"
exit(2)
dbs = options.dbs.split(' ')
for db in dbs:
r_source = redis.Redis(host=source, db=db, password="", port=int(options.port))
r_dest = redis.Redis(host=dest, db=db, password="", port=int(options.port))
print "______________db____________:%s"%db
time.sleep(2)
if options.type == "normal":
mv_str(r_source, r_dest, options.quiet)
elif options.type == "lpoplist":
mv_list(r_source, r_dest, options.quiet)
del r_source
del r_dest

you can try my own tool, rdd
it's a command line utility,
can dump database to a file, work on it (filter, match, merge, ...), and back it in a redis instance
take care, alpha stage, https://github.com/r043v/rdd/

Now that redis has scripting using lua, you can easily write a command that loops through all the keys, checks their type and moves them accordingly to a new database.

I suggest you can try it as below:
1. copy the rdb file to another dir;
2. modify the rdb file name;
3. modify the redis configure file adapter to the new db;

Related

Why do I get a `java.nio.file.ProviderMismatchException` when I access `isEmpty()` on a staged file

I am getting a java.nio.file.ProviderMismatchException when I run the following script:
process a {
output:
file _biosample_id optional true into biosample_id
script:
"""
touch _biosample_id
"""
}
process b {
input:
file _biosample_id from biosample_id.ifEmpty{file("_biosample_id")}
script:
def biosample_id_option = _biosample_id.isEmpty() ? '' : "--biosample_id \$(cat _biosample_id)"
"""
echo \$(cat ${_biosample_id})
"""
}
i'm using a slightly modified version of Optional Input pattern.
Any ideas on why I'm getting the java.nio.file.ProviderMismatchException?
In your script block, _biosample_id is actually an instance of the nextflow.processor.TaskPath class. So to check if the file (or directory) is empty you can just call it's .empty() method. For example:
script:
def biosample_id_option = _biosample_id.empty() ? '' : "--biosample_id \$(< _biosample_id)"
I like your solution - I think it's neat. And I think it should be robust (but I haven't tested it). The optional input pattern that is recommended will fail when attempting to stage missing input files to a remote filesystem/object store. There is a solution however, which is to keep an empty file in your $baseDir and point to it in your scripts. For example:
params.inputs = 'prots/*{1,2,3}.fa'
params.filter = "${baseDir}/assets/null/NO_FILE"
prots_ch = Channel.fromPath(params.inputs)
opt_file = file(params.filter)
process foo {
input:
file seq from prots_ch
file opt from opt_file
script:
def filter = opt.name != 'NO_FILE' ? "--filter $opt" : ''
"""
your_commad --input $seq $filter
"""
}

Python3/Redis: redis.exceptions.ResponseError: unknown command 'JSON.SET'

I'm trying to run the sample program from this RedisLabs page.
I chose Option A - which was to set up the free Redis cloud server.
(Seems like if you install manually, then you have to add the JSON as a plugin.)
I'm able to connect and use other "set" commands, but getting error on JSON:
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 915, in parse_response
response = connection.read_response()
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'JSON.SET'
My Python test program (except put in the sample endpoint before posting):
import redis
import json
import pprint
host_info = "redis.us-east-1-1.ec2.cloud.redislabs.com"
redisObj = redis.Redis(host=host_info, port=18274, password='xxx')
print ("Normal call to Redis")
redisObj.set('foo', 'bar')
value = redisObj.get('foo')
print(value)
capitals = {
"Lebanon": "Beirut",
"Norway": "Oslo",
"France": "Paris"
}
print ("capitals - before call to Redis")
pprint.pprint(capitals)
print("JSON call to Redis")
redisObj.execute_command('JSON.SET', 'doc', '.', json.dumps(capitals))
print("Data Saved, now fetch data back from redis")
reply = json.loads(redisObj.execute_command('JSON.GET', 'doc'))
print("reply from Redis get")
pprint.pprint(reply)
This is the screen shot from their website where I created the database. I didn't see any option to enable JSON or add any modules.
Not sure this was available when I created the REDIS database, but it is now. When you create it on redislabs.com, you can turn on the modules, and pick one from the list.
Then use this library: "rejson" from https://pypi.org/project/rejson/ to get the method "jsonset" method, using such code such as this:
rj = Client(host=config_dict['REDIS_CONFIG_HOST'], port=config_dict['REDIS_CONFIG_PORT'], password=config_dict['REDIS_CONFIG_PASSWORD'], decode_responses=True)
out_doc = {}
out_doc['firstname'] = "John"
out_doc['lastname'] = "Doe"
rj.jsonset('config', Path.rootPath(), out_doc)
get_doc = rj.jsonget('config', Path.rootPath())
pprint.pprint(get_doc)
I'm not used the cloud redis, in my local the Python don't load the JSON.SET
I just so make done, in this sample https://onelinerhub.com/python-redis/save-json-to-redis

insert_many in pymongo not persisting

I'm having some issues with persisting documents with pymongo when using insert_many.
I'm handing over a list of dicts to insert_many and it works fine from inside the same script that does the inserting. Less so once the script has finished.
def row_to_doc(row):
rowdict = row.to_dict()
for key in rowdict:
val = rowdict[key]
if type(val) == float or type(val) == np.float64:
if np.isnan(val):
# If we want a SQL style document collection
rowdict[key] = None
# If we want a NoSQL style document collection
# del rowdict[key]
return rowdict
def dataframe_to_collection(df):
n = len(df)
doc_list = []
for k in range(n):
doc_list.append(row_to_doc(df.iloc[k]))
return doc_list
def get_mongodb_client(host="localhost", port=27017):
return MongoClient(host, port)
def create_collection(client):
db = client["material"]
return db["master-data"]
def add_docs_to_mongo(collection, doc_list):
collection.insert_many(doc_list)
def main():
client = get_mongodb_client()
csv_fname = "some_csv_fname.csv"
df = get_clean_csv(csv_fname)
doc_list = dataframe_to_collection(df)
collection = create_collection(client)
add_docs_to_mongo(collection, doc_list)
test_doc = collection.find_one({"MATERIAL": "000000000000000001"})
When I open up another python REPL and start looking through the client.material.master_data collection with collection.find_one({"MATERIAL": "000000000000000001"}) or collection.count_documents({}) I get None for the find_one and 0 for the count_documents.
Is there a step where I need to call some method to persist the data to disk? db.collection.save() in the mongo client API sounds like what I need but it's just another way of inserting documents from what I have read. Any help would be greatly appreciated.
The problem was that I was getting my collection via client.db_name.collection_name and it wasn't getting the same collection I was creating with my code. client.db_name["collection-name"] solved my issue. Weird.

Shared dict of multiprocessing.Lock in python3

Is it possible to have a shared dict of Locks in python3? I need multiple locks because I want to protect a dict of shared resources. Each resource gets a lock:
manager = multiprocessing.Manager()
locks = manager.dict({key : manager.Lock() for key in range(100)})
shared_resource = manager.dict({key : SomeClass() for key in range(100)})
# later in a multi-processed function
def foo(key):
# ...
locks[key].acquire()
shared_resource[key] = ...
locks[key].release()
# ...
This toy example would fail with:
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Unserializable message: ('#RETURN', <unlocked _thread.lock object at 0x7f9a4c9dc468>)
Any idea how to get around this problem? I could use conditional variables? Or how would you protect a list of resources?
Ok, seems like it's a bug with ptyhon3.5.
With python3.6 it works like a charm.
I'm sure it's possible. I don't get an error when I run this code...I just substituted SomeClass with 'x'. So maybe there's an issue there.
Also, using a context manager to acquire and release the lock is a nice little abstraction...
manager = multiprocessing.Manager()
locks = {key : manager.Lock() for key in range(100)}
shared_resource = {key : 'x' for key in range(100)}
# later in a multi-processed function
def foo(key):
# ...
with locks[key]:
shared_resource[key] = 'xoyo'
if __name__ == '__main__':
p = Process(target=foo, args=(1,))
p.start()
p = Process(target=foo, args=(1,))
p.start()
p = Process(target=foo, args=(1,))
p.start()
p.join()

Freezing the time and date of an array of VMs

Is there a way to freeze the time and date of a VM, so that it is not synchronized with the guest BIOS and/or the internet? I have seen some solutions which talk about killing services inside the VM, but I wish to avoid this as it changes the "clean state" of the VM (used for testing purposes).
To clarify: I don't want to set the time offset for a VM, I want to set the exact time that will be passed to the OS at boot time.
From there, is there a way to do this across a large number of VMs?
Ended up solving the issue with a python script.
To use it, you edit the VM_NAMES list to contain the names of the VMs as they appear in VirtualBox, then set the RESET_TIME_VALUE according to the date and time you wish to send the VMs.
If you have installed VirtualBox in a non-default location, edit the VIRTUAL_BOX_MANAGE_PATH variable as well.
To run, call the main method.
import datetime
import subprocess
VIRTUAL_BOX_MANAGE_PATH = r"C:\Program Files\Oracle\VirtualBox\vboxmanage.exe"
SET_EXTRA_DATA_COMMAND = r"setextradata"
GET_HOST_TIME_DISABLE_COMMAND = "\"VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled\" \"1\""
MODIFY_VM_COMMAND = r"modifyvm"
BIOS_SYSTEM_TIME_OFFSET = r"--biossystemtimeoffset"
# Edit this list to add more Virtual Machines
VM_NAMES = ("xxx",
"yyy",
"zzz")
RESET_TIME_VALUE = datetime.datetime(2014, 6, 7, 13, 0, 0, 0)
def main():
for vm in VM_NAMES:
reset_time(vm)
disable_time_sync(vm)
def reset_time(vm_name):
"""
Resets the VM to the clean install time
"""
args = get_subprocess_args_set_bios_time(vm_name, RESET_TIME_VALUE)
print("Resetting time on VM [" + vm_name + "] to " + str(RESET_TIME_VALUE) + " ...")
subprocess.call(args)
print("\tDone.")
def disable_time_sync(vm_name):
"""
Disables the time synchronization of a VM with the BIOS
"""
args = [
VIRTUAL_BOX_MANAGE_PATH,
SET_EXTRA_DATA_COMMAND,
vm_name,
GET_HOST_TIME_DISABLE_COMMAND
]
print("Disabling time synchronization on VM [" + vm_name + "] ...")
subprocess.call(args)
print("\tDone.")
def get_subprocess_args_set_bios_time(vm_name, datetime_to_set):
"""
Returns a list containing the arguments to pass to the subprocess method
to start the Virtual Box Manage program and set the BIOS time to the supplied value
"""
return [
VIRTUAL_BOX_MANAGE_PATH,
MODIFY_VM_COMMAND,
vm_name,
BIOS_SYSTEM_TIME_OFFSET,
str(get_msec_time_difference(datetime.datetime.now(), datetime_to_set))
]
def get_msec_time_difference(reference_point, check_point):
"""
Computes the offset in msec from the reference point to the check point
"""
return int(round( (check_point - reference_point).total_seconds() * 1000 ))
" To run, call the main method."
..or just add at the end of the file
if __name__ == '__main__':
main()