I made a user keys on REDIS like that.
The pattern > u:info:(userNumber):userId
example >
mset u:info:1:userId Jack, u:info:2:userId Mason, u:info:3:userId Justin
And I wanna have userId list.
I have a solution for that. First, use SCAN and parsing all keys and make an array using parsed userNumber like that arr = [1,2,3]. Second, execute a command like below.
command > mget u:info: + arr[0] + :userid, u:info: + arr[1] + :userid ...
But It needs the parsing stuff. I wanna do it just using a command. Is there any good way for that?
Related
ch_files = Channel.fromPath("myfiles/*.csv")
ch_parameters = Channel.from(['A','B, 'C', 'D'])
ch_samplesize = Channel.from([4, 16, 128])
process makeGrid {
input:
path input_file from ch_files
each parameter from ch_parameters
each samplesize from ch_samplesize
output:
tuple path(input_file), parameter, samplesize, path("config_file.ini") into settings_grid
"""
echo "parameter=$parameter;sampleSize=$samplesize" > config_file.ini
"""
}
gives me a number_of_files * 4 * 3 grid of settings files, so I can run some script for each combination of parameters and input files.
How do I add some ID to each line of this grid? A row ID would be OK, but I would even prefer some unique 6-digit alphanumeric code without a "meaning" because the order in the table doesn't matter. I could extract out the last part of the working folder which is seemingly unique per process; but I don't think it is ideal to rely on sed and $PWD for this, and I didn't see it provided as a runtime metadata variable provider. (plus it's a bit long but OK). In a former setup I had a job ID from the LSF cluster system for this purpose, but I want this to be portable.
Every combination is not guaranteed to be unique (e.g. having parameter 'A' twice in the input channel should be valid).
To be clear, I would like this output
file1.csv A 4 pathto/config.ini 1ac5r
file1.csv A 16 pathto/config.ini 7zfge
file1.csv A 128 pathto/config.ini ztgg4
file2.csv A 4 pathto/config.ini 123js
etc.
Given the input declaration, which uses the each qualifier as an input repeater, it will be difficult to append some unique id to the grid without some refactoring to use either the combine or cross operators. If the inputs are just files or simple values (like in your example code), refactoring doesn't make much sense.
To get a unique code, the simple options are:
Like you mentioned, there's no way, unfortunately, to access the unique task hash without some hack to parse $PWD. Although, it might be possible to use BASH parameter substitution to avoid sed/awk/cut (assuming BASH is your shell of course...) you could try using: "${PWD##*/}"
You might instead prefer using ${task.index}, which is a unique index within the same task. Although the task index is not guaranteed to be unique across executions, it should be sufficient in most cases. It can also be formatted for example:
process example {
...
script:
def idx = String.format("%06d", task.index)
"""
echo "${idx}"
"""
}
Alternatively, create your own UUID. You might be able to take the first N characters but this will of course decrease the likelihood of the IDs being unique (not that there was any guarantee of that anyway). This might not really matter though for a small finite set of inputs:
process example {
...
script:
def uuid = UUID.randomUUID().toString()
"""
echo "${uuid}"
echo "${uuid.take(6)}"
echo "${uuid.takeBefore('-')}"
"""
}
When getting all the keys from Redis, like this:
redis.server.com:6379> keys *
1) "z13235jxby03knne1w1gucl5"
Instead of manually copying the long key to execute get z13235jxby03knne1w1gucl5, I'd like to run something like get $(1) (pseudo code) to get the value at position 1, as output by the keys command.
Is this possible, if not, is there any workaround to not have to manually copy paste?
Note, I don't want to solve this with a script, then I prefer just copy and paste it
off the top of my head, I'm not aware of a way from inside the cli.
But, Regardless of any performance implications, you can pipe the cli command lines together , but you have to do it from the shell
redis-cli --raw KEYS "*" | sed -n 1p | xargs redis-cli GET
where the 1 in :
sed -n 1p
is the line number (1-based index) inside KEYS output.
but still you need to do your validations; like making sure the index is withing the nuber of keys returned by the KEYS command all keys are of simple string type ; not sets, hash maps, etc...
I have a lot of analytics data that I'm adding to redis. I plan on incrementally moving the data out of redis and into my database.
I know I can use KEYS [the_key]:* to get all keys that match. For example, I can do that to get the following:
127.0.0.1:6379> KEYS c_Track:*
1) "c_Track:6c93a5c1-77e9-4c4a-9232-bf182713a02e"
2) "c_Track:2c9d99c2-af37-4de9-ac64-b48f339e97a9"
3) "c_Track:9e7fd190-86d9-4b4a-9a70-7bf4c7768eef"
4) "c_Track:7f2d2e98-7440-4fd7-a80a-2af309ab15a4"
Is there a recommended way to get these values easily? I can get the keys, but how can I get all the values as well? I can loop through the keys to get the values, but is there some one-shot method for doing this?
Also I know I shouldn't use keys, but this is just an example. Thanks
Thanks
Also I know I shouldn't use keys
So don't. Use SCAN instead.
is there some one-shot method for doing this?
No, not as a core Redis command, but given the need this is fairly simple to achieve with a server-side Lua script. For example, assuming that your values are strings, you could do something like the following:
local cursor = tonumber(ARGV[1])
local pattern = ARGV[2]
local scan = redis.call('SCAN', cursor, 'MATCH', pattern)
for i, v in ipairs(scan[2]) do
local val = redis.call('GET', v)
scan[2][i] = { v, val }
end
return scan
Assuming that this script is saved under "scan.lua", you can run it as follows:
$ redis-cli SET foo bar
OK
$ redis-cli SET baz qaz
OK
$ redis-cli --eval scan.lua , 0 "*"
1) "0"
2) 1) 1) "baz"
2) "qaz"
2) 1) "foo"
2) "bar"
To scan your entire keyspace, call the script with the returned cursor until it returns 0.
Notes:
1) If your keys are of different types, you should change the script accordingly (e.g. https://github.com/itamarhaber/redis-lua-scripts/blob/master/scanfetch.lua).
2) While this script goes against the common recommendation of generating key names inside a script, it is still safe to run as SCAN returns keys that are in the server's keyspace (whether single-instance or clustered).
My redis collection contains many keys
I want to be able to flush them all except all the keys that start with:
"configurations::"
is this possible?
You can do this
redis-cli KEYS "*" | grep -v "configurations::" | xargs redis-cli DEL
List all keys into the redis, remove from the list keys that contains "configurations::" and delete them from the redis
Edit
As #Sergio Tulentsev notice it keys is not for use in production. I used this python script to remove keys on prodution redis. I stoped replication from master to slave before call the script.
#!/usr/bin/env python
import redis
import time
pattern = "yourpattern*"
poolSlave = redis.ConnectionPool(host='yourslavehost', port=6379, db=0)
redisSlave = redis.Redis(connection_pool=poolSlave)
poolMaster = redis.ConnectionPool(host='yourmasterhost', port=6379, db=0)
redisMaster = redis.Redis(connection_pool=poolMaster)
cursor = '0'
while cursor != 0:
cursor, data = redisSlave.scan(cursor, pattern, 1000)
print "cursor: "+str(cursor)
for key in data:
redisMaster.delete(key)
print "delete key: "+key
# reduce call per second on production server
time.sleep(1)
The SCAN & DEL approach (as proposed by #khanou) is the best ad-hoc solution. Alternatively, you could keep an index of all your configurations:: key names with a Redis Set (simply SADD the key's name to it whenever you create a new configurations:: key). Once you have this set you can SSCAN it to get all the relevant key names more efficiently (don't forget to SREM from it whenever you DEL though).
Yes, it's possible. Enumerate all the keys, evaluate each one and delete if it fits the criteria for deletion.
There is no built-in redis command for this, if this is what you were asking.
It might be possible to cook up a Lua script that will do this (and it'll look to your app that it's a single command), but still it's the same approach under the hood.
I want to use Redis's HSCAN command in my assignment but I have no idea how it works. Redis's official page (http://redis.io/commands/hscan) for this command gives me blank page.
I am getting continuous input data and saving them instantaneously in multiple hashes in Redis and I would like to iterate through all of them at later point of time.
I'm using following command to save my data
HMSET myhash ordertype "neworder" order_ts "1234" act_type "order_ack" ack_ts "1240"
HMSET myhash2 ordertype "neworder" order_ts "2234" act_type "order_ack" ack_ts "2240"
Can anyone give me some examples of how to use HSCAN?
In my case I would like to get following output
1) myhash
2) myhash2
3) myhash3
.
.
.
.
Commands
Start a full hash scan with:
HSCAN myhash 0
Start a hash scan with fields matching a pattern with:
HSCAN myhash 0 MATCH order_*
Start a hash scan with fields matching a pattern and forcing the scan command to do more scanning with:
HSCAN myhash 0 MATCH order_* COUNT 1000
Note
Don't forget that MATCH can return little to no element for each iteration, as explained in the documentation:
It is important to note that the MATCH filter is applied after
elements are retrieved from the collection, just before returning data
to the client. This means that if the pattern matches very little
elements inside the collection, SCAN will likely return no elements in
most iterations.
And that's why you can use COUNT to force more scanning for each iteration.
[Update] As Didier Spezia specified, you'll need Redis 2.8+ to use the *SCAN commands.
As mentioned by you. you need to get the output of hash keys
myhash
myhash2
myhash3
HSCAN is not for this purpose. HSCAN is to scan the fields of a particular HASH. so you can scan the fields of myhash or myhash2. But if you want to find the keys on the basis of patterns you have two options.
Create a SET with the HASH keys
SADD hashkeys "myhash" "myhash1" "myhash2"
SMEMBERS hashkeys
1) "myhash"
2) "myhash2"
3) "myhash1"
If you are using 2.8 I would recommend you use sscan instead of smembers
sscan hashkeys 0 match my*
USE ONLY SCAN COMMAND
scan 0 MATCH myhash* count 1000
USE key matching (NOT RECOMMENDED AS it will block the redis server)
keys myhash*
So long story cut sort to fetch the keys you can use SMEMBERS, SSCAN or KEYS. Ofcourse best is SSCAN if you are using redis 2.8