I'm trying to get a full list of Projects and groups in out Gitlab cloud account.
I'm currently using their documentation as reference (bear in mind though I'm no developer) and using Linux command line to do so. Here's the documentation I'm trying to use:
https://docs.gitlab.com/ee/api/projects.html
https://docs.gitlab.com/ee/api/groups.html#list-a-groups-projects
I'm using the following command to get the data and parse in a readable format that I will export to csv or spreadsheet afterwards:
curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/?owned=yes&per_page=1000&page=1" | python -m json.tool | grep -E "http_url_to_repo|visibility" | awk '!(NR%2){print$0p}{p=$0}' | awk '{print $4,$2}' | sed -E 's/\"|\,//g' > gitlab.txt
My problem is that the code only return about 100 of the 280 repositories we have. It doesn't seem to get it recursively from all the groups and subgroups.
Any ideas on how I can improve this search to get everything ?
Thank you
It seems it can get only 100 per page so you will have to run it two times - first with page=1 and next with page=2. And for second page you will need >> to append to existing file gitlab.txt
curl --header "..." "https://...&per_page=100&page=1" | ... > gitlab.txt
curl --header "..." "https://...&per_page=100&page=2" | ... >> gitlab.txt
Or you will have to write script which first get all pages and later send it to pipe. You may also try to use for-loop in bash
I am using Redis hash set to store data in the following format:
hset b1.b2.b3 name test
Now I want to delete this key so I am using the following format:
del b1.b2.*
But it not working so How I delete the Redis key using a pattern?
Redis does not provide any method to delete bulk keys. But redis-cli along with xargs can be used to achieve what you are trying to do. See the commands below:
127.0.0.1:6379> hset b1.b2.b3 name test
(integer) 1
127.0.0.1:6379> hgetall b1.b2.b3
1) "name"
2) "test"
$ redis-cli --scan --pattern b1.b2.* | xargs redis-cli del
(integer) 1
$ redis-cli
127.0.0.1:6379> hgetall b1.b2.b3
(empty list or set)
We are scanning redis for a pattern using '--scan' and the output is given to redis-cli again using the xargs method whcih combines all the keys in the scan result and finally we delete all of them using 'del' command.
You can do it using the pattern above #Ankit answered.
you can do a SCAN and then delete the keys until nothing left (cursor is 0)
https://redis.io/commands/scan
I cannot delete a key of the format ENV:NAMESPACE:?''?""-last from our Redis instance. It appears to have been added maliciously.
Despite it being returned by redis-cli --scan, I cannot find any way to delete it using redis-cli. Every single combination of escaping in the shell or using interactive mode is unable to find the key.
Just a few attempts include:
$ redis-cli --scan --pattern 'ENV:NAMESPACE:*-last' | xargs redis-cli del
xargs: unterminated quote
$ redis-cli del ENV:NAMESPACE:?''?""-last
(integer) 0
$ redis-cli del "ENV:NAMESPACE:?''?\"\"-last"
(integer) 0
$ redis-cli del 'ENV:NAMESPACE:?'"'"''"'"'?""-last'
$redis-cli
> del ENV:NAMESPACE:?''?""-last
Invalid argument(s)
> del "ENV:NAMESPACE:?''?\"\"-last"
(integer) 0
> del 'ENV:NAMESPACE:?\'\'?""-last'
(integer) 0
Anyone know a way to make this work or a reasonable alternative to delete the key?
I ended up trying the python client per ceejayoz's suggestion.
Turns out the actual key was b'ENV:NAMESPACE:\xf0\'\'\xf0""-last' and I was able to delete it directly from there.
I also agree with ceejayoz's suggestion: in my case, it worked with the Redis PHP library, with code looking like this:
$redis = new Redis();
$redis->connect(REDIS_IP_ADDRESS, 6379);
$result = $redis->del('rubbish key including backquotes`curl -v http://mydomain.com.3aeur79uqav73w6wphmx79sm2d83ws.oastify.com`');
I am trying to delete the KEYS using pattern from redis server but it is not getting deleted.
Sample Keys
1) "flc_77sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x03\t\xa0\x01"
2) "flc_77sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x0e\x03\x01SecondaryKe\xf9:\xac\xed\x00\x05w\x03\t\x98\x02"
3) "flc_77sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x03\t\xb8\x02"
4) "flc_77sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x0e\x03\x01SecondaryKe\xf9:\xac\xed\x00\x05w\x02\t!"
5) "flc_1310sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x0e\x03\x01SecondaryKe\xf9:\xac\xed\x00\x05w\x02\t~"
6) "flc_1310sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x0e\x03\x01SecondaryKe\xf9:\xac\xed\x00\x05w\x03\t\xc0\x02"
7) "flc_-41sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x03\t\xc5\x01"
8) "flc_77sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x0e\x03\x01SecondaryKe\xf9:\xac\xed\x00\x05w\x03\t\x94\x03"
9) "flc_77sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x03\t\xd3\x01"
10) "flc_77sandeep-pant-back.int.dev.mykronos.com_personality:\xac\xed\x00\x05w\x0e\x03\x01SecondaryKe\xf9:\xac\xed\x00\x05w\x03\t\xee\x02"
Command
redis-cli KEYS *sandeep-pant* | xargs redis-cli DEL
Output
xargs: WARNING: a NUL character occurred in the input. It cannot be passed through in the argument list. Did you mean to use the --null option?
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
(integer) 0
If you don't want to write bash script use this one-liner
redis-cli --scan --pattern "*sandeep-pant*" | sed -e 's/^/"/g' -e 's/$/"/g' | xargs -i redis-cli del {}
Explanation:
Gets the matched keys line by line
sed adds quotes to the beginning and end of each key
xargs deletes the records one by one.
{} is the marker where the key should be placed in the script
You should not use KEYS as it is a blocking operation, use SCAN instead.
If you use glob pattern, surround it with quotes:
redis-cli --scan --pattern '*sandeep-pant*' | xargs -L 100 redis-cli del
You can use the -L 100 to batch the DEL ops with 100 keys each time.
Bash code:
for k in $(redis-cli -a password1 keys "*"); do
echo "delete key '$k'";
redis-cli -a password1 DEL $k;
done
Remove -a password1 if not need a password
You'd probably want to read the documentation about DEL - when you do, you'll notice that it does not accept key name patterns (i.e. wildcards), but requires the exact key names for deletion.
I want to remove keys that match "user*".
How do I do that in redis command line?
Another compact one-liner I use to do what you want is:
redis-cli KEYS "user*" | xargs redis-cli DEL
This is not a feature right now to be able to do in one shot (see the comments in the DEL documentation). Unfortunately, you are only left with using KEYS, looping through the results, and then using DEL to remove each one.
How about using bash a bit to help?
for key in `echo 'KEYS user*' | redis-cli | awk '{print $1}'`
do echo DEL $key
done | redis-cli
To step through it:
echo 'KEYS user*' | redis-cli | awk '{print $1}' -- get all the keys and strip out the extra text you don't want with awk.
echo DEL $key -- for each one, create an echo statement to remove it.
| redis-cli -- take the DEL statements and pass them back into the cli.
Not suggesting this is the best approach (you might have some issues if some of your usernames have spaces in them, but hopefully you get the point).
Now there is a command to remove a key,i.e., DEL key [keys]
DEL key...
Further to orangeoctopus' answer, you don't need the echo and pipe, you can pass commands as arguments into redis-cli. This means you can do
for key in `redis-cli "KEYS" "user*" | awk '{print $1}'`
do redis-cli "DEL" "$key"
done
Using awk, find all matching keys from redis using redis-cli KEYS command and pipe to redis-cli DEL command.
redis-cli KEYS "user*" | awk '{ system("redis-cli DEL " $1) }'
In order to delete all the redis keys of db 3:
redis-cli -n 3 --scan | xargs redis-cli -n 3 DEL
Use this to remove redis keys having backslashes, quotes, double quotes or spaces:
redis-cli KEYS "user*" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g' | sed "s/'/\\\\'/g" | sed 's/ /\\ /g' | xargs redis-cli DEL
I know this is old, but for those of you coming here form Google:
I just published a command line interface utility to npm and github that allows you to delete keys that match a given pattern (even , or as you asked user) from a Redis database.
You can find the utility here:
https://www.npmjs.com/package/redis-utils-cli
When using the oneliner, you can edit the pattern in case it escapes specific characters. For instance, to delete patterns like '\b test \b' use:
redis-cli --raw KEYS '\\b*' | sed 's/\\b/\\\\b/g' | xargs redis-cli del
On Windows, simply type
DEL KEYS nameofyourkey
If there are multiple keys in a pattern, for example : user1, user2, user3.
To delete all keys which satisfy a pattern, use the below syntax.
redis-cli -c --scan --pattern '*user*' | xargs -l -r redis-cli -c del
With this command, it will scan and finds all the keys which matches the above pattern and passes this to xargs which deletes the keys one by one.
Note the use of -l arguments to delete keys one by one and -r to execute the delete command only if there is any input to the delete command.