I would like to go over million of keys and delete all the ones that remained 10 days or less to live.
I've got this
local cursor='0';
repeat
local keysVar = {};
local scanResult = redis.call('SCAN', cursor, 'MATCH', 'lucee-storage:session:*', 'COUNT', 100);
local keys = scanResult[2];
for i = 1, #keys do
ttl = redis.call('TTL', keys[i]);
if ttl < 864000 then
keysVar[i] = keys[i];
end;
end;
redis.call('UNLINK', unpack(keysVar));
cursor = scanResult[1];
until cursor == '0';
The problem is that this runs forever which impacts prod. Ideally I would like something with redid-cli and pipeline.
Do you have ideas how to tune it?
Try to do it from bash as I wanted:
redis-cli -h $HOST --scan --pattern 'keys:*' |
xargs -L 1 echo "eval \"local ttl=redis.call('TTL', ARGV[1]);
if ttl < 864000 then
redis.call('UNLINK', ARGV[1]);
end;
\"
0 " > pipe
cat pipe | redis-cli -h $HOST --pipe
There's a pipe logic where I read in bulk of 5000 and send to reds
Related
I have a shell script that running a count in sql but when I put the script via crontab the query doesn't take enough time and the script run quickly, and no results while it should take about 25 min as it count millions of data, when I run it manually it gives the right results, so what is the reason, this is a sample of the script,
#!/bin/bash
COUNT=`sqlplus -s username/pass#192.168.1.10:1521/oracl << EOF
SELECT count(*) AS COUNT from table1 where
trunc(generation_timestamp)=trunc(sysdate-1);
EOF << EOF EXIT; EOF`
if [[ $COUNT -ne 0 ]];
then sleep 0
else echo "No Data in table1"> /home/data/file.txt
fi
0 * * * * sh /home/date/count.sh
I have a csv that has around 25k rows. I have to pick 1000 rows from column#1 and column#2 at a time and then next 1000 rows and so on.
I am using below command, and its working fine in picking up all the values from column#1 and Column#2 i.e 25K fields from both the columns, I want to pick value like 1-1000, put them in my sql export query then 1001-2000,2001-3000 and so on and put the value in WHERE IN in my export query and append the result in dbData.csv file.
My code is below:
awk -F ',' 'NR > 2 {print $1}' $INPUT > column1.txt
i=$(cat column1.txt | sed -n -e 'H;${x;s/\n/,/g;s/^,//;p;}')
awk -F ',' 'NR > 2 {print $2}' $INPUT > column2.txt
j=$(cat column2.txt | sed -n -e 'H;${x;s/\n/,/g;s/^,//;p;}')
echo "Please wait - connecting to database..."
db2 connect to $sourceDBStr user user123 using pas123
db2 "export to dbData.csv of del select partnumber,language_id as LanguageId from CATENTRY c , CATENTDESC cd where c.CATENTRY_ID=cd.CATENTRY_ID and c.PARTNUMBER in ($i) and cd.language_id in ($j)"
Let's assume the two first fields of your input CSV are "simple" (no spaces, no commas...) and do not need any kind of quoting. You could generate the tricky part of your query string with an awk script:
# foo.awk
NR >= first && NR <= last {
c1[n+0] = $1
c2[n++] = $2
}
END {
for(i = 0; i < n-1; i++) printf("%s,", c1[i])
printf("%s) %s (%s", c1[n-1], midstr, c2[0])
for(i = 1; i < n; i++) printf(",%s", c2[i])
}
And then use it in a bash loop to process 1000 records per iteration, store the result of the query in a temporary file (e.g., tmp.csv in the following bash script) that you concatenate to your dbData.csv file. The following example bash script uses the same parameters as you do (INPUT, sourceDBStr) and the same constants (dbData.csv, 1000, user123, pas123). Adapt if you need more flexibility. Error management (input file not found, DB connection error, DB query error...) is left as a bash exercise (but should be done).
prefix="export to tmp.csv of del select partnumber,language_id as LanguageId from CATENTRY c , CATENTDESC cd where c.CATENTRY_ID=cd.CATENTRY_ID and c.PARTNUMBER in"
midstr="and cd.language_id in"
rm -f dbData.csv
len=$(cat "$INPUT" | wc -l)
for (( first = 2; first <= len - 999; first += 1000 )); do
(( last = len < first + 999 ? len : first + 999 ))
query=$(awk -F ',' -f foo.awk -v midstr="$midstr" -v first="$first" \
-v last="$last" "$INPUT")
echo "Please wait - connecting to database..."
db2 connect to $sourceDBStr user user123 using pas123
db2 "$prefix ($query)"
cat tmp.csv >> dbData.csv
done
rm -f tmp.csv
But there are other ways using split, bash arrays and simpler awk or sed scripts. Example:
declare -a arr=()
prefix="export to tmp.csv of del select partnumber,language_id as LanguageId from CATENTRY c , CATENTDESC cd where c.CATENTRY_ID=cd.CATENTRY_ID and c.PARTNUMBER in"
midstr="and cd.language_id in"
awk -F, 'NR>1 {print $1, $2}' "$INPUT" | split -l 1000 - foobar
rm -f dbData.csv
for f in foobar*; do
arr=($(awk '{print $1 ","}' "$f"))
i="${arr[*]}"
arr=($(awk '{print $2 ","}' "$f"))
j="${arr[*]}"
echo "Please wait - connecting to database..."
db2 connect to $sourceDBStr user user123 using pas123
db2 "$prefix (${i%,}) $midstr (${j%,})"
cat tmp.csv >> dbData.csv
rm -f "$f"
done
rm -f tmp.csv
Easiest to give an example.
bash-$ psql -c 'select relname, reltype from pg_catalog.pg_class limit 5;
relname | reltype
------------------------+---------
bme_reltag_02 | 0
bme_reltag_type1_type2 | 0
bme_reltag_10 | 0
bme_reltag_11 | 0
bme_reltag_cvalue3 | 0 👈 what I care about
But what I am really interested in is anything with cvalue in it. Rather than modifying each query by hand (yes, I know I could do it), I can egrep what I care about.
psql -c 'select relname, reltype from pg_catalog.pg_class limit 5;' | egrep 'cvalue'
but that strips out the first two lines with the column headers.
bme_reltag_cvalue3 | 0
I know I can also do this:
psql -c 'select relname, reltype from pg_catalog.pg_class limit 5;' | head -2 && psql -c 'select relname, reltype from pg_catalog.pg_class limit 5;' | egrep 'cvalue'
relname | reltype
------------------------+---------
bme_reltag_cvalue3 | 0
but what I really want to do is to keep the head (or tail) of some lines one way and then process the rest another.
My particular use case here is grepping the contents of arbitrary psql selects, but I'm curious as to what bash capabilities are in this domain.
I've done this before by writing to a temp file and then processing the temp file in multiple steps, but that's not what I am looking for.
A while read loop and grep, if that is acceptable.
#!/usr/bin/env bash
while IFS= read -r lines; do
[[ $lines == [12]* ]] && echo "${lines#*:}"
[[ $lines == *cvalue[0-9]* ]] && echo "${lines#*:}"
done < <(psql -c 'select relname, reltype from pg_catalog.pg_class limit 5;' | grep -n .)
Without the grep an alternative is a counter to know the line number, which will be a pure bash solution.
#!/usr/bin/env bash
counter=1
while IFS= read -r lines; do
[[ $counter == [12] ]] && echo "$lines"
[[ $lines == *cvalue[0-9]* ]] && echo "$lines"
((counter++))
done < <(psql -c 'select relname, reltype from pg_catalog.pg_class limit 5;')
If bash4+ is available.
#!/usr/bin/env bash
mapfile -t files < <(psql -c 'select relname, reltype from pg_catalog.pg_class limit 5;')
printf '%s\n' "${files[0]}" "${files[1]}"
unset 'files[0]' 'files[1]'
for file in "${files[#]}"; do
[[ $file == *cvalue[0-9]* ]] && echo "$file"
done
By default the builtin read strips the leading and trailing white spaces, so in this case we don't want that, so we use IFS=
grep -n . adds the line number with a :
[12] is a glob not regex which means either 1 or 2 and the glob * will match if it is the first character of the line.
*cvalue[0-9]* will match cvalue and any amount of int/digit next to it.
"${lines#*:}" is a parameter expansion that strips the leading :
<( ) is called process substitution.
$ psql -c ... | awk 'NR<3 || /cvalue/' file
This can be done with sed using its range feature to only operate on lines 3 and beyond
sed '3,${/cvalue/!{d;};}'
Proof of Concept
$ cat ./psql
relname | reltype
------------------------+---------
bme_reltag_02 | 0
bme_reltag_type1_type2 | 0
bme_reltag_10 | 0
bme_reltag_11 | 0
bme_reltag_cvalue3 | 0
$ sed '3,${/cvalue/!{d;};}' ./psql
relname | reltype
------------------------+---------
bme_reltag_cvalue3 | 0
Explanation
3,${...;}: Start processing from line 3 until the end of file $
/cvalue/!{d;}: Delete d any line that does not match (!) the regex /cvalue/
You can use bash.. tail.and head commands
cat file.sql | head -n 15 > head.sql
Replace the 15 with the number of lines
Or replace head with tail... for the bottom of the file
Can anyone please help me with understanding the below script from the "typeset" part in the script and what the while and if statements are doing in the script:
# Set effective dates for tasks
set -A EDATE `sqlplus -s / << ENDSQL
set pages 0 feed off
set timing off
alter session set nls_date_format='DD-MM-YYYY';
select sysdate + 42, sysdate + 51, sysdate + 50 from dual;
ENDSQL`
# Check effective dates set
# ${EDATE[0]} = SYSDATE + 42 for tasks NORMALISED
# ${EDATE[1]} = SYSDATE + 51 for tasks SUBTOTAL, SUBTOTAL_RAT
# ${EDATE[2]} = SYSDATE + 50 for tasks NORMALISED_EV,CHARGE
typeset -i C=0
while [[ $C -lt 3 ]] ; do
if [[ -z "${EDATE[C]}" ]] ; then
echo "FAILED TO SET ROTATE PARTITION TASKS EFFECTIVE DATE! PLEASE CHECK."
sms "${SV_SL}" "Failed to set Rotate Partition Tasks effective date. Please check."
exit -1
fi
let C+=1
done
In the first line you've used builtins which is used for permit modifying the properties of variables. You can use 'declare' as well. Both are nearly equal.
In your if statement line you've included semicolon ; which isn't really required at that place. The right if..else syntax in unix is as following given.
Same for while loop also, just remove the semicolon
If..else statement
x=5
y=20
if [ $x == $y ]
then
echo "x is equal to y"
else
echo "x is not equal to y"
fi
Above code's output: x is not equal to y
for while loop
a=0
while [ $u -lt 5 ]
do
echo $u
a=`expr $u + 1`
done
Above code's output:
0
1
2
3
4
-lt 5 indicates limits up to 5 for the while loop.
Also -z option used in if condition that is for variable or string is null.
i.e.
$ip="";
$[-n "$ip"] && echo "ip is not null"
$[-z "$fip"] && echo "ip is null"
Above code's output: ip is null.
How do I predefine a range of a Counter in Redis while setting it up. I want the Counter to have a predefined MAX and a MIN value(specifically in my case MIN value is 0) such that INCR or DECR return an error if the value is surpassing this Range. I went through the Redis Documentation and i didn't find any answer.
Redis does not provide this built-in, but you can use it to build it yourself. There are many ways to do this, my personal preference is using Lua scripts - read EVAL for more background.
In this case, I'd use this script:
local val = tonumber(redis.call('GET', KEYS[1]))
if not val then
val = 0
end
local inc = val + tonumber(ARGV[1])
if inc < tonumber(ARGV[2]) or inc > tonumber(ARGV[3]) then
error('Counter is out of bounds')
else
return redis.call('SET', KEYS[1], inc)
end
Here's the output of a sample run from the command line:
$ redis-cli --eval incrbyminmax.lua foo , 5 0 10
(integer) 5
$ redis-cli --eval incrbyminmax.lua foo , 5 0 10
(integer) 10
$ redis-cli --eval incrbyminmax.lua foo , 5 0 10
(error) ERR Error running script (call to f_ada0f9d33a6278f3e55797b9b4c89d5d8a012601): #user_script:8: user_script:8: Counter is out of bounds
$ redis-cli --eval incrbyminmax.lua foo , -9 0 10
(integer) 1
$ redis-cli --eval incrbyminmax.lua foo , -9 0 10
(error) ERR Error running script (call to f_ada0f9d33a6278f3e55797b9b4c89d5d8a012601): #user_script:8: user_script:8: Counter is out of bounds