Not able to perform mass insertion in redis - redis

I am working on centos 6. I want to perform mass insertion into redis cache from mySQL table that has more than 10 million records. I have already read about redis protocol and its format. I am able to copy table data into redis protocol format in text file. But when I try to execute pipe command an error is coming.
Error reading from the server: Connection reset by peer
or
ERR Protocol error: expected '$', got '3'
I am using following command:
cat insert.txt | redis-cli --pipe
insert.txt contains data in this format:
*3
$3
SET
$2
K1
$2
V1
If possible,please tell me format for multiple commands in this text file as in above example, this file only contains one set command. I will be thankful if you will give me example of text file which has at least two commands in it.
I have also tried file data in this format.
"*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n"
It gives the following error:
ERR unknown command '*3 $3 SET $3 key $5 value '

Right, so it's really important that you send the data with CRLF line separators -- not the string \r\n but actually those characters. This means no regular unix line endings, which is just LF, or \n.
How you insert those is depending on the editor you use, but the easiest way to work with what you have would be to have each symbol on it's own line:
*3
$3
SET
$2
K1
$2
V2
And use sed:
cat insert.txt | sed 's/$/\r/g' | redis-cli --pipe
Should do the trick.
Then, in order to send many commands in one go (pipelining), you just add that to the end of your file, like so:
*3
$3
SET
$2
K1
$2
V2
*3
$3
SET
$2
K2
$2
V3

Related

how to obtain the service of a computer in a csv format

I want to get the service of a computer using :
systemctl list-unit-files --no-legend -t service | awk '{ sub(/\.service$/, "", $1); print "SERVICE:" $1 "|" $2 }'
but it gives me the Following result :
SERVICE: syslog|enable
(for example)
and i would love to get this :
"SERVICE","syslog","enable",
I tried several syntax but it generates errors because of the ' "" '.
i even tried to put some other printf '","' but it brake the awk sentence because of the the ' at the beginning.
i tried to use sed and grep but it modifies the
systemctl list-unit-files --no-legend -t service
directly which breaks my script.
How can i parse the result in a CSV format ?
Thanks a lot for your help.
Assuming you intend to remove .service from the service names, you can try this sed
$ systemctl list-unit-files --no-legend -t service | sed 's/\([^.]*\)\.[^[:space:]]*[[:space:]]\+\([^[:space:]]*\)/"SERVICE","\1","\2",/'
"SERVICE","syslog","enabled",
\([^.]*\) - Match until the next occurance of a period. As it is within a parenthesis, group it to be returned at a later time with back reference \1
\.[^[:space:]]*[[:space:]]\+ - As this part is not within a parenthesis, it will be excluded. This will exclude from the period . up to the the space.
\([^[:space:]]*\) - Group the rest up to the next space, it is grouped and will be returned with back reference \2
Suggesting to try this solution:
systemctl list-unit-files --no-legend -t service| awk '{printf("\"SERVICE\",\"%s\",\"%s\"\n",$1,$2)}'

Bash script process csv file line by line while updateing $6 with different value but keeping other values unchanged

I am beginner at bash scripting and I have been trying to fix this for more than 8 hours.
I have searched on StackOwerflow and tried the answers to fit my needs, but without success.
I want to use bash script to change csv file's date value to current date.
I am using a dummy .csv file ( http://eforexcel.com/wp/wp-content/uploads/2017/07/100-Sales-Records.zip ) and I want to change the 6th value (date) to the current date.
What I have been doing so far:
I have created one line csv to test the script
cat oneline.csv:
Australia and Oceania,Tuvalu,Baby Food,Offline,H,5/28/2010,669165933,6/27/2010,9925,255.28,159.42,2533654.00,1582243.50,951410.50
then I have tested the one line script:
echo `cat oneline.csv | awk -F, '{ print $1"," $2"," $3"," $4"," $5","}'` `date` `cat oneline.csv |awk -F, '{print $7"," $8"," $9"," $10"," $11"," $12"," $13"," $14"\n"}'
then I have this code for the whole 100 line files in source.sh:
#I want to change 6th value for every line of source.csv to current date and keep the rest and export it to output.csv
while read
do
part1=$(`cat source.csv | awk -F, '{ print $1"," $2"," $3"," $4"," $5","}'`)
datum=$(`date`)
part2=$(`cat source.csv |awk -F, '{print $7"," $8"," $9"," $10"," $11"," $12"," $13"," $14"\n"}'`)
echo `$part1 $datum $part2`
done
and I expect to run the command like ./source.sh > output.csv
What I want for the full 100 lines file is to have result like:
Food,Offline,H,Thu Jan 17 06:34:03 EST 2019,669165933,6/27/2010,9925,255.28,159.42,2533654.00,1582243.50,951410.50
Could you guide me how to change the code to get the result?
Refactor everything to a single Awk script; that also avoids the echo in backticks.
awk -v datum="$(date)" -F , 'BEGIN { OFS=FS }
{ $6 = datum } 1' source.csv >output.csv
Briefly, we split on comma (-F ,) and replace the value of the sixth field with the value of the variable we passed in with -v. OFS=FS sets the output field separator to the input field separator (comma). Then the 1 means "print unconditionally".
Generally speaking, you should probably avoid while read.
Tangentially, your quoting looks wacky; you don't want backticks around $part1 unless it is a command you want the shell to run (which in turn is probably a bad idea in itself). Also, backticks have long been deprecated in favor of $(command) syntax which is more legible and offers some syntactic advantages.

awk command - define the size of a word

I'm learning AWK and trying to count the number of sessions to a particular destination.
Using this command:
awk '{print $9}' traffic-log-cust.txt | sort | uniq -c
and I am getting the below output.
#awk '{print $9}' traffic-log-cust.txt | sort | uniq -c
1
1 10.10.17.72/38403->157.55.235.140/40046
1 10.10.17.72/38403->157.55.235.146/40006
1 10.10.17.72/38403->157.55.235.148/40039
1 10.10.17.72/38403->157.55.235.159/40019
1 10.10.17.72/38403->157.55.235.160/40019
1 10.10.17.72/38403->157.55.56.156/40046
1 10.10.17.72/38403->157.55.56.174/40018
1 10.10.17.72/38403->64.4.23.156/40017
1 10.10.17.72/38403->64.4.23.164/40011
1 10.10.17.72/38403->64.4.23.166/40053
1 10.10.17.72/38403->65.55.223.16/40003
1 10.10.17.72/38403->65.55.223.44/40002
#
and I believe word 9 have no space and contains destination IP as well.
I would like to know how I can count the sessions based on destination IP's.
Thanks in Advance.
I am going to guess that you are having issues deciding how big each field is. (Your question is unclear.) I would argue you don't need to; just split each row into 2 fields and deal with the second field.
With awk, you specify what the delimiter is with the -F option, and since the greater-than sign (>) is meaningful in many shells, you have to escape it somehow. In Linux, you can use a backslash to do so.
Since you are using awk, you don't need sort and uniq; associative arrays can be used.
Assuming that you are NOT ignoring the ports:
awk -F\> '{dest_ips[$2]++}
END {
for (ip in dest_ips) {
printf "%s: %d\n", ip, dest_ips[ip]
}
}' traffic-log-cust.txt
If you are ignoring the ports, you have to parse that second field first (perhaps using split()).

When columns are seperated by multiple spaces, how to change a column and keep the spaces?

I have a file where the columns are seperated by multiple spaces.
How can I set, let's say, the second column, and keep the spaces on the line?
For example, in postgres' pg_hba.conf there is a line
local all all peer
How can I change 'peer' to 'trust' and keep the spaces as they are on the line? When I let awk do $4="trust" there will be one space between the columns. In principle that is OK, but it makes the file harder to read, because of the position of headers above the line in the file.
here is one approach
$ echo "local all all peer" |
awk 'gsub("peer","trust")'
local all all trust
here is another approach if you're changing the field not by value but by index. For example change third field to "trust" this time
$ echo "local all all peer" |
awk -v RS='[^ ]+' -v ORS="" '{print $0 (NR==3?"trust":RT)}'
local all trust peer
You need to use a regexp and operate on the whole record by specifying how many initial fields (\S+) plus separators (\s+) to skip before making your change, e.g. with GNU awk for gensub() and \s/\S:
$ awk '{$0=gensub(/(\s*(\S+\s+){3})\S+/,"\\1trust",1)}1' file
local all all trust
Changing field 3 is a harder/better test of any potential solution since it's contents (all) appear earlier in the line too:
$ awk '{$0=gensub(/(\s*(\S+\s+){2})\S+/,"\\1trust",1)}1' file
local all trust peer
The regexp starts with \s* since by default awk ignores leading white space but that's not actually needed for your specific data.
With other awks you'd do:
$ awk 'match($0,/[[:space:]]*([^[:space:]]+[[:space:]]+){2}/) {
head = substr($0,1,RLENGTH)
tail = substr($0,RLENGTH+1)
sub(/[^[:space:]]+/,"trust",tail)
print head tail
}' file
local all trust peer
You really can't. If the amount of space varies between fields. Changing a field value will lead to rebuilding the record which means that the field separator FS is replaced by output field separator OFS.
You can try your luck with regex, though.

Understanding awk command

I need help in understanding the following awk command
awk -F "<name>|</name>|<machine>|</machine>" '{if($0 ~ "<name>" && $0 ~ "</name>") nm=$2;else if($0 ~ "<machine>" && $0 ~ "</machine>") {print nm,$2}}' config.xml
This command is giving me the output of weblogic managed servers and their respective hosts in the following format.
managed_server1 host1
managed_server2 host2
managed_server3 host3
It's not particularly well written script but extracts values from lines in this format (and in this order)
<name>xxx</name>
<machine>yyy</machine>
and outputs
xxx yyy
Sets the field delimiter to open/close xml tags and if the first pair is present in the line sets a variable to the value from the second field and when the second pair is present print the previous value set and current second field.