I have an Apache server log and am trying to determine what IP address has generated the most traffic. I've already managed to get it formatted so its just the IPs and their traffic in bytes:
xxx.xxx.xxx.xxx 915925
yyy.yyy.yyy.yyy 1193
zzz.zzz.zzz.zzz 2356
So now I'm looking for a method to combine and add the bytes of identical IPs and then just find the top value.
Any ideas?
If you have the ip and traffic bytes in a file use the following to get the work done.
cat file | perl -ane '$h{ $F[0] } += $F[1]; END { for ( sort keys %h ) { printf qq[%s %d\n], $_, $h{ $_ } } }' | sort -k2 -n -r
awk '{A[$1]+=$2;next}END{for(i in A){print i,A[i]}}' file | sort -k2 -n -r
Related
There are many awk attacks to my server. I have tried to block them , but too much of them .
is there a way to block them one time?
I use this command :
netstat -an|awk -F: '{print $2}'|sort|uniq -c|sort -nr|head
show the result
[root#local ~]# netstat -an|awk -F: '{print $2}'|sort|uniq -c|sort -nr|head
1080 80 107.189.8.33
864 80 185.129.61.5
485 80 23.154.177.11
386 80 183.245.24.27
318 80 185.243.218.32
309 80 185.220.101.2
276 80 61.153.251.150
259 80 59.148.106.164
235 80 185.175.119.113
And after list a ip , I will find the connection ips to 80 port more than 100 ones . and block them .
firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address="107.189.8.0/24" drop'
Anyway to make a .sh file , to find out the awk ips which more than 200 connections , and add them to the droplist of the firewall?
in this case , need to exclude 127.0.0.1 and our own ips .
hope anyone can help thanks.
I have tried to output the ips with problem using this code .
netstat -an|awk -F: '{print $2}'|sort|uniq -c|sort -nr|head > ccips.txt
after this , I use :
awk '{sub("IP:", "", $3); print $3}' /root/ccips.txt | xargs -n1 -I{} firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address={} drop'
this can block all the ips with attack .
I just don't know how to import this to a .sh , which can be do this in one command .
I THINK what you're trying to do is create a file like this of IPs you don't want to block:
$ cat allowedIPs
127.0.0.1
whatever...
and then have a script like this to block all IPs not in that file connecting to port 80 (untested and guessing at what the netstat -an output looks like by reading your code):
$ cat blockIPs
#!/usr/bin/env bash
netstat -an |
awk -F: '
NR == FNR {
allowedIPs[$1]
next
}
{
split($2,portIP," ")
port = portIP[1]
ip = portIP[2]
}
(port == 80) && !(ip in allowedIPs) && !seen[ip]++ {
print ip
}
' allowedIPs - |
xargs -n1 -I{} firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address={} drop'
Let's say.
I have one file with the name of the computer and some other information.
E.g.
Computer1
There's another file with the ip address and some other information.
192.168.100.2
I have 2 greps for example:
grep -i computer /etc/hosts
grep -i ips /etc/hosts
They give me answers like
Computer1
19.168.100.2
Well, I would like to get a file with headers and the information organized as this:
Name
Ip
oser1313
19.168.100.1
I'm quite lost I have no idea how could I format this I usually copy-paste it on Excel but I don't want to do it anymore and since I have to do this on several computers from a server It would be great if I can format it.
Just do something like this:
awk '
{ lc = tolower($0) }
lc ~ /computer/ { name = $0 }
lc ~ /ips/ { ip = $0 }
END {
print "Name", "Ip"
print name, ip
}
' /etc/hosts
The above is untested since you didn't provide a sample input file to test with and it's just mimicing what your grep commands do but there may be a better way to do it if we knew what your input looked like.
I suppose that your two files have the same number of lines and that line numbers match between one file and the other: if oser1313 is line n in the output of grep from /etc/hosts then same for 19.168.100.1 in /etc/hosts.
So it turns pretty simple as bash script:
grep -i computer /etc/hosts > part1.dat
grep -i ips /etc/hosts > part2.dat
echo "Name,IP" > out.dat
paste -d"," part1.dat part2.dat >> out.dat
rm part1.dat part2.dat
Or a oneliner, as suggested in comments:
printf "Name,IP\n$(grep -i computer /etc/hosts),$(grep -i ips /etc/hosts)\n" > out.dat
I often use this to check website access logs by IP address. The problem is that it only includes IPV4 and not IPV6.
Any idea what regex I can use so that it includes (or runs a separate) command for IPV6?
cat access.log | sed -e 's/^\([[:digit:]\.]*\).*"\(.*\)"$/\1 \2/' | sort -n | uniq -c | sort -nr | head -50
Matching IP addresses via regular expressions can be tricky - yours matches lots of things that aren't valid IPv4 addresses, like 100000.55, for example.
There's a perl module, Regexp::Common that provides well tested regular expressions for matching all sorts of things, including both IPv4 and IPv6 addresses. If you install it (The Ubuntu package is libregexp-common-perl), you can replace the sed part of that pipeline with
perl -MRegexp::Common=net -lne '/^($RE{net}{IPv4}|$RE{net}{IPv6}).*"(.*)"$/ && print "$1 $2"'
to match both address families.
Is it possible to get a list of all vhosts, IPs in a server?
So I don't have to check every file at /etc/apache2/sites-enabled/ to know what is it on every server.
I need to get something like this:
+--------------+-----------------+---------------------+
| Server Alias | IP Address | Document Root |
+--------------+-----------------+---------------------+
| mysite.test | 192.168.0.12:80 | /home/User/mysite |
+--------------+-----------------+---------------------+
You could use the following command
# apache2ctl -S
And with some awk magic have it in the desired format
To be exact try the following
# apachectl -S 2>&1 | perl -ne 'm#.*port\s+([0-9]+)\s+\w+\s+(\S+)\s+\((.+):.*# && do { print "$2:$1\n\t$3\n"; $root = qx{grep DocumentRoot $3}; $root =~ s/^\s+//; print "\t$root\n" };'
I need load ip list from file, scan it, and create output format such as ip:port. I tried this:
nmap -iL mylistwithip.txt -p 80,21 -oG -PS 80,21 | awk '/open/{print $2}' >` output.txt
but it gives me only "open" and that's all.
While I need only opened ports from list of IP addresses, for example:
192.168.2.1
192.168.2.2
192.168.2.3
after scan ports, sample output.txt:
192.168.2.1:80
192.168.2.1:21
192.168.2.3:80
(only scanned ip addresses with opened ports)
Try this awk oneliner:
nmap -Pn -oG - 192.168.1.1 | awk '/open/{ s = $2; for (i = 5; i <= NF-4; i++) s = substr($i,1,length($i)-4) "\n"; split(s, a, "/"); print $2 ":" a[1]}'
try one more solution with single awk only.
nmap -vv -iL file | awk -F'[ /]' '/Discovered open port/{print $NF":"$4}'
Quick and ugly hack to achieve that:
nmap -vv -iL mylistwithip.txt | grep "Discovered open port" | awk {'print $6":"$4'} | awk -F/ {'print $1'} > output.txt
With -vv output includes lines like
Discovered open port 22/tcp on 192.168.2.1
Discovered open port 80/tcp on 192.168.2.1
Discovered open port 22/tcp on 192.168.2.107
Discovered open port 80/tcp on 192.168.2.107
First awk selects "ip address" and "port number/protocol" fields, and second cuts off "/protocol".
This will probably break in some future update of nmap. Using -sG (greppable output) would be a better idea.