For a port-knocking scheme, I'm wondering how to make the iptables recent module temporarily (for just a few seconds) list-name matched source addresses. My intuition tells me that I need the --set function of the recent module to accept the --seconds option, making the list assignment temporary, but all I can see is the way to assign the list name to an address permanently and have another rule remove the list-name assignment only upon receipt of some later packet. The reason that doesn't fit into my mental concept is because the removal (--reap or --remove, I suppose) of the address from the list will only occur upon reception of a future packet, whereas my intuition wants the address removed at a certain time expiration regardless of whether any packets arrive to trigger said name removal. The only way I can see to do something vaguely similar to this is very non-intuitive to me, and therefore suspicious to me that I'm missing something about how it all works: I would need a recent module rule and --rcheck option to ensure the listed packet's name matches and that it had gotten assigned within the previous x seconds and remove it with a jump destination, the rule in that jump destination would be to assign the next list name to the source address. In the meantime, the length of the lists just keeps growing (don't they?), filling up with stray source addresses that never completed the knock sequence[s]. What a simple solution it would be for the recent module to accept the --seconds option with --set! Can anyone help me help me see this more clearly?
( I've looked at other knocking solutions using iptables, but they are limited to only using each port-protocol combination for one knock in the sequence, while a good knocking solution should, IMHO, allow for the same port-protocol combo to be used as many times in the knock sequence as the user wants it to be used. knockd had that same limitation, as well as exhibiting terrible non-robust operation. I tried to obtain the pknock module for iptables, but it appears that not all its components exist [specifically two shell scripts referred to in the documentation, knock.sh and knock-orig.sh, supposedly "found in doc/pknock/util", wherever that is...certainly not SF, Github, nor anywhere else I could see], making me very suspicious of using it.)
EDIT: I'm seeing that the ruleset needs to be even more complex than described - the knock steps numbered two and above all need to match the packet by name first, then jump it to their own chain that removes the name, determines whether the correct timing and port-protocol matches, then jump it to yet another chain to rename it, or don't jump it if timing or knock is off which falls into a drop rule. WHEW
My initial solution is shown below. The knocks in this example just happen to be unique, but non-unique knocks will function fine as well. As you can see, I make every knocking packet reap every list because I don't know if the lists self-limit the length of time they'll keep entries otherwise. It seems like the only way to ensure that no list can get too long.
$--> iptables -wnvL
Chain INPUT (policy DROP)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- $internal_net_interface * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate RELATED,ESTABLISHED /* extract ssh for knock testg frm private side in ssh */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: authorized side: source mask: 255.255.255.255 ctstate NEW /* 1-packet pass: 1 chance to establish or then knock higher */
0 0 knockerstest all -- $internal_net_interface * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 200 name: knocker side: source mask: 255.255.255.255 /* for knock capability */
0 0 knockstage1 tcp -- $internal_net_interface * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1 flags:0x17/0x02 recent: SET name: knocker side: source mask: 255.255.255.255 /* for knock capability, 1st port */
0 0 knockers all -- $external_net_interface * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 200 name: knocker side: source mask: 255.255.255.255 ctstate NEW /* for port knock capability */
0 0 knockstage1 tcp -- $external_net_interface * 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1 flags:0x17/0x02 recent: SET name: knocker side: source mask: 255.255.255.255 /* for port knock capability, 1st port */
Chain knockerreap (10 references)
pkts bytes target prot opt in out source destination
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 60 reap name: knocker side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage1 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage2 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage3 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage4 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage5 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage6 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage7 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage8 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage1 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage2 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage3 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage4 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage5 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage6 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage7 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage8 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage1 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage2 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage3 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage4 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage5 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage6 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage7 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage8 side: source mask: 255.255.255.255
Chain knockers (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0 ! ctstate NEW /* for port knock capability */
0 0 knockersort all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* for port knock capability */
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix "pktfail:knock|late|ctstate "
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 60 reap name: knocker side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knocker side: source mask: 255.255.255.255
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockersort (2 references)
pkts bytes target prot opt in out source destination
0 0 knockstage2 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage2 side: source mask: 255.255.255.255 /* knock to stage 2 successful */
0 0 knockstage3 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage3 side: source mask: 255.255.255.255 /* knock to stage 3 successful */
0 0 knockstage4 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage4 side: source mask: 255.255.255.255 /* knock to stage 4 successful */
0 0 knockstage5 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage5 side: source mask: 255.255.255.255 /* knock to stage 5 successful */
0 0 knockstage6 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage6 side: source mask: 255.255.255.255 /* knock to stage 6 successful */
0 0 knockstage7 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:7 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage7 side: source mask: 255.255.255.255 /* knock to stage 7 successful */
Chain knockerstest (1 references)
pkts bytes target prot opt in out source destination
0 0 knockersort all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix "knockertest fail "
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 60 reap name: knocker side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knocker side: source mask: 255.255.255.255
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage1 (2 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage2 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage1 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage2 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage3 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage2 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage3 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage4 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage3 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage4 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage5 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage4 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage5 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage6 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage5 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage6 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage7 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage6 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage7 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: authorized side: source mask: 255.255.255.255 /* allows time-limited access */ LOG flags 0 level 4 prefix "knock full success "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Related
I'm getting the following output when I do a tcpdump
2017-07-26 00:00:00.062388 IP (tos 0x0, ttl 64, id 55135, offset 0, flags [none], proto UDP (17), length 78)
192.168.1.69.26818 > 192.168.1.5.53: 8603+ AAAA? pkcba.ad.roy.com. (50)
2017-07-26 00:00:00.062859 IP (tos 0x0, ttl 128, id 7617, offset 0, flags [none], proto UDP (17), length 128)
192.168.1.5.53 > 192.168.1.69.26818: 8603* 0/1/0 (100)
What do the numbers in the brackets (50) & (100) mean?
In this particular case they're the length of the DNS payload within the UDP packet (note that they're 28 bytes less than the total packet length reported in the lines above - 20 bytes for a standard IP header without options and 8 bytes for UDP)
I have installed ssl certificate on nginx using certbot, I am able to access https://URL from browser only if i stop iptables, So I checked if 443 port was open
I created rule in iptables to open 80 and 443 ports:
iptables -A INPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate ESTABLISHED -j ACCEPT
service iptables save
service iptables restart
and checked by nginx:
$netstat -anltp
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 27432/nginx
...........
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 27432/nginx
in mysite.conf file
server {
listen 443 ssl;
}
and nginx
$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
and iptables..
iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
133 9875 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
3543 271K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:587
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:110
1496 131K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:143
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:110
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:143
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:993
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:995
2274 3696K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
1 56 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
45 2700 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:25
53 3288 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
180 32695 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 state NEW
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:25
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:465
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:110
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:995
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:143
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:993
0 0 DROP all -- * * 58.218.204.189 0.0.0.0/0
0 0 DROP all -- * * 58.218.204.189 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:53
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:53
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 ctstate NEW,ESTABLISHED
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443 ctstate NEW,ESTABLISHED
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 6686 packets, 10M bytes)
pkts bytes target prot opt in out source destination
133 9875 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp spt:443 ctstate ESTABLISHED
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443 ctstate ESTABLISHED
but still I am getting connection refused in browser if iptables service is running, Nothing in nginx logs as it seems firewall issue.
nginx version: nginx/1.12.0
Centos 6, uname -a Linux server.name 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Any help? thanks
I had issue with iptables I should have used -I instead of -A and SSL worked, as mentioned by Leader at Let's Encrypt Community
Here is my text file
No.,Time,Source,Destination,Protocol,Length,Info,SrcPort,DstPort,src_dst_pair
1401,0.397114,145.95.225.186,210.218.218.164,UDP,100,Source port: hsrp Destination port: hsrp,hsrp,1985,"('145.95.225.186', '210.218.218.164')"
8999,3.229111,145.95.225.186,210.218.218.164,UDP,100,Source port: hsrp Destination port: hsrp,hsrp,1985,"('145.95.225.186', '210.218.218.164')"
18504,5.877098,145.95.225.186,210.218.218.164,UDP,100,Source port: hsrp Destination port: hsrp,hsrp,1985,"('145.95.225.186', '210.218.218.164')"
23755,8.695843,145.95.225.186,210.218.218.164,UDP,100,Source port: hsrp Destination port: hsrp,hsrp,1985,"('145.95.225.186', '210.218.218.164')"
28027,11.24121,145.95.225.186,210.218.218.164,UDP,100,Source port: hsrp Destination port: hsrp,hsrp,1985,"('145.95.225.186', '210.218.218.164')"
33304,14.117213,145.95.225.186,210.218.218.164,UDP,100,Source port: hsrp Destination port: hsrp,hsrp,1985,"('145.95.225.186', '210.218.218.164')"
700443,222.305789,145.95.41.251,145.95.81.118,UDP,50,Source port: 36477 Destination port: snmp,36477,161,"('145.95.41.251', '145.95.81.118')"
700495,222.351933,145.95.41.251,145.95.81.118,UDP,50,Source port: 36477 Destination port: snmp,36477,161,"('145.95.41.251', '145.95.81.118')"
700496,222.352372,145.95.41.251,145.95.81.118,UDP,50,Source port: 36477 Destination port: snmp,36477,161,"('145.95.41.251', '145.95.81.118')"
708982,225.913385,145.95.41.251,145.95.81.118,UDP,50,Source port: 36477 Destination port: snmp,36477,161,"('145.95.41.251', '145.95.81.118')"
709797,226.130847,145.95.41.251,145.95.81.118,UDP,50,Source port: 36477 Destination port: snmp,36477,161,"('145.95.41.251', '145.95.81.118')"
710340,226.372421,145.95.41.251,145.95.81.118,UDP,50,Source port: 36477 Destination port: snmp,36477,161,"('145.95.41.251', '145.95.81.118')"
I want to group the data based on Source and Destination, then :
accumulate the Length column within the group
find the diff between the max and min Time within the group
I got the results but I need to format it in the way as I have shown in the expected output. I also want to know if there is better way to do this.
Below is my try
import pandas as pd
data = pd.read_csv('simple_udp.csv')
# getting the accumulated sum for the group
length = data.groupby(['Source','Destination']).Length.sum()
# getting the difference in time between the max and min in the group
time = data.groupby(['Source','Destination']).Time.max() - data.groupby(['Source','Destination']).Time.min()
# This is were I have problem. How can i format the result so that
# I can get the expected output(shown below)
print length, time
Expected output
Source Destination Length Time
145.95.225.186 210.218.218.164 600 13.720099
145.95.41.251 145.95.81.118 300 4.066632
Use agg
data.groupby(['Source','Destination']).agg({'Length': 'sum', 'Time': lambda x: x.max() - x.min()})
My first guess would be
import pandas as pd
data = pd.read_csv('simple_udp.csv')
# Creating a DataFramGroupBy object
group = data.groupby(['Source','Destination'])
df_length = g['Length'].sum()
df_time = g['Time'].max() - g['Time'].min()
df = pd.DataFrame([df_length,df_time])
or if you want to have it on less lines, but also less readable use the agg method on group
I'm trying to use curl in SAS to download data from an api. I have a Consumer key,Consumer secret,OAuth Token, & OAuth Token Secret. It appears that I can connect to the api but I get an error stating that "HTTP Transport: Couldn't determine the content length". Does anyone have any insights or thoughts? Thanks so much.
Code:
dm 'clear log';
options;
%let consumer_key =;
%let consumer_secret=;
%let oauth_token=;
%let oauth_token_secret=;
filename curl pipe "curl -X POST -k https://api.tradeking.com/v1/market/ext/quotes.xml?symbols=aapl --verbose --header
'Authorization:OAuth oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
oauth_signature_method=HMAC-SHA1,oauth_consumer_key={&consumer_key.},oauth_consumer_key_secret={&consumer_secret.},
oauth_token={&oauth_token.},oauth_token_secret={&oauth_token_secret.}";
data _null_;
infile curl lrecl=32767;
input;
put _infile_;
run;
Log:
dm 'clear log';
options;
636
637 %let consumer_key =;
638 %let consumer_secret=;
639 %let oauth_token=;
640 %let oauth_token_secret=;
641
642 filename curl pipe "curl -X POST -k
642! https://api.tradeking.com/v1/market/ext/quotes.xml?symbols=aapl --verbose --header
643 'Authorization:OAuth
643! oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
WARNING: The quoted string currently being processed has become more than 262 characters long.
You might have unbalanced quotation marks.
644 oauth_signature_method=HMAC-SHA1,oauth_consumer_key={&consumer_key.},oauth_consumer_ke
644! y_secret={&consumer_secret.},
645 oauth_token={&oauth_token.},oauth_token_secret={&oauth_token_secret.}";
646
647 data _null_;
648 infile curl lrecl=32767;
649 input;
650 put _infile_;
651 run;
NOTE: The infile CURL is:
Unnamed Pipe Access Device,
PROCESS=curl -X POST -k https://api.tradeking.com/v1/market/ext/quotes.xml?symbols=aapl
--verbose --header 'Authorization:OAuth
oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
oauth_signature_method=HMAC-SHA1,oauth_consumer_key={
JXpV},oauth_consumer_key_secret={},
oauth_token={},oauth_token_secret={},
RECFM=V,LRECL=32767
Fault Name: HttpRequestReceiveError
Error Type: Default
Description: Http request received failed
Root Cause Code: -19013
Root Cause : HTTP Transport: Couldn't determine the content length
Binding State: CLIENT_CONNECTION_ESTABLISHED
Service: null
Endpoint: null
Operation (Client):
--_curl_--oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
--_curl_--oauth_token=,oauth_token_secret=Q
Stderr output:
* Adding handle: conn: 0x1ea8850
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1ea8850) send_pipe: 1, recv_pipe: 0
Total Received Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to
connect() to api.tradeking.com port 443 (#0)
* Trying 206.132.7.9...
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to
api.tradeking.com (206.132.7.9) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* SSLv3, TLS
handshake, Finished (20):
{ [data not shown]
* SSL connection using RC4-MD5
* Server certificate:
* subject: OU=Domain Control Validated; OU=COMODO SSL Wildcard; CN=*.tradeking.com
* start date: 2013-06-09 00:00:00 GMT
* expire date: 2016-06-08 23:59:59 GMT
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO SSL CA
* SSL certificate verify result: self signed certificate in certificate chain (19),
continuing anyway.
> POST /v1/market/ext/quotes.xml?symbols=aapl HTTP/1.1
> User-Agent: curl/7.33.0
> Host: api.tradeking.com
> Accept: */*
> 'Authorization:OAuth
>
< HTTP/1.1 411 Length Required
< Connection: close
< Content-Length: 284
<
{ [data not shown]
100 284 100 284 0 0 178 0 0:00:01 0:00:01 --:--:-- 178
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
* Rebuilt URL to:
oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,/
* Adding handle: conn: 0x1f00560
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 1 (0x1f00560) send_pipe: 1, recv_pipe: 0
* Could not resolve host:
oauth_nonce=,oauth_timestamp=1359019570,oauth_version=1.0,
* Closing connection 1
curl: (6) Could not resolve host:
oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
[1/3]:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
--> <stdout>
* Rebuilt URL to:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
/
* Adding handle: conn: 0x1ef10e0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 2 (0x1ef10e0) send_pipe: 1, recv_pipe: 0
* Could not resolve host:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
* Closing connection 2
curl: (6) Could not resolve host:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
[1/2]:
oauth_token=,oauth_token_secret=
--> <stdout>
* Rebuilt URL to:
oauth_token=,oauth_token_secret=
/
* Adding handle: conn: 0x1ef56c0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 3 (0x1ef56c0) send_pipe: 1, recv_pipe: 0
* Could not resolve host:
oauth_token=,oauth_token_secret=
* Closing connection 3
curl: (6) Could not resolve host:
oauth_token=,oauth_token_secret=
NOTE: 12 records were read from the infile CURL.
The minimum record length was 0.
The maximum record length was 174.
NOTE: DATA statement used (Total process time):
real time 1.84 seconds
cpu time 0.23 seconds
The first thing to do is to try running your cURL statement from the command line. If you get a valid response it's a SAS issue, otherwise it's an issue with your cURL request.
Have you tried adding the -k option to your cURL statement seeing as you're making an https request?
Oh also, do any of the macro variables contain values that include ampersands, percent symbols, double quotes, or single quotes? These could all cause issues - especially ampersands as they need to be further masked with a caret symbol ^.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm Trying to write a Cisco Command Line Parser to have an automated Graphical User Interface replacement for the Cisco console output. I have been able to get the ping time using Regular Expressions from a ping output and graph it, but am now stuck with more detailed out put of other commands like "Show interfaces" command,
any ideas how I can parse the Show Interface command output and extract all the useful info which i need?
Here is a "Show Interfaces" out put example:
FastEthernet0/0 is up, line protocol is up
Hardware is MV96340 Ethernet, address is 0018.189d.1df0 (bia 0018.189d.1df0)
Description: IP+ connection
Internet address is 164.128.251.50/24
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/3718/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 6 packets/sec
5 minute output rate 3000 bits/sec, 10 packets/sec
152817108 packets input, 1043050554 bytes
Received 77347880 broadcasts (67140888 IP multicasts)
0 runts, 0 giants, 3351 throttles
381823 input errors, 0 CRC, 0 frame, 0 overrun, 381823 ignored
0 watchdog
0 input packets with dribble condition detected
--More-- 99065802 packets output, 440637782 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
300246 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
FastEthernet0/1 is administratively down, line protocol is down
Hardware is MV96340 Ethernet, address is 0018.189d.1df1 (bia 0018.189d.1df1)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto Speed, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes
Received 0 broadcasts (0 IP multicasts)
--More-- 0 runts, 0 giants, 0 throttles
--More-- 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Tunnel0 is up, line protocol is up
Hardware is Tunnel
Interface is unnumbered. Using address of FastEthernet0/0 (164.128.251.50)
MTU 17912 bytes, BW 100 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel source 164.128.251.50 (FastEthernet0/0), destination 164.128.32.1
Tunnel Subblocks:
src-track:
Tunnel0 source tracking subblock associated with FastEthernet0/0
Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface
Tunnel protocol/transport PIM/IPv4
--More-- Tunnel TOS/Traffic Class 0xC0, Tunnel TTL 255
--More-- Tunnel transport MTU 1472 bytes
Tunnel is transmit only
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Last input never, output 28w1d, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
44 packets output, 2464 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
Virtual-Access1 is up, line protocol is up
Hardware is Virtual Access interface
Description: Internally created by SSLVPN context TEST
MTU 1406 bytes, BW 100000 Kbit/sec, DLY 100000 usec,
--More-- reliability 255/255, txload 1/255, rxload 1/255
--More-- Encapsulation SSL
Internal vaccess
Vaccess status 0x0, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 5 seconds on reset
Last input never, output never, output hang never
Last clearing of "show interface" counters 29w5d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Interface_Long_Split = Regex.Split(Result_Long, "(POS[0-9]/[0-9]/[0-9])|(POS[0-9]/[0-9])|(GigabitEthernet[0-9]/[0-9])|(FastEthernet[0-9]/[0-9])")
Dim count As Integer = 0
For i = 0 To Interface_Long_Split.Length
If Regex.IsMatch(Interface_Long_Split(i), "(POS[0-9]/[0-9]/[0-9])|(POS[0-9]/[0-9])|(GigabitEthernet[0-9]/[0-9])|(FastEthernet[0-9]/[0-9])") = True Then
ReDim Preserve Interfaces_List(count)
Interfaces_List(count) = Interface_Long_Split(i)
count = count + 1
End If
imho you are probably on a hiding to nothing.
you could try parsing those complex outputs a line at a time rather than as one big blob.