Get IP addresses from PCAP file in scapy - optimization

Is there a smart and fast way to get all IP addresses from a PCAP file?
I need only (destination address, source address) tuples.
Currently I'm using Scapy's rdpcap function like this:
from scapy.all import *
pcap = rdpcap('file.pcap')
ips = set([(p[IP].fields['src'], p[IP].fields['dst']) for p in pcap if p.haslayer(IP) == 1])
But it takes about two minutes on my machine to parse a 70MB PCAP file with 370 unique extracted entries...

The "best" way to do what I think (based on the code you provided, I suppose you want the couples (IP source address, IP destination address) rather than IP packets) you want is :
>>> set((p[IP].src, p[IP].dst) for p in PcapReader('file.pcap') if IP in p)
You don't waste memory with the whole PCAP file, the set object is built packet by packet.
If it's not fast enough, you can instruct Scapy not to dissect packets after the IP layer since you don't use the IP payload:
>>> IP.payload_guess = []
And then run the previous command.

Related

what does 'ip -4 rule add table main suppress_prefixlength 0' meaning?

ip -4 rule add table main suppress_prefixlength 0
This command appears in the process of connecting to wireguard vpn.
It's perhaps easiest to explain this in terms of three potential route considerations for wireguard.
When you create a wireguard interface, you obviously want packets going to the IP ranges you want to access through wireguard to go through that interface. So if you're accessing, say, 10.2.0.0/16 via wireguard on wg0, you could just do ip route add 10.2.0.0/16 dev wg0.
That works if you're just connecting to a private network. But what if you want everything routed through wireguard, for it to be your default route? That poses a complexity, because when you want everything routed through a wireguard interface, you obviously don't want wireguard's own packets routed through that interface; otherwise, they'd never go anywhere at all. Try ip route add 0.0.0.0/0 dev wg0 and now nothing will work: packets transporting wg0 would try to go through wg0. You could add specific routes to each wireguard peer, but you might have many wireguard peers, so that would be inconvenient.
Instead, wg-quick uses a firewall mark (fwmark) so that routing can recognize packets for that interface, and handle them differently. By setting something like wg setconf wg0 fwmark 51820, wireguard can then add rules to treat wireguard packets and non-wireguard packets differently. Then, it creates a different routing table (eg, 51820, which you can see by ip route list table 51820), which non-wireguard packets go through, and routes them all through wg0, while wireguard packets go through the main table (what you see with ip route list). It uses a rule (not from all fwmark 0xca6c lookup 51820) to divert anything that isn't a wireguard packet to the 51820 table (you can see these rules with ip rule or ip -6 rule).
So now, why the from all lookup main suppress_prefixlength 0? You don't actually need this in many cases: wireguard packets will be routed on the main table, and non-wireguard packets will be routed on the table wg-quick creates. But what if your main routing table isn't just simply a default route? What if you've added other, more specific, routes, maybe for some private-address space VPN (the reason why I just had to figure out what this command did), or maybe to get to some of the peers in the first place: maybe they're on different interfaces?
To cover these situations, this third command adds a rule that first, looks up what the route for any packet (from all) would be on the main table (lookup main). Then, it sees how specific that route is, eg, what its prefix length is. If it's 0 (ie, a default route, 0.0.0.0/0 or ::/0), it suppresses that route (suppress_prefixlength 0, which suppresses anything with a prefix length of its argument or less), and continues looking at the next rules. If it's more than 0 (eg, 10.1.0.0/16), then it uses that route.
Thus, you end up with rules that look like these from ip rule list (annoyingly, ip rule shows the firewall mark as hexadecimal, so 0xca6c, while wireguard sets it as decimal, so 51820):
32764: from all lookup main suppress_prefixlength 0
32765: not from all fwmark 0xca6c lookup 51820
32766: from all lookup main
A main table that might look something like this (ip route):
default via {gateway_ip} dev wlan0 proto dhcp metric 600
10.2.0.0/24 dev other_vpn proto kernel scope link src 10.2.0.210
And a "51820" table that looks like this (ip route list table 51820):
default dev wg0 scope link
So when a packet goes through these rules and tables:
At 32764, we go to the main table.
If the packet is going to, say, 10.2.0.5, then it will hit the 10.2.0.0/24 route, and with the 24 prefix, that's where it will go.
If, on the other hand, matches default (0.0.0.0/0), then rather than going to gateway_ip on wlan0, it will get suppressed by the suppress_prefixlength 0, and we'll continue.
At 32765, we check the fwmark. If it's not 0xca6c (ie, it's not a packet wireguard is sending to implement wg0), then we'll go to table 51820. And that's very simple: everything goes over wg0.
If the fwmark is 0xca6c, and thus it is a wireguard packet, then we'll go to 32766. That will bring us back to the main table, where, in this case, we'll match the default route, and the wireguard packets will go out through wlan0, as we want: we do, eventually, need to use our physical connection.
What it does?
For shorter and more straightforward explanation:
$ ip rule
from all lookup main suppress_prefixlength 0
This means, "check main table for routing decisions but discard any routes that have prefix 0 (or less)" (docs). So what it exactly mean? Given main table like this:
$ ip route show table main
default via 192.168.0.1 dev wlp0s20f3 proto dhcp metric 600
10.0.0.0/8 dev wlp0s20f3 scope link
Rule above will discard default (which translates to 0.0.0.0/0) therefore effective lookup table would look like:
#default via 192.168.0.1 dev wlp0s20f3 proto dhcp metric 600 # ignored
10.0.0.0/8 dev wlp0s20f3 scope link
How to use it?
Rules are processed from up-to-down (contrary to most-specific-to-least-specific route used by tables). Therefore it can be used to override just default route in main table while leaving more specific routes alone. For example:
$ ip rule
32764: from all lookup main suppress_prefixlength 0
32765: from all lookup 1234
32766: from all lookup main
$ ip route
default via 192.168.0.1 dev wlp0s20f3 proto dhcp metric 600
10.0.0.0/8 dev wlp0s20f3 scope link
$ ip route show table 1234
default dev ppp0 scope link
In setup like above whole traffic will go through device ppp0 (via rule 32765, table 1234) with exception of 10.0.0.0/8 which will go through device wlp0s20f3 (via rule 32764, table main).
This is super useful when you have dynamic routes (or just lot of custom ones) in main and you can't easily copy them to custom table (vide VPN case).
How to test it?
$ ip route get 8.8.8.8
8.8.8.8 dev ppp0 table 1234
$ ip route get 10.0.0.0
10.0.0.0 dev wlp0s20f3
if you use ip rule you will see the following output:
0: from all lookup local
32764: from all lookup main suppress_prefixlength 0
32765: not from all fwmark 0xca6c lookup 51820
32766: from all lookup main
32767: from all lookup default
your question is about line no.32764, it is used to routes all traffic via the tunnel instead of looking up in the [main] table.
reference:
https://man7.org/linux/man-pages/man8/ip-rule.8.html

Bro Script: Hardcoded IP addresses

Ich have one assignment and I need a little help. I have infected.pcap and the following task:
Hardcoded IP addresses Sometimes, malware contains hardcoded IP addresses to download their payload or to communicate with their command and control (C&C) server. Find all such communication. Hint: Such IPs have no preceding DNS request.
I need to solve it with Bro script. This was my idea, but unfortunatelly all my connections have no DNS request:
#load base/protocols/dns/main.bro
event file_timeout(f: fa_file)
{
for ( cid in f$conns )
{
if(f$conns[cid]?$dns){
print f$conns[cid]$dns;
print "DNS";
}else {
print "No DNS";
}
}
}
Do you know maybe what is wrong with my code?
I would suggest that you're using the wrong event for this. The file_timeout only occurs if a file transfer was occurring and then stopped without completing. A much more interesting event correlation would be:
Track DNS address lookup responses (I would likely use event
dns_A_reply(c: connection, msg: dns_msg, ans: dns_answer, a:
addr)).
Record the addresses returned in a set; this will provide
you a set of all addresses that were discovered through a DNS query.
Examine outbound requests (where orig_h on the SYN is an internal
address)
Check to see if the address in id$resp_h is in the set of
addresses step 2. If it is, return, if it isn't,
generate a notice since you have an outbound connection attempt with
no corresponding DNS lookup.

ip address updation in openflow

I am trying to modify the destination address for an incoming ping request at the switch using a POX controller. I use packet.next to modify the destination address. Once this address is modified I create a new packet with the incoming source IP and the new destination IP. But my pings aren't getting through. I also make sure that the nw destination of the message is modified before it is sent to the switch.
It will be really helpful if someone can help me solve this issue.
I'm using the l3_learning.py sample program present in Mininet.
I've added this condition in the handle_PacketIn function to the ifinstance(packet.next,arp).
My code
: : if str(packet.src)==str("00:00:00:00:00:19") and (inport)==19: packet.src = EthAddr("00:00:00:00:00:22") inport = 22 if str(packet.dst)==str("00:00:00:00:00:19") and inport==19: a1.protosrc = IPAddr("10.0.0.6") a1.hwsrc = EthAddr("00:00:00:00:00:22")
I then send an ARP packet.
I have changed the nw_dst using ofp.match() –
I figured out what I was doing wrong. Instead of programming flows I was directly trying to modify the packets to redirect to the hosts. That was why I was unable to get a ping response.

Wireshark dissector that works with tls/ssl

I have a protocol that uses SSL/TLS over a non-standard port and transmits non-standard data (not http) through it. I'm trying to make a wireshark dissector (in lua) to dissect this protocol.
How do I do this? I can register a dissector that gets called for tcp fragments on that port
local dissector_table_tcp = DissectorTable.get("tcp.port")
dissector_table_tcp:add(1234, myprotocol)
I can get the SSL dissector to then decode all the fragments as SSL
function myprotocol.dissector(tvb, pinfo, root)
local ssl_dissector = Dissector.get("ssl")
local ssl_dissected_len = ssl_dissector:call(tvb, pinfo, root)
pinfo.cols.protocol:set("My Protocol")
At this point, if I have a premaster key file set in Wireshark (Preferences->Protocols->SSL->Master key file), I can see the decrypted contents of the packets and all is good. Sort of.
But I want to create fields for my protocol and put them in the protocol tree. How do I get at the decrypted data that the ssl dissector produced?
Update:
I'm trying to muddle through this as best as I can; there's no tutorial on how exactly you're supposed to do this. It sort of looks like Wireshark has a programming model based on fields/variables that are populated by dissectors, and that in theory it should be possible to interrogate those variables to find the output of a dissector.
To that end, I've been running the SSL dissector and then looking at fields that it declares, but it doesn't actually seem to populate them. When I run a post-dissector after the SSL dissector, none of the seemingly-useful fields, like ssl.segments or ssl.segment.data, are set:
protocol_foo = Proto("foo", "Foo protocol")
port = 4172
g_field_segment = Field.new("ssl.segment")
g_field_segment_data = Field.new("ssl.segment.data")
g_field_segments = Field.new("ssl.segments")
g_field_reassembled_data = Field.new("ssl.reassembled.data")
function protocol_foo.dissector(tvb, pinfo, root)
print("====== protocol_foo")
for k,v in pairs({ g_field_segment, g_field_segment_data, g_field_segments, g_field_reassembled_data }) do
if v() ~= nil then
print("Field " .. v.name .. " is NOT nil")
else
print("Field " .. v.name .. " is nil")
end
end
end
-- post-dissector registration
local ssl_dissector = Dissector.get("ssl")
local dissector_table_tcp = DissectorTable.get("tcp.port")
dissector_table_tcp:add(port, ssl_dissector)
register_postdissector(protocol_foo)
When I run this code on my protocol, none of those ssl.segment* variables test positive; lots of variables (like the ssl.handshake.*) variables do test positive (at least with handshake pdus), but not the ones with the decrypted contents.
Does anyone have any ideas?

How to display the ip address and port number in an text box that should be generated dynamically

Is there a way to display the system ip address and port number in a text box that is generated dynamically???
I want the system to put the ip address into a text box according to the machine.
Siddharth
Since you mentioned a text box, I can only postulate that you are talking about a web browser, and in that case 99.9% of the time you are talking about http and then 99.999% of the time a TCP connection. This means that your connection will have a 4-Tuple consisting of the source ip:port and the destination ip:port. In most cases the port numbers are fairly standard (80) for the destination (client).
Then you get into the very common issues of NAT and the like, so again I think you need to clarify what type of ip address you want. The publicly routable ip address is obtained server side and the LAN address will be obtained from the localhost.
For the more interesting case (publicly routable ip) I would just use a server side script (python, PHP, C, etc...) to read the incoming ip address and then use a little ajax to set the value of the text box. I did something similar for a project and it worked really well. Our client program was written in Python and C but this will give you an idea...
# Returns the client's public IP address (past any NATs)
def get_public_ip():
return urllib.urlopen('http://ddih.org/ip.php').read().strip()
I think something like set the inner html... from that webpage...
Hope this helps.
Your system does not have a port number. Port numbers are a software concept to differentiate different IP or UDP applications that might want to listen for connections on your IP address.
Also, it is quite possible to have more than one IP address. In fact, your system almost always has two if you count the loopback address (127.0.0.1). Even if you don't these days even many consumer PC's have multiple ethernet jacks.
You didn't say you were using Win32 so I don't know that it will be useful to you, but here's some code I wrote once that puts all local IP addresses (loopback excepted) into a an MFC CComboBox. It's a bit more C-ish than I'd like to see these days, but here it is.
size_t const Max_Expected_Addresses = 20; // Something rediculous
unsigned long IPADDRTBL_Size = sizeof(DWORD) + sizeof(MIB_IPADDRROW) * Max_Expected_Addresses;
PMIB_IPADDRTABLE IP_Address_Table = (PMIB_IPADDRTABLE) malloc (IPADDRTBL_Size);
if (GetIpAddrTable (IP_Address_Table, &IPADDRTBL_Size, TRUE) == NO_ERROR) {
for (DWORD i = 0; i < IP_Address_Table->dwNumEntries; i++) {
// Skip the loopback.
if (IP_Address_Table->table[i].dwAddr == 0x0100007f) continue;
if (m_IP_Address == "") m_IP_Address = String_Address(IP_Address_Table->table[i].dwAddr);
m_IP_Address_List.AddString (String_Address(IP_Address_Table->table[i].dwAddr));
};
}
m_IP_Address_List is an MFC control defined as a CComboBox which gets filled in by this snippet.
m_IP_Address is a CString tied to an MFC textbox control (IIRC) which I use to store the currently selected (or first found on startup) IP address.