Forwarding rules apply with delay (UDP traffic) - sdn

I am working on a project where Open vSwitch 2.9 (with kernel module 2.9) is installed on Ubuntu 16.04. I dynamically install forwarding rules for flows and remove them when the flow completes (using Floodlight controller 1.2). I am forwarding UDP traffic across VMs. My problem is that sometimes rules don't function.
For example, I see that a rule's packet counter increasing at one switch and on the next switch when I run tcpdump I see no packets coming in (although on the previous switch tcpdump shows packets on the correct outgoing interface). In some cases the forwarding suddenly starts working after a few minutes which is way too late!
Running netstat does not show any dropped UDP packets. I tried different versions of OVS and also tried using the userspace mode but the problem persists. I also noticed that depending on other flows that are installed, the switches where this problem occurs change but as long as I install all preceding flows the same way, the same switch and the same flow observes the problem. Any ideas what I might do next to figure out how to fix this?
Also, my forwarding rules use the following fields:
ovs0:~$ sudo ovs-ofctl dump-flows br0 -O OpenFlow13
cookie=0xafffff84962f92, duration=263.813s, table=0, n_packets=199405, n_bytes=269333926, send_flow_rem priority=12345,udp,in_port=eth2,nw_src=10.10.10.1,nw_d
st=10.10.10.2,tp_dst=3330 actions=output:eth3
cookie=0xafffff84962f95, duration=263.743s, table=0, n_packets=170236, n_bytes=229866720, send_flow_rem priority=12345,udp,in_port=eth3,nw_src=10.10.10.2,nw_d
st=10.10.10.1,tp_dst=1159 actions=output:eth2
cookie=0xafffff84962f9a, duration=263.437s, table=0, n_packets=202470, n_bytes=273485588, send_flow_rem priority=12345,udp,in_port=eth2,nw_src=10.10.10.1,nw_d
st=10.10.10.5,tp_dst=8450 actions=output:eth4
cookie=0xafffff84963066, duration=263.362s, table=0, n_packets=200642, n_bytes=271012852, send_flow_rem priority=12345,udp,in_port=eth4,nw_src=10.10.10.5,nw_d
st=10.10.10.1,tp_dst=9330 actions=output:eth2
cookie=0x0, duration=1294.627s, table=0, n_packets=447, n_bytes=381687, priority=0 actions=CONTROLLER:65535
I also tried using VLANs, same problem. This is the set of rules at the switch where packets are forwarded but not received at the next switch (flow with tp_dst=1966):
ovs7:~$ sudo ovs-ofctl dump-flows br0 -O OpenFlow13
cookie=0xafffff84963703, duration=158.640s, table=0, n_packets=75083, n_b
ytes=101947446, send_flow_rem priority=2,ip,in_port=eth5,dl_vlan=1895 acti
ons=pop_vlan,set_field:02:9b:dc:eb:80:c8->eth_dst,output:eth1
cookie=0xafffff8496e272, duration=93.641s, table=0, n_packets=3539, n_byt
es=4793142, send_flow_rem priority=2,ip,in_port=eth2,dl_vlan=1105 actions=
pop_vlan,set_field:02:9b:dc:eb:80:c8->eth_dst,output:eth1
cookie=0xafffff8496e419, duration=78.626s, table=0, n_packets=0, n_bytes=
0, send_flow_rem priority=2,ip,in_port=eth2,dl_vlan=1458 actions=output:et
h4
cookie=0xafffff8496de55, duration=148.614s, table=0, n_packets=66432, n_b
ytes=89933600, send_flow_rem priority=2,udp,in_port=eth1,nw_src=10.10.10.8
,nw_dst=10.10.10.12,tp_dst=1597 actions=push_vlan:0x8100,set_field:5693->v
lan_vid,output:eth5
cookie=0xafffff8496e0c9, duration=108.644s, table=0, n_packets=4592, n_by
tes=6203544, send_flow_rem priority=2,udp,in_port=eth1,nw_src=10.10.10.8,n
w_dst=10.10.10.12,tp_dst=1363 actions=push_vlan:0x8100,set_field:5459->vla
n_vid,output:eth4
*cookie=0xafffff8496e341, duration=83.644s, table=0, n_packets=3145, n_byt
es=4245630, send_flow_rem priority=2,udp,in_port=eth1,nw_src=10.10.10.8,nw
_dst=10.10.10.11,tp_dst=1966 actions=push_vlan:0x8100,set_field:6062->vlan
_vid,output:eth2*
cookie=0x0, duration=15540.491s, table=0, n_packets=683847, n_bytes=92457
8766, priority=0 actions=CONTROLLER:65535

Related

AWS|Traffic mirroring by using iptables

I am trying to achieve Network Traffic Mirroring with iptables,In my scenario, i am mirroring the traffic on Server 1 to Server 2's IP address. Apparently configurations are straight foreword as follow
Server 1
echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -t mangle -I POSTROUTING -j TEE --gateway 172.31.34.228 (Server 2 IP)
iptables -t mangle -I PREROUTING -j TEE --gateway 172.31.34.228
But when i run tcpdump on Server 2's interface(172.31.34.228), it is not showing any results.
Both servers are on AWS and under same subnet, OS is AWS latest IAM,
[root#ip-172-31-37-29 ~]# iptables --version
iptables v1.4.21
Kernel Modules
[root#ip-172-31-37-29 ~]# ls /lib/modules/`uname -r`/kernel/net/netfilter/
ipset nf_log_common.ko nf_tables_inet.ko nft_objref.ko xt_cluster.ko xt_hashlimit.ko xt_nat.ko xt_sctp.ko
ipvs nf_log_netdev.ko nf_tables.ko nft_queue.ko xt_comment.ko xt_helper.ko xt_NETMAP.ko xt_SECMARK.ko
nf_conntrack_amanda.ko nf_nat_amanda.ko nf_tables_netdev.ko nft_redir.ko xt_connbytes.ko xt_hl.ko xt_nfacct.ko xt_set.ko
nf_conntrack_broadcast.ko nf_nat_ftp.ko nft_compat.ko nft_reject_inet.ko xt_connlabel.ko xt_HL.ko xt_NFLOG.ko xt_socket.ko
nf_conntrack_ftp.ko nf_nat_irc.ko nft_counter.ko nft_reject.ko xt_connlimit.ko xt_HMARK.ko xt_NFQUEUE.ko xt_state.ko
nf_conntrack_h323.ko nf_nat.ko nft_ct.ko nft_rt.ko xt_connmark.ko xt_IDLETIMER.ko xt_osf.ko xt_statistic.ko
nf_conntrack_irc.ko nf_nat_redirect.ko nft_exthdr.ko nft_set_bitmap.ko xt_CONNSECMARK.ko xt_ipcomp.ko xt_owner.ko xt_string.ko
nf_conntrack.ko nf_nat_sip.ko nft_fib_inet.ko nft_set_hash.ko xt_conntrack.ko xt_iprange.ko xt_physdev.ko xt_tcpmss.ko
nf_conntrack_netbios_ns.ko nf_nat_tftp.ko nft_fib.ko nft_set_rbtree.ko xt_cpu.ko xt_ipvs.ko xt_pkttype.ko xt_TCPMSS.ko
nf_conntrack_netlink.ko nfnetlink_acct.ko nft_fib_netdev.ko x_tables.ko xt_CT.ko xt_l2tp.ko xt_policy.ko xt_TCPOPTSTRIP.ko
nf_conntrack_pptp.ko nfnetlink_cthelper.ko nft_hash.ko xt_addrtype.ko xt_dccp.ko xt_length.ko xt_quota.ko xt_tcpudp.ko
nf_conntrack_proto_gre.ko nfnetlink_cttimeout.ko nft_limit.ko xt_AUDIT.ko xt_devgroup.ko xt_limit.ko xt_rateest.ko xt_TEE.ko
nf_conntrack_sane.ko nfnetlink.ko nft_log.ko xt_bpf.ko xt_dscp.ko xt_LOG.ko xt_RATEEST.ko xt_time.ko
nf_conntrack_sip.ko nfnetlink_log.ko nft_masq.ko xt_cgroup.ko xt_DSCP.ko xt_mac.ko xt_realm.ko xt_TPROXY.ko
nf_conntrack_snmp.ko nfnetlink_queue.ko nft_meta.ko xt_CHECKSUM.ko xt_ecn.ko xt_mark.ko xt_recent.ko xt_TRACE.ko
nf_conntrack_tftp.ko nf_synproxy_core.ko nft_nat.ko xt_CLASSIFY.ko xt_esp.ko xt_multiport.ko xt_REDIRECT.ko xt_u32.ko
[root#ip-172-31-37-29 ~]# rpm -ql kernel | grep xt_TEE
/lib/modules/4.14.62-70.117.amzn2.x86_64/kernel/net/netfilter/xt_TEE.ko
/lib/modules/4.14.70-72.55.amzn2.x86_64/kernel/net/netfilter/xt_TEE.ko
I am really stuck, and any help will be really appreciated.
According to SO thread target must be in same network as your computer. https://code.google.com/archive/p/port-mirroring/ is hinted to be alternative way of achieving your goal but it seems to be openwrt specific.

How to prevent loopback publish when mosquitto bridged with RabbitMQ MQTT?

I have two mosquitto brokers installed on PC1 (mosquitto v1.4.8) and PC2 (RabbitMQ v3.6.2 with MQTT Adapter).
Bridging initiated at PC1 like below
sensor/room1/ <-> office/room1/
But I noticed there is always a duplicate message being published back whenever the bridge is active, means all my application (on PC1) which subscribes to the same topic will receives the same message twice. What setting I did wrong here?
PC1 mosquitto.conf
connection bridge-pc1-to-pc2
address pc2-address.com
topic room1/# both 2 sensor/ office/
bridge_protocol_version mqttv311
notifications true
cleansession true
try_private false
To test loopback issue, I had PC1 subscribed to topic sensor/#
mosquitto_sub -t sensor/# -v -d
Then at PC1 I publish a test message
mosquitto_pub -t sensor/room1/temperature -m '{"value":27.3, "timestamp":"2016-06-03 14:02:38"}'
Broker at cloud (PC2) received the message correctly (message received only once)
Client mosqsub/3121-Dennis-iMa sending CONNECT
Client mosqsub/3121-Dennis-iMa received CONNACK
Client mosqsub/3121-Dennis-iMa sending SUBSCRIBE (Mid: 1, Topic: office/#, QoS: 0)
Client mosqsub/3121-Dennis-iMa received SUBACK
Subscribed (mid: 1): 0
Client mosqsub/3121-Dennis-iMa received PUBLISH (d0, q0, r0, m0, 'office/room1/temperature', ... (14 bytes))
office/room1/temperature {"value":27.3, "timestamp":"2016-06-03 14:02:38"}
But PC1 received the same message twice! Below is the Pi's output
Received CONNACK
Received SUBACK
Subscribed (mid: 1): 0
Received PUBLISH (d0, q0, r0, m0, 'sensor/room1/temperature', ... (14 bytes))
sensor/room1/temperature {"value":27.3, "timestamp":"2016-06-03 14:02:38"}
Received PUBLISH (d0, q0, r0, m0, 'sensor/room1/temperature', ... (14 bytes))
sensor/room1/temperature {"value":27.3, "timestamp":"2016-06-03 14:02:38"}
Why there is loopback published message and how to solve this?
Update 3 Jun 2016
This is not the same question with this question, as it does not involve horizontal scaling (1-to-many brokers)
Change try_private false to try_private true. This is exactly what it is intended for. If rabbit doesn't support that feature (it is currently not in the spec, but widely used) then you're out of luck.
RabbitMQ doesn't support try_private and doesn't know anything about the bridge. So messages published to office/* in RabbitMQ will be sent back to subscribed mosquitto without taking to account any private flags.
To remove cycles you can use different topic names for in and out connections or use two mosquitto servers.

{active, N} mode in erlang secure tcp

I write this simple code with ssltcp:
ssl:start().
{ok, ListenSocket} = ssl:listen(9999, [{certfile, "cert.pem"}, {keyfile, "key.pem"},{reuseaddr, true}]).
{ok, Socket} = ssl:transport_accept(ListenSocket).
ssl:ssl_accept(Socket).
ssl:setopts(Socket, [{active, once}]).
it works fine but when i replace {active, once} with {active, 3}, returns this error:
{error,{options,{socket_options,{active,3}}}}
How can use {active, N} mode in secure tcp?
The {active,N} mode is not implemented for SSL connections. I originally wrote the {active,N} mode and when I looked into possibly implementing it for SSL, I found that the way Erlang SSL sockets are implemented over the top of underlying TCP sockets involves changes on those sockets between active and passive modes as part of the protocol implementation, and so implementing {active,N} for SSL is not simply a matter of opening an underlying socket in that mode.

java.net.ConnectException: JBAS012144: Could not connect to remote://nnn.nn.nn.88:9999. The connection timed out

I am trying to run in jboss instance in domain mode. While I do that I am getting the following issue......
[Host Controller] 12:45:56,535 WARN [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010900: Could not connect to remote domain controller at remote://nnn.nn.nn.88:9999 -- java.net.ConnectException: JBAS012144: Could not connect to remote://nnn.nn.nn.88:9999. The connection timed out
I had ran two JBoss instance in domain mode after configuring...
First JBoss instance->
./domain.sh -b nnn.nn.nn.88 -Djboss.bind.address.management=nnn.nn.nn.88
Second JBoss Instance ->
./domain.sh -b nnn.nn.nn.89 -Djboss.domain.master.address=nnn.nn.nn.88 --host-config=host-slave.xml
nnn.nn.nn.88 host.xml configuration is as follows...
<domain-controller>
<local/>
</domain-controller>
nnn.nn.nn.89 host-slave.xml configuration is as follows...
<domain-controller>
<remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>
<domain-controller>
I am able to telnet to port 9999 on host nnn.nn.nn.88 from 89..... as I configured by removing loopback ip for public & management port...... Although is it the implication that <domain-controller> has <local/>....
Please help me to solve this issue... JDK version is JDK 7 Update 80.... EAP 6.3....
In HC host.xml and if we use --host-config=host-slave.xml that particular xml has to connected with DC under <domain-controller> node....
jboss.domain.master.address should be Domain Controller address nnn.nn.nn.88....
<domain-controller>
<remote host="${jboss.domain.master.address:nnn.nn.nn.88}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>
<domain-controller>
As per the solution article from redhat....
https://access.redhat.com/solutions/218053#
I ran following command for the same configuration which I had while posting this question..... And I got succeeded.....
DC->
./domain.sh -b my-host-ip1 -bmanagement my-host-ip1
HC->
./domain.sh -Djboss.domain.master.address=my-host-ip1 -b my-host-ip2 -bmanagement my-host-ip2
Although is this way of configuring gives clustering capability to DC and HCs..... I had raised same question to Redhat on the same solution article..... The answer must be yes I hope....
https://access.redhat.com/solutions/218053#comment-975683

Calling COM Library From XBAP

I am trying to call an old COM library from my XBAP and continue to receive the following exception:
System.AccessViolationException was unhandled
Message: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
I have tried adding the HKLM value for RunUnrestricted to no avail.
I don't get anything else but this error when calling the library. Any ideas? (This library even works from a pure ASP.NET app)
EDIT:
The COM library makes socket calls to a server. It looks like that is happenning but somewhere after the last packet, it bombs with this error.
No. Time Source Destination Protocol Info
10 8.452945 10.10.10.210 10.10.10.250 TCP 50736 > 22700 [SYN] Seq=0 Win=8192 Len=0 MSS=1260 WS=8
14 11.459350 10.10.10.210 10.10.10.250 TCP 50736 > 22700 [SYN] Seq=0 Win=8192 Len=0 MSS=1260 WS=8
21 17.459690 10.10.10.210 10.10.10.250 TCP 50736 > 22700 [SYN] Seq=0 Win=8192 Len=0 MSS=1260
try trusting the site. e.g. trusted sites for IE.
I wish there was a way to cancel a question cause this one is my fault. I was tunneled through a VPN and just had a bad connection no matter how many times I reconnected. After a restart, I was then able to interact with the API.