I am a little bit confused when interpreting the action part for the following rule
cookie=0x2b000000000000a5, duration=528.939s, table=0, n_packets=176, n_bytes=33116, idle_age=0, priority=2,in_port=1 actions=output:4,output:2
we have multiple action ports in a certain order, when checking the "restconf/operational/opendaylight-inventory:nodes/" in ODL controller we have different order for each port
"action": [
{ "order": 0,"output-action": {
"max-length": 65535,
"output-node-connector": "2" }
{"order": 1, "output-action": {
"max-length": 65535,
"output-node-connector": "4" }
}
I am not sure how the packets hitting such entry will be forwarded, are they replicated and send over both? are they load balanced over all ports?
what does the max-length refer to?
Is there any documentation explaining all fields in detail?
It seems, this is a group-table flow.
You can use group table functionality to support multiports in action part. You can read Openflow 1.3 spec documentation for details. (Part. 5.6, 5.6.1)
For max length, again from the same document (Part A.2.5):
An Output action uses the following structure and fields:
Action structure for OFPAT_OUTPUT, which sends packets out 'port'. When the
'port' is the OFPP_CONTROLLER, 'max_len' indicates the max number of
bytes to send. A 'max_len' of zero means no bytes of the packet should
be sent. A 'max_len' of OFPCML_NO_BUFFER means that the packet is not
buffered and the complete packet is to be sent to the controller.
Related
I have a list of jobs in my application queue(RabbitMQ).
Some of these jobs are group together and must do in the order.(not continuous, but by order of dispatch time)
For example, consider this 4 jobs in the queue:
[
{ "group": "x", "dispatched_timestamp": 10001, "field1": "some data", "field2": "some other data"},
{ "group": "g", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
{ "group": "x", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
{ "group": "t", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"}
]
I must sure the first job in group "x" execute successfully before the thirth job(same group).
But i don't care if the fourth job execute sooner than the first(or whatever).
Because sometimes it may happen which all three job deliver to 3 consumer, but the first job fail for some reason(but the second and thirth job has done successful).
I know with this conditions there will be some situations which all jobs in the queue are belongs to same group, so multiple consumers can't work on them and they must deliver one by one. that's ok.
There's no such thing in AMQ protocol that can lead to this exact solution, there're some ways to solve this problem.
Define queue for each message group
Set concurrency as 1
Let me quote the message ordering from the doc
Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Ref: https://www.rabbitmq.com/semantics.html
First of the foremost things for you is to preserve the message ordering, once we have ordered messages we can utilize the concurrency to handle the messages in order.
Let's say your queue has 5 messages as shown
Queue: Queue1
+--------------+
Head-->|m1|m2|m3|m4|m5| <---- Tail
+--------------+
There's the concept of competing consumers, competing consumers means there're more than consumers/subscribers for the same queue. If there is more than one consumer than each of them will run autonomously, which means ordering on the consumer side won't be preserved. To preserve the ordering on consumer side, we should not use competing consumers.
Even though now consumers are not competing, we can still lose message ordering, if we have more than one executor. More than one executor simply means we can poll the queue, send a polled message to any of the executors. Based on the CPU execution policy etc we will still lose the ordering, so now we need to restrict the number of executors to 1.
As we have only one executor each of the polled messages will be executed in orders, so it will become a serial execution.
For Queue1
The executor will consume the message in the following order
-> m1
-> m2
-> m3
-> m4
-> m5
Still, there's one missing piece, what happens if the execution of m1 is failing?
You can retry for N number of times before consuming the next message, to achieve this don't acknowledge unless you have successfully executed any polled message.
From design points of view, this does not look good, since you're processing messages in serial instead of parallel, though you don't have any other alternatives.
I am using TcpClient to connect to a simple TCP echo server. Messages consist of the message size in 4 bytes followed by the message itself. For instance, to send the message "hello", the server will expect "0005hello", and respond with "0005hello".
When testing under load (approximately 300+ concurrent users), adjacent requests sometimes result in responses "piling up", e.g. sending "0004good" followed by "0003day" might result in the client receiving "0004good0003" followed by "day".
In a conventional, non-WebFlux-based TCP client, one would normally read the first 4 bytes from the socket into a buffer, determine the length of the message N, then read the following N bytes from the socket into a buffer, before returning the response. Is it possible to achieve such fine-grained control, perhaps by using TcpClient's underlying Channel?
I have also considered the approach of accumulating responses in some data structure (Queue, StringBuffer, etc.) and having a daemon parse the result, but this has not had the desired performance in practice.
I solved this by adding a handler of type LengthFieldBasedFrameDecoder to the Connection:
TcpClient.create()
.host(ADDRESS)
.port(PORT)
.doOnConnected((connection) -> {
connection.addHandler("parseLengthFromFirstFourBytes", new LengthFieldBasedFrameDecoder(9999, 0, 4) {
#Override
protected long getUnadjustedFrameLength(ByteBuf buf, int offset, int length, ByteOrder order) {
ByteBuf lengthBuffer = buf.copy(0, 4);
byte[] messageLengthBytes = new byte[4];
lengthBuffer.readBytes(messageLengthBytes);
String messageLengthString = new String(messageLengthBytes);
return Long.parseLong(messageLengthString);
}
});
})
.connect()
.subscribe();
This solves the issue with the caveat that responses still "pile up" (as described in the question) when the application is subjected to sufficient load.
I had declare a queue like below:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length-bytes", 2 * 1024 * 1024); // Max length is 2G
channel.queueDeclare("queueName", true, false, false, args);
When the queue messages count bytes is large than 2G, It will auto remove the message on the head of the queue.
But what I expected is That it reject produce the last message and return exception to the producer.
How can I get it?
A possible workaround is check the queue size before send your message using the HTTP API.
For example if you have a queue called: myqueuetest with max size = 20.
Before send the message you can call the HTTP API in this way:
http://localhost:15672/api/queues/
the result is a JSON like this:
"message_bytes":10,
"message_bytes_ready":10,
"message_bytes_unacknowledged":0,
"message_bytes_ram":10,
"message_bytes_persistent":0,
..
"name":"myqueuetest",
"vhost":"test",
"durable":true,
"auto_delete":false,
"arguments":{
"x-max-length-bytes":20
},
then you cloud read the message_bytes field before send your message and then decide if send or not.
Hope it helps
EDIT
This workaround could kill your application performance
This workaround is not safe if you have multi-threading/more publisher
This workaround is not a very "best practise"
Just try to see if it is ok for your application.
As explained on the official docs:
Messages will be dropped or dead-lettered from the front of the queue to make room for new messages once the limit is reached.
https://www.rabbitmq.com/maxlength.html
If you think RabbitMQ should drop messages form the end of the queue, feel free to open an issue here so we can discuss about it https://github.com/rabbitmq/rabbitmq-server/issues
netty 4.0.24
I am passing XML over UDP. When receiving the UPD packet, the packet is always of length 2048, truncating the message. Even though, I have attempted to set the receive buffer size to something larger (4096, 8192, 65536) but it is not being honored.
I have verified the UDP sender using another UDP ingest mechanism. A standalone Java app using java.net.DatagramSocket. The XML is around 45k.
I was able to trace the stack to DatagramSocketImpl.createChannel (line 281). Stepping into DatagramChannelConfig, it has a receiveBufferSize of whatever I set (great), but a rcvBufAllocator of 2048.
Does the rcvBufAllocator override the receiveBufferSize (SO_RCVBUF)? Is the message coming in multiple buffers?
Any feedback or alternative solutions would be greatly appreciated.
I also should mention, I am using an ESB called vert.x which uses netty heavily. Since I was able to trace down to netty, I was hopeful that I could find help here.
The maximum size of incoming datagrams copied out of the socket is actually not a socket option, but rather a parameter of the socket read() function that your client passes in each time it wants to read a datagram. One advantage of this interface is that programs accepting datagrams of unknown/varying lengths can adaptively change the size of the memory allocated for incoming datagram copies such that they do not over-allocate memory while still getting the whole datagram. (In netty this allocation/prediction is done by implementors of io.netty.channel.RecvByteBufAllocator.)
In contrast, SO_RCVBUF is the size of a buffer that holds all of the datagrams your client hasn't read yet.
Here's an example of how to configure a UDP service with a fixed max incoming datagram size with netty 4.x using a Bootstrap:
import io.netty.bootstrap.Bootstrap;
import io.netty.channel.ChannelOption;
import io.netty.channel.FixedRecvByteBufAllocator;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioDatagramChannel;
int maxDatagramSize = 4092;
String bindAddr = "0.0.0.0";
int port = 1234;
SimpleChannelInboundHandler<DatagramPacket> handler = . . .;
InetSocketAddress address = new InetSocketAddress(bindAddr, port);
NioEventLoopGroup group = new NioEventLoopGroup();
Bootstrap b = new Bootstrap()
.group(group)
.channel(NioDatagramChannel.class)
.handler(handler);
b.option(ChannelOption.RCVBUF_ALLOCATOR, new FixedRecvByteBufAllocator(maxDatagramSize));
b.bind(address).sync().channel().closeFuture().await();
You could also configure the allocator with ChannelConfig.setRecvByteBufAllocator
As far as I understand source code, net device is already prepared fot UDP, but how to make UDP scheme?
In this line of the source https://github.com/rebol/r3/blob/master/src/core/c-port.c#L612, the comments say
In order to add a port scheme:
In mezz-ports.r add a make-scheme.
Add an Init_*_Scheme() here.
Be sure host-devices.c has the device enabled.
I think the first instruction refers to mezz/sys-ports.r. So, we have this example https://github.com/rebol/r3/blob/master/src/mezz/sys-ports.r#L254, we could add something like
make-scheme [
title: "UDP Networking"
name: 'udp
spec: system/standard/port-spec-net
info: system/standard/net-info ; for C enums
awake: func [event] [print ['UDP-event event/type] true]
]
You would then have to write an INIT_UDP_SCHEME like this one for TCP https://github.com/rebol/r3/blob/master/src/core/p-net.c#L299 where TCP_Actor starts here https://github.com/rebol/r3/blob/master/src/core/p-net.c#L92, and then initialize it here https://github.com/rebol/r3/blob/master/src/core/c-port.c#L626
And as you say, the UDP does seem to be otherwise ready.