netty 4.0.24
I am passing XML over UDP. When receiving the UPD packet, the packet is always of length 2048, truncating the message. Even though, I have attempted to set the receive buffer size to something larger (4096, 8192, 65536) but it is not being honored.
I have verified the UDP sender using another UDP ingest mechanism. A standalone Java app using java.net.DatagramSocket. The XML is around 45k.
I was able to trace the stack to DatagramSocketImpl.createChannel (line 281). Stepping into DatagramChannelConfig, it has a receiveBufferSize of whatever I set (great), but a rcvBufAllocator of 2048.
Does the rcvBufAllocator override the receiveBufferSize (SO_RCVBUF)? Is the message coming in multiple buffers?
Any feedback or alternative solutions would be greatly appreciated.
I also should mention, I am using an ESB called vert.x which uses netty heavily. Since I was able to trace down to netty, I was hopeful that I could find help here.
The maximum size of incoming datagrams copied out of the socket is actually not a socket option, but rather a parameter of the socket read() function that your client passes in each time it wants to read a datagram. One advantage of this interface is that programs accepting datagrams of unknown/varying lengths can adaptively change the size of the memory allocated for incoming datagram copies such that they do not over-allocate memory while still getting the whole datagram. (In netty this allocation/prediction is done by implementors of io.netty.channel.RecvByteBufAllocator.)
In contrast, SO_RCVBUF is the size of a buffer that holds all of the datagrams your client hasn't read yet.
Here's an example of how to configure a UDP service with a fixed max incoming datagram size with netty 4.x using a Bootstrap:
import io.netty.bootstrap.Bootstrap;
import io.netty.channel.ChannelOption;
import io.netty.channel.FixedRecvByteBufAllocator;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioDatagramChannel;
int maxDatagramSize = 4092;
String bindAddr = "0.0.0.0";
int port = 1234;
SimpleChannelInboundHandler<DatagramPacket> handler = . . .;
InetSocketAddress address = new InetSocketAddress(bindAddr, port);
NioEventLoopGroup group = new NioEventLoopGroup();
Bootstrap b = new Bootstrap()
.group(group)
.channel(NioDatagramChannel.class)
.handler(handler);
b.option(ChannelOption.RCVBUF_ALLOCATOR, new FixedRecvByteBufAllocator(maxDatagramSize));
b.bind(address).sync().channel().closeFuture().await();
You could also configure the allocator with ChannelConfig.setRecvByteBufAllocator
Related
I am using TcpClient to connect to a simple TCP echo server. Messages consist of the message size in 4 bytes followed by the message itself. For instance, to send the message "hello", the server will expect "0005hello", and respond with "0005hello".
When testing under load (approximately 300+ concurrent users), adjacent requests sometimes result in responses "piling up", e.g. sending "0004good" followed by "0003day" might result in the client receiving "0004good0003" followed by "day".
In a conventional, non-WebFlux-based TCP client, one would normally read the first 4 bytes from the socket into a buffer, determine the length of the message N, then read the following N bytes from the socket into a buffer, before returning the response. Is it possible to achieve such fine-grained control, perhaps by using TcpClient's underlying Channel?
I have also considered the approach of accumulating responses in some data structure (Queue, StringBuffer, etc.) and having a daemon parse the result, but this has not had the desired performance in practice.
I solved this by adding a handler of type LengthFieldBasedFrameDecoder to the Connection:
TcpClient.create()
.host(ADDRESS)
.port(PORT)
.doOnConnected((connection) -> {
connection.addHandler("parseLengthFromFirstFourBytes", new LengthFieldBasedFrameDecoder(9999, 0, 4) {
#Override
protected long getUnadjustedFrameLength(ByteBuf buf, int offset, int length, ByteOrder order) {
ByteBuf lengthBuffer = buf.copy(0, 4);
byte[] messageLengthBytes = new byte[4];
lengthBuffer.readBytes(messageLengthBytes);
String messageLengthString = new String(messageLengthBytes);
return Long.parseLong(messageLengthString);
}
});
})
.connect()
.subscribe();
This solves the issue with the caveat that responses still "pile up" (as described in the question) when the application is subjected to sufficient load.
I’m using SSL for reading data from various remote services over secure websockets as follows: I create the socket, embed it in the SSL context and add the socket to the reading list for Unix.select. When the socket fires, I use Ssl.read to get the data.
4 services are working well. And with one I get Ssl.Read_error.Error_syscall: error:00000000:lib(0):func(0):reason(0) after receiving each websocket frame (size ~5-6Kb). By the way, frames here are much bigger than on other services, but I’m not sure it’s the reason.
I ignore syscall errors (and most probably loose some data) because frames continue to arrive. Then, always after one minute I get Ssl.Read_error.Error_zero_return: error:00000000:lib(0):func(0):reason(0), which means the peer closed SSL socket for writing and I have to restart the process because no new data will be received from this socket.
Problem is perfectly reproducible. At the same time examples for this service and my own test implementation with Node.JS receive the data for hours without any problems.
I assume I do something wrong or setup socket/SSL too straightforward (see below).
Any help or ideas would be strongly appreciated.
let sock = Unix.socket PF_INET SOCK_STREAM 0 in
let laddr = Unix.inet_addr_of_string p.interface in
Unix.bind sock (ADDR_INET (laddr,0));
Unix.connect sock addr;
let (sock, res) =
let req = Bytes.of_string http_request in
if ssl then begin
Ssl.init ();
let ctx = create_context TLSv1_2 Client_context in
let sock = Ssl.embed_socket sock ctx in
Ssl.connect sock;
(SslSock sock, (write sock req 0 http_request_len))
end else
(UnixSock sock, (Unix.write sock req 0 http_request_len))
WireShark did the trick: this “bad” service sends two websocket frames in one tcp packet where second frame has zero payload length. Naturally, my Websocket implementation improperly handled frames with zero payload which lead to missing of Ping frames and closing of TCP connection by remote server.
We are working with NRF52840 dongles and want to be able to have them relay data over an OpenThread mesh network through UDP automatically. We have found within the OpenThread API a solid Udp.h library with all the Udp functions we need to create code that runs on the dongles from the main.c.
Below is our code that should broadcast the message: "Hallo" to all nodes that have an open socket on port 1994.
We have read that the ipv6 address ff03::1 is reserved for multicast UDP broadcasting and it works perfectly when manually performed with the CLI udp commands.
CLI: Udp open, udp send ff03::1 1994 Hallo
With all the nodes that have udp open, udp bind :: 1994, receiving the Hallo message from the sending node.
We are trying to recreate this in the main.c of our nodes so that we can provide the nodes with some intelligence of their own.
This piece of code is run once when the push button on the dongle is pressed.
The code compiles perfectly and we have tested the functions that have a return with the RGB led (green OK, red not) to confirm that there weren't any errors produced (sadly not all functions return a no_error value)
void udpSend(){
const char *buf = "Hallo";
otMessageInfo messageInfo;
otInstance *myInstance;
myInstance = thread_ot_instance_get();
otUdpSocket mySocket;
memset(&messageInfo, 0, sizeof(messageInfo));
// messageInfo.mPeerAddr = otIp6GetUnicastAddresses(myInstance)->mNext->mNext->mAddress;
otIp6AddressFromString("ff03::1", &messageInfo.mPeerAddr);
messageInfo.mPeerPort = 1994;
messageInfo.mInterfaceId = OT_NETIF_INTERFACE_ID_THREAD;
otUdpOpen(myInstance, &mySocket, NULL, NULL);
otMessage *test_Message = otUdpNewMessage(myInstance, NULL);
otMessageSetLength(test_Message, sizeof(buf));
if (otMessageAppend(test_Message, &buf, sizeof(buf)) == OT_ERROR_NONE){
nrf_gpio_pin_write(LED2_G, 0);
}
else{
nrf_gpio_pin_write(LED2_R, 0);
}
otUdpSend(&mySocket, test_Message, &messageInfo);
otCliUartOutputFormat("Done.\0");
otUdpClose(&mySocket);
}
Now, we aren't exactly experts, so we are not sure why this isn't working as we had a lot of trouble figuring out how everything is called/initialised.
We hope to create a way to send and receive data through UDP through the code, so that they can operate autonomously.
We would really appreciate it if someone could assist us with our project!
Thanks!
Jonathan
There are a few errors in your code:
Remove the call to otMessageSetLength(). The message length is automatically increased as part of otMessageAppend().
The call to otMessageAppend() should be: otMessageAppend(test_message, buf, (uint16_t)strlen(buf)).
Removed the & before buf.
Replaced sizeof() with strlen().
Couple other things you should consider:
After calling otUdpNewMessage(), if any following call returns an error, make sure to call otMessageFree() on the message buffer.
Custody is only given to OpenThread after a successful call to otUdpSend().
Do not call udpSend() from interrupt context.
OpenThread library was designed to assume a single thread of execution.
Hope that helps.
At the interface level DCCP is like TCP: you connect and then send/receive bytes.
I was wondering it's possible to make dccp connections in twisted by just adapting the wrappers for tcp...
According to the sample code (below) what needs to be changed is:
at socket instantiation: use different parameters
before using the socket: set some options
Then everything else would be the same...
Hints: I've spotted addressFamily and socketType in the sources of twisted but I have no idea on how to cleanly set them in the protocol factory. Also the protocol number, the 3rd parameter, here IPPROTO_DCCP, is always keeped to default. I have no clue either on how to access the socket to call setsockopt
import socket
socket.DCCP_SOCKOPT_PACKET_SIZE = 1
socket.DCCP_SOCKOPT_SERVICE = 2
socket.SOCK_DCCP = 6
socket.IPPROTO_DCCP = 33
socket.SOL_DCCP = 269
packet_size = 256
address = (socket.gethostname(),12345)
# Create sockets
server,client = [socket.socket(socket.AF_INET, socket.SOCK_DCCP,
socket.IPPROTO_DCCP) for i in range(2)]
for s in (server,client):
s.setsockopt(socket.SOL_DCCP, socket.DCCP_SOCKOPT_PACKET_SIZE, packet_size)
s.setsockopt(socket.SOL_DCCP, socket.DCCP_SOCKOPT_SERVICE, True)
# Connect sockets
server.bind(address)
server.listen(1)
client.connect(address)
s,a = server.accept()
# Echo
while True:
client.send(raw_input("IN: "))
print "OUT:", s.recv(1024)
More about DCCP:
https://www.sjero.net/research/dccp/
https://wiki.linuxfoundation.org/networking/dccp
TL;DR: dccp is a protocol that provides congestion control (like tcp) without guaranteeing reliability or in-order delivery of data (like udp). The standard linux kernel implements dccp.
I write client-server application like this:
client(c#) <-> server (twisted; ftp proxy and additional functional) <-> ftp server
Server has two classes: my own class-protocol inherited from LineReceiever protocol and FTPClient from twisted.protocols.ftp.
But when client sends or gets big files (10 Gb - 20 Gb) server catches MemoryError. I don't use any buffers in my code. It happens when after call transport.write(data) data appends to inner buffer of reactor's writers (correct me if I wrong).
What should I use to avoid this problem? Or should I change approach to the problem?
I found out that for big streams, I should use IConsumer and IProducer interfaces. But finally it will invoke transfer.write method and effect will be the same. Or am I wrong?
UPD:
Here is logic of file download/upload (from ftp through Twisted server to client on Windows):
Client sends some headers to Twisted server and after that begins send of file. Twisted server receive headers and after that (if it needs) invoke setRawMode(), open ftp connection and recieves/sends bytes from/to client and after all close connections. Here is a part of code that uploads files:
FTPManager class
def _ftpCWDSuccees(self, protocol, fileName):
self._ftpClientAsync.retrieveFile(fileName, FileReceiver(protocol))
class FileReceiver(Protocol):
def __init__(self, proto):
self.__proto = proto
def dataReceived(self, data):
self.__proto.transport.write(data)
def connectionLost(self, why = connectionDone):
self.__proto.connectionLost(why)
main proxy-server class:
class SSDMProtocol(LineReceiver)
...
After SSDMProtocol object (call obSSDMProtocol) parse headers it invoke method that open ftp connection (FTPClient from twisted.protocols.ftp) and set object of FTPManager field _ftpClientAsync and call _ftpCWDSuccees(self, protocol, fileName) with protocol = obSSDMProtocol and when file's bytes recieved invokes dataReceived(self, data) of FileReceiver object.
And when self.__proto.transport.write(data) invoked, data appends to inner buffer faster than sending back to client, therefore memory runs out. May be I can stop reading when the buffer reaches a certain size and resume reading after buffer will be all send to client? or something like that?
If you're passing a 20 gigabyte (gigabit?) string to transport.write, you're going to need at least 20 gigabytes (gigabits?) of memory - probably more like 40 or 60 due to the extra copying necessary when dealing with strings in Python.
Even if you never pass a single string to transport.write that is 20 gigabytes (gigabits?), if you repeatedly call transport.write with short strings at a rate faster than your network can handle, the send buffer will eventually grow too large to fit in memory and you'll encounter a MemoryError.
The solution to both of these problems is the producer/consumer system. The advantage that using IProducer and IConsumer gives you is that you'll never have a 20 gigabyte (gigabit?) string and you'll never fill up a send buffer with too many shorter strings. The network will be throttled so that bytes are not read faster than your application can deal with them and forget about them. Your strings will end up on the order of 16kB - 64kB, which should easily fit in memory.
You just need to adjust your use of FileReceiver to include registration of the incoming connection as a producer for the outgoing connection:
class FileReceiver(Protocol):
def __init__(self, outgoing):
self._outgoing = outgoing
def connectionMade(self):
self._outgoing.transport.registerProducer(self.transport, streaming=True)
def dataReceived(self, data):
self._outgoing.transport.write(data)
Now whenever self._outgoing.transport's send buffer fills up, it will tell self.transport to pause. Once the send buffer empties out, it will tell self.transport to resume. self.transport nows how to undertake these actions at the TCP level so that data coming into your server will also be slowed down.