I’m using SSL for reading data from various remote services over secure websockets as follows: I create the socket, embed it in the SSL context and add the socket to the reading list for Unix.select. When the socket fires, I use Ssl.read to get the data.
4 services are working well. And with one I get Ssl.Read_error.Error_syscall: error:00000000:lib(0):func(0):reason(0) after receiving each websocket frame (size ~5-6Kb). By the way, frames here are much bigger than on other services, but I’m not sure it’s the reason.
I ignore syscall errors (and most probably loose some data) because frames continue to arrive. Then, always after one minute I get Ssl.Read_error.Error_zero_return: error:00000000:lib(0):func(0):reason(0), which means the peer closed SSL socket for writing and I have to restart the process because no new data will be received from this socket.
Problem is perfectly reproducible. At the same time examples for this service and my own test implementation with Node.JS receive the data for hours without any problems.
I assume I do something wrong or setup socket/SSL too straightforward (see below).
Any help or ideas would be strongly appreciated.
let sock = Unix.socket PF_INET SOCK_STREAM 0 in
let laddr = Unix.inet_addr_of_string p.interface in
Unix.bind sock (ADDR_INET (laddr,0));
Unix.connect sock addr;
let (sock, res) =
let req = Bytes.of_string http_request in
if ssl then begin
Ssl.init ();
let ctx = create_context TLSv1_2 Client_context in
let sock = Ssl.embed_socket sock ctx in
Ssl.connect sock;
(SslSock sock, (write sock req 0 http_request_len))
end else
(UnixSock sock, (Unix.write sock req 0 http_request_len))
WireShark did the trick: this “bad” service sends two websocket frames in one tcp packet where second frame has zero payload length. Naturally, my Websocket implementation improperly handled frames with zero payload which lead to missing of Ping frames and closing of TCP connection by remote server.
Related
I've implemented an unsecured mosquitto broker which works fantastically to send large amount of data periodically (~200kb file once per minute) over port 1883.
Since i've implemented TLS, the broker seems to reject data >128kb automatically over port 8883 despite setting the message_size_limit = 0.
heres my mosquitto.conf:
listener 1883 localhost
listener 8883
certfile /etc/letsencrypt/live/example.com/cert.pem
cafile /etc/letsencrypt/live/example.com/chain.pem
keyfile /etc/letsencrypt/live/example.com/privkey.pem
And heres my script which is used to test the broker which works fine without TLS over 1883
client = mqtt.Client("test")
client.tls_set(certfile="./mqtt/cert.pem", keyfile="./mqtt/key.pem")
client.connect("example.com", 8883)
#publish file as zip
with open("./mqtt/20180319171000.gz", 'rb') as f:
byte_array = f.read()
m.update(byte_array)
file_hash = m.hexdigest()
payload_json = {'byte_array': byte_array, 'md5': file_hash}
client.publish("topic", pickle.dumps(payload_json), 0)
time.sleep(1)
client.disconnect()
Is there a limit on the payload size with TLS or is something wrong with my setting/script?
The problem here is that the MQTT Client loop is not being run.
When the payload is larger than can fit in a single TCP packet the call to client.publish() needs to queue up the rest of the message and this is then broken up into multiple packets and sent via the client loop.
The correct response is not to increase the keepalive period. There are 2 ways to solve this with the python Paho library.
First you can use the Publish class instead of the Client class. This includes a one function that handles all background tasks required to ensure the whole message is delivered.
import paho.mqtt.publish as publish
tls_opt = {
'certfile':"./mqtt/cert.pem",
'keyfile':"./mqtt/key.pem"
}
with open("./mqtt/20180319171000.gz", 'rb') as f:
byte_array = f.read()
m.update(byte_array)
file_hash = m.hexdigest()
payload_json = {'byte_array': byte_array, 'md5': file_hash}
publish.single("topic", payload=pickle.dumps(payload_json), qos=0, hostname="example.com", port=8883, tls=tls_opt)
Second is to start the network loop as follows:
client = mqtt.Client("test")
client.tls_set(certfile="./mqtt/cert.pem", keyfile="./mqtt/key.pem")
client.connect("example.com", 8883)
client.loop_start()
#publish file as zip
with open("./mqtt/20180319171000.gz", 'rb') as f:
byte_array = f.read()
m.update(byte_array)
file_hash = m.hexdigest()
payload_json = {'byte_array': byte_array, 'md5': file_hash}
client.publish("topic", pickle.dumps(payload_json), 0)
time.sleep(1)
client.loop_stop()
client.disconnect()
An old question, but I experienced the same issue with large messages (>500kb). My solution was to increase the keepalive on the client from (default) 60 to 300 sec.
This is probably related to timeout for TLS encrypton on large messages taking longer than keepalive.
Edit: Added python-code for connect:
client.connect(
host="example.com",
port=8883,
keepalive=300)
Update:
I found this question looking for answers to a problem that looked similar to mine, that is MQTT publish failed for large (> 500kb) paylods when using MQTT TLS. As #hardillb indicates in his answer, OP is missing client.loop_start(). This does not fix my problem, however.
keepalive should have no impact, but that is just not the case. Increasing the value definetely fixes the problem. My theory is that the broker failes the connection on timeout because it tries to PING the client, but the client refuses to respond withion keepalive because it is busy trying to encrypt the message. This is just a theory, though.
I've created some test code to illustrate the problem. I also included a "last will" to check if the connection is lost without a proper disconnect(), and it seems to fit my theory. Using too small keepalive definetely activates the last will on the broker, indicating a "timeout".
Increasing the keepalive does not activate "last will" on the broker.
Here is my code I used to test different keepalive values and payload sizes.
import paho.mqtt.client as mqtt_client
import time
from datetime import datetime
password = 'somepassword'
user = 'someuser'
address = 'somebroker.no'
connected = False
def on_connect(client, userdata, flags, rc):
global connected
connected = True
print("Connected!")
def on_disconnect(client, userdata, rc):
global connected
connected = False
print("Disconnected!")
client = mqtt_client.Client()
client.username_pw_set(user, password)
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.tls_set()
client.will_set(topic='tls_test/connected', payload='False', qos=0, retain=True)
client.connect(host=address, port=8883, keepalive=100)
client.loop_start()
while not connected:
time.sleep(1)
topic = 'tls_test/abc'
payload = 'a'*1000000
start = time.time()
print('Start: {}'.format(datetime.fromtimestamp(start).strftime('%H:%M:%S')))
result = client.publish(topic='tls_test/connected', payload='True', qos=0, retain=True)
result = client.publish(topic=topic, payload=payload)
if result.rc != 0:
print("MQTT Publish failed: {}".format(result.rc))
exit()
client.loop_stop()
client.disconnect()
stop = time.time()
print('Stop: {}, delta: {} sec'.format(datetime.fromtimestamp(stop).strftime('%H:%M:%S'), stop-start))
Usig the code above (keepalive=100), it sends 1000.000 bytes and tls_test/connected has the value True on the broker after finishing.
Data is transmitted successfully, The console output is:
python3 .\mqtt_tls.py
Connected!
Start: 10:51:16
Disconnected!
Stop: 10:53:01, delta: 105.57992386817932 sec
Reducing the keepalive (keepalive=10), transmission fails and tls_test/connected has the value False on the broker after finishing.
Data transmit fails, and the console output is:
python3 .\mqtt_tls.py
Connected!
Start: 11:08:23
Disconnected!
Disconnected!
Stop: 11:08:43, delta: 19.537118196487427 sec
Tailing /var/log/mosquitto/mosquitto.log on the broker gives the following error message:
1612346903: New client connected from x.x.x.x as xxx (c1, k10, u'someuser').
1612346930: Socket error on client xxx, disconnecting.
My conclusion is: keepalive does have an impact on large payloads when using TLS
netty 4.0.24
I am passing XML over UDP. When receiving the UPD packet, the packet is always of length 2048, truncating the message. Even though, I have attempted to set the receive buffer size to something larger (4096, 8192, 65536) but it is not being honored.
I have verified the UDP sender using another UDP ingest mechanism. A standalone Java app using java.net.DatagramSocket. The XML is around 45k.
I was able to trace the stack to DatagramSocketImpl.createChannel (line 281). Stepping into DatagramChannelConfig, it has a receiveBufferSize of whatever I set (great), but a rcvBufAllocator of 2048.
Does the rcvBufAllocator override the receiveBufferSize (SO_RCVBUF)? Is the message coming in multiple buffers?
Any feedback or alternative solutions would be greatly appreciated.
I also should mention, I am using an ESB called vert.x which uses netty heavily. Since I was able to trace down to netty, I was hopeful that I could find help here.
The maximum size of incoming datagrams copied out of the socket is actually not a socket option, but rather a parameter of the socket read() function that your client passes in each time it wants to read a datagram. One advantage of this interface is that programs accepting datagrams of unknown/varying lengths can adaptively change the size of the memory allocated for incoming datagram copies such that they do not over-allocate memory while still getting the whole datagram. (In netty this allocation/prediction is done by implementors of io.netty.channel.RecvByteBufAllocator.)
In contrast, SO_RCVBUF is the size of a buffer that holds all of the datagrams your client hasn't read yet.
Here's an example of how to configure a UDP service with a fixed max incoming datagram size with netty 4.x using a Bootstrap:
import io.netty.bootstrap.Bootstrap;
import io.netty.channel.ChannelOption;
import io.netty.channel.FixedRecvByteBufAllocator;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioDatagramChannel;
int maxDatagramSize = 4092;
String bindAddr = "0.0.0.0";
int port = 1234;
SimpleChannelInboundHandler<DatagramPacket> handler = . . .;
InetSocketAddress address = new InetSocketAddress(bindAddr, port);
NioEventLoopGroup group = new NioEventLoopGroup();
Bootstrap b = new Bootstrap()
.group(group)
.channel(NioDatagramChannel.class)
.handler(handler);
b.option(ChannelOption.RCVBUF_ALLOCATOR, new FixedRecvByteBufAllocator(maxDatagramSize));
b.bind(address).sync().channel().closeFuture().await();
You could also configure the allocator with ChannelConfig.setRecvByteBufAllocator
I've got two test programs (A & B)that are nearly identical, that use the same boost asio UDP async code.
Here is the receive call:
_mSocket.async_receive_from(
boost::asio::buffer(_mRecvBuffer), _mReceiveEndpoint,
boost::bind(&UdpConnection::handle_receive, this,_mReceiveEndpoint,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
// _mReceiveEndpoint is known and good. the buffer is good too.
// here's the handler
void handle_receive(const udp::endpoint recvFromEP, const boost::system::error_code& error,std::size_t bytesRecv/*bytes_transferred*/)
{
boost::shared_ptr<std::string> message(new std::string(_mRecvBuffer.c_array(),bytesRecv));
if (!error)
{
doSomeThingGood();
}
else {
cerr << "UDP Recv error : " << error << endl;
}
}
So here's what happens, all on localhost.
If I start program 'A' first, then program 'B', 'A' gives a UDP Recv error : server:10061.
Program 'A' continues to send just fine and 'B' receives just fine.
You can swap 'A' and 'B' in the above sentence and it is still true.
IF I attempt a reset of the bad read condition by calling mSocket.async_receive_from again, I get error 10054.
I've looked these errors up on the web..... not very helpful.
Anybody have any ideas as to what these mean, and how I can recover inside the program if this condition occurs? Is there a way to reset the socket?
Sanity check.... can both programs operate on loopback with only two ports?
A send = 20000, A receive = 20001
B send = 20001, B receive = 20000
TL;DR
It appears as though if I try to listen before I'm sending, I get an error & I can't recover from it. If I listen after sending, I'm fine.
-- EDIT - It appears that McAfee host intrusion prevention is doing something nasty to me.... If I debug in VS2010, I get stuck in their DLL.
Thanks
In my receive handler, I wasn't calling _mSocket.async_receive_from() again.... I just printed the error and exited.
Silly mistake, just posting here in case it helps anyone else.
Also for a similar problem with a different resolution:
_mSocket.set_option(boost::asio::socket_base::reuse_address(true));
helps if you have multiple listeners.
Several sources explain that you should use SO_REUSEADDR on windows. But none mention that it is possible to receive UDP message with and without binding the socket.
The code below binds the socket to a local listen_endpoint, that is essential, because without that you can and will still receive your UDP messages, but by default your will have exclusive ownership of the port.
However if you set reuse_address(true) on the socket (or on the acceptor when using TCP), and bind the socket afterwards, it will enable multiple applications, or multiple instances of your own application to do it again, and everyone will receive all messages.
// Create the socket so that multiple may be bound to the same address.
boost::asio::ip::udp::endpoint listen_endpoint(
listen_address, multicast_port);
// == important part ==
socket_.open(listen_endpoint.protocol());
socket_.set_option(boost::asio::ip::udp::socket::reuse_address(true));
socket_.bind(listen_endpoint);
// == important part ==
boost::array<char, 2000> recvBuffer;
socket_.async_receive_from(boost::asio::buffer(recvBuffer), m_remote_endpoint,
boost::bind(&SocketReader::ReceiveUDPMessage, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)
source:
http://www.boost.org/doc/libs/1_45_0/doc/html/boost_asio/example/multicast/receiver.cpp
I try to make peer to peer connection between a server and a client. I send local video stream, through peer connection,
from the client to the server and when once the server received it in onAddStream() event it takes the stream and add it to peer connection with addStream() to send it back to the client, where it came from initially. The source on the server side looks like this:
void ServerPeerConnection::OnAddStream(webrtc::MediaStreamInterface* stream)
{
this->AddStream(stream);
}
I know it seems senseless but it's the first step to implement before to go further.
So I'm asking you if it's allowed to the sequence? Should I addStream() before SDP parameters are transferred between the peers or can I call addStream() after. Now doing so I have the following error log:
Error(statscollector.cc:192): The SSRC 2128160837 is not associated with a track
Error(statscollector.cc:192): The SSRC 0 is not associated with a track
Transport::ConnectChannels_w: No local description has been set. Will generate o
ne.
Jingle:Channel[audio|1|]: NULL DTLS identity supplied. Not doing DTLS
Jingle:Channel[audio|2|]: NULL DTLS identity supplied. Not doing DTLS
You can attach the remote stream like this:
var MediaStream = window.webkitMediaStream || window.MediaStream;
firstPeer.onaddstream = function(remoteSteam) {
remoteStream = new MediaStream(remoteSteam.audioTracks, remoteSteam.videoTracks);
otherPeer.addStream(remoteStream); /* attaching remote stream */
};
https://github.com/muaz-khan/WebRTC-Experiment/issues/2
I use C# program for client UDP application. Application listens for a connection, and then communicates.
Socket udpClient = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
udpClient.Bind(new IPEndPoint(IPAddress.Any, ListenPort));
udpClient.Blocking = true;
int count = 0;
while (count == 0) udpClient.ReceiveFrom(receiveBuffer, ref ePoint);
udpClient.SendTo(data, endPoint);
udpClient.ReceiveFrom(receiveBuffer, ref ep);
...
I use Wireshark to debug the application. The problem is that after sometime my application starts sending malformed STUN packets, and I think that because of that they get rejected by a router on the internet.
The question: is it possible to prevent sending malformed UDP/STUN packets?
When your application sends malformed UDP packets, it has a bug. The minimal fragment of your code has only one SendTo call. You can add a check function for the content/length of data.
BTW: UDP is connectionless. I would say, your application waits for a request or a kind of start command not for a connection.