My understanding is that TCP is considered "reliable" because the receiver acknowledges packet receipt and requests a resend if there is any problem. My file transfer program currently sends files in 32767 byte packets, though I have experimented with all sizes. Sending a 10 meg file that requires 340 packets consistently results in three or four packets on the receiver being significantly smaller than what was sent. I always end up with a file that is very slightly different from the original.
As an example, my log records the size of all packets received:
TCP packet received (32767 bytes)
TCP packet received (32767 bytes)
TCP packet received (14600 bytes)
TCP packet received (32767 bytes)
My sending thread reads the file in 32767 byte chunks and calls a sending sub:
MyFile.Read(Buffer, 0, BufferSize)
SendTCPData(Address, Buffer)
My TCP code is very simple:
Shared Sub SendTCPData(Address As String, ByVal Data As Byte())
Dim Client As New TcpClient(Address, PortNumber)
Dim Stream As NetworkStream = Client.GetStream()
Stream.Write(Data, 0, Data.Length)
Stream.Close()
Client.Close()
End Sub
Can anyone help?
(The post "TCP Client to Server communication" does not deal with with how to handle received packet sizes, which is my question.)
TCP doesn't provide a packet interface to applications, it provides a byte stream interface. TCP does not "glue" bytes together into messages -- it's not a message protocol. If you want code that reads and writes messages using TCP, you'll have to actually write it or use someone else's code that does that.
If you know the sender is sending exactly 32,767 bytes, just keep calling TCP receive functions until you get 32,767 bytes. If you don't know exactly how many bytes the sender is going to send or can't identify the end of the data with some kind of marker, then it's impossible to know when you've gotten all of them.
For the future, before you write any networking code, it's worth taking the effort to document the protocol you're going to use. Take a look at some specifications for existing protocols that layer over TCP (such as SMTP, NNTP, FTP, or HTTP) to see what you need to decide on and document.
If you're sending files over TCP, look closely at some standard for sending files over TCP (such as FTP) and either implement that standard or choose a rational subset of it. At a minimum, reading the standard will give you an idea of the types of decisions you need to make to wind up with a protocol that works. Also, it's essential for debugging -- if the program doesn't work, without a standard it's difficult to determine if it's the server or the client that's at fault because there is no reference to compare them to.
Related
I am writing a program for receiving udp multicast packet. I came across short read. Is that applicable to udp? How do I ensure that I read one packet at a time? Is that possible?
My packet has a fixed size header followed by a variable length body.
https://www.boost.org/doc/libs/1_60_0/doc/html/boost_asio/overview/core/streams.html#:~:text=When%20a%20short%20read%20or,write()%20and%20async_write()%20.
"I came across short read"
What does that mean? You mean you "heard about it"? Or received the error?
Is that applicable to udp?
No. UDP is connectionless and DATAGRAM.
How do I ensure that I read one packet at a time?
That's how UDP works.
Is that possible?
Something else is not possible, although you can set flags that allow an incomplete datagram when the supplied buffer is too small to receive the complete datagram.
My packet has a fixed size header followed by a variable length body.
SUMMARY
The title of the page you linked was Streams, Short Reads and Short Writes. Your sockets are UDP, that is, not streams.
I have 1 server and several (maybe up to 20) clients. All clients are sending UDP datagram at random time. Each datagram is quite short (about 10B), but I must make sure all the data from each client is received correctly.
If I let all clients send datagram to the same port, and client B sends it datagram at the exact time when the server is receiving data from client A, it seems the server will miss the data from client A.
So what's the correct method to do this job? Do I need to create a listener for each of the 20 clients?
When you bind a UDP socket to a port, the networking stack will allocate a buffer for a finite number of incoming UDP packets for you, so that (assuming you call recv() in a relatively timely manner), no incoming packets should get lost.
If you want see your buffer size in terminal, you can take a look at:
/proc/sys/net/core/rmem_default for recv
and
/proc/sys/net/core/wmem_default for send
I think the default buffer size on Linux is 131071B.
On Linux, you can change the UDP buffer size (e.g. to 26214400) by (as root):
sysctl -w net.core.rmem_max=26214400
You can also make it permanent by adding this line to /etc/sysctl.conf:
net.core.rmem_max=26214400
Since each packet is only 10B, shouldnt be a problem.
If you are still worried about packet loss you could implement a protocol where your client waits for a ACK from the server or it will resend. Many protocols use such a feature, but this is only possible if timing allows it. For example in streaming data it is not useful because there is no time to resend.
or consider using tcp ( if it is an option)
I am consuming real-time data from a network stream using a blocking read as follows:
Do
NetworkStream.Read(Bytes, 0, ReceiveBufferSize)
'Do stuff with data here
Loop
Watching packets come in on the wire in Wireshark, I see that sometimes when a new packet comes in, .NET sees it immediately and unblocks, letting me process it. Other times, multiple packets will come in on the wire before the NetworkStream.Read unblocks and returns the whole lot in one go - I've seen up to 8 packets buffer before the NetworkStream read unblocks.
Is this expected behaviour? Is there a way to grab and process each packet immediately as it is received across the wire? Will an Async receive model make any difference here? Or am I just fundamentally misunderstanding the way that TCP streams work?
If sendto fails according to the manpage
"On success, these calls return the number of characters sent. On error, -1 is returned, and errno is set appropriately."
I know that with TCP that is definately the case and you should really attempt to send the remaining data as pointed out in Beej's guide to network programming.
However, partially sending a UDP packet makes no sense to me, and this comment seems to imply it.
If the message is too long to pass atomically through the underlying protocol, the error EMSGSIZE is returned, and the message
is not transmitted.
Can someone confirm for me that if I call sendto (or send) with a UDP packet that if it actually doesn't fit in the outbound buffer then I'll get -1 returned with errno set to EMSGSIZE and no partial send as with a stream (TCP) socket?
There is no hidden meaning, the function just returns the count of bytes sent. It is a standard pattern for Unix APIs. Datagrams are all or nothing delivery, receipt is more complicated if the network caused fragmentation to occur but generally the stack hides all the details and presents each complete packet as it is reconstructed.
EMSGSIZE indicates that "the socket requires that the message be sent atomically, but the size of the message to be sent makes this impossible" (see man sendto).
However, the outbound buffer being full isn't necessarily the reason - Linux (for instance) apparently won't fragment UDP packets by default (see man udp).
We use an embedded device to send packets from a serial port over a serial-to-Ethernet converter to a server. One manufacturer we use, Moxa, will always send the packets in the same manner which they are constructed. Meaning, if we construct a packet size of 255, it will always send the packet in a 255 length. The other manufacturer, Tibbo, if we send the packet size 255, it will break the packet up if it is greater than 128. This is the answer I received from the Tibbo engineers at the time:
"From what I understand and what the
engineers said, even if the other
devices provide you with the right
packet size now does not guarantee
that when implemented in other
networks the same will happen. This
is the reason why we feel that packet
size based data transfer through TCP
is not reliable as it was not the way
TCP was designed to be used."
I understand that this may not be how TCP was designed to be used, but if I create a packet of 255 bytes and TCP allows it, then how is this outside of how TCP works? I understand that at some point the packet may get broken up but if the server is expecting a certain packet size and Moxa's offering does not have the same problem as the Tibbo device.
So, is it possible to guarantee a reasonable TCP packet size?
No. TCP is not a packet protocol, it is a stream protocol. It guarantees that the bytes you send will all arrive, and in the right order, but nothing else. In particular, TCP does not give you any kind of message or packet boundaries. If you want such things, they need to be implemented at a higher level by your protocol.