I am wondering one thing; How can I serialize multiple objects to a byte array. My goal is to send the serialized object over tcp, receive it, then deserialize it, and recreate it.
My concept is:
The first thing in the byte array will be the "Packet Header" -- This will tell the receiver what type of packet it is; "Chat Message", "File Transfer", etc etc. Then after the header I will add the packet itself. Then at last there will be an "EOF Header" (This will tell the server if the whole packet is received). - The headers are enumerables(as Byte).
where you get these errors would be helpful (essential even) but it is probably related to this:
Public Shared Function Deserialize(Data As Byte()) As Packet
Dim MS As New MemoryStream(Data)
Dim BF As New BinaryFormatter
MS.Position = 0
' or
'MS.Seek(0, SeekOrigin.Begin)
Return DirectCast(BF.Deserialize(MS), Packet)
End Function
After seeding the memstream, the stream position is left at the end. You need to reset it so the BF can read all the bytes. (and you really dont need things like BOF and EOF in the serialized data - even if you are sending multiple things, if you put them in a list, they will either de/serialize in toto or not).
Also look at ProtoBuf-NET - much faster serializer making much smaller packets, and it will let you deserialize into a different assembly-culture-class which NET's BF does not do without basically tricking it.
Related
I am building a communication library based on the Net.Sockets.TcpClient class. During some unit tests I wanted to test how large a datapacket could be before running into problems. My theory was that the actual size would not matter because the TcpClient would split the data into parts because of its internal sendbuffer. But the actual size did matter because somewhere around 600KB I discovered loss of data.
What my test does is create a local server and a local client that connect with each other. Then it sends a specific (large) package in a loop to test if the server receives it well. I left out all checks but after sending the data a check runs that the data is exactly the same as what was send before looping again. So there is never more data in the pipe than the size that I specified. This code is the client part in my unit test. The server part is a whole library so I cannot post that.
Using myClient As New Net.Sockets.TcpClient()
myClient.BeginConnect("127.0.0.1", ServerPort, Nothing, Nothing)
'Code that checks if the connection has been made
'Create a large string
Dim SendString as String = StrDup(1024000, "A")
Dim SendBytes as Byte() = Text.Encoding.ASCII.GetBytes(SendString)
'Loop the test
For i As Int32 = 1 To 10000
Wait.Reset()
myClient.Client.Send(SendBytes)
Wait.WaitOne(10000) 'Wait for the server to acknowledge
'Run checks to make sure the data is good, otherwise end loop
Next
myClient.Close()
End Using
What happens is that at some random point the server does not receive all data. It looks like sending 1.024.000 bytes of data works most of the times but not always. The iteration at which it fails is random but a loop will never finish 10.000 iterations successfully. I tested the loop with 512.000 bytes and that works. I also tested 600.000 bytes and that failed. I do not know what the actual size is at which it starts failing because it does not seem to be a hard limit. I cannot figure out the problem. Is the TcpClient somehow limited or do I exceed an internal buffer of some kind? I checked the SendBufferSize of the TcpClient and it was 65536. I have no idea if that has anything to do with it. Packages larger than that buffer seem to be sending just fine.
I'm using CocoaAsyncSocket for an iOS project. I'm trying to read VarInts through an asynchronous interface. The problem is unlike something else like a String, where I can prefix a length, I don't know the length of a varint beforehand. It needs to be processed one byte at a time, but since each read operation is asynchronous other read calls may have been queued in between.
I considered reading into a buffer then processing it, say reading 5 bytes (the max length for a varint-32), and pushing extra bytes back, but that may hang unnecessarily if the varint is only 4 bytes and I'm waiting for a 5th byte to be available.
How can I do this? Also, I cannot change the protocol on the other end, to use fixed size ints.
Here's a snippet of code as Josh requested
- (void)readByte:(void (^)(int8_t))onComplete {
NSUInteger size = 1;
int32_t tag = OSAtomicAdd32(1, &_nextTag);
dispatch_async(self.dispatchQueue, ^{
[self.onCompleteHandlers setObject:(^void (NSData* data) {
int8_t x = 0;
[data getBytes:&x length:size];
onComplete(x);
}) forKey:[NSNumber numberWithInteger:((NSInteger) tag)]];
[self.socket readDataToLength:size withTimeout:-1 tag:tag];
});
}
A callback is saved in a dictionary, which is used in the delegate method socket: didReadData: withTag.
Suppose I'm reading a VarInt byte by byte:
execute read first byte for varint
don't know if we need to read another byte for a varint or not; that depends on the result of the first read
(possible) read another byte for something else
read second byte for varint, but now it's actually the 3rd byte being read
I can imagine using a flag to indicate whether or not I'm in a multipart-read, and a queue to hold reads that should be executed after the multipart-read, and I've started writing it but it's quite messy. Just wondering if there is a standard/recommended/better way to approach this problem.
in short there are 4 ways to know how much to read from a socket...
read some format that you can infer the length from like the Content-Length header... only works if the whole request can be put together before the body is sent.
read until some pattern: like \r\n\r\n at the end of the headers
read until some timeout... after you get no bytes after n seconds you flush the buffers and close the connection.
read until the server closes the connection... actually used to be pretty common.
these each have problems and I would probably lean in your case from using some existing protocol.
of course there is overhead to doing it that way, and you may find that you don't want to use any of that application level stuff and your requests may be like:
client>"doMath(2+5)\0"
server>"(7)\0"
but it is hard to answer your general question specifically.
edit:
So I looked into the varint base-128 issue a little more and I think really only a timeout or the server closing the connection will work, if you are writing these right at the TCP level which is horrible...
I have a sending application that uses TCP to send files. Sometimes these files contain one message, and other times the file may contain multiple messages. Unfortunately, I do not have access to the Sending application's code.
I am working on editing legacy code to receive these messages. I have managed to get the legacy application to accept a file when there is a single message sent. However, since I disconnect the socket after receiving a single message, the Sender gives a disconnect error.
I wrote a small process to help determine whether there was another message. If it worked, I was going to incorporate it into the code, but I had mixed results:
Dim check(1) As Byte
If (handler.Receive(check, SocketFlags.Peek) > 0) Then
Dim bytesRec As Integer
ReDim bytes(1024)
bytesRec = handler.Receive(bytes)
End If
If there is another message being sent, this will detect it. However, if the file only has a single message, it locks up on Receive until I send another file, and then it is accepted.
Is there a way to tell if there is another message pending that will not lock up if the stream is empty?
I won't post all of the code for accepting the message, as it is a legacy rat's nest, but the general idea is below:
s2 = CType(ar.AsyncState, Socket)
handler = s2.EndAccept(ar)
bytes = New Byte(1024) {}
Dim bytesRec As Integer = handler.Receive(bytes)
' Send Ack/Nak.
numAckBytesSent = handler.Send(myByte)
Thank you in advance for any assistance.
Socket.Select can be used as a quick way of polling a socket for readability. Pass in a timeout of 0 seconds, and the socket in question in the readability list, and it will simply check and report back immediately.
Two other options might be to set Socket.ReceiveTimeout on your socket, or make the socket non-blocking using Socket.Blocking, so that you can find out (as part of the Receive call) whether there is incoming data. These look a bit inconvenient to do in .NET, though, as they throw exceptions rather than simply returning a value, which might make the code a little longer.
Just keep reading. If there is nothing left you will get an end-of-stream indication of some kind, depending on your API.
I have a streamed WCF service. In one operation, I receive a file, for upload purposes.
If I try to do something like this
request.FileContent.Length
Then I receive an OperationNotSupported exception. That's Ok.
But how could I get the file size without actually transfering it entirely?
I know I could send this information along with the call, as a Header, but I don't want to go this way.
If WCF is able to limit the request size trough maxReceiveMessageSize. How can I use the same information to check the message/stream size?
In general, you can't know the size of a byte stream without reading it all and counting the bytes, unless there is data at the start of the stream which tells you how many bytes there are in the entire stream, or some other out-of-band way to communicate the length of the stream, such as in the WCF message headers. You will have to go with the Header approach if you want to know the size without reading the stream.
The WCF maxReceiveMessageSize works by counting the bytes as they are received and throwing an exception if the limit is exceeded... it doesn't know the stream length either, and can't pre-emptively prevent the message being received without first reading the maximum allowed number of bytes.
But how could I get the file size without actually transfering it entirely? I know I could send this information along with the call, as a Header, but I don't want to go this way.
You're going to have to send the size of the byte stream down the pipe first there is no other way. (if there was some inbuilt way thats all it would be doing anyway)
It doesn't add much complexity to prepend it to the stream:
var bytes = File.ReadAllBytes("somefile.txt");
stream.Write(BitConverter.GetBytes((Int32)bytes.Length), 0, 4);
stream.Write(bytes, 0, bytes.Length);
and then on the other side when reading the stream:
byte[] fileLengthBytes =new byte[4];
stream.Read(fileLengthBytes, 0, 4);
int length = BitConverter.ToInt32(fileLengthBytes, 0);
//you know the size of the file now, log it or show the user
var fileBytes = new byte[length];
stream.Read(fileBytes, 0, fileBytes.Length);
this is only an example - you may not want to create a byte[] buffer if your stream is large.
I'd like to reduce the message size when sending serialized integer over the network.
In the below section buff.Length is 256 - to great an overhead to be efficient!
How it can be reduced to the minimum (4 bytes + minimum overhead)?
int val = RollDice(6);
// Should 'memoryStream' be allocated each time!
MemoryStream memoryStream = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(memoryStream, val);
byte[] buff = memoryStream.GetBuffer();
Thanks in advance,
--- KostaZ
Have a look at protobuf.net...it is a very good serialization lib (you can get it on NuGet). Also, ideally you should be using a "using" statement around your memory stream.
To respond to the comment below, then the most efficient method depends on your use case. If you know exactly what you need to serialize and don't need a general purpose serializer then you could write your own binary formatter, which might have no overhead at all (there is some detail here custom formatters).
This link has a comparison of the BinaryFormatter and protobuf.net for your reference.