I'm writing a very specific application protocol to enable communication between 2 nodes. Node 1 is an embedded platform (a microcontroller), while node 2 is a common computer.
Such protocol defines messages of variable length. This means that sometimes node 1 sends a message of 100 bytes to node 2, while another time it sends a message of 452 bytes.
Such protocol shall be independent on how the messages are transmitted. For instance, the same message can be sent over USB, Bluetooth, etc.
Let's assume that a protocol message is defined as:
| Length (4 bytes) | ...Payload (variable length)... |
I'm struggling about how the receiver can recognise how long is the incoming message. So far, I have thought about 2 approaches.
1st approach
The sender sends the length first (4 bytes, always fixed size), and the message afterwards.
For instance, the sender does something like this:
// assuming that the parameters of send() are: data, length of data
send(msg_length, 4)
send(msg, msg_length - 4)
While the receiver side does:
msg_length = receive(4)
msg = receive(msg_length)
This may be ok with some "physical protocols" (e.g. UART), but with more complex ones (e.g. USB) transmitting the length with a separate packet may introduce some overhead. The reason being that an additional USB packet (with control data, ACK packets as well) is required to be transmitted for only 4 bytes.
However, with this approach the receiver side is pretty simple.
2nd approach
The alternative would be that the receiver keeps receiving data into a buffer, and at some point tries to find a valid message. Valid means: finding the length of the message first, and then its payload.
Most likely this approach requires adding some "start message" byte(s) at the beginning of the message, such that the receiver can use them to identify where a message is starting.
Related
I am working on a project with a master computer connected via a CANOpen network to 4 slaves.
At each time step, the computer receives a measurement message from each slave, and sends them a control message. In total, 4 messages are received and 4 messages are sent at each time sample.
The message sent is a PDO with 6 data bytes (8 bytes including COB-ID)
The message received is a PDO with 8 data bytes (10 bytes including COB-ID)
My CAN network is configured at 1Mbit/s, and I run my program at 1000 Hz (1 ms sampling time). As the total load resulting from the messages described is 576 bits/cycle, the total load expected in the network is 576kbit/s, or 57%.
What I see, however, is that:
The controlling computer measures a load of ~86% (with minima of 68% and peaks of 100%).
A USB CAN bus analyser I connect to the network registers a traffic
of messages (count-wise) that is around half of what I nominally
expect (i.e., 4 sent, 4 received each cycle, for 50 seconds should result in 50k messages, while I only see 18-25k). Moreover, I receive
1-2 error messages per cycle from the slave devices that the
network is overloaded. Before it is pointed out, even counting the
size of these messages as part of traffic wouldn't get close to
explain the anomaly in load.
What I'd like to know is whether my way of calculating the CANOpen network load is correct. For instance, are there any protocol-specific handshakes, CRCs, or any sort of extra bytes sent to make the network simply work? It's nothing I could see in the wiki page of CANOpen, but I do know there are such appendices to messages in the original CAN bus standard.
In a CAN message, there is more than the data to be transmitted.
There is also the arbitration ID (11- or 29bits, depending on whether you use CAN 2.0A or 2.0B), there is a 15 bit CRC, an 7 bit EOF marker, the control field and also some other reserved bits.
Depending on the data, there may also be stuff bits.
Using CAN2.0B and assuming 48 bits (6 bytes) of data, you will get a message size of roughly 132 bits and roughly 151 bits for your 64 bits messages.
Summing this up, you will get roughly 1132 bits per cycle which is too much for a 1Mbit/s bus and 1000 Hz.
Hope that helps.
I have a problem in my contrib implementation for distributed TensorFlow. Not to bother you with non-relevant details, the solution applies a certain message protocol in order to utilize RDMA writes directly from/to source/destination tensors, to save memory copies on the CPU.
Let's say I have 2 sides, A and B, and A wants to receive a tensor from B.
The protocol is as follows:
A sends a REQUEST message to B.
B lookups the tensor locally (BaseRendezvousMgr::RecvLocalAsync) and sends a META-DATA response to A.
A uses the meta data to allocate the destination tensor, and sends a ACK to B containing the destination address.
B receives the ACK, and performs a remote DMA write to the destination address.
Between the REQUEST and the ACK, B keeps the local tensor alive (and Ref() > 0), by saving it in a local map (the REQUEST copies the tensor to the local map, the ACK pops it from the map).
To validate my solution, I added a checksum calculation at each step. Occasionally, I see that the checksum changes between the REQUEST and the ACK. This happens when I run PS with two workers:
Line 1 is REQUEST to worker 0.
Line 2 is ACK to worker 0.
Line 3 is REQUEST to worker 1.
Line 4 is ACK to worker 1.
The last value on each line is the checksum. The errors happens about 50% of the times. I always see it on line 4.
I also saw that the problematic tensor has a shared buffer for all step-ids (this is a given. I can't control it). So it is very likely that some other thread changed the tensor's content between lines 3 and 4, which is something I want to prevent.
So the question is how? What prevented the content from changing between lines 1 and 2, and 2 and 3? To emphasize, the time elapsed between lines 3 and 4 is less than 0.04 seconds, while the time elapsed between 2 and 3 is almost 2.5 seconds.
Thanks for your help. Code will be posted if required.
Are you using tf.Variable for the shared buffer? If so using tfe.Variable (to enable reasonable read-write semantics) or tf.get_variable(..., use_resource=True) to construct it will make any synchronization issues go away.
Otherwise this is hard to understand without knowing more about the generating graph.
I am writing a java applet that will print what a telnet client sends to the connection. Unfortunately, the client splits at 1448 characters.
The code that is proving to be a problem:
char[] l = new char[5000];
Reader r = new BufferedReader(new InputStreamReader(s.getInputStream(), "US-ASCII"));
int i = r.read(line);
I cannot change the source of what the telnet client reads from, so I am hoping it is an issue with the above three lines.
You're expecting to get telnet protocol data units from the TCP layer. It just doesn't work that way. You can only extract telnet protocol data units from the code that implements the telnet protocol. The segmentation of bytes of data at the TCP layer is arbitrary and it's the responsibility of higher layers to reconstruct the protocol data units.
The behavior you are seeing is normal, and unless you're diagnosing a performance issue, you should completely ignore the way the data is split at the TCP level.
The reason you're only getting 1448 bytes at a time is that the underlying protocols divide the transmission into packets. Frequently, this size is around 1500, and there are some bytes used for bookkeeping, so you're left with a chunk of 1448 bytes. The protocols don't guarantee that if you send X bytes in a 'single shot', that the client will receive X bytes in a single shot (e.g. a single call to the receive method).
As has been noted already in the comments above, its up to the receiving program to re-assemble these packets in a way that is meaningful to the client. In generally, you perform receives and append the data you receive to some buffer until you find an agreed-upon 'end of the block of data' marker (such as an end-of-line, new-line, carriage return, some symbol that won't appear in the data, etc.).
If the server is genuinely a telnet server--its output might be line-based (e.g. a single block of data is terminated with a 'end of line': carriage return and linefeed characters). RFC 854 may be helpful--it details the Telnet protocol as originally specified.
I want both the client and server to write and read resp. at a constant rate (which can be configured on the GUI of the client) to the buffer.
Say,
I am able to send from the client at 150 bytes per packet
Now, I should be able to read also at 150 bytes per packet on the server too
Since, both are connected through a socket, can we retrieve the socket params (set on the client size, like 150 here) from the server end, using the tcpServer object.
Or is it must to send an initial setup packet which tells about these client params and so accordingly the server can continue?
It's kinda usual to sort message sizes out at the application level. You could indeed send a 'setup message' as the first data after a successful connect. You should send this setup message in a form that will not be misunderstood due to endianness or the number of bytes received per read call. Perhaps a fixed-size messge in ASCII, maybe five bytes:
'00150'
The server can then read five bytes only, convert to integer, save it in the server-client socket object so that the server always knows how many bytes to send and then issue a read call for that number of bytes.
Alternatively, you could use a simple protocol that embeds the size into each message, eg:
SOH
"0"
"0"
"1"
"5"
"0"
[150 bytes of data]
EOT
Rgds,
Martin
I want to read specific number of bytes from udp socket. In tcp socket I can use socket.read where I can specify the amount of data to receive. I don't find similar function for UDP socket. I am using receive_from() where I can specify the amount of data to read, but if there is more data then no data is read and I get following error.
"A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram into was smaller than the datagram itself" std::basic_string<char,std::char_traits<char>,std::allocator<char> >
I am not able to find what value do I need to give for message_flags (3rd arg to receive_from) so that it will read the number of bytes specified. Currenly I am using the following code to read data but it either reads all data or no data.
size_t size=socket.receive_from(boost::asio::buffer((const void*)&discRsp,sizeof(DataStructure)),remote_endpoint,0,errors);
Try this:
socket.set_option(boost::asio::socket_base::receive_buffer_size(65536));