I read redis source code recently, and I'm now studying the networking codes.
Redis use nonblock mode and epoll(or something simliar) for networking data read/write. When read data event arrived,"readQueryFromClient" function will be called, and in this function request data will be readed into buffer.
In "readQueryFromClient" function, if there are really data arrived, data will be readed into buffer through one 'read' function, and then the request will be handled.
nread = read(fd, c->querybuf+qblen, readlen); // **one read function**
//... some other codes to check read function retuen value
processInputBuffer(c);// **request will be handled in this function**
My question is: how redis ensure all request data can be readed into buffer by only one 'read' function call, maybe all data will be gotten by more 'read' function call?
processInputBuffer(c);// request will be handled in this function
That part is not true. Redis protocol is designed to include length of every chunk of data passed around. So the server always knows how much data it has to read to make a complete request out of it. Inside processInputBuffer if neither processInlineBuffer nor processMultibulkBuffer returns REDIS_OK (i.e. request terminator was not found in the buffer/not enough arguments), control simply falls out of the function. All that processInputBuffer did in this case was filling up a chunk of the client buffer and updating the parsing state. Then, on the next iteration of event loop, in the call to aeProcessEvents, if there is unread data remaining in the socket buffer, readQueryFromClient callback will be triggered again to parse the remaining data.
Related
Hi I am really new at embedded programming. I am using stm32cube IDE. I am trying to read a string to the DMA buffer but I need to implement start-marker as well as an end-marker. For example I only need to read the serial data in between '<' and '>' to the DMA buffer and when soon as it gets to the end-marker I want to call following call back function and process the data.
void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart)
{
//Process Data
}
I want this to run in the background all the time. Is this possible?
No, the DMA controller cannot check the value of the data and stop or interrupt when a delimiter or marker byte is received. The DMA controller can only copy the received byte to memory. If you want to read variable length packets between delimiters then you need to use the CPU to check whether each byte is a delimiter. You can use the UART's RX interrupt to check for the delimiter as each byte is received.
DMA might be useful for receiving a continuous stream of bytes or a packet of known length.
I'm setting up a market data back-testing using Chronicle Queue (CQ), reading data from a binary file then writing into a single CQ and simultaneously reading the data from that CQ and dumping the statistics. I am doing a POC to replace our existing real-time market data feed handler worker queue.
While doing basic read/writes testing on Linux/SSD setup, I see reads are lagging behind writes - in fact latency is accumulating. Both Appender and Tailer are running as separate processes on same host.
Would like to know, if there is any issue in the code I am using?
Below is the code snippet -
Writer -
In constructor -
myQueue = SingleChronicleQueueBuilder.binary(queueName).build();
myAppender = myQueue.acquireAppender();
In data callback -
myAppender.writeDocument(myDataPacket);
myQueue.close();
where myDataPacket is Java object wrapping the byte[] and other fields.
Tailer -
In Constructor -
myQueue = SingleChronicleQueueBuilder.binary(queueName).build();
myTailer = myQueue.createTailer();
In Read method -
while (notLastRecord)
{
if(myTailer.readDocument(myDataPacket))
{
notLastRecord = ;
//do stuff
}
}
myQueue.close();
Any help is highly appreciated.
Thanks,
Pavan
First of all I assume by "reads are lagging behind writes - in fact latency is accumulating" you mean that for every every subsequent message, the time the message is read from the queue is further from the time the event was written to the queue.
If you see latency accumulating like that, most likely the data is produced much quicker then you can consume it which from the use case you described is very much possible - if all you need at the write side is parsing simple text line and dump it into a queue file, it's quick, but if you do some processing when you read the entry from the queue - it might be slower.
From the code it's not clear what/how much work your code is doing, and the code looks OK to me, except you probably shouldn't call queue.close() after each appender.writeDocument() call but most likely you are not doing this otherwise it would blow up.
Without seeing actual code or test case it's impossible to say more.
I'm using CocoaAsyncSocket for an iOS project. I'm trying to read VarInts through an asynchronous interface. The problem is unlike something else like a String, where I can prefix a length, I don't know the length of a varint beforehand. It needs to be processed one byte at a time, but since each read operation is asynchronous other read calls may have been queued in between.
I considered reading into a buffer then processing it, say reading 5 bytes (the max length for a varint-32), and pushing extra bytes back, but that may hang unnecessarily if the varint is only 4 bytes and I'm waiting for a 5th byte to be available.
How can I do this? Also, I cannot change the protocol on the other end, to use fixed size ints.
Here's a snippet of code as Josh requested
- (void)readByte:(void (^)(int8_t))onComplete {
NSUInteger size = 1;
int32_t tag = OSAtomicAdd32(1, &_nextTag);
dispatch_async(self.dispatchQueue, ^{
[self.onCompleteHandlers setObject:(^void (NSData* data) {
int8_t x = 0;
[data getBytes:&x length:size];
onComplete(x);
}) forKey:[NSNumber numberWithInteger:((NSInteger) tag)]];
[self.socket readDataToLength:size withTimeout:-1 tag:tag];
});
}
A callback is saved in a dictionary, which is used in the delegate method socket: didReadData: withTag.
Suppose I'm reading a VarInt byte by byte:
execute read first byte for varint
don't know if we need to read another byte for a varint or not; that depends on the result of the first read
(possible) read another byte for something else
read second byte for varint, but now it's actually the 3rd byte being read
I can imagine using a flag to indicate whether or not I'm in a multipart-read, and a queue to hold reads that should be executed after the multipart-read, and I've started writing it but it's quite messy. Just wondering if there is a standard/recommended/better way to approach this problem.
in short there are 4 ways to know how much to read from a socket...
read some format that you can infer the length from like the Content-Length header... only works if the whole request can be put together before the body is sent.
read until some pattern: like \r\n\r\n at the end of the headers
read until some timeout... after you get no bytes after n seconds you flush the buffers and close the connection.
read until the server closes the connection... actually used to be pretty common.
these each have problems and I would probably lean in your case from using some existing protocol.
of course there is overhead to doing it that way, and you may find that you don't want to use any of that application level stuff and your requests may be like:
client>"doMath(2+5)\0"
server>"(7)\0"
but it is hard to answer your general question specifically.
edit:
So I looked into the varint base-128 issue a little more and I think really only a timeout or the server closing the connection will work, if you are writing these right at the TCP level which is horrible...
I have a sending application that uses TCP to send files. Sometimes these files contain one message, and other times the file may contain multiple messages. Unfortunately, I do not have access to the Sending application's code.
I am working on editing legacy code to receive these messages. I have managed to get the legacy application to accept a file when there is a single message sent. However, since I disconnect the socket after receiving a single message, the Sender gives a disconnect error.
I wrote a small process to help determine whether there was another message. If it worked, I was going to incorporate it into the code, but I had mixed results:
Dim check(1) As Byte
If (handler.Receive(check, SocketFlags.Peek) > 0) Then
Dim bytesRec As Integer
ReDim bytes(1024)
bytesRec = handler.Receive(bytes)
End If
If there is another message being sent, this will detect it. However, if the file only has a single message, it locks up on Receive until I send another file, and then it is accepted.
Is there a way to tell if there is another message pending that will not lock up if the stream is empty?
I won't post all of the code for accepting the message, as it is a legacy rat's nest, but the general idea is below:
s2 = CType(ar.AsyncState, Socket)
handler = s2.EndAccept(ar)
bytes = New Byte(1024) {}
Dim bytesRec As Integer = handler.Receive(bytes)
' Send Ack/Nak.
numAckBytesSent = handler.Send(myByte)
Thank you in advance for any assistance.
Socket.Select can be used as a quick way of polling a socket for readability. Pass in a timeout of 0 seconds, and the socket in question in the readability list, and it will simply check and report back immediately.
Two other options might be to set Socket.ReceiveTimeout on your socket, or make the socket non-blocking using Socket.Blocking, so that you can find out (as part of the Receive call) whether there is incoming data. These look a bit inconvenient to do in .NET, though, as they throw exceptions rather than simply returning a value, which might make the code a little longer.
Just keep reading. If there is nothing left you will get an end-of-stream indication of some kind, depending on your API.
I'm executing 4 startup commands and also expecting to receive 4 responses. The server is already implemented and another dev who is developing android, is able to receive those 4 separate responses, however, I'm getting 2 good responses (separate) and then 3rd and 4th responses come as one response. I'v placed NSLog of NSData result in completeCurrentRead, and it outputs me merged packet "0106000000000b0600000000" instead of separate packets "010600000000" and "0b0600000000". I'v also tested those 3rd and 4th commands separatedly (only one at a time) and everything is OK with the server, it sends them separately, however there occurs merge (with 3rd and 4th) if all four commands are executed in a line. Any ideas?
UPDATE: I think I'v traced to the problem roots. There's a call that reads packet data from a stream in doBytesAvailable method:
CFIndex result = [self readIntoBuffer:subBuffer maxLength:bytesToRead];
And in readIntoBuffer:maxLength, there's a call (length == 256) :
return CFReadStreamRead(theReadStream, (UInt8 *)buffer, length);
So, CFReadStreamRead returns incorrect length of packet - it return length of 12 (instead of 6), and also grabs merged data. Hm, what might causing CFReadStreamRead to read two packets into one, instead of reading them separately...
UPDATE2: I'm using onSocket:didReadData:withTag: delegate method and expecting to receive response data with the tag of request I performed. I have realized recently, streams are streams, not packets but how I can solve that? Server responses does not have terminating chars at start and end of response, just response size, that comes as 2 - 5 bytes. I can cut the first part of response (first packet) and ignore the second part but how AsyncSocket will make another callback with the second part of the response (second packet)? If I will cut only the first parts and ignore the second then IMHO the second "packet" will be lost...
How to cut the first part of response and tell AsyncSocket to make another callback with tag and the second part of response as separate callback?
UPDATE3: In onSocket:didReadData:withTag:, I manually cut merged response, handle the first part (first packet) and then at the end, throwing a call to onSocket:didReadData:withTag: again:
if (isMergedPacket) {
...
[self onSocket:sock didReadData:restPartOfTheResponse withTag:myCommandTag];
}
However, it looks like AsyncSocket itself pairs every request packet with its response packet (via AsyncReadPacket class) using tags. So, my manual cutting works, but AsyncSocket does not know that I already handled both packets, and it still tries to read the second packet. So, I'm getting sock:shouldTimeoutReadWithTag:... callback which is called when a read operation has reached its timeout without completing.
Found solution. It's not necessary to change and dig into AsyncSocket. You just need to define the length of each response - how much bytes are you interested in reading and getting your callback. More info you can on other post here