I am using pipes to transfer information between two vxWorks tasks.
Here is a code sample:
Init()
{
fd = open("/pipe/mydev", O_RDWR, 0777);
...
}
taskRx()
{
...
len = read(fd, rxbuf, MAX_RX_LEN);
...
}
taskTx()
{
...
len = write(fd, txbuf, txLen);
...
}
If we send a message that is longer than MAX_RX_LEN, (ie txLen > MAX_RX_LEN) we do 2 reads to get the remainder of the message.
What we noticed is that the 2nd read didn't receive any data!
Why is that?
VxWorks' pipe mechanism is not stream based (unlike unix named pipes).
It is a layer on top of the vxWorks message Queue facility. As such, it has the same limitations as a message queue: when reading from the pipe, you are really reading the entire message. If your receive buffer does not have enough space to store the received data, the overflow is simply discarded.
When doing a receive on a message Queue or a pipe, always make sure the buffer is set to the maximum size of a queue element.
Related
in my kafka streams app, I need to re-try processing a message whenever a particular type of exception is thrown in the processing logic.
Rather than wrapping my logic in the RetryTemplate (am using springboot), am considering just simply writing the message back into the input topic, my assumption is that this message will be added to the back of the log in the appropriate partition and it will eventually be re-processed.
Am aware that this would mess up the ordering and am okay with that.
My question is, would kafka streams have an issue when it encounters a message that was supposedly already processed in the past (am assuming kafka streams has a way of marking the messages it has processed especially when exactly is enabled)?
Here is an example of the code am considering for this solution.
val branches = streamsBuilder.stream(inputTopicName)
.mapValues { it -> myServiceObject.executeSomeLogic(it) }
.branch(
{ _, value -> value is successfulResult() }, // success
{ _, error -> error is errorResult() }, // exception was thrown
)
branches[0].to(outputTopicName)
branches[1].to(inputTopicName) //write them back to input as a way of retrying
I am using TcpClient to connect to a simple TCP echo server. Messages consist of the message size in 4 bytes followed by the message itself. For instance, to send the message "hello", the server will expect "0005hello", and respond with "0005hello".
When testing under load (approximately 300+ concurrent users), adjacent requests sometimes result in responses "piling up", e.g. sending "0004good" followed by "0003day" might result in the client receiving "0004good0003" followed by "day".
In a conventional, non-WebFlux-based TCP client, one would normally read the first 4 bytes from the socket into a buffer, determine the length of the message N, then read the following N bytes from the socket into a buffer, before returning the response. Is it possible to achieve such fine-grained control, perhaps by using TcpClient's underlying Channel?
I have also considered the approach of accumulating responses in some data structure (Queue, StringBuffer, etc.) and having a daemon parse the result, but this has not had the desired performance in practice.
I solved this by adding a handler of type LengthFieldBasedFrameDecoder to the Connection:
TcpClient.create()
.host(ADDRESS)
.port(PORT)
.doOnConnected((connection) -> {
connection.addHandler("parseLengthFromFirstFourBytes", new LengthFieldBasedFrameDecoder(9999, 0, 4) {
#Override
protected long getUnadjustedFrameLength(ByteBuf buf, int offset, int length, ByteOrder order) {
ByteBuf lengthBuffer = buf.copy(0, 4);
byte[] messageLengthBytes = new byte[4];
lengthBuffer.readBytes(messageLengthBytes);
String messageLengthString = new String(messageLengthBytes);
return Long.parseLong(messageLengthString);
}
});
})
.connect()
.subscribe();
This solves the issue with the caveat that responses still "pile up" (as described in the question) when the application is subjected to sufficient load.
I am trying to code a simple consumer using librabbitmq. It is working, but when I do execute amqp_basic_consume, it consumes the entire queue.
What I want is for it to get a single message, process it and repeat.
I tried using a basic_qos to have the consumer prefetch 1 at a time, but that seems to have no effect at all.
The basic setup and loop:
// set qos of 1 message at a time
if (!amqp_basic_qos(conn, channel, 0, 1, 0)) {
die_on_amqp_error(amqp_get_rpc_reply(conn), "basic.qos");
}
// Consuming the message
amqp_basic_consume(conn, channel, queue, amqp_empty_bytes, no_local, no_ack, exclusive, amqp_empty_table);
while (run) {
amqp_rpc_reply_t result;
amqp_envelope_t envelope;
amqp_maybe_release_buffers(conn);
result = amqp_consume_message(conn, &envelope, &timeout, 0);
if (AMQP_RESPONSE_NORMAL == result.reply_type) {
strncpy(message, envelope.message.body.bytes, envelope.message.body.len);
message[envelope.message.body.len] = '\0';
printf("Received message size: %d\nbody: -%s-\n", (int) envelope.message.body.len, message );
if ( strncmp(message, "DONE",4 ) == 0 )
{
printf("XXXXXXXXXXXXXXXXXX Cease message received. XXXXXXXXXXXXXXXXXXXXX\n");
run = 0;
}
amqp_destroy_envelope(&envelope);
}else{
printf("Timeout.\n");
run = 0;
}
}
I expect to have a queue filled that I can start processing and if I hit ^C, the remaining messages are still in the queue. Instead, even if I have only processed one message, the entire queue is emptied.
This is the behavior when noAck is true. What will happen is that the messages will be pushed to the connected consumer as fast as the broker can send them, because it assumes that the consumer is able to accept them as they are acknowledged immediately upon delivery.
You would want to change noAck to false, then explicitly ack each message back to the broker in this case.
Alternatively, you could use a basic.get to pull messages from the broker one at a time as opposed to using a push-based consumer (there are folks out there who don't like this idea). Your use case will determine what is most appropriate, but based on the fact that you seem to have a full queue and fairly process-intensive messages, I would assume a basic.get would be just fine in this scenario. The question then would be to decide how often to poll when the queue is empty.
I made a setup which consists of 3 Zigbee's, 2 routers(Zigbee S2C's) and 1 coordinator(Zigbee S2). The routers are each connected to arduino nano which collects data from 2 FSR's and an IMU(frame type: zigbee transmit request and packet size 46 bytes) and sends it to the Coordinator attached to an arduino UNO. All the Xbees are in API mode 2 and working at a baud rate of 115200. I am using a library called "Simple Zigbee Library" to send all the collected data to the Coordinator. The collection and sending of data works fine except that there are packets lost in the way. The nano's sample data at a frequency of around 25Hz independently. The coordinator tries to read the data send from the zigbees(using the library of course) in every loop, but unfortunately, it seems to receive only around 40-45 samples.(Should have been 25*2=50 samples total from the 2 xbees). Can anybody suggest why this is happening. I need as less data loss as possible for my setup to achieve its motive. Any kind of help is appreciated.
P.S: It may be important to mention that the coordinator is reading the data only from one xbee in each loop.
As can be seen under the "Source" heading of this image of data received by the coordinator, "19" and "106" are the addresses of the routers and there are data packets dropped intermittently
Thank you.
void setup()
{
// Start the serial ports ...
Serial.begin( 115200 );
while( !Serial ){;} // Wait for serial port (for Leonardo only).
xbeeSerial.begin( 115200 );
// ... and set the serial port for the XBee radio.
xbee.setSerial( xbeeSerial );
// Set a non-zero frame id to receive Status packets.
xbee.setAcknowledgement(true);
}
void loop()
{
// While data is waiting in the XBee serial port ...
while( xbee.available() )
{
// ... read the data.
xbee.read();
// If a complete message is available, display the contents
if( xbee.isComplete() ){
Serial.print("\nIncoming Message: ");
printPacket( xbee.getIncomingPacketObject() );
}
}
delay(10); // Small delay for stability
// That's it! The coordinator is ready to go.
}
// Function for printing the complete contents of a packet //
void printPacket(SimpleZigBeePacket & p)
{
//Serial.print( START, HEX );
//Serial.print(' ');
//Serial.print( p.getLengthMSB(), HEX );
//Serial.print(' ');
//Serial.print( p.getLengthLSB(), HEX );
//Serial.print(' ');
// Frame Type and Frame ID are stored in Frame Data
uint8_t checksum = 0;
for( int i=10; i<p.getFrameLength(); i++){
Serial.print( p.getFrameData(i), HEX );
Serial.print(' ');
checksum += p.getFrameData(i);
}
// Calculate checksum based on summation of frame bytes
checksum = 0xff - checksum;
Serial.print(checksum, HEX );
Serial.println();
}
Although you claim to be using 115,200bps, posted code shows you opening the serial ports at 9600 baud, definitely not fast enough for 2500 bytes/second (50 packets/second * 45 bytes/packet * 110% for overhead) received from XBee and dumped by printPacket()). Remember that 802.15.4 is always 250kbps over the air, and the XBee module's serial port configuration is just for local communications with the host.
Make sure your routers are sending unicast (and not broadcast) packets to keep the radio traffic down.
You should verify that sending is working before troubleshooting code on the coordinator. Update the code on your routers to see if you get a successful Transmit Status packet for every packet sent. Aiming for 50Hz seems like a bit much -- you're trying to send 45 bytes (is that the full size of the API frame?) every 20ms.
Are you using a hardware serial port on the Arduino for both the XBee module and Serial.print()? How much time does each call to printPacket() take? If you reduce the code in printPacket() to a bare minimum (last byte of sender's address and the 1-byte frame ID), do you see all packets come through (an indication that you're spending too much time dumping the packets).
I'm concerned with the code you're using in loop. I don't know the deep internals of how the Arduino works, but does that 10ms delay block other code from processing data? What if you simplify it:
void loop()
{
xbee.read();
// Process any complete frames.
while (xbee.isComplete()){
Serial.print("\nIncoming Message: ");
printPacket( xbee.getIncomingPacketObject() );
}
}
But before going too far, you should isolate the problem would by connecting the coordinator to a terminal emulator on a PC to monitor the frame rate. If all frames arrive then there's an issue on the coordinator. If they don't, work on your router code first.
I had declare a queue like below:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length-bytes", 2 * 1024 * 1024); // Max length is 2G
channel.queueDeclare("queueName", true, false, false, args);
When the queue messages count bytes is large than 2G, It will auto remove the message on the head of the queue.
But what I expected is That it reject produce the last message and return exception to the producer.
How can I get it?
A possible workaround is check the queue size before send your message using the HTTP API.
For example if you have a queue called: myqueuetest with max size = 20.
Before send the message you can call the HTTP API in this way:
http://localhost:15672/api/queues/
the result is a JSON like this:
"message_bytes":10,
"message_bytes_ready":10,
"message_bytes_unacknowledged":0,
"message_bytes_ram":10,
"message_bytes_persistent":0,
..
"name":"myqueuetest",
"vhost":"test",
"durable":true,
"auto_delete":false,
"arguments":{
"x-max-length-bytes":20
},
then you cloud read the message_bytes field before send your message and then decide if send or not.
Hope it helps
EDIT
This workaround could kill your application performance
This workaround is not safe if you have multi-threading/more publisher
This workaround is not a very "best practise"
Just try to see if it is ok for your application.
As explained on the official docs:
Messages will be dropped or dead-lettered from the front of the queue to make room for new messages once the limit is reached.
https://www.rabbitmq.com/maxlength.html
If you think RabbitMQ should drop messages form the end of the queue, feel free to open an issue here so we can discuss about it https://github.com/rabbitmq/rabbitmq-server/issues