What is the impact of increasing the MAXMSGL parm of a receiver channel?" Does it automatically increase the amount of memory allocated for the channel, regardless of the size of the messages that flow across the channel? For a cluster-receiver channel, which typically supports multiple channel instances, does it increase the memory allocation for each channel instances? (example: if the channel is supporting 10 connections, and we increase MAXMSGL from 4MB to 100MB, does it increase memory usage from 40MB to 1GB?
We are using MQ v7.5.0.3 on AIX.
Thanks!!
The short answer is no. If the message size is 100KB that flows across the channel then you will see no difference. On the other hand, if the message size jumps to 100MB then the memory size will increase but I don't believe it will hit 1GB.
Related
I'm assuming that during replica resynchronisation (full or partial), the master will attempt to send data as fast as possible to the replica. Wouldn't this mean the replica output buffer on the master would rapidly fill up since the speed the master can write is likely to be faster than the throughput of the network? If I have client-output-buffer-limit set for replicas, wouldn't the master end up closing the connection before the resynchronisation can complete?
Yes, Redis Master will close the connection and the synchronization will be started from beginning again. But, please find some details below:
Do you need to touch this configuration parameter and what is the purpose/benefit/cost of it?
There is a zero (almost) chance it will happen with default configuration and pretty much moderate modern hardware.
"By default normal clients are not limited because they don't receive data
without asking (in a push way), but just after a request, so only asynchronous clients may create a scenario where data is requested faster than it can read." - the chunk from documentation .
Even if that happens, the replication will be started from beginning but it may lead up to infinite loop when slaves will continuously ask for synchronization over and over. Redis Master will need to fork whole memory snapshot (perform BGSAVE) and use up to 3 times of RAM from initial snapshot size each time during synchronization. That will be causing higher CPU utilization, memory spikes network utilization (if any) and IO.
General recommendations to avoid production issues tweaking this configuration parameter:
Don't decrease this buffer and before increasing the size of the buffer make sure you have enough memory on your box.
Please consider total amount of RAM as snapshot memory size (doubled for copy-on-write BGSAVE process) plus the size of any other buffers configured plus some extra capacity.
Please find more details here
I am wondering how much memory space does enabling TTL for a particular key will use. To be more precise, if I enable TTL for a key, how much memory does this settings will consume and how to check the memory consumption by setting such TTL?
TTL doesn't consume any extra space. The internal memory allocation for it is performed when creating a key, whether you use it or not.
P.S. IIRC, the TTL is a 64-bit variable, so it "costs" 8 bytes.
I was testing enqueue and dequeue rate of redis over the network which has 1Gbps LAN speed, and both the machines has 1Gbps ethernet card.
Redis version:3.2.11
lpush 1L items having 1 byte per item using python client.
Dequeuing items using rpop took around 55 secs over the network which is just 1800 dequeues sec. Whereas the same operation completes within 5 secs which I dequeue from local which is around 20,000 dequeues sec.
Enqueue rates are almost close to dequeue rate.
This is done using office network when no much usage are there. The same is observed on production environments too!
A drop of less than 3x over the network is accepted. Around 10x looks like I am doing something wrong.
Please suggest if I need to make any configuration changes on server or client side.
Thanks in Advance.
Retroactively replying in case anyone else discovers this question.
Round-trip latency and concurrency are likely your bottlenecks here. If all of the dequeue calls are in serial, then you are stacking that network latency. With 1 million calls at 2ms latency, you'd have at least 2 million ms of latency overhead, or 33 mins). This is to say that your application is waiting for the server to receive the payload, do something, and reply to acknowledge the operation was successful. Some redis clients also perform multiple calls to enqueue / dequeue a single job (pop & ack/del), potentially doubling that number.
The following link illustrates different approaches for using redis keys by different libraries (ruby's resque vs. clojure's carmine, pay note to the use of multiple redis commands that are executed on the redis server for a single message). This is likely the cause of the 10x vs. 3x performance you were expecting.
https://kirshatrov.com/2018/07/20/redis-job-queue/
An oversimplified example of two calls per msg dequeue (latency of 1ms and redis server operations take 1 ms):
|client | server
~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1ms | pop msg >--(1ms)--> receive pop request
2ms | [process request (1ms)]
3ms | receive msg <--(1ms)--< send msg to client
4ms | send del >--(1ms)--> receive del
5ms | [delete msg from queue (1ms)]
6ms | receive ack <--(1ms)--< reply with delete ack
Improving dequeue times often involves using a client that supports multi-threaded or multi-process concurrency (i.e. 10 concurrent workers would significantly reduce the overall time to completion). This ensures your network is better utilized by sending a stream of dequeue requests, instead of waiting for one request to complete before grabbing the next one.
As for 1 byte vs 500 bytes, the default TCP MTU is 1500 bytes. Subtracting TCP headers, the payload is ~ 1460 bytes (less if tunneling with GRE/IPsec, more if using jumbo frames). Since both payload sizes would fit in a single TCP packet, they will have similar performance characteristics.
A 1gbps ethernet interface can deliver anywhere between 81,274 and 1,488,096 packets per second (depending on payload size).
So really, it's a question of how many processes & threads you can run concurrently on the client to keep the network & redis server busy.
Redis is generally I/O bound, not CPU bound. It may be hitting network bandwidth limits. Given the small size of your messages most of the bandwidth may be eaten by TCP overhead.
On a local machine you are bound by memory bandwidth, which is much faster than your 1Gbps network bandwidth. You can likely increase network throughput by increasing the amount of data you grab at a time.
What's the maximum number of queues that RabbitMQ can handle on a single server?
Does it depend on RAM? Does it depends on erlang processes?
There are not any hard-coded limits inside RabbitMQ broker. The broker will utilize all available resources (unless you set limits on some of them, they are called watermarks in RabbitMQ terminology).
There are some limitations put by Erlang itself, like maximum number of concurrent processes, but if you theoretically can reach them on single node then it is always good idea to use distributed features.
There are a lot of discussions about RabbitMQ resource usage and limits,
How many queues can one broker support on RabbitMQ mailing list
Max messages allowed in a queue in RabbitMQ? on RabbitMQ mailing list
Rabbitmq - Reasonable performance/scale expectations on Server Fault
Is there a limit to the number of exchanges for rabbitmq? on Stack Overflow
P.S. There are AMQP protocol limit though. They are described in section 4.9
Limitations
The AMQP specifications impose these limits on future extensions of
AMQP or protocols from the same wire-level format:
Number of channels per connection: 16-bit channel number.
Number of protocol classes: 16-bit class id.
Number of methods per protocol class: 16-bit method id.
The AMQP specifications impose these limits on data:
Maximum size of a short string: 255 octets.
Maximum size of a long string or field table: 32-bit size.
Maximum size of a frame payload: 32-bit size.
Maximum size of a content: 64-bit size.
The server or client may also impose its own limits on resources such
as number of simultaneous connections, number of consumers per
channel, number of queues, etc. These do not affect interoperability
and are not specified.
This post can help you:
http://rabbitmq.1065348.n5.nabble.com/Max-messages-allowed-in-a-queue-in-RabbitMQ-tp26063p26066.html
Max queues allowed in RabbitMQ?
Thousands (or even tens of thousands) of queues should be no problem
at all, though each object (e.g., queues, exchanges, bindings, etc)
will take up some memory and/or disk space. By default, Erlang will
enforce a maximum number of concurrent processes (i.e., lightweight
threads) at around 32768 IIRC. Each queue is managed by its own
process and each connection can result in several more, so if you're
planning on having a very large number of active queues in a single
node (?) and using them all at the same time, then you may need to
tweak the emulator arguments rabbit passes the VM by setting +P <new
limit> to a higher limit.
You're also likely to use up many Gb just with the overhead for each
queue / connection pretty fast, so you're going to need a pretty meaty
server to handle millions of both. Tens of thousands should be no
problem at all, providing they fit into RAM.
While testing the performance of RabbitMQ, I found that after increasing the message size above around 250 KB the throughput was decreasing very quickly. Then I changed the frame_size to 5MB from 256KB and I got, as expected increase in the throughput. This increase was also for the message of smaller size i.e. for message of size 50 KB, throughput increases from 700 to 1100 messages per sec. and that for messages of size 10,000 KB it increases from 2 to 3 messages per sec.
My question is : Can somebody explain me how the messages are passed on wire from client to the broker and vice versa. If the size of message is greater than the size of frame then, I guess multiple frames will be made for that message(that's why, I think, when frame size was 256 KB and message size was increased from 256 KB there was large drop in throughput). But if the size of message is say around 1/10th the size of frame, then is it that each frame will be now holding 10 messages (if 10 messages are ready to be delivered).
If it is so, then say if 10 messages are not ready then for how much time the frame will wait so as to have 10 messages, means there has to be some time out after which the frame will deliver any number of message it is currently holding.
I think I have put all required observations in the question. Any help is warmly welcome.