I am trying to understand the use case in which EngineCPUUtilization Matric is more relevant than CPUUtilization for AWS ElastiCache. Since Redis is single threaded, it is expected that EngineCPUUtilization would peak apriori the CPUUtilization, but in my monitoring I can see both these matrices having similar values. Can any one help me understand the exact use of these two matrices. Also even though, the memory is available, SwapUsages matric has non zero value. why is this? and does SwapUsages matric is used even if physical memory is available which seems contrary to usual swap definition.
Related
I am trying to understand how different Load Balancing strategy works.
One way is to use consistent hashing algorithm where we divide the entire space into multiple virtual nodes and each physical node takes a set of vnodes.
I am not able to understand how would hot partition problem will be solved? It could happen that one particular node is busier than the rest?
Can someone add their experience handling similar use cases? Any pointer to the right document/literature would be helpful.
I came to 2 options on how to solve the problem I have with (AWS ElastiCache (REDIS)).
I was able to find all the differences for these two approaches in scope of Time complexity (Big O) and other stuff.
However, there is one question that still bothers me:
Is there any difference for REDIS cluster (in memory consumption, CPU or any other resources) to handle:
500K larger Sorted Sets (https://redis.io/commands#sorted_set) containing ~100K elements each
48MLN smaller Simple Sets (https://redis.io/commands#set) containing ~500 elements each
?
Thanks in advance for the help :)
You are comparing two different data types, it is better to be benchmarked to decide which one's memory consumption is better with info memory. But I assume both are used with the same length for entries inside.
If you use the config set-max-intset-entries and stay in the limits of it while adding to this set(let's say 512), then your memory consumption will be lower than your first option(same value lengths and equality of the total entries). But it doesn't come for free.
The documentation states that
This is completely transparent from the point of view of the user and API. Since this is a CPU / memory trade off it is possible to tune the maximum number of elements and maximum element size for special encoded types using the following redis.conf directives.
I normally resolve this by allocating more memory, however sometimes I find mys elf allocating quite a lot and I wonder if there are other alternatives I can use to solve this, is there a way to make the map jobs smaller, perhaps in aggregate they use same memory, but not all the time
I am mostly trying to learn. I can provide an example but I am not asking about a query or a specific instance, more in general
In the module tensorflow.contrib.memory_stats:
As I understand MaxBytesInUse, it will give me the maximum memory usage so far since the lifetime of the session, right?
I was wondering about BytesInUse. I thought it would give me the maximum memory usage from a single session run but that doesn't seem to be the case because it returns a quite low number but I know that I'm quite close to the limit of my memory. So I guess it returns the current memory usage when that operation kernel is executed? But that is then very much non-deterministic. If it starts right at the beginning, it will be low (only the variables). Via tf.control_dependencies, I could make it such that it executes at the very end, but there TF likely will have released any used memory already, so it will also be low (again only the variables), right? So, where does it make sense to put it in the graph? But probably this never will give me what I want, i.e. the maximum memory usage during a single session run, right?
So, is there any way to get what I want, i.e. the maximum memory usage of a single session run?
I have a distributed application across which I'd like to replicate a single, eventually consistent state. The data is suitable for a CRDT (http://pagesperso-systeme.lip6.fr/Marc.Shapiro/papers/RR-6956.pdf) which has the excellent property that each node, given the same set of messages, will deterministically converge to the same value without complicated consensus protocols.
However, I need another messaging/log layer that will ensure that each node actually sees every message, even in the face of adverse network conditions.
Specifically, I'm looking for an algorithm that has the following properties:
Works on an asynchronous network.
Nodes are only necessarily aware of their neighbors, not the whole network.
Nodes may be added or dropped at any time (that is, the network is not of a fixed size or topology).
The network can be acyclic (this can be a requirement, if necessary).
Is capable of bringing up to date a node that has become behind due to temporary network outage or dropped messages.
Is capable of bringing a new, empty node joining the cluster up to date.
There is not a hard limit on the time taken for the network to converge on a value (that is, for every node to recieve every message), but given no partitions it should be fairly quick (in fuzzy terms, a matter of seconds, not minutes).
Is bounded in size. Algorithms that keep the entire message history (which will grow boundlessly) are unsuitable.
Is anyone aware of an algorithm with these properties?