Is there any limitation on the number of redisgraph databases that I can create in a single redis instance?
GRAPH.QUERY database1 "Cypher"
Is there any performance issues regarding a high number of these keys?
Thanks
The number of graphs on a single Redis server is limited by the number of keys the server can accommodate, as each graph is associated with a key.
In terms of performance, RedisGraph the module will initialise a single global thread pool which will serve all of the graph keys, obviously each graph will have its memory footprint. but this is not different than having multiple keys of different types: list, set, hash.
Related
My doubt is,
I have the same instance of Redis, with multiple databases (one for each service).
If more than one service used the same database, would the prefix search be slower? (having the data of all the services in one place and having to go through all of them, as opposed to only going through the selected base)
Partitioning in Redis serves two main goals:
It allows for much larger databases, using the sum of the memory of
many computers. Without partitioning you are limited to the amount of
memory a single computer can support.
It allows scaling the computational power to multiple cores and multiple computers, and the network bandwidth to multiple computers and network adapters.
Ref
Integration through database should be avoided, read more here: https://martinfowler.com/bliki/IntegrationDatabase.html
If you use KEYS command for it, then read this from documentation:
Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using SCAN or sets.
Redis doesn't support prefix index, but you can use sorted set to do prefix search to some degree.
Redis is in-memory store, so all reads are pretty fast as long as you model it the right way.
Better query multiple times than doing integration through db...
Btw if you have multiple services owning the same data, then you should probably model your services differently...
In general, I would avoid key prefix searching. REDIS isn't a standard database and key searches are slow.
Since REDIS is a key/value store, it's optimized as such.
To take advantage of REDIS you want to hit the desired key directly.
I expect key search time increases with the total amount of keys, so splitting the database would potentially reduce the key search time.
However, if your doing key searches, I would put the desired keys into a key list and just do a direct look up there.
desired_prefix = ["desired_prefix-a", "desired_prefix-b", ...]
lpush "prefix_x_keys" "x-a"
If you run multiple databases on a single instance, they all will try to use and acquire the same resource and memory plus they will also run their core process which will not help you.
you can think like this, 2 OS running on the same machine will never match the performance of a single OS utilizing all resources of the machine.
What you can do to increase the performance is make more tables or use the partition concept. This will not put too much data in a single table and the search will work faster.
I have a Redis cluster that supports a number of massive sorted sets. These sorted sets are operated on with very high frequency; a single node in the cluster can only reasonably support two sets.
For example, if I have 2 nodes and 3 sets, I can't tolerate all 3 of the sets being assigned to hash slots served by a single node.
Is there any way that I can reasonably ensure that these keys will be evenly distributed across the nodes in my cluster?
I believe there is no way to ensure what you need out-of-the-box.
If you google redis hotkeys you can find some literature explaining the problem. Usually the "solution" they provide is not to use big sets, but in some scenarios that may not be an valid answer.
Redis provides commands to move slots from one shard to another. But if the big keys are assigned to the same slot, the problem persists.
But maybe you can play with heys-hash-tags. It is an option provided by redis to ensure some keys to be allocated in the same slot. You can use this approach to obtain just the opposite, ensure some keys are not allocated in the same slot.
If a key contains a {...} pattern only that part of the key name is passed to the hash function that assigns the shard. You can try-test renaming your keys and adding for instance bigset:1{1}, bigset:2{2}, bigset:3{3} and find a pattern that makes each key be assigned to a different slot.
It's my understanding that best practice for redis involves many keys with small values.
However, we have dozens of keys that we'd like to have store a few MB each. When traffic is low, this works out most of the time, but in high traffic situations, we find that we have timeout errors start to stack up. This causes issues for all of our tiny requests to redis, which were previously reliable.
The large values are for optimizing a key part of our site's functionality, and a real performance boost when it's going well.
Is there a good way to isolate these large values so that they don't interfere with the network I/O of our best practice-sized values?
Note, we don't need to dynamically discover if a value is >100KB or in the MBs. We have a specific method that we could have use a separate redis server/instance/database/node/shard/partition (I'm not a hardware guy).
Just install/configure as many instances as needed (2 in the case), each managing independently on a logical subset if keys (e.g. big and small), with routing done by the application. Simple and effective - divide and converter conquer
The correct solution would would be to have 2 separate redis clusters, one for big sized keys, and another one for small sized keys. These 2 clusters could run on the same set of physical or virtual machines, aka multitenancy (You would want to do that to fully utilize the underlying cores on your machine, as redis server is single threaded). This way you would be able to scale both the clusters separately, and your problem of small requests timing out because of queueing behind the bigger ones will be alleviated.
please I need a simple explanation as regards this.
Redis claims that time complexity for get key is o(1)
As such, whether i have key value pair of 1,000,000 or 1,000,000,000,000,000 the time to get key is the same.
My question now is
I have a requirement to hold about 1 billion key-value pair, If memory is not a problem (meaning assume i have a single server with enough memory to hold that much data), is there any advantage of sharding? that is to say, will there be any performance advantage of seperating this 1billion key-value pair to 10 redis instances each holding 100million records as against just a single redis instance holding the entire records?
Thanks so much for your anticipated response
There is a definite advantage to sharding in terms of performance, as it can use multiple CPU cores (ideally, one per shard). Being (mostly) single threaded, a single instance Redis can use only one core (and a little more). Sharding effectively increases the parallelism of the deployment, thus contributing positively to performance (but adding to the administrative overhead).
Hi I'm going to be using multiple Redis instances and some sharding between instances.
My question is will performance suffer [a noticeable amount] if loading a webpage requires multiple shards accessed.
My basic overview is to have load balanced between multiple Redis shards*footnote below, possibly using Twemproxy for this. And have everything pertaining to a particular users' data on only one shard, (for things like 'likes','user-information','save-list' etc.) but also have multiple instances of Redis containing objects (which many different users will access) and data about said objects which will load for users also. I will not need to have Redis operations on multiple keys in different databases, but I will need to have Redis instances return m keys from n instances in real time.
To come completely clean with you I'm also planning on using something like this https://github.com/mpalmer/redis/blob/nds-2.6/README.nds so that I can use Redis while saving many keys to disc when not in use.
FOOTNOTE: (I am aware of Redis's Master-Slave replication, but prefer sharding for the extra storage in place of just more access)
Please, if your only comment is along the lines of, ""don't bother to shard until you absolutely have to"", keep it to yourself. I'm not interested in hearing responses that sharding is only important for a certain percentage of sites. That may be your opinion and that may even be fact but that is not what I am asking here.
IMO, if you're going to perform multiple reads from multiple shards instead of a single instance, you're most likely to get better performance as long as:
1. The sharding layer isn't slowing you down
2. The app can pull the data from the different shards asynchronously