I am using redis to save jsonwebtokens. I am confused a little about the consumption of memory for every record.
Let's say I have an instance on Google cloud that has 4GB Memory allocated to it, I want to know that how many records can it handle.
Given that a record has on an average 1 string values excluding he identifier and every string has on an average 200 characters.
It's all about how you store them. Using hashes (sizing them properly), or plain key value pair.
Do read this doc for more info http://redis.io/topics/memory-optimization
For 1 million keys (simple key value pair) of 200 characters it takes about 300 MB. So for 4 GB you can store more or less 14 million keys I guess. To make sure this, install redis in your machine, run a simple java (using jedis) snippet, and check the memory consumption before and after the insertion.
Jedis jedis = new Jedis("localhost");
for i=0 to N
jedis.set("Key_"+i,string);
Redis wraps strings into sds struct, which requires 3 extra bytes (or more) for each string.
Each sds is stored in a redisObject struct (using a pointer pointing to that sds object). It takes about 16 extra bytes if you're on a 64-bit machine.
You may also consider the entries in the hash table. Each one takes 24 bytes.
So you can assume each of your string occupies 243 bytes. 1 million strings will use more than 250 MB (Redis itself needs memory).
Related
In working on a project, where:
A dataset is collected every 10 seconds, which is stored in an SQlite file on an server.
After being processed, the data is sent to an SQL-Database every 5 minutes.
Afterwards the data in the SQlite file, which isn't needed anymore, gets deleted.
The collecting of the data continues and at the moment the id doens't get reset.
I didn't get how much an integer in SQLite can store according to the documentation (https://www.sqlite.org/datatype3.html).
In MySQL-Databases the maximum value of an interger column is 2.147.483.647. If my script would run for 10 years the id would be 31.449.600. Although this would be much lower less than the maximum, I wondered,
if there is any problem with storing high values in SQlite.
Could this affect the performance?
That page mentions that integer numbers can be stored in up to 8 bytes, i.e., 64 bits.
As mentioned elsewhere, this means that the largest allowed integer is 9,223,372,036,854,775,807.
An integer in SQLite can store values up to 9223372036854775807 (8 Bytes signed so 63 bits as 1 bit is for the sign), which is the same as a MySQL BIGINT (which is in addition to the Standard INT).
I have around 256 keys. Against each key I have to store a large number of non-repitative integers.
Following are the top 7 keys with number of total values (entries) against each key. Each value is a unique integer with large value.
Key No. of integers (values) in the list
Key 1 3394967
Key 2 3385081
Key 3 2172866
Key 4 2171779
Key 5 1776702
Key 6 1772936
Key 7 1748858
By default Redis consumes lot of memory in storing this data. I read that changing following parameters can result in memory usage reduction highly.
list-max-zipmap-entries 512
list-max-zipmap-value 64
Can anyone please explain me these above configuration commands (are 512 and 64 bytes?) and what changes I can make in the above configuration settings for my case to achieve memory usage reduction?
What should be kept in mind while selecting the values for entries and value in above command?
list-max-mipmap-entries 512:
list-max-zipmap-value 64
If the number of entries in a List exceeds 512, or if the size of any given element in the list > 64 bytes, Redis will switch to a less-efficient in-memory storage structure. More specifically, below those thresholds it will use a ziplist, and above it will use a linked list.
So in your case, you would need to use an entries value of > 1748858 to see any change (and then only in keys 8-end). Also note that for Redis to re-encode them to the smaller object size you would also need to make the change in the config and restart Redis as it doesn't re-encode down automatically.
To verify a given key is using a ziplist vs. linked list, use the OBJECTcommand.
For more details, see Redis Memory Optimization
IMO you can't achieve redis' memory optimization. In your case the entries in each list/set is around 3 million. In order to achieve memory optimization if you give the value of list-max-zipmap-entries as 3 million.
Redis doc says,
This operation is very fast for small values, but if you change the
setting in order to use specially encoded values for much larger
aggregate types the suggestion is to run some benchmark and test to
check the conversion time.
As per this encoding and decoding will take more time/CPU for that huge number. So it is better to run a benchmark test and then decide.
One alternative suggestion, if you only look up this sets to see whether a key is available or not. then you can change the Structure to a bucket kind of a thing.
For example a value 123456 set to key1 can be stored like this
Sadd key1:bucket:123 456
123 = 123456/1000
456 = 123456%1000
Note this won't work if you want to retrieve all the values for key1. In that case you would be looping through 1000 of sets. similarly for total size of key1 you have to loop through 1000 keys.
But the memory will be reduced to about 10 times.
I have a New Dictionary(Of String, Long()) with 3,125,000 unique (string) keys.
I am distributing close to 1 billion (935,984,413) values (all longs) amongst the keys, and populate a long() array for each key.
This works fine and very fast for average datasets, let's say for 1,5000,000 string Keys and 500,000,000 Long values to be distributed, this is done in about 2 hours.
However, for the abovementioned dataset, once I get halfway through my data, the process is running extremely slow and at the current trend may never finish ...
I think I am running out of memory, the application is using 5GB of memory, and I believe it is now limited by my system (8GB of RAM).
How can I calculate the amount of memory I need for the above situation?
The size of the string Keys average around 5 characters.
Thanks!
Long data type is 8-byte each. For string, it is more complicated. Check out this post by famous Jon Skeet.
Quote:
In the current implementation at least, strings take up 20+(n/2)*4
bytes (rounding the value of n/2 down)
(Note: in his blog post, he has some updates on this string calculation)
Given your case, each of your 5 chars string would take around:
20 + (5/2) * 4 = 20 + 8 = 28 bytes
Nevertheless, you could simplify your calculation by computing only the significant figure - in your case is the Long since it has a lot more members than the string while your string key is of small size (5 chars).
Thus if you have 1 billions of Long, you would have around 8GB memory used only for the Long. Some other overheads + the string would be less significant, but at least almost 8 GB (935,984,413 x 8 = 7,487,875,304) would be needed.
The string, in your example, would be:
28 * 3,125,000 = 87.5 MB
Thus totaling in 7.5~7.6 GB just for the string and the Long()
I'm currently working on a school project to design a network, and we're asked to assess traffic on the network. In our solution (dealing with taxi drivers), each driver will have a smartphone that can be used to track its position to assign him the best ride possible (through Google Maps, for instance).
What would be the size of data sent and received by a single app during one day? (I need a rough estimate, no real need for a precise answer to the closest bit)
Thanks
Gps Positions compactly stored, but not compressed needs this number of bytes:
time : 8 (4 bytes is possible too)
latitude: 4 (if used as integer or float) or 8
longitude 4 or 8
speed: 2-4 (short: 2: integer 4)
course (2-4)
So binary stored in main memory, one location including the most important attributes, will need 20 - 24 bytes.
If you store them in main memory as single location object, additonal 16 bytes per object are needed in a simple (java) solution.
The maximum recording frequence is usually once per second (1/s): Per hour this need: 3600s * 40 byte = 144k. So a smartphone easily stores that even in main memory.
Not sure if you want to transmit the data:
When transimitting this to a server data usually will raise, depending of the transmit protocoll used.
But it mainly depends how you transmit the data and how often.
If you transimit every 5 minutes a position, you dont't have to care, even
when you use a simple solution that transmits 100 times more bytes than neccessary.
For your school project, try to transmit not more than every 5 or better 10 minutes.
Encryption adds an huge overhead.
To save bytes:
- Collect as long as feasible, then transmit at once.
- Favor binary protocolls to text based. (BSON better than JSON), (This might be out of scope for your school project)
Does anyone know what the maximum value size you can store in redis? I want to use redis as a message queue with celery to store some small documents that need to be processed by a worker on another server, and I want to make sure the documents aren't going to be too big.
I found one page with a reference to 1GB, but when I followed the link on the page for where they got that answer the link wasn't valid anymore. Here is the link:
http://news.ycombinator.com/item?id=1182005
All string values are limited to 512 MiB. This is the size limit you probably care most about.
EDIT: Because keys in Redis are strings, the maximum key size is 512 MiB. The maximum number of keys is 2^32 - 1 = 4,294,967,295.
Values, on the other hand, can vary in size depending on their type. For aggregate data types (i.e. hash, list, set, and sorted set), the maximum value size is 512 MiB for each element, although the data structure itself can have up to 2^32 - 1 elements.
https://redis.io/topics/data-types
https://redis.io/topics/faq#what-is-the-maximum-number-of-keys-a-single-redis-instance-can-hold-and-what-is-the-max-number-of-elements-in-a-hash-list-set-sorted-set
http://groups.google.com/group/redis-db/browse_thread/thread/1c7e33fbc98734b3?fwc=2
Article about Redis Memory Usage can help you to roughly determine how much memory your database would take.
It's in the order of the amount of RAM you have, at least, so unless you plan on puting multi-gigabyte objects in there I wouldn't worry. I've had sets that were hundreds of megabytes big without a problem, but I don't know the exact limits.
A String value can accommodate the size of max 512MB. But according to this link, the size can be increased.