Related
I'm trying to reduce Redis's objects size as much as I can and I've taken this whole week to experiment with it.
While testing different data representations I found out that an int representation of the string "hello" results in a smaller object. It may not look like much, but if you have a lot of data it can make a difference between using a few GB memory vs dozens of it.
Look at the following example (you can try it yourself if you want):
> SET test:1 "hello"
> debug object test:1
> Value at:0xb6c9f380 refcount:1 encoding:raw serializedlength:6 lru:9535350 lru_seconds_idle:7
In particular you should look at serializedlength which is 6 (bytes) in this case.
Now, look at the following int representation of it:
> SET test:2 "857715"
> debug object test:2
> Value at:0xb6c9f460 refcount:1 encoding:int serializedlength:5 lru:9535401 lru_seconds_idle:2
As you see, it results in a byte shorter object (note also encoding:int which I think is suggesting that ints get handled in a more efficient way).
With the string "hello w" (you'll see in a few moments why I didn't use "hello world" instead) we get an even bigger saving when it's represented as an int:
> SET test:3 "hello w"
> SET test:4 "857715023" <- Int representation. Notice that I inserted a "0", if I don't, it results in a bigger object and the encoding is set to "raw" instead (after all a space is not an int).
>
> debug object test:3
> Value at:0xb6c9f3a0 refcount:1 encoding:raw serializedlength:8 lru:9535788 lru_seconds_idle:6
> debug object test:4
> Value at:0xb6c9f380 refcount:1 encoding:int serializedlength:5 lru:9535809 lru_seconds_idle:5
It looks cool as long as you don't exceed 7 bytes string.. Look at what happens by a "hello wo" int representation:
> SET test:5 "hello wo"
> SET test:6 "85771502315"
>
> debug object test:5
> Value at:0xb6c9f430 refcount:1 encoding:raw serializedlength:9 lru:9535907 lru_seconds_idle:9
> debug object test:6
> Value at:0xb6c9f470 refcount:1 encoding:raw serializedlength:12 lru:9535913 lru_seconds_idle:5
As you can see the int (12 bytes) is bigger than the string representation (9 bytes).
My question here is, what's going on behind the scenes when you represent a string as an int, that it is smaller until you reach 7 bytes?
Is there a way to increase this limit as you do with "list-max-ziplist-entries/list-max-ziplist-value" or a clever way to optimize this process so that it always (or nearly) results in a smaller object than a string?
UPDATE
I've further experimented with other tricks, and you can actually have smaller ints than string, regardless of its size, but that would involve a little more work as of data structure modelling.
I've found out that if you split the int representation of a string in chunks of ~8 numbers each, it ends up being smaller.
Take as an example the word "Hello World Hi Universe" and create both a string and int SET:
> HMSET test:7 "Hello" "World" "Hi" "Universe"
> HMSET test:8 "74111114" "221417113" "78" "2013821417184"
The results are as follows:
> debug object test:7
> Value at:0x7d12d600 refcount:1 encoding:ziplist serializedlength:40 lru:9567096 lru_seconds_idle:296
>
> debug object test:8
> Value at:0x7c17d240 refcount:1 encoding:ziplist serializedlength:37 lru:9567531 lru_seconds_idle:2
As you can see we got the int set smaller by 3 bytes.
The problem in this will be how to organize such a thing, but it shows that it's possible nonetheless.
Still, don't know where this limit is set. The ~700K persistent use of memory (even when you have no data inside) makes me think that there is a pre-defined "pool" dedicated to the optimization of int sets.
UPDATE2
I think I've found where this intset "pool" is defined in Redis source.
At line 81 in the file redis.h there is the def REDIS_SHARED_INTEGERS set to 10000
REDISH_SHARED_INTEGERS
I suspect it's the one defining the limit of an intset byte length.
I have to try to recompile it with an higher value and see if I can use a longer int value (it'll most probably allocate more memory if it's the one I think of).
UPDATE3
I want to thank Antirez for the reply! Didn't expect that.
As he made me notice, len != memory usage.
I got further in my experiment and saw that the objects get already slightly compressed (serialized). I may have missed something from the Redis documentation.
The confirmation comes from analyzing a Redis key wih the command redis-memory-for-key key, which actually returns the memory usage and not the serialized length.
For example, let's take the "hello" string and int we used before, and see what's the result:
~ # redis-memory-for-key test:1
Key "test:1"
Bytes 101
Type string
~ #
~ # redis-memory-for-key test:2
Key "test:2"
Bytes 87
Type string
As you can notice the intset is smaller (87 bytes) than the string (101 bytes) anyway.
UPDATE4
Surprisingly a longer intset seems to affect its serializedlength but not memory usage..
This makes it possible to actually build a 2digit-char mapping while it still being more memory efficient than a string, without even chunking it.
By 2digit-char mapping I mean that instead of mapping "hello" to "85121215" we map it to digits with a fixed length of 2 each, prefixing it with "0" if digit < 10 like "0805121215".
A custom script would then proceed by taking every two digit apart and converting them to their equivalent char:
08 05 12 12 15
\ | | | /
h e l l o
This is enough to avoid disambiguation (like "o" and "ae" which both result in the digit "15").
I'll show you this works by creating another set and therefore analyzing its memory usage like I did before:
> SET test:9 "0805070715"
Unix shell
----------
~ # redis-memory-for-key test:9
Key "test:9"
Bytes 87
Type string
You can see that we have a memory win here.
The same "hello" string compressed with Smaz for comparison:
>>> smaz.compress('hello')
'\x10\x98\x06'
// test:10 would be unfair as it results in a byte longer object
SET post:1 "\x10\x98\x06"
~ # redis-memory-for-key post:1
Key "post:1"
Bytes 99
Type string
My question here is, what's going on behind the scenes when you represent a
string as an int, that it is smaller until you reach 7 bytes?
Notice that the integer you supplied as test #6 is no longer actually encoded
as an integer, but as raw:
SET test:6 "85771502315"
Value at:0xb6c9f470 refcount:1 encoding:raw serializedlength:12 lru:9535913 lru_seconds_idle:
So we see that a "raw" value occupies one byte plus the length of its string representation. In memory
you get that plus the overhead of the value.
The integer encoding, I suspect, encodes a number as a 32-bit integer; then it will always
need five bytes, one to tell its type, and four to store those 32 bits.
As soon as you overflow the maximum representable integer in 32 bits, which is either 2 billions or 4 depending on whether you use a sign or not, you need to revert to raw encoding.
So probably
2147483647 -> five bytes (TYPE_INT 0x7F 0xFF 0xFF 0xFF)
2147483649 -> eleven bytes (TYPE_RAW '2' '1' '4' '7' '4' '8' '3' '6' '4' '9')
Now, how can you squeeze a string representation PROVIDED THAT YOU ONLY USE AN ASCII SET?
You can get the string (140 characters):
When in the Course of human events it becomes necessary for one people
to dissolve the political bands which have connected them with another
and convert each character to a six-bit representation; basically its index in the string
"ABCDEFGHIJKLMNOPQRSTUVWXYZ01234 abcdefghijklmnopqrstuvwxyz56789."
which is the set of all the characters you can use.
You can now encode four such "text-only characters" in three "binary characters", a sort of "reverse base 64 encoding"; base64 encoding will get three binary characters and create a four-byte sequence of ASCII characters.
If we were to code it as groups of integers, we would save a few bytes - maybe get it down
to 130 bytes - at the cost of a larger overhead.
With this type of "reverse base64" encoding, we can get 140 character to 35 groups of four characters, which become a string of 35x3 = 105 binary characters, raw encoded to 106 bytes.
As long, I repeat, as you never use characters outside the range above. If you do, you can
enlarge the range to 128 characters and 7 bits, thus saving 12.5% instead of 25%; 140 characters will then become 126, raw encoded to 127 bytes, and you save (141-127) = 14 bytes.
Compression
If you have much longer strings, you can compress them (i.e., you use a function such as deflate() or gzencode() or gzcompress() ). Either straight; in which case the above string becomes 123 bytes. Easy to do.
Compressing many small strings: the Rube Goldberg approach
Since compression algorithms learn, and at the beginning they dare assume nothing, small strings will not compress greatly. They're "all beginning", so to speak. Just as an engine, when running cold the performances are inferior.
If you have a "corpus" of text these strings come from, you can use a time-consuming trick that "warms up" the compression engine and may double (or better) its performances.
Suppose you have two strings, COMMON and TARGET (the second one is the one you're interested in). If you z-compressed COMMON you would get, say, ZCMN. If you compressed TARGET you would get ZTRGT.
But as I said, since the gz compression algorithm is stream oriented, and it learns as it goes by, the compression ratio of the second half of any text (provided there aren't freakish statistical distribution changes between halves) is always appreciably higher than that of the first half.
So if you were to compress, say, COMMONTARGET, you'd get ZCMGHQI.
Notice that the first part of the string, as far as almost the end, is the same as before. Indeed if you compressed COMMONFOOBAR, you'd get something like ZCMQKL. And the second part is compressed better than before, even if we count the area of overlap as belonging entirely to the second string.
And this is the trick. Given a family of strings (TARGET, FOOBAR, CASTLE BRAVO), we compress not the strings, but the concatenation of those strings with a large prefix. Then we discard from the result the common compressed prefix. Thus TARGET is taken from the compression of COMMONTARGET (which is ZCMGHQI), and becomes GHQI instead of ZTRGT, with a 20% gain.
The decoder does the reverse: given GHQI, it first applies the common compressed prefix ZCM (which it must know); then it decodes the result, and finally discards the common uncompressed prefix, of which it need only know the length beforehand.
So the first sentence above (140 characters) becomes 123 when compressed by itself; if I take the rest of the Declaration and use it as a prefix, it compresses to 3355 bytes. This prefix plus my 140 bytes becomes 3409 bytes, of which 3352 are common, leaving 57 bytes.
At the cost of storing once the uncompressed prefix in the encoder, and the compressed prefix once in the decoder, and the whole thingamajig running five times as slow, I can now get those 140 bytes down to 57 instead of 123 - less than half of before.
This trick works great for small strings; for larger ones, the advantage isn't worth the pain. Also, different prefixes yield different results. The best prefixes are those that contain most of the sequences that are likely to appear in the string pool, ordered by increasing length.
Added bonus: the compressed prefix also doubles as a sort of weak encryption, as without that, you can't easily decode the compressed strings, even if you might be able to recover some pieces thereof.
In a given file record, I need to read the first two integer elements at first, and then the rest of the line (a large number of real elements), because the assignment depend on the first 2. Suppose the format of the first two integer elements is not really well defined.
The best way to solve the problem could be something:
read(unitfile, "(I0,I0)", advance='no') ii, jj
read(unitfile,*) aa(ii,jj,:)
But it seems to me the "(I0)" specification is not allowed in gfortran.
Basically the file read in unitfile could be something like:
0 0 <floats>
0 10 <floats>
10 0 <floats>
100 0 <floats>
100 100 <floats>
which is hard to be read with any fortran-like fixed field format specification.
Is there any other way to get around this, apparently trivial, problem?
This applies string manipulations to get the individual components, separated by blanks ' ' and/or tabs (char(9)):
program test
implicit none
character(len=256) :: string, substring
integer :: ii, jj, unitfile, stat, posBT(2), pos
real, allocatable :: a(:)
open(file='in.txt', newunit=unitfile, status='old' )
read(unitfile,'(a)') string
! Crop whitespaces
string = adjustl(trim(string))
! Get first part:
posBT(1) = index(string,' ') ! Blank
posBT(2) = index(string,char(9)) ! Tab
pos = minval( posBT, posBT > 0 )
substring = string(1:pos)
string = adjustl(string(pos+1:))
read(substring,*) ii
! Get second part:
posBT(1) = index(string,' ') ! Blank
posBT(2) = index(string,char(9)) ! Tab
pos = minval( posBT, posBT > 0 )
substring = string(1:pos)
string = adjustl(string(pos+1:))
read(substring,*) jj
! Do stuff
allocate( a(ii+jj), stat=stat )
if (stat/=0) stop 'Cannot allocate memory'
read(string,*) a
print *,a
! Clean-up
close(unitfile)
deallocate(a)
end program
For a file in.txt like:
1 2 3.0 4.0 5.0
This results in
./a.out
3.00000000 4.00000000 5.00000000
NOTE: This is just a quick&dirty example, adjust it to your needs.
[This answer has been significantly revised: the original was unsafe. Thanks to IanH for pointing that out.]
I generally try to avoid doing formatted input which isn't list-directed, when I can afford it. There's already an answer with string parsing for great generality, but I'll offer some suggestions for a simpler setting.
When you are relaxed about trusting the input, such as when it's just the formatting that's a bit tricky (or you 're happy leaving it to your compiler's bounds checking), you can approach your example case with
read(unitfile, *) ii, jj, aa(ii, jj, :)
Alternatively, if the array section is more complicated than given directly by the first two columns, it can be by an expression, or even by functions
read(unitfile, *) ii, jj, aa(fi(ii,jj), fj(ii,jj), :fn(ii,jj))
with pure integer function fi(ii,jj) etc. There is even some possibility of having range validation in those functions (returning a size 0 section, for example).
In a more general case, but staying list-directed, one could use a buffer for the real variables
read(unitfile, *) ii, jj, buffer(:) ! Or ... buffer(:fn(ii,jj))
! Validate ii and jj before attempting to access aa with them
aa(.., .., :) = buffer
where buffer is of suitable size.
Your first considered approach suggests you have some reasonable idea of the structure of the lines, including length, but when the number of reals is unknown from ii and jj, or when the type (and polymorphism reading isn't allowed) is not known, then things do indeed get tricky. Also, if one is very sensitive about validating input, or even providing meaningful detailed user feedback on error, this is not optimal.
Finally, iostat helps.
I've written several IDL programs to analyse some data. To keep it simple the programs read in some time varying data and calculate the fourier spectrum. This spectrum is written to file using this code:
openw,3,filename
printf,3,[transpose(freq),transpose(power)],format='(e,e)'
close,3
The file is then read by another program using this code:
rdfloat,filename,freq,power,/double
The rdfloat procedure can be found here: http://idlastro.gsfc.nasa.gov/
The error i get when trying to read the a file is: "Input conversion error. Unit: 101"
When i delve in to the file being read, i notice several types of unrecognised characters. I dont know if these are a result of the writing to the file or some thing else related to the number of files being created (over 300 files)
These symbols/characters are in the place of a single number:
< dle> < dc1> < dc2> < dc3> < dc4> < can> < nak> < em> < soh> < syn>
Example of what appears in the file being read, Note they are not consecutive lines.
7.7346< dle>18165493007e+01 8.4796811549010105e+00
7.7354408697119453e+01 1.04459538071< dc2>1749e+01
7.7360701595839< can>28e+01 3.0447318983094189e+00
Whenever I run the procedures that write the files, there is always at least one file that has some or all of these characters. The file/s that contains these characters is always different.
Can anyone explain what these symbols are and what I might be doing to create them as well as how to ensure they are not written to file?
I see two things that may be causing a problem. But first, I want to suggest a few tips.
When you open a file, it is useful to use the /GET_LUN keyword because it allows IDL to find and use a logical unit number (LUN) that is available (e.g., in case you left LUN 3 open somewhere else). When you print formatted data, you should specify the total width and number of decimal places. It will make things easier because then you need not worry about changing spacings between numbers in a file.
So I would change your first set of code to the following (or some variant of the following):
OPENW,gunit,filename[0],/GET_LUN,ERROR=err
FOR j=0L, N_ELEMENTS(freq) - 1L DO BEGIN
PRINTF,gunit,freq[j],power[j],FORMAT='(2e20.12)'
ENDFOR
FREE_LUN,gunit ;; this is better than using the CLOSE routine
So the first potential issue I see is that if your variable power was calculated using something like FFT.pro, then it will be a complex float or complex double, depending on the input and keywords used.
The second potential issue may be due to an incorrect format statement. You did not tell PRINTF how many columns or rows to expect. It might not know how to handle the input properly, so it guesses and may result in those characters you show. Those characters may be spacing characters due to the vague format statement or the software you are using to look at the files (e.g., I would not recommend using Word to open text files, use a text editor).
Side Note: You can open and read the file you just wrote in a similar fashion to what I showed above, but changed to the following:
n = FILE_LINES(filename[0])
freq = DBLARR(n)
power = DBLARR(n)
OPENR,gunit,filename[0],/GET_LUN,ERROR=err
FOR j=0L, N_ELEMENTS(freq) - 1L DO BEGIN
READF,gunit,freq[j],power[j],FORMAT='(2e20.12)'
ENDFOR
FREE_LUN,gunit ;; this is better than using the CLOSE routine
I have made a program which takes a 1000 digit number as input.
It is fixed, so I put this input into the code file itself.
I would obviously be storing it as Integer type, but how do I do it?
I have tried the program by having 1000 digits in the same line. I know this is the worst possible code format! But it works.
How can assign the variable this number, and split its lines. I read somewhere something about eos? Ruby, end of what?
I was thinking that something similar to comments could be used here.
Help will be appreciated.
the basic idea is to make this work:
a=3847981438917489137897491412341234
983745893289572395725258923745897232
instead of something like this:
a=3847981438917489137897491412341234983745893289572395725258923745897232
Haskell doesn't have a way to split (non-String) literals across multiple lines. Since Strings are an exception, we can shoehorn in other literals by parsing a multiline String:
v = read
"32456\
\23857\
\23545" :: Integer
Alternately, you can use list syntax if you think it's prettier:
v = read . concat $
["32456"
,"24357"
,"23476"
] :: Integer
The price you pay for this is that some work will be done (once) at runtime, namely, the parsing (e.g. read).
this is my first time trying to program in Fortran. I'm trying to write a program that prints the first 1476 terms of the Fibonacci sequence, then examines the first digit of each term and stores the number of 1s, 2s, 3s, ..., 9s that occur in an array.
The problem that I can't seem to figure out is how to read the first digit of each term. I've tried several things but am having difficulty with my limited knowledge of Fortran techniques. I write the terms to a text file and the idea is to read the first digit of each line and accumulate the respective number in the array. Does anyone have any suggestions of how to do this?
Here is my code so far:
(edit: I included the code I have for reading the file. Right now it just prints out 3.60772951994415996E-313,
which seems like an address of some sort, because it's not one of the Fibonacci numbers. Also, it is the only thing printed, I expected that it would print out every line of the file...)
(edit edit: After considering this, perhaps there's a way to format the writing to the text file to just the first digit. Is there a way to set the number of significant digits of a real number to one? :P)
subroutine writeFib(n)
integer :: i
real*8 :: prev, current, newFib
prev = 0
current = 1
do i = 1, n
newFib = prev + current
prev = current
current = newFib
write(7,*) newFib
end do
return
end subroutine
subroutine recordFirstDigits(a)
integer :: openStat, inputStat
real*8 :: fibNum
open(7, file = "fort.7", iostat = openStat)
if (openStat > 0) stop "*** Cannot open the file ***"
do
read(7, *, iostat = inputStat) fibNum
print *,fibNum
if (inputStat > 0) stop "*** input error ***"
if (inputStat < 0) exit ! end of file
end do
close(7)
end subroutine
program test
integer :: k, a(9)
k = 1476
call writeFib(k)
call recordFirstDigits(a)
end program
Although the suggestions were in place, there were also several things that were forgotten. Range of the REAL kind, and some formatting problems.
Anyways, here's one patched up solution, compiled and working, so try to see if this will work for you. I've took the liberty of choosing my own method for fibonacci numbers calculation.
program SO1658805
implicit none
integer, parameter :: iwp = selected_real_kind(15,310)
real(iwp) :: fi, fib
integer :: i
character(60) :: line
character(1) :: digit
integer :: n0=0, n1=0, n2=0, n3=0, n4=0, n5=0, n6=0, n7=0, n8=0, n9=0
open(unit=1, file='temp.txt', status='replace')
rewind(1)
!-------- calculating fibonacci numbers -------
fi = (1+5**0.5)/2.
do i=0,1477
fib = (fi**i - (1-fi)**i)/5**0.5
write(1,*)fib,i
end do
!----------------------------------------------
rewind(1)
do i=0,1477
read(1,'(a)')line
line = adjustl(line)
write(*,'(a)')line
read(line,'(a1)')digit
if(digit.eq.' ') n0=n0+1
if(digit.eq.'1') n1=n1+1
if(digit.eq.'2') n2=n2+1
if(digit.eq.'3') n3=n3+1
if(digit.eq.'4') n4=n4+1
if(digit.eq.'5') n5=n5+1
if(digit.eq.'6') n6=n6+1
if(digit.eq.'7') n7=n7+1
if(digit.eq.'8') n8=n8+1
if(digit.eq.'9') n9=n9+1
end do
close(1)
write(*,'("Total number of different digits")')
write(*,'("Number of digits 0: ",i5)')n0
write(*,'("Number of digits 1: ",i5)')n1
write(*,'("Number of digits 2: ",i5)')n2
write(*,'("Number of digits 3: ",i5)')n3
write(*,'("Number of digits 4: ",i5)')n4
write(*,'("Number of digits 5: ",i5)')n5
write(*,'("Number of digits 6: ",i5)')n6
write(*,'("Number of digits 7: ",i5)')n7
write(*,'("Number of digits 8: ",i5)')n8
write(*,'("Number of digits 9: ",i5)')n9
read(*,*)
end program SO1658805
Aw, ... I just read you need the number of digits stored in to an array. While I just counted them.
Oh well, ... "left as an exercise for the reader ..." :-)
Can you read with a FORMAT(A1)? It's been 20 years so I don't remember the exact syntax.
I wonder why the open statement succeeds when file 7 hasn't been closed. I think you need an endfile statement and/or a rewind statement in between writing and reading.
Paul Tomblin posted what you have to do after you solve your problem in getting reads to work in the first place.
I am getting an 'end of line' runtime error
You don't show the ! code to read here... which makes it kind of difficult to guess what you are doing wrong :-)
Perhaps you need a loop to read each line and then jump out of the loop to a continue statement when there are no more lines.
Something like this:
do
read(7,*,end=10) fibNumber
end do
10 continue
Better still - look at the more modern style used in this revcomp program.
here are some hints:
You don't need to use characters,
much less file i/o for this problem
(unless you forgot to state that a
file must be created).
Therefore, use math to find your statistics. There are lots of resources on Fibonacci numbers that might provide a simplifying insight or at least a way to independently spot check your answers.
Here is a complicated hint in non-Fortran lingo:
floor(10^(frac(log_10(7214989861293412))))
(Put this in Wolfram Alpha to see what it does.)
A simpler hint (for a different approach) is that you can do very
well in Fortran with simple
arithmetic inside of looping
constructs--at least for a first pass at the solution.
Accumulate your statistics as you
go. This advice would even apply to your character-driven approach. (This problem is ideally suited
for coming up with a cute indexing
scheme for your statistics, but some
people hate cute schemes in
programming. If you don't fear
cuteness ... then you can have associative
arrays in Fortran as long as your
keys are integers ;-)
The most important aspect of this
problem is the data type you will
use to calculate your answers. For
example, here's the last number you
will have to print.
Cheers, --Jared