I must process some huge file with gawk. My main problem is that I have to print some floats using thousand separators. E.g.: 10000 should appear as 10.000 and 10000,01 as 10.000,01 in the output.
I (and Google) come up with this function, but this fails for floats:
function commas(n) {
gsub(/,/,"",n)
point = index(n,".") - 1
if (point < 0) point = length(n)
while (point > 3) {
point -= 3
n = substr(n,1,point)"."substr(n,point + 1)
}
sub(/-\./,"-",n)
return d n
}
But it fails with floats.
Now I'm thinking of splitting the input to an integer and a < 1 part, then after formatting the integer gluing them again, but isn't there a better way to do it?
Disclaimer:
I'm not a programmer
I know that via some SHELL env. variables the thousand separators can be set, but it must be working in different environments with different lang and/or locale settings.
English is my 2nd language, sorry if I'm using it incorrectly
It fails with floats because you're passing in European type numbers (1.000.000,25 for a million and a quarter). The function you've given should work if you just change over commas and periods. I'd test the current version first with 1000000.25 to see if it works with non-European numbers.
The following awk script can be called with "echo 1 | awk -f xx.gawk" and it will show you both the "normal" and European version in action. It outputs:
123,456,789.1234
123.456.789,1234
Obviously, you're only interested in the functions, real-world code would use the input stream to pass values to the functions, not a fixed string.
function commas(n) {
gsub(/,/,"",n)
point = index(n,".") - 1
if (point < 0) point = length(n)
while (point > 3) {
point -= 3
n = substr(n,1,point)","substr(n,point + 1)
}
return n
}
function commaseuro(n) {
gsub(/\./,"",n)
point = index(n,",") - 1
if (point < 0) point = length(n)
while (point > 3) {
point -= 3
n = substr(n,1,point)"."substr(n,point + 1)
}
return n
}
{ print commas("1234,56789.1234") "\n" commaseuro("12.3456789,1234") }
The functions are identical except in their handling of commas and periods. We'll call them separators and decimals in the following description:
gsub removes all of the existing separators since we'll be putting them back.
point finds where the decimal is since that's our starting point.
if there's no decimal, the if-statement starts at the end.
we loop while there's more than three characters left.
inside the loop, we adjust the position for inserting a separator, and insert it.
once the loop is finished, we return the adjusted value.
To go with Pax's answer:
Read the "Conversion" section of the GNU awk manual which talks explicitly about the effect of your LOCALE environment variable on the string representation of numeric types.
Related
from GNU gawk's page
https://www.gnu.org/software/gawk/manual/html_node/Checking-for-MPFR.html
they have a formula to check arbitrary precision
function adequate_math_precision(n) { return (1 != (1+(1/(2^(n-1))))) }
My question is : wouldn't it be more efficient by staying within integer math domain with a formula such as
( 2^abs(n) - 1 ) % 2 # note 2^(n-1) vs. 2^|n| - 1
Since any power of 2 must also be even, then subtracting 1 must always be odd, then its modulo (%) over 2 becomes indicator function for is_odd() for n >= 0, while the abs(n) handles the cases where it's negative.
Or does the modulo necessitate a casting to float point, thus nullifying any gains ?
Good question. Let's tackle it.
The proposed snippet aims at checking wether gawk was invoked with the -M option.
I'll attach some digression on that option at the bottom.
The argument n of the function is the floating point precision needed for whatever operation you'll have to perform. So, say your script is in a library somewhere and will get called but you have no control over it. You'll run that function at the beginning of the script to promptly throw exception and bail out, suggesting that the end result will be wrong due to lack of bits to store numbers.
Your code stays in the integer realm: a power of two of an integer is an integer. There is no need to use abs(n) here, because there is no point in specifying how many bits you'll need as a negative number in the first place.
Then you subtract one from an even, integer number. Now, unless n=0, in which case 2^0=1 and then your code reads (1 - 1) % 2 = 0, your snippet shall always return 1, because the quotient (%) of an odd number divided by two is 1.
Problem is: you are trying to calculate a potentially stupidly large number in a function that should check if you are able to do so in the first place.
Since any power of 2 must also be even, then subtracting 1 must always
be odd, then its modulo (%) over 2 becomes indicator function for
is_odd() for n >= 0, while the abs(n) handles the cases where it's
negative.
Except when n=0 as we discussed above, you are right. The snippet will tell that any power of 2 is even, and any power of 2, minus 1, is odd. We were discussing another subject entirely thought.
Let's analyze the other function instead:
return (1 != (1+(1/(2^(n-1)))))
Remember that booleans in awk runs like this: 0=false and non zero equal true. So, if 1+x where x is a very small number, typically a large power of two (2^122 in the example page) is mathematically guaranteed to be !=1, in the digital world that's not the case. At one point, floating computation will reach a precision rock bottom, will be rounded down, and x=0 will be suddenly declared. At that point, the arbitrary precision function will return 0: false: 1 is equal 1.
A larger discussion on types and data representation
The page you link explains precision for gawk invoked with the -M option. This sounds like technoblahblah, let's decipher it.
At one point, your OS architecture has to decide how to store data, how to represent it in memory so that it can be accessed again and displayed. Terms like Integer, Float, Double, Unsigned Integer are examples of data representation. We here are addressing Integer representation: how is an integer stored in memory?
A 32-bit system will use 4 bytes to represent and integer, which in turn determines how larger the integer will be. The 32 bits are read from most significative (MSB) to less significative (LSB) and if signed, one bit will represent the sign (the MSB typically, drastically reducing the max size of the integer).
If asked to compute a large number, a machine will try to fit in in the max number available. If the end result is larger than that, you have overflow and end up with a wrong result or an error. Many online challenges typically ask you to write code for arbitrary long loops or large sums, then test it with inputs that will break the 64bit barrier, to see if you master proper types for indexes.
AWK is not a strongly typed language. Meaning, any variable can store data, regardless of the type. The data type can change and it is determined at runtime by the interpreter, so that the developer doesn't need to care. For instance:
$awk '{a="this is text"; print a; a=2; print a; print a+3.0*2}'
-| this is text
-| 2
-| 8
In the example, a is text, then is an integer and can be summed to a floating point number and printed as integer without any special type handling.
The Arbitrary Precision Page presents the following snippet:
$ gawk -M 'BEGIN {
> s = 2.0
> for (i = 1; i <= 7; i++)
> s = s * (s - 1) + 1
> print s
> }'
-| 113423713055421845118910464
There is some math voodoo behind, we will skip that. Since s is interpreted as a floating point number, the end result is computed as floating point.
Try to input that number on Windows calculator as decimal, and it will fail. Although you can compute it as a binary. You'll need the programmer setting and to add up to 53 bits to be able to fit it as unsigned integer.
53 is a magic number here: with the -M option, gawk uses arbitrary precision for numbers. In other words, it commandeers how many bits are necessary, track them and breaks free of the native OS architecture. The default option says that gawk will allocate 53 bits for any given arbitrary number. Fun fact, the actual result of that snippet is wrong, and it would take up to 100 bits to compute correctly.
To implement arbitrary large numbers handling, gawk relies on an external library called MPFR. Provided with an arbitrary large number, MPFR will handle the memory allocation and bit requisition to store it. However, the interface between gawk and MPFR is not perfect, and gawk can't always control the type that MPFR will use. In case of integers, that's not an issue. For floating point numbers, that will result in rounding errors.
This brings us back to the snippet at the beginning: if gawk was called with the -M option, numbers up to 2^53 can be stored as integers. Floating points will be smaller than that (you'll need to make the comma disappear somehow, or rather represent it spending some of the bits allocated for that number, just like the sign). Following the example of the page, and asking an arbitrary precision larger than 32, the snippet will return TRUE only if the -M option was passed, otherwise 1/2^(n-1) will be rounded down to be 0.
In a given file record, I need to read the first two integer elements at first, and then the rest of the line (a large number of real elements), because the assignment depend on the first 2. Suppose the format of the first two integer elements is not really well defined.
The best way to solve the problem could be something:
read(unitfile, "(I0,I0)", advance='no') ii, jj
read(unitfile,*) aa(ii,jj,:)
But it seems to me the "(I0)" specification is not allowed in gfortran.
Basically the file read in unitfile could be something like:
0 0 <floats>
0 10 <floats>
10 0 <floats>
100 0 <floats>
100 100 <floats>
which is hard to be read with any fortran-like fixed field format specification.
Is there any other way to get around this, apparently trivial, problem?
This applies string manipulations to get the individual components, separated by blanks ' ' and/or tabs (char(9)):
program test
implicit none
character(len=256) :: string, substring
integer :: ii, jj, unitfile, stat, posBT(2), pos
real, allocatable :: a(:)
open(file='in.txt', newunit=unitfile, status='old' )
read(unitfile,'(a)') string
! Crop whitespaces
string = adjustl(trim(string))
! Get first part:
posBT(1) = index(string,' ') ! Blank
posBT(2) = index(string,char(9)) ! Tab
pos = minval( posBT, posBT > 0 )
substring = string(1:pos)
string = adjustl(string(pos+1:))
read(substring,*) ii
! Get second part:
posBT(1) = index(string,' ') ! Blank
posBT(2) = index(string,char(9)) ! Tab
pos = minval( posBT, posBT > 0 )
substring = string(1:pos)
string = adjustl(string(pos+1:))
read(substring,*) jj
! Do stuff
allocate( a(ii+jj), stat=stat )
if (stat/=0) stop 'Cannot allocate memory'
read(string,*) a
print *,a
! Clean-up
close(unitfile)
deallocate(a)
end program
For a file in.txt like:
1 2 3.0 4.0 5.0
This results in
./a.out
3.00000000 4.00000000 5.00000000
NOTE: This is just a quick&dirty example, adjust it to your needs.
[This answer has been significantly revised: the original was unsafe. Thanks to IanH for pointing that out.]
I generally try to avoid doing formatted input which isn't list-directed, when I can afford it. There's already an answer with string parsing for great generality, but I'll offer some suggestions for a simpler setting.
When you are relaxed about trusting the input, such as when it's just the formatting that's a bit tricky (or you 're happy leaving it to your compiler's bounds checking), you can approach your example case with
read(unitfile, *) ii, jj, aa(ii, jj, :)
Alternatively, if the array section is more complicated than given directly by the first two columns, it can be by an expression, or even by functions
read(unitfile, *) ii, jj, aa(fi(ii,jj), fj(ii,jj), :fn(ii,jj))
with pure integer function fi(ii,jj) etc. There is even some possibility of having range validation in those functions (returning a size 0 section, for example).
In a more general case, but staying list-directed, one could use a buffer for the real variables
read(unitfile, *) ii, jj, buffer(:) ! Or ... buffer(:fn(ii,jj))
! Validate ii and jj before attempting to access aa with them
aa(.., .., :) = buffer
where buffer is of suitable size.
Your first considered approach suggests you have some reasonable idea of the structure of the lines, including length, but when the number of reals is unknown from ii and jj, or when the type (and polymorphism reading isn't allowed) is not known, then things do indeed get tricky. Also, if one is very sensitive about validating input, or even providing meaningful detailed user feedback on error, this is not optimal.
Finally, iostat helps.
Currently studying bitwise arithmetic. It's really easy, because I have some CS background. But I just don't understand one moment with this operator.
For example:
variable3 = variableOne & 3;
or
variable3 &= 3;
Actually this doesn't matter.
I don't understand how the process of setting the bits to 0 is going on. And how you can process it on the paper?
Let’s say 5&3, four-bit width:
0101b = 5dec
0011b = 3dec
------------
0001b = 1dec
You just & the bits in the same column. And since the & operator only returns 1 when both arguments are 1, the higher bits from 5 not present in 3 are masked out.
As for your example from the comments:
$ perl -E 'printf "%b\n", 0x76'
1110110
And now:
1110110 = 0x76
0000011 = 3dec
-------
0000010 = 2dec
…and just to validate:
$ perl -E 'say 0x76&3'
2
The schema is simple, you just & each column:
x
y
-
z
Where z is x&y.
Aha, judging by your comments in the neighbouring answer the problem is elsewhere. Numeric variables do not contain “hexadecimal values” in them. Numeric variables contain a bit pattern representing a number. “A number” is never binary, decimal or hexadecimal. When you say “three”, there’s no number system in play, three is a three no matter what.
When you say something like var x = 0x76 in the source code, the machine reads the hexadecimal representation of the number, creates a bit pattern representing this number and stores it in the memory corresponding to the variable. And when you then say something like x &= 3, the machine creates a bit pattern representing number three, combines that with the bit pattern stored in the variable and stores the result in the variable.
I'm learning MMIX so I tried making a simple program to add one to itself and print the result. Unfortunately it doesn't print anything. Here is my program:
n IS $4
y IS $3
t IS $255
LOC #100
Main SET n,1 %let n = 1
ADD y,n,1 %add 1 to n and store the result in y
LDA t,y
TRAP 0,Fputs,StdOut
TRAP 0,Halt,0
What am I doing wrong?
I ended up figuring it out after seeing the code here. I had to first create a byte, then store the value of the register into the byte. Then by printing out that byte, I get the result of ADD y,n,1.
The link in Robert's own response is broken. Also the explanation is unsatisfactory.
The main issue is there is no printf in MMIX assembly. So you can't just print a number directly. It needs to be converted to a string for Fputs to work.
Once you know this the solution is easy. The challenge is to code it in MMIX. The program below handles one unsigned number.
// printnum.mms
// run with MMIX simulator or visual debugger: https://mmix.cs.hm.edu
n IS $4
y IS $3
t IS $255
// a register for extracting a digit
digit IS $5
// a 16-byte buffer for the converted string
buf OCTA 0
LOC #100
Main SET n,1 %let n = 1
ADD y,n,1 %add 1 to n and store the result in y
// convert y to ascii digits and store in buf
GETA t,buf+16
// divide and set digit to the remainder register rR
1H DIV y,y,10
GET digit,rR
// convert digit to ascii character
INCL digit,'0'
// fill buffer from the end
SUB t,t,1
STBU digit,t,0
// loop back to 1H for more digits
PBNZ y,1B
// print the converted string
// this works because string offset is already in register $255
TRAP 0,Fputs,StdOut
TRAP 0,Halt,0
this is my first time trying to program in Fortran. I'm trying to write a program that prints the first 1476 terms of the Fibonacci sequence, then examines the first digit of each term and stores the number of 1s, 2s, 3s, ..., 9s that occur in an array.
The problem that I can't seem to figure out is how to read the first digit of each term. I've tried several things but am having difficulty with my limited knowledge of Fortran techniques. I write the terms to a text file and the idea is to read the first digit of each line and accumulate the respective number in the array. Does anyone have any suggestions of how to do this?
Here is my code so far:
(edit: I included the code I have for reading the file. Right now it just prints out 3.60772951994415996E-313,
which seems like an address of some sort, because it's not one of the Fibonacci numbers. Also, it is the only thing printed, I expected that it would print out every line of the file...)
(edit edit: After considering this, perhaps there's a way to format the writing to the text file to just the first digit. Is there a way to set the number of significant digits of a real number to one? :P)
subroutine writeFib(n)
integer :: i
real*8 :: prev, current, newFib
prev = 0
current = 1
do i = 1, n
newFib = prev + current
prev = current
current = newFib
write(7,*) newFib
end do
return
end subroutine
subroutine recordFirstDigits(a)
integer :: openStat, inputStat
real*8 :: fibNum
open(7, file = "fort.7", iostat = openStat)
if (openStat > 0) stop "*** Cannot open the file ***"
do
read(7, *, iostat = inputStat) fibNum
print *,fibNum
if (inputStat > 0) stop "*** input error ***"
if (inputStat < 0) exit ! end of file
end do
close(7)
end subroutine
program test
integer :: k, a(9)
k = 1476
call writeFib(k)
call recordFirstDigits(a)
end program
Although the suggestions were in place, there were also several things that were forgotten. Range of the REAL kind, and some formatting problems.
Anyways, here's one patched up solution, compiled and working, so try to see if this will work for you. I've took the liberty of choosing my own method for fibonacci numbers calculation.
program SO1658805
implicit none
integer, parameter :: iwp = selected_real_kind(15,310)
real(iwp) :: fi, fib
integer :: i
character(60) :: line
character(1) :: digit
integer :: n0=0, n1=0, n2=0, n3=0, n4=0, n5=0, n6=0, n7=0, n8=0, n9=0
open(unit=1, file='temp.txt', status='replace')
rewind(1)
!-------- calculating fibonacci numbers -------
fi = (1+5**0.5)/2.
do i=0,1477
fib = (fi**i - (1-fi)**i)/5**0.5
write(1,*)fib,i
end do
!----------------------------------------------
rewind(1)
do i=0,1477
read(1,'(a)')line
line = adjustl(line)
write(*,'(a)')line
read(line,'(a1)')digit
if(digit.eq.' ') n0=n0+1
if(digit.eq.'1') n1=n1+1
if(digit.eq.'2') n2=n2+1
if(digit.eq.'3') n3=n3+1
if(digit.eq.'4') n4=n4+1
if(digit.eq.'5') n5=n5+1
if(digit.eq.'6') n6=n6+1
if(digit.eq.'7') n7=n7+1
if(digit.eq.'8') n8=n8+1
if(digit.eq.'9') n9=n9+1
end do
close(1)
write(*,'("Total number of different digits")')
write(*,'("Number of digits 0: ",i5)')n0
write(*,'("Number of digits 1: ",i5)')n1
write(*,'("Number of digits 2: ",i5)')n2
write(*,'("Number of digits 3: ",i5)')n3
write(*,'("Number of digits 4: ",i5)')n4
write(*,'("Number of digits 5: ",i5)')n5
write(*,'("Number of digits 6: ",i5)')n6
write(*,'("Number of digits 7: ",i5)')n7
write(*,'("Number of digits 8: ",i5)')n8
write(*,'("Number of digits 9: ",i5)')n9
read(*,*)
end program SO1658805
Aw, ... I just read you need the number of digits stored in to an array. While I just counted them.
Oh well, ... "left as an exercise for the reader ..." :-)
Can you read with a FORMAT(A1)? It's been 20 years so I don't remember the exact syntax.
I wonder why the open statement succeeds when file 7 hasn't been closed. I think you need an endfile statement and/or a rewind statement in between writing and reading.
Paul Tomblin posted what you have to do after you solve your problem in getting reads to work in the first place.
I am getting an 'end of line' runtime error
You don't show the ! code to read here... which makes it kind of difficult to guess what you are doing wrong :-)
Perhaps you need a loop to read each line and then jump out of the loop to a continue statement when there are no more lines.
Something like this:
do
read(7,*,end=10) fibNumber
end do
10 continue
Better still - look at the more modern style used in this revcomp program.
here are some hints:
You don't need to use characters,
much less file i/o for this problem
(unless you forgot to state that a
file must be created).
Therefore, use math to find your statistics. There are lots of resources on Fibonacci numbers that might provide a simplifying insight or at least a way to independently spot check your answers.
Here is a complicated hint in non-Fortran lingo:
floor(10^(frac(log_10(7214989861293412))))
(Put this in Wolfram Alpha to see what it does.)
A simpler hint (for a different approach) is that you can do very
well in Fortran with simple
arithmetic inside of looping
constructs--at least for a first pass at the solution.
Accumulate your statistics as you
go. This advice would even apply to your character-driven approach. (This problem is ideally suited
for coming up with a cute indexing
scheme for your statistics, but some
people hate cute schemes in
programming. If you don't fear
cuteness ... then you can have associative
arrays in Fortran as long as your
keys are integers ;-)
The most important aspect of this
problem is the data type you will
use to calculate your answers. For
example, here's the last number you
will have to print.
Cheers, --Jared