In D, I'm writing to a file:
File opfile = File(opdir~opname, "w");
... //first gap
opfile.writeln("somestuff");
... //second gap
opfile.writeln("otherstuff");
In this case, the stuff in the first gap takes several minutes to run, and the stuff in the second gap takes several hours, and I'd like to see "somestuff" written to the file before the end of the program as a sanity check.
It looks to me like D is using buffered output, and consequently, all the output is written at once after second gap. In C++ I'd use ostream::flush to manually flush opfile prior to the second gap.
What is the equivalent operation in D? I cannot find it in the documentation for std.file.
See opfile.flush()
Calls fflush for the file handle.
Related
I have a code of more than 400 lines, and it takes a long time to run it. I'm at the stage of debugging and was wondering whther its possible to just run only a display command in a particular line? I mean like the option we have in PyCharm that we jsut press shift+enter at the desired line and it'll execute only that line.
Thanks
Short answer: No, you can not run just a single line (unless it is the first one).
Little longer, some way that could still be useful to save some time while debugging:
You could add $exit to run just the lines before that and skip the rest of your model.
If you have "static code" that takes some time at the start before you have other code that is work in progress and you want to save the time for the first part, you can use the "Save and Restart" facility of GAMS (see https://www.gams.com/41/docs/UG_SaveRestart.html). For this, you split the model in parts, lets say, the first part will be saved in longPrep.gms, the second part is wip.gms.
Then, you run gams longPrep.gms s=prep, which will generate a save file prep which will be your starting point for your second part by running gams wip.gms r=prep.
I have 500,000 files which I need to read in Fortran and each file has ~14,000 entries in it (each entry is only about 100 characters long). I need to process each line for each file at a time. For example, I need to process line 1 for all 500,000 files before moving on to line 2 from the files and so forth.
I cannot open them all at once (I tried making an array of file pointers and opening them all) because there will be too many files open at once. Instead, I would like to do something as follows:
do iline = 1,Nlines
do ifile = 1,Nfiles
! open the file
! read a line
! close the file
enddo
end
In hopes that this would allow me to read one line at a time (from each file) and then move on to the next line (in each file). Unfortunately, each time I open the file it starts me off at line 1 again. Is there any way to open/close a file and then open it again where you left off previously?
Thanks
Unfortunately it is not possible in this way in standard Fortran. Even If you specify
position="ASIS"
the actual position will be unspecified for a not already connected unit and will be in fact the beginning of the file on most systems.
That means You have to use
read(*,*)
enough times to get on the right place in the file.
You could also use stream access. The file would be again opened at the beginning, but you can use
read(u,*,pos=n) number
where n is the position saved from the previous open. You can get the position from
inquire(unit=u, pos=n)
n = n
You would open the file with acess="STREAM".
Also 500000 opened files is indeed too much. There are ways how to inquire for the system limits and how to control them, but also your compiler may have some limits http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Other solution: Couldn't you store the content of the files in memory? Today couple of Gigabytes is OK, but it may be not enough for you.
You can try using fseek and ftell in something like the following.
! initialize an array of 0's
do iline = 1,Nlines
do ifile = 1,Nfiles
! open the file
! fseek(fd, array(ifile))
! read a line
! array(ifile)=ftell(fd)
! close the file
enddo
end
The (untested) idea is to store the offset of each file in an array and position the cursor at that place upon opening the file. Then, once a line is read, the ftell retrieves the current position which is saved to memory for next round. If all entries have the same length, you can spare the array and just store one value.
If the files have fixed, i.e., constant, record lengths, you could use direct access. Then you could "directly" read a specific record. A big "if" however.
the overhead of all the file opening/closing will be a big performance bottleneck.
You should try to read as much as you can for each open operation given whatever memory you have:
pseudocode:
loop until done:
loop over all files:
open
fseek !as in damiens answer
read N lines into array ! N=100 eg.
save ftell value for file
close
end file loop
loop over N output files:
open
write array data
close
The title says it all.
I have 4 awk processes logging to the same file, and output seems fine, not mangled, but I'm not sure that just redirecting print output like this: print "xxx" >> file in every process is the right way to do it.
There are many similar questions around the site, but this one is particularly about awk and a pragmatic, code-correct way to approach the problem.
EDIT
Sorry folks, of course I wasn't "just redirecting" like I wrote, I was appending.
No it is not safe.
the awk print "foo" > "file" will open the file and overwrite the file content, till the end of script.
That is, if your 4 awk processes started writing to the same file on different time, they overwrite the result of each other.
To reproduce it, you could start two (or more) awk like this:
awk '{while(++i<9){system("sleep 2");print "p1">"file"}}' <<<"" &
awk '{while(++i<9){system("sleep 2");print "p2">"file"}}' <<<"" &
and same time you monitoring the content of file, you will see finally there are not exactly 8 "p1" and 8 "p2".
using >> could avoid the losing of entries. but the entry sequence from 4 processes could be messed up.
EDIT
Ok, the > was a typo.
I don't know why you really need 4 processes to write into same file. as I said, with >>, the entries won't get lost (if you awk scripts works correctly). however personally I won't do in this way. If I have to have 4 processes, i would write to different files. well I don't know your requirement, just speaking in general.
outputting to different files make the testing, debugging easier.. imagine when one of your processes had problem, you want to solve it. etc...
I think using the operating system print command is save. As in fact this will append the file write buffer with the string you provide as log. So the system will menage the actual writing process of the data to disc, also if another process will want to use the same file the system will see that the resource is already claimed and will wait for 1st thread to finish its processing, than will allow the 2nd process to write to the buffer.
I have a FORTRAN code which performs a numerical integration. All the computed data are written in an external file (data.out). Here is a simple sketch of the code
OPEN(UNIT=10,FILE='data.out')
DO i=1,n
........
WRITE(10,'(7(E22.16,1x))')a,b,c,d,e,f,g
........
ENDDO
CLOSE(10)
The program keeps running for long time (about 1.5 h) until the numerical integration is finished. During the execution I would like to see the results in the .out file. However, when I try to open the .out file while the .exe is running I get the following message:
"The document data.out is used by another application and cannot be accessed." So, is there a way to open the .out file during the execution? It is important for me to observe the output values (they are not only seven as in the above example) so, it is not convenient to send them in screen output (is reduces significantly the speed of the code).
Many thanks in advance.
* EDIT *
This is another scenario quite similar to the above mentioned case. Here, the integration routine reads the initial conditions from an input file and writes the outputs to another external file. Below I present the corresponding skeleton of the code
OPEN(UNIT=10,FILE='input.par',STATUS='UNKNOWN')
2 READ(10,*,END=1) x_0,y_0
! INTEGRATION LOOP
DO i=1,n
........
ENDDO
OPEN(UNIT=12,FILE='data.out')
WRITE(12,'(7(E22.16,1x))')a,b,c,d,e,f,g
GOTO 2
1 CLOSE(10)
CLOSE(12)
So, the routine opens UNIT 10 reads the initial conditions, performs the integration and at the end of the integration it writes the outputs to UNIT 12. Then, it takes another set of initial conditions and repeat the same procedure until it there is no more initial conditions in UNIT 10. Again, I want to be able to open and monitor UNIT 12. I tried your approach but it does not work properly at this case. I can open UNIT 12 any time I want but the routine does not write all the outputs there. In fact, it writes only the outputs of the last set of initial conditions. Any ideas? I strongly believe that a minor modification to your approach could do the job.
Alternative to printing out the result to the screen, you could just close your file after each write operation and reopen it in the next integration cycle to append the new data to it.
program test
implicit none
integer :: ii
! Create emtpy file
open(10, file='data.out', status='REPLACE', action='WRITE')
close(10)
do ii = 1, 100
!...
! Reopen file, append new information and close it again.
open(10, file='data.out', status='OLD', action='WRITE', position='APPEND')
write(10,'(7(E23.16,1x))') 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0
close(10)
end do
end program test
EDIT: THe main idea is, that before you start any kind of looping, you create an empty file by the first open statement:
open(10, file='data.out', status='REPLACE', action='WRITE')
close(10)
Then, inside the loop you just append to that file, to make sure you do not replace content already being there:
open(10, file='data.out', status='OLD', action='WRITE', position='APPEND')
write(10,'(7(E23.16,1x))') 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0
close(10)
Please note the differences in the arguments passed to the open statement.
The logical way to do this would be to echo the output to the terminal. By default, this is unit 6. So I would change your output to:
WRITE(6,'(7(E22.16,1x))')a,b,c,d,e,f,g
WRITE(10,'(7(E22.16,1x))')a,b,c,d,e,f,g
Try it and see. It may even work.
;'2706 410'
~{.#\%.}do;
From the GCD example.
It looks like the pop and discard at the start will do nothing, so why is it there?
The program starts with the contents of standard input at the top of the stack. The pop discards this unused input so that it is not printed when the program exits.
From the tutorial:
There is no explicit input command in GolfScript, instead when your script is executed, all input from stdin is read first and placed as a string onto the stack.
and:
When your script reaches the end. The contents of the stack are printed automatically.