Fortran program sometimes hangs while writing large files to disk - file-io

Question

I have written a meshing application that runs quite well except for one thing:
While writing the results to disk, the program sometimes hangs. Well effectively it is still running, I can see it produces 100% CPU load and occupying memory using top. But there is no new data written to the file and this state would probably last indefinitely.
The job can not even be killed, the only option I had so far was rebooting the machine to get rid of the job.
However, most of the time the exact same job executes until the end without any problems. It should be noted that this only happens for large jobs where the final file size is somewhere above the order of 50GB. I haven't hat this behavior with smaller jobs so far. The result file is written up to a different point each time. Sometimes its 45GB, sometimes its 60GB or something completely different.
The workstation I use operates OpenSuse 13.1, has enough RAM and Disk space available. Yet the same behavior could be observed on different machines.
What I tried so far (apart from the usual debugging) was using Gfortran and Intel Fortran compilers to no avail. I messed around with the -mcmodel=large compiler option, but as I understood this is not related to the size of files written by the program and did not help anything.
I don't know what information to provide since I am really running out of ideas what could be causing the problem. I append the routine that writes an ASCII result file here because it produces the largest file size, but I have the same issues when writing unformatted files with a similar routine.
ANY hint on what could possibly cause this behavior is highly appreciated.
SUBROUTINE write_legacymesh
use types
use parameters
use data
implicit none
INTEGER :: l, n, i, j, k, number, c, neighbor
REAL(DP), DIMENSION(3) :: pos_d
CHARACTER(LEN=100) :: dummy_char
INTEGER(I4B), DIMENSION(26) :: neighborbuffer
LOGICAL :: file_exists
call write_header
call system ('sync')
! create the mesh file
dummy_char = folder_mesh // '/' // file_mesh
write(*,'(A,A)') ' writing mesh: ', trim(adjustl(dummy_char))
open(unit=20, file=trim(adjustl(dummy_char)), status='replace', action='write')
! write the mesh file
write(20,'(A)') ' LB database'
l = level_max
! write the header for this mesh size
write(20,'(A)') ' number of collision centers'
write(20,'(I16)') nf(l)
write(20,'(A)') ' number of cut links'
write(20,'(I16)') ncut(l)
write(20,'(A)') ' number of boundary collision centers'
write(20,'(I16)') nb(l)
write(20,'(A)') ' dummy value'
write(20,'(I16)') 0
write(20,'(A)') ' number of inflow boundary collision centers'
write(20,'(I16)') nb1(l)
write(20,'(A)') ' number of pressure boundary collision centers'
write(20,'(I16)') nb2(l)
write(20,'(A)') ' number of slip boundary collision centers'
write(20,'(I16)') nb3(l)
write(20,'(A)') ' number of noslip boundary collision centers'
write(20,'(I16)') nb4(l)
write(20,'(A)') ' lattice spacing'
write(20,'(E16.8)') dx(l)*scaling
! write the coordinates
write(20,'(A)') ' cc RBT value cc coordinates'
do n=1, nf(l)
number = order_level(l)%order(n)
i = levels(l)%blocks(number)%pos(1)
j = levels(l)%blocks(number)%pos(2)
k = levels(l)%blocks(number)%pos(3)
! rescale and move the geometry to get the original size and position
pos_d(1) = (real(i,kind=dp) - 0.5_dp) * dx(l) + corner_ibc(1)
pos_d(2) = (real(j,kind=dp) - 0.5_dp) * dx(l) + corner_ibc(2)
pos_d(3) = (real(k,kind=dp) - 0.5_dp) * dx(l) + corner_ibc(3)
write(20,'(2I16,3E20.7)') n, levels(l)%blocks(number)%state, pos_d*scaling
end do
! write the link info
write(20,'(A)') ' link information'
do n=1, nf(l)
number = order_level(l)%order(n)
do c=1, nlinks
neighbor = levels(l)%blocks(number)%neighbors(c)
if(neighbor .GT. 0) then
neighborbuffer(c) = levels(l)%blocks(neighbor)%number
else
neighborbuffer(c) = neighbor
end if
end do
write(20,'(7I16)') n, neighborbuffer(1:6)
write(20,'(A,6I16)') ' ', neighborbuffer(7:12)
write(20,'(A,6I16)') ' ', neighborbuffer(13:18)
end do
! write the q-values
write(20,'(A)') ' q-values'
do n=1, ncut(l)
write(20,'(I16,E20.7)') n, max(qmin, min(qmax, aux_level(l)%q_values(n)))
end do
write(20,'(A)') ' end of data'
close(unit=20)
END SUBROUTINE write_legacymesh
Edit1: Following the suggestions of flushing write buffers, I added this to every loop that writes data:
if(mod(n,10000000) .EQ. 0) then
call flush(20)
close(unit=20)
call system('sync')
open(unit=20, file=trim(adjustl(dummy_char)), status='old', position='append', action='write')
end if
This did not change anything, the routine still hangs at a random point, this time after writing 18.5GB of data or 211'149'016 write events in the first loop.
Edit2: Deactivating buffered I/O with the Intel Fortran compiler (not using the -assume buffered_io flag) kind of "solved" the issue. But of course this leaves me with slow unbuffered writes for large files, a particularly unpleasant combination.
Edit3: Thanks for the suggestion to add manual write buffers. The file format is not well suited to make use of them efficiently, but its still better than nothing. Anyway, this is a legacy file format and I am positive that I can convince my co-workers that a different file format or even unformatted files are the better option.
While I still have no idea what causes the error, at least I have a tolerable way to avoid it now. Thanks for all the valuable suggestions.