I know that from their input and output signatures, it's possible to determine the size of a gnuradio block's input and output items. I am wondering whether it's also possible to directly determine input and output type (float vs complex etc) from a block.
Within the GNU Radio runtime, only the size is stored. Type information only exists in the source code, and in GNU Radio Companion if you are using that.
So, no, you cannot get type information from a block object that already exists — except by imperfect outside-information strategies like looking up the block's name in the installed GRC data files to guess what the type is.
Because there are no types but only sizes, items can be reinterpreted if they are the same size, which may occasionally be useful; for example, you can connect a block producing "complex" to one expecting "vector of 2 floats" and get a useful result since a complex is represented as two floats.
Suppose I have a file that each line contains an array index followed by the array value
i array(i)
Can I read in the data by just a naive read(unit=10, *) i, array(i)? Will Fortran always read i first and then use this i value to assign array(i)? Will certain read specifications or compiler flags influence the behavior?
The data transfer statement
read(unit=10,*) i, array(i)
is a legitimate one, and its behaviour is as desired: from the record the value for i is first read, then that value is used to identify the element of the array array(i) for the second value read.
This is a requirement of the Fortran specification, such as with (Fortran 2018, 12.6.4.5.1):
All values needed to determine which entities are specified by an input/output list item are determined at the beginning of the processing of that item.
Of course, although this data transfer statement potentially works, that doesn't mean that it is desirable in all but the simplest cases where you trust the input data. In particular, it is not possible to do any checking of the bounds during this read statement. If the i value read corresponds to an invalid array element specification, the program is broken. You may want to use an intermediate value for the array element merely to handle potential problems with the input file.
I'm having trouble reading an unformatted F77 binary file in Python.
I've tried the SciPy.io.FortraFile method and the NumPy.fromfile method, both to no avail. I have also read the file in IDL, which works, so I have a benchmark for what the data should look like. I'm hoping that someone can point out a silly mistake on my part -- there's nothing better than having an idiot moment and then washing your hands of it...
The data, bcube1, have dimensions 101x101x101x3, and is r*8 type. There are 3090903 entries in total. They are written using the following statement (not my code, copied from source).
open (unit=21, file=bendnm, status='new'
. ,form='unformatted')
write (21) bcube1
close (unit=21)
I can successfully read it in IDL using the following (also not my code, copied from colleague):
bcube=dblarr(101,101,101,3)
openr,lun,'bcube.0000000',/get_lun,/f77_unformatted,/swap_if_little_endian
readu,lun,bcube
free_lun,lun
The returned data (bcube) is double precision, with dimensions 101x101x101x3, so the header information for the file is aware of its dimensions (not flattend).
Now I try to get the same effect using Python, but no luck. I've tried the following methods.
In [30]: f = scipy.io.FortranFile('bcube.0000000', header_dtype='uint32')
In [31]: b = f.read_record(dtype='float64')
which returns the error Size obtained (3092150529) is not a multiple of the dtypes given (8). Changing the dtype changes the size obtained but it remains indivisible by 8.
Alternately, using fromfile results in no errors but returns one more value that is in the array (a footer perhaps?) and the individual array values are wildly wrong (should all be of order unity).
In [38]: f = np.fromfile('bcube.0000000')
In [39]: f.shape
Out[39]: (3090904,)
In [42]: f
Out[42]: array([ -3.09179121e-030, 4.97284231e-020, -1.06514594e+299, ...,
8.97359707e-029, 6.79921640e-316, -1.79102266e-037])
I've tried using byteswap to see if this makes the floating point values more reasonable but it does not.
It seems to me that the np.fromfile method is very close to working but there must be something wrong with the way it's reading the header information. Can anyone suggest how I can figure out what should be in the header file that allows IDL to know about the array dimensions and datatype? Is there a way to pass header information to fromfile so that it knows how to treat the leading entry?
I played a bit around with it, and I think I have an idea.
How Fortran stores unformatted data is not standardized, so you have to play a bit around with it, but you need three pieces of information:
The Format of the data. You suggest that is 64-bit reals, or 'f8' in python.
The type of the header. That is an unsigned integer, but you need the length in bytes. If unsure, try 4.
The header usually stores the length of the record in bytes, and is repeated at the end.
Then again, it is not standardized, so no guarantees.
The endianness, little or big.
Technically for both header and values, but I assume they're the same.
Python defaults to little endian, so if that were the the correct setting for your data, I think you would have already solved it.
When you open the file with scipy.io.FortranFile, you need to give the data type of the header. So if the data is stored big_endian, and you have a 4-byte unsigned integer header, you need this:
from scipy.io import FortranFile
ff = FortranFile('data.dat', 'r', '>u4')
When you read the data, you need the data type of the values. Again, assuming big_endian, you want type >f8:
vals = ff.read_reals('>f8')
Look here for a description of the syntax of the data type.
If you have control over the program that writes the data, I strongly suggest you write them into data streams, which can be more easily read by Python.
Fortran has record demarcations which are poorly documented, even in binary files.
So every write to an unformatted file:
integer*4 Test1
real*4 Matrix(3,3)
open(78,format='unformatted')
write(78) Test1
write(78) Matrix
close(78)
Should ultimately be padded by an np.int32 values. (I've seen references that this tells you the record length, but haven't verified persconally.)
The above could be read in Python via numpy as:
input_file = open(file_location,'rb')
datum = np.dtype([('P1',np.int32),('Test1',np.int32),('P2',np.int32),('P3',mp.int32),('MatrixT',(np.float32,(3,3))),('P4',np.int32)])
data = np.fromfile(input_file,datum)
Which should fully populate the data array with the individual data sets of the format above. Do note that numpy expects data to be packed in C format (row major) while Fortran format data is column major. For square matrix shapes like that above, this means getting the data out of the matrix requires a transpose as well, before using. For non square matrices, you will need to reshape and transpose:
Matrix = np.transpose(data[0]['MatrixT']
Transposing your 4-D data structure is going to need to be done carefully. You might look into SciPy for automated ways to do so; the SciPy package seems to have Fortran related utilities which I have not fully explored.
I am a beginner to Unix programming and C and I have two questions regarding the stat struc and its field st_mode:
When accessing the st_mode field as below, what type of number is returned ( octal, decimal, etc.)?
struct stat file;
stat( someFilePath, &file);
printf("%d", file.st_mode );
I thought the number is in octal but when I ran this code, and I got the value 33188. What is the base?
I found out that the st_mode encodes a 16 bit binary number that represents the file type and file permissions. How do I get the 16-bit number from the above output (especially when it doesn't seem to be in octal). And which parts of the 16-bit digit encode which information?
Thanks for any help.
The actual type behind mode_t and how it encodes information is implementation defined. The only thing that's certain is that it's a bitmask.
To work with st_mode, use the flags and macros defined in the sys/stat.h header. For a list of those defines, consult:
man 2 stat
If you truly need to know what each bit represents, or are simply curious, read the header or use printf to inspect the flags.
I'm trying to visualize some data in vtr format. For this purposes I've created a couple npy files by this library, then I've converted this files by PyEVTK into the vtr format (like in the lowlevel.py example). But when I'm trying to visualize this data by ParaView, an error appears:
ERROR: In /var/tmp/portage/sci-visualization/paraview-4.0.1-r1/work/ParaView-v4.0.1-source/VTK/IO/XML/vtkXMLDataReader.cxx, line 510
vtkXMLRectilinearGridReader (0x36bb080): Cannot read point data array "Pressure" from PointData in piece 0. The data array in the element may be too short.
Can anybody explain, what exactly means this error message, and what's wrong with the my visualization data?
Solved:
I made a stupid mistake - data size in header was different from the actual data size, and this was the cause of error.
This error may be coming from the XML header declaration, that may not contain all data needed. You may miss the header_type that contains the size of each info written between each set of data.
<VTKFile type="UnstructuredGrid" version="0.1" byte_order="BigEndian" header_type="UInt64">