I have a binary file (capture.bin) from the rtl_sdr tool. I convert it to a .cfile with this manual http://sdr.osmocom.org/trac/wiki/rtl-sdr#Usingthedata
Where can I get the data in this file? The goal is to get a numerical format output from the the source. Is this possible?
That actually is covered by a GNU Radio FAQ entry.
What is the file format of a file_sink? How can I read files produced by a file sink?
All files are in pure binary format. Just bits. That’s it. A floating point data stream is saved as 32 bits in the file, one after the other. A complex signal has 32 bits for the real part and 32 bits for the imaginary part. Reading back a complex number means reading in 32 bits, saving that to the real part of a complex data structure, and then reading in the next 32 bits as the imaginary part of the data structure. And just keep reading the data.
Take a look at the Octave and Python files in gr-utils for reading in data using Octave and Python’s Scipy module.
The exception to the format is when using the metadata file format. These files are produced by the File Meta Sink: http://gnuradio.org/doc/doxygen/classgr_1_1blocks_1_1file__meta__sink.html block and read by the File Meta Source block. >See the manual page on the metadata file format for more information about how to deal with these files.
A one-line Python command to read the entire file into a numpy array is:
f = scipy.fromfile(open("filename"), dtype=scipy.uint8)
Replace the dtype with scipy.int16, scipy.int32, scipy.float32, scipy.complex64 or >whatever type you were using.
Update
scipy.fromfile will be deprecated in v2.0 so instead use numpy library
f = numpy.fromfile(open("filename"), dtype=numpy.uint8)
Related
I haven't used openvms for 20+ years. It was my 1st OS. I've been asked if it possible to copy the data from RMS files from openvms server to windows as a text file - so that it's readable.
No-one has experience or knowledge of the record structures etc.
The files are xyz.DAT and are relative files. I'm hoping the dat files are fixed length.
My 1st attempt would be to try and use Datatrieve (DTR) but get an error that the image isn't loaded.
Thought it might be as easy using CONVERT/FDL = nnnn.FDL - by changing the Relative to Sequential. The file seems still to be unreadable.
Is there an easy way to stream an RMS index file to a flat ASCII file?
I use to use COBOL and C to access the data in the past but had lots of libraries to help....
I've notice some solution may use odbc to connect but not sure what I can or cannot install on the server.
I can FTP using Filezilla to the server....
Another plan writing C application to read a file and output out as string.....or DCL too.....doesn't have to be quick...
Any ideas
Has mentioned before
The simple solution MIGHT be to to just use: $ TYPE/OUT=test.TXT test.DAT.
This will handle Relatie and Indexed files alike.
It is much the same as $ CONVERT / FDL=NL: test.DAT test.TXT
Both will just read records from the source and transfer the bytes, byte for byte, to the records in a sequential file.
FTP in ASCII mode will transfer that nicely to windows.
You can also use an 'inline' FDL file to generate a 'unix' LF file like:
$ conv /fdl="record; format stream_lf" test.DAT test.TXT
Or CR-LF file using:
$ conv /fdl="record; format stream" test.DAT test.TXT
Both can be transferring in Binary or Ascii with FTP.
MOSTLY - because this really only works well for TEXT ONLY source .DAT file.
There should be no CR, LF, FF or NUL characters in the source or things will break.
As 'habo' points out, use DUMP /RECORD=COUNT=3 to see how 'readable' the source data is.
If you spot 'binary' data using DUMP then you will need to find a record defintion somewhere which maps byte to Integers or Floating points or Dates as needed.
These defintions can be COBOL LIB files, or BASIC MAPS and are often stores IN the CDD (Common Data Dictionary) or indeed in DATATRIEVE .DIC DICTIONARIES
To use such definition you likely need a program to just read following the 'map' and write/print as text. Normally that's not too hard - notably not when you can find an example program on the server to tweak.
If it is just one or two 'suspect' byte ranges, then you can create a DCL loop to read and write and use F$EXTRACT to select the chunks you like.
If you want further help, kindly describe in words what kind of data is expected and perhaps provide the output from DUMP for 3 or 5 rows.
Good luck!
Hein.
I am new to geospatial analytics and using NetCDF and GeoTIFF files. I am trying to convert the NetCDF file into GeoTIFF file. I came across this reference: netcdf-to-geotiff-file-conversion. I have successfully installed and can run gdal_translate in my MacOS terminal. However, I get this message I am trying to understand what this means and what I am missing? This appears to be warning, however it didn't generate any output.
My code executed:
gdal_translate some_nc_file_name.nc output.tif
Error/Message:
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Input file contains subdatasets. Please, select one of them for reading.
Here is my data output I previewed in Python:
You appear to have multiple variables in the file, so need to select one. Example (following https://nsidc.org/support/how/how-convert-golive-netcdf-variables-geotiff).
gdal_translate NETCDF:"Input_FileName.nc":variable_name Output_FileName.tif
Based on the warning message, the netCDF file lacks certain important attributes, so some issues may exist with the coordinates etc. in the tif.
TL;DR: can I do an incremental read of binary files with Red or Rebol?
I would like to use Red to process some large (13MB to 2GB) structured binary files (Kurzweil synthesizer files). I've used other languages (C, Go, Tcl, Ruby, Dart) to walk through these files, and now I'd like to do the same with Red or Rebol.
Is there a way to incrementally read binary files, byte by byte? All I see is read/binary which seems to slurp the entire file at once (or a part of a file).
I'll need to jump around a little bit, too (either peek at the next byte, or skip to the end of a section, or skip past variable length strings to the start of data).
(Yes, I could make some helpers that tracked the position and used read/part/seek.)
I would like to make a call to the low level OS read/seek if that is possible - something new to learn.
This is on macos, but a portable solution would be great.
Thanks!
PS: "open/read %abc" gives an error "*** Script Error: open does not allow file! for its port argument", even though the help message say the port argument is "port [port! file! url! block!]"
Rebol has ports for that, which are planned for 0.7.0 release in Red. So, current I/O is very basic and buffer-only, and open is a preliminary stub.
I would like to make a call to the low level OS read/seek if that is possible - something new to learn.
You can leverage Rebol or Red/System FFI as a learning excercise.
Here is how you would do it in Rebol:
>> file: open/direct/binary %file.dat
>> until [none? probe copy/part file 20]
>> close file
#{732F7072696E74657253657474696E6773312E62}
#{696E504B01022D00140006000800000021006149}
#{0910890100001103000010000000000000000000}
...
#{000000006A290000646F6350726F70732F617070}
#{2E786D6C504B0506000000000D000D0068030000}
#{292C00000000}
none
first file or pick file 1 will return the next byte value (integer!)
This even works with text files: open/lines/direct, in that case copy/part file 20 will return 20 lines, or you can use pick file 1 or first file to get the next line.
Soon this will be available on Red too.
Please guide me to convert a GenBank sequence to its equivalent FASTA format using biosmalltalk (Pharo edition).
I have already figured out to read a GenBank file from disk:
| file x y m |
x:=Time millisecondClockValue .
file := BioFile on: (FileStream readOnlyFileNamed: BioObject testFilesDirectoryName asFileReference / 'AF165912.gbk').
m:=BioParser tokenizeMultiFasta: file contents.
y:=Time millisecondClockValue.
Transcript open.
Transcript clear.
Transcript show:m;cr.
Now I want to get its FASTA equivalent.
The GenBank format is (supposed to be) a human-readable format but it is not really easily parsable. Lot of efforts has been spent, and still today, in programming libraries to parse the flat GenBank format, when the XML format wasn't available or used at all.
One of the goals behind BioSmalltalk is to focus on complexity reduction, which implies using the right tools. For that reason a GenBank flat parser is not included expecting to favor the usage of the GenBank XML format.
To give a try first install the latest BioSmalltalk in clean Pharo 3.0 evaluating the following command:
$ pharo Pharo.image "config" "http://smalltalkhub.com/mc/hernan/BioSmalltalk" "ConfigurationOfBioSmalltalk" --printVersion --install=development
or its equivalence from inside the image:
Gofer it
smalltalkhubUser: 'hernan' project: 'BioSmalltalk';
configuration;
loadDevelopment.
To parse a GenBank XML formatted file, I highly recommend you to re-download your files in XML format in a reproducible way. If you downloaded your files from NCBI, you can use the Entrez e-Utils BioSmalltalk client (currently the NCBI removed the XML download option from the web page):
The following script downloads two GenBank records in XML, filter nodes by sequence definition and sequence string, and exports them in FASTA format. The sequence is in the GBSeq_sequence node.
| gbReader fastaCollection seqsWithDefs |
fastaCollection := BioFastaMultiRecord new.
gbReader := (BioEntrezClient new nuccore
uids: #(57240072 57240071);
setModeXML;
fetch) reader.
seqsWithDefs := gbReader
selectNodes: #('GBSeq_definition' 'GBSeq_sequence')
in: gbReader contents.
(seqsWithDefs at: 'GBSeq_definition') with: (seqsWithDefs at: 'GBSeq_sequence') do: [ : defs : seqs |
fastaCollection addFastaRecord: (BioFastaRecord named: defs value sequence: seqs value) ].
BioFASTAFormatter new exportFrom: fastaCollection sequences.
If you are starting with Smalltalk, remember you have pure objects and almost everything can be discovered through the inspector/explorer tools.
I'm trying to generate excel file from Oracle by FILE_UTIL. In Oracle document they gave some mode of operations like
W - Write
R - Read
WB -Write Byte
RB - Read Byte
Unable to understand the difference between W and WB. Thanks in advance.
The documentation you're referring to seems to the this, which says slightly more than you indicated in the question:
Specifies how the file is opened. Modes include:
r -- read text
w -- write text
a -- append text
rb -- read byte mode
wb -- write byte mode
ab -- append byte mode
The documentation also says:
byte_mode Indicates whether the file was open as a binary file, or as a text file
So the b indicates byte mode rather than text mode. The file is accessed as a character stream if it's in text mode, so the file should be encoded in the database character set, as it says in the operational notes for that package. And it's accessed as a binary stream in byte mode. Several methods, such as get_line, will raise an exception for a file opened in byte mode as a 'line' has no meaning for binary data.
So it you're processing a file that is text, which could be stored as a CLOB, then use the text-mode flags. If you're processing a file that contains binary data like an image or PDF, which could be stored as a BLOB, use the byte-mode flags.
Excel files contain binary data whether you have a .xls or .xlsx file, so you'd need to use byte mode. If you were generating a .csv file though, you'd probably want text-mode.