I'm having this error in a python script:
TypeError: 'str' does not support the buffer interface
The line that is generating the error is
username = cred_file.readlines()[0].split(';')[0]
I'm a python beginner, any help is appreciated.
You're running a python 2 script with python 3. Python 3 now returns bytes no longer str when reading from a binary stream.
3 choices:
run it with python 2. That if you don't have the rights/time to adapt the script, not recommended as python 3 is becoming more and more the norm.
change your code to insert a decode function (it will continue to work in python 2):
username = cred_file.readlines()[0].decode().split(';')[0]
If file is opened in read/binary mode, readlines returns a list of bytes not str. You have do decode the bytes into a str to apply str methods.
open the file in "r" instead of "rb". readlines then returns a list of str and your code will work. Sometimes it can be problematic on windows because of need to preserve the carriage return (\r) chars, so look out for side effects in your code.
Note: cred_file.readlines()[0] is a questionable construction: you're reading the whole file lines, and drop all the lines but the first. Not very efficient I/O and CPU wise.
Prefer that: cred_file.readline() which is equivalent to read the first line.
If you need to read all the lines for further processing, then store the result of readlines in a list.
Related
TL;DR: can I do an incremental read of binary files with Red or Rebol?
I would like to use Red to process some large (13MB to 2GB) structured binary files (Kurzweil synthesizer files). I've used other languages (C, Go, Tcl, Ruby, Dart) to walk through these files, and now I'd like to do the same with Red or Rebol.
Is there a way to incrementally read binary files, byte by byte? All I see is read/binary which seems to slurp the entire file at once (or a part of a file).
I'll need to jump around a little bit, too (either peek at the next byte, or skip to the end of a section, or skip past variable length strings to the start of data).
(Yes, I could make some helpers that tracked the position and used read/part/seek.)
I would like to make a call to the low level OS read/seek if that is possible - something new to learn.
This is on macos, but a portable solution would be great.
Thanks!
PS: "open/read %abc" gives an error "*** Script Error: open does not allow file! for its port argument", even though the help message say the port argument is "port [port! file! url! block!]"
Rebol has ports for that, which are planned for 0.7.0 release in Red. So, current I/O is very basic and buffer-only, and open is a preliminary stub.
I would like to make a call to the low level OS read/seek if that is possible - something new to learn.
You can leverage Rebol or Red/System FFI as a learning excercise.
Here is how you would do it in Rebol:
>> file: open/direct/binary %file.dat
>> until [none? probe copy/part file 20]
>> close file
#{732F7072696E74657253657474696E6773312E62}
#{696E504B01022D00140006000800000021006149}
#{0910890100001103000010000000000000000000}
...
#{000000006A290000646F6350726F70732F617070}
#{2E786D6C504B0506000000000D000D0068030000}
#{292C00000000}
none
first file or pick file 1 will return the next byte value (integer!)
This even works with text files: open/lines/direct, in that case copy/part file 20 will return 20 lines, or you can use pick file 1 or first file to get the next line.
Soon this will be available on Red too.
I try to compile my code using gfortran. I got this error:
**INQUIRE(inpunit,flen=iflen)
1
Error: Syntax error in INQUIRE statement at (1)**
This code was compiled before with lahey. With a quick research I find that parameters of INQUIRE have different meaning in gfortran compared to lahey.
inpunit is a scalar INTEGER expression that evaluates to the
input/output unit number of an external file.
flen is a scalar default INTEGER variable having the length of the file in bytes.
My question is when using gfortran is this statement correct to get the same functionality as in Lahey:
**INQUIRE(inpunit,RECL=iflen) **
Are these two statements similar?
Thanks
No, these two are completely different.
flen= is a nonstandard extension specific to the Leahy compiler and returns the length of the file.
recl= is the maximum record length in the file (if the file is connected - opened, otherwise it is 0) https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference-inquire-recl-specifier
To be standard conforming you should use size=. Be aware that the result will be in file storage units. Gfortran uses bytes, but other compilers may use 4-byte words.
See What is a good way to get file size in bytes using Fortran (ifort)? Find input file size in fortran90
I have a binary file (capture.bin) from the rtl_sdr tool. I convert it to a .cfile with this manual http://sdr.osmocom.org/trac/wiki/rtl-sdr#Usingthedata
Where can I get the data in this file? The goal is to get a numerical format output from the the source. Is this possible?
That actually is covered by a GNU Radio FAQ entry.
What is the file format of a file_sink? How can I read files produced by a file sink?
All files are in pure binary format. Just bits. That’s it. A floating point data stream is saved as 32 bits in the file, one after the other. A complex signal has 32 bits for the real part and 32 bits for the imaginary part. Reading back a complex number means reading in 32 bits, saving that to the real part of a complex data structure, and then reading in the next 32 bits as the imaginary part of the data structure. And just keep reading the data.
Take a look at the Octave and Python files in gr-utils for reading in data using Octave and Python’s Scipy module.
The exception to the format is when using the metadata file format. These files are produced by the File Meta Sink: http://gnuradio.org/doc/doxygen/classgr_1_1blocks_1_1file__meta__sink.html block and read by the File Meta Source block. >See the manual page on the metadata file format for more information about how to deal with these files.
A one-line Python command to read the entire file into a numpy array is:
f = scipy.fromfile(open("filename"), dtype=scipy.uint8)
Replace the dtype with scipy.int16, scipy.int32, scipy.float32, scipy.complex64 or >whatever type you were using.
Update
scipy.fromfile will be deprecated in v2.0 so instead use numpy library
f = numpy.fromfile(open("filename"), dtype=numpy.uint8)
I am trying to read in a .gda file into IDL for plotting purposes. I am not familiar with the format and my research indicates it is an unformatted binary data file type. Anyways, here is what I am doing:
pro omidi_contour
openr, 1, 'data.gda'
a = fltarr(128,128,128)
readu, 1, a
close, 1
end
However when I look at the variable definition at the left panel of IDL, it indicates that a is 'undefined'. When I try to print:
print, a[0,0,0]
I get:
Variable is undefined: A
How can I solve this?
I found out that there was nothing wrong with my program. It was reading the right values from the file. However, the IDL "forgot" the value of the variables once the program was done. Solution: DO not run this as a program i.e. remove the following lines:
pro omidi_contour
end
This makes the code to run as if each line were typed into the IDL prompt and IDL does remember the values this time round.
I like to scan larger(>500M) binary files for structs/patterns. I am new to the language an hope that someone can give me start. Actually the files are a database containing Segments. A Segment starts with a fixed sized header followed by a fixed sized optional part followed by the payload/data part of variable lenght. For a first test i just like to log the number of segments in the file. I googled already for some tutorial but found nothing which helped. I need a hint or a tutorial which is not too far from my use case to get started.
Greets
Stefan
you need to learn about Bit Syntax and Binary Comprehensions. More useful links to follow: http://www.erlang.org/documentation/doc-5.6/doc/programming_examples/bit_syntax.html, and http://goto0.cubelogic.org/a/90.You will also need to learn how to process files, reading from files (line-by-line, chunk-by-chunk, at given positions in a file, e.t.c.), writing to files in several ways. The file processing functions are explained here You can also choose to look at the source code of large file processing libraries within the erlang packages e.g. Disk Log, Dets and mnesia. These libraries heavily read and write into files and their source code is open for you to see. I hope that helps
Here is a synthesized sample problem: I have a binary file (test.txt) that I want to parse. I want to find all the binary patterns of <<$a, $b, $c>> in the file.
The content of "test.txt":
I arbitrarily decide to choose the string "abc" as my target string for my test. I want to find all the abc's in my testing file.
A sample program (lab.erl):
-module(lab).
-compile(export_all).
find(BinPattern, InputFile) ->
BinPatternLength = length(binary_to_list(BinPattern)),
{ok, S} = file:open(InputFile, [read, binary, raw]),
loop(S, BinPattern, 0, BinPatternLength, 0),
file:close(S),
io:format("Done!~n", []).
loop(S, BinPattern, StartPos, Length, Acc) ->
case file:pread(S, StartPos, Length) of
{ok, Bin} ->
case Bin of
BinPattern ->
io:format("Found one at position: ~p.~n", [StartPos]),
loop(S, BinPattern, StartPos + 1, Length, Acc + 1);
_ ->
loop(S, BinPattern, StartPos + 1, Length, Acc)
end;
eof ->
io:format("I've proudly found ~p matches:)~n", [Acc])
end.
Run it:
1> c(lab).
{ok,lab}
2> lab:find(<<"abc">>, "./test.txt").
Found one at position: 43.
Found one at position: 103.
I've proudly found 2 matches:)
Done!
ok
Note that the code above is not very efficient (the scanning process shifts one byte at a time) and it is sequential (not utilizing all the "cores" on your computer). It is meant only to get you started.
When your data fits into memory, best thing what you can to do is read data in whole using file:read_file/1. If you can't use file in raw mode. Then you can parse data using bit_syntax. If you write it in right manner you can achieve parsing speed in tens of MB/s when parsing module is compile using HiPE. Exact techniques of parsing depends on exact segment data format and how robust/accurate result you are looking for. For parallel parsing you can inspire by Tim Bray's Wide Finder project.