Hello I fairly new to the DWARF standard and ELF format. I have a few questions. I am using the DWARF 2 standard and I have a pretty basic understanding of how DIEs work and I was needing more clarity on how they are represented in bytes.
ELF Wiki provides a good table for in which order the bytes go in the program header, sections, and segments. But what is the correct way to represent DIEs in bytes for the DWARF 2 standard?
I have tried to dive deep into Dwarf Standards pdf documents to try to understand how DIEs are represented in bytes. Perhaps there is a section I am missing?
I would like to use this information to be able to delete certain DIEs to save space in the debugging section. I am only interest in the DIEs that provide variable address's.
I recommend that anyone starting out in DWARF begin with the Introduction to the DWARF Debugging Format. It's a very concise overview that provides an excellent foundation for exploring the format in further depth. Armed with this background, compile a debug version of a very simple program and compare a hex dump of the two ELF sections .debug_abbrev and .debug_info with the output of dwarfdump or readelf.
Once you are broadly familiar with the encoding of a DIE you will see that simply deleting its corresponding bytes from .debug_info would corrupt the entire file — in terms of both DWARF and ELF. For example, each DIE is identified by its relative file offset; deleting one DIE's bytes would alter the offsets of all subsequent DIEs and any references to them would therefore be broken. A robust solution would require parsing the DWARF to create an internal representation of the tree before eliminating unwanted nodes and writing out new DWARF. After modifying .debug_info you'd then need to edit the fabric of the ELF itself: at the very least, this would involve updating the section header table to reflect the new offsets for any shifted sections and updating any relocations.
If your principal concern is indeed space saving then I suggest you instead investigate what compiler options you have. The Oracle Studio Compilers, for example, allow fine control over the content included in the DWARF. Depending on your compiler and OS it may also be possible to emit files with compressed DWARF sections (e.g. .zdebug_info) or even leave the DWARF in different files altogether. The problem of DWARF bloat is well known and, if you are interested in tackling it at a low level yourself, you will find other suggestions in Michael Eager's introduction and in later versions of the standard.
The format is explained page 66 in sections 7.5.2 and 7.5.3.
The example in appendix 2, page 93 is is much clearer:
Each DIE references a corresponding entry in .debug_abbrev which defines a given DIE "signature" i.e.
its type (DW_TAG_*)
it has child DIE
its attribute (DW_AT_*) and their form (DW_FORM_*).
The format od the DIE is:
reference to a abbreviation (LEB128 i.e. variable length);
0 is used for ending a list of children̄;
une value per attribute (using the encoding associated with the given form).
In protobuf, we only have the choice of using signed or unsigned 32- or 64-bit integer to limit the range of a value.
However, the datastructure I want to define contains a mixture of 8-bit, 16-bit and 32-bit integers to save space on embedded devices. On them, the datastructure is also implemented somewhat differently and requires reserved special values for some fields, so the maximum number for them is not a power of 2.
On these embedded devices, the protobuf definition is only used for transmission to and from them, not for actual storage. So I could just limit the numbers when reading them in.
However, I'd rather define these maximum values in the .proto or .options file to make sure all client applications are aware of these limitations.
Is there a way to do this?
I know there are field options, but the ones listed here does not include an option for this. It is possible to create custom options, but that seems to require writing a compiler extension, which means I have to manually implement this limit checking for every language I want to compile to, and that costs more time than it will ever save.
This is not possible in protobuf by default, and the specification includes no syntax to enforce limits like this.
However, some third party implementations do include such support.
For example, my own nanopb has the int_size option:
int_size: Override the integer type of a field. (To use e.g. uint8_t to save RAM.)
This will return an error at runtime from pb_decode() if the value does not fit in the field.
No there is no syntax for expressing that intent and no inbuilt tool / codegen that will enforce the rule you want to add. You would need to handle this manually.
Lets say you have some code in your app with a hard coded string.
If somevalue = "test123" Then
End If
Once the application is compiled, is it possible for someone to modify the .exe file and change 'test123' to something else? If so, would it only work if the string contained the same number of characters?
It's possible but not necessarily straightforward. For example, if your string is loaded in memory, someone could use a memory manager tool to modify the value of the string's address directly.
Alternatively, they could decompile your app, change the string, and recompile it to create a new assembly with the new string. However, whether this is likely to happen depends on your app and how important it is for that string to be changed.
You could use an obfuscator to make it a bit harder to do but, ultimately, a determined cracker would be able to do it. The question is whether that string is important enough to worry about and, if so, maybe consider an alternative approach such as using a web service to provide the string.
Strings hard-coded without any obfuscation techniques can easily be found inside compiled executables by openign them up in any HEX-editor. Once found, replacing the string is possible in 2 ways :
1. Easy way (*conditions apply)
If the following conditions apply in your case, this is a very quick-fire way of modifying the hard-coded strings in the executable binary.
length(new-string) <= length(old-string)
No logic in the code to check for executable modification using CRC.
This is a viable option ONLY if the new string is equal or shorter than the old string. Use a hex-editor to find occurrences of the old string and replace it with the new string. Pad an extra space with NULL i.e. 0x00
For example old-long-string in the binary
is modified to a shorter new-string and padded with null characters to the same length as the original string in the binary executable file
Note that such modifications to the executable files are detected by any code that verifies the checksum of the binary file against the pre-calculate checksum of the original binary executable file.
2. Harder way (applicable in almost all cases)
De-compiling the binary to native code opens up the possibility to modify any strings (and even code) and rebuild it to obtain the new binary executable.
There exist dozens of such de-compiler tools to decompile vb.net (Visual Studio.net, in general). An excellent detailed comparison of the most popular ones (ILspy, JustDecompile, DotPeek, .NET Reflector to name a few ) can be found here.
There do exist scenarios in which even the harder way will NOT be successful. This is the case when the original developer has used obfuscation techniques to prevent the strings from being detected and modified in the executable binary. One such obfuscation technique is storing encrypted strings.
I have a Serial number string "1080910" embedded in a programmable device which has been downloaded to a binary file using the ALL-100 programmer. This is my Master file as it were. I need to change this serial number to that of the unit that I need to re-flash using the Master file - the ALL-100 programmer uses XACCESS User Interface which has Edit feature showing Address location, Hex data field and Ascii field. Somewhere in this file is the serial number string - can anybody assist me in how to locate and edit the serial number string as I have been unable to locate it using the search function and have not been able to visually pick up the sequence of numbers. Help !!!
If the data has a symbolic address in the source code, and is not a local variable, its address will appear in the map file generated by the linker. If it is a local variable initialised with a literal constant, then the data will exist in the static initialisation data the location of which should also be identified in the map file.
Another possibility is that your application image is compressed and the start-up code expands it into RAM at run-time. This will be obvious in the map file if the data and code addresses are in RAM rather than ROM. If this is the case then what you are attempting will be very difficult. You would have to know the compression algorithm used, and which part of the image is the commpressed part (part of it will be the decompression code that runs from ROM). You would then have to decompress the image, modify the string, and then recompress it. Further, if the decompression performs any kind of checksum on the compressed or decompressed data, you will have to recalculate and modify that too.
If this was a requirement from the outset, you would have done better to reserve the space in the linker script or use compiler specific extensions to absolutely locate the data at a specific location.
Maybe it is stored in Unicode, so alternate chars are 00.
I am looking for a scripting (or higher level programming) language (or e.g. modules for Python or similar languages) for effortlessly analyzing and manipulating binary data in files (e.g. core dumps), much like Perl allows manipulating text files very smoothly.
Things I want to do include presenting arbitrary chunks of the data in various forms (binary, decimal, hex), convert data from one endianess to another, etc. That is, things you normally would use C or assembly for, but I'm looking for a language which allows for writing tiny pieces of code for highly specific, one-time purposes very quickly.
Any suggestions?
Things I want to do include presenting arbitrary chunks of the data in various forms (binary, decimal, hex), convert data from one endianess to another, etc. That is, things you normally would use C or assembly for, but I'm looking for a language which allows for writing tiny pieces of code for highly specific, one-time purposes very quickly.
Well, while it may seem counter-intuitive, I found erlang extremely well-suited for this, namely due to its powerful support for pattern matching, even for bytes and bits (called "Erlang Bit Syntax"). Which makes it very easy to create even very advanced programs that deal with inspecting and manipulating data on a byte- and even on a bit-level:
Since 2001, the functional language Erlang comes with a byte-oriented datatype (called binary) and with constructs to do pattern matching on a binary.
And to quote informIT.com:
(Erlang) Pattern matching really starts to get
fun when combined with the binary
type. Consider an application that
receives packets from a network and
then processes them. The four bytes in
a packet might be a network byte-order
packet type identifier. In Erlang, you
would just need a single processPacket
function that could convert this into
a data structure for internal
processing. It would look something
like this:
processPacket(<<1:32/big,RestOfPacket>>) ->
% Process type one packets
...
;
processPacket(<<2:32/big,RestOfPacket>>) ->
% Process type two packets
...
So, erlang with its built-in support for pattern matching and it being a functional language is pretty expressive, see for example the implementation of ueencode in erlang:
uuencode(BitStr) ->
<< (X+32):8 || <<X:6>> <= BitStr >>.
uudecode(Text) ->
<< (X-32):6 || <<X:8>> <= Text >>.
For an introduction, see Bitlevel Binaries and Generalized Comprehensions in Erlang.You may also want to check out some of the following pointers:
Parsing Binaries with erlang, lamers inside
More File Processing with Erlang
Learning Erlang and Adobe Flash format same time
Large Binary Data is (not) a Weakness of Erlang
Programming Efficiently with Binaries and Bit Strings
Erlang bit syntax and network programming
erlang, the language for network programming (1)
Erlang, the language for network programming Issue 2: binary pattern matching
An Erlang MIDI File Reader/Writer
Erlang Bit Syntax
Comprehending endianness
Playing with Erlang
Erlang: Pattern Matching Declarations vs Case Statements/Other
A Stream Library using Erlang Binaries
Bit-level Binaries and Generalized Comprehensions in Erlang
Applications, Implementation and Performance Evaluation of Bit Stream Programming in Erlang
perl's pack and unpack ?
Take a look at python bitstring, it looks like exactly what you want :)
The Python bitstring module was written for this purpose. It lets you take arbitary slices of binary data and offers a number of different interpretations through Python properties. It also gives plenty of tools for constructing and modifying binary data.
For example:
>>> from bitstring import BitArray, ConstBitStream
>>> s = BitArray('0x00cf') # 16 bits long
>>> print(s.hex, s.bin, s.int) # Some different views
00cf 0000000011001111 207
>>> s[2:5] = '0b001100001' # slice assignment
>>> s.replace('0b110', '0x345') # find and replace
2 # 2 replacements made
>>> s.prepend([1]) # Add 1 bit to the start
>>> s.byteswap() # Byte reversal
>>> ordinary_string = s.bytes # Back to Python string
There are also functions for bit-wise reading and navigation in the bitstring, much like in files; in fact this can be done straight from a file without reading it into memory:
>>> s = ConstBitStream(filename='somefile.ext')
>>> hex_code, a, b = s.readlist('hex:32, uint:7, uint:13')
>>> s.find('0x0001') # Seek to next occurence, if found
True
There are also views with different endiannesses as well as the ability to swap endianness and much more - take a look at the manual.
I'm using 010 Editor to view binary files all the time to view binary files.
It's especially geared to work with binary files.
It has an easy to use c-like scripting language to parse binary files and present them in a very readable way (as a tree, fields coded by color, stuff like that)..
There are some example scripts to parse zipfiles and bmpfiles.
Whenever I create a binary file format, I always make a little script for 010 editor to view the files. If you've got some header files with some structs, making a reader for binary files is a matter of minutes.
Any high-level programming language with pack/unpack functions will do. All 3 Perl, Python and Ruby can do it. It's matter of personal preference. I wrote a bit of binary parsing in each of these and felt that Ruby was easiest/most elegant for this task.
Why not use a C interpreter? I always used them to experiment with snippets, but you could use one to script something like you describe without too much trouble.
I have always liked EiC. It was dead, but the project has been resurrected lately. EiC is surprisingly capable and reasonably quick. There is also CINT. Both can be compiled for different platforms, though I think CINT needs Cygwin on windows.
Python's standard library has some of what you require -- the array module in particular lets you easily read parts of binary files, swap endianness, etc; the struct module allows for finer-grained interpretation of binary strings. However, neither is quite as rich as you require: for example, to present the same data as bytes or halfwords, you need to copy it between two arrays (the numpy third-party add-on is much more powerful for interpreting the same area of memory in several different ways), and, for example, to display some bytes in hex there's nothing much "bundled" beyond a simple loop or list comprehension such as [hex(b) for b in thebytes[start:stop]]. I suspect there are reusable third-party modules to facilitate such tasks yet further, but I can't point you to one...
Forth can also be pretty good at this, but it's a bit arcane.
Well, if speed is not a consideration, and you want perl, then translate each line of binary into a line of chars - 0's and 1's. Yes, I know there are no linefeeds in binary :) but presumably you have some fixed size -- e.g. by byte or some other unit, with which you can break up the binary blob.
Then just use the perl string processing on that data :)
If you're doing binary level processing, it is very low level and likely needs to be very efficient and have minimal dependencies/install requirements.
So I would go with C - handles bytes well - and you can probably google for some library packages that handle bytes.
Going with something like Erlang introduces inefficiencies, dependencies, and other baggage you probably don't want with a low-level library.