I'm working on a program with a plugin system which works by the plugins deriving from an abstract class. The program is written in c++ and built with g++.
Obviously when new functions are added to the abstract class it will make the plugin interface incompatible with previous versions. I would like to automatically detect this and mark the plugins as incompatible.
To do this I would like to get the size of the virtual method table for the abstract class. Is there any way to do so for a program compiled with g++ (I know this will not be possible using only standard c++ features).
I would prefer a solution that works within the program, but a method that involves looking at the object files with external tools would also be acceptable.
What I ended up doing (guided somewhat by the comments). This goes a bit further than simply the size of the virtual method table.
First I used sed to replace "= 0" with "{}", the reason for doing this is that the dumped class hierarchy has more information for real methods than for pure virtual ones. Then I piped the output of that into g++ with the -fdump-class-hierarchy option.
sed 's/= 0/{}/' <my header> | g++ -c -fdump-class-hierarchy <search path arguments> -x c++ -o dummy.o -"
This created a file called "-.002t.class" (I understand the middle part varies with the g++ version) with details of all the VMTs and the size and alignment of the classes (but sadly not full details of their fields)
I then used python to pick out the sections for the class I was interested in, edit out the large hexadecimal numbers (I'm not sure of the meaning of said numbers but they do not appear to be related to the structure of the classes, i'm guessing they are addresses in the object file or something) and feed them (along with some other data) to a hash function to generate a "plugin compatibility ID".
Related
In https://opensource.apple.com/source/cctools/cctools-836/RelNotes/CompilerTools.html
The static linker has an option to do dead code stripping The static
link editor now can do dead code stripping. The new ld(1) option to do
this is -dead_strip. There is also an additional option
-no_dead_strip_inits_and_terms that can be used when -dead_strip is specified to cause all constructors and destructors to never be dead
code stripped. The load map printed with the ld(1) option -M notes
what was dead stripped from the input files.
However, due to dynamic runtime property of Objective-C, dead_strip does not work well on Objective-C.
So in my understanding, it does not work on the level of functions.
Can it work on the level of classes, i.e. if one class is not used, can it be removed effectively?
I'm trying to learn PyQt5 and I am finding it very difficult since I can't just guess what methods are available. I've just spent an entire week trying to find a method to simulate a button push. I eventually found the solution ( QPushButton.animateClick() ) only after stumbling across an example someone left out there (how did this person know this?). It's very difficult to develop without some reference to what's available for tools!
Riverbank has a version of what I'm looking for but it is not complete making it virtually useless.
pyqt5 being a qt5 binding has almost all the functionalities (there are minimal known incompatibilities) so the qt5 documentation: https://doc.qt.io/ is valid for pyqt5 except for small exceptions.
Although the target of the documentation is c++ the description of the classes and methods are generic, so they also validly apply for pyqt5, on the other hand many of the examples are written in c++ but the translation to python in many cases is trivial .
So to avoid doing a double task it seems that Riverbank Computing Limited only documents the exceptions indicated in the pyqt5 docs: https://www.riverbankcomputing.com/static/Docs/PyQt5/
The next part of my answer will propose tips to handle the Qt documentation.
The Qt documentation also has an easy to understand structure, for example let's analyze the QPushButton class (https://doc.qt.io/qt-5/qpushbutton.html):
At the top there is a table:
This table indicates how to include the class in a C++ project, how to add it to qmake, from which class it inherits, and which classes inherit from it. From the above, relevant information for PyQt5 can be extracted, such as to which sub-module the class belongs to: In this case we use QT += widgets that inform us that it belongs to the QtWidgets sub-module, in general if Qt += submodulefoo belongs to QtSubModuleFoo (camelcase)
If you want to know all the methods of the QPushButton class, you must use the "List of all members, including inherited members" link below the table, in this case the link will be https://doc.qt.io/qt-5/qpushbutton-members.html where is the complete list of all class methods, enumerations, etc.
Other tips to understand the conversion between Qt/C++ and PyQt5/Python are:
Some methods use pointers to receive information such as:
void QLayout::getContentsMargins(int *left, int *top, int *right, int *bottom) const
bool QProcess::startDetached(qint64 *pid = nullptr), etc
those transformed to PyQt5 as:
lay = QtWidgets.QXLayout()
left, top, right, bottom = lay.getContentsMargins()
process = QProcess()
# ...
ok, pid = process.startDetached()
Some methods collide with reserved words such as exec , raise, print, etc so to avoid incompatibilities, the underscore is added at the end: exec_, raise_, print_, etc
In Qt, the Q_SLOT and Q_SIGNAL that are translated into python are used through the #pyqtSlot and #pyqtSignal decorators.
In conclusion, my recommendation is that you use the Qt and PyQt5 documentation at the same time to know all the functionalities, in addition there are many Q&A in SO about translations from one language to another so you could learn from there.
The Qt documentation can also be consulted using the Qt Assistant tool.
The main PyQt5 documentation is on the official website:
https://www.riverbankcomputing.com/static/Docs/PyQt5/
But it's still incomplete, and most parts refer to the official Qt documentation:
https://doc.qt.io/qt-5/
While that's C++ oriented, consider that almost every module, class and function behave exactly in the same way as it does in python, so it's usually better to use that.
Consider that:
in the function lists you'll always see the returned type on the left of each function;
"void" means that the function returns None;
when overriding some existing method (expecially private and virtual), you always have to return the expected types listed for that function;
function arguments are usually written in the form [const] Type argName=default: you can usually ignore the "const" part (it's a C++ term), while the argName for keyword arguments might be different in PyQt;
some functions could have different names, since they're reserved on python (print, raise, etc); in those cases, an underscore is always appended;
some positional or keyword arguments might be different, or the return type signature might; that's because in C++ you can use a pointer to a variable as an argument, and the function will change that variable using the pointer (this is an oversimplification);
all "properties" are not python properties, and they are only accessible through their parenthesis functions, such as self.width() an self.setWidth();
some methods have different overrides, in some cases python has special cases with different arguments that are not available in C++, and viceversa; also, some methods don't exist at all in one case or the other;
My suggestion is to always use the official documentation, it's only a matter of habit to get used to the C++ references (and you'll see that it is educational too); whenever some doubt raises, check the PyQt documentation to see what's different and use the help command in the python shell.
When using external libraries, you often have to decide whether you use the static or the dynamic version of the library. Typically, you can not exchange them: If the library is build as dynamic library, you can not link statically against it.
Why is this the case?
Example: I am building a C++ program on windows and use a library that provides a small .lib file for the linker and a large .dll file that must be present when running my executable. If the library code in the .dll can be resolved at runtime, why can't it be resolved at compile time and directly put into my executable?
Why is this the case?
Most linkers (AIX linker is a notable exception) discard information in the process of linking.
For example, suppose you have foo.o with foo in it, and bar.o with bar in it. Suppose foo calls bar.
After you link foo.o and bar.o together into a shared library, the linker merges code and data sections, and resolves references. The call from foo to bar becomes CALL $relative_offset. After this operation, you can no longer tell where the boundary between code that came from foo.o and code that came from bar.o was, nor the name that CALL $relative_offset used in foo.o -- the relocation entry has been discarded.
Suppose now you want to link foobar.so with your main.o statically, and suppose main.o already defines its own bar.
If you had libfoobar.a, that would be trivial: the linker would pull foo.o from the archive, would not use bar.o from the archive, and resolve the call from foo.o to bar from main.o.
But it should be clear that none of above is possible with foobar.so -- the call has already been resolved to the other bar, and you can't discard code that came from bar.o because you don't know where that code is.
On AIX it's possible (or at least it used to be possible 10 years ago) to "unlink" a shared library and turn it back into an archive, which could then be linked statically into a different shared library or a main executable.
If foo.o and bar.o are linked into a foobar.so, wouldn't it make sense that the call from foo to bar is always resolved to the one in bar.o?
This is one place where UNIX shared libraries work very differently from Windows DLLs. On UNIX (under common conditions), the call from foo to bar will resolve to the bar in main executable.
This allows one to e.g. implement malloc and free in the main a.out, and have all calls to malloc use that one heap implementation consistently. On Windows you would have to always keep track of "which heap implementation did this memory come from".
The UNIX model is not without disadvantages though, as the shared library is not a self-contained mostly hermetic unit (unlike a Windows DLL).
Why would you want to resolve it to another bar from main.o?
If you don't resolve the call to main.o, you end up with a totally different program, compared to linking against libfoobar.a.
I've got a program written in C++, with some subfolders containing libraries linked in. There's a top level SConscript, which calls SConscript files in the subfolders/libraries.
Inside a library cpp, there is a GTest test function:
TEST(X, just_a_passing_test) {
EXPECT_EQ(true, true);
}
There is main() in the top level program source, which just calls GTests main, and has another GTest test within it:
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
TEST(Dummy, should_pass){
EXPECT_EQ(true, true);
}
Now the issue is that when I run the program, GTest only runs the test in the main.cpp source. Ignoring the test in the library. Now it gets bizarre when I reference an unrelated class in the same library cpp in main.cpp, in a no side-effect kind of way (eg. SomeClass foo;), the test magically appears. I've tried using -O0 and other tricks to force gcc to not optimize out code that isn't called. I've even tried Clang.
I suspect it's something to do with how GTest does test discovery during compilation, but I can't find any info on this issue. I believe it uses static initialization, so maybe there's some weird ordering going on there.
Any help/info is greatly appreciated!
Update: Found a section in the FAQ that sounds like this problem, despite it referring specifically to Visual C++. Which includes a trick/hack to force the compiler to not discard the library if not referenced.
It recommends not putting tests in libraries, but that leaves me wondering how else would you test libraries, without having an executable for every one, making quickly running them a pain and with bloated output.
https://code.google.com/p/googletest/wiki/Primer#Important_note_for_Visual_C++_users
From the scene-setting one gathers that the library whose gtest test case
goes missing is statically linked in the application build. Also that the
GNU toolchain is in use.
The cause of the problem behaviour is straightforward. The test
program contains no references to anything in the library that contains
TEST(X, just_a_passing_test). So the linker doesn't need to link any
object file from that library to link the program. So it doesn't. So the
gtest runtime doesn't find that test in the executable, because it's not there.
It helps to understand that a static library in GNU format is an archive
of object files, adorned with a house-keeping header block and a global symbol table.
The OP discovered that by coding in the program an ad hoc reference to
any public symbol in the problem library, he could "magically" compel its
test case into the program.
No magic. To satisfy the reference to that public symbol, the linker is
now obliged to link an object file from the library - the one that contains
the definition of the symbol. And the OP imparts that the library is made
from a .cpp. So there is only one object file in the library, and it
contains the definition of the test case, too. With that object file in the
linkage, the test case is in program.
The OP twiddled in vain with the compiler options, switching from GCC to clang,
in search of a more respectable way to achieve the same end. The compiler is
irrelevant. GCC or clang, it gets its linking done by the system linker, ld
(unless unusual measures have been taken to replace it).
Is there a more respectable way to get ld to link an object file from a
static library even when the program refers to no symbols in that object file?
There is. Say the problem program is app and the problem static library is
libcool.a
Then the usual GCC commandline that links app resembles this, in the relevant
points:
g++ -o app -L/path/to/the/libcool/archive -lcool
This delegates a commandline to ld, with additional linker options and
libraries that g++ deems to be defaults for the system where it finds itself.
When the linker comes to consider -lcool, it will figure out this is a request
for the archive /path/to/the/libcool/archive/libcool.a. Then it will figure
out whether at this point it has still got any unresolved symbol references in hand
whose definitions are compiled in object files in libcool.a. If there are
any, then it will link those object files into app. If not, then it links
nothing from libcool.a and passes on.
But we know there are symbol definitions in libcool.a that we want to
link, even though app does not refer to them. In that case, we can tell
the linker to link the object files from libcool.a even though they are
not referenced. More precisely, we can tell g++ to tell the linker to do that,
like so:
g++ -o app -L/path/to/the/libcool/archive -Wl,--whole-archive -lcool -Wl,-no-whole-archive
Those -Wl,... options tell g++ to pass the options ... to ld. The --whole-archive
option tells ld to link all object files from subsequent archives, whether they
are referenced or not, until further notice. The -no-whole-archive tells the
ld to stop doing that and resume business as usual.
It may look as if -Wl,-no-whole-archive is redundant, as it's the last thing on the
g++ commandline. But it's not. Remember that g++ appends system default libraries
to the commandline, behind the scenes, before passing it to the ld. You definitely
do not want --whole-archive to be in force when those default libraries are linked.
(The linkage will fail with multiple definition errors).
Apply this solution to the problem case and TEST(X, just_a_passing_test)
will be executed, without the hack of forcing the program to make some no-op
reference into the object file that defines that test.
There's an obvious downside to this solution in the general case. If it happens that the library from
which we want to force linkage of some unreferenced object file contains a
bunch of other unreferenced object files that we really don't need.
--whole-archive links them all of them too, and they're just bloat in the program.
The --whole-archive solution may be more respectable that the no-op reference
hack, but it's not respectable. It doesn't even look respectable.
The real solution here is just to do the reasonable thing. If you want the
linker to link the definition of something in your program, then don't keep that a secret from
the linker. At least declare the thing in each compilation unit where you
expect its definition to be used.
Doing the reasonable thing with gtest test-cases involves understanding that
a gtest macro like TEST(X, just_a_passing_test) expands to a class definition,
in this case:
class X_just_a_passing_test_Test : public ::testing::Test {
public:
X_just_a_passing_test_Test() {}
private:
virtual void TestBody();
static ::testing::TestInfo* const test_info_ __attribute__ ((unused));
X_just_a_passing_test_Test(X_just_a_passing_test_Test const &);
void operator=(X_just_a_passing_test_Test const &);
};
(plus a static initializer for test_info_ and a definition for TestBody()).
Likewise for the TEST_F, TEST_P variants. Consequently, you can deploy these
macros in your code with just the same constraints and expectations that would
apply to class definitions.
In this light, if you have a library libcool defined in cool.h, implemented in cool.cpp
and want gtest unit tests for it, to be executed by a test program tests
that is implemented in tests.cpp, the reasonable thing is:-
Write a header file, cool_test.h
#include "cool.h" in it
#include <gtest/gtest.h> in it.
Then define your libcool test cases in it
#include "cool_test.h" in tests.cpp,
Compile and link tests.cpp with libcool and libgtest
And it's obvious why you wouldn't do what the OP has done. You would not define
classes that are needed by tests.cpp, and not needed by cool.cpp, within cool.cpp
and not in tests.cpp.
The OP was averse to the advice against defining the test cases in the library
because:
how else would you test libraries, without having an executable for every one,
making quickly running them a pain.
As a rule of thumb I would recommend the practice of maintaining a gtest executable
per library to be unit-tested: running them quickly is painless with commonplace automation tools
such a make, and it's far better to get a pass/fail verdict per library than
just a verdict for a bunch of libraries. But if you don't want to do that there's still nothing to the
objection:
// tests.cpp
#include "cool_test.h"
#include "cooler_test.h"
#include "coolest_test.h"
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
Compile and link with libcool, libcooler, libcoolest and libgtest
I am splitting off some of the code in my project into a separate library to be reused in another application. This new library has various functions defined but not implemented, and both my current project and the other application will implement their own versions of these functions.
I implemented these functions in my original project, but they are not called anywhere inside it. They are only called by this new library. As a result, the compiler optimizes them away, and I get linking failures. When I add a dummy call to these functions, the linking failures disappear.
Is there any way to tell GCC to compile these functions even if they're not being called?
I am compiling with gcc 4.2.2 using -O2 on SuSE linux (x86-64_linux_2.6.5_ImageSLES9SP3-3).
You could try __attribute__ ((used)) - see Declaring Attributes of Functions in the gcc manual.
Being a pragmatist, I would just put:
// Hopefully not a name collision :-)
void *xyzzy_plugh_zorkmid_3141592653589_2718281828459[] = {
&functionToForceIn,
&anotherFunction
};
at the file level of one of your source files (or even a brand new source file, something along the lines of forcedCompiledFunctions.c, so that it's obvious what it's for).
Because this is non-static, the compiler won't be able to take a chance that you won't need it elsewhere, so should compile it in.
Your question lacks a few details but I'll give it a shot...
GCC generally removes functions in very few cases:
If they are declared static
In some cases (like when using -fno-implement-inlines) if they are declared inline
Any others I missed
I suggest using 'nm' to see what symbols are actually exported in the resulting .o files to verify this is actually the issue, and then see about any stray 'static' keywords. Not necessarily in this order...
EDIT:
BTW, with the -Wall or -Wunused-function options GCC will warn about unused functions, which will then be prime targets for removal when optimising. Watch out for
warning: ‘xxx’ defined but not used
in your compile logs.
Be careful as the -Wunused-functions doesn't warn of unused functions as stated above. It warns of ununsed STATIC functions.
Here's what the man page for gcc says:
-Wunused-function
Warn whenever a static function is declared but not defined or a non-inline static function is unused. This warning is
enabled by -Wall.
This would have been more appropriate as a comment but I can't comment on answers yet.