Extra build/missing object files with header-tracking Makefile - g++

I have written a (GNU make) Makefile designed to perform automatic dependency tracking in header includes. Everything works great except that upon typing make a second time, the entire code base rebuilds. Only typing make the third time and successive times gives the message that nothing is to be done.
SRCDIR := src
INCDIR := inc
ifeq ($(DEBUG),1)
OBJDIR := debug_obj
BINDIR := debug_bin
else
OBJDIR := obj
BINDIR := bin
endif
BINS := prog1 prog2 prog3 prog4
SRCS := $(wildcard $(SRCDIR)/*.cpp)
OBJS := $(patsubst $(SRCDIR)/%,$(OBJDIR)/%,$(SRCS:.cpp=.o))
DEPS := $(OBJS:.o=.d)
CC := g++
COMMON_FLAGS := -Wall -Wextra -Werror -std=c++11 -pedantic
ifeq ($(DEBUG),1)
CXX_FLAGS := $(COMMON_FLAGS) -Og -g
else
CXX_FLAGS := $(COMMON_FLAGS) -O3 -D NDEBUG
endif
all: $(addprefix $(BINDIR)/,$(BINS)) | $(BINDIR)
$(OBJDIR) $(BINDIR):
# mkdir -p $#;
$(BINDIR)/%: $(OBJDIR)/%.o | $(BINDIR)
$(CC) $(CPP_FLAGS) $< -o $#;
$(OBJDIR)/%.o: $(SRCDIR)/%.cpp | $(OBJDIR)
$(CC) $(CPP_FLAGS) -MMD -MP -c $< -o $#;
-include $(DEPS)
.PHONY: all clean
clean:
- rm -f $(OBJS);
- rm -f $(DEPS);
- rm -f $(addprefix $(BINDIR)/,$(BINS));
- rmdir $(OBJDIR) $(BINDIR) 2> /dev/null || true
Clearly some dependency had changed, so I tried running make -n -d | grep 'newer' following the first invocation of make, which shows this:
Prerequisite obj/prog1.o' is newer than targetbin/prog1'.
Prerequisite obj/prog2.o' is newer than targetbin/prog2'.
Prerequisite obj/prog3.o' is newer than targetbin/prog3'.
Prerequisite obj/prog4.o' is newer than targetbin/prog4'.
And ls -la obj/*
Showed the existence of the dependency (*.d) files but not the object (*.o) files. I assume that this is related to how g++ -MMD -MP works, but despite the apparent absence of object files, binaries are present after the first make.
The answer to this question suggests that both are generated at the same time, and man g++ does not dispute this as far as I can tell.
I've read a couple other questions and answers related to automatic dependency tracking, but I don't see this issue arising. Why is this happening? Can you suggest a fix?
Update
A more careful look at the first invocation of make shows this unexpected (to me) line at the end:
rm obj/prog1.o obj/prog2.o obj/prog3.o obj/prog4.o
That answers one question but raises another.
Update
I also found this in the debugging output.
Considering target file `prog1'.
File `prog1' does not exist.
make: *** No rule to make target `prog1'. Stop.
No implicit rule found for `prog1'.
Finished prerequisites of target file `prog1'.
Must remake target `prog1'.
For which I note that prog1 is missing the bin/ prefix. Nothing explains why the first run removes the object files, but the second run leaves them, however. That seems to be at the heart of the issue.

make was treating the object files as intermediates and deleting them accordingly. Adding:
.SECONDARY: $(OBJS)
solved the problem. I do not know why it was doing this the first invocation but not the second invocation. Comments are welcome.

The reason that the .o files are not present is that they're considered intermediate files so make deletes them. However, that shouldn't cause any problems in your build, because as long as make can envision the intermediate file it will realize it doesn't need to be rebuilt if its prerequisites are older than its parents (in this case, as long as prog1 is newer than prog1.cpp for example).
I was not able to reproduce your experience with the second build rebuilding everything. More details will be needed. The output you showed is not interesting because that's just saying that make does NOT need to rebuild the .o file (it's newer than the prerequisite). You need to find the lines in the output that explain why make does need to rebuild the .o file. If you provide that info we may be able to help.
Just a couple of comments on your makefile: first, I don't think it's a good idea to force the mkdir rule to always succeed. If the mkdir fails you WANT your build to fail. Probably you did this so it would not be a problem if the directory already exists, but that's not needed because the mkdir -p invocation will never fail just because the directory exists (but it will fail if the directory can't be created for other reasons such as permissions). Also you can combine those into a single rule with multiple targets:
$(BINDIR) $(OBJDIR):
#mkdir -p $#
Next, you don't need the semicolons in your command lines and in fact, adding them will cause your builds to be slightly slower.
Finally, a small nit, but the correct order of options in the compile line is -c -o $# $<; the source file is not (this is a common misconception) an argument to the -c option. The -c option, like -E, -s, etc. tells the compiler what output to create; in the case of -c it means compile into an object file. Those options do not take arguments. The filename is a separate argument.

Related

view command before build code in Cmake [duplicate]

I'm trying to debug a compilation problem, but I cannot seem to get GCC (or maybe it is make??) to show me the actual compiler and linker commands it is executing.
Here is the output I am seeing:
CCLD libvirt_parthelper
libvirt_parthelper-parthelper.o: In function `main':
/root/qemu-build/libvirt-0.9.0/src/storage/parthelper.c:102: undefined reference to `ped_device_get'
/root/qemu-build/libvirt-0.9.0/src/storage/parthelper.c:116: undefined reference to `ped_disk_new'
/root/qemu-build/libvirt-0.9.0/src/storage/parthelper.c:122: undefined reference to `ped_disk_next_partition'
/root/qemu-build/libvirt-0.9.0/src/storage/parthelper.c:172: undefined reference to `ped_disk_next_partition'
/root/qemu-build/libvirt-0.9.0/src/storage/parthelper.c:172: undefined reference to `ped_disk_next_partition'
collect2: ld returned 1 exit status
make[3]: *** [libvirt_parthelper] Error 1
What I want to see should be similar to this:
$ make
gcc -Wall -c -o main.o main.c
gcc -Wall -c -o hello_fn.o hello_fn.c
gcc main.o hello_fn.o -o main
Notice how this example has the complete gcc command displayed. The above example merely shows things like "CCLD libvirt_parthelper". I'm not sure how to control this behavior.
To invoke a dry run:
make -n
This will show what make is attempting to do.
Build system independent method
make SHELL='sh -x'
is another option. Sample Makefile:
a:
#echo a
Output:
+ echo a
a
This sets the special SHELL variable for make, and -x tells sh to print the expanded line before executing it.
One advantage over -n is that is actually runs the commands. I have found that for some projects (e.g. Linux kernel) that -n may stop running much earlier than usual probably because of dependency problems.
One downside of this method is that you have to ensure that the shell that will be used is sh, which is the default one used by Make as they are POSIX, but could be changed with the SHELL make variable.
Doing sh -v would be cool as well, but Dash 0.5.7 (Ubuntu 14.04 sh) ignores for -c commands (which seems to be how make uses it) so it doesn't do anything.
make -p will also interest you, which prints the values of set variables.
CMake generated Makefiles always support VERBOSE=1
As in:
mkdir build
cd build
cmake ..
make VERBOSE=1
Dedicated question at: Using CMake with GNU Make: How can I see the exact commands?
Library makefiles, which are generated by autotools (the ./configure you have to issue) often have a verbose option, so basically, using make VERBOSE=1 or make V=1 should give you the full commands.
But this depends on how the makefile was generated.
The -d option might help, but it will give you an extremely long output.
Since GNU Make version 4.0, the --trace argument is a nice way to tell what and why a makefile do, outputing lines like:
makefile:8: target 'foo.o' does not exist
or
makefile:12: update target 'foo' due to: bar
Use make V=1
Other suggestions here:
make VERBOSE=1 - did not work at least from my trials.
make -n - displays only logical operation, not command line being executed. E.g. CC source.cpp
make --debug=j - works as well, but might also enable multi threaded building, causing extra output.
I like to use:
make --debug=j
https://linux.die.net/man/1/make
--debug[=FLAGS]
Print debugging information in addition to normal processing. If the FLAGS are omitted, then the behavior is the same as if -d was specified. FLAGS may be a for all debugging output (same as using -d), b for basic debugging, v for more verbose basic debugging, i for showing implicit rules, j for details on invocation of commands, and m for debugging while remaking makefiles.
Depending on your automake version, you can also use this:
make AM_DEFAULT_VERBOSITY=1
Reference: AM_DEFAULT_VERBOSITY
Note: I added this answer since V=1 did not work for me.
In case you want to see all commands (including the compiled ones) of the default target run:
make --always-make --dry-run
make -Bn
show commands executed the next run of make:
make --dry-run
make -n
You are free to choose a target other than the default in this example.

wxWidgets library build garbled file name in linker

On attempting to build wxWidgets 3.1.0 libraries, I am getting this error
ar: gcc_mswu\moolib_fontmap.o: No such file or directory
A file does exists with a slightly different spelling
Directory of C:\Users\James\code\wxwidgets-3.1.0\build\msw\gcc_mswu
2017-01-05 02:01 PM 98,886 monolib_fontmap.o
It looks like a typo in the makefile, or like a letter is missed reading the makefile. Is that possible?
Except: if I redirect the console output to a file and open in an editor , the correct spelling shows up:
So the correct command is going to the linker, but the linker is looking for a garbled filename!
Here is the complete recipe for what I am doing:
Download wxWidgets source code from https://github.com/wxWidgets/wxWidgets/releases/download/v3.1.0/wxWidgets-3.1.0.7z
Unpack to a folder. On my system, I use C:\Users\James\code\wxwidgets-3.1.0
Open a command window.
cd to the code::blocks mingw folder. On my system this is C:\Program Files (x86)\CodeBlocks16\MinGW
Type mingwvars.bat
cd to wxwidgets folder. On my system C:\Users\James\code\wxwidgets-3.1.0
cd to ./build/msw
Type mingw32-make SHELL=CMD.exe -j4 -f makefile.gcc BUILD=release UNICODE=1 SHARED=0 MONOLITHIC=1
This seems to be the same problem as the one mentioned in the wiki. Apparently there is a bug in mingw32-make with very long command lines which makes it (sometimes?) eat characters in them...
Yes. Applying the recipe in the link you posted fixed the problem. Here are the details:
modify makefile.gcc as following:
From:
ifeq ($(MONOLITHIC),1)
ifeq ($(SHARED),0)
$(LIBDIRNAME)\libwx$(PORTNAME)$(WXUNIVNAME)$(WX_RELEASE_NODOT)$(WXUNICODEFLAG)$(WXDEBUGFLAG)$(WX_LIB_FLAVOUR).a: $(MONOLIB_OBJECTS)
if exist $# del $#
ar rcu $# $(MONOLIB_OBJECTS)
ranlib $#
endif
endif
Replace second occurence of $(MONOLIB_OBJECTS) with gcc_mswu\monolib*.o:
ifeq ($(MONOLITHIC),1)
ifeq ($(SHARED),0)
$(LIBDIRNAME)\libwx$(PORTNAME)$(WXUNIVNAME)$(WX_RELEASE_NODOT)$(WXUNICODEFLAG)$(WXDEBUGFLAG)$(WX_LIB_FLAVOUR).a: $(MONOLIB_OBJECTS)
if exist $# del $#
ar rcu $# gcc_mswu\\monolib*.o
ranlib $#
endif
endif

syntax error near unexpected token `AX_VALGRIND_CHECK'

I am trying to integrate valgrind into my unit test framework by using the following m4 macro described at https://www.gnu.org/software/autoconf-archive/ax_valgrind_check.html. In my configure.ac I have
AC_CONFIG_MACRO_DIR([m4])
...
AX_VALGRIND_DFLT()
AX_VALGRIND_CHECK
I have placed the .m4 script provided, in both ./m4 and in /usr/share/aclocal. To generate the configure script etc, I run the following:
aclocal && autoconf && autoreconf --no-recursive --install && \
autoheader && libtoolize --force && automake --force-missing \
--add-missing
However when I go an run ./configure I get the following error
./configure: line 12914: syntax error near unexpected token `AX_VALGRIND_CHECK'
./configure: line 12914: `AX_VALGRIND_CHECK'
What do I need to do to get my configure script to work with the macros provided by the .m4 script above. I am not sure what other information to provide.
Below is my configure.ac. I will try to find at which point things break using this configure.ac vs the one generated by autoreconf -i as posted by #Kusalananda.
AC_INIT([binary_balanced], [0.1], [mehoggan#gmail.com])
AM_INIT_AUTOMAKE([-Wall -Werror foreign subdir-objects])
AC_CONFIG_SRCDIR([./src/])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_MACRO_DIR([m4])
AC_PROG_CC
AM_PROG_AR
AM_PATH_CHECK
LT_INIT
# Checks for programs.
AC_PROG_CC
# Checks for libraries.
AX_VALGRIND_DFLT()
AX_VALGRIND_CHECK
# Checks for header files.
# Checks for typedefs, structures, and compiler characteristics.
# Checks for library functions.
AC_CONFIG_FILES([Makefile
src/Makefile
tests/Makefile])
AC_OUTPUT
I can not re-create your problem.
I also very seldom run anything other than autoreconf -i. This will re-run the other autotools as needed.
I put the ax_valgrind_check.m4 into a ./m4 directory and created a stub configure.ac:
AC_PREREQ([2.69])
AC_INIT([test],[0.0.0-dev])
AM_INIT_AUTOMAKE([foreign])
AC_CONFIG_MACRO_DIR([m4])
AX_VALGRIND_DFLT()
AX_VALGRIND_CHECK
Running autoreconf -i creates a configure script that does the following:
$ ./configure
checking for a BSD-compatible install... /Users/kk/sw/bin/ginstall -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /Users/kk/sw/bin/gmkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for valgrind... no
So the macros are picked up (which they weren't in your case).
So, run autoreconf -i to see if that sorts things out for you.
If you can't get this to work, try installing the autoconf-archive package for whatever Unix you're on. It will also contain this macro.

make does not rebuild target on header file modification

I have a makefile of this kind:
program: \
a/a.o \
b/b.o
$(CXX) $(CXXFLAGS) -o program \
a/a.o \
b/b.o
a.o: \
a/a.cpp \
a/a.h
$(CXX) $(CXXFLAGS) -c a/a.cpp
b.o: \
b/b.cpp \
b/b.h
$(CXX) $(CXXFLAGS) -c b/b.cpp
So in the directory of the makefile I have two subdirectories a and b
that contain respectively a.h, a.cpp and b.h, b.cpp.
The problem is that if I modify a .cpp file, issuing a make rebuilds the target program
but if I modify an .h file make do not rebuilds anything but says
make: `program' is up to date.
I can't understand why, because the .h files are in the prerequisites line
along with the .cpp files.
Interestingly, if I issue a make on an object file target like
$ make a.o
instead, the modifications to a/a.h
are detected and the target a/a.o is rebuild.
Where is the problem?
The subdirectories that you added to the question later are causing the problem indeed. The target program depends on a/a.o and b/b.o, but there are no explicit rules to make those to .o files -- only the targets a.o and b.o are present but those are not in the subdirectories.
Therefore, make will look for implicit rules to build a/a.o and b/b.o. That rule does exist, you will see it being found when you run make -d. That implicit rule depends on a/file_a.cpp only, not on a/file_a.h. Therefore, changing a/file_a.cpp will make a/a.o out of date according to that implicit rule, whereas a/file_a.h will not.
For your reference, the make User's Manual has a section Catalogue of Implicit Rules. That also explains that you can use the argument --no-builtin-rules to avoid that implicit behavior. If you use that, you will see that make can not find any rules to make a/a.o and b/b.o.
Finally, running make a.o will run the recipe for the target a.o as defined in your makefile. That target does have a/a.h as its prerequisite so any change to that file will result in a recompile. But essentially, that has nothing to do with the target program, which has different prerequisites.

Makefile to archive/link together auto-generated source files

Basically, I have a file 'blah.txt'. That files gets parsed by a 'compiler' and from it will generate N output .c files. I want a makefile that will from that .txt file generate the c files then compile all of them and archive them in a libmystuff.a
I tought of something like this :
all: dogen libmystuff.a
dogen: source.txt
mycompiler $^
libmystuff.a: $(addsuffix .o, $(shell ls *.c))
$(AR) rcs $# $^
.PHONY: dogen
But obviously that doesn't work because the dependencies are evaluated at the start and at that point the *.c just doesn't yield anything yet since they don't exist.
Does anyone see how to accomplish that (without listing all the generated *.c explicitely) ?
Use sentry "makefile" to force make to re-read makefile and substitute correct list at *.c:
include sources-sentry
sources-sentry: source.txt
mycompiler $^
touch $#
libmystuff.a: $(addsuffix .o, $(shell ls *.c))
$(AR) rcs $# $^
include directive is used to include other makefiles (just like C's #include). It has a nice pecularity that if makefile it includes is a target itself, make program first considers it as a target and tries to update. If it is not up-to-date, make invokes the commands needed to update it and then re-reads makefile, substituting all the variables again.
Thus, if source.txt changed since the last time you processed it (the time being recorded as timestamp of sources-sentry file), the sources will be updated and make will be re-invoked, the *.c being substituted to the updates set of c-files.
Pavel Shved is right(*), you must rerun Make. Here is a trick I'm rather proud of. It will handle dependencies on objects that may not yet exist, and won't run unnecessarily.
SOURCES = $(wildcard *.c)
OBJECTS = $(SOURCES:.c=.o)
all: libmystuff.a
ifeq ($(MAKELEVEL),0)
libmystuff.a: source.txt
mycompiler $^
#$(MAKE) -s $#
else
libmystuff.a: $(OBJECTS)
$(AR) rcs $# $^
endif
(*) My old nemesis, we meet again.
EDIT:
If some other make calls this make... I hadn't thought of that. But I think this will solve it:
SOURCES = $(wildcard *.c)
OBJECTS = $(SOURCES:.c=.o)
all: libmystuff.a
libmystuff.a: source.txt
mycompiler $^
#$(MAKE) -s phonyLib
.PHONY: phonyLib
phonyLib: $(OBJECTS)
$(AR) rcs libmystuff.a $^
(Yes, I know, if you feel an urge to build a file called "phonyLib" you won't be able to do it with this makefile, but let's not be perverse.)
If your .c files are only produced by the .txt, then you can let the libmystuff.a depend on the txt, and evaluate the $(shell ls *.c) in the rule body instead.