Make pattern match variables are not expanded - pdf

I'm trying to build some PDFs in a Makefile using Sphinx. The resulting PDF has broken references, so I want to fix those using pdftk.
Goal
So what I want to do for all PDFs I build is this:
# Creates the PDF files.
$(SPHINXBUILD) -b pdf $(ALLSPHINXOPTS) source/pdf/ $(BUILDDIR)/pdf_broken
# Go through all PDFs and fix them.
pdftk $(BUILDDIR)/pdf_broken/thepdf.pdf output $(BUILDDIR)/pdf/thepdf.pdf
Attempt with Make
So to do this with Make I have written this Makefile:
# Build PDF (results in broken references)
$(BUILDDIR)/pdf_broken/%.pdf:
$(SPHINXBUILD) -b pdf $(ALLSPHINXOPTS) source/pdf/ $(BUILDDIR)/pdf_broken
# This fixes the broken pdfs and produces the final result.
$(BUILDDIR)/pdf/%.pdf: $(BUILDDIR)/pdf_broken/%.pdf
mkdir -p $(BUILDDIR)/pdf/
pdftk $^ output $#
pdf: $(BUILDDIR)/pdf/%.pdf
Expected result
I'm using Pattern matching as I understand it from reading the manual:
http://www.tack.ch/gnu/make-3.82/make_91.html
Where $< as I understand it should be the prerequisite expanded so from my above example:
$(BUILDDIR)/pdf_broken/thepdf.pdf
and then $# should be the target:
$(BUILDDIR)/pdf/thepdf.pdf
So my recipe pdftk $^ output $# should run the command:
pdftk $(BUILDDIR)/pdf_broken/thepdf.pdf output $(BUILDDIR)/pdf/thepdf.pdf
Actual result
But this is not what is happening. Instead, this is run:
pdftk build/pdf_broken/%.pdf output build/pdf/%.pdf
Which obviously gives me an error:
Error: Unable to find file.
Error: Failed to open PDF file:
build/pdf_broken/%.pdf
Question
So my question is, what have I missundestood with how the pattern matching works, and how is the correct way to solve this using Make?

You should likely lookup pattern rules. In any case, it looks like you have a single command to generate all the files in the broken directory. This should have its own rule, and should likely output a dummy file to indicate it is complete. Your rule to fix the pdf files should be dependent on this dummy target being created.
It should be something like:
// get a list of expected output files:
PDF_SOURCES:=$(wildcard source/pdf/*)
PDF_OUTS:=$(patsubst $(PDF_SOURCES),source/pdf/%.pdf,$(BUILDDIR)/pdf/%.pdf);
// just for debugging:
$(info PDF_SOURCES = $(PDF_SOURCES))
$(info PDF_OUTS = $(PDF_OUTS))
// default rule
all: $(PDF_OUTS)
#echo done
// rule to build BUILDIR:
$(BUILDDIR)/pdf:
mkdir -p $#
// rule to build all broken files in one go:
// (note: generates a file .dosphynx, which is used to keep track
// of when the rule was run last. This rule will be run if the
// timestamp of any of the sources are newer.
.do_sphynx: $(PDF_SOURCES) | $(BUILDDIR)/pdf
$(SPHINXBUILD) -b pdf $(ALLSPHINXOPTS) source/pdf/ $(BUILDDIR)/pdf_broken
touch $#
// create a dependency of all output files on do_sphynx
$(PDF_OUTS): .do_sphynx
// patern rule to fix pdf files
$(BUILDDIR)/pdf/%.pdf : $(BUILDDIR)/pdf_broken/%.pdf
pdftk $< output $#
I've not tested this, so its possible it may have a syntax error in it..
---------------------- EDIT -------------
Ok, since $(PDF_OUTS) cannot be determined at makefile read time, perhaps you should do:
// get a list of expected output files:
PDF_SOURCES:=$(wildcard source/pdf/*)
all: .do_fix
#echo done
$(BUILDDIR)/pdf:
mkdir -p $#
.do_sphynx: $(PDF_SOURCES) | $(BUILDDIR)/pdf
$(SPHINXBUILD) -b pdf $(ALLSPHINXOPTS) source/pdf/ $(BUILDDIR)/pdf_broken
touch $#
.do_fix: .do_sphynx
#for src in $$(ls source/pdf/*.pdf); do \
trg=$${src/#"source/pdf"/"$(BUILD_DIR)/pdf"}; \
[[ $$src -nt $$trg ]] && \
echo "$$src ==> $$trg" && pdftk $$src output $$trg; \
done
touch $#
One note -- the -nt comparator in the if will return true if $trg does not exist, so it will cover the case where the file is missing, or the target is older than the source. Again not tested, but it should work.

Related

running metabat2 with snakemake but not getting the bin files

I have been trying to run metabat2 with snakemake. I can run it but the output files in metabat2/ are missing. The checkM that works after it does use the data and can work I just cant find the files later. There should be files created with numbers but it is imposible to predict how many files will be created. Is there a way I can specify it to make sure that the files are created in that file?
rule all:
[f"metabat2/" for sample in samples],
[f"checkm/" for sample in samples]
rule metabat2:
input:
"input/consensus.fasta"
output:
directory("metabat2/")
conda:
"envs/metabat2.yaml"
shell:
"metabat2 -i {input} -o {output} -v"
rule checkM:
input:
"metabat2/"
output:
c = "bacteria/CheckM.txt",
d = directory("checkm/")
conda:
"envs/metabat2.yaml"
shell:
"checkm lineage_wf -f {output.c} -t 10 -x fa {input} {output.d}"
the normal code to run metabat2 would be
metabat2 -i path/to/consensus.fasta -o /outputdir/bin -v
this will create in outputdir files with bin.[number].fa
I can't tell what the problem is but I have a couple of suggestions...
[f"metabat2/" for sample in samples]: I doubt this will do what you expect as it will simply create a list with the string metabat2/ repeat len(samples) times. Maybe you want [f"metabat2/{sample}" for sample in samples]? The same for [f"checkm/" for sample in samples]
The samples variable is not used anywhere in the rules following all. I suspect somewhere it should be used and/or you should use something like output: directory("metabat2/{sample}")
Execute snakemake with -p option to see what commands are executed. It may be useful to post the stdout from it.

Extra build/missing object files with header-tracking Makefile

I have written a (GNU make) Makefile designed to perform automatic dependency tracking in header includes. Everything works great except that upon typing make a second time, the entire code base rebuilds. Only typing make the third time and successive times gives the message that nothing is to be done.
SRCDIR := src
INCDIR := inc
ifeq ($(DEBUG),1)
OBJDIR := debug_obj
BINDIR := debug_bin
else
OBJDIR := obj
BINDIR := bin
endif
BINS := prog1 prog2 prog3 prog4
SRCS := $(wildcard $(SRCDIR)/*.cpp)
OBJS := $(patsubst $(SRCDIR)/%,$(OBJDIR)/%,$(SRCS:.cpp=.o))
DEPS := $(OBJS:.o=.d)
CC := g++
COMMON_FLAGS := -Wall -Wextra -Werror -std=c++11 -pedantic
ifeq ($(DEBUG),1)
CXX_FLAGS := $(COMMON_FLAGS) -Og -g
else
CXX_FLAGS := $(COMMON_FLAGS) -O3 -D NDEBUG
endif
all: $(addprefix $(BINDIR)/,$(BINS)) | $(BINDIR)
$(OBJDIR) $(BINDIR):
# mkdir -p $#;
$(BINDIR)/%: $(OBJDIR)/%.o | $(BINDIR)
$(CC) $(CPP_FLAGS) $< -o $#;
$(OBJDIR)/%.o: $(SRCDIR)/%.cpp | $(OBJDIR)
$(CC) $(CPP_FLAGS) -MMD -MP -c $< -o $#;
-include $(DEPS)
.PHONY: all clean
clean:
- rm -f $(OBJS);
- rm -f $(DEPS);
- rm -f $(addprefix $(BINDIR)/,$(BINS));
- rmdir $(OBJDIR) $(BINDIR) 2> /dev/null || true
Clearly some dependency had changed, so I tried running make -n -d | grep 'newer' following the first invocation of make, which shows this:
Prerequisite obj/prog1.o' is newer than targetbin/prog1'.
Prerequisite obj/prog2.o' is newer than targetbin/prog2'.
Prerequisite obj/prog3.o' is newer than targetbin/prog3'.
Prerequisite obj/prog4.o' is newer than targetbin/prog4'.
And ls -la obj/*
Showed the existence of the dependency (*.d) files but not the object (*.o) files. I assume that this is related to how g++ -MMD -MP works, but despite the apparent absence of object files, binaries are present after the first make.
The answer to this question suggests that both are generated at the same time, and man g++ does not dispute this as far as I can tell.
I've read a couple other questions and answers related to automatic dependency tracking, but I don't see this issue arising. Why is this happening? Can you suggest a fix?
Update
A more careful look at the first invocation of make shows this unexpected (to me) line at the end:
rm obj/prog1.o obj/prog2.o obj/prog3.o obj/prog4.o
That answers one question but raises another.
Update
I also found this in the debugging output.
Considering target file `prog1'.
File `prog1' does not exist.
make: *** No rule to make target `prog1'. Stop.
No implicit rule found for `prog1'.
Finished prerequisites of target file `prog1'.
Must remake target `prog1'.
For which I note that prog1 is missing the bin/ prefix. Nothing explains why the first run removes the object files, but the second run leaves them, however. That seems to be at the heart of the issue.
make was treating the object files as intermediates and deleting them accordingly. Adding:
.SECONDARY: $(OBJS)
solved the problem. I do not know why it was doing this the first invocation but not the second invocation. Comments are welcome.
The reason that the .o files are not present is that they're considered intermediate files so make deletes them. However, that shouldn't cause any problems in your build, because as long as make can envision the intermediate file it will realize it doesn't need to be rebuilt if its prerequisites are older than its parents (in this case, as long as prog1 is newer than prog1.cpp for example).
I was not able to reproduce your experience with the second build rebuilding everything. More details will be needed. The output you showed is not interesting because that's just saying that make does NOT need to rebuild the .o file (it's newer than the prerequisite). You need to find the lines in the output that explain why make does need to rebuild the .o file. If you provide that info we may be able to help.
Just a couple of comments on your makefile: first, I don't think it's a good idea to force the mkdir rule to always succeed. If the mkdir fails you WANT your build to fail. Probably you did this so it would not be a problem if the directory already exists, but that's not needed because the mkdir -p invocation will never fail just because the directory exists (but it will fail if the directory can't be created for other reasons such as permissions). Also you can combine those into a single rule with multiple targets:
$(BINDIR) $(OBJDIR):
#mkdir -p $#
Next, you don't need the semicolons in your command lines and in fact, adding them will cause your builds to be slightly slower.
Finally, a small nit, but the correct order of options in the compile line is -c -o $# $<; the source file is not (this is a common misconception) an argument to the -c option. The -c option, like -E, -s, etc. tells the compiler what output to create; in the case of -c it means compile into an object file. Those options do not take arguments. The filename is a separate argument.

How do I create a makefile rule to run astyle?

I'd like to create a makefile rule to run astyle on any writable source files. Currently, I have a rule like the following:
style:
find . -perm -200 -regex ".*[.][CHch]p*" -exec astyle --suffix=none --style=ansi --convert-tabs "{}" \;
This rule basically works but doesn't seem to be the make way of doing things.
Assuming you have a list of source files (or can create them with the shell function), something like:
style : $(SOURCES:.cpp=.astyle-check-stamp)
astyle $(ASTYLEFLAGS) $< && touch $#
would be the make-style. It would re-check each changed source file with astyle and skipped already checked files.

Portable makefile creation of directories

I'm looking to save myself some effort further down the line by making a fairly generic makefile that will put together relatively simple C++ projects for me with minimal modifications required to the makefile.
So far I've got it so it will use all .cpp files in the same directory and specified child directories, place all these within a matching structure in a obj subdir and place the resulting file in another subdir called bin. Pretty much what I want.
However, trying to get it so that the required obj and bin directories is created if they don't exist is providing awkward to get working cross-platform - specifically, I'm just testing with Windows 7 & Ubuntu (can't remember version), and I can't get it to work on both at the same time.
Windows misreads mkdir -p dir and creates a -p directory and obviously the two platforms use \ and / respectively for the path separator - and I get errors when using the wrong one.
Here is a few selected portions of the makefile that are relevant:
# Manually edited directories (in this example with forward slashes)
SRC_DIR = src src/subdir1 src/subdir2
# Automagic object directories + the "fixed" bin directory
OBJ_DIR = obj $(addprefix obj/,$(SRC_DIR))
BIN_DIR = bin
# Example build target
debug: checkdirs $(BIN)
# At actual directory creation
checkdirs: $(BIN_DIR) $(OBJ_DIR)
$(BIN_DIR):
#mkdir $#
$(OBJ_DIR):
#mkdir -p $#
This has been put together by me over the last week or so from things I've been reading (mostly on Stack Overflow), so if it happens to be I'm following some horrible bad practice or anything of that nature please let me know.
Question in a nutshell:
Is there a simple way to get this directory creation to work from a single makefile in a way that provides as much portability as possible?
I don't know autoconf. Every experience I've had with it has been tedious. The problem with zwol's solution is that on Windows mkdir returns an error, unlike mkdir -p on Linux. This could break your make rule. The workaround is to ignore the error with - flag before the command, like this:
-mkdir dir
The problem with this is that make still throws an ugly warning for the user. The workaround for this is to run an "always true" command after the mkdir fails as described here, like this:
mkdir dir || true
The problem with this is that Windows and Linux have different syntax for true.
Anyway, I spent too much time on this. I wanted a make file that worked in both POSIX-like and Windows environments. In the end I came up with the following:
ifeq ($(shell echo "check_quotes"),"check_quotes")
WINDOWS := yes
else
WINDOWS := no
endif
ifeq ($(WINDOWS),yes)
mkdir = mkdir $(subst /,\,$(1)) > nul 2>&1 || (exit 0)
rm = $(wordlist 2,65535,$(foreach FILE,$(subst /,\,$(1)),& del $(FILE) > nul 2>&1)) || (exit 0)
rmdir = rmdir $(subst /,\,$(1)) > nul 2>&1 || (exit 0)
echo = echo $(1)
else
mkdir = mkdir -p $(1)
rm = rm $(1) > /dev/null 2>&1 || true
rmdir = rmdir $(1) > /dev/null 2>&1 || true
echo = echo "$(1)"
endif
The functions/variables are used like so:
rule:
$(call mkdir,dir)
$(call echo, CC $#)
$(call rm,file1 file2)
$(call rmdir,dir1 dir2)
Rationale for the definitions:
mkdir: Fix up the path and ignore any errors.
del: In Windows del doesn't delete any files if one of the files is specified to be in a directory that doesn't exist. For example, if you try to delete a set of files and dir/file.c is in the list, but dir doesn't exist, no files will be deleted. This implementation works around that issue by invoking del once for each file.
rmdir: Fix up the path and ignore any errors.
echo: The output's appearance is preserved and doesn't show the extraneous "" in Windows.
I spent a lot of time on this. Perhaps I would have been better off spending my time learning autoconf.
See also:
OS detecting makefile
Windows mkdir always does what Unix mkdir does with the -p switch on. And you can deal with the backslash problem with $(subst). So, on Windows, you want this:
$(BIN_DIR) $(OBJ_DIR):
mkdir $(subst /,\\,$#)
and on Unix you want this:
$(BIN_DIR) $(OBJ_DIR):
mkdir -p -- $#
Choosing between these is not practical to do within a makefile. This is what Autoconf is for.
As a side note, never, ever use the #command feature in your makefiles. There will come a day when you need to debug your build process on a machine you do not have direct access to, and on that day, you will regret it.
I solved the portability problem by creating a Python script called mkdir.py and calling it from the Makefile. A limitation is that Python must be installed, but this is most likely true for any version of UNIX.
#!/usr/bin/env python
# Cross-platform mkdir command.
import os
import sys
if __name__=='__main__':
if len(sys.argv) != 2:
sys.exit('usage: mkdir.py <directory>')
directory = sys.argv[1]
try:
os.makedirs(directory)
except OSError:
pass

Makefile to archive/link together auto-generated source files

Basically, I have a file 'blah.txt'. That files gets parsed by a 'compiler' and from it will generate N output .c files. I want a makefile that will from that .txt file generate the c files then compile all of them and archive them in a libmystuff.a
I tought of something like this :
all: dogen libmystuff.a
dogen: source.txt
mycompiler $^
libmystuff.a: $(addsuffix .o, $(shell ls *.c))
$(AR) rcs $# $^
.PHONY: dogen
But obviously that doesn't work because the dependencies are evaluated at the start and at that point the *.c just doesn't yield anything yet since they don't exist.
Does anyone see how to accomplish that (without listing all the generated *.c explicitely) ?
Use sentry "makefile" to force make to re-read makefile and substitute correct list at *.c:
include sources-sentry
sources-sentry: source.txt
mycompiler $^
touch $#
libmystuff.a: $(addsuffix .o, $(shell ls *.c))
$(AR) rcs $# $^
include directive is used to include other makefiles (just like C's #include). It has a nice pecularity that if makefile it includes is a target itself, make program first considers it as a target and tries to update. If it is not up-to-date, make invokes the commands needed to update it and then re-reads makefile, substituting all the variables again.
Thus, if source.txt changed since the last time you processed it (the time being recorded as timestamp of sources-sentry file), the sources will be updated and make will be re-invoked, the *.c being substituted to the updates set of c-files.
Pavel Shved is right(*), you must rerun Make. Here is a trick I'm rather proud of. It will handle dependencies on objects that may not yet exist, and won't run unnecessarily.
SOURCES = $(wildcard *.c)
OBJECTS = $(SOURCES:.c=.o)
all: libmystuff.a
ifeq ($(MAKELEVEL),0)
libmystuff.a: source.txt
mycompiler $^
#$(MAKE) -s $#
else
libmystuff.a: $(OBJECTS)
$(AR) rcs $# $^
endif
(*) My old nemesis, we meet again.
EDIT:
If some other make calls this make... I hadn't thought of that. But I think this will solve it:
SOURCES = $(wildcard *.c)
OBJECTS = $(SOURCES:.c=.o)
all: libmystuff.a
libmystuff.a: source.txt
mycompiler $^
#$(MAKE) -s phonyLib
.PHONY: phonyLib
phonyLib: $(OBJECTS)
$(AR) rcs libmystuff.a $^
(Yes, I know, if you feel an urge to build a file called "phonyLib" you won't be able to do it with this makefile, but let's not be perverse.)
If your .c files are only produced by the .txt, then you can let the libmystuff.a depend on the txt, and evaluate the $(shell ls *.c) in the rule body instead.