clang/LLVM project level optimization - optimization

Question

So periodically I try LLVM as I have this theory it should outperform GNU. And then it sadly doesn't.
Part of the theory has to do with its ability to link modules/objects together and THEN optimize, where normally optimization happens on a per file/object basis.
Instead of using a generic one, I see how to build for a specific default target
rm -rf llvm-project
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
git checkout llvmorg-10.0.0
mkdir build
cd build
cmake -DLLVM_ENABLE_PROJECTS='clang;lld' -DCMAKE_CROSSCOMPILING=True -DCMAKE_INSTALL_PREFIX=/opt/llvm/llvm10armv6m -DLLVM_DEFAULT_TARGET_TRIPLE=armv6m-none-eabi -DLLVM_TARGET_ARCH=ARM -DLLVM_TARGETS_TO_BUILD=ARM -G "Unix Makefiles" ../llvm
make -j 8
make -j 4
make
sudo make install
And the test files
test.c
unsigned int one ( void )
{
return(1);
}
unsigned int two ( void );
unsigned int testone ( void )
{
return(one());
}
unsigned int testtwo ( void )
{
return(two());
}
two.c
unsigned int two ( void )
{
return(2);
}
basic run
clang -O2 -fomit-frame-pointer -c test.c -o test.o
llvm-objdump -D test.o
00000000 one:
0: 01 20 movs r0, #1
2: 70 47 bx lr
00000004 testone:
4: 01 20 movs r0, #1
6: 70 47 bx lr
00000008 testtwo:
8: 80 b5 push {r7, lr}
a: ff f7 fe ff bl #-4
e: 80 bd pop {r7, pc}
as one would expect, one() has been inlined into testone().
The desire is to get testwo() inlined as well.
clang -fomit-frame-pointer -c -emit-llvm test.c -o test.bc
clang -fomit-frame-pointer -c -emit-llvm two.c -o two.bc
llvm-link test.bc two.bc -o both.bc
llc both.bc -o both.s
cat both.s
opt -O2 both.bc -o both.opt.bc
llc both.opt.bc -o both.opt.s
cat both.opt.s
gives
testone:
.fnstart
# %bb.0: # %entry
.save {r7, lr}
push {r7, lr}
bl one
pop {r7, pc}
testtwo:
.fnstart
# %bb.0: # %entry
.save {r7, lr}
push {r7, lr}
bl two
pop {r7, pc}
and
testone:
.fnstart
# %bb.0: # %entry
.save {r7, lr}
push {r7, lr}
bl one
pop {r7, pc}
testtwo:
.fnstart
# %bb.0: # %entry
.save {r7, lr}
push {r7, lr}
bl two
pop {r7, pc}
that is worse.
opt -std-link-opts both.bc -o both.opt.bc
same, no better
Now this works
clang -O2 -fomit-frame-pointer -c -emit-llvm test.c -o test.bc
clang -O2 -fomit-frame-pointer -c -emit-llvm two.c -o two.bc
llvm-link test.bc two.bc -o both.bc
opt -O2 both.bc -o both.opt.bc
llc both.opt.bc -o both.opt.s
cat both.opt.s
testone:
.fnstart
# %bb.0: # %entry
movs r0, #1
bx lr
testtwo:
.fnstart
# %bb.0: # %entry
movs r0, #2
bx lr
One would think that not optimizing the parts would give more meat for the optimization of the whole to chew on. Yes? Although this indicates otherwise.
clang -fomit-frame-pointer -c -emit-llvm test.c -o test.bc
clang -fomit-frame-pointer -c -emit-llvm two.c -o two.bc
llvm-link test.bc two.bc -o both.bc
opt -O3 both.bc -o both.opt.bc
llc both.opt.bc -o both.opt.s
cat both.opt.s
testone:
.fnstart
# %bb.0: # %entry
.save {r7, lr}
push {r7, lr}
bl one
movs r0, #1
pop {r7, pc}
testtwo:
.fnstart
# %bb.0: # %entry
.save {r7, lr}
push {r7, lr}
bl two
movs r0, #2
pop {r7, pc}
-O3 doesn't help either, and this output is as pretty bad it calls the function AND inlines it. What is going on there?!
llvm-dis both.opt.bc
cat both.opt.ll
; ModuleID = 'both.opt.bc'
source_filename = "llvm-link"
target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
target triple = "thumbv6m-none-unknown-eabi"
; Function Attrs: noinline nounwind optnone
define dso_local i32 #one() local_unnamed_addr #0 {
entry:
ret i32 1
}
; Function Attrs: noinline nounwind optnone
define dso_local i32 #testone() local_unnamed_addr #0 {
entry:
%call = call i32 #one()
ret i32 1
}
; Function Attrs: noinline nounwind optnone
define dso_local i32 #testtwo() local_unnamed_addr #0 {
entry:
%call = call i32 #two()
ret i32 2
}
; Function Attrs: noinline nounwind optnone
define dso_local i32 #two() local_unnamed_addr #0 {
entry:
ret i32 2
}
How does one undo that?
clang -O2 -fomit-frame-pointer -c -emit-llvm test.c -o test.bc
clang -O2 -fomit-frame-pointer -c -emit-llvm two.c -o two.bc
llvm-link test.bc two.bc -o both.bc
llvm-dis both.bc
cat both.ll
opt -O3 both.bc -o both.opt.bc
llvm-dis both.opt.bc
cat both.opt.ll
gives
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #one() local_unnamed_addr #0 {
entry:
ret i32 1
}
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #testone() local_unnamed_addr #0 {
entry:
ret i32 1
}
; Function Attrs: nounwind
define dso_local i32 #testtwo() local_unnamed_addr #1 {
entry:
%call = tail call i32 #two() #2
ret i32 %call
}
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #two() local_unnamed_addr #0 {
entry:
ret i32 2
}
and
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #one() local_unnamed_addr #0 {
entry:
ret i32 1
}
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #testone() local_unnamed_addr #0 {
entry:
ret i32 1
}
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #testtwo() local_unnamed_addr #0 {
entry:
ret i32 2
}
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #two() local_unnamed_addr #0 {
entry:
ret i32 2
}
So is it correct that you have to apply the optimizations everywhere, at the file/object level in order to get the project level to optimize?
And then there is the question of tail call or leaf, etc optimization, if nothing else testtwo: even in the first case
clang -O2 -fomit-frame-pointer -c test.c -o test.o
could simply branch to two() and not setup a stack frame not do any of that. Or is this a thumb thing? b cant reach?
one:
0: b8 01 00 00 00 movl $1, %eax
5: c3 retq
testone:
10: b8 01 00 00 00 movl $1, %eax
15: c3 retq
testtwo:
20: e9 00 00 00 00 jmp 0 <testtwo+5>
In gnu the linker patches up any branch reaching or mode issues with trampolines
arm-none-eabi-gcc -c -O2 -mcpu=cortex-m0 test.c -o test.o
arm-none-eabi-objdump -D test.o
00000000 <one>:
0: 2001 movs r0, #1
2: 4770 bx lr
00000004 <testone>:
4: 2001 movs r0, #1
6: 4770 bx lr
00000008 <testtwo>:
8: b510 push {r4, lr}
a: f7ff fffe bl 0 <two>
e: bd10 pop {r4, pc}
Okay I stand corrected...
clang --version
clang version 10.0.0 (https://github.com/llvm/llvm-project.git d32170dbd5b0d54436537b6b75beaf44324e0c28)
Target: armv6m-none-unknown-eabi
Thread model: posix
InstalledDir: /opt/llvm/llvm10armv6m/bin
arm-none-eabi-gcc --version
arm-none-eabi-gcc (GCC) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I guess the question is if one wants to do a project level optimization using llvm-link and opt, is optimization of each of the individual items required or is there a command line option I am missing. Not interested in compiler specific attributes that go into the source code itself, want the code infected with neither gcc nor llvm specifics.
After gcc 5.x.x the code got more bloated was hoping that llvm would have a chance but whenever I try this (on projects not just 10 lines of code) gcc ends up with fewer executed instructions, and/or fewer memory accesses, etc, etc. For simple demonstration functions like the ones above, with some exceptions they produce the same/equivalent output.
Is there something, another one of the tools, or command line options, that I am missing in order to get more out of clang/llvm?
Is it that this is too trivial of an example for the tool to shine?
EDIT based on answer
clang -c start.s -o start.o
clang -O2 -flto=thin -fomit-frame-pointer -c test.c
clang -O2 -flto=thin -fomit-frame-pointer -c two.c
ld.lld start.o test.o two.o -o test.elf
llvm-objdump -D test.elf
000110fc testtwo:
110fc: 02 20 movs r0, #2
110fe: 70 47 bx lr
00011100 two:
11100: 02 20 movs r0, #2
11102: 70 47 bx lr
so getting rid of the -emit-llvm and using lto basically gives the desired result.
Looking at the bc disassembly
clang -O2 -flto=thin -fomit-frame-pointer -c test.c
llvm-dis test.o
cat test.o.ll
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #one() local_unnamed_addr #0 {
entry:
ret i32 1
}
; Function Attrs: norecurse nounwind readnone
define dso_local i32 #testone() local_unnamed_addr #0 {
entry:
ret i32 1
}
; Function Attrs: nounwind
define dso_local i32 #testtwo() local_unnamed_addr #1 {
entry:
%call = tail call i32 #two() #3
ret i32 %call
}
enables/adds the tail call. I really dislike using the compiler/shell as a linker (for embedded projects that have their own bootstrap and linker script), llvm-ldd usage wasn't easy to figure out or basically couldn't figure out, but ld.lld also supports the tlo stuff, so that worked out.

Answer 1

The answer is pretty easy actually: one should never want to use llc / opt / llvm-link for performing "end-user" project level kind of optimizations. These are developer-side tools with different defaults, thresholds, etc. Basically, they are just simple command-line frontends to various pieces of LLVM toolbox.
In order to perform the proper link-time-optimization you'd need to use the pipelines that were intended for such task. Basically, compiling everything using "clang -flto" and then linking everything again via "clang -flto" would work. Using LTO-aware linker like lld is a prerequisite as well.
Some further information about ThinLTO could also be found here: https://clang.llvm.org/docs/ThinLTO.html and http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html