LTO Problems with sys-devel/llvm-10, and PGO problems with www-client/firefox

[TheGreatMcPain]

This is more of a continuation of #490 regarding an issue with Firefox and LLVM 10.

If sys-devel/llvm-10 is compiled using GCC, and LTO, clang will segfault with this error that contains.

Two passes with the same argument (-amdgpu-argument-reg-usage-info) attempted to be registered!

(I haven't tried GCC 10 yet, but I know this happens with GCC 9)

You can either disable LLVM_TARGETS="AMDGPU", or compile llvm without LTO to fix that segfault.

If LLVM is compiled using clang the segfault will clear up as well, but the problem with Firefox happens if LLVM is compiled with clang, and LTO. Here's the package.env that I use to compile packages with clang, and LTO.

USE="clang"
CC="clang"
CXX="clang++"
CFLAGS="${CFLAGS} -fno-math-errno -fno-trapping-math -flto=thin"
CXXFLAGS="${CXXFLAGS} -fno-math-errno -fno-trapping-math -flto=thin"
LDFLAGS="-Wl,--lto-O2 -Wl,-O2 -Wl,--as-needed -fuse-ld=lld"
AR="llvm-ar"
NM="llvm-nm"
RANLIB="llvm-ranlib"

NOLDADD=1
USE_NONGNU=1

When I attempt to emerge Firefox with the useflags clang, pgo, and lto while LLVM has been compiled with LTO. Firefox will fail with an error that I think relates to PGO: [build.log.tar.gz(https://github.com/InBetweenNames/gentooLTO/files/4639143/build.log.tar.gz)

Now if I rebuild LLVM without -flto=thin and -Wl,--lto-O2 Firefox will successfully build.

I am currently using the default GentooLTO CFLAGS, but with -falign-functions=32, and with pie disabled on sys-devel/gcc.

[leandrolnh]

I've had some problems with LLVM and for me the solution was to also remove -fipa-pta, in addition to what is already removed from ltoworkarounds.conf.
Maybe you can try this.

[TheGreatMcPain]

@leandrolnh I've rebuild LLVM 10 without -fipa-pta using GCC which also cleared that segfault, and it looks like Firefox may successfully build.

Yup, Firefox was able to build.

[InBetweenNames]

@TheGreatMcPain to be clear, removing -fipa-pta from Firefox resolved the issue for you?

[TheGreatMcPain]

@InBetweenNames I removed it from sys-devel/llvm-10.

Although, if I'm compiling LLVM using clang I have to disable LTO.

[eandry77]

I experienced same issue with Steam client. Rebuilding llvm without optimizations fixed it. I didn't try rebuilding it with only removing -fipa-pta, but i'll give it a try.

[telans]

Has anyone enountered/been able to fix profileserver.py hanging when using pgo with firefox?

Same as these:
https://bugs.gentoo.org/703256
https://www.reddit.com/r/Gentoo/comments/gs7nsg/hi_new_gentoo_user_here_and_im_having_some/
https://bugzilla.mozilla.org/show_bug.cgi?id=1570787

[TheGreatMcPain]

Unfortunately I haven't ran into that problem yet.

[barolo]

@telans you can't build firefox with PGO enabled under GCC, atm, there was note somewhere about that. It works with clang.
It's unrelated to lto/gentoo-lto

[telans]

@telans you can't build firefox with PGO enabled under GCC, atm, there was note somewhere about that. It works with clang.
It's unrelated to lto/gentoo-lto

@barolo I haven't actually tried with GCC, just clang & llvm built with -=-fipa-pta. Makes it all the stranger as to why it hangs

[barolo]

@telans I remember it happening to me at some point, don't remember what fixed that. Possibly new ebuild version. Do you have custom cflags enabled? The profiling server is python based maybe it's related?
I'm running pg_overlay firefox builds atm you might try that

[telans]

Makes it all the stranger as to why it hangs

@barolo
Well, turns out you need a display server to build with PGO. Makes sense now that I think about it, I was trying to build it within a systemd-nspawn container previously. Thanks for the suggestions, but it seems to be fixed

I haven't seen pg_overlay's version of it before, thanks

[barolo]

@telans ah, yeah, I've made that mistake when trying to build it via ssh once. PG build has lto enabled for rust parts too, it applies tons of privacy patches though, disabling things like ff accounts/sync. Which might not be for everyone

[StefanSalewski]

BorisCarvajal , your hint works great for me.

My box recently tried to update to clang 10, and I got the error "Two passes with the same argument (-amdgpu-argument-reg-usage-info)" when building some tools like compiler-rt. Your tip fixed it:

$ grep llvm /etc/portage/package.cflags/ltoworkarounds.conf
sys-devel/llvm *FLAGS-="-fipa-pta"

Then rebuild llvm and after that the other packages like compiler-rt.

[jiblime]

I've never been unable to build Firefox with USE="clang lto pgo". All of my LLVM/clang tools have been built with -O2 at most. Unsure why...I think it has to do with rust?

LLVM build flags

sys-devel/llvm-10.0.1::gentoo was built with the following:
USE="debug gold libffi ncurses xar xml -doc -exegesis -libedit -test -z3" ABI_X86="32 (64) (-x32)" LLVM_TARGETS="AMDGPU ARM BPF NVPTX (X86) -AArch64 -ARC -AVR -Hexagon -Lanai -MSP430 -Mips -PowerPC -RISCV -Sparc -SystemZ -WebAssembly -XCore"
CFLAGS="-O2 -march=znver2 -pipe -fno-strict-aliasing -fno-finite-math-only"
CXXFLAGS="-O2 -march=znver2 -pipe -fno-strict-aliasing -fno-finite-math-only"

Firefox compile error

ld.lld: /var/tmp/portage/sys-devel/llvm-10.0.1/work/llvm/lib/Transforms/Instrumentation/ControlHeightReduction.cpp:453: bool shouldApply(llvm::Function&, llvm::ProfileSummaryInfo&): Assertion `PSI.hasProfileSummary() && "Empty PSI?"' failed.
...
clang-10: error: unable to execute command: Aborted (core dumped)
clang-10: error: linker command failed due to signal (use -v to see invocation)

LTO/PGO with GCC 10.2.0 is fine, however.

www-client/firefox-79.0-r1::gentoo was built with the following:
USE="custom-cflags custom-optimization geckodriver gmp-autoupdate hardened hwaccel lto openh264 pgo pulseaudio screenshot system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp -bindist -clang -debug -eme-free -jack -screencast (-selinux) -test -wayland -wifi" ABI_X86="(64)" CPU_FLAGS_X86="avx2" L10N="-ach -af -an -ar -ast -az -be -bg -bn -br -bs -ca -cak -cs -cy -da -de -dsb -el -en-CA -en-GB -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -ia -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mk -mr -ms -my -nb -nl -nn -oc -pa -pl -pt-BR -pt-PT -rm -ro -ru -si -sk -sl -son -sq -sr -sv -ta -te -th -tr -uk -ur -uz -vi -xh -zh-CN -zh-TW"
CFLAGS="-march=native -mtune=native -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fno-semantic-interposition -flimit-function-alignment -falign-functions=32 -malign-data=cacheline -pipe"
CXXFLAGS="-march=native -mtune=native -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fno-semantic-interposition -flimit-function-alignment -falign-functions=32 -malign-data=cacheline -pipe -flifetime-dse=1"

I didn't append that last bit in my CXXFLAGS myself, that's from the package itself. 1

[TheGreatMcPain]

@jiblime Not sure if I can help with your LLVM/CLANG issue, but I'm glad to see GCC working again. Maybe you could send a build.log, so I could get a better idea what your issue is, or you could see if disabling ccache (if you're using it) will make it work.

The only reason I was using clang for firefox is because the pgo useflag wasn't compatible with GCC at the time I created this issue.

[barolo]

@jiblime I'm building firefox with lto, cross-lto, pgo and O3 successfully for a couple of versions now, I'm using firefox from pg_overlay [ llvm 10 ]
This is how upstream [from mozilla] firefox is built btw
also, emerge --info won't show you what really was used during compilation, for many packages it gets stripped inside
about:buildconfig is more reliable

[jiblime]

It's alright as long as it works, it must be my system if everyone else's fine. What's cross-lto?

Always fails here:

ld.lld: /var/tmp/portage/sys-devel/llvm-10.0.1/work/llvm/lib/Transforms/Instrumentation/ControlHeightReduction.cpp:453: bool shouldApply(llvm::Function&, llvm::ProfileSummaryInfo&): Assertion `PSI.hasProfileSummary() && "Empty PSI?"' failed.

65:08.52 Stack dump:
65:08.52 0.     Running pass 'CallGraph Pass Manager' on module '../../../security/nss/lib/mozpkix/mozpkix_mozpkix/pkixbuild.o'.
65:08.52 1.     Running pass 'Reduce control height in the hot paths' on function '@_ZN7mozilla4pkix11TrustDomain13IssuerCheckerC2Ev'
65:08.52  #0 0x00007f946456b88a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/usr/lib/llvm/10/lib64/libLLVM-10.so+0xa4888a)
65:08.52  #1 0x00007f94645697a4 llvm::sys::RunSignalHandlers() (/usr/lib/llvm/10/lib64/libLLVM-10.so+0xa467a4)
65:08.52  #2 0x00007f9464569925 (/usr/lib/llvm/10/lib64/libLLVM-10.so+0xa46925)
65:08.52  #3 0x00007f9463ac6400 __restore_rt (/lib64/libpthread.so.0+0x13400)
65:08.52  #4 0x00007f9463753cc0 raise (/lib64/libc.so.6+0x38cc0)
65:08.52  #5 0x00007f946373d547 abort (/lib64/libc.so.6+0x22547)
65:08.52  #6 0x00007f946373d42f (/lib64/libc.so.6+0x2242f)
65:08.52  #7 0x00007f946374c3b2 (/lib64/libc.so.6+0x313b2)
65:08.52  #8 0x00007f94651c45f0 (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x16a15f0)
65:08.52  #9 0x00007f94651c49bb (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x16a19bb)
65:08.52 #10 0x00007f94646b989e llvm::FPPassManager::runOnFunction(llvm::Function&) (/usr/lib/llvm/10/lib64/libLLVM-10.so+0xb9689e)
65:08.52 #11 0x00007f9465894e29 (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x1d71e29)
65:08.52 #12 0x00007f94646ba422 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/lib/llvm/10/lib64/libLLVM-10.so+0xb97422)
65:08.52 #13 0x00007f9465afb2a1 (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x1fd82a1)
65:08.52 #14 0x00007f9465afd47a llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<std::unique_ptr<llvm::lto::NativeObjectStream, std::default_delete<llvm::lto::NativeObjectStream> > (unsigned int)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long> >, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*> > const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int> >, std::vector<std::pair<llvm::StringRef, llvm::BitcodeModule>, std::allocator<std::pair<llvm::StringRef, llvm::BitcodeModule> > > >&) (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x1fda47a)
65:08.52 #15 0x00007f9465ae7d44 (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x1fc4d44)
65:08.52 #16 0x00007f9465aeb683 (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x1fc8683)
65:08.52 #17 0x00007f946450bf25 (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x9e8f25)
65:08.52 #18 0x00007f94644ef68b (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x9cc68b)
65:08.52 #19 0x00007f9463ac33a7 __pthread_once_slow (/lib64/libpthread.so.0+0x103a7)
65:08.52 #20 0x00007f946450d35f (/usr/lib/llvm/10/lib64/libLLVM-10.so+0x9ea35f)
65:08.52 #21 0x00007f94639a7e40 (/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libstdc++.so.6+0xd2e40)
65:08.52 #22 0x00007f9463abafc4 start_thread (/lib64/libpthread.so.0+0x7fc4)
65:08.52 #23 0x00007f94638156ff clone (/lib64/libc.so.6+0xfa6ff)
65:08.52 clang-10: error: unable to execute command: Aborted (core dumped)
65:08.52 clang-10: error: linker command failed due to signal (use -v to see invocation)
65:08.52 gmake[4]: *** [/var/tmp/portage/www-client/firefox-79.0-r1/work/firefox-79.0/config/rules.mk:606: libxul.so] Error 254
65:08.52 gmake[4]: Leaving directory '/var/tmp/portage/www-client/firefox-79.0-r1/work/firefox-79.0/ff/toolkit/library/build'
65:08.52 gmake[3]: *** [/var/tmp/portage/www-client/firefox-79.0-r1/work/firefox-79.0/config/recurse.mk:72: toolkit/library/build/target] Error 2
65:08.52 gmake[2]: *** [/var/tmp/portage/www-client/firefox-79.0-r1/work/firefox-79.0/config/recurse.mk:34: compile] Error 2
65:08.52 gmake[1]: *** [/var/tmp/portage/www-client/firefox-79.0-r1/work/firefox-79.0/config/rules.mk:390: default] Error 2
65:08.52 gmake: *** [client.mk:125: build] Error 2
65:08.52 139 compiler warnings present.
 * ERROR: www-client/firefox-79.0-r1::gentoo failed (compile phase):
 *   Failed to run './mach build --verbose'

[barolo]

@jiblime cross-lto allows for LTO between rust and C++ if I'm recalling correctly. No idea about that stack though

[Titaniumtown]

Is there a performance improvement with compiling firefox with clang over gcc?

[TheGreatMcPain]

I not sure, but the only reason I was using clang at the time, was because gcc wouldn't compile it at the time I opened this issue.

Also, Mozilla uses clang in their pre-built packages of Firefox.

[barolo]

@Titaniumtown @TheGreatMcPain It depends, there was point where skia rendering library used by FF was significantly slower under gcc than clang due to clanfg specific optimizations, it was fixed some time ago in gcc afaik. On the javascript side gcc usually produced faster results in benches. There's crossLTO for rust parts under clang now too...
Quite few variables to consider, I can't build FF most of the time anyway so I gave up on it for the time being