New GentooLTO layout

[InBetweenNames]

Hello everyone,

Just wanted to run an idea by you. Currently, we symlink make.conf.lto and make.conf.lto.defines to /etc/portage. This makes it hard to extend with different configurations. What I'd like to do instead is only install things to package.cflags and bashrc.d and leave the rest of /etc/portage untouched.

One idea I had was to have users source directly from the overlay. For example, instead of:

source make.conf.lto

One could do:

source /path/to/lto-overlay/sys-config/ltoize/files/configs/make.conf.lto

It's longer, but this way we could more easily introduce more default configurations, such as:

• GentooLTO Experimental - LTO, -O3, Graphite, -fipa-pta, -fno-plt, -fno-common...
• GentooLTO Testing - LTO, -O3
• GentooLTO Stable - LTO
  etc

The layout I'm proposing is:

• lto-overlay/sys-config/ltoize/files/configs/*.lto for default configurations
• lto-overlay/sys-config/ltoize/files/configs/make.conf.lto.defines for definitions

For a migration, we could provide a transition period where the symlinks are still installed, then afterwards we could remove them. An eselect news entry could be made to warn users of the change.

I'd appreciate your thoughts on the matter!

Thanks,
Shane

[javashin]

Whats The Flags Recommended For Speed/Stability Configuration ? not yet . Can Someone Recommend A Standard Group Of *FLAGS That We Can Make Default For Future Reference Is a Lot To Process Reading All The Issues Looking For The Right Combination . Not Just For Me For Anyone new on The Overlay There Are A lot Choose ending With Smash Potato Gentoo Thanks.

[gcs-github]

Speaking only for myself:

In order not to get surprises on my systems in case changes are made in the make.conf configuration, I don't source it at all, and instead choose to track the changes in this repository as well as copy-pasting what I find of interest.

Therefore, I wouldn't be affected by such a change.

For context, I find this combination of no sourcing + tracking changes to this repo to be a good way to stay up-to-date personally with what's going on here, plus learn as soon as new ideas are introduced when I don't already know about them.

[pchome]

IMHO LTO, -O3, Graphite, -fipa-pta is pretty stable. Me and, I believe, many users have whole system built using this flags(with only few exceptions for failing packages). So those flags received good testing.

All other flags, should be Safe (Testing), Unsafe (Experimental/Optional) or Optional.
E.g. if -fno-plt will introduce regressions for small applications, but can improve big ones -- it should be enabled only for some individual packages.

Also consider to expand this Stable/Testing/Experimental logic on ltoworkarounds.conf.
We have a lot of packages which can compile with LTO in some cases, but disabled for all users.

Make them, let's say:

• Stable - current
• Testing - if LTO confirmed, then remove from list or add conditional fix
• Experimental - nothing, but runtime "surprises" listed in BEGIN: LTO not recommended section
• nothing

I know, maintenance hell, but still.

[javashin]

me i have a whole system compiled with LTO, -O3, Graphite, -fipa-pta but after add IPA some startup time in apps and boot time increased . ATM im Compiling Whole system with LTO, -O3, Graphite, -fipa-pta -fno-plt and FASTMATH with no pie and no ssb , with openrc profile 17.1

[javashin]

i dont know anything about -fno-common

[nivedita76]

Speaking only for myself:

In order not to get surprises on my systems in case changes are made in the make.conf configuration, I don't source it at all, and instead choose to track the changes in this repository as well as copy-pasting what I find of interest.

Therefore, I wouldn't be affected by such a change.

For context, I find this combination of no sourcing + tracking changes to this repo to be a good way to stay up-to-date personally with what's going on here, plus learn as soon as new ideas are introduced when I don't already know about them.

Yup, I'm only sourcing the defines, and the workarounds, not make.conf.lto itself.

[InBetweenNames]

@pchome to add to this, we could separate ltoworkarounds.conf into different configs that may be sourced depending on whether the user is on GentooLTO "Safe" or other things. The section you mention, LTO not recommended, could be moved to a completely different file in that setup.

I would like to have a configuration for a -fno-trapping-math -fno-math-errno and also a -Ofast setup (which is even more aggressive) as well. Some users here are already doing that, and it would be nice to have some official support for that.

[funghetto]

I'm waiting for this to rebuild my systems with gcc 9.1.

[InBetweenNames]

@funghetto you don't need to wait -- it's only a new layout for the repo, it won't change how your system functions. The codegen will be the same at default settings. You can go ahead and rebuild right now if you want and migrate over later without any penalty.

[pchome]

@InBetweenNames

The section you mention, LTO not recommended, could be moved to a completely different file in that setup.

BTW, maybe some of non-*FLAGS-=-flto* workarounds should be proposed to upstream gentoo?

EDIT: like "silent" ones, from Deliberate -O3 workarounds and -fipa-pta workarounds

[funghetto]

@InBetweenNames What would be the full list of flags of the experimental branch?

...
GentooLTO Experimental - LTO, -O3, Graphite, -fipa-pta, -fno-plt, -fno-common...
...

Edit: Will there be a Ofast branch too?

You can go ahead and rebuild right now if you want and migrate over later without any penalty.

Yes but I would need to rebuild world again to use the new flags.

[InBetweenNames]

@pchome Agreed -- it's time to get some of those fixed where they should be

@funghetto I won't be revising the default build that much in the refactor -- I'll probably push those updates out before I end up doing the actual refactor. The refactor is just for maintainability and introducing some different configs, including -Ofast. It's nothing that you can't already do in make.conf.

Over the next month, I'd like to look at adding -fipa-pta and -fdevirtualize-at-ltrans to the defaults because they don't change the semantics of the code they are compiled with. I'm already testing both on my system. -fno-pta is another candidate, but I'm hesitant to make it default because I'm not completely convinced it's necessarily better than lazy binding. It makes a lot of sense on hardened systems though, with Full RELRO, etc. I'm testing it out for now. If I can get prelink working safely, then it'll be a clear win over lazy binding.

I would like to have an Ofast config, or something like it -- but I can't condone using that because it does break the C and C++ standards. A restricted version like -fno-math-errno and -fno-trapping-math is a lot safer and does get a lot of the way there. Clear Linux uses those by default, for example.

I may introduce -fno-common after this as well, with the sets of workaround contributed here already. But this one requires many workarounds in different packages and I'm going to have to spend a lot of time on it probably.

I've also been testing out -mtls-dialect=gnu2 for a while without issues. I may make that default around -fno-common as well.

Last but not least, I'm strongly considering enforcing usage of the Gold linker as it seems to resolve a number of problems that BFD has, and only overriding for BFD using specific packages.

[nivedita76]

What packages work with gold but not bfd?

[nivedita76]

I’m currently using bfd for everything (unless the ebuild specifically overrides it like Firefox)

[nivedita76]

Re no-plt I think it makes sense to turn it on if hardened is turned on. In that case you’re anyway given up on lazy binding so there’s only upside.

[InBetweenNames]

@nivedita76 It was only a couple, here's the list:

• dev-libs/nss
• app-office/texstudio
• sys-devel/metashell
• sys-fs/cryfs

I think it may make sense to have a "no-PLT" configuration intended for use with prelink, with the glibc exception. But opt-in, not as default.

[elsandosgrande]

While you would like to avoid any symbolic links being placed inside of the /etc/portage directory, I do believe that a compromise might be beneficial. What I have in mind is that you create a symbolic link to the most basic and well tested configuration, which contains further instructions for our more intrepid users with regards to other configurations that do not have symbolic links, amongst other instructions and explanations.