No Boot with lvm / initramfs

[ghost]

Hey there,
my kernel as generated by genkernel does not boot and is stuck on booting initramfs.
As far as I understand it LTO does not work with initramfs, or genkernel, but you cannot boot an LVM root file system without initramfs, even with unencrypted /boot.
According to #31 a patch is already applied to remedy this, but apparently it didn't automatically apply the workaround.

How can I check whether my emerge of genkernel honored the workaround?
As I understand it genkernel does not do any LTO stuff and nothing has to be changed there, just with the emerge of genkernel itself...

[InBetweenNames]

I actually have never used genkernel, so I'm not sure. From a theoretical perspective, there's no reason LTO shouldn't work with an initramfs. But there may be some practical considerations that are at fault.

Questions:

• Could you post all log files related to the genkernel process?
• Also, is the kernel itself being built with LTO? I haven't had much luck with using LTOed kernels, mainly due to proprietary drivers.
• When you boot, are you able to see the init log?
• If so, could you obtain a copy, via screenshot or serial port, and post it at the point it halts?

[ghost]

I actually have never used genkernel, so I'm not sure. From a theoretical perspective, there's no reason LTO shouldn't work with an initramfs. But there may be some practical considerations that are at fault.

Questions:

* Could you post all log files related to the genkernel process?

* Also, is the kernel itself being built with LTO?  I haven't had much luck with using LTOed kernels, mainly due to proprietary drivers.

* When you boot, are you able to see the init log?

* If so, could you obtain a copy, via screenshot or serial port, and post it at the point it halts?

I do no specific commands to enable LTO with the kernel's generation. Unless this is somehow done by genkernel when invoking GCC 8.2.0, I guess the kernel generated without LTO, although I don't know exactly.
I'm not sure how to post genkernel's log files. There isn't much info going to the terminal, just the usual: Compiling 4.14.83-gentoo bzImage, Compiling 4.14.83-gentoo modules.
Grub doesn't get far, so no init log Just "Loading x86_64-4.14.83-gentoo... Loading initial ramdisk ..."
Nothing from there on out. Just frozen.

Next compile attempt I'll write stdout to a file to submit, not that there is much useful info there, unless there is some better way.

I created "genkernel.conf" in /etc/portage/package.cflags with "sys-kernel/genkernel FLAGS-=-flto" and reinstalled genkernel. Is this the correct way to disable LTO for a package?
I just finished compiling the kernel and again, crash. I guess I have to disable flto for genkernel's dependencies as well, as noted in that commit, unless the way I described didn't actually disable it.

[InBetweenNames]

In the linked PR (https://github.com/InBetweenNames/gentooLTO/pull/31/files), if you add those exceptions, re-emerge them, and do your genkernel does it work?

[ghost]

In the linked PR (https://github.com/InBetweenNames/gentooLTO/pull/31/files), if you add those exceptions, re-emerge them, and do your genkernel does it work?

Will do just that and report back :]
I presume the way I added the exemptions was correct and will add those to the genkernel.conf file, I hope that is correct as well.

[zhuyifei1999]

Grub doesn't get far, so no init log Just "Loading x86_64-4.14.83-gentoo... Loading initial ramdisk ..."

Do you have an argument quiet in the kernel boot parameters? If so, please remove that and see if you get more messages. Without quiet, the kernel should output lots of message before initramfs's /init is called, and the issue more likely is an issue with the kernel rather than initramfs.

If the issue is indeed within the kernel, the kernel might have failed before the console loads. You can consider using earlyprintk and see if that gives anything (earlyprintk must be compiled in). I use earlyprintk=efi if I have such issues.

If even with earlyprintk nothing is printed, the kernel might be very broken, eg. the kernel self-decompression code failed. In that case, yeah, check the architecture.

For reference, I have gentooLTO + systemd + gummiboot + dracut-generated initramfs. It boots fine.

[ghost]

In the linked PR (https://github.com/InBetweenNames/gentooLTO/pull/31/files), if you add those exceptions, re-emerge them, and do your genkernel does it work?

Again, no luck, hang on boot.
I added those lines to my genkernel.conf inside /etc/portage/package.cflags and did "emerge -a atom" on every package those lines mentioned, that were installed, which I checked qith "qlist -I | grep atom"
genkernellog.log

This time I did the Logfile and it is attached to this comment.
Maybe my flto exeptions are not honored, (how can I check that btw?).

Grub doesn't get far, so no init log Just "Loading x86_64-4.14.83-gentoo... Loading initial ramdisk ..."

Do you have an argument quiet in the kernel boot parameters? If so, please remove that and see if you get more messages. Without quiet, the kernel should output lots of message before initramfs's /init is called, and the issue more likely is an issue with the kernel rather than initramfs.

If the issue is indeed within the kernel, the kernel might have failed before the console loads. You can consider using earlyprintk and see if that gives anything (earlyprintk must be compiled in). I use earlyprintk=efi if I have such issues.

If even with earlyprintk nothing is printed, the kernel might be very broken, eg. the kernel self-decompression code failed. In that case, yeah, check the architecture.

For reference, I have gentooLTO + systemd + gummiboot + dracut-generated initramfs. It boots fine.

The booting proccess never leaves Grub. Those two lines I got from grub. My grub config does not have quiet set as boot param under GRUB_CMDLINE_LINUX, but then again as I understand it it doesn't matter, since Grub is never left. Just those two frozen messages.

I read about dracut before solving LTO and the initramfs issues, but I have always relied on genkernel to do the heavy lifting.

[zhuyifei1999]

The booting proccess never leaves Grub. Those two lines I got from grub.

Grub has already jumped to the kernel entry point. Kernel won't print anything, by default, until the console is loaded; this is why I'm asking if earlyprintk shows anything.

Have you tried booting without initramfs? Do you see any kernel messages if booted that way?

[InBetweenNames]

In addition to what @zhuyifei1999 has suggested, could you post your kernel boot command line from Grub as well?

[ghost]

The booting proccess never leaves Grub. Those two lines I got from grub.

Grub has already jumped to the kernel entry point. Kernel won't print anything, by default, until the console is loaded; this is why I'm asking if earlyprintk shows anything.

Have you tried booting without initramfs? Do you see any kernel messages if booted that way?

I'm pretty sure the kernel is just broken beyond believe. I tried earlyprintk with efi and vga param and checked that it is indeed compiled. In get nothing.
I tried booting via grub command line manually (without initramfs) and get the same freeze, but this time after 5 secs the Capslock LED starts flashing on and off.
I don't have a knowledge with USB debug to pull that off.

In addition to what @zhuyifei1999 has suggested, could you post your kernel boot command line from Grub as well?

As read from the generated grub.cfg:
linux /boot/kernel-genkernel-x86_64-4.14.83-gentoo root=/dev/mapper/nameofmypartition ro earlyprintk=efi dolvm crypt_root=UUID=uuidofmylvmpart root=/dev/mapper/nameofmypart(again?) roottypefs=ext4

[zhuyifei1999]

Does a precompiled known-working (say, from another distro) kernel work? By work I mean you can some kernel messages after the grub Loading initial ramdisk ..., even if the kernel panics in the end.

Also, I'm wondering if grub is at fault here, though I can't imagine a reason why it would.

[ghost]

I had gentoo all well and running before trying the LTO overlay. Also genkernel generated that kernel. I guess I'll try full manual route next. It is definetly related to this overlay or the way my gentoo is set up.
I'm on a librebooted T500 Laptop, so not the standard grub being on an unencrypted boot partition.

Just to be sure, how can I check whether genkernel was emerged without LTO? Just in case the exception didn't get honored...

[zhuyifei1999]

Just to be sure, how can I check whether genkernel was emerged without LTO?

AFAICT genkernel isn't a compiled binary, but lots of bash scripts. There isn't some sort of "LTO for bash scripts"...
sys-kernel/gentoo-sources also shouldn't have LTO inside.

[ghost]

AFAICT genkernel isn't a compiled binary, but lots of bash scripts. There isn't some sort of "LTO for bash scripts"...

That explains the fast emerge time :D

sys-kernel/gentoo-sources also shouldn't have LTO inside.

Ohh, I didn't write an exception to that, so I guess the overlay already provided an exception?
Can I check THAT somehow?

Except watching the emerge live and seeing -flto, I have no I idea how to check whether a package was LTO'ed afterwards or not...

[mortebrume]

Hey.

I just had this problem some weeks ago and it kept getting me crazy for a lot of time.

I have a T440p, it’s provsbly some vendor issues, I don’t think it’s linked with LTO.

You can boot this kernel with ”acpi=off", it’s the only acpi flag that will work, and there will be obviously no acpi, battery management, suspend etc.

The other solution is to switch to the latest mainline kernel (4.19.something), the latest beta gentoo sources (or pf-patchset) doesn’t have the issue anymore.

Hope this will fix it. (And use dracut it’s more up to date even if the doc is uterly shit).

[InBetweenNames]

In addition to what @aphypnise and @zhuyifei1999 said, @SairesArt , to my knowledge your CFLAGS are entirely ignored during genkernel and emerging sys-kernel/gentoo-sources. The only way to affect your kernel's compilation flags is by modifying KCFLAGS, which deliberately doesn't pull in your CFLAGS. I do this on both my desktop and my router to bring in -O3 and friends (without LTO) into my kernels. To my knowledge, work has been done in getting LTO support in linux mainline, but it is still not yet supported -- my own efforts a couple months ago worked only when no proprietary drivers were in use. Even getting the kernel building using LTO was a few hours of work and certainly couldn't happen by accident, even if -flto is in your CFLAGS.

Now when talking about userspace, there's a few checks you can do to see if your package was emerged with LTO. The easiest one is by just looking at the symbols in a static library that was installed during the emerge of the package. You'll see lto symbols all over the place instead of the usual ones. The second one is to check the emerge log and look for the presence of compile lines using -flto. Simply seeing it in your CFLAGS at the start of the emerge isn't necessarily enough. Unfortunately, getting Portage itself to track LTO is a very complicated task for a variety of reasons, so there's no direct way to just look it up.

By the way, are you using USE=override-flagomatic on sys-config/ltoize by any chance?