• [gentoo-dev] [PATCH 1/7] linux-mod-r1.eclass: add USE=initramfs

    From Andrew Ammerlaan@21:1/5 to All on Tue May 14 14:00:01 2024
    Adds a new variable that adds the "initramfs" flag when set. This new
    flag controls whether or not the modules that were built should be
    included in the initramfs. If the modules should be included, then we
    also rebuild the initramfs/uki in post_install using installkernel.

    Bug: https://bugs.gentoo.org/923025
    Bug: https://bugs.gentoo.org/928271
    Signed-off-by: Andrew Ammerlaan <andrewammerlaan@gentoo.org>
    ---
    eclass/linux-mod-r1.eclass | 54 ++++++++++++++++++++++++++++++++++++++
    1 file changed, 54 insertions(+)

    diff --git a/eclass/linux-mod-r1.eclass b/eclass/linux-mod-r1.eclass
    index 4b267aae4db1..d9651f962604 100644
    --- a/eclass/linux-mod-r1.eclass
    +++ b/eclass/linux-mod-r1.eclass
    @@ -131,6 +131,15 @@ IDEPEND="
    sys-apps/kmod[tools]
    "

    +if [[ ${MODULES_INITRAMFS_IUSE} ]]; then
    + IUSE+=" ${MODULES_INITRAMFS_IUSE}"
    + IDEPEND+="
    + ${MODULES_INITRAMFS_IUSE#+}? (
    + sys-kernel/installkernel
    + )
    + "
    +fi
    +
    if [[ -n ${MODULES_OPTIONAL_IUSE} ]]; then
    : "${MODULES_OPTIONAL_IUSE#+}? ( | )"
    RDEPEND=${_/|/${RDEPEND}} DEPEND=${_/|/${DEPEND}} \
    @@ -179,6 +188,22 @@ fi
    #
    # May want to look at KERNEL_CHOST before considering this.

    +# @ECLASS_VARIABLE: MODULES_INITRAMFS_IUSE
    +# @DEFAULT_UNSET
    +# @PRE_INHERIT
    +# @DESCRIPTION:
    +# If set, adds the specified USE flag. When this flag is enabled the
    +# installed kernel modules are registered for inclusion in the dracut
    +# initramfs.
  • From Andrew Ammerlaan@21:1/5 to All on Tue May 14 14:00:01 2024
    Signed-off-by: Andrew Ammerlaan <andrewammerlaan@gentoo.org>
    ---
    profiles/use.desc | 1 +
    1 file changed, 1 insertion(+)

    diff --git a/profiles/use.desc b/profiles/use.desc
    index d00e21ff01e2..0b96f6add385 100644
    --- a/profiles/use.desc
    +++ b/profiles/use.desc
    @@ -142,6 +142,7 @@ imagemagick - Enable optional support for the ImageMagick or GraphicsMagick imag
    imap - Add support for IMAP (Internet Mail Application Protocol)
    imlib - Add support for imlib, an image loading and rendering library
    infiniband - Enable Infiniband RDMA transport support
    +initramfs - Include kernel modules in the initramfs, and re-install the kernel (only effective for distribution kernels)
    inotify - Enable inotify filesystem monitoring support
    introspection - Add support for GObject based introspection
    iodbc - Add support for iODBC library
    --
    2.45.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrew Ammerlaan@21:1/5 to All on Tue May 14 14:10:01 2024
    The /boot/amd-ucode is useless when we use a dist-kernel since dracut is responsible for generating the initramfs, and it (by default) includes the microcode.

    Instead add USE=dist-kernel and enable USE=initramfs by default. The
    initramfs flag toggles the early_microcode setting of dracut, and if both flags are enabled we trigger initramfs re-installation. This ensures that when we update this package, we also get the latest microcode in our dist-kernel initramfs or UKI.

    Signed-off-by: Andrew Ammerlaan <andrewammerlaan@gentoo.org>
    ---
    .../linux-firmware-20240220-r2.ebuild | 32 +++++++++++++++----
    .../linux-firmware-20240312.ebuild | 32 +++++++++++++++----
    .../linux-firmware-20240410.ebuild | 32 +++++++++++++++----
    .../linux-firmware-99999999.ebuild | 32 +++++++++++++++----
    4 files changed, 100 insertions(+), 28 deletions(-)

    diff --git a/sys-kernel/linux-firmware/linux-firmware-20240220-r2.ebuild b/sys-kernel/linux-firmware/linux-firmware-20240220-r2.ebuild
    index 8ea7565f00ce..0ecca9dae446 100644
    --- a/sys-kernel/linux-firmware/linux-firmware-20240220-r2.ebuild
    +++ b/sys-kernel/linux-firmware/linux-firmware-20240220-r2.ebuild
    @@ -2,7 +2,7 @@
    # Distributed under the terms of the GNU General Public License v2

    EAPI=8
    -inherit linux-info mount-boot savedconfig multiprocessing
    +inherit dist-kernel-utils linux-info mount-boot savedconfig multiprocessing

    # In case this is a real snapshot, fill in commit below.
    # For normal, tagged releases, leave blank
    @@ -29,7 +29,7 @@ L
  • From Andrew Ammerlaan@21:1/5 to All on Tue May 14 14:10:01 2024
    Closes: https://bugs.gentoo.org/931213
    Bug: https://bugs.gentoo.org/926063
    Signed-off-by: Andrew Ammerlaan <andrewammerlaan@gentoo.org>
    ---
    eclass/linux-mod-r1.eclass | 5 +++++
    1 file changed, 5 insertions(+)

    diff --git a/eclass/linux-mod-r1.eclass b/eclass/linux-mod-r1.eclass
    index d9651f962604..6060074d8734 100644
    --- a/eclass/linux-mod-r1.eclass
    +++ b/eclass/linux-mod-r1.eclass
    @@ -661,6 +661,11 @@ _modules_check_migration() {
    # Handles linux-info bits to provide usable sources, KV_ variables,
    # and CONFIG_CHECK use.
    _modules_prepare_kernel() {
    + # The modules we build are specific to each kernel version, we don't
    + # want to reset the environment to use the user selected kernel version.
    + # Bug 931213, 926063
    + SKIP_KERNEL_BINPKG_ENV_RESET=1
    +
    get_version

    # linux-info allows skipping checks if SKIP_KERNEL_CHECK is set and
    --
    2.45.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrew Ammerlaan@21:1/5 to All on Tue May 14 14:10:01 2024
    The /boot/intel-uc.img is useless when we use a dist-kernel since dracut is responsible for generating the initramfs, and it (by default) includes the microcode.

    Instead add USE=dist-kernel and enable USE=initramfs by default. The
    initramfs flag toggles the early_microcode setting of dracut, and if both flags are enabled we trigger initramfs re-installation. This ensures that when we update this package, we also get the latest microcode in our dist-kernel initramfs or UKI.

    Signed-off-by: Andrew Ammerlaan <andrewammerlaan@gentoo.org>
    ---
    .../intel-microcode-20221108_p20221102.ebuild | 43 ++++++++++++++-----
    .../intel-microcode-20230214_p20230212.ebuild | 41 ++++++++++++++----
    .../intel-microcode-20230512_p20230512.ebuild | 41 ++++++++++++++----
    .../intel-microcode-20230613_p20230520.ebuild | 40 +++++++++++++----
    .../intel-microcode-20230808_p20230804.ebuild | 40 +++++++++++++----
    .../intel-microcode-20230808_p20231007.ebuild | 40 +++++++++++++----
    .../intel-microcode-20231114_p20231114.ebuild | 40 +++++++++++++----
    .../intel-microcode-20240312_p20240312.ebuild | 38 +++++++++++++---
    8 files changed, 256 insertions(+), 67 deletions(-)

    diff --git a/sys-firmware/intel-microcode/intel-microcode-20221108_p20221102.ebuild b/sys-firmware/intel-microcode/intel-microcode-20221108_p20221102.ebuild
    index 006f4811618a..a8cf0d39ff33 100644
    --- a/sys-firmware/intel-microcode/intel-microcode-20221108_p20221102.ebuild +++ b/sys-firmware/intel-microcode/intel-microcode-20221108_p20221102.ebuild
    @@ -1,9 +1,9 @@
    -# Copyright 1999-20
  • From Andrew Nowa Ammerlaan@21:1/5 to All on Tue May 14 14:10:02 2024
    Hi all,

    This patch series essential moves the dist-kernel initramfs/UKI
    rebuilding out off zfs-kmod.ebuild and into linux-mod-r1.eclass.

    In the process we fix several issues:
    - zfs-kmod.ebuild binpkgs no longer rebuild currently running kernel,
    but uses instead the kernel version that matches the version the modules
    were built for (Bug 931213, commit 1,2,3,6,7).
    - zfs.ebuild binpkgs no longer rebuild currently running kernel, but use
    the eselected kernel instead (commit 6,7). In most "normal" cases the
    eselected kernel version will match the kernel version zfs-kmod modules
    were built for due to the slot operator on the virtual/dist-kernel
    dependency in both ebuilds, and the automatic adjustments of the
    /usr/src/linux symlink when dist-kernels are used. (TODO: consider
    merging the zfs and zfs-kmod ebuilds to make this 100% robust).
    - Which modules are and which ones are not included in the Dracut
    generated initramfs is now properly defined and won't change after
    random dist-kernel reinstallations (commit 1,2).
    - As a result of the above, Dracut's "--reproducible" setting actually
    makes something that is reproducible (apart from package upgrades and configuration changes of course).
    - It is now super easy for kernel module package maintainers to define
    that a module can be included in the initramfs, USE flags and
    (initramfs/UKI) rebuilds are taken care off by the eclass (commits 1,2)
    - We mirror the changes to the kernel modules in the packages that
    install CPU microcode (linux-firmware and intel-microcode) by
    introducing the "dist-kernel" flag there which alters the behaviour of
    the "initramfs" flag to instead configure Dracut to include the
    microcode and then rebuild the Dracut initramfs/UKI. This ensures that microcode updates are always propagated into the initramfs. (commits 4,5)

    Most users will not have to do anything, we are only setting defaults in /usr/lib/dracut/dracut.conf.d, the (existing) user configuration in /etc/dracut.conf.d takes precedence. Nonetheless, I wrote a news item to accompany these changes, users should ensure that they do not
    unintentionally overwrite USE=initramfs, and they may also want to get
    rid of custom configuration/workarounds they have set in /etc/dracut.conf.

    Best regards,
    Andrew


    Title: Changes to dracut kernel module/microcode handling
    Author: Andrew Ammerlaan <andrewammerlaan@gentoo.org>
    Posted: 2024-05-10
    Revision: 1
    News-Item-Format: 2.0
    Display-If-Installed: sys-kernel/dracut
    Display-If-Installed: virtual/dist-kernel
    Display-If-Installed: sys-kernel/linux-firmware
    Display-If-Installed: sys-firmware/intel-microcode


    Impact
    ====================

    Several changes were made regarding out-of-tree kernel modules, CPU
    microcode, and how these are handled in initial RAM file systems
    (initramfs) generated by sys-kernel/dracut for distribution kernels.
    Depending on the local Dracut and USE flag configuration, some
    configuration adjustments may be required as a result of these changes.

    Background (the problem)
    ====================

    Previously Dracut implicitly included all out-of-tree kernel modules
    it could find. This leads to several problems:
    - It unnecessarily increases the size of the initramfs
    - It creates a bit of a mess when using distribution kernels, consider
    the following:
    1) Distribution kernel is upgraded
    2) Initramfs for the new kernel is generated, it does not include
    any out-of-tree kernel modules.
    3) Portage triggers rebuild of the out-of-tree kernel modules
    4) If zfs is installed, its rebuild will trigger an initramfs
    re-installation. Otherwise no rebuild is triggered.
    Problem: What is and is not included in the initramfs is now
    ambiguous. It depends on the emerge order of the kernel modules
    when zfs is used. And will completely change if at some later stage
    regeneration of the initramfs is triggered manually via e.g.:
    emerge --config sys-kernel/gentoo-kernel
    As a result, Dracut's "--reproducible" setting is not working. And
    the functionality of the initramfs may change (seemingly) at random.

    Background (the fix)
    ====================

    Several things have been changed:
    - Out-of-tree kernel modules installed by portage are explicitly omitted
    from the initramfs generated by Dracut by default.
    - Packages that install a kernel module for which it might make sense to
    have it in the initramfs, have gained the "initramfs" USE flag. When
    this flag is enabled, Dracut is instructed to include the installed
    kernel modules. Packages for which it is essential that its kernel
    modules are included in the initramfs have this new flag enabled
    by default.
    - When distribution kernels are used (USE=dist-kernel), and a module
    that should be in the initramfs is installed (USE=initramfs) the
    initramfs is always re-generated.
    - The packages installing CPU microcode (sys-kernel/linux-firmware
    and sys-firmware/intel-microcode) have been adjusted to mirror the
    above changes for out-of-tree kernel modules. Both packages
    have gained the "dist-kernel" USE flag, and the "initramfs" flag is
    now enabled by default. When both flags are enabled, Dracut is
    configured to include the installed microcode in the initramfs, and
    then the initramfs is regenerated. When the "dist-kernel" flag is
    disabled, the "initramfs" flag behaves as it previously did.

    User Action Required
    ====================

    Users should double check two things:
    1) Please ensure that you are *not* globally enabling or disabling
    the "initramfs" USE flag. Enabling it globally might result in an
    unnecessarily large initramfs. Disabling it globally might result
    in missing functionality in the initramfs. Which could lead to boot
    failure if, for example, the zfs module is missing while the root
    partition is a zfs.
    2) Any add_drivers, or omit_drivers lines in /etc/dracut.conf or
    /etc/dracut.conf.d/* may override the Dracut configuration snippets
    installed by the kernel module packages in
    /usr/lib/dracut/dracut.conf.d. Please review your Dracut
    configuration files to ensure that you are not unintentionally
    overriding the settings set by Portage.

    Frequently Asked Questions
    ====================

    A package installing a kernel module I would like in my initramfs has
    not gained the "initramfs" USE flag. How do I proceed?

    Please report a new bug on bugs.gentoo.org, requesting that the
    package maintainer consider adding support to the package for
    including the modules in the initramfs. In the meantime you can
    locally override the configuration provided by the package (see
    below). Note though that when distribution kernels are used,
    regeneration of the initramfs must be triggered manually via e.g.:
    emerge --config sys-kernel/gentoo-kernel

    How do I override the provided Dracut configuration snippets to
    include/exclude a custom list of modules?

    To override the provided configuration snippet, create a new file
    /etc/dracut.conf.d/10-PACKAGENAME.conf, replacing PACKAGENAME with
    the name of the package providing the module. Add to this file:
    omit_drivers+=" my list of drivers to omit "
    and/or
    add_drivers+=" my list of drivers to include "

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrew Ammerlaan@21:1/5 to All on Tue May 14 14:10:02 2024
    Commit f51cd5b64c14ddfb83488a12d538c66a4a309376 resets kernel environment variables when binpkgs are merged, this makes sense since we care about the system that the binpkg will be installed on, not the system the binpkg was built on. This introduced Bug 931213 as a side-effect, which will be fixed
    in a separate commit to linux-mod-r1.eclass. In preparation of that, we document
    and rename the LINUX_INFO_BINARY_RESET to SKIP_KERNEL_BINPKG_ENV_RESET.

    However, commit f51cd5b64c14ddfb83488a12d538c66a4a309376 also makes binpkgs always use the running kernel version. This behaviour is surprising and confusing. In principle the 'eselect kernel' setting should still be respected when binpkgs are used.

    Bug: https://bugs.gentoo.org/926063
    Bug: https://bugs.gentoo.org/931213
    Signed-off-by: Andrew Ammerlaan <andrewammerlaan@gentoo.org>
    ---
    eclass/linux-info.eclass | 14 +++++++++++---
    1 file changed, 11 insertions(+), 3 deletions(-)

    diff --git a/eclass/linux-info.eclass b/eclass/linux-info.eclass
    index 864594f607ca..9449a6359d2a 100644
    --- a/eclass/linux-info.eclass
    +++ b/eclass/linux-info.eclass
    @@ -158,6 +158,14 @@ KERNEL_DIR="${KERNEL_DIR:-${ROOT%/}/usr/src/linux}"
    # This is a user flag and should under _no circumstances_ be set in the ebuild.
    : "${SKIP_KERNEL_CHECK:=""}"

    +# @ECLASS_VARIABLE: SKIP_KERNEL_BINPKG_ENV_RESET
    +# @DEFAULT_UNSET
    +# @DESCRIPTION:
    +# If set, do not reset the kernel environment variables when merging a package +# as a binpkg.
    +# Main use-case is for kernel modules, i.e. linux-mod-r1.eclass.
    +# This should be set before running linux-info_pkg_setup
    +
    # And to ensure all the weirdness with crosscompile
    inherit toolchain-funcs
    [[ ${EAPI} == 6 ]] && inherit eapi7-ver
    @@ -696,12 +704,12 @@ linux-info_get_any_version() {
    die "${FUNCNAME}() called on non-Linux system, please fix the ebu
  • From Mike Gilbert@21:1/5 to andrewammerlaan@gentoo.org on Wed May 15 18:10:01 2024
    On Tue, May 14, 2024 at 7:59 AM Andrew Ammerlaan
    <andrewammerlaan@gentoo.org> wrote:
    Commit f51cd5b64c14ddfb83488a12d538c66a4a309376 resets kernel environment variables when binpkgs are merged, this makes sense since we care about the system that the binpkg will be installed on, not the system the binpkg was built on. This introduced Bug 931213 as a side-effect, which will be fixed
    in a separate commit to linux-mod-r1.eclass. In preparation of that, we document
    and rename the LINUX_INFO_BINARY_RESET to SKIP_KERNEL_BINPKG_ENV_RESET.

    However, commit f51cd5b64c14ddfb83488a12d538c66a4a309376 also makes binpkgs always use the running kernel version. This behaviour is surprising and confusing. In principle the 'eselect kernel' setting should still be respected
    when binpkgs are used.

    Seems reasonable to me as the author of f51cd5b64c14ddfb83488a12d538c66a4a309376.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)