• Bug#1100629: systemd-ukify: ability to use a different zstd implementat

    From Simon McVittie@21:1/5 to All on Sun Mar 16 13:50:01 2025
    Package: systemd-ukify
    Version: 257.4-3
    Severity: wishlist
    Tags: upstream

    systemd-ukify currently supports zstd decompression by using
    python3-zstandard, which is a hard dependency; but python3-zstandard
    seems to require source changes every time libzstd is upgraded as a
    result of having been designed with a vendored libzstd in mind
    (#1100475) which makes it more fragile than seems ideal, especially if
    the long-term goal is for Debian's boot process to make more use of UKIs
    in future.

    Looking at the source of systemd-ukify, it seems like it only makes a
    very simple use of this module: import the module, and decompress one compressed binary blob into an uncompressed byte array in RAM. This
    could presumably be done equally well with any implementation of zstd.

    Perhaps it would be worthwhile to have a code path that implements zstd decompression by using python3-pyzstd (which is somewhat higher-popcon
    than python3-zstandard at the moment), or python3-zstd (although that
    one is lower-popcon), or by invoking the zstd CLI tool (which is a
    Recommends of initramfs-tools-core) via subprocess? That would make it
    easier to swap between the implementations if one of them has
    problematic properties.

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luca Boccassi@21:1/5 to All on Sun Mar 16 14:10:01 2025
    Control: tags -1 wontfix
    Control: close -1

    On Sun, 16 Mar 2025 12:44:57 +0000 Simon McVittie <smcv@debian.org>
    wrote:
    Package: systemd-ukify
    Version: 257.4-3
    Severity: wishlist
    Tags: upstream

    systemd-ukify currently supports zstd decompression by using python3-zstandard, which is a hard dependency; but python3-zstandard
    seems to require source changes every time libzstd is upgraded as a
    result of having been designed with a vendored libzstd in mind
    (#1100475) which makes it more fragile than seems ideal, especially
    if
    the long-term goal is for Debian's boot process to make more use of
    UKIs
    in future.

    Looking at the source of systemd-ukify, it seems like it only makes a
    very simple use of this module: import the module, and decompress one compressed binary blob into an uncompressed byte array in RAM. This
    could presumably be done equally well with any implementation of
    zstd.

    Perhaps it would be worthwhile to have a code path that implements
    zstd
    decompression by using python3-pyzstd (which is somewhat higher-
    popcon
    than python3-zstandard at the moment), or python3-zstd (although that
    one is lower-popcon), or by invoking the zstd CLI tool (which is a Recommends of initramfs-tools-core) via subprocess? That would make
    it
    easier to swap between the implementations if one of them has
    problematic properties.

    Sadly that's not possible, as python3-pyzstd does not work with
    compressed kernels, it cannot extract them correctly as it doesn't
    support stream decompression. That's what it used before, but it never
    worked. If anybody can fix python3-pyzstd to work with kernel images it
    can be revisited, but it seems unlikely as upstream explicitly rejected
    it in the past:

    https://github.com/sergey-dryabzhinsky/python-zstd/issues/53

    This is only a corner case anyway, kernel images should use zboot
    instead of shipping compressed files, as firmwares cannot handle that
    anyway. It's only an issue with some Ubuntu architectures other than
    x86_64.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luca Boccassi@21:1/5 to Simon McVittie on Sun Mar 16 15:40:01 2025
    On Sun, 16 Mar 2025 at 14:26, Simon McVittie <smcv@debian.org> wrote:

    On Sun, 16 Mar 2025 at 13:07:50 +0000, Luca Boccassi wrote:
    This is only a corner case anyway, kernel images should use zboot
    instead of shipping compressed files, as firmwares cannot handle that >anyway. It's only an issue with some Ubuntu architectures other than >x86_64.

    Should python3-zstandard perhaps be a Recommends (or lower) on the architectures that don't need it, if not all do?

    Cross-building is supported, and arm64 needs it, which is a release architecture, so I don't think that would improve the situation much.
    I'll try to look into fixing python3-zstandard, it's unfortunate that
    the zstd transition was started to late

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to Luca Boccassi on Sun Mar 16 15:30:02 2025
    On Sun, 16 Mar 2025 at 13:07:50 +0000, Luca Boccassi wrote:
    This is only a corner case anyway, kernel images should use zboot
    instead of shipping compressed files, as firmwares cannot handle that
    anyway. It's only an issue with some Ubuntu architectures other than
    x86_64.

    Should python3-zstandard perhaps be a Recommends (or lower) on the architectures that don't need it, if not all do?

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luca Boccassi@21:1/5 to All on Mon Mar 17 18:40:01 2025
    On Sun, 16 Mar 2025 14:33:23 +0000 Simon McVittie <smcv@debian.org>
    wrote:
    On Sun, 16 Mar 2025 at 14:28:52 +0000, Luca Boccassi wrote:
    I'll try to look into fixing python3-zstandard, it's unfortunate
    that
    the zstd transition was started to late

    It's a bit of a weird transition, because upgrading a library to a
    new
    micro version with no new SONAME and no API/ABI breaks would normally
    be
    treated as a completely routine thing that doesn't need transitions
    or
    coordination, but python3-zstandard seems to be unusually
    tightly-coupled to an exact version of libzstd.

    I have fixed python3-zstandard, so things are back to working in
    unstable. I'm hand-holding it so that it transitions to testing in a
    reasonable time too.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)