• Bug#1100677: Pending autoremoval of debian-reference* packages

    From textshell@uchuujin.de@21:1/5 to helmut@subdivi.de on Sun Apr 13 16:20:01 2025
    XPost: linux.debian.devel

    I belive debian wants to provide its documentation as packages in stable releases.

    But currently debian-reference is scheduled for auto removal on 2025-04-15 (which this mail bumps a bit into the future).
    Maybe "nobody" is aware of this?

    Anyone up to taking a look at this?

    On Mon, 17 Mar 2025 08:14:02 +0100 Helmut Grohne <helmut@subdivi.de> wrote:
    Package: debian-reference-de,debian-reference-en,debian-reference-es,debian-reference-fr,debian-reference-id,debian-reference-it,debian-reference-ja,debian-reference-pt,debian-reference-pt-br,debian-reference-zh-cn,debian-reference-zh-tw
    Version: 2.125
    Severity: serious
    User: debian-qa@lists.debian.org
    Usertags: fileconflict
    Control: affects -1 + debian-reference-common

    The debian-reference packages have a tricky undeclared file conflict
    that may break bookworm to trixie upgrades. In bookworm, debian-reference-common contains a symlink /usr/share/doc/debian-reference-common/docs pointing to ../../debian-reference whereas the debian-references-* packages in
    trixie install the same location as a directory. On the face of it, this
    is an undeclared file conflict. Really though, a bad unpack order can
    cause the unpack of the trixie files to be redirected and then go
    missing as this is a symlink to directory conversion moving between
    packages. The debian-refefence-* packages really need to prevent
    concurrent unpack with bookworm's debian-reference-common. Breaks and Replaces is not sufficient here. I think the options basically are using Conflicts or upgrading the versioned dependency on
    debian-reference-common to a Pre-Depends (requires consultation with d-devel). The latter option is a larger hammer and prevents a weird
    corner case that is not covered by Conflicts.

    Helmut

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Osamu Aoki@21:1/5 to Osamu Aoki on Thu Apr 17 14:50:01 2025
    Hi again, (dropping -devel)

    I see dpkg-maintscript-helper is already used to address

    Debian Bug report logs - #1058960
    dpkg: warning: unable to delete old directory '/usr/share/debian-reference': Directory not empty

    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058960

    https://salsa.debian.org/debian/debian-reference/-/blob/latest/debian/debian-reference-common.maintscript?ref_type=heads


    Funny thing is I use 2.109~ . Maybe this version needs to be bumped to 2.126~ or unspecified. I need to read manpage and think.

    Also, d/control needs update:

    https://salsa.debian.org/debian/debian-reference/-/blob/latest/debian/control?ref_type=heads

    Here:

    Depends: debian-reference-common (>= 2.115) ...

    These must be

    Depends: debian-reference-common (>= 2.126) ...

    Osamu


    On Thu, 2025-04-17 at 20:58 +0900, Osamu Aoki wrote:
    Hi,

    Following up on my previous post.

    How about adding simpler versioned depends (no pre-depends) with pre-rm script?

    I am talking about tricks using the "dpkg-maintscript-helper symlink_to_dir ..."
    command.  Any thought?


    Osamu


    On Thu, 2025-04-17 at 20:40 +0900, Osamu Aoki wrote:
    Hi,


    I now see this as a bug.  I think this was caused by my post-bookworm change
    in
    debian-reference (2.109) on Mon, 18 Dec 2023.

    If I remember correctly, the intent of this change was to move all HTML/PDF/Plain_Text document to a path under /usr/share/doc/ for better policy
    compliance.

    This regression was partly addressed in debian-reference (2.114) by closing #1063590 on Sat, 10 Feb 2024. What is reported is the remaining issue.

    So the question is the best known method to mitigate potential upgrade problem.

    I now understand "adding Conflicts or upgrading the versioned dependency on debian-reference-common to a Pre-Depends" is one way.

    Before doing it, I would like to know the best established method.

    How about adding simpler versioned depends (no pre-depends) with pre-rm script?
    Isn't it simpler?

    Or is there any other simpler and cleaner established solution. (A pointer to
    a
    package using such trics will be good for me.)

    Any suggestion?

    Osamu


    On Sun, 2025-04-13 at 16:08 +0200, textshell@uchuujin.de wrote:
    I belive debian wants to provide its documentation as packages in stable releases.

    But currently debian-reference is scheduled for auto removal on 2025-04-15
    (which this mail bumps a bit into the future).
    Maybe "nobody" is aware of this?

    Anyone up to taking a look at this?

    On Mon, 17 Mar 2025 08:14:02 +0100 Helmut Grohne <helmut@subdivi.de> wrote:
    Package: debian-reference-de,debian-reference-en,debian-reference- es,debian- reference-fr,debian-reference-id,debian-reference-it,debian-reference- ja,debian-reference-pt,debian-reference-pt-br,debian-reference-zh- cn,debian-
    reference-zh-tw
    Version: 2.125
    Severity: serious
    User: debian-qa@lists.debian.org
    Usertags: fileconflict
    Control: affects -1 + debian-reference-common

    The debian-reference packages have a tricky undeclared file conflict that may break bookworm to trixie upgrades. In bookworm, debian-reference-common contains a symlink /usr/share/doc/debian-reference-common/docs pointing to ../../debian-reference whereas the debian-references-* packages in trixie install the same location as a directory. On the face of it, this
    is an undeclared file conflict. Really though, a bad unpack order can cause the unpack of the trixie files to be redirected and then go missing as this is a symlink to directory conversion moving between packages. The debian-refefence-* packages really need to prevent concurrent unpack with bookworm's debian-reference-common. Breaks and Replaces is not sufficient here. I think the options basically are using
    Conflicts or upgrading the versioned dependency on debian-reference-common to a Pre-Depends (requires consultation with d-devel). The latter option is a larger hammer and prevents a weird corner case that is not covered by Conflicts.

    Helmut




    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Helmut Grohne@21:1/5 to Osamu Aoki on Sun Apr 27 11:20:02 2025
    Control: severity -1 serious

    On Thu, Apr 17, 2025 at 10:41:08PM +0900, Osamu Aoki wrote:
    As I assessed, since this bug hits user using testing only, this is not serious
    bug per Policy. Excuse me for my pedantic reasoning.

    As I reproduced the problem in bookworm to trixie upgrades with 2.126,
    the reasoning for downgrading to important no longer applies and I am
    upgrading it back to serious.

    Rationale: There is no error seen by the official testing of Debian package transition checks.

    I asked the release team how to best integrate conflict checking into
    testing migration checks and the answer was to use automated rc bugs. Therefore, please consider this rc bug an official testing transition
    check. It would also be possible to feed conflict checking directly into britney2, but we opted to use rc bugs here.

    The safe way to fix this problem is upgrading the dependencies to
    Pre-Depends and given that debian-reference-$LANG has no reverse
    dependencies, that has a low chance of causing further problems while at
    the same time reliably fixing the problem at hand by forcing correct
    ordering of the symlink transition.

    Helmut

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Osamu Aoki@21:1/5 to Helmut Grohne on Sun Apr 27 12:10:01 2025
    Hi Helmut,

    I only see three ways to get around this:
     * debian-reference-de Pre-Depends debian-reference-common (>= ...) to    ensure the correct unpack order.
     * debian-reference-de Conflicts debian-reference-common (<< ...) to    ensure the correct unpack order.
     * debian-reference-de takes care of removing debian-reference-common's    symbolic link in its own preinst.

    I wanted to follow your third suggestion. I guess I failed. I will think it again.

    Since you seem to be very knowlegeable, I have a question:

    Isn't the APT system smart enough to run all preinst scripts of all downloaded packages?

    Regards,

    Osamu


    On Sun, 2025-04-27 at 10:55 +0200, Helmut Grohne wrote:
    Control: severity -1 serious

    On Thu, Apr 17, 2025 at 10:41:08PM +0900, Osamu Aoki wrote:
    As I assessed, since this bug hits user using testing only, this is not serious
    bug per Policy.  Excuse me for my pedantic reasoning.

    As I reproduced the problem in bookworm to trixie upgrades with 2.126,
    the reasoning for downgrading to important no longer applies and I am upgrading it back to serious.

    Rationale: There is no error seen by the official testing of Debian package transition checks.

    I asked the release team how to best integrate conflict checking into
    testing migration checks and the answer was to use automated rc bugs. Therefore, please consider this rc bug an official testing transition
    check. It would also be possible to feed conflict checking directly into britney2, but we opted to use rc bugs here.

    The safe way to fix this problem is upgrading the dependencies to
    Pre-Depends and given that debian-reference-$LANG has no reverse dependencies, that has a low chance of causing further problems while at
    the same time reliably fixing the problem at hand by forcing correct
    ordering of the symlink transition.

    Helmut


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Osamu Aoki@21:1/5 to Helmut Grohne on Mon Apr 28 18:10:01 2025
    Control: tags -1 moreinfo
    Control: severity -1 important
    thanks

    Hi,

    It took me some time to set up my private apt repository to test next upload in advance without using the real Debian repository ;-) That leads me to the following conclusion.

    1. There were a bug in version 2.125
    2. Your testing script lead me to realize the bug.
    3. I see no issue here on 2.126 packages. (See below log)
    4. My fix use no pre-depends following policy guidance (See the bottom)

    Can you show me actual failure log you observed on your system with the current 2.126 packages. Until then, let me downgrade this bug

    My test result is the following:

    (https://osamuaoki.github.io/debian is my private APT repo holding debian- reference 2.126 only. I am using this with my Debain stable machine)

    ```
    osamu@goofy:~/salsa/debian-reference/test 00:02:54 ↵
    $ ls -l
    total 0
    osamu@goofy:~/salsa/debian-reference/test 00:03:05 ↵
    $ dpkg -l debian-reference\*
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name Version Architecture Description +++-=======================-============-============- =====================================================================
    ii debian-reference 2.100 all metapackage to install (all) translations of Debian Reference
    ii debian-reference-common 2.100 all Debian system administration guide, common files
    ii debian-reference-de 2.100 all Debian system administration guide, German translation
    ii debian-reference-en 2.100 all Debian system administration guide, English original
    ii debian-reference-es 2.100 all Debian system administration guide, Spanish translation
    ii debian-reference-fr 2.100 all Debian system administration guide, French translation
    ii debian-reference-id 2.100 all Debian system administration guide, Indonesian translation
    ii debian-reference-it 2.100 all Debian system administration guide, Italian translation
    ii debian-reference-ja 2.100 all Debian system administration guide, Japanese translation
    ii debian-reference-pt 2.100 all Debian system administration guide, Portuguese translation
    ii debian-reference-pt-br 2.100 all Debian system administration guide, Portuguese translation
    ii debian-reference-zh-cn 2.100 all Debian system administration guide, Chinese (Simplified) translation
    ii debian-reference-zh-tw 2.100 all Debian system administration guide, Chinese (Traditional) translation osamu@goofy:~/salsa/debian-reference/test 00:07:03 ↵
    $ dpkg -l debian-reference\*
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name Version Architecture Description +++-=======================-============-============- =====================================================================
    ii debian-reference 2.100 all metapackage to install (all) translations of Debian Reference
    ii debian-reference-common 2.100 all Debian system administration guide, common files
    ii debian-reference-de 2.100 all Debian system administration guide, German translation
    ii debian-reference-en 2.100 all Debian system administration guide, English original
    ii debian-reference-es 2.100 all Debian system administration guide, Spanish translation
    ii debian-reference-fr 2.100 all Debian system administration guide, French translation
    ii debian-reference-id 2.100 all Debian system administration guide, Indonesian translation
    ii debian-reference-it 2.100 all Debian system administration guide, Italian translation
    ii debian-reference-ja 2.100 all Debian system administration guide, Japanese translation
    ii debian-reference-pt 2.100 all Debian system administration guide, Portuguese translation
    ii debian-reference-pt-br 2.100 all Debian system administration guide, Portuguese translation
    ii debian-reference-zh-cn 2.100 all Debian system administration guide, Chinese (Simplified) translation
    ii debian-reference-zh-tw 2.100 all Debian system administration guide, Chinese (Traditional) translation osamu@goofy:~/salsa/debian-reference/test 00:07:12 ↵
    $ apt list debian-reference\*
    Listing... Done
    debian-reference-common/stable,now 2.100 all [installed,automatic] debian-reference-de/stable,now 2.100 all [installed,automatic] debian-reference-en/stable,now 2.100 all [installed,automatic] debian-reference-es/stable,now 2.100 all [installed,automatic] debian-reference-fr/stable,now 2.100 all [installed,automatic] debian-reference-id/stable,now 2.100 all [installed,automatic] debian-reference-it/stable,now 2.100 all [installed,automatic] debian-reference-ja/stable,now 2.100 all [installed,automatic] debian-reference-pt-br/stable,now 2.100 all [installed] debian-reference-pt/stable,now 2.100 all [installed,automatic] debian-reference-zh-cn/stable,now 2.100 all [installed,automatic] debian-reference-zh-tw/stable,now 2.100 all [installed,automatic] debian-reference/stable,now 2.100 all [installed] osamu@goofy:~/salsa/debian-reference/test 00:07:31 ↵
    $ sudo apt update
    Hit:1 https://deb.debian.org/debian bookworm InRelease
    Hit:2 https://deb.debian.org/debian bookworm-updates InRelease
    Hit:3 https://deb.debian.org/debian bookworm-backports InRelease
    Hit:4 https://dl.google.com/linux/chrome/deb stable InRelease
    Get:5 https://osamuaoki.github.io/debian sid InRelease [1,846 B]
    Hit:6 https://security.debian.org/debian-security bookworm-security InRelease Get:7 https://osamuaoki.github.io/debian sid/main amd64 Packages [7,468 B] Fetched 9,314 B in 1s (8,446 B/s)
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    13 packages can be upgraded. Run 'apt list --upgradable' to see them. osamu@goofy:~/salsa/debian-reference/test 00:07:47 ↵
    $ apt list debian-reference\*
    Listing... Done
    debian-reference-common/unknown 2.126 all [upgradable from: 2.100] debian-reference-de/unknown 2.126 all [upgradable from: 2.100] debian-reference-en/unknown 2.126 all [upgradable from: 2.100] debian-reference-es/unknown 2.126 all [upgradable from: 2.100] debian-reference-fr/unknown 2.126 all [upgradable from: 2.100] debian-reference-id/unknown 2.126 all [upgradable from: 2.100] debian-reference-it/unknown 2.126 all [upgradable from: 2.100] debian-reference-ja/unknown 2.126 all [upgradable from: 2.100] debian-reference-pt-br/unknown 2.126 all [upgradable from: 2.100] debian-reference-pt/unknown 2.126 all [upgradable from: 2.100] debian-reference-zh-cn/unknown 2.126 all [upgradable from: 2.100] debian-reference-zh-tw/unknown 2.126 all [upgradable from: 2.100] debian-reference/unknown 2.126 all [upgradable from: 2.100] osamu@goofy:~/salsa/debian-reference/test 00:08:02 ↵
    $ apt-get download debian-reference-de
    Get:1 https://osamuaoki.github.io/debian sid/main amd64 debian-reference-de all 2.126 [1,925 kB]
    Fetched 1,925 kB in 1s (2,714 kB/s) osamu@goofy:~/salsa/debian-reference/test 00:11:17 ↵
    $ ls -l
    total 1892
    -rw-r--r-- 1 osamu osamu 1925248 Apr 28 23:23 debian-reference-de_2.126_all.deb osamu@goofy:~/salsa/debian-reference/test 00:11:28 ↵
    $ sudo dpkg --unpack --auto-deconfigure debian-reference-de_2.126_all.deb (Reading database ... 636494 files and directories currently installed.) Preparing to unpack debian-reference-de_2.126_all.deb ...
    Unpacking debian-reference-de (2.126) over (2.100) ...
    Processing triggers for doc-base (0.11.1) ...
    Processing 1 changed doc-base file...
    osamu@goofy:~/salsa/debian-reference/test 00:13:12 ↵
    $ sudo apt-get -y -f install
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    Correcting dependencies... Done
    The following additional packages will be installed:
    debian-reference-common
    Suggested packages:
    calibre
    The following packages will be upgraded:
    debian-reference-common
    1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.
    1 not fully installed or removed.
    Need to get 0 B/38.1 kB of archives.
    After this operation, 7,168 B of additional disk space will be used.
    N: Make snapshot (system snapshot mode): BASE=/ DEST=/disk_volume/main/@rootfs-snapshots TIME=2025-04-28T15:13:54+00:00 TYPE=pre
    Create a snapshot of '/' in '/disk_volume/main/@rootfs-snapshots/2025-04- 28T15:13:54+00:00.pre'
    apt-listchanges: Reading changelogs...
    (Reading database ... 636493 files and directories currently installed.) Preparing to unpack .../debian-reference-common_2.126_all.deb ...
    Unpacking debian-reference-common (2.126) over (2.100) ...
    Setting up debian-reference-common (2.126) ...
    Setting up debian-reference-de (2.126) ...
    Processing triggers for mailcap (3.70+nmu1) ...
    Processing triggers for desktop-file-utils (0.26-1) ...
    Processing triggers for gnome-menus (3.36.0-1.1) ...
    Processing triggers for man-db (2.11.2-2) ...
    N: Make snapshot (system snapshot mode): BASE=/ DEST=/disk_volume/main/@rootfs-snapshots TIME=2025-04-28T15:13:57+00:00 TYPE=post
    Create a snapshot of '/' in '/disk_volume/main/@rootfs-snapshots/2025-04- 28T15:13:57+00:00.post'
    Scanning processes... Scanning processor microcode... Scanning linux images...

    Running kernel seems to be up-to-date.

    The processor microcode seems to be up-to-date.

    No services need to be restarted.

    No containers need to be restarted.

    No user sessions are running outdated binaries.

    No VM guests are running outdated hypervisor (qemu) binaries on this host. osamu@goofy:~/salsa/debian-reference/test 00:13:59 ↵
    $
    ```

    I see symlink /usr/share/doc/debian-reference-common/doc is properly changed to directory ;-) No more issue seen here.


    FYI: Your previous log had:
    | W: Download is performed unsandboxed as root as file '//debian-reference- de_2.125_all.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

    This is just warning since you run this "apt-get download ..." command as root.


    On Sun, 2025-04-27 at 17:15 +0200, Helmut Grohne wrote:
    Hi Osamu,

    On Sun, Apr 27, 2025 at 07:01:44PM +0900, Osamu Aoki wrote:
    Since you seem to be very knowlegeable, I have a question:

    Isn't the APT system smart enough to run all preinst scripts of all downloaded
    packages?

    apt is only half the story. In principle, you may perform upgrades with dselect or dpkg directly. It just so happens that everyone uses apt. Now
    apt instructs dpkg to do its thing, but subtle changes in dependencies
    may instruct apt to order its instructions differently. Ultimately, what needs to happen here is making valid dpkg interactions work.

    Generally, dpkg may choose to unpack debian-reference-$LANG before
    unpacking debian-reference-common even though it may only configure debian-reference-$LANG after having configured debian-reference-common.
    In particular, debian-reference-$LANG may be unpacked before running debian-reference-common.preinst. I see how that is not desirable. We
    would like to tell apt and dpkg to always unpack debian-reference-common before unpacking debian-reference-$LANG. The way to do that is declaring suitable Conflicts or better Pre-Depends.

    I argue that debian-reference-$LANG should really declare:

        Depends: debian-reference-common (= ${source:Version})
        Pre-Depends: debian-reference-common (>= 2.109~)

    https://www.debian.org/doc/debian-policy/ch-relationships.html#binary-dependencies-depends-recommends-suggests-enhances-pre-depends

    In this section, Policy states:

    Pre-Depends are also required if the preinst script depends on the named package. It is best to avoid this situation if possible.

    Pre-Depends should be used sparingly, preferably only by packages whose premature upgrade or installation would hamper the ability of the system to continue with any upgrade that might be in progress.

    You should not specify a Pre-Depends entry for a package before this has been discussed on the debian-devel mailing list and a consensus about doing that has been reached. See Dependencies.


    In my test, the updated package seems to work without pre-depends, I should avoid using it, I think.

    Regards,

    Osamu

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Helmut Grohne@21:1/5 to Osamu Aoki on Mon Apr 28 22:50:01 2025
    Control: tags -1 - moreinfo
    Control: severity -1 serious

    On Tue, Apr 29, 2025 at 01:00:58AM +0900, Osamu Aoki wrote:
    It took me some time to set up my private apt repository to test next upload in
    advance without using the real Debian repository ;-) That leads me to the following conclusion.

    1. There were a bug in version 2.125
    2. Your testing script lead me to realize the bug.
    3. I see no issue here on 2.126 packages. (See below log)
    4. My fix use no pre-depends following policy guidance (See the bottom)

    Can you show me actual failure log you observed on your system with the current
    2.126 packages. Until then, let me downgrade this bug

    Sure. Take my previous reproducer and replace trixie with sid since
    2.126 has not yet migrated.

    | $ mmdebstrap --variant=apt '' /dev/null 'deb http://deb.debian.org/debian bookworm main' --include=debian-reference-common --chrooted-customize-hook='sed -i -e s/bookworm/sid/ /etc/apt/sources.list && apt-get update && apt-get download debian-reference-
    de && dpkg --unpack --auto-deconfigure *.deb && apt-get -y -f install && dpkg --verify'
    | I: automatically chosen mode: unshare
    | I: chroot architecture amd64 is equal to the host's architecture
    | I: automatically chosen format: null
    | I: using /tmp/mmdebstrap.Njtt81dugF as tempdir
    | I: running apt-get update...
    | done
    | I: downloading packages with apt...
    | done
    | I: extracting archives...
    | done
    | I: installing essential packages...
    | done
    | I: installing remaining packages inside the chroot...
    | done
    | done
    | I: running --chrooted-customize-hook in shell: sh -c 'sed -i -e s/bookworm/sid/ /etc/apt/sources.list && apt-get update && apt-get download debian-reference-de && dpkg --unpack --auto-deconfigure *.deb && apt-get -y -f install && dpkg --verify'
    | Get:1 http://deb.debian.org/debian sid InRelease [205 kB]
    | Get:2 http://deb.debian.org/debian sid/main amd64 Packages [10.1 MB]
    | Fetched 10.3 MB in 1s (8266 kB/s)
    | Reading package lists... Done
    | Get:1 http://deb.debian.org/debian sid/main amd64 debian-reference-de all 2.126 [1927 kB]
    | Fetched 1927 kB in 0s (63.3 MB/s)
    | W: Download is performed unsandboxed as root as file '//debian-reference-de_2.126_all.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
    | Selecting previously unselected package debian-reference-de.
    | (Reading database ... 6748 files and directories currently installed.)
    | Preparing to unpack debian-reference-de_2.126_all.deb ...
    | Unpacking debian-reference-de (2.126) ...
    | Reading package lists... Done
    | Building dependency tree... Done
    | Reading state information... Done
    | Correcting dependencies... Done
    | The following additional packages will be installed:
    | debian-reference-common
    | Suggested packages:
    | calibre debian-reference debian-reference-en debian-reference-es debian-reference-fr debian-reference-id debian-reference-it debian-reference-ja debian-reference-pt
    | debian-reference-zh-cn debian-reference-zh-tw mc vim
    | Recommended packages:
    | w3m | www-browser
    | The following packages will be upgraded:
    | debian-reference-common
    | 1 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
    | 1 not fully installed or removed.
    | Need to get 38.1 kB of archives.
    | After this operation, 7168 B of additional disk space will be used.
    | Get:1 http://deb.debian.org/debian sid/main amd64 debian-reference-common all 2.126 [38.1 kB]
    | Fetched 38.1 kB in 0s (3663 kB/s)
    | (Reading database ... 6771 files and directories currently installed.)
    | Preparing to unpack .../debian-reference-common_2.126_all.deb ...
    | Unpacking debian-reference-common (2.126) over (2.100) ...
    | dpkg: warning: unable to delete old directory '/usr/share/debian-reference': Directory not empty
    | Setting up debian-reference-common (2.126) ...
    | Setting up debian-reference-de (2.126) ...
    | missing /usr/share/doc/debian-reference-common/docs/apa.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch01.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch02.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch03.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch04.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch05.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch06.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch07.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch08.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch09.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch10.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch11.de.html
    | missing /usr/share/doc/debian-reference-common/docs/ch12.de.html
    | missing /usr/share/doc/debian-reference-common/docs/debian-reference.de.pdf
    | missing /usr/share/doc/debian-reference-common/docs/debian-reference.de.txt.gz
    | missing /usr/share/doc/debian-reference-common/docs/index.de.html
    | missing /usr/share/doc/debian-reference-common/docs/pr01.de.html
    | I: cleaning package lists and apt cache...
    | done
    | done
    | I: removing tempdir /tmp/mmdebstrap.Njtt81dugF...
    | I: success in 11.5543 seconds
    | $


    $ apt-get download debian-reference-de
    Get:1 https://osamuaoki.github.io/debian sid/main amd64 debian-reference-de all
    2.126 [1,925 kB]
    Fetched 1,925 kB in 1s (2,714 kB/s)
    osamu@goofy:~/salsa/debian-reference/test 00:11:17 ↵
    $ ls -l
    total 1892
    -rw-r--r-- 1 osamu osamu 1925248 Apr 28 23:23 debian-reference-de_2.126_all.deb
    osamu@goofy:~/salsa/debian-reference/test 00:11:28 ↵
    $ sudo dpkg --unpack --auto-deconfigure debian-reference-de_2.126_all.deb (Reading database ... 636494 files and directories currently installed.) Preparing to unpack debian-reference-de_2.126_all.deb ...
    Unpacking debian-reference-de (2.126) over (2.100) ...
    Processing triggers for doc-base (0.11.1) ...
    Processing 1 changed doc-base file... osamu@goofy:~/salsa/debian-reference/test 00:13:12 ↵

    This is closely following my reproducer. At this point, the damage
    should be there. Whilst, debian-reference-de.preinst attempts to perform
    the link to directory conversion, the link is owned by the "wrong"
    package and hence the helper does not act. At this point, dpkg --verify
    should tell you about missing files.

    $ sudo apt-get -y -f install
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    Correcting dependencies... Done
    The following additional packages will be installed:
    debian-reference-common
    Suggested packages:
    calibre
    The following packages will be upgraded:
    debian-reference-common
    1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.
    1 not fully installed or removed.
    Need to get 0 B/38.1 kB of archives.
    After this operation, 7,168 B of additional disk space will be used.
    N: Make snapshot (system snapshot mode): BASE=/ DEST=/disk_volume/main/@rootfs-snapshots TIME=2025-04-28T15:13:54+00:00 TYPE=pre
    Create a snapshot of '/' in '/disk_volume/main/@rootfs-snapshots/2025-04- 28T15:13:54+00:00.pre'
    apt-listchanges: Reading changelogs...
    (Reading database ... 636493 files and directories currently installed.) Preparing to unpack .../debian-reference-common_2.126_all.deb ...
    Unpacking debian-reference-common (2.126) over (2.100) ...
    Setting up debian-reference-common (2.126) ...

    This is when the link really got converted into a directory.

    Setting up debian-reference-de (2.126) ...
    Processing triggers for mailcap (3.70+nmu1) ...
    Processing triggers for desktop-file-utils (0.26-1) ...
    Processing triggers for gnome-menus (3.36.0-1.1) ...
    Processing triggers for man-db (2.11.2-2) ...
    N: Make snapshot (system snapshot mode): BASE=/ DEST=/disk_volume/main/@rootfs-snapshots TIME=2025-04-28T15:13:57+00:00 TYPE=post
    Create a snapshot of '/' in '/disk_volume/main/@rootfs-snapshots/2025-04- 28T15:13:57+00:00.post'
    Scanning processes...
    Scanning processor microcode...
    Scanning linux images...

    Running kernel seems to be up-to-date.

    The processor microcode seems to be up-to-date.

    No services need to be restarted.

    No containers need to be restarted.

    No user sessions are running outdated binaries.

    No VM guests are running outdated hypervisor (qemu) binaries on this host. osamu@goofy:~/salsa/debian-reference/test 00:13:59 ↵
    $
    ```

    Yeah. It looks as if all is right, but some files are now elsewhere and
    dpkg --verify tells you which.

    I see symlink /usr/share/doc/debian-reference-common/doc is properly changed to
    directory ;-) No more issue seen here.

    Yes, the migration from link to directory works, but debian-reference-de
    was unpacked before said migration. Therefore files installed beneath
    the link have been redirected elsewhere.

    In this section, Policy states:

    Pre-Depends are also required if the preinst script depends on the named package. It is best to avoid this situation if possible.

    Pre-Depends should be used sparingly, preferably only by packages whose premature upgrade or installation would hamper the ability of the system to continue with any upgrade that might be in progress.

    You should not specify a Pre-Depends entry for a package before this has been
    discussed on the debian-devel mailing list and a consensus about doing that has been reached. See Dependencies.


    In my test, the updated package seems to work without pre-depends, I should avoid using it, I think.

    I am aware of this policy aspect. I argue that this is one of those
    exceptional cases where Pre-Depends really is the cure and the risk of downsides is manageable, because the packages that shall issue
    Pre-Depends do not have any reverse dependencies.

    There is another relatively simple argument that might help with
    understanding the matter. Take a moment to think about file ownership in
    a packaging sense. Initially,
    /usr/share/doc/debian-reference-common/docs is a symbolic link owned by debian-reference-common. Now you unpack debian-reference-de, which
    installs the same location as a directory. Which package is now the
    proper owner of that file? After unpacking debian-reference-de, there is
    no obligation to configure it. You may as well remove it again. Now that location will be gone, but debian-reference-common (which hasn't been
    upgraded yet) would still own it in principle. In a normal situation,
    you'd declare Breaks+Replaces from debian-reference-de for debian-reference-common, because you are transferring ownership of a
    file. However, that method does not work for links being turned into directories and therefore we need the stronger cousin Conflicts.
    Unfortunately, Conflicts are not sufficient either, because dpkg
    actually allows concurrent unpacks despite declared Conflicts if the
    other package is going away. Therefore, there is no reliable mechanism
    weaker than Pre-Depends that reliably avoids the file-loss situation.

    And yeah, by all means, discuss the use of Pre-Depends with
    d-devel@l.d.o.

    Helmut

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Helmut Grohne@21:1/5 to Osamu Aoki on Tue Apr 29 13:10:02 2025
    On Tue, Apr 29, 2025 at 11:02:04AM +0900, Osamu Aoki wrote:
    I still see APT warning in your updated test install log.

    That's purely from apt-get download. We may separate the invocations to
    make the source apparent.

    Are you insisting that "Preparing to unpack" process run under warned condition
    is a valid test case to warrant declaring pre-depends against the policy guidance.

    I am insisting that the warning comes from apt-get download and not from
    dpkg and that dpkg is not run by apt here. The unpack condition is
    otherwise valid.

    "Preparing to unpack" runs preinst in advance. If this process is run by _apt,
    it may be unable to find preinst and skipped. This may be the case in your test
    case due to this permission problem.

    It is not run by _apt.

    Unpacking debian-reference-de (2.126) ...

    It looks like it failed since preinst script was skipped when it really meant to
    be executed.

    The preinst script was run, but the helper concluded that it should not
    act.

    This is closely following my reproducer. At this point, the damage
    should be there. Whilst, debian-reference-de.preinst attempts to perform the link to directory conversion, the link is owned by the "wrong"
    package and hence the helper does not act. At this point, dpkg --verify should tell you about missing files.


    You say "should". But really? I don't see such problem in my test case since
    debian-reference-de.preinst is accessible and executed. Here is the situation:

    osamu@goofy:~/salsa/debian-reference/test 10:19:23 ↵
    $ sudo dpkg --unpack --auto-deconfigure debian-reference-de_2.126_all.deb (Reading database ... 636574 files and directories currently installed.) Preparing to unpack debian-reference-de_2.126_all.deb ...
    Unpacking debian-reference-de (2.126) over (2.126) ...
    Processing triggers for doc-base (0.11.1) ...
    Processing 1 changed doc-base file... osamu@goofy:~/salsa/debian-reference/test 10:21:20 ↵
    $ ls -l /usr/share/doc/debian-reference-common/
    total 24
    -rw-r--r-- 1 root root 6616 Apr 24 11:18 changelog.gz
    -rw-r--r-- 1 root root 1430 Apr 24 11:18 copyright
    drwxr-xr-x 1 root root 548 Apr 29 10:20 docs
    -rw-r--r-- 1 root root 8893 Apr 24 11:18 README.md.gz osamu@goofy:~/salsa/debian-reference/test 10:21:28 ↵

    Now this is where it truly gets interesting. Thanks for adding that ls
    -l output at the right spot: You run it after the unpacking of debian-reference-de before the unpacking of debian-reference-common
    which is exactly the spot where I claim the problem to occur.

    I still cannot reproduce your output. There are two sensible options I
    see here. Either, the dpkg-maintscript-helper acted. Then it will have
    backed up the docs symlink and thus there should be a link
    "docs.dpkg-backup". Evidently this is not the case for you. The other
    option is that it has not acted and then "docs" should be a symlink. In
    no case does dpkg-maintscript-helper delete the docs link during
    preinst. This strongly suggests that your test environment was not in a
    sane state at the beginning.

    I see "own" reference only in "Switching a directory to symlink" for "dpkg- maintscript-helper dir_to_symlink ...". Your argument doesn't make sense.

    I stand corrected. There is no ownership checking in that code path.
    However, there is version checking code.

    dpkg --compare-versions -- "$2" le-nl "$LASTVERSION"

    That "$2" is the version of the old debian-reference-de. If there is one installed, the dpkg-maintscript-helper will move the symlink and there
    is docs.dpkg-backup. When debian-reference-de is not installed, the
    version is empty and the "-nl" suffix renders the comparison false
    skipping the action.

     $ sudo apt-get -y -f install
    ...
    Unpacking debian-reference-common (2.126) over (2.100) ...
    Setting up debian-reference-common (2.126) ...

    This is when the link really got converted into a directory.

    No. It was done by debian-reference-de.preinst as shown in the above and debian-reference-common.preinst here becomes NOP since there is no symlink.

    We need to add precision here. What you describe is what happens when debian-reference-de is initially installed. It can also happen the other
    way round when it is not.

    So here is a slightly improved reproducer.

    mmdebstrap --variant=apt '' /dev/null 'deb http://deb.debian.org/debian bookworm main' --include=debian-reference-common --chrooted-customize-hook='sed -i -e s/bookworm/sid/ /etc/apt/sources.list && apt-get update && apt-get download debian-reference-de'
    --chrooted-customize-hook='DPKG_DEBUG=1 dpkg --unpack --auto-deconfigure *.deb' --chrooted-customize-hook='ls -la /usr/share/doc/debian-reference-common/ && apt-get -y -f install && dpkg --verify'

    I split the unpacking step into a separate hook for you to clearly see
    how they're disconnected. I also added DPKG_DEBUG to let us see that the maintscript helper indeed is being run and copied your ls -l invocation
    to observe its effects. You will see that in that reproducer, the
    symlink is not converted and encourage you to run it locally so you can
    play with it in a fresh and clean environment rather than operating on
    your host system and therefore skip including any output here.
    mmdebstrap operates as a regular user account and given that the target
    is /dev/null it does not write any files outside $TMPDIR and cleans up
    behind itself. A useful trick for interactive examination is --chrooted-customize-hook=bash as that results in you being dropped into
    a shell prompt until you exit it.

    If you change --include=debian-reference-common to --include=debian-reference-de, you shall observe that at that point the
    "le-nl" comparison becomes successful and therefore, the symlink is
    converted during debian-reference-de.preinst and thus the problem is no
    longer observable.

    There are other corner cases that should be broken if you initially
    start with bookworm's debian-reference-de installed. Consider:
    * unpack sid debian-reference-de (renames the docs link)
    * purge debian-reference-de (deletes the docs.dpkg-backup link)
    * dpkg --verify now complains about the docs link owned by
    debian-reference-common going missing

    Helmut

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Osamu Aoki@21:1/5 to All on Tue Apr 29 18:10:01 2025
    Hi,

    Thanks for your response and patience.

    I now realize my test environment installed both debian-refeence-common/2.100 and debian-refeence-de/2.100 before upgrade while your test environment installed only debian-refeence-common/2.100. I see that is the situation causing problem.

    I thought I followed your code but I was wrong.

    The current d/maintscript-in doesn't isn't run when it is new install as you pointed out.

    However, there is version checking code.

        dpkg --compare-versions -- "$2" le-nl "$LASTVERSION"

    That "$2" is the version of the old debian-reference-de. If there is one installed, the dpkg-maintscript-helper will move the symlink and there
    is docs.dpkg-backup. When debian-reference-de is not installed, the
    version is empty and the "-nl" suffix renders the comparison false
    skipping the action.


    I suppose I need to come up with custom preinst script which always rename even when previous version is not installed.

    If you change --include=debian-reference-common to --include=debian-reference-de, you shall observe that at that point the "le-nl" comparison becomes successful and therefore, the symlink is
    converted during debian-reference-de.preinst and thus the problem is no longer observable.

    Yes.

    There are other corner cases that should be broken if you initially
    start with bookworm's debian-reference-de installed. Consider:
     * unpack sid debian-reference-de (renames the docs link)
     * purge debian-reference-de (deletes the docs.dpkg-backup link)
     * dpkg --verify now complains about the docs link owned by
       debian-reference-common going missing

    I understand "dpkg --verify now complains..." is ugly. But doesn't the following sid's debian-reference-common installation induced by dependency fixes
    situation.

    I need to look into this tomorrow sometime. Good night.

    Osamu

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Osamu Aoki@21:1/5 to All on Tue Apr 29 19:10:01 2025
    Hi,

    It looks like

    diff --git a/debian/all.maintscript-in b/debian/all.maintscript-in
    index bf02b4f9..ab378d0a 100644
    --- a/debian/all.maintscript-in
    +++ b/debian/all.maintscript-in
    @@ -1 +1 @@
    -symlink_to_dir /usr/share/doc/debian-reference-common/docs ../../debian-reference 2.126~ debian-reference-@@
    +symlink_to_dir /usr/share/doc/debian-reference-common/docs ../../debian-reference 2.127~ debian-reference-common

    with version bumps on d/control etc. may work. I will test it tomorrow.

    Regards,

    Osamu

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Osamu Aoki@21:1/5 to Osamu Aoki on Wed Apr 30 02:40:01 2025
    Hi,

    It is not as guessed. I need some time.

    Osamu


    On Wed, 2025-04-30 at 02:01 +0900, Osamu Aoki wrote:
    Hi,

    It looks like

    diff --git a/debian/all.maintscript-in b/debian/all.maintscript-in
    index bf02b4f9..ab378d0a 100644
    --- a/debian/all.maintscript-in
    +++ b/debian/all.maintscript-in
    @@ -1 +1 @@
    -symlink_to_dir /usr/share/doc/debian-reference-common/docs ../../debian-reference 2.126~ debian-reference-@@
    +symlink_to_dir /usr/share/doc/debian-reference-common/docs ../../debian-reference 2.127~ debian-reference-common

    with version bumps on d/control etc. may work. I will test it tomorrow.

    Regards,

    Osamu


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Osamu Aoki@21:1/5 to All on Wed Apr 30 04:50:01 2025
    Hi,

    I just uploaded packages using pre-depends.

    There are so many complicated corner cases to worry with maintscript and I run out of my time.

    TBH, Policy needs to be updated to make APT compatible depends usages OK.

    Since I didn't have enough sleep etc., I may have made errors.

    Please remind me if this upload didn't fix right.

    Regards,

    Osamu

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)