Hey Ken,

ma 21.4.2025 klo 1.42 Ken Milmore (ken.milmore@gmail.com) kirjoitti:

Package: dhcpcd-base
Version: 1:10.1.0-9

Description:

When using dhcpcd-base together with systemd-resolved, the name server settings for an interface are initially configured correctly from DHCP but may then subsequently disappear, leaving no DNS server configured.

This is because the dhcpcd hook scripts contain support for resolvconf, but the implementation of resolvconf provided by systemd-resolved is very limited and does not support the full semantics required (see below).

Steps to Reproduce:

1. Install systemd-resolved (257.5-2) along with dhcpcd-base (1:10.1.0-9) and ifupdown (0.8.44)

2. Configure an interface for DHCP in /etc/network/interfaces.
As an example I am using a usermode (slirp) interface in QEMU:

allow-hotplug enp1s0
iface enp1s0 inet dhcp

3. Reboot, or restart the interface with ifdown/ifup.

4. Observe the DNS server configuration with resolvectl, repeating a few minutes later.

Expected Behaviour:

resolvectl should show the DNS server address configured by DHCP. Name resolution should work.

Observed Behaviour:

Shortly after the interface is brought up, resolvectl does indeed show that a DNS server has been configured, e.g.:

Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp1s0)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 172.17.2.3
DNS Servers: 172.17.2.3
Default Route: yes

But a few minutes later, the server seems to have disappeared again and DNS is no longer working:

Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp1s0)
Current Scopes: none
Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Default Route: no

Investigation:

Turning on the debug option for dhcpcd shows a number of hook script calls after the interface has been configured, e.g:

Apr 20 21:07:36 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks PREINIT
Apr 20 21:07:36 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks PREINIT
Apr 20 21:07:36 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks NOCARRIER
Apr 20 21:07:36 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks NOCARRIER
Apr 20 21:07:36 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks CARRIER
Apr 20 21:07:36 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks CARRIER
Apr 20 21:07:39 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:07:39 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:07:41 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks BOUND
Apr 20 21:07:41 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks BOUND
Apr 20 21:12:20 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:21:56 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:28:29 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT

Examining the hook script /usr/lib/dhcpcd/dhcpcd-hooks/20-resolv.conf, shows that when a resolvconf binary is present, each hook call will ultimately invoke resolvconf to add or delete the DNS settings for a particular protocol on the interface. This

results in a series of invocations such as the following:

resolvconf -d enp1s0.dhcp -f
resolvconf -d enp1s0.link -f
resolvconf -d enp1s0.link -f
resolvconf -d enp1s0.ra -f
resolvconf -a enp1s0.dhcp (with a resolv.conf-formatted file piped to stdin)
resolvconf -d enp1s0.ra -f

This is where the difference in semantics becomes apparent between the versions of Debian resolvconf/openresolv and the resolvconf provided by systemd-resolved:

The "traditional" resolvconf maintains a separate state for each protocol on the interface, and generates a superposition of the information from all protocols. By contrast, the systemd version ignores the protocol suffix completely and stores a single

state for all protocols. The last information supplied for any protocol typically wins out.

In the particular example above, it appears that an IPv6 router advertisement for "enp1s0.ra" has deleted the DNS configuration previously set for "enp1s0.ra". I expect that various other sequences of events may lead to a similar result.

Possible Solutions:

I attach a patch which modifies the dhcpcd hook scripts to work around the problem: We detect when the systemd version of resolvconf is being used and provide it with a resolv.conf fragment that contains the superposition of all the protocols on a

given interface (retrieved from state stored under /run/dhcpcd).

I have not tested the patch extensively, although it has worked well enough in my test cases.

Note that the ifupdown and isc-dhcp-client packages already have a their own hook scripts (borrowed from Ubuntu) for dealing with systemctl-resolved. These scripts call resolvectl to set the DNS servers, and there is a shared state that is maintained

by both packages so they are to some extent integrated together. Since dhcpcd has now replaced isc-dhcp-client as the default DHCP client for trixie, this integration now seems to be rather pointless. See as follows:

https://sources.debian.org/src/ifupdown/0.8.44/debian/if-up.d/resolved/ https://sources.debian.org/src/isc-dhcp/4.4.3-P1-7/debian/resolved/

Remarks:

It seems to me that the need for all these hacky hook scripts has come about because systemd's implementation of resolvconf is too limited to be of use as a drop-in replacement for resolvconf with the various "legacy" networking packages. Rather than

propagating these bespoke workarounds into different packages, it may be better if they could all continue to use the "traditional" resolvconf/openresolv system to configure resolved.

I think this could be accomplished fairly cleanly and easily if the systemd-resolved package were to simply omit installing the "resolvconf" symbolic link, and drop the "Provides: resolvconf" and "Conflicts: resolvconf" dependences, allowing it to be

installed alongside the "traditional" resolvconf. At the same time, resolvconf/openresolv could be equipped with a unified hook script to detect systemd-resolved and call resolvectl. Then all the hacks elsewhere could go away!

If anyone can suggest a better forum to raise that wider issue then please let me know and I will do so.

Thanks for this bug report and very thorough analysis.

Can you submit a report upstream and link to this bug?

https://github.com/NetworkConfiguration/dhcpcd/issues/new

Martin-Éric

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Ken,

Thanks for raising this, I've had a hunch things are still a bit dodgy in
this area for a while now.

I'm overall not very happy with Debian's DNS configuration story and I've
been meaning to have a go at improving the situation, but so far I haven't
been able to focus on it.

If you want to have a stab at collaborating on improving the situation
consider me your condiuit. As a DD with a good grasp of Debian's "process"
and politics I can help you make the necessary changes happen.

On Mon, Apr 21, 2025 at 04:12:08PM +0100, Ken Milmore wrote:

With trixie release coming up, I would not expect to see any enthusiasm
for making these sorts of changes across multiple packages.

Consider me, at least, enthused :-).

As long as potential breakage is contained to one or two packages and the
time window is reasonably small I'd consider this a "small, targeted
fix". I don't see this violating the current soft freeze policy.

We can get pre-clearance from release-team in any case, develop in
experimental and upload to unstable when ready. Business as usual.

Hard freeze on 2025-05-15 will make changes more beurocratic and slow, so
if we'd want to have a go at this it'd be best to get something together
before then.

However any DNS breakage easily qualifies for a severity:serious RC bug
which we can fix any time before the release as long as the change diff is reasonably small and we can motivate the change to release team.

--Daniel <3

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEV6G/FbT2+ZuJ7bKf05SBrh55rPcFAmgGbNwACgkQ05SBrh55 rPegUxAAk0XbYUlKB2dxRdY0+mx92aCU5aLTgfV+qgox2lfQQssCZHqqTqCxgiR3 GAEW90aHqvBxdGsfgIAUxdYhISvpwnm2wjnLOpzfTvR4z8Vdq56sJ1LX9BK5O34S UA5O6MHblw5E+bYEsGNtLg5WJ0uq4CmISSPG5cJTIDRl3BQoPmVw+1AQ84iz/4fh 8YITfAqDTzCnCp+twFx3yPC1qq6Cg5o8aApyvyeriH75zkYt08yhrgCVpWKDk9yJ J4pOYiXGqzH8x3SP86CY6OmtB8lzv7kHPzhmWUThak2WUf1yFgK+F1Mjh+67KGkR crOa01/xEOgUkbwiPMuQjyztlPvyps3ad/963iRHLLIRkl+mbzHopL2wuHwLd0I3 wOkHOp98GWZeh2GVRhSoIYytG3SZLv1JkOS0SnY8LsmwNi+1DUlZpfKw9cGozE7c +ghtWc3rgbHtN9BLP78uTj661KnKjki+jBHVEATE1Et8nywxw3lvFCwj+3kNsyQW 8MI3lDgJ2SgDgwBBEtyz1m3YpyRAm/Z2voSHvdRUwWDhzC8WZMUkw12BcKm1Kcjh US6aET6Z58L2+zYJE37BNZDL0kFb4vUlMxrGZ/Zxfrv7HLRklIxNNjrlQ5eAOd4/ vIwZPvwXa2sb2ioMOIf6Wr3eBGE+6ypg8QK5+6XRfm+iXcjEUhs=
=cGdC
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

---- On Sat, 26 Apr 2025 23:27:43 +0100 Ken Milmore <ken.milmore@gmail.com> wrote ---

The issue I raised upstream has now been closed [1], after Roy produced a new release of openresolv [2] with support for systemd-resolved.

Unfortunately, IMHO the direction taken in openresolv doesn't help us much with the present bug, for the following reasons:

- It violates the principle of least surprise (from the POV of network interface configuration), by placing all name servers into the global pool, rather than assigning servers and their routing domains to specific network interfaces.
I am sure there are valid arguments for doing it that way. But it is completely at odds with how resolved is "normally" configured by all the other networking tools such as systemd-networkd, NetworkManager and dhclient.
By contrast, almost anyone installing systemd-resolved on Debian is going to expect per-interface configuration.

Why are people going to expect that?
I have a counter argument - people installing stuff expect it to just work.

resolved is the *only* DNS resolver that can be configured per interface.
DNS by it's nature from configuration is global, but you can still see the interface or source where it's derived by using `resolvconf -l`

The only time it is not is when it's from a VPN and that's a whole new issue because some VPN's such as strongswan sit on top
of existing interfaces and with the current design you the have two programs (say dhclient and strongswan) wanting to update the same interface via resolvectl with competing information.
And by competing dhclient could say "here be public nameservers" and strongswan could say "here be private nameservers for this domain" and resolved currently does not support this.
This is why you need resolvconf to manage this - and by that I mean openresolv or the Debian one.
https://github.com/strongswan/strongswan/discussions/2085
systemd-resolved does not yet support Split Horizon DNS - https://github.com/systemd/systemd/issues/20485
There is a PR to resolve this via the concept of DNS Delegates from poettering - https://github.com/systemd/systemd/pull/34368
Once/if that get's merged then resolved is on feature parity with other resolvers.

Then let's talk about state. If resolved is restarted, it's state learned via resolvectl is lost. How do we get that back?
If it's baked in a config file like other DNS servers it's still there.

Lastly, let's talk about features.
openresolv offers the admin the ability to massage the resolv.conf data before it's applied to the DNS server.
You can append, prefix, remove and change pretty much anything as it comes in. And like Debian's resolvconf you can also use it to just discard all inputs and keep /etc/resolv.conf alone.

Now, none of the above might be important to you, but it's important to others because they've opened many bugs about it.
I didn't just implement these features on a whim, they are asked for.

- In any case, an openresolv-based solution still wouldn't circumvent the package management issues we currently have in Debian, namely that the systemd-resolved package conflicts with anything providing resolvconf.
there's also the problem that we would have to duplicate any new openresolv behaviour into the resolvconf package to keep the two in step.

You could split it out into it's own package, or manage it via alternatives? https://wiki.debian.org/DebianAlternatives

For the above reasons I've given up on the resolvconf approach, decided to eat my earlier words and reconsider the "hacky script" technique once more:

I have ported the resolved configuration script (originally borrowed from Ubuntu) from dhclient [3] to work with dhcpcd.
This turned out to be completely trivial, since the hook script environments provided by dhclient and dhcpcd are so similar.
Basically, it should just work.

See the attached patch.

Functionally speaking, this bespoke script solution has a lot to commend it:

- It configures resolved in the "expected" way, i.e. per interface, the same way as NetworkManager and the rest do.

I find it interesting that NetworkManager does this per interface for resolved and yet just treats resolvconf as a global.

- It integrates with ifupdown settings configured in /etc/network/interfaces:
Specifically, dns-nameservers, dns-search and dns-default-route can optionally be set to manually override what comes from DHCP.

- It is low-impact and follows the solution already in use, so we would hope for little or nothing in the way of unexpected breakage.

So what's not to like? Well I have never been too keen on the implementation details:

- There is code duplication and shared state between the ifupdown, dhclient and (now) dhcpcd packages.
The script needs to be maintained in step between all of these packages, which implies fragility.
Strictly speaking, we could avoid this interdependency, but at the cost of removing the useful ability to override things in /etc/network/interfaces.
I think it would be preferable if the same common script could be used by all of the above packages:
Probably the central place to put it would be inside the ifupdown package. This would not necessarily require a package dependency, if things were handled correctly.

So you realise you're effectively replicating what resolvconf does here per package? This was exactly why resolvconf was born back in the day ....
You're also discarding DNS received via ND. https://github.com/NetworkConfiguration/dhcpcd/blob/master/hooks/20-resolv.conf#L117

- The script modifies some internal resolved state files under /run/systemd/resolve/netif/.
I think the only reason for doing this is in case DNS gets configured before resolved has started up: I would have thought that might be better handled with systemd service dependencies or suchlike?

- There are some other minor things, such as the use of mktemp, which I would have preferred to avoid.

Notwithstanding the above wrinkles, I do think it would be worthwhile getting something along these lines into trixie:
It will allow someone who has installed a base system with ifupdown to simply "apt install systemd-resolved" and still have things work as expected.

If there is any enthusiasm for pursuing this further then I would be prepared to do some work on tweaking the solution to order.
Suggestions welcome...

[1] https://github.com/NetworkConfiguration/dhcpcd/issues/500#issuecomment-2831237831
[2] https://github.com/NetworkConfiguration/openresolv/releases/tag/v3.15.0 [3] https://salsa.debian.org/debian/isc-dhcp/-/commit/ab10fa481ee012c52de95adfc92691c882bf78ac

If you really want to have things pushed via resolvectl then I'm happy to revisit the resolvectl helper for openresolv as I committed here:
https://github.com/NetworkConfiguration/openresolv/commit/a21cc3bd201217e2d4e271c87e1990e03622ba99
And we could work this together to satisfy your needs.

openresolv can ship with both helpers and the admin can choose which one to use.
What do you think?

Roy

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Ken

---- On Mon, 28 Apr 2025 23:20:43 +0100 Ken Milmore <ken.milmore@gmail.com> wrote ---

Thanks for responding Roy.

On 28/04/2025 11:52, Roy Marples wrote:

By contrast, almost anyone installing systemd-resolved on Debian is going to expect per-interface configuration.

Why are people going to expect that?

Because all of the other major networking solutions in Debian do it that way.

resolved isn't a networking solution, it's a DNS resolver - nothing more, nothing less.
It's a baby version of dnsmasq, unbound or named, but specific to systemd.

So, knowing that, which other DNS resolver can be setup per interface.

Try configuring an interface in NetworkManager, networkd or (lately) ifupdown with dhclient and you will see what I mean.
I'm certainly not trying to make a point about whether this is right or wrong, but it is undeniably the "usual" way that systemd-resolved gets configured by everyone else.

Buit isn't this why we are here?
resolvconf and openresolv are configured in the usual way of $interface.$protocol or $interface.$program and have been since before systemd was born and by more than just
dhcpcd or the packages you've listed.

I have a counter argument - people installing stuff expect it to just work.

See above. resolved is completely optional in Debian, as are openresolv/resolvconf.
I expect they each have their own users with possibly different expectations.

There is a PR to resolve this via the concept of DNS Delegates from poettering - https://github.com/systemd/systemd/pull/34368
Once/if that get's merged then resolved is on feature parity with other resolvers.

Presumably this isn't getting into trixie though, and I assume NetworkManager et al will still continue to configure it the "old" way unless and until they are explicitly given support for delegates?

As I understand it, delegates appear like an interface in resolvectl output, so the work alongside the existing setup.
So we could have a ens5.dhcp delegate and ens5.dhcp6 delegate for example if you just wanted passhthrough.

Then let's talk about state. If resolved is restarted, it's state learned via resolvectl is lost. How do we get that back?
If it's baked in a config file like other DNS servers it's still there.

The "hacky patch" handles this by persisting the state under /run/network. As I've been at pains to point out, I'm not much of a fan of the implementation, but it is already in use by ifupdown and dhclient.
Better the devil you know I suppose.

Hmmm, if that's the case, could be just not use resovlectl and just write to /run/network and then send SIGHUP to re-read it's config?

Because of the interop with ifupdown, it does have the advantage that you can override things manually from /etc/network/interfaces.
I found that particularly useful for controlling the "default route" settings in resolved, which can force it to be more pedantic about prefix matching.

So at least we agree that we both like something in the middle which can adjust things before they get into systemd :)

Lastly, let's talk about features.
openresolv offers the admin the ability to massage the resolv.conf data before it's applied to the DNS server.
You can append, prefix, remove and change pretty much anything as it comes in.
And like Debian's resolvconf you can also use it to just discard all inputs and keep /etc/resolv.conf alone.

Yes, but I assume there would be considerable pushback against making either package mandatory in Debian.

I don't think anyone is trying to make things mandatory here.

At the moment, the systemd-resolved package has a hard conflict with both openresolv and resolvconf so they can never be installed together.
I have raised a feature request to get this changed, but...

If that restriction were relaxed then there are various subtleties that would need to be taken care of to avoid breaking more things:
Mostly regarding the ownership and function of /usr/sbin/resolvconf and /etc/resolv.conf with various different combinations of packages installed.
And since we have *two* different resolvconf packages, any behaviour would have to be replicated in *both* of them.

No, I'm sure we don't need to replicate in both of them.
Just document that as of now, resolvconf does not support resolved but openresolv does.

And this might be a silly question, but have you considered putting your work into a script for Debian's resolvconf?

Functionally speaking, this bespoke script solution has a lot to commend it:

- It configures resolved in the "expected" way, i.e. per interface, the same way as NetworkManager and the rest do.

I find it interesting that NetworkManager does this per interface for resolved and yet just treats resolvconf as a global.

Yes I noticed that too! It's a case of "when in Rome" I guess.

It is not uncommon in Debian desktop installs to use both ifupdown and NetworkManager at the same time, on different interfaces:
The former is most often used for wired interfaces and the latter for wireless networks.
This was another reason why I thought it was better to go per-interface with resolved, because it keeps things consistent in this sort of mixed setup.

If you really want to have things pushed via resolvectl then I'm happy to revisit the resolvectl helper for openresolv as I committed here:
https://github.com/NetworkConfiguration/openresolv/commit/a21cc3bd201217e2d4e271c87e1990e03622ba99
And we could work this together to satisfy your needs.

At the moment, I think the biggest hurdle we face is those package conflicts: dhcpcd can coexist with resolved, but openresolv cannot.
As I've pointed out above, removing that restriction would create other problems for us to solve.

Well, the packaging is purely a Debian concen. Please understand that all my thoughts and arguments take other OS's into account.
I'm a NetBSD person for example.

Anyway, I don't see removing the restriction as creating other problems to solve at all.
Not all programs are equal - for example bmake and GNU make which are both in Debian only work with POSIX Makefiles and yet many are just GNU.
What's the expectation here? Understand this and understand why I don't see any problems here with removing that restriction.

For now, doing anything that calls resolvconf in the presence of systemd-resolved is likely to remain broken. That's why I raised this bug! :-)

And it's a good bug to raise!
But really it's against the wrong package ;)

Roy

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 26 Apr 2025 23:27:43 +0100 Ken Milmore <ken.milmore@gmail.com> wrote:

The issue I raised upstream has now been closed [1], after Roy produced a new release of openresolv [2] with support for systemd-resolved.

Unfortunately, IMHO the direction taken in openresolv doesn't help us much with the present bug, for the following reasons:

- It violates the principle of least surprise (from the POV of network interface configuration), by placing all name servers into the global pool, rather than assigning servers and their routing domains to specific network interfaces.
I am sure there are valid arguments for doing it that way. But it is completely at odds with how resolved is "normally" configured by all the other networking tools such as systemd-networkd, NetworkManager and dhclient.
By contrast, almost anyone installing systemd-resolved on Debian is going to expect per-interface configuration.

So openresolv-3.16.4 will allow two approaches.
The limited per interface approach via resolvectl or the write to global and dns delegate files like any other resolver approach.
Pick your poison.
Both ways offer more resolvconf support than what systemd's resolvconf shim offers as well as anything discussed thus far and support the systemd notion of marking an entries domains as both non searchable and non routeable.

As far as trixie is concerned, both have limitations - resolvectl approach only works per interface, systemd-resolved approach relies on DNS delegate support which isn't in any systemd release yet.
Also, the resolvectl approach is slow because of the number of times we call resolvectl. A minor point, but worth noting.
strongswan - a debian package - will only work with the systemd-resolved approach as it doesn't have an interface.

For the above reasons I've given up on the resolvconf approach, decided to eat my earlier words and reconsider the "hacky script" technique once more:

I have ported the resolved configuration script (originally borrowed from Ubuntu) from dhclient [3] to work with dhcpcd.
This turned out to be completely trivial, since the hook script environments provided by dhclient and dhcpcd are so similar.
Basically, it should just work.

See the attached patch.

[3] https://salsa.debian.org/debian/isc-dhcp/-/commit/ab10fa481ee012c52de95adfc92691c882bf78ac

I looked at this and have some issues

1) /run/systemd/resolve/netif is stated as internal private data and not part of the API
as such any long term hack around this is a Bad Idea Which Could Break on Upgrade
2) While you can write to it and it magically appears in resolvectl after a SIGHUP, any futher action on
this file is futile - you have to go via resolvectl. At least this is true on Fedora 41.
3) Each time you add a new hack into /run/network you need to change all the hacks

But at the end, I will say this - systemd's resolvconf shim
1) Embraces - resolvconf is a known and recognised standard of managing DNS setup from various sources (iface.prog)
2) Extends - it can configure systemd-resolved!
3) Extinguish
- it breaks the orignal design goal of resolvconf which was to make managing different sources of DNS setup easier (rejects iface.prog, just allows iface)
- only supports systemd-resolved

Point 3 violates the Principle Of Least Astonishment from the POV of the system administrator who has used Debian for a long time.

Roy

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)