package: apparmor
version: 4.1.0~beta5-3
severity: important
x-debbugs-cc: podman@packages.debian.org, pasta@packages.debian.org, golang-github-containers-common@packages.debian.org, tim.miller@hadronindustries.com

Recently I started running into the following error shutting down
containers with podman stop:

* rootless netns: kill network process: permission denied
This error is produced by
golang-github-containers-common/libnetwork/internal/rootlessnetns/netns_linux.go
in the cleanup function:
if err := n.cleanupRootlessNetns(); err != nil {
multiErr = multierror.Append(multiErr, wrapError("kill network process", err))
}

And that function effectively just finds and kills the pasta or
slirp4netns process:
if err == nil {
// kill the slirp/pasta process so we do not leak it
err = unix.Kill(pid, unix.SIGTERM)
if err == unix.ESRCH {
err = nil
}

Looking at my kernel logs, I see that
[ 462.337636] audit: type=1400 audit(1741711021.533:118): apparmor="DENIED" ope
ration="signal" class="signal" profile="pasta" pid=4552 comm="exe" requested_mas
k="receive" denied_mask="receive" signal=term peer="podman"
In other words, apparmor is preventing podman from cleaning up pasta.

Podman is effectively supposed to be unconfined:
Quoting /etc/apparmor.d/podman:
# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"

Although it turns out it's not really true that podman can do anything
because it turns out that the base abstraction specifically special
cases the unconfined profile:
(quoting /etc/apparmor.d/base)

# Allow unconfined processes to send us signals by default
signal (receive) peer=unconfined,

# Allow us to signal ourselves
signal peer=@{profile_name},

# Checking for PID existence is quite common so add it by default for now
signal (receive, send) set=("exists"),


So, in other words, podman could send the signal if it were in the
unconfined profile, but cannot because it has a profile at all.

There's clearly a race condition involved somewhere. Some of my
containers exhibit this behavior, but some do not.
However, once it starts happening, it keeps happening, and it is common
enough that it is breaking a lot of automation.



On the apparmor side, I'd like to either see the bogus/empty podman
profile removed or to see signals from podman permitted by the pasta
profile.
I also think that golang-github-containers-common should treat EACCES
from killing pasta as a warning not an error.

whether

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Sam, Stefano, others,

(almost fully quoting because the Sam's original report Cc'ed pasta@packages.debian.org, while I believe he meant to write to passt@packages.debian.org)

Sam Hartman (2025-03-11):

Recently I started running into the following error shutting down
containers with podman stop:

* rootless netns: kill network process: permission denied
This error is produced by
golang-github-containers-common/libnetwork/internal/rootlessnetns/netns_linux.go
in the cleanup function:
if err := n.cleanupRootlessNetns(); err != nil {
multiErr = multierror.Append(multiErr, wrapError("kill network process", err))
}

And that function effectively just finds and kills the pasta or
slirp4netns process:
if err == nil {
// kill the slirp/pasta process so we do not leak it
err = unix.Kill(pid, unix.SIGTERM)
if err == unix.ESRCH {
err = nil
}

Looking at my kernel logs, I see that
[ 462.337636] audit: type=1400 audit(1741711021.533:118): apparmor="DENIED" ope
ration="signal" class="signal" profile="pasta" pid=4552 comm="exe" requested_mas
k="receive" denied_mask="receive" signal=term peer="podman"
In other words, apparmor is preventing podman from cleaning up pasta.

Podman is effectively supposed to be unconfined:
Quoting /etc/apparmor.d/podman:
# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"

Although it turns out it's not really true that podman can do anything because it turns out that the base abstraction specifically special
cases the unconfined profile:
(quoting /etc/apparmor.d/base)

# Allow unconfined processes to send us signals by default
signal (receive) peer=unconfined,

# Allow us to signal ourselves
signal peer=@{profile_name},

# Checking for PID existence is quite common so add it by default for now
signal (receive, send) set=("exists"),

So, in other words, podman could send the signal if it were in the
unconfined profile, but cannot because it has a profile at all.

Thank you Sam for educating me! I did not remember we had these rules
for unconfined processes.

There's clearly a race condition involved somewhere. Some of my
containers exhibit this behavior, but some do not.
However, once it starts happening, it keeps happening, and it is common enough that it is breaking a lot of automation.

On the apparmor side, I'd like to either see the bogus/empty podman
profile removed or to see signals from podman permitted by the pasta profile.

Thank you Sam for having gone to great lengths to investigate the
problem you were experiencing, up to the point where you could propose
these solutions.

I agree we should do 1 of those.

I would prefer we modify the pasta profile to allow signals
from podman, for 2 reasons:

- It'll be necessary on Ubuntu, where removing the podman profile is
not an option. It's not needed *yet* solely because the profile is
not included in the Ubuntu package, which I'm guessing is a mistake
that will be fixed at some point
(https://bugs.launchpad.net/ubuntu/+source/passt/+bug/2077158).
So we can as well fix this proactively. And the fix should probably
be upstreamed.

- It's 1 tiny but still useful step towards being able to some day
stop accepting signals from unconfined processes.

Stefano, what do you think?

I believe this (untested) rule should do the job:

signal (receive) peer=podman,

If we don't do that, then I'm fine with removing the podman profile,
which has limited value anyway in the context of Debian.

Cheers,
--
intrigeri

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Control: reassign -1 passt

Hi,

Stefano Brivio (2025-03-12):

On Wed, 12 Mar 2025 14:41:14 +0100
intrigeri <intrigeri@debian.org> wrote:
Thanks for fixing the address, yes, I didn't get the original report.

Thanks for the quick reply!

- It'll be necessary on Ubuntu, where removing the podman profile is
not an option. It's not needed *yet* solely because the profile is
not included in the Ubuntu package, which I'm guessing is a mistake
that will be fixed at some point
(https://bugs.launchpad.net/ubuntu/+source/passt/+bug/2077158).
So we can as well fix this proactively. And the fix should probably
be upstreamed.

I'm not sure what fix you mean here, but Launchpad #2077158 is already
fixed on Debian, and there's no further fix needed upstream.

OK, so Ubuntu is already affected by the Debian bug we're
discussing here.

(I haven't checked the current status in Ubuntu and I was blindly
trusting the status encoded in the Launchpad bug. I see current Ubuntu
Plucky now has the same passt version as current Debian testing/sid so
I suppose the Launchpad bug could be marked as fixed in that version.
I've left a comment on LP about this.)

If we don't do that, then I'm fine with removing the podman profile,
which has limited value anyway in the context of Debian.

Well, eventually, it would make sense to have an actual profile, I
guess.

Anyway, let me know. If somebody is willing to add to change Podman's
profile in the way I mentioned (I can also submit a merge request
eventually, but that will be in a while), I'd prefer that, but I can
also just add a rule in pasta's profile for the moment.

Developing a real, enforcing AppArmor profile for podman would
be great!

That said, we're getting close to the freeze for Debian 13 (Trixie) so
to me it feels it's too late to aim for this solution as far as Trixie
is concerned, so please "just add a rule in pasta's profile for the
moment".

I'm reassigning this bug accordingly.

Cheers,
--
intrigeri

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Stefano Brivio (2025-03-13):

Actually, if you need something quick, you don't really need a
complete/real profile for Podman. You can just add to the current stub (untested, but I'm fairly confident):

Thank you for proposing more options!

Sadly, this ventures too far away from my domain of expertise for me
to take responsibility to include this in the Debian-specific delta of
the AppArmor package, or to propose this change to AppArmor upstream
myself so I can then cherry-pick it into Debian.

So at this stage, as far as Debian Trixie is concerned, I'm now
tempted to simply remove the stub podman profile from the apparmor
package: it seems none of us is super comfortable with the workaround
they would have to carry to make it play nicer with pasta. And we
would not be losing much value for our users.

Does this sound reasonable?

Cheers,
--
intrigeri

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Control: reassign -1 apparmor

Hi,

Stefano Brivio (2025-03-14):

On Thu, 13 Mar 2025 18:18:28 +0100
intrigeri <intrigeri@debian.org> wrote:

So at this stage, as far as Debian Trixie is concerned, I'm now
tempted to simply remove the stub podman profile from the apparmor
package: it seems none of us is super comfortable with the workaround
they would have to carry to make it play nicer with pasta. And we
would not be losing much value for our users.

If we lose zero value (do we? what's the value of the stub?) then I
would go ahead with that, definitely.

The main value of the stub is as a stepping stone for finer-grained confinement, such as what we're discussing on this thread. I think
it's great to take advantage of it eventually but perhaps not at this
time in the dev cycle as far as Debian is concerned, so I'll go ahead
and remove the stub podman profile for now. Happy to bring it back
once other components such as passt are ready to take advantage of it,
even happier if a proper profile is created and maintained instead of
the stub :)

Cheers,
--
intrigeri

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Control: Tag -1 upstream

The full context of this conversation is archived at https://bugs.debian.org/1100135

Sam Hartman <hartmans@debian.org> writes:

package: apparmor
version: 4.1.0~beta5-3
severity: important
x-debbugs-cc: podman@packages.debian.org, pasta@packages.debian.org, golang-github-containers-common@packages.debian.org, tim.miller@hadronindustries.com

Recently I started running into the following error shutting down
containers with podman stop:

* rootless netns: kill network process: permission denied
This error is produced by
golang-github-containers-common/libnetwork/internal/rootlessnetns/netns_linux.go
in the cleanup function:
if err := n.cleanupRootlessNetns(); err != nil {
multiErr = multierror.Append(multiErr, wrapError("kill network process", err))
}

And that function effectively just finds and kills the pasta or
slirp4netns process:
if err == nil {
// kill the slirp/pasta process so we do not leak it
err = unix.Kill(pid, unix.SIGTERM)
if err == unix.ESRCH {
err = nil
}

Sam, I think your analyis makes sense. I see that you have now re-assigned this to
the golang-github-containers-common package, which does contain the code above.

May I ask you to file this bug at https://github.com/containers/common/issues/new?template=BLANK_ISSUE and
tag me and dwalsh to it? I'd make sure that we make the right decision
here. As a heads-up, I've CC'ed Dan and Paul to this email.

Thanks!

-rt

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)