The systemd docs (https://manpages.debian.org/unstable/systemd/systemd.generator.7.en.html) indicate that generators are executed synchronously during the boot
process and should thus take care to not "delay the entire boot if they
are slow". However, there's no mention of a specific runtime
requirement. Is the deadline dynamic in some way?
"systemd.log_level=debug systemd.log_target=console" on the kernel
command line gives some more insight. But this is a lot of output, so requires serial console output.
Yep. One of my earlier updates contains the full debug output from
systemd. Interestingly, it doesn't seem like systemd logs anything at
all when it sends a SIGTERM (at least to generators).
You mentioned on IRC that you've observed this on the generic images,
and I've now observed the same thing. Note that it doesn't occur only
on first boot; it appears that it can happen on any reboot (or
presumably on daemon-reload, as generators are executed then as well).
One potentially interesting entry from the debug logs is:
[ 3.398752] boot-failure1 (sd-gens)[271]: (sd-exec-strv) failed with exit status 1.
Yep. One of my earlier updates contains the full debug output from
systemd. Interestingly, it doesn't seem like systemd logs anything at
all when it sends a SIGTERM (at least to generators).
That mail did not make it to the list. I also don't see any code path
that would do explicit SIGTERM.
You mentioned on IRC that you've observed this on the generic images,
and I've now observed the same thing. Note that it doesn't occur only
on first boot; it appears that it can happen on any reboot (or
presumably on daemon-reload, as generators are executed then as well).
One potentially interesting entry from the debug logs is:
[ 3.398752] boot-failure1 (sd-gens)[271]: (sd-exec-strv) failed with exit status 1.
Yes. This code bails out on all and every error with exit(1).
Okay, found it:
wait_for_terminate_and_check:
| log_full(prio, "%s terminated by signal %s.", strna(name), signal_to_string(status.si_status));
| return -EPROTO;
do_execute:
| r = wait_for_terminate_and_check(…);
| if (r < 0) return r;
execute_strv:
| r = do_execute(…)
| _exit(r < 0 ? EXIT_FAILURE : r);
So the SIGABRT of netplan short circuits the error handling and kills
off all other generators.
On Thu, Mar 13, 2025 at 07:23:24PM +0100, Bastian Blank wrote:
You mentioned on IRC that you've observed this on the generic images,
and I've now observed the same thing. Note that it doesn't occur only
on first boot; it appears that it can happen on any reboot (or
presumably on daemon-reload, as generators are executed then as well).
One potentially interesting entry from the debug logs is:
[ 3.398752] boot-failure1 (sd-gens)[271]: (sd-exec-strv) failed with exit status 1.
Yes. This code bails out on all and every error with exit(1).
Okay, found it:
wait_for_terminate_and_check:
| log_full(prio, "%s terminated by signal %s.", strna(name), signal_to_string(status.si_status));
| return -EPROTO;
do_execute:
| r = wait_for_terminate_and_check(…);
| if (r < 0) return r;
execute_strv:
| r = do_execute(…)
| _exit(r < 0 ? EXIT_FAILURE : r);
So the SIGABRT of netplan short circuits the error handling and kills
off all other generators.
Yep, that's basically the path I was on, too. That explains the
intermittent nature of the failures, too, since it's possible for the
ABRT to happen after some generators have already completed
successfully.
I wonder what's going on with netplan-generator...
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 481 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 14:27:14 |
| Calls: | 9,540 |
| Calls today: | 8 |
| Files: | 13,653 |
| Messages: | 6,139,621 |
| Posted today: | 1 |