On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
Bug: https://bugs.gentoo.org/570534
Signed-off-by: Sam James <sam@gentoo.org>
---
eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
<div class="">On 4 Jan 2022, at 22:58, Sam James <<a href="mailto:sam@gentoo.org" class="">sam@gentoo.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high forthe<br class="">amount of RAM available (uses amount declared as needed<br class="">in the ebuild). Typically should be ~2GB per job.<br class=""><br class="">Bug: <a href="https://bugs.gentoo.org/570534" class="">https://bugs.gentoo.org/570534</a><br
On Tue, 04 Jan 2022, Sam James wrote:
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
+# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
+# to lower memory usage.
+: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for theOn Tue, 04 Jan 2022, Sam James wrote:
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
Where does this number 2 GB come from? The amount of RAM strongly
depends on the programming language and other factors, so I don't
believe that there's one number that can be used for everything.
+# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
+# to lower memory usage.
+: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
If anything, the feature should be opt-in rather than opt-out.
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
Bug: https://bugs.gentoo.org/570534
Signed-off-by: Sam James <sam@gentoo.org>
---
eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
index 2130e2e349141..8c4adc8b4f121 100644
--- a/eclass/check-reqs.eclass
+++ b/eclass/check-reqs.eclass
@@ -43,6 +43,8 @@ case ${EAPI} in
*) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
esac
+inherit multiprocessing
+
EXPORT_FUNCTIONS pkg_pretend pkg_setup
if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
@@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
# @DESCRIPTION:
# How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M
+# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
+# to lower memory usage.
+: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
+
# @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
# @DEFAULT_UNSET
# @DESCRIPTION:
@@ -346,9 +355,36 @@ _check-reqs_memory() {
eend 0
else
eend 1
- _check-reqs_unsatisfied \
- ${size} \
- "RAM"
+
+ # Has the user allowed us to mangle their MAKEOPTS?
+ if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} == "yes" ]] ; then
+ local jobs=$(makeopts_jobs)
+
+ local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes 1G)))
+ if [[ $((jobs*2)) -gt ${estimated_max_memory} ]] ; then
+ # Number of jobs exceeds RAM/2GB, so clamp it.
+ local new_jobs=$(($(_check-reqs_get_number ${estimated_max_memory}G)*10/20))
+
+ # This might _still_ be too big on small machines. Give up in such cases.
+ # (Users can still set the do nothing variable which is independent of this.)
+ if [[ $((new_jobs*2)) -gt ${estimated_max_memory} ]] ; then
+ _check-reqs_unsatisfied \
+ ${size} \
+ "RAM"
+ else
+ # The clamped jobs seem to be enough to satisfy the check-reqs requirement from the ebuild.
+ ewarn "Clamping MAKEOPTS jobs to -j${new_jobs} to reduce memory usage"
+ ewarn "Compiler jobs may use around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS"
+ ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no."
+
+ MAKEOPTS+=" -j${new_jobs}"
+ fi
+ fi
+ else
+ _check-reqs_unsatisfied \
+ ${size} \
+ "RAM"
+ fi
fi
else
eend 1
On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
Bug: https://bugs.gentoo.org/570534
Signed-off-by: Sam James <sam@gentoo.org>
---
eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
Note that we discussed this on GitHub a bit when I just posted it there
for some rough feedback: https://github.com/gentoo/gentoo/pull/23311.
I think this is valuable for reducing invalid bug reports from OOM and
easing user experience.
Still kind of a WIP/rough draft, but may be ready in this state. Need
more testing, so not planning on pushing yet or anything.
On Wed, 05 Jan 2022, Florian Schmaus wrote:
On 05/01/2022 09.28, Ulrich Mueller wrote:
Where does this number 2 GB come from? The amount of RAM stronglyCrank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for theOn Tue, 04 Jan 2022, Sam James wrote:
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
depends on the programming language and other factors, so I don't
believe that there's one number that can be used for everything.
Surely not, but 2 GiB seems like a good start. I guess it could become
an eclass variable that ebuilds could modify later (if the need
emerges).
It appears to me that the motivation for this change is to prevent
users from running into OOM situations when emerging a package. And I
believe that many users, especially novices, are not directly able to determine that portage failed due to OOM, simply because there is no
direct hint in the build log. Hence opt-out appears to be sensible to
me here.
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
Bug: https://bugs.gentoo.org/570534
Signed-off-by: Sam James <sam@gentoo.org>
---
eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
index 2130e2e349141..8c4adc8b4f121 100644
--- a/eclass/check-reqs.eclass
+++ b/eclass/check-reqs.eclass
@@ -43,6 +43,8 @@ case ${EAPI} in
*) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
esac
+inherit multiprocessing
+
EXPORT_FUNCTIONS pkg_pretend pkg_setup
if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
@@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
# @DESCRIPTION:
# How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M
+# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Allow packages to reduce the number of multiprocessing (e.g. make,
ninja) jobs
+# to lower memory usage.
+: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
+
# @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
# @DEFAULT_UNSET
# @DESCRIPTION:
@@ -346,9 +355,36 @@ _check-reqs_memory() {
eend 0
else
eend 1
- _check-reqs_unsatisfied \
- ${size} \
- "RAM"
+
+ # Has the user allowed us to mangle their
MAKEOPTS?
+ if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} ==
"yes" ]] ; then
+ local jobs=$(makeopts_jobs)
+
+ local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes
1G)))
+ if [[ $((jobs*2)) -gt ${estimated_max_memory} ]] ; then
+ # Number of jobs exceeds RAM/2GB,
so clamp it.
+ local new_jobs=$(($(_check-reqs_get_number
${estimated_max_memory}G)*10/20))
+
+ # This might _still_ be too big
on small machines. Give up in such cases.
+ # (Users can still set the do
nothing variable which is independent of this.)
+ if [[ $((new_jobs*2)) -gt ${estimated_max_memory}
]] ; then
+ _check-reqs_unsatisfied \
+ ${size} \
+ "RAM"
+ else
+ # The clamped jobs seem to be enough to satisfy the check-reqs requirement from the ebuild.
+ ewarn "Clamping MAKEOPTS jobs to -j${new_jobs} to reduce memory usage"
+ ewarn "Compiler jobs may use around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS"
+ ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no."
+
+ MAKEOPTS+=" -j${new_jobs}"
+ fi
+ fi
+ else
+ _check-reqs_unsatisfied \
+ ${size} \
+ "RAM"
+ fi
fi
else
eend 1
--
2.34.1
On Wed, 05 Jan 2022, Florian Schmaus wrote:
On 05/01/2022 09.28, Ulrich Mueller wrote:
Where does this number 2 GB come from? The amount of RAM stronglyCrank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for theOn Tue, 04 Jan 2022, Sam James wrote:
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
depends on the programming language and other factors, so I don't
believe that there's one number that can be used for everything.
Surely not, but 2 GiB seems like a good start. I guess it could become
an eclass variable that ebuilds could modify later (if the need
emerges).
We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
to introduce another one that is counting memory per job? I doubt that
would be possible, because the amount of memory greatly varies within a single build. For example, it is different for compiler vs linker vs
building the documentation.
It appears to me that the motivation for this change is to prevent
users from running into OOM situations when emerging a package. And I believe that many users, especially novices, are not directly able to determine that portage failed due to OOM, simply because there is no
direct hint in the build log. Hence opt-out appears to be sensible to
me here.
That applies to all parallel builds though, not only to ebuilds
inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
telling the user that the --jobs setting in their make.conf is wrong,
in the first place.
On Wed, 05 Jan 2022, Florian Schmaus wrote:
On 05/01/2022 09.28, Ulrich Mueller wrote:
Where does this number 2 GB come from? The amount of RAM stronglyCrank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for theOn Tue, 04 Jan 2022, Sam James wrote:
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
depends on the programming language and other factors, so I don't
believe that there's one number that can be used for everything.
Surely not, but 2 GiB seems like a good start. I guess it could become
an eclass variable that ebuilds could modify later (if the need
emerges).
We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
to introduce another one that is counting memory per job? I doubt that
would be possible, because the amount of memory greatly varies within a single build. For example, it is different for compiler vs linker vs
building the documentation.
It appears to me that the motivation for this change is to prevent
users from running into OOM situations when emerging a package. And I
believe that many users, especially novices, are not directly able to
determine that portage failed due to OOM, simply because there is no
direct hint in the build log. Hence opt-out appears to be sensible to
me here.
That applies to all parallel builds though, not only to ebuilds
inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
telling the user that the --jobs setting in their make.conf is wrong,
in the first place.
On Wed, 05 Jan 2022, Florian Schmaus wrote:
That applies to all parallel builds though, not only to ebuilds
inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
telling the user that the --jobs setting in their make.conf is wrong,
in the first place.
Yes, exactly. And it is a bandaid solution. But I believe that it will
do more good than evil. And it is probably only used until portage is
able to report to the user that the emerge failed due to OOM (which I
believe to be non-trivial to implement, but I am happy to be proven otherwise).
On 5 Jan 2022, at 17:51, Alec Warner <antarus@gentoo.org> wrote:
On Tue, Jan 4, 2022 at 3:03 PM Sam James <sam@gentoo.org> wrote:
On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
[snip]
I'm still not sure I grasp why we cannot make OOMs easier to discover
from portage.
Most packages don't even use check-reqs, so your solution is very
narrow (and I get why, because you get a lot of bug reports from the
big packages that do use it.)
Can we write a build log analyzer?
-A
PS: If this was a global change I'd downvote it. It's only for
check-reqs though and most packages don't use check-reqs; I don't
really care. I'd be concerned about adopting this kind of approach
wider; its very much a bandaid.
On 5 Jan 2022, at 19:53, Ulrich Mueller <ulm@gentoo.org> wrote:
On Wed, 05 Jan 2022, Florian Schmaus wrote:
That applies to all parallel builds though, not only to ebuilds
inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
telling the user that the --jobs setting in their make.conf is wrong,
in the first place.
Yes, exactly. And it is a bandaid solution. But I believe that it will
do more good than evil. And it is probably only used until portage is
able to report to the user that the emerge failed due to OOM (which I
believe to be non-trivial to implement, but I am happy to be proven
otherwise).
Obviously I disagree. Tweaking the value for a subset of packages isn't
a solution to the problem.
MAKEOPTS applies to all parallel builds, and users should set it to a
value suitable for their system (i.e. number of CPUs, available memory, etc.). Maybe our documentation needs to be improved? I see that make.conf.example says "The suggested number for parallel makes is
CPUs+1" which may not be the best possible advice.
On Tue, 04 Jan 2022, Sam James wrote:
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.
Where does this number 2 GB come from? The amount of RAM strongly
depends on the programming language and other factors, so I don't
believe that there's one number that can be used for everything.
(If only considering C and C++ 2 GB seems to be excessive, i.e. it
will limit parallel build more than necessary. When linking, the
number
may be _much_ larger.)
Also not sure if I understand the arithmetic. Shouldn't it use CHECKREQS_MEMORY as the basis of the calculation?
+# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Allow packages to reduce the number of multiprocessing (e.g.
make, ninja) jobs
+# to lower memory usage.
+: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
If anything, the feature should be opt-in rather than opt-out.
Ulrich
On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>wrote:
Sam,
Do users with FEATURES=distcc still have to opt out of this
MAKEOPTS clamping?
Great point! I think we could add an exemption for that and make it a
noop or warning-only.
Best,
sam
On Wed, 05 Jan 2022, Sam James wrote:
On 5 Jan 2022, at 08:28, Ulrich Mueller <ulm@gentoo.org> wrote:
Where does this number 2 GB come from? The amount of RAM strongly
depends on the programming language and other factors, so I don't
believe that there's one number that can be used for everything.
(If only considering C and C++ 2 GB seems to be excessive, i.e. it
will limit parallel build more than necessary. When linking, the number
may be _much_ larger.)
This is essentially "common law" (or "common lore" if you like!) and
is well-accepted as a good rule of thumb.
The number being larger doesn't really make any difference here;
the point is to help in cases where we're pretty sure there's going
to be a problem (only users of check-reqs.eclass, and we can
Introduce a variable to give ~RAM per job too if needed).
This is also about reducing the number of support queries about
builds which failed for trivial reasons, as someone who has to handle
quite a lot of those (until such a time as we can implement better
detection, but this is also a generally nice UX improvement -- as
per what Florian/flow said).
On 2022.01.05 20:22, Sam James wrote:
On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>wrote:
Sam,Great point! I think we could add an exemption for that and make it a
Do users with FEATURES=distcc still have to opt out of this
MAKEOPTS clamping?
noop or warning-only.
Best,
sam
Sam,
You are building a better mousetrap here. That's not a reason to try.
Do users of I_KNOW_WHAT_I_AM_DOING, who have already
opted to shoot themselves in both feet, get a free pass here?
There are users who run emerge --jobs=X with MAKEOPTS='-jY"
and get firefox, thunderbird and libreoffice all building concurrently
as they allow X * Y MAKE threads, reduced by this proposed
throttling, still triggering the OOM.
I don't think you can head that off beforehand.
On 2022.01.05 20:22, Sam James wrote:
On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>wrote:
Sam,
Do users with FEATURES=distcc still have to opt out of this
MAKEOPTS clamping?
Great point! I think we could add an exemption for that and make it
a
noop or warning-only.
Best,
sam
Sam,
You are building a better mousetrap here. That's not a reason to try.
Do users of I_KNOW_WHAT_I_AM_DOING, who have already
opted to shoot themselves in both feet, get a free pass here?Â
There are users who run emerge --jobs=X with MAKEOPTS='-jY"
and get firefox, thunderbird and libreoffice all building concurrently
as they allow X * Y MAKE threads, reduced by this proposed
throttling, still triggering the OOM.
I don't think you can head that off beforehand.
On Wed, 2022-01-05 at 21:06 +0000, Roy Bamford wrote:
On 2022.01.05 20:22, Sam James wrote:
On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>wrote:
Sam,
Do users with FEATURES=distcc still have to opt out of this
MAKEOPTS clamping?
itGreat point! I think we could add an exemption for that and make
a
noop or warning-only.
Best,
sam
Sam,
You are building a better mousetrap here. That's not a reason totry.
Do users of I_KNOW_WHAT_I_AM_DOING, who have already
opted to shoot themselves in both feet, get a free pass here?
There are users who run emerge --jobs=X with MAKEOPTS='-jY"concurrently
and get firefox, thunderbird and libreoffice all building
as they allow X * Y MAKE threads, reduced by this proposed
throttling, still triggering the OOM.
I don't think you can head that off beforehand.
What's your proposed alternative?
On 5 Jan 2022, at 19:18, Kai Krakow <kai@kaishome.de> wrote:
Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller <ulm@gentoo.org>:
[...]
That applies to all parallel builds though, not only to ebuilds
inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
telling the user that the --jobs setting in their make.conf is wrong,
in the first place.
Well, I'm using a safe combination of jobs and load-average, maybe the
documentation should be tweaked instead.
I think "safe" is doing some heavy lifting here...
I'm using
[...]
The "--jobs" parameter is mostly a safe-guard against "make" or
"emerge" overshooting the system resources which would happen if
running unconstrained without "--load-average". The latter parameter
OTOH tunes the parallel building processes automatically to the
available resources. If the system starves of memory, thus starts to
swap, load will increase, and make will reduce the jobs. It works
pretty well.
I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
won't starve emerge from running configure phases of parallel ebuilds.
... because it's quite hard for this logic to work correctly enough
of the time without jobserver integration (https://bugs.gentoo.org/692576).
But indeed, I'd say you're not the target audience for this (but I appreciate the input).
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 399 |
| Nodes: | 16 (0 / 16) |
| Uptime: | 29:36:44 |
| Calls: | 8,327 |
| Calls today: | 4 |
| Files: | 13,153 |
| Messages: | 5,890,081 |
| Posted today: | 1 |